Difference between revisions of "L10n Vendors and Integration"
(→Advanced FTP Vendor issues) |
|||
Line 73: | Line 73: | ||
* Often FTP uses port 22 and SSH. Change this if the configuration differs. |
* Often FTP uses port 22 and SSH. Change this if the configuration differs. |
||
− | ==== How do I create the SSH keys? ==== |
||
− | |||
− | See http://www.linuxproblem.org/art_9.html. |
||
− | |||
− | ==== How do I hide the password? ==== |
||
− | Use a '''.netrc''' file, The .netrc file is located in the $HOME directory. |
||
− | |||
− | Example of a .netrc: |
||
− | |||
− | <pre> |
||
− | machine ftp.company.net |
||
− | login username |
||
− | password myPassword |
||
− | </pre> |
||
− | |||
− | Then <code>ftp ftp.company.net</code> will use the login and username in the .netrc file to connect and the username password are not visible. |
||
− | |||
− | ==== Translation FTP Vendor Integration==== |
||
− | Lingoport pushes zip files to the configured FTP outbound endpoint and consumes zip files sent on the incoming endpoint. The structure of the zip file is defined below: |
||
− | |||
− | The files sent back and forth over FTP have the following characteristics: |
||
− | |||
− | * The data files to be translated or returned from translation are sent in a '''zip file'''. |
||
− | * The '''zip file''' is named according to this convention: '''<code><group-name>.<project-name>.<kit-version>.<locale>.zip</code>'''. For example, a zip file could be named '''<code>Queens.Champions.1.fr_fr.zip</code>''': The group-name is Queens, the project-name is Champions, the kit version is 1, and the target locale is fr_FR. |
||
− | * The zip file consists of '''one top level directory''', <code><group-name>.<project-name>.<kit-version>.<locale></code>, with all the files to be translated under it. |
||
− | * There is '''one zip file per target locale'''. You may target 10 locales for translation for project "Champions" under group "Queens". You then would have 10 zip files to send over FTP. |
||
− | * A zip file may contain '''more than one file and more than one file type'''. |
||
− | * Each file to be translated has a comment at the top of the file which needs to be kept when returning the file from translation vendor. |
||
− | * The '''translated files''' will be named the same as the original files (so they could be _en.properties even if the content is for French) |
||
− | |||
− | Two FTP endpoints are necessary: a ''''to translation'''' end point and a ''''from translation'''' endpoint. |
||
− | |||
− | ==== Example of the Zip File Structure ==== |
||
− | The structure of the zip file for target locale ''de'' is as follows: |
||
− | <pre> |
||
− | Acme.frontend.1.de.zip |
||
− | | -- Acme.frontend.1.de |
||
− | | -- resources.properties |
||
− | | -- messages.resx |
||
− | | -- errors.json |
||
− | </pre> |
||
− | |||
− | This is the structure of the zip file sent for translation on the FTP outbound endpoint and also the structure and name of the file received from translation on the FTP inbound endpoint. |
||
− | |||
− | Notes: |
||
− | * The names of the files are the base resource file names |
||
− | * The names of the files (resources.properties) must be the same exact name when the translation is returned as it was in the zip file sent for translation. |
||
− | * The files in one prep kit must all be returned in the translated zip file with a complete translation. If prep kit #1 had three files, the same three files must be returned in, say, German with every key/value pair returned. |
||
=== Lingotek Vendor === |
=== Lingotek Vendor === |
Revision as of 23:23, 19 December 2019
Contents
- 1 What are L10n Vendors?
- 2 Supported L10n Vendors
- 3 Recommended Approach
- 4 How do I set the transmission protocol?
- 5 InContext Server Configuration
- 6 Advanced FTP Vendor issues
- 6.1 How do I create the SSH keys?
- 6.2 How do I hide the password?
- 6.3 Translation FTP Vendor Integration
- 6.4 Example of the Zip File Structure
- 6.5 How do I setup a proxy for a project's FTP server?
- 6.6 Encoding of translated files
- 6.7 I configured a vanilla FTP (not SSH or SSL) and I get an SSH error
- 7 Other Integrations
What are L10n Vendors?
LRM packages the resources files and send them to localization service providers and translation management systems to translate the files. There are many ways to send the files and they can be done a number of different ways that depend on a different factors including the translation management system.
Supported L10n Vendors
To get a current list of LSP (localization service provider) and TMS (translation management system) companies that are integrated with LRM, please email sales@lingoport.com.
Recommended Approach
Lingoport FTP Protocol
Lingoport strongly recommends using the FTP protocol. With that approach,
- Lingoport automatically pushes valid resource files to be translated following a specific naming convention to an FTP end point
- The translation group picks up the files and on-boards them in the TMS following their own automated pattern
- The translation group returns the translated resource files to another FTP end point as per their system
- LRM picks up the translated files, validates them, and imports them into the proper repository
Advantages
- Robustness: FTP has been in place for decades and is one of the most reliable protocols for transmitting files from one system to another
- Simplicity: FTP and zip are well known and are well understood in the industry
- Security: FTP support SSH and SSL encryption; The FTP system can allow only some IP ranges to access the FTP port(s)
- Automation: LRM pushes and retrieves files automatically; The translation group supplies the automation to and from the automation end point
- Network Failure Handling: If the network goes down when accessing FTP, files are resent later when the network is up; Files are not lost due to downtime
- Asynchronicity: The LRM system and the TMS do not need any special connection; Files can be sent and received independently
- Scalability: The volume sent to an FTP system does not slow down either LRM or a TMS due to traffic. Each system can push or retrieve files without slowing down the other. FTP supports large volume of data
- Configurability: Configuring the FTP end point location is something well understood by IT departments and simple to configure (for LRM, see below)
- Testability: Checking the proper configuration is straightforward
- Fast Set Up: Given the above, setting up LRM with FTP is the fastest and simplest route
Note: Lingoport has set up LRM over FTP with many Translation Groups and the TMS of their choice and any TMS vendor can add their system to this integration pattern.
Resource File Transmission Protocols
For most use cases, Lingoport recommends sending files to translation over SFTP. Direct API-based connections are also available to some TMS systems.
Overall, LRM supports several localization paths for sending files to be translated. They are:
Typical:
- Local
- Files are kept on the system LRM resides on. Good for debugging.
- FTP
- FTP/SFTP. Typical method of sending resource files to/from translation.
API Based:
- GlobalLink
- Lingotek
- WorldServer
- Memsource
- XTM
How do I set the transmission protocol?
Setting the transmission protocol has changed significantly over the releases. Setting the protocol is now done through Jenkins configuration. In Jenkins, select Manage Jenkins in the left menu and then Configure System. Scroll down to LRM L10n Vendor Setup.
Transmission protocols can be set to either Group or Project. If the protocol is set to Group, then all the projects in that group will use the same protocol and no other configuration is needed. If the protocol is set to Project then individual projects can test a different protocol than the rest in the group. When configuring a project, the L10n vendor name needs to be selected. Here is what it looks like when configuring and on boarding a new job:
Local Vendor
A local vendor uses the current system for the incoming and outgoing files.
- Set the name of the vendor. This can be anything decriptive that can be selected in the automation jobs.
- Set the names of the incoming and outgoing file locations
FTP Vendor
- Set the name of the vendor. This can be anything decriptive that can be selected in the automation jobs.
- Set the hostname for uploading files and for downloading files and the respective file location, username, and password. Test the Connection to make sure that the system can communicate with the host.
- Often FTP uses port 22 and SSH. Change this if the configuration differs.
Lingotek Vendor
- Set the name of the vendor. This can be anything decriptive that can be selected in the automation jobs.
- Set the Lingotek credentials. If these are unknown, contact Lingotek or support@lingoport.com.
- The Jenkins URL will be something like: https://<IP address>/jenkins. Probably of the current system.
WorldServer Vendor
- Set the name of the vendor. This can be anything decriptive that can be selected in the automation jobs.
- Set the credentials for WorldServer. Contact WorldServer for these values.
- Set the host, port, location and username and password for the files to download and import.
Historically, the type of L10n Vendor was defined in the config_l10n_vendor.properties
file that can exist at either the group or project level. However, since it is now part of the Jenkins configuration, this is obsolete. Manual changes to this file may be overwritten by the Jenkins configuration.
The default location of the config_l10n_vendor.properties file is at the group level in /var/lib/jenkins/Lingoport_Data/L10nStreamlining/<group-name>/config
.
InContext Server Configuration
In order for a L10n Vendor to be able to communicate to the Lingoport InContext Server, there must be a valid LRM LQA license for the group as well as a valid configuration for the InContext Server. See InContext License for more information. The InContext Server configuration is located in the L10n Vendor configuration file with the following keys.
l10n.vendor.incontext.server.url=<valid http or https URL pointing to the InContext Server > l10n.vendor.incontext.auth.token=<valid authorization token>
The authorization token will be supplied by your InContext Server administrator. During the prep kit process, the configured URL and token will be used to access the InContext Server. If these are not valid then an error will occur and the prep kit process will fail. If the defined InContext server is valid then all the resource files will be instrumented and the prep kit files will contain the InContext decoration.
InContext Decoration
When prepping a kit where there is a valid InContext server configuration, each of the keys that need to be translated will have an InContext tag above the key. The default tag prefix is LRM_INCONTEXT. Your L10n vendor may have a specific tag that is used for filtering. If this is the case, then a unique InContext tag can be configured in the L10n Vendor configuration file using the following key:
l10n.vendor.incontext.unique.tag=
If this key is missing or has a blank value then the default, LRM_INCONTEXT, tag will be used.
InContext decoration examples
- L10n Vendor Configuration
l10n.vendor.incontext.server.url=http://ec5-4-3-2-1.amazonaws.com:8081 l10n.vendor.incontext.auth.token=0qhZ9yTvtW5
- XML type prep kit file
<?xml version="1.0" encoding="UTF-8"?> <!--MyProject_8_6 - CHANGES-ONLY--> <resources> <!--LRM_INCONTEXT http://ec5-4-3-2-1.amazonaws.com:8081/incontext-server/lookup?key=3130_192&token=0qhZ9yTvtW5--> <string name="Hello">Hello</string>
- Properties type prep kit file
#MyProject_1_4 - CHANGES-ONLY #LRM_INCONTEXT http://ec5-4-3-2-1.amazonaws.com:8081/incontext-server/lookup?key=422_238&token=0qhZ9yTvtW5 myString=Hello
InContext decoration examples using unique InContext tag
- L10n Vendor Configuration
l10n.vendor.incontext.server.url=http://ec5-4-3-2-1.amazonaws.com:8081 l10n.vendor.incontext.auth.token=0qhZ9yTvtW5 l10n.vendor.incontext.unique.tag=MY_SPECIAL_TAG
- XML type prep kit file
<?xml version="1.0" encoding="UTF-8"?> <!--MyProject_8_6 - CHANGES-ONLY--> <resources> <!--MY_SPECIAL_TAG http://ec5-4-3-2-1.amazonaws.com:8081/incontext-server/lookup?key=3130_192&token=0qhZ9yTvtW5--> <string name="Hello">Hello</string>
- Properties type prep kit file
#MyProject_1_4 - CHANGES-ONLY #MY_SPECIAL_TAG http://ec5-4-3-2-1.amazonaws.com:8081/incontext-server/lookup?key=422_238&token=0qhZ9yTvtW5 myString=Hello
Advanced FTP Vendor issues
How do I create the SSH keys?
See http://www.linuxproblem.org/art_9.html.
How do I hide the password?
Use a .netrc file, The .netrc file is located in the $HOME directory.
Example of a .netrc:
machine ftp.company.net login username password myPassword
Then ftp ftp.company.net
will use the login and username in the .netrc file to connect and the username password are not visible.
Translation FTP Vendor Integration
Lingoport pushes zip files to the configured FTP outbound endpoint and consumes zip files sent on the incoming endpoint. The structure of the zip file is defined below:
The files sent back and forth over FTP have the following characteristics:
- The data files to be translated or returned from translation are sent in a zip file.
- The zip file is named according to this convention:
<group-name>.<project-name>.<kit-version>.<locale>.zip
. For example, a zip file could be namedQueens.Champions.1.fr_fr.zip
: The group-name is Queens, the project-name is Champions, the kit version is 1, and the target locale is fr_FR. - The zip file consists of one top level directory,
<group-name>.<project-name>.<kit-version>.<locale>
, with all the files to be translated under it. - There is one zip file per target locale. You may target 10 locales for translation for project "Champions" under group "Queens". You then would have 10 zip files to send over FTP.
- A zip file may contain more than one file and more than one file type.
- Each file to be translated has a comment at the top of the file which needs to be kept when returning the file from translation vendor.
- The translated files will be named the same as the original files (so they could be _en.properties even if the content is for French)
Two FTP endpoints are necessary: a 'to translation' end point and a 'from translation' endpoint.
Example of the Zip File Structure
The structure of the zip file for target locale de is as follows:
Acme.frontend.1.de.zip | -- Acme.frontend.1.de | -- resources.properties | -- messages.resx | -- errors.json
This is the structure of the zip file sent for translation on the FTP outbound endpoint and also the structure and name of the file received from translation on the FTP inbound endpoint.
Notes:
- The names of the files are the base resource file names
- The names of the files (resources.properties) must be the same exact name when the translation is returned as it was in the zip file sent for translation.
- The files in one prep kit must all be returned in the translated zip file with a complete translation. If prep kit #1 had three files, the same three files must be returned in, say, German with every key/value pair returned.
How do I setup a proxy for a project's FTP server?
SSH/SFTP proxy server
If the ftp.out.protocol/ftp.in.protocol is SSH and there is a SFTP proxy server then the proxy host and proxy port are set up via java options.
There are 3 proxy protocols supported
- ProxyHTTP
- ProxySOCKS4
- ProxySOCKS5
ProxyHTTP
The expected java options for ProxyHTTP are:
- ftp.proxyHost
- ftp.proxyPort
Linux example if setting java options within a shell session.
- export _JAVA_OPTIONS='-Dftp.proxyHost=theproxyhost -Dftp.proxyPort=123'
ProxySOCKS4
The expected java options for ProxySOCKS4 are:
- ftp.socks4.proxyHost
- ftp.socks4.proxyPort
Linux example if setting java options within a shell session.
- export _JAVA_OPTIONS='-Dftp.socks4.proxyHost=theproxyhost -Dftp.socks4.proxyPort=123'
ProxySOCKS5
The expected java options for ProxySOCKS5 are:
- ftp.socks5.proxyHost
- ftp.socks5.proxyPort
Linux example if setting java options within a shell session.
- export _JAVA_OPTIONS='-Dftp.socks5.proxyHost=theproxyhost -Dftp.socks5.proxyPort=123'
Encoding of translated files
We are expecting the .properties to be returned in UTF-8 (no BOM) and so they should not have unicode escaped characters.
I configured a vanilla FTP (not SSH or SSL) and I get an SSH error
If you configure an FTP end point and you get a message from SSH, it means you are reaching an SSH FTP endpoint and that endpoint does not understand the request which follows a different protocol (FTP).
For instance, you may get this error:
org.apache.commons.net.MalformedServerReplyException: Could not parse response code.
Server Reply: SSH-2.0-OpenSSH_4.3
A possible issue may be the port number: port 22 is typically used for FTP/SSH; port 21 is typically used for vanilla FTP. Change the port number in config_l10n_vendor.properties and test again.