Difference between revisions of "L10n Vendors and Integration"

From Lingoport Wiki
Jump to: navigation, search
(→‎How do I hide the password)
(→‎InContext Token)
 
(113 intermediate revisions by 4 users not shown)
Line 1: Line 1:
 
=What are L10n Vendors?=
 
=What are L10n Vendors?=
   
−
LRM packages the resources files and send them to localization service providers and translation management systems to do translate the files. There are many ways to send the files and they can be done a number of different ways that depend on a different factors including the translation management system.
 +
Localyzer packages the resources files and send them to localization service providers (LSP) and translation management systems (TMS) to translate the files. There are many ways to send the files and they can be done a number of different ways that depend on different factors including the translation management system.
   
 
=Supported L10n Vendors=
  
=Supported L10n Vendors=
−
To get a current list of LSP ([[Terms_and_Definitions#localizationVendor|localization service provider]]) and TMS ([[Terms_and_Definitions#TMD|translation management system]]) companies that are integrated with LRM, please email ''sales@lingoport.com''.
 +
To get a current list of LSP ([[Terms_and_Definitions#localizationVendor|localization service provider]]) and TMS ([[Terms_and_Definitions#TMD|translation management system]]) companies that are integrated with
  +
Localyzer, please email ''sales@lingoport.com''.
   
 
= Recommended Approach =
  
= Recommended Approach =
Line 12: Line 13:
 
* The translation group picks up the files and [[Terms_and_Definitions#onboarding|on-boards]] them in the TMS following their own automated pattern
  
* The translation group picks up the files and [[Terms_and_Definitions#onboarding|on-boards]] them in the TMS following their own automated pattern
 
* The translation group returns the translated resource files to another FTP end point as per their system
  
* The translation group returns the translated resource files to another FTP end point as per their system
−
* LRM picks up the translated files, validates them, and imports them into the proper repository
 +
* Localyzer picks up the translated files, validates them, and imports them into the proper repository
   
 
==Advantages==
  
==Advantages==
Line 18: Line 19:
 
* '''Simplicity''': FTP and zip are well known and are well understood in the industry
  
* '''Simplicity''': FTP and zip are well known and are well understood in the industry
 
* '''Security''': FTP support SSH and SSL encryption; The FTP system can allow only some IP ranges to access the FTP port(s)
  
* '''Security''': FTP support SSH and SSL encryption; The FTP system can allow only some IP ranges to access the FTP port(s)
−
* '''Automation''': LRM pushes and retrieves files automatically; The translation group supplies the automation to and from the automation end point
 +
* '''Automation''':Localyzer pushes and retrieves files automatically; The translation group supplies the automation to and from the automation end point
 
* '''Network Failure Handling''': If the network goes down when accessing FTP, files are resent later when the network is up; Files are not lost due to downtime
  
* '''Network Failure Handling''': If the network goes down when accessing FTP, files are resent later when the network is up; Files are not lost due to downtime
−
* '''Asynchronicity''': The LRM system and the TMS do not need any special connection; Files can be sent and received independently
 +
* '''Asynchronicity''': The Localyzer system and the TMS do not need any special connection; Files can be sent and received independently
−
* '''Scalability''': The volume sent to an FTP system does not slow down either LRM or a TMS due to traffic. Each system can push or retrieve files without slowing down the other. FTP supports large volume of data
 +
* '''Scalability''': The volume sent to an FTP system does not slow down either Localyzer or a TMS due to traffic. Each system can push or retrieve files without slowing down the other. FTP supports large volume of data
−
* '''Configurability''': Configuring the FTP end point location is something well understood by IT departments and simple to configure (for LRM, see below)
 +
* '''Configurability''': Configuring the FTP end point location is something well understood by IT departments and simple to configure (for Localyzer, see below)
 
* '''Testability''': Checking the proper configuration is straightforward
  
* '''Testability''': Checking the proper configuration is straightforward
−
* '''Fast Set Up''': Given the above, setting up LRM with FTP is the fastest and simplest route
 +
* '''Fast Set Up''': Given the above, setting up Localyzer with FTP is the fastest and simplest route
   
−
Note: Lingoport has set up LRM over FTP with many Translation Groups and the TMS of their choice and any TMS vendor can add their system to this integration pattern.
 +
Note: Lingoport has set up Localyzer over FTP with many Translation Groups and the TMS of their choice and any TMS vendor can add their system to this integration pattern.
   
 
==Resource File Transmission Protocols==
  
==Resource File Transmission Protocols==
 
For most use cases, Lingoport recommends sending files to translation over '''SFTP'''. Direct API-based connections are also available to some TMS systems.
  
For most use cases, Lingoport recommends sending files to translation over '''SFTP'''. Direct API-based connections are also available to some TMS systems.
   
−
Overall, LRM supports 5 [[Terms_and_Definitions#localization|localization]] paths for sending files to be translated. They are:
 +
Overall, Localyzer supports several [[Terms_and_Definitions#localization|localization]] paths for sending files to be translated. They are:
   
 
'''Typical:'''
  
'''Typical:'''
−
* '''Local'''
 +
* [[L10n_Vendors_and_Integration#Local_Vendor|'''Local''']]
−
** Files are kept on the system LRM resides on. Good for debugging.
 +
** Files are kept on the system Localyzer resides on. Good for debugging.
  +
* [[L10n_Vendors_and_Integration#FTP_Vendor|'''FTP''']]
−
* '''FTP'''
  
 
** FTP/SFTP. Typical method of sending resource files to/from translation.
  
** FTP/SFTP. Typical method of sending resource files to/from translation.
   
 
'''API Based:'''
  
'''API Based:'''
  +
* [[L10n_Vendors_and_Integration#Lingotek_Vendor|'''Lingotek''']]
−
* '''GlobalLink'''
  
  +
* [[L10n_Vendors_and_Integration#WorldServer_Vendor|'''WorldServer''']]
−
* '''Lingotek'''
  
−
* '''WorldServer'''
 +
* '''Memsource'''
  +
* '''XTM'''
  +
* '''Machine Translation'''
  +
** [[L10n_Vendors_and_Integration#AWS_Amazon_Translate|'''AWS Amazon Translate''']]
  +
** [[L10n_Vendors_and_Integration#Systran|'''SYSTRAN Pure Neural® Server''']]
   
 
=How do I set the transmission protocol?=
  
=How do I set the transmission protocol?=
   
  +
Setting the transmission protocol has changed significantly over the releases. Setting the protocol is now done through Jenkins configuration. In Jenkins, select '''Manage Jenkins''' in the left menu and then '''Configure System'''. Scroll down to '''Localyzer L10n Vendor Setup'''.
−
The type of L10n Vendor is defined in the '''<code>config_l10n_vendor.properties<code>''' file that can exist at either the group or project level. See [[Group Configuration Files]]. A template of this file can be found at <code><HOME>/lingoport/lrm-server-x.y/deploy/templates/dir_structure/group/config</code>
  
   
  +
[[File:JenkinsConfigL10n.png]]
−
The default location of the config_l10n_vendor.properties file is at the group level in <code>/var/lib/jenkins/Lingoport_Data/L10nStreamlining/<group-name>/config</code>.
  
   
  +
=== Standard Vendor Info ===
−
The default settings do not define a vendor so that an error will occur when prepping a kit, forcing the user to chose a vendor. The global L10n Vendor attributes are:
  
  +
[[File:StandardInfo.png]]
−
*'''l10n.vendor=ftp''' - uncomment if using FTP/SFTP, the recommended approach
  
   
  +
==== Name ====
−
*'''l10n.vendor=lingotek''' - uncomment if using Lingotek
  
  +
The vendor name is used by a Jenkins job/Localyzer Project to define the L10n Vendor configuration to use. Vendor Name must be unique across all defined L10n Vendors.
−
*'''l10n.vendor=local''' - uncomment if storing the prep/translated files locally
  
−
*'''l10n.vendor=worldserver''' - uncomment if using Worldserver
  
−
*'''l10n.vendor=globallink''' - uncomment if using GlobalLink
  
−
*'''l10n.vendor.nonsupported.extensions''' - a vendor may not support a specific extension such as .json. Enter any extensions that are not supported by the vendor but are supported by LRM. See [[LRM Fixing Issues]] for LRM supported extensions. An error will occur if there is no conversion protocol for a non-supported extension.
  
   
  +
==== Access Level ====
−
Example file:
  
−
<pre>
  
−
l10n.vendor=ftp
  
−
#l10n.vendor=lingotek
  
−
#l10n.vendor=local
  
−
#l10n.vendor=worldserver
  
−
#l10n.vendor=globallink
  
−
l10n.vendor.nonsupported.extensions=
  
   
  +
===== Group Level =====
−
</pre>
  
  +
If the group access level is selected, then the configuration will be available to a Localyzer project at the group level.
  +
When these settings are saved, a L10n Vendor configuration file will be created in the selected group's configuration folder.
  +
If the access level is changed to project or configuration deleted, then on save, the group's configuration file will be deleted.
   
  +
The group drop-down is populated with all the Localyzer groups. The selected group must be unique across all the L10n Vendors otherwise a fatal error will occur when the global settings are saved.
−
=InContext Server Configuration=
  
−
In order for a L10n Vendor to be able to communicate to the Lingoport InContext Server, there must be a valid LRM LQA license for the group as well as a valid configuration for the InContext Server. See [[LRM_Instrumentation|InContext License]] for more information. The InContext Server configuration is located in the L10n Vendor configuration file with the following keys.
  
−
<pre>
  
−
l10n.vendor.incontext.server.url=&lt;valid http or https URL pointing to the InContext Server &gt;
  
−
l10n.vendor.incontext.auth.token=&lt;valid authorization token&gt;
  
−
</pre>
  
   
  +
====== Linking the Localyzer project to the defined group vendor ======
−
The authorization token will be supplied by your InContext Server administrator.
  
  +
When setting up the L10n Vendor with a group access level, a Localyzer group was selected. When the Localyzer Automation Job is configured for a particular Localyzer project then the corresponding group L10n Vendor information is displayed. If there is no corresponding group L10n Vendor that has been created then a message will be displayed. See [[On_Boarding_an_LRM_Project#On_Boarding_the_LRM_Automation_Job|On Boarding the Localyzer Automation Job]] for more information.
−
During the prep kit process, the configured URL and token will be used to access the InContext Server. If these are not valid then an error will occur and the prep kit process will fail. If the defined InContext server is valid then all the resource files will be [[LRM_Commands_Reference#Create_Instrumented_files|instrumented]] and the prep kit files will contain the InContext decoration.
  
   
  +
In the example below, a group access level L10n Vendor has been created for group ''Acme''. The ''Use Group Vendor'' option is selected because the project can use the L10 Vendor config file that was created during the Vendor config setup.
−
==InContext Decoration==
  
−
When prepping a kit where there is a valid InContext server configuration, each of the keys that need to be translated will have an InContext tag above the key. The default tag prefix is '''LRM_INCONTEXT'''. Your L10n vendor may have a specific tag that is used for filtering. If this is the case, then a unique InContext tag can be configured in the L10n Vendor configuration file using the following key:
  
−
<pre>
  
−
l10n.vendor.incontext.unique.tag=
  
−
</pre>
  
   
−
If this key is missing or has a blank value then the default, LRM_INCONTEXT, tag will be used.
  
   
  +
[[File:LinkGroupVendorToProject.png]]
−
===InContext decoration examples===
  
   
  +
===== Project Level =====
−
:'''L10n Vendor Configuration'''
  
  +
When setting up the L10n Vendor with a ''project'' access level, a Localyzer group was not selected. This means that the L10n Vendor configuration can be used by any Localyzer project.<br>
  +
If the configuration access level is changed from project to group (or deleted), then the configuration file for any Localyzer Automation Job that use this vendor name's configuration will be deleted.
   
  +
====== Linking the Localyzer project to a project vendor ======
−
<pre>
  
  +
There can be multiple L10n Vendors defined with a ''project'' access level. When the Localyzer Automation Job is configured for a particular Localyzer project then the user can select the L10n Vendor type, such as ''FTP'' to see what ''project'' access level vendors have been defined.
−
l10n.vendor.incontext.server.url=http://ec5-4-3-2-1.amazonaws.com:8081
 
  +
−
l10n.vendor.incontext.auth.token=0qhZ9yTvtW5
  
  +
In the example below, a project access level L10n Vendor has been created for a FTP vendor. The ''Use Project Specific Vendor'' option is selected because the project has a different protocol then the group level vendor. The next time that the job is built, the configuration defined for the ''FtpVendor'' will be used to create the L10n Vendor Config file in the L10nStreamling project's config file.
−
</pre>
  
  +
  +
[[File:LinkProjectVendorToProject.png]]
  +
  +
==== Incontext Translation ====
  +
For more info see [[InContext_Server_Users_Guide|Incontext User Guide]]
  +
  +
[[File:UseIncontextTranslation.png]]
  +
  +
===== InContext URL =====
  +
The URL of your InContext Server for InContext Translation.
  +
  +
===== InContext Token =====
  +
The Incontext Token value must match one of the tokens configured on the Settings page of the InContext Server. Together with the InContext Server URL, the token is used to display the content to the translator for InContext Translation. This authorization token will be supplied by your InContext Server administrator.
  +
  +
===== InContext Unique Tag =====
  +
A special tag is used when sending changed content to your L10n Vendor. The default tag is '''LRM_INCONTEXT'''.
  +
Some L10nVendors may require their own tag to delineate InContext information. If so then enter the tag here.
  +
  +
===== InContext File Decoration =====
  +
'''Example of Default Incontext Tag'''
  +
  +
[[File:IncontextTranslationExample.png]]
  +
  +
When prepping a kit where there is a valid InContext server configuration as above, each of the keys that need to be translated will have an InContext tag above the key. The default tag prefix is '''LRM_INCONTEXT''' since there is no ''InContext Unique Tag''.<br>
  +
Below are examples of the decoration that occurs in the prep kit files.
   
 
::'''XML type prep kit file'''
  
::'''XML type prep kit file'''
Line 113: Line 128:
 
</pre>
  
</pre>
   
  +
'''Example of Unique Incontext Tag'''
−
===InContext decoration examples using unique InContext tag===
  
   
  +
[[File:IncontextTranslationUnique.png]]
−
:'''L10n Vendor Configuration'''
  
   
  +
When prepping a kit where there is a ''Incontext Unique Tag'' defined, each of the keys that need to be translated will have the defined ''InContext Unique Tag'' above the key.<br>
−
<pre>
  
  +
Below are examples of the decoration that occurs in the prep kit files for a unique incontext tag.
−
l10n.vendor.incontext.server.url=http://ec5-4-3-2-1.amazonaws.com:8081
 
−
l10n.vendor.incontext.auth.token=0qhZ9yTvtW5
  
−
l10n.vendor.incontext.unique.tag=MY_SPECIAL_TAG
  
−
</pre>
  
   
 
::'''XML type prep kit file'''
  
::'''XML type prep kit file'''
Line 128: Line 140:
 
<!--MyProject_8_6 - CHANGES-ONLY-->
  
<!--MyProject_8_6 - CHANGES-ONLY-->
 
<resources>
  
<resources>
−
<!--MY_SPECIAL_TAG http://ec5-4-3-2-1.amazonaws.com:8081/incontext-server/lookup?key=3130_192&token=0qhZ9yTvtW5-->
 +
<!--SOME_UNIQUE_TAG http://ec5-4-3-2-1.amazonaws.com:8081/incontext-server/lookup?key=3130_192&token=0qhZ9yTvtW5-->
 
<string name="Hello">Hello</string>
  
<string name="Hello">Hello</string>
 
</pre>
  
</pre>
Line 135: Line 147:
 
<pre>
  
<pre>
 
#MyProject_1_4 - CHANGES-ONLY
  
#MyProject_1_4 - CHANGES-ONLY
−
#MY_SPECIAL_TAG http://ec5-4-3-2-1.amazonaws.com:8081/incontext-server/lookup?key=422_238&token=0qhZ9yTvtW5
 +
#SOME_UNIQUE_TAG http://ec5-4-3-2-1.amazonaws.com:8081/incontext-server/lookup?key=422_238&token=0qhZ9yTvtW5
 
myString=Hello
  
myString=Hello
 
</pre>
  
</pre>
   
  +
==== Vendor has unsupported file types ====
−
=How to set up FTP as the transmission protocol=
  
  +
Some L10 Vendors do not supported all the extensions types that Localyzer supports. If this is true for one of your Vendors then select the checkbox to display the supported extensions that have an OTB conversion. If an extension cannot be transformed by Localyzer (not in the drop down) then special transforms can be created by Lingoport.
   
−
==FTP Configuration==
 +
=== Local Vendor ===
−
To choose FTP as the vendor, uncomment the <code>l10n.vendor=ftp</code> in the config_l10n_vendor.properties file.
  
   
  +
[[File:L10nLocalVendor.png |700px]]
−
The information that is needed to upload the files to be translated as well as retrieve translated files is:
  
−
<pre>
  
−
#FTP Attributes
  
−
## FTP inbound attributes for import kit files
  
−
ftp.in.host=
  
−
ftp.in.port=
  
−
ftp.in.location.path=
  
−
ftp.in.password=
  
−
ftp.in.username=
  
−
##SSH, SSL or empty
  
−
ftp.in.protocol=
  
−
## ssl implicit flag. Set to 0 for explicit, set to 1 for implicit
  
−
ftp.in.ssl.implicit=0
  
   
  +
A local vendor uses the current system for the incoming and outgoing files.
−
## FTP outbound attributes for prep kit files
  
  +
* Set the name of the vendor. This can be anything descriptive that can be selected in the automation jobs.
−
ftp.out.host=
  
  +
* Set the names of the incoming and outgoing file locations.
−
ftp.out.port=
  
  +
* [[XTM]] and [[Memsource]] use the local vendor for their configurations. Continuing to these links gives more information.
−
ftp.out.location.path=
  
−
ftp.out.username=
  
−
ftp.out.password=
  
−
##SSH, SSL or empty
  
−
ftp.out.protocol=
  
−
## ssl implicit flag. Set to 0 for explicit, set to 1 for implicit
  
−
ftp.out.ssl.implicit=0
  
   
  +
==== Translation Local Vendor Integration ====
−
</pre>
  
  +
See [[Local_Vendor|advanced Local vendor info]] for more vendor details.
   
  +
=== FTP Vendor ===
−
The <code>ftp.out</code> attributes refer to locations for files being sent out for translation. The <code>ftp.in</code> refers to locations for files coming back in (import) from translation.
  
   
  +
[[File:L10nFTPVendor.png |850px]]
−
Note: For special cases, like using One Planet, you may need to create an .ssh key http://www.linuxproblem.org/art_9.html.
  
   
  +
* Set the name of the vendor. This can be anything descriptive that can be selected in the automation jobs.
−
=== FTP attributes ===
  
  +
* Set the hostname for uploading files and for downloading files and the respective file location, username, and password. '''Test the Connection''' to make sure that the system can communicate with the host.
−
If the end point is FTP, the
 
  +
* Often FTP uses port 22 and SSH. Change this if the configuration differs.
−
l10n.vendor=ftp
  
−
must be uncommented. The ''''in'''' attributes refer to the files coming back ''to'' Lingoport Resource Manager after translation. The ''''out'''' attributes refer to the prep-kit files which are send ''from'' Lingoport Resource Manager.
  
−
* '''ftp.in.host''' - the hostname where the translated files reside
  
−
* '''ftp.in.location.path''' - the directory path on the host containing the translated files
  
−
* '''ftp.in.password''' - password associated with the username that is using the protocol
 
−
* '''ftp.in.port''' - the port for the protocol service. In general, SSH uses port 22 and FTP uses port 21
  
−
* '''ftp.in.protocol''' - this can be SSH or SSL. If left blank, the protocol is FTP
  
−
* '''ftp.in.username''' - the username used with the protocol to receive the translated files
  
   
  +
==== Translation FTP Vendor Integration ====
  +
See [[Zip_Files_For_Prep_and_Import | Prep/Import Zip Files]] for more information.
   
  +
=== Lingotek Vendor ===
−
* '''ftp.out.host''' - The hostname that the prep-kits prepared by LRM are being sent to.
  
−
* '''ftp.out.location.path''' - the directory path on the host for the files to be translated
  
−
* '''ftp.out.password''' - password associated with the username that is using the protocol
  
−
* '''ftp.out.port''' - the port for the protocol service. In general, SSH uses port 22 and FTP uses port 21
  
−
* '''ftp.out.protocol''' - this can be SSH or SSL. If left blank, the protocol is FTP
  
−
* '''ftp.out.username''' - the username used with the protocol to send the files to be translated.
  
   
  +
[[File:L10nLingotekVendor.png |700px]]
−
From a command line attempt to use the protocol (ssh, ssl or ftp) and the username and password to log into the host. This can verify the settings.
  
−
On the host, verify that the in and out paths exist.
  
   
  +
* Set the name of the vendor. This can be anything descriptive that can be selected in the automation jobs.
−
==== How do I create the SSH keys? ====
  
  +
* Set the Lingotek credentials. If these are unknown, contact Lingotek or ''support@lingoport.com''.
  +
* The Jenkins URL will be something like: '''https://<IP address>/jenkins'''. Probably of the current system.
  +
  +
==== Configuration Details ====
  +
See [[Lingotek|advanced Lingotek vendor info]] for more Lingotek configuration details.
  +
  +
=== WorldServer Vendor ===
  +
  +
[[File:L10nWorldserverVendor.png |800px]]
  +
  +
*Set the name of the vendor. This can be anything descriptive that can be selected in the automation jobs.
  +
*Set the credentials for WorldServer. Contact WorldServer for these values.
  +
*Set the host, port, location and username and password for the files to download and import.
  +
*Enable the following Jenkins Jobs under the Jenkins 'TMS' tab - WorldserverCheckDownloads, WorldserverDownload, WorldserverImport
  +
*See [[WorldServer|advanced WorldServer vendor issues]] for more WorldServer configuration details.
  +
  +
Historically, the type of L10n Vendor was defined in the '''<code>config_l10n_vendor.properties</code>''' file that can exist at either the group or project level. However, since it is now part of the Jenkins configuration, this is obsolete. Manual changes to this file may be overwritten by the Jenkins configuration.
  +
  +
The default location of the config_l10n_vendor.properties file is at the group level in <code>/var/lib/jenkins/Lingoport_Data/L10nStreamlining/<group-name>/config</code>.
  +
  +
==== Configuration Details ====
  +
See [[WorldServer|advanced WorldServer vendor info]] for more WorldServer configuration details.
  +
  +
=== Machine Translation Vendors ===
  +
  +
==== AWS Amazon Translate ====
  +
See the [[AWS_Amazon_Translate|AWS Amazon Translate]] wiki page for information on how to set up a AWS Amazon Translate Vendor.
  +
  +
==== Systran ====
  +
See the [[Systran|Systran]] wiki page for information on how to set up a Systran Vendor.
  +
  +
=== Microsoft Azure Cognitive Services Translator ===
  +
See the [[Microsoft Translator]] wiki page for information on how to set up a Microsoft Vendor.
  +
  +
=== Google Cloud Platform Translation Service ===
  +
See the [[Google Translate]] wiki page for information on how to set up a Google Vendor.
  +
  +
=Advanced FTP Vendor issues=
  +
  +
=== How do I create the SSH keys? ===
   
 
See http://www.linuxproblem.org/art_9.html.
  
See http://www.linuxproblem.org/art_9.html.
   
−
==== How do I hide the password? ====
 +
=== How do I hide the password? ===
 
Use a '''.netrc''' file, The .netrc file is located in the $HOME directory.
 
Use a '''.netrc''' file, The .netrc file is located in the $HOME directory.
   
Line 213: Line 237:
   
 
Then <code>ftp ftp.company.net</code> will use the login and username in the .netrc file to connect and the username password are not visible.
  
Then <code>ftp ftp.company.net</code> will use the login and username in the .netrc file to connect and the username password are not visible.
  +
=== I configured a vanilla FTP (not SSH or SSL) and I get an SSH error===
   
  +
If you configure an FTP end point and you get a message from SSH, it means you are reaching an SSH FTP endpoint and that endpoint does not understand the request which follows a different protocol (FTP).
−
=== Translation FTP Vendor Integration===
  
−
Lingoport pushes zip files to the configured FTP outbound endpoint and consumes zip files sent on the incoming endpoint. The structure of the zip file is defined below:
  
   
  +
For instance, you may get this error:
−
The files sent back and forth over FTP have the following characteristics:
  
   
  +
<code>org.apache.commons.net.MalformedServerReplyException: Could not parse response code.</code>
−
* The data files to be translated or returned from translation are sent in a '''zip file'''.
  
  +
<code>Server Reply: SSH-2.0-OpenSSH_4.3</code>
−
* The '''zip file''' is named according to this convention: '''<code><group-name>.<project-name>.<kit-version>.<locale>.zip</code>'''. For example, a zip file could be named '''<code>Queens.Champions.1.fr_fr.zip</code>''': The group-name is Queens, the project-name is Champions, the kit version is 1, and the target locale is fr_FR.
  
−
* The zip file consists of '''one top level directory''', <code><group-name>.<project-name>.<kit-version>.<locale></code>, with all the files to be translated under it.
  
−
* There is '''one zip file per target locale'''. You may target 10 locales for translation for project "Champions" under group "Queens". You then would have 10 zip files to send over FTP.
 
−
* A zip file may contain '''more than one file and more than one file type'''.
 
−
* Each file to be translated has a comment at the top of the file which needs to be kept when returning the file from translation vendor.
  
−
* The '''translated files''' will be named the same as the original files (so they could be _en.properties even if the content is for French)
  
   
  +
A possible issue may be the port number: port 22 is typically used for FTP/SSH; port 21 is typically used for vanilla FTP. Change the port number in '''config_l10n_vendor.properties''' and test again.
−
Two FTP endpoints are necessary: a ''''to translation'''' end point and a ''''from translation'''' endpoint.
  
   
  +
= How do I setup a proxy for a project's FTP server? =
−
=== Example of the Zip File Structure ===
  
−
The structure of the zip file for target locale ''de'' is as follows:
  
−
<pre>
  
−
Acme.frontend.1.de.zip
  
−
| -- Acme.frontend.1.de
  
−
| -- resources.properties
  
−
| -- messages.resx
  
−
| -- errors.json
  
−
</pre>
  
−
 
−
This is the structure of the zip file sent for translation on the FTP outbound endpoint and also the structure and name of the file received from translation on the FTP inbound endpoint.
 
−  
−
Notes:
 
−
* The names of the files are the base resource file names
  
−
* The names of the files (resources.properties) must be the same exact name when the translation is returned as it was in the zip file sent for translation.
 
−
* The files in one prep kit must all be returned in the translated zip file with a complete translation. If prep kit #1 had three files, the same three files must be returned in, say, German with every key/value pair returned.
  
−  
−
== How do I setup a proxy for a project's FTP server? ==
  
 
=== SSH/SFTP proxy server ===
  
=== SSH/SFTP proxy server ===
 
If the ftp.out.protocol/ftp.in.protocol is SSH and there is a SFTP proxy server then the proxy host and proxy port are set up via java options.
  
If the ftp.out.protocol/ftp.in.protocol is SSH and there is a SFTP proxy server then the proxy host and proxy port are set up via java options.
Line 282: Line 284:
 
:: <span style="background-color: #eaecf0;>export _JAVA_OPTIONS='-Dftp.socks5.proxyHost=theproxyhost -Dftp.socks5.proxyPort=123'</span>
  
:: <span style="background-color: #eaecf0;>export _JAVA_OPTIONS='-Dftp.socks5.proxyHost=theproxyhost -Dftp.socks5.proxyPort=123'</span>
   
−
==Encoding of translated files==
 +
=Encoding of translated files=
−
We are expecting the .properties to be returned in UTF-8 (no BOM) and so they should not have unicode escaped characters.
 +
We are expecting the .properties to be returned in UTF-8 (no BOM) and so they should not have Unicode escaped characters.
−  
−
= Other Integrations =
  
−
<!-- [[Special L10n Vendor Integrations | Special L10n Vendor Integrations ]] -->
  
−  
−
[[Local Vendor]]
  
−  
−
[[GlobalLink]]
  
−  
−
[[Lingotek]]
  
−  
−
[[WorldServer]]
  

Latest revision as of 19:02, 26 August 2022

What are L10n Vendors?

Localyzer packages the resources files and send them to localization service providers (LSP) and translation management systems (TMS) to translate the files. There are many ways to send the files and they can be done a number of different ways that depend on different factors including the translation management system.

Supported L10n Vendors

To get a current list of LSP (localization service provider) and TMS (translation management system) companies that are integrated with Localyzer, please email sales@lingoport.com.

Recommended Approach

Lingoport FTP Protocol

Lingoport strongly recommends using the FTP protocol. With that approach,

  • Lingoport automatically pushes valid resource files to be translated following a specific naming convention to an FTP end point
  • The translation group picks up the files and on-boards them in the TMS following their own automated pattern
  • The translation group returns the translated resource files to another FTP end point as per their system
  • Localyzer picks up the translated files, validates them, and imports them into the proper repository

Advantages

  • Robustness: FTP has been in place for decades and is one of the most reliable protocols for transmitting files from one system to another
  • Simplicity: FTP and zip are well known and are well understood in the industry
  • Security: FTP support SSH and SSL encryption; The FTP system can allow only some IP ranges to access the FTP port(s)
  • Automation:Localyzer pushes and retrieves files automatically; The translation group supplies the automation to and from the automation end point
  • Network Failure Handling: If the network goes down when accessing FTP, files are resent later when the network is up; Files are not lost due to downtime
  • Asynchronicity: The Localyzer system and the TMS do not need any special connection; Files can be sent and received independently
  • Scalability: The volume sent to an FTP system does not slow down either Localyzer or a TMS due to traffic. Each system can push or retrieve files without slowing down the other. FTP supports large volume of data
  • Configurability: Configuring the FTP end point location is something well understood by IT departments and simple to configure (for Localyzer, see below)
  • Testability: Checking the proper configuration is straightforward
  • Fast Set Up: Given the above, setting up Localyzer with FTP is the fastest and simplest route

Note: Lingoport has set up Localyzer over FTP with many Translation Groups and the TMS of their choice and any TMS vendor can add their system to this integration pattern.

Resource File Transmission Protocols

For most use cases, Lingoport recommends sending files to translation over SFTP. Direct API-based connections are also available to some TMS systems.

Overall, Localyzer supports several localization paths for sending files to be translated. They are:

Typical:

  • Local
    • Files are kept on the system Localyzer resides on. Good for debugging.
  • FTP
    • FTP/SFTP. Typical method of sending resource files to/from translation.

API Based:

How do I set the transmission protocol?

Setting the transmission protocol has changed significantly over the releases. Setting the protocol is now done through Jenkins configuration. In Jenkins, select Manage Jenkins in the left menu and then Configure System. Scroll down to Localyzer L10n Vendor Setup.

JenkinsConfigL10n.png

Standard Vendor Info

StandardInfo.png

Name

The vendor name is used by a Jenkins job/Localyzer Project to define the L10n Vendor configuration to use. Vendor Name must be unique across all defined L10n Vendors.

Access Level

Group Level

If the group access level is selected, then the configuration will be available to a Localyzer project at the group level. When these settings are saved, a L10n Vendor configuration file will be created in the selected group's configuration folder. If the access level is changed to project or configuration deleted, then on save, the group's configuration file will be deleted.

The group drop-down is populated with all the Localyzer groups. The selected group must be unique across all the L10n Vendors otherwise a fatal error will occur when the global settings are saved.

Linking the Localyzer project to the defined group vendor

When setting up the L10n Vendor with a group access level, a Localyzer group was selected. When the Localyzer Automation Job is configured for a particular Localyzer project then the corresponding group L10n Vendor information is displayed. If there is no corresponding group L10n Vendor that has been created then a message will be displayed. See On Boarding the Localyzer Automation Job for more information.

In the example below, a group access level L10n Vendor has been created for group Acme. The Use Group Vendor option is selected because the project can use the L10 Vendor config file that was created during the Vendor config setup.


LinkGroupVendorToProject.png

Project Level

When setting up the L10n Vendor with a project access level, a Localyzer group was not selected. This means that the L10n Vendor configuration can be used by any Localyzer project.
If the configuration access level is changed from project to group (or deleted), then the configuration file for any Localyzer Automation Job that use this vendor name's configuration will be deleted.

Linking the Localyzer project to a project vendor

There can be multiple L10n Vendors defined with a project access level. When the Localyzer Automation Job is configured for a particular Localyzer project then the user can select the L10n Vendor type, such as FTP to see what project access level vendors have been defined.

In the example below, a project access level L10n Vendor has been created for a FTP vendor. The Use Project Specific Vendor option is selected because the project has a different protocol then the group level vendor. The next time that the job is built, the configuration defined for the FtpVendor will be used to create the L10n Vendor Config file in the L10nStreamling project's config file.

LinkProjectVendorToProject.png

Incontext Translation

For more info see Incontext User Guide

UseIncontextTranslation.png

InContext URL

The URL of your InContext Server for InContext Translation.

InContext Token

The Incontext Token value must match one of the tokens configured on the Settings page of the InContext Server. Together with the InContext Server URL, the token is used to display the content to the translator for InContext Translation. This authorization token will be supplied by your InContext Server administrator.

InContext Unique Tag

A special tag is used when sending changed content to your L10n Vendor. The default tag is LRM_INCONTEXT. Some L10nVendors may require their own tag to delineate InContext information. If so then enter the tag here.

InContext File Decoration

Example of Default Incontext Tag

IncontextTranslationExample.png

When prepping a kit where there is a valid InContext server configuration as above, each of the keys that need to be translated will have an InContext tag above the key. The default tag prefix is LRM_INCONTEXT since there is no InContext Unique Tag.
Below are examples of the decoration that occurs in the prep kit files.

XML type prep kit file
<?xml version="1.0" encoding="UTF-8"?>
<!--MyProject_8_6 - CHANGES-ONLY-->
<resources>
<!--LRM_INCONTEXT http://ec5-4-3-2-1.amazonaws.com:8081/incontext-server/lookup?key=3130_192&token=0qhZ9yTvtW5-->
<string name="Hello">Hello</string>
Properties type prep kit file
#MyProject_1_4 - CHANGES-ONLY
#LRM_INCONTEXT http://ec5-4-3-2-1.amazonaws.com:8081/incontext-server/lookup?key=422_238&token=0qhZ9yTvtW5
myString=Hello

Example of Unique Incontext Tag

IncontextTranslationUnique.png

When prepping a kit where there is a Incontext Unique Tag defined, each of the keys that need to be translated will have the defined InContext Unique Tag above the key.
Below are examples of the decoration that occurs in the prep kit files for a unique incontext tag.

XML type prep kit file
<?xml version="1.0" encoding="UTF-8"?>
<!--MyProject_8_6 - CHANGES-ONLY-->
<resources>
<!--SOME_UNIQUE_TAG http://ec5-4-3-2-1.amazonaws.com:8081/incontext-server/lookup?key=3130_192&token=0qhZ9yTvtW5-->
<string name="Hello">Hello</string>
Properties type prep kit file
#MyProject_1_4 - CHANGES-ONLY
#SOME_UNIQUE_TAG http://ec5-4-3-2-1.amazonaws.com:8081/incontext-server/lookup?key=422_238&token=0qhZ9yTvtW5
myString=Hello

Vendor has unsupported file types

Some L10 Vendors do not supported all the extensions types that Localyzer supports. If this is true for one of your Vendors then select the checkbox to display the supported extensions that have an OTB conversion. If an extension cannot be transformed by Localyzer (not in the drop down) then special transforms can be created by Lingoport.

Local Vendor

L10nLocalVendor.png

A local vendor uses the current system for the incoming and outgoing files.

  • Set the name of the vendor. This can be anything descriptive that can be selected in the automation jobs.
  • Set the names of the incoming and outgoing file locations.
  • XTM and Memsource use the local vendor for their configurations. Continuing to these links gives more information.

Translation Local Vendor Integration

See advanced Local vendor info for more vendor details.

FTP Vendor

L10nFTPVendor.png

  • Set the name of the vendor. This can be anything descriptive that can be selected in the automation jobs.
  • Set the hostname for uploading files and for downloading files and the respective file location, username, and password. Test the Connection to make sure that the system can communicate with the host.
  • Often FTP uses port 22 and SSH. Change this if the configuration differs.

Translation FTP Vendor Integration

See Prep/Import Zip Files for more information.

Lingotek Vendor

L10nLingotekVendor.png

  • Set the name of the vendor. This can be anything descriptive that can be selected in the automation jobs.
  • Set the Lingotek credentials. If these are unknown, contact Lingotek or support@lingoport.com.
  • The Jenkins URL will be something like: https://<IP address>/jenkins. Probably of the current system.

Configuration Details

See advanced Lingotek vendor info for more Lingotek configuration details.

WorldServer Vendor

L10nWorldserverVendor.png

  • Set the name of the vendor. This can be anything descriptive that can be selected in the automation jobs.
  • Set the credentials for WorldServer. Contact WorldServer for these values.
  • Set the host, port, location and username and password for the files to download and import.
  • Enable the following Jenkins Jobs under the Jenkins 'TMS' tab - WorldserverCheckDownloads, WorldserverDownload, WorldserverImport
  • See advanced WorldServer vendor issues for more WorldServer configuration details.

Historically, the type of L10n Vendor was defined in the config_l10n_vendor.properties file that can exist at either the group or project level. However, since it is now part of the Jenkins configuration, this is obsolete. Manual changes to this file may be overwritten by the Jenkins configuration.

The default location of the config_l10n_vendor.properties file is at the group level in /var/lib/jenkins/Lingoport_Data/L10nStreamlining/<group-name>/config.

Configuration Details

See advanced WorldServer vendor info for more WorldServer configuration details.

Machine Translation Vendors

AWS Amazon Translate

See the AWS Amazon Translate wiki page for information on how to set up a AWS Amazon Translate Vendor.

Systran

See the Systran wiki page for information on how to set up a Systran Vendor.

Microsoft Azure Cognitive Services Translator

See the Microsoft Translator wiki page for information on how to set up a Microsoft Vendor.

Google Cloud Platform Translation Service

See the Google Translate wiki page for information on how to set up a Google Vendor.

Advanced FTP Vendor issues

How do I create the SSH keys?

See http://www.linuxproblem.org/art_9.html.

How do I hide the password?

Use a .netrc file, The .netrc file is located in the $HOME directory.

Example of a .netrc: 

machine ftp.company.net
login username
password myPassword

Then ftp ftp.company.net will use the login and username in the .netrc file to connect and the username password are not visible.

I configured a vanilla FTP (not SSH or SSL) and I get an SSH error

If you configure an FTP end point and you get a message from SSH, it means you are reaching an SSH FTP endpoint and that endpoint does not understand the request which follows a different protocol (FTP).

For instance, you may get this error: 

   org.apache.commons.net.MalformedServerReplyException: Could not parse response code.
   Server Reply: SSH-2.0-OpenSSH_4.3

A possible issue may be the port number: port 22 is typically used for FTP/SSH; port 21 is typically used for vanilla FTP. Change the port number in config_l10n_vendor.properties and test again.

How do I setup a proxy for a project's FTP server?

SSH/SFTP proxy server

If the ftp.out.protocol/ftp.in.protocol is SSH and there is a SFTP proxy server then the proxy host and proxy port are set up via java options.

There are 3 proxy protocols supported

  • ProxyHTTP
  • ProxySOCKS4
  • ProxySOCKS5

ProxyHTTP

The expected java options for ProxyHTTP are:

  • ftp.proxyHost
  • ftp.proxyPort

Linux example if setting java options within a shell session.

export _JAVA_OPTIONS='-Dftp.proxyHost=theproxyhost -Dftp.proxyPort=123'

ProxySOCKS4

The expected java options for ProxySOCKS4 are:

  • ftp.socks4.proxyHost
  • ftp.socks4.proxyPort

Linux example if setting java options within a shell session.

export _JAVA_OPTIONS='-Dftp.socks4.proxyHost=theproxyhost -Dftp.socks4.proxyPort=123'

ProxySOCKS5

The expected java options for ProxySOCKS5 are:

  • ftp.socks5.proxyHost
  • ftp.socks5.proxyPort

Linux example if setting java options within a shell session.

export _JAVA_OPTIONS='-Dftp.socks5.proxyHost=theproxyhost -Dftp.socks5.proxyPort=123'

Encoding of translated files

We are expecting the .properties to be returned in UTF-8 (no BOM) and so they should not have Unicode escaped characters.

Retrieved from "http://wiki.lingoport.com/index.php?title=L10n_Vendors_and_Integration&oldid=96576"