Difference between revisions of "Localyzer Jenkins Jobs"

From Lingoport Wiki
Jump to: navigation, search
(→‎Infrastructure Tab Jobs)
(→‎Checksend)
 
(39 intermediate revisions by 4 users not shown)
Line 1: Line 1:
  +
Jobs are organized under views (or tabs). If you have not setup the basic infrastructure jobs or the tabs, run the scripts to do so.
−
LRM Jenkins Jobs are setup under 4 tabs:
  
−
* The '''0 - Infrastructure''' tab contains Jenkins job which run independently of particular projects.
 
−
* The '''1 - Automation''' tab contains on-boarded jobs which generate the status of the project and pushes it to the dashboard, and pseudo-localizes resource files if necessary. These jobs tend to run whenever files are pushed to the repository for the project.
  
−
* The '''2 - Nightlies''' tab contains on-boarded jobs which run every day or every night and provide an update to the dashboard and an email to a set of recipients based on the configured frequency.
  
−
* The '''3- PrepKit''' tab contains on-boarded jobs which are triggered by a human intervention to send the kits for translation based on the configured translation endpoint.
  
   
  +
=== Execute <code>create-views.sh</code> and <code>create-lingoport-jobs.sh</code> ===
−
We focus on the infrastructure jobs
  
  +
This way to setup involves downloading the <code>jenkins-cli.jar</code> file, setting some variables, and running the script.
  +
  +
==== Download <code>jenkins-cli.jar</code> ====
  +
If you have not already done so:
  +
* Navigate to your Jenkins URL followed by cli, for example <code>http://localhost:8080/cli</code>.
  +
* Click on the download link and copy the downloaded jenkins-cli.jar to a location you have access to.
  +
* From the '/cli' page, click on the ''login'' command in the list. An example is given on how to login Jenkins using jenkins-cli.jar. This will be useful soon.
  +
  +
==== Setup Environment Variables ====
  +
If you have not already done so:
  +
Two global variables need to be setup:
  +
* Set JENKINS_CLI to the absolute location of the downloaded jenkins-cli.jar file, for example
  +
: <code>export JENKINS_CLI=/home/jenkins/jenkins-cli.jar</code>
  +
* Set JENKINS_URL to the URL for Jenkins, for example
  +
: <code>export JENKINS_URL=http://locahost:8080/</code>
  +
  +
==== Login using <code>jenkins-cli.jar</code> ====
  +
If you have not already done so:
  +
The login command example you saw above should help you login with an account with administrative privileges. It should look like:
  +
: <code>java -jar $JENKINS_CLI -s $JENKINS_URL login --username admin --password adminPassword</code>
  +
==== Execute the <code>create-views.sh</code> and <code>create-lingoport-jobs.sh</code> ====
  +
Go to the directory where the script is located and execute the script. For example:
  +
* Either
  +
:: <code>cd /var/lib/jenkins/lingoport/lrm-server-x.x/jenkins/views</code>
  +
:: <code>cd /home/jenkins/lingoport/lrm-server-x.x/jenkins/views</code>
  +
* <code>./create-views.sh</code>
  +
  +
Go to the directory where the next script is located and execute the script. For example:
  +
* Either
  +
:: <code>cd /var/lib/jenkins/lingoport/lrm-server-2.2/jenkins/base-jobs</code>
  +
:: <code>cd /home/jenkins/lingoport/lrm-server-2.2/jenkins/base-jobs</code>
  +
* <code>./create-lingoport-jobs.sh</code>
  +
  +
==== Restart Jenkins ====
  +
You may be able to do so with a browser URL such as <code>http://localhost:8080/restart</code>
  +
==== Verify the Views and Jobs have been Created ====
  +
* Navigate to the Jenkins URL, such as <code>http://localhost:8080/</code>: You should see new views (tabs) and jobs in those tabs.
  +
  +
= LRM Jobs =
  +
  +
LRM Jenkins Jobs are setup under tabs:
  +
== Infrastructure ==
  +
This tab contains Jenkins job which run independently of particular projects.
  +
See [[LRM_Jenkins_Jobs#Infrastructure_Tab_Jobs| Infrastructure Jobs]]
  +
  +
== Automation ==
  +
This tab contains on-boarded jobs which generate the status of the project and pushes it to the dashboard, and pseudo-localizes resource files if necessary. These jobs tend to run whenever files are pushed to the repository for the project.
  +
== Nightlies ==
  +
This tab contains on-boarded jobs which run every day or every night and provide an update to the dashboard and an email to a set of recipients based on the configured frequency.
  +
== PrepKit ==
  +
This tab contains on-boarded jobs which are triggered by a human intervention to send the kits for translation based on the configured translation endpoint. See [[LRM_Process_Commands_Reference#Send_out_prep_kits|LRM Process Prep Kit Command]]
   
 
= Infrastructure Tab Jobs =
  
= Infrastructure Tab Jobs =
  +
== CheckSend ==
−
* '''checksend''': This job sweeps through unsent prep kits and tries to send them again to the configured translation endpoint, such as Lingotek, FTP, or WorldServer. It is best to keep this job triggered by a manual click so that the error is first corrected (translation group network access or authorization, etc.) before trying to resend the prep kit.
  
  +
This job sweeps through unsent prep kits, located in the staging area, and tries to send them again to the configured translation endpoint, such as Lingotek, FTP, or WorldServer. It is best to keep this job triggered by a manual click so that the error is first corrected (translation group network access or authorization, etc.) before trying to resend the prep kit.
−
* '''debug''': This job is a free type of job setup to enter any kind of command to debug the installation or anything else in the system.
 
   
  +
See [[LRM_Process_Commands_Reference#--check-send_.28Upload_pending_prep_kits.29|LRM Process --check-send]]
−
== Lingotek Jobs ==
  
−
* LingotekCallback: (<code>lingotek_webhook.sh</code>This job is invoked by Lingotek upon the last step of the workflow. Files are downloaded to the staging area and ready to be imported. This needs to be a <code>Parameterized Build</code> with the following <code>String Parameters</code> (default NOTSET):
  
−
<code>
  
−
** project_id
  
−
** document_id
  
−
** locale_code
  
−
** locale
  
−
** progress
  
−
** type
  
−
** complete
  
−
** (Build Triggered Remotely:) LINGOSQUARE
  
−
</code>
  
−
* LingotekCheckCallback: (<code>lingotek_checkcallbacks.sh</code>) This job makes sure that files which needed to be downloaded from Lingotek have been downloaded and moved to the staging area
  
−
* LingotekImportKit: (<code>lingotek_import.sh</code>This job runs based on a frequency (every 2 hours) and runs the import for Lingotek files stashed under the staging area.
  
   
 
== FTP Jobs ==
  
== FTP Jobs ==
  +
=== FTPDownload ===
−
* FTPDownload: This job makes sure that files which needed to be downloaded from FTP 'in' have been downloaded and moved to the staging area and are ready to be imported. It runs on a regular (every hour) schedule. It calls the <code>ftp_download.sh</code> script.
  
−
* FTPImport: This job runs based on a frequency (every 2 hours) and runs the import for FTP files stashed under the staging area. It calls the <code>ftp_import.sh</code> script.
 +
This job makes sure that files which needed to be downloaded from FTP 'in' have been downloaded and moved to the staging area and are ready to be imported. It runs on a regular (every hour) schedule. It calls the <code>ftp_download.sh</code> script.
  +
=== FTPImport ===
  +
This job runs based on a frequency (every 2 hours) and runs the import for FTP files stashed under the staging area.
  +
  +
== GlobalLink Jobs ==
  +
  +
== Lingotek Jobs ==
  +
=== LingotekCallback ===
  +
This job is invoked by Lingotek upon the last step of their workflow. Files are first downloaded to the download staging area and then moved to the Lingotek import staging area.
  +
  +
=== LingotekCheckCallback===
  +
This job runs once a day and moves any files left in the download staging area to the Lingotek import staging area.
  +
  +
=== LingotekImportKit ===
  +
This job runs based on a frequency (every 2 hours) and runs the import on any files in the Lingotek import staging area.
  +
  +
== Local Vendor Jobs ==
  +
=== LocalVendorDownload ===
  +
This job runs every 2 hours. It downloads the zip files that are in the expected ''incoming'' location and moves the files to the LocaleVendor import staging area.
  +
=== LocalVendorImport ===
  +
This job is triggered after the LocalVendorDownload completes and imports all files that are in the import staging area.
   
 
== WorldServer Jobs ==
  
== WorldServer Jobs ==
  +
  +
=== WorldserverDownload ===
  +
When using the Worldserver protocol in the config_l10n_vendor.properties file, this job is invoked to download any zip file located in the expected incoming folder. The files are moved from the expected incoming folder into the download staging area.
  +
  +
=== WorldserverCheckDownloads ===
  +
This job moves any files that are in from the download staging area into the importkit staging area.
  +
  +
=== WorldserverImport ===
  +
This job runs after files have been downloaded using WSDownload. It checks the translated files for LRM correctness and pushes the translated files if they pass the tests. If they fail the tests, the files are not pushed into the corresponding repository at the correct location. A Notification email is sent in either case, success or failure
  +
  +
== Next Steps ==
  +
* If your projects are under Git, follow those additional instructions: [[Git]]
  +
* If your projects are under SVN, follow these additional instructions: [[SVN]]
  +
  +
The overall system is setup and configured. You are now ready to on-board your first project. The first project will prepare for all subsequent LRM projects.
  +
[[On-Boarding New Projects]]

Latest revision as of 23:12, 26 August 2022

Jobs are organized under views (or tabs). If you have not setup the basic infrastructure jobs or the tabs, run the scripts to do so.

Execute create-views.sh and create-lingoport-jobs.sh

This way to setup involves downloading the jenkins-cli.jar file, setting some variables, and running the script.

Download jenkins-cli.jar

If you have not already done so:

  • Navigate to your Jenkins URL followed by cli, for example http://localhost:8080/cli.
  • Click on the download link and copy the downloaded jenkins-cli.jar to a location you have access to.
  • From the '/cli' page, click on the login command in the list. An example is given on how to login Jenkins using jenkins-cli.jar. This will be useful soon.

Setup Environment Variables

If you have not already done so: Two global variables need to be setup:

  • Set JENKINS_CLI to the absolute location of the downloaded jenkins-cli.jar file, for example
export JENKINS_CLI=/home/jenkins/jenkins-cli.jar
  • Set JENKINS_URL to the URL for Jenkins, for example
export JENKINS_URL=http://locahost:8080/

Login using jenkins-cli.jar

If you have not already done so: The login command example you saw above should help you login with an account with administrative privileges. It should look like:

java -jar $JENKINS_CLI -s $JENKINS_URL login --username admin --password adminPassword

Execute the create-views.sh and create-lingoport-jobs.sh

Go to the directory where the script is located and execute the script. For example:

  • Either
cd /var/lib/jenkins/lingoport/lrm-server-x.x/jenkins/views
cd /home/jenkins/lingoport/lrm-server-x.x/jenkins/views
  • ./create-views.sh

Go to the directory where the next script is located and execute the script. For example:

  • Either
cd /var/lib/jenkins/lingoport/lrm-server-2.2/jenkins/base-jobs
cd /home/jenkins/lingoport/lrm-server-2.2/jenkins/base-jobs
  • ./create-lingoport-jobs.sh

Restart Jenkins

You may be able to do so with a browser URL such as http://localhost:8080/restart

Verify the Views and Jobs have been Created

  • Navigate to the Jenkins URL, such as http://localhost:8080/: You should see new views (tabs) and jobs in those tabs.

LRM Jobs

LRM Jenkins Jobs are setup under tabs:

Infrastructure

This tab contains Jenkins job which run independently of particular projects. See Infrastructure Jobs

Automation

This tab contains on-boarded jobs which generate the status of the project and pushes it to the dashboard, and pseudo-localizes resource files if necessary. These jobs tend to run whenever files are pushed to the repository for the project.

Nightlies

This tab contains on-boarded jobs which run every day or every night and provide an update to the dashboard and an email to a set of recipients based on the configured frequency.

PrepKit

This tab contains on-boarded jobs which are triggered by a human intervention to send the kits for translation based on the configured translation endpoint. See LRM Process Prep Kit Command

Infrastructure Tab Jobs

CheckSend

This job sweeps through unsent prep kits, located in the staging area, and tries to send them again to the configured translation endpoint, such as Lingotek, FTP, or WorldServer. It is best to keep this job triggered by a manual click so that the error is first corrected (translation group network access or authorization, etc.) before trying to resend the prep kit.

See LRM Process --check-send

FTP Jobs

FTPDownload

This job makes sure that files which needed to be downloaded from FTP 'in' have been downloaded and moved to the staging area and are ready to be imported. It runs on a regular (every hour) schedule. It calls the ftp_download.sh script.

FTPImport

This job runs based on a frequency (every 2 hours) and runs the import for FTP files stashed under the staging area.

GlobalLink Jobs

Lingotek Jobs

LingotekCallback

This job is invoked by Lingotek upon the last step of their workflow. Files are first downloaded to the download staging area and then moved to the Lingotek import staging area.

LingotekCheckCallback

This job runs once a day and moves any files left in the download staging area to the Lingotek import staging area.

LingotekImportKit

This job runs based on a frequency (every 2 hours) and runs the import on any files in the Lingotek import staging area.

Local Vendor Jobs

LocalVendorDownload

This job runs every 2 hours. It downloads the zip files that are in the expected incoming location and moves the files to the LocaleVendor import staging area.

LocalVendorImport

This job is triggered after the LocalVendorDownload completes and imports all files that are in the import staging area.

WorldServer Jobs

WorldserverDownload

When using the Worldserver protocol in the config_l10n_vendor.properties file, this job is invoked to download any zip file located in the expected incoming folder. The files are moved from the expected incoming folder into the download staging area.

WorldserverCheckDownloads

This job moves any files that are in from the download staging area into the importkit staging area.

WorldserverImport

This job runs after files have been downloaded using WSDownload. It checks the translated files for LRM correctness and pushes the translated files if they pass the tests. If they fail the tests, the files are not pushed into the corresponding repository at the correct location. A Notification email is sent in either case, success or failure

Next Steps

  • If your projects are under Git, follow those additional instructions: Git
  • If your projects are under SVN, follow these additional instructions: SVN

The overall system is setup and configured. You are now ready to on-board your first project. The first project will prepare for all subsequent LRM projects. On-Boarding New Projects

Retrieved from "http://wiki.lingoport.com/index.php?title=Localyzer_Jenkins_Jobs&oldid=96578"