GitHub Pull Requests

GitHub Pull Requests & Commits Globalyzer Analysis Webhook

Follow these steps to set up a Webhook to automate Globalyzer's analysis on your repository's pull requests and/or commits.

• Video Guide: https://vimeo.com/817611647

Jenkins Set Up

Install Globalyzer Lite

1. If not done so already, move the root Globalyzer Lite folder to your Jenkins file system. Create a `lingoport` directory at the root of your Jenkins system and place the downloaded globalyzer-lite folder there (e.g., /var/lib/jenkins/lingoport/globalyzer-lite). To avoid any errors, please do not modify or move the files in this folder unless specifically directed to do so.

2. Install Globalyzer Lite on your Jenkins system by following the instructions in the `README.txt` file (located in the globalyzer-lite folder).

Make sure to install Lite as the `jenkins` user!

Note: You should see the success message "Globalyzer Lite installation completed successfully" before proceeding.

3. You will need to download the Globalyzer.license file from here: https://www.globalyzer.com/gzserver/home/installclient (you may need to log in with your Globalyzer.com credentials). Once downloaded, place the Globalyzer.license file into the `lingoport` directory from step 1.

Set Up GitHub Credentials

4. In this folder, you will find the `github.properties` file. Your GitHub credentials will need to be set in this file before proceeding (failure to set these credentials will result in an error).

• For `github.login`, enter your GitHub username. Note this is not your GitHub login email, it is the username as it appears after you sign in.
• For `github.oauth`, you will need to enter a Personal Access Token created for your GitHub account. To generate an access token:
  • Sign in to GitHub.
  • Select your profile icon on the top right hand side and select "Settings".
  • Scroll down and select "Developer settings".
  • Select "Personal access tokens" > "Tokens (classic)".
  • Select "Generate new token" > "Generate new token (classic)".
  • Add a note and expiration date as desired.
  • Under "Select scopes", select "**repo**".
  • Select "Generate token".
  • Copy the generated token and paste it into the file. Note that the token will disappear after the window is closed/changed.

Create & Configure Jenkins Job

5. On Jenkins, navigate to "Manage Jenkins" > "Configure System" > "Global properties" > select the "Environment variables" box > select "Add".

Under name, enter `GITHUB_PRC_FILES` and for the value, enter the file path to the webhook file directory in the globalyzer-lite folder, for example: `$JENKINS_HOME/lingoport/globalyzer-lite-*.*.*_*.*/GitHub_PRC`

• Note: Whenever a new version of Lite is downloaded, only this environment variable's value needs to be changed rather than having to reconfigure every Jenkins job.

6. On Jenkins, select "New Item" and create a new "Freestyle project". Give it a name, we recommend the format "Team.RepoName-GitHub-COMMAND" where COMMAND could be PullRequest, Commit, or PRC (for both PRs and commits) depending on what you would like to be scanned.

7. Configure the project:

• General:
  • Check the "This project is parameterized" box. Two **string** parameters are needed for the job to run properly.
  • payload: The first parameter is named `payload` and the default value is `NOT_SET`. This parameter is NOT set by hand. GitHub's webhooks generate data based on the GitHub event and send the JSON data to Jenkins as the payload parameter. See the [GitHub Page](https://developer.github.com/webhooks/#payloads) for more details.
  • LITE_PROJECT_DEFINITION: The second parameter is named `LITE_PROJECT_DEFINITION` and it's default value is `DEFAULT`. This parameter is the location of the project definition file that will be used when scanning (such as: $JENKINS_HOME/jobs/$JOB_NAME/workspace/lingoport/LiteProjectDefinition.xml). If DEFAULT, it will use the lingoport/LiteProjectDefinition.xml file from the working branch of the pull request/commit. We highly recommend setting up concurrent build throttling as well to allow builds to process one at a time. Please refer to the [Throttle Concurrent Builds](#throttle-concurrent-builds) section at the bottom of the page for more details.
  • Source Code Management: You may leave this as "None". The job will locate the corresponding repo and branch through the webhook's payload data.
  • Build Triggers: Check the "Trigger builds remotely (e.g., from scripts)" box. Under "Authentication Token", enter a token to trigger the build remotely, we recommend `HOOK`. This token will be used to set up the webhook in the following section.

Note: If your job does not have the "Trigger builds remotely" option, it is likely that your Jenkins user does not have the necessary permissions.

• • Build: Select "Add build step" and select the "Execute shell" option. Under "Command", enter the code to execute the `job_lite_pr.sh` file and pass `pr_message.html` as an argument. If you haven't already, configure the `GITHUB_PRC_FILES` environment variable (refer to step 5) before proceeding.

       bash
       set +x
       $GITHUB_PRC_FILES/job_lite_pr.sh $GITHUB_PRC_FILES/pr_message.html
       

8. Select "Save" to save the project configuration. 9. Install jq on the system (if not already done): https://stedolan.github.io/jq/download/. jq is used to process the incoming webhook's JSON data.

Create The Github Webhook

10. Navigate to the repository where the webhook will be created. 11. Select "Settings" > "Webhooks" > "Add webhook".

• Note: You will need admin access to the repository to be able to access the settings.

12. Under "Payload URL", use the following URL to trigger the build remotely: `https://JENKINS_URL/buildByToken/buildWithParameters/build?job=JOB_NAME&token=TOKEN`

• `JENKINS_URL` is the URL to your Jenkins instance.
• `JOB_NAME` is the Jenkins job name you entered in step 6.
• `TOKEN` is the the token you entered in the "Build Triggers" subsection of step 7.

13. Under "Content type", select `application/x-www-form-urlencoded`. 14. Under "Which events would you like to trigger this webhook?", select "Let me select individual events" and then check "Pull Requests" and/or "Pushes" depending on if you want Globalyzer to scan pull requests, commits, or both. 15. Ensure that the webhook is set to "Active". 16. Select "Add webhook".

17. Navigate to the repository where the webhook was added. 18. Open/update a pull request or perform a commit depending on the option(s) you selected in step 14. 19. Once complete, navigate to the Jenkins job that was created in step 6. If all of the connections were completed correctly, a build should have automatically been started.

• If the build did not start automatically, navigate to the Github repo and select "Settings" > "Webhooks". Locate and select the webhook created in the last section and click "Recent Deliveries". Check for any potential issues that may have prevented the webhook from delivering the payload before trying again. Once you are ready to try again, select "Redeliver" > "Yes, redeliver this payload".
  • If you see an error that says "We couldn’t deliver this payload: failed to connect to host", there may be a proxy or firewall issue that is blocking the webhook. See GitHub's guide to GitHub IP addresses here, and check with your IT and networking departments to set up access or forwarding of GitHub's connections before trying again.
  • If you see a 403 error ("No valid crumb was included in the request"), you may need to use a Jenkins API token as an added security measure. From your Jenkins home page, select the user on the top right hand side of the page > select "Configure". Under API Tokens, select "Add New Token" > enter a name > select "Generate". Copy the token (note that it will disappear once the page is changed/closed). Navigate to the webhook created in step 16 and modify the URL to include the Jenkins credentials like so: `http://JENKINS_USERNAME:JENKINS_TOKEN@JENKINS_URL/...` (Note the : and @ characters.)
    • `JENKINS_USERNAME`: The username used to log into Jenkins. Note that this is not the full name of the user.
    • `JENKINS_TOKEN`: The API token generated from Jenkins.
  • If you see a 404 error, please double check your URL from step 12. If everything looks correct, you may need to use this URL format instead for the GitHub webhook: `https://JENKINS_URL/job/JOB_NAME/buildWithParameters?token=TOKEN`.

20. If the build...

• Finished Successfully: Navigate to the pull request/commit from step 18 and look at the comments.
  • If everything was set up correctly, you should see a new comment titled "Globalyzer Analysis Summary" with a concise summary of i18n issues detected by Globalyzer Lite. With that, you are done! Future pull requests and/or commits should automatically trigger the corresponding Jenkins job and run the Globalyzer analysis.
  • If a comment was not created on the corresponding pull request/commit, there may be an issue. Navigate to the Jenkins job > select the most recent build > select "Console Output" > locate and fix any errors that may have occurred. If there are no visible errors and the output "Globalyzer Analysis Summary added to {commit/pull-request}" is present but there is no comment, please contact support@lingoport.com for assistance.
• Failed: Select the most recent build > select "Console Output". Locate and fix any errors that may have occurred before trying step 18 again.
  • If you see the error "Host key verification failed" or "git@github.com: Permission denied (publickey). fatal: Could not read from remote repository.", you will need to create an ssh key. If you do not already have a .ssh directory (it may be hidden) on the root of your Jenkins system, create the directory before proceeding.
    • If you already have a .ssh directory, check if you have a file named `id_rsa.pub`. If so, you may skip this step. If you do not, navigate to the .ssh directory and run the command `ssh-keygen`. It will ask where you want to save the key, it should default to the .ssh directory (enter the path to the Jenkins .ssh directory if not). Enter a passphrase if desired (it will ask you to enter it twice) to proceed. Once the key has been saved, navigate to the .ssh folder and you should see two new files: `id_rsa.pub` and `id_rsa`.
    • Next, sign in to GitHub > select your profile icon on the top right hand side and select "Settings" > select "SSH and GPG keys" > select "New SSH key". Enter a title and leave the "Key type" as Authentication Key. For the "Key" field, paste the contents of the `id_rsa.pub` file and select "Add SSH Key". It is also worth making sure that GitHub is on your list of known hosts. To check, navigate to the .ssh folder on your Jenkins system and view the `known_hosts` file. If github.com is not present in the file (or the file does not exist), you will need to add it either manually with the command: `ssh-keyscan -t rsa github.com >> <PATH_TO_SSH_FOLDER>/.ssh/known_hosts` or by letting ssh do it for you when you run a git command and entering "yes" when prompted. Once complete, try step 18 again.
  • If you see the error "Error: Lite failed, GlobalyzerLicense: Cannot find license file", refer to step 3. Ensure that the file is in the `lingoport` directory (the program will attempt to look here: `/var/lib/jenkins/lingoport` for the license file). If your license has expired, please download the license again.
  • If the build finished successfully but you see an error like so: "main ERROR Unable to create file /root/.globalyzer/log/globalyzer.log", you may have installed Globalyzer Lite as the root user instead of the jenkins user (refer to step 2). To fix this, navigate to your Jenkins file system > locate the Globalyzer Lite folder > locate the `log4j2.xml` file. Modify the LOG_DIR variable like so: `<Property name="LOG_DIR">/var/lib/jenkins/.globalyzer/log/globalyzer.log</Property>`.
    • If you are using an older version of Lite, you may have a `log4j.properties` file instead. If so, you want to modify the log4j.appender.FILE.File variable like so: `log4j.appender.FILE.File=/var/lib/jenkins/.globalyzer/log/globalyzer.log`.
    • If you see the error "File /var/lib/jenkins/.globalyzer/log/globalyzer.log exists and is not a directory. Unable to create directory" after this fix, you just need to delete the existing .globalyzer directory from your Jenkins system by running: `rm -rf /var/lib/jenkins/.globalyzer`.

Optional Additions

Throttle Concurrent Builds

- We highly recommend installing the "Throttle Concurrent Builds" plugin: https://plugins.jenkins.io/throttle-concurrents/throttling if not done so already. This allows multiple webhook calls to this job to execute one after the other, rather than all at the same time. - After installing the plugin, navigate to the Jenkins job created in step 6 and select "Configure". Select the "Throttle Concurrent Builds" checkbox. Select the "Throttle this project alone" option and configure the "Maximum Total Concurrent Builds" and "Maximum Concurrent Builds Per Node", we recommend leaving both values as 1 to allow the job to process each build one at a time.

Discard Old Builds

- We also recommend installing the "Discard Old Build" Jenkins plugin: https://plugins.jenkins.io/discard-old-build/ to automatically remove old builds. This plugin is highly configurable and helps to reduce build clutter. - After installing the plugin, navigate to the Jenkins job created in step 6 and select "Configure". Under "Post-build Actions", select "Add post-build action", and select "Discard Old Builds". You can customize how the builds are discarded here (i.e., over a set number of days, keep a max number of builds, etc.).

Jenkins Upgrades

Upgrade Required Indication

When logging into Jenkins, there will be a message on the Manage Jenkins page indicating that a new version is available.

Upgrade Procedure

1. Log into the VM containing docker with a user that has sudo privileges.

2. Stop the existing jenkins/jenkins:lts-jdk11 container

  sudo docker stop <CONTAINER ID>

3. Make a backup of the existing Jenkins directory that is mounted as a volume. All Jenkins files are contained on the file system, there is no associated DB.

  $ cp -pr /home/<USER_DIR>/docker/jenkins /home/USER_DIR/docker/jenkins_<MMDDYYYY>
 

4. Pull the latest version of the jenkins/jenkins:lts-jdk11 from docker.io. Note that it will indicate a newer image has been downloaded.

       sudo docker pull jenkins/jenkins:lts-jdk11
       lts-jdk11: Pulling from jenkins/jenkins
       bd73737482dd: Pull complete
       747b9186aa97: Pull complete
       77e4b3fa9f98: Pull complete
       4185da018337: Pull complete
       fbfe8a536a91: Pull complete
       c28d8031aee7: Pull complete
       d90093f5fc84: Pull complete
       b7bf9221df23: Pull complete
       22a3392c99b4: Pull complete
       bbe8cad20a08: Pull complete
       c81331c127c6: Pull complete
       c125b4d0161b: Pull complete
       da080ad7c1d0: Pull complete
       Digest: sha256:90f7f78a7e114516216b6c0a06e00c597d6490597caec9d7bd7a95ce5c7dada0
       Status: Downloaded newer image for jenkins/jenkins:lts-jdk11
       docker.io/jenkins/jenkins:lts-jdk11

5. Start the new container with the updated image using one of the two options below depending upon whether a proxy is required to be leveraged from inside of the container or not.

Option #1 - Run the newly updated container to start Jenkins with the latest version (Command is the same as the one used initially to run)

   $ sudo docker run -d -u 1000:1000 -p 8085:8080 -p 50000:50000 \
     --env JENKINS_OPTS="--prefix=/jenkins" \
     --restart=unless-stopped \
     -v /home/centos/docker/jenkins:/var/jenkins_home jenkins/jenkins:lts-jdk11
   $
   $ sudo docker container ls
   CONTAINER ID   IMAGE                       COMMAND                  CREATED         STATUS         PORTS                                                                                      NAMES
   1924e0a6bf0d   jenkins/jenkins:lts-jdk11   "/usr/bin/tini -- /u…"   2 minutes ago   Up 2 minutes   0.0.0.0:50000->50000/tcp, :::50000->50000/tcp, 0.0.0.0:8085->8080/tcp, :::8085->8080/tcp   peaceful_snyder

Option #2 - Run the newly updated container to start Jenkins with the latest version of Jenkins when a proxy is needed for outgoing communications version (Command is the same as the one used initially to run)

   $ sudo docker run -d -u 1000:1000 -p 8085:8080 -p 50000:50000 \
     --env JENKINS_OPTS="--prefix=/jenkins" \
     --env http_proxy="http://<PROXY_URL>:<PORT>" \
     --env https_proxy="http://<PROXY_URL>:<PORT>" \
     --restart=unless-stopped \
     -v /home/centos/docker/jenkins:/var/jenkins_home jenkins/jenkins:lts-jdk11
   $
   $ sudo docker container ls
   CONTAINER ID   IMAGE                       COMMAND                  CREATED          STATUS          PORTS                                                                                      NAMES
   9e2650b5ec1c   jenkins/jenkins:lts-jdk11   "/usr/bin/tini -- /u…"   33 seconds ago   Up 32 seconds   0.0.0.0:50000->50000/tcp, :::50000->50000/tcp, 0.0.0.0:8085->8080/tcp, :::8085->8080/tcp   elegant_engelbart

6. Confirm that the new version is running properly by logging in the web interface and going to the manage jenkins section. There should not be mention of a new version being available.

7. At the bottom of this page, the current version will be displayed in the footer.

8. The old container can be removed if desired at this point.

       sudo docker rm <CONTAINER ID>

9. The old image can also be removed if desired at this point.

       sudo docker rmi <IMAGE ID>

10. The system will now be running the latest version of the Jenkins docker image and both the old images and containers will have been removed.