Difference between revisions of "GitHub Pull Requests"

From Lingoport Wiki
Jump to: navigation, search
(GitHub WebHook)
 
(84 intermediate revisions by 5 users not shown)
Line 1: Line 1:
  +
= GitHub Pull Requests & Commits Globalyzer Analysis Webhook =
= GitHub =
 
GitHub is a Web-based Git repository hosting service. It offers all of the distributed revision control and source code management (SCM) functionality of Git as well as adding its own features.
 
   
  +
Follow these steps to set up a Webhook to automate Globalyzer's analysis on your repository's pull requests and/or commits.
= Git Pull Requests =
 
Files can be committed in a Git branch or directly in Master. When committing files in a branch, pull requests let you know what changes you've made to a repository before they are committed to the repository's master branch. Once a pull request is sent, interested parties can review the set of changes, discuss potential modifications, and even push follow-up commits if necessary.
 
   
  +
* '''Video Guide:''' https://vimeo.com/817611647
= SonarQube GitHub Pull Request Plugin =
 
The SonarQube GitHub Plugin serves only one purpose: analyse GitHub pull requests in a preview mode, and inlay a summary analysis within the Pull Request's comments. This summary analysis showcases the top Globalyzer and LRM issues detected from scan. The latest version of this plugin is found [https://sonarsource.bintray.com/Distribution/sonar-github-plugin/ here]. Further documentation can be found [http://docs.sonarqube.org/display/PLUG/GitHub+Plugin here].
 
:: '''Note:''' After downloading the latest version of the plugin, place the <code>sonar-github-plugin.jar</code> file in your <code>.../Dashboard-Server/extensions/plugins</code> directory.
 
   
  +
== Jenkins Set Up ==
= Dashboard and Pull Requests =
 
Globalyzer Dashboard is based on SonarQube and leverages that platform's feature. Starting with <b>Lingoport Dashboard 5.1.2</b>, the GitHub Plugin is available. It allows to show Dashboard issues within GitHub as comments of the pull request. Globalyzer and LRM issues can then be detected and shown on the code changes before the pull request is merged with the main (master) branch. Therefore this GitHub feature is only to be used on working branches and never on master.
 
Sonar Scanner is used to push Pull Request issues to GitHub.com and to push issues on the master branch to the Globalyzer Dashboard. '''However the differences between these two workflows is very imperative to the capability of these tools! Within your ''sonar-project.properties'' file, publish mode should only be configured/run on your master branch and preview mode should only be configured/run on a working branch.'''
 
   
  +
=== Install Globalyzer Lite ===
====Example Dashboard and Pull Request Scanning Workflows====
 
  +
1. If not done so already, move the root Globalyzer Lite folder to your Jenkins file system. Create a `lingoport` directory at the root of your Jenkins system and place the downloaded globalyzer-lite folder there (e.g., /var/lib/jenkins/lingoport/globalyzer-lite). ''To avoid any errors, please do not modify or move the files in this folder unless specifically directed to do so.''
:::[[File:NewPrScanFlow2.png|caption|This diagram showcases the differences between scanning on a master branch ('''publishing''' to dashboard) and scanning on a pull request ('''previewing''' to GitHub.com).]]
 
:This diagram showcases the differences between scanning on a master branch ('''publishing''' to dashboard) and scanning on a pull request ('''previewing''' to GitHub.com).
 
   
  +
2. Install Globalyzer Lite on your Jenkins system by following the instructions in the `README.txt` file (located in the globalyzer-lite folder).
== Example of a Pull Request Analyzed with Globalyzer ==
 
   
  +
'''Make sure to install Lite as the `jenkins` user!'''
[[File:PrSummaryAnalysis.png]]
 
   
  +
'''Note:''' You should see the success message "Globalyzer Lite installation completed successfully" before proceeding.
This is pull request <b>#44</b>. The change to the code in the pull request was to add multiple embedded strings and a locale-sensitive method to the files <code>ExampleMain.java</code> & <code>ExamplePanel.java</code>. This pull request is reporting 25 different issues, however since the <code>sonar-project.properties</code> attribute is set to <code>sonar.github.disableInlineComments=true</code>, the top ten issues (sorted by severity, by component, then by line) are put into a summary analysis as a pull request comment.
 
   
  +
3. You will need to download the Globalyzer.license file from here: https://www.globalyzer.com/gzserver/home/installclient (you may need to log in with your Globalyzer.com credentials). Once downloaded, place the Globalyzer.license file into the `lingoport` directory from step 1.
Globalyzer analysis flagged these lines as critical issues. Clicking on the link next to the issue number will send the user to the line of code in that respective file.
 
   
  +
=== Set Up GitHub Credentials ===
Before merging the code back to master, the developer would typically externalize these strings, fix the locale-sensitive method issue, and re-submit the changed code / resource files for further analysis.
 
  +
4. In this folder, you will find the `github.properties` file. Your GitHub credentials will need to be set in this file before proceeding (failure to set these credentials will result in an error).
   
  +
* For `github.login`, enter your GitHub username. Note this is not your GitHub login email, it is the username as it appears after you sign in.
====External Dashboard Link====
 
  +
* For `github.oauth`, you will need to enter a Personal Access Token created for your GitHub account. To generate an access token:
  +
** Sign in to GitHub.
  +
** Select your profile icon on the top right hand side and select "Settings".
  +
** Scroll down and select "Developer settings".
  +
** Select "Personal access tokens" > "Tokens (classic)".
  +
** Select "Generate new token" > "Generate new token (classic)".
  +
** Add a note and expiration date as desired.
  +
** Under "Select scopes", select "**repo**".
  +
** Select "Generate token".
  +
** Copy the generated token and paste it into the file. Note that the token will disappear after the window is closed/changed.
   
  +
=== Create & Configure Jenkins Job ===
The GitHub pull request issues also showcase a <code>...</code> link to send the user to their dashboard server as configured in the <code>/sonar-runner-2.5.1/conf/sonar-runner.properties</code> file:
 
  +
5. On Jenkins, navigate to "Manage Jenkins" > "Configure System" > "Global properties" > select the "Environment variables" box > select "Add".
* For Example: Changing the url attribute to <code>sonar.host.url=<nowiki>http://localhost:9010</nowiki></code> (your local Globalyzer Dashboard Server) will redirect the user to the following link to give a more detailed report of this unique issue:
 
   
  +
Under name, enter `GITHUB_PRC_FILES` and for the value, enter the file path to the webhook file directory in the globalyzer-lite folder, for example: `$JENKINS_HOME/lingoport/globalyzer-lite-*.*.*_*.*/GitHub_PRC`
:::::[[File:GitHubPullRequestLink.png]]
 
   
  +
* '''Note''': Whenever a new version of Lite is downloaded, only this environment variable's value needs to be changed rather than having to reconfigure every Jenkins job.
== Pull Request Analysis Configuration ==
 
Globalyzer Dashboard uses the SonarQube sonar-project.properties file for configuration. This file should be in your repositories base directory. For the i18n analysis of the pull request, the sonar-project.properties file needs the following attributes:
 
   
  +
6. On Jenkins, select "New Item" and create a new "Freestyle project". Give it a name, we recommend the format "Team.RepoName-GitHub-COMMAND" where COMMAND could be PullRequest, Commit, or PRC (for both PRs and commits) depending on what you would like to be scanned.
* <code> sonar.analysis.mode=preview</code>
 
** It is imperative this attribute is set to preview to work. If not set, it defaults to publish mode, and could create future issues for analysis on this pull request.
 
* <code> sonar.github.login= </code>
 
** Your GitHub login name
 
* <code> sonar.github.oauth= </code>
 
** GitHub oauth is your unique GitHub personal access token: created @ https://github.com/settings/tokens
 
* <code> sonar.github.repository= </code>
 
** The repository you are working in. If repository link is https://github.com/ursulaLingoport/indexing2, this attribute should be set to <code> sonar.github.repository=ursulaLingoport/indexing2 </code>
 
* <code> sonar.github.pullRequest= </code>
 
** The number of the pull request you plan to run an i18n analysis on.
 
* <code> sonar.github.disableInlineComments= </code>
 
** If set to true (highly recommended), issues will not be reported as inline comments but only in the global analysis summary in sorted order. False by default.
 
** This Wiki page is based around this attribute being set to '''true'''. [https://github.com/ecrawfordLingoport/LiteDemo/pull/44 Here is an example of a Pull Request when this attribute is set to '''false''' (issues appear as individual inline comments within the pull request).]
 
   
  +
7. Configure the project:
===GitHub Personal Access Token Security===
 
  +
* '''General:'''
Since the sonar-project.properties file includes a personal access token (OAuth), this file cannot be pushed to a remote repository. This exposes the users personal access token to the public, and therefore will void the token. If this happens, you should receive an email from GitHub acknowledging the user of the exposure and the token's decommission. In order to avoid this, the user must avoid staging the sonar-project.properties file for commit. In order to do this, you can create a <code>.gitignore</code> file in your repository's root directory to ignore all files with the <code>.properties</code> extension. It is recommended to do this in your <code>master</code> branch, so all forked and branched repositories will not face this issue.
 
  +
** Check the "This project is parameterized" box. Two **string** parameters are needed for the job to run properly.
  +
** payload: The first parameter is named `payload` and the default value is `NOT_SET`. This parameter is NOT set by hand. GitHub's webhooks generate data based on the GitHub event and send the JSON data to Jenkins as the payload parameter. See the [GitHub Page](https://developer.github.com/webhooks/#payloads) for more details.
  +
** LITE_PROJECT_DEFINITION: The second parameter is named `LITE_PROJECT_DEFINITION` and it's default value is `DEFAULT`. This parameter is the location of the project definition file that will be used when scanning (such as: $JENKINS_HOME/jobs/$JOB_NAME/workspace/lingoport/LiteProjectDefinition.xml). If DEFAULT, it will use the lingoport/LiteProjectDefinition.xml file from the working branch of the pull request/commit. We highly recommend setting up concurrent build throttling as well to allow builds to process one at a time. Please refer to the [Throttle Concurrent Builds](#throttle-concurrent-builds) section at the bottom of the page for more details.
  +
** Source Code Management: You may leave this as "None". The job will locate the corresponding repo and branch through the webhook's payload data.
  +
** Build Triggers: Check the "Trigger builds remotely (e.g., from scripts)" box. Under "Authentication Token", enter a token to trigger the build remotely, we recommend `HOOK`. This token will be used to set up the webhook in the following section.
  +
'''Note:''' If your job does not have the "Trigger builds remotely" option, it is likely that your Jenkins user does not have the necessary permissions.
  +
** Build: Select "Add build step" and select the "Execute shell" option. Under "Command", enter the code to execute the `job_lite_pr.sh` file and pass `pr_message.html` as an argument. If you haven't already, configure the `GITHUB_PRC_FILES` environment variable (refer to step 5) before proceeding.
  +
bash
  +
set +x
  +
$GITHUB_PRC_FILES/job_lite_pr.sh $GITHUB_PRC_FILES/pr_message.html
  +
  +
8. Select "Save" to save the project configuration.
  +
9. Install jq on the system (if not already done): https://stedolan.github.io/jq/download/. jq is used to process the incoming webhook's JSON data.
   
  +
=== Create The Github Webhook ===
Here is an example <code>.gitignore</code> file that will prevent this issue from occurring:
 
  +
10. Navigate to the repository where the webhook will be created.
  +
11. Select "Settings" > "Webhooks" > "Add webhook".
  +
* '''Note:''' You will need admin access to the repository to be able to access the settings.
  +
12. Under "Payload URL", use the following URL to trigger the build remotely: `https://JENKINS_URL/buildByToken/buildWithParameters/build?job=JOB_NAME&token=TOKEN`
  +
* `JENKINS_URL` is the URL to your Jenkins instance.
  +
*`JOB_NAME` is the Jenkins job name you entered in step 6.
  +
*`TOKEN` is the the token you entered in the "Build Triggers" subsection of step 7.
  +
13. Under "Content type", select `application/x-www-form-urlencoded`.
  +
14. Under "Which events would you like to trigger this webhook?", select "Let me select individual events" and then check "Pull Requests" and/or "Pushes" depending on if you want Globalyzer to scan pull requests, commits, or both.
  +
15. Ensure that the webhook is set to "Active".
  +
16. Select "Add webhook".
   
  +
17. Navigate to the repository where the webhook was added.
[[File:GitIgnore.png]]
 
  +
18. Open/update a pull request or perform a commit depending on the option(s) you selected in step 14.
  +
19. Once complete, navigate to the Jenkins job that was created in step 6. If all of the connections were completed correctly, a build should have automatically been started.
  +
* If the build did not start automatically, navigate to the Github repo and select "Settings" > "Webhooks". Locate and select the webhook created in the last section and click "Recent Deliveries". Check for any potential issues that may have prevented the webhook from delivering the payload before trying again. Once you are ready to try again, select "Redeliver" > "Yes, redeliver this payload".
  +
** If you see an error that says "We couldn’t deliver this payload: failed to connect to host", there may be a proxy or firewall issue that is blocking the webhook. See GitHub's guide to GitHub IP addresses [https://docs.github.com/en/enterprise-cloud@latest/authentication/keeping-your-account-and-data-secure/about-githubs-ip-addresses here], and check with your IT and networking departments to set up access or forwarding of GitHub's connections before trying again.
  +
** If you see a 403 error ("No valid crumb was included in the request"), you may need to use a Jenkins API token as an added security measure. From your Jenkins home page, select the user on the top right hand side of the page > select "Configure". Under API Tokens, select "Add New Token" > enter a name > select "Generate". Copy the token (note that it will disappear once the page is changed/closed). Navigate to the webhook created in step 16 and modify the URL to include the Jenkins credentials like so: `http://JENKINS_USERNAME:JENKINS_TOKEN@JENKINS_URL/...` (Note the : and @ characters.)
  +
*** `JENKINS_USERNAME`: The username used to log into Jenkins. Note that this is not the full name of the user.
  +
*** `JENKINS_TOKEN`: The API token generated from Jenkins.
  +
** If you see a 404 error, please double check your URL from step 12. If everything looks correct, you may need to use this URL format instead for the GitHub webhook: `https://JENKINS_URL/job/JOB_NAME/buildWithParameters?token=TOKEN`.
  +
20. If the build...
  +
* '''Finished Successfully:''' Navigate to the pull request/commit from step 18 and look at the comments.
  +
** If everything was set up correctly, you should see a new comment titled "Globalyzer Analysis Summary" with a concise summary of i18n issues detected by Globalyzer Lite. With that, '''you are done!''' Future pull requests and/or commits should automatically trigger the corresponding Jenkins job and run the Globalyzer analysis.
  +
** If a comment was not created on the corresponding pull request/commit, there may be an issue. Navigate to the Jenkins job > select the most recent build > select "Console Output" > locate and fix any errors that may have occurred. If there are no visible errors and the output "Globalyzer Analysis Summary added to {commit/pull-request}" is present but there is no comment, please contact support@lingoport.com for assistance.
  +
* '''Failed:''' Select the most recent build > select "Console Output". Locate and fix any errors that may have occurred before trying step 18 again.
  +
** If you see the error "Host key verification failed" or "git@github.com: Permission denied (publickey). fatal: Could not read from remote repository.", you will need to create an ssh key. If you do not already have a .ssh directory (it may be hidden) on the root of your Jenkins system, create the directory before proceeding.
  +
*** If you already have a .ssh directory, check if you have a file named `id_rsa.pub`. If so, you may skip this step. If you do not, navigate to the .ssh directory and run the command `ssh-keygen`. It will ask where you want to save the key, it should default to the .ssh directory (enter the path to the Jenkins .ssh directory if not). Enter a passphrase if desired (it will ask you to enter it twice) to proceed. Once the key has been saved, navigate to the .ssh folder and you should see two new files: `id_rsa.pub` and `id_rsa`.
  +
*** Next, sign in to GitHub > select your profile icon on the top right hand side and select "Settings" > select "SSH and GPG keys" > select "New SSH key". Enter a title and leave the "Key type" as Authentication Key. For the "Key" field, paste the contents of the `id_rsa.pub` file and select "Add SSH Key". It is also worth making sure that GitHub is on your list of known hosts. To check, navigate to the .ssh folder on your Jenkins system and view the `known_hosts` file. If github.com is not present in the file (or the file does not exist), you will need to add it either manually with the command: `ssh-keyscan -t rsa github.com >> <PATH_TO_SSH_FOLDER>/.ssh/known_hosts` or by letting ssh do it for you when you run a git command and entering "yes" when prompted. Once complete, try step 18 again.
  +
** If you see the error "Error: Lite failed, GlobalyzerLicense: Cannot find license file", refer to step 3. Ensure that the file is in the `lingoport` directory (the program will attempt to look here: `/var/lib/jenkins/lingoport` for the license file). If your license has expired, please download the license again.
  +
** If the build finished successfully but you see an error like so: "main ERROR Unable to create file /root/.globalyzer/log/globalyzer.log", you may have installed Globalyzer Lite as the root user instead of the jenkins user (refer to step 2). To fix this, navigate to your Jenkins file system > locate the Globalyzer Lite folder > locate the `log4j2.xml` file. Modify the LOG_DIR variable like so: `<Property name="LOG_DIR">/var/lib/jenkins/.globalyzer/log/globalyzer.log</Property>`.
  +
*** If you are using an older version of Lite, you may have a `log4j.properties` file instead. If so, you want to modify the log4j.appender.FILE.File variable like so: `log4j.appender.FILE.File=/var/lib/jenkins/.globalyzer/log/globalyzer.log`.
  +
*** If you see the error "File /var/lib/jenkins/.globalyzer/log/globalyzer.log exists and is not a directory. Unable to create directory" after this fix, you just need to delete the existing .globalyzer directory from your Jenkins system by running: `rm -rf /var/lib/jenkins/.globalyzer`.
   
=== GitHub WebHook ===
+
=== Optional Additions ===
The Pull Request on GitHub needs to trigger a Web hook back to Lingoport's Jenkins instance. To do so:
 
* Navigate to the GitHub project as an 'Administer' account
 
* In Settings for the GitHub project (available if the account has Administer permissions), add a Webhook
 
* The Web hook should follow this pattern:
 
   
  +
==== Throttle Concurrent Builds ====
http://JENKINS_URL/jenkins/buildByToken/buildWithParameters/build?job=GROUP_NAME.PROJECT_NAME-PullRequest&token=HOOK
 
  +
- We highly recommend installing the "Throttle Concurrent Builds" plugin: https://plugins.jenkins.io/throttle-concurrents/throttling if not done so already. This allows multiple webhook calls to this job to execute one after the other, rather than all at the same time.
  +
- After installing the plugin, navigate to the Jenkins job created in step 6 and select "Configure". Select the "Throttle Concurrent Builds" checkbox. Select the "Throttle this project alone" option and configure the "Maximum Total Concurrent Builds" and "Maximum Concurrent Builds Per Node", we recommend leaving both values as 1 to allow the job to process each build one at a time.
   
  +
==== Discard Old Builds ====
application/x-www-form-urlencoded
 
  +
- We also recommend installing the "Discard Old Build" Jenkins plugin: https://plugins.jenkins.io/discard-old-build/ to automatically remove old builds. This plugin is highly configurable and helps to reduce build clutter.
 
  +
- After installing the plugin, navigate to the Jenkins job created in step 6 and select "Configure". Under "Post-build Actions", select "Add post-build action", and select "Discard Old Builds". You can customize how the builds are discarded here (i.e., over a set number of days, keep a max number of builds, etc.).
Pull Request event selected
 
 
Active
 
 
For example:
 
 
[[File:GitHubWebhook.png|300px]]
 
 
* Test the Web hook and especially check for firewall issues going back to the Jenkins system
 
 
=== Sample sonar-project.properties file: ===
 
 
For instance, here is a test sonar-project.properties with some properties modified for security reasons:
 
<pre>
 
sonar.projectKey=Lingoport.indexing2:scan
 
sonar.projectName=Lingoport.indexing2
 
sonar.projectVersion=5.1
 
sonar.sources=C:/Users/Joe/Documents/GitHub/indexing2
 
sonar.importSources=true
 
sonar.lingoport.project.root=.
 
sonar.language=lport
 
 
#sonar.lingoport.extensions=as,mxml,asp,asax,ascx,ashx,aspx,awk,c,c++,cc,cpp,cxx,h,hpp,hxx,sqc,sqx,qml,cs,dfm,dpk,dpr,pas,java,jsp,jspf,js,perl,pl,pm,plx,inc,php,sql,sqc,sqx,bas,cls,ctl,dob,
 
dsr,frm,pag,vb,vbs,htm,html,shtml,vm,html5,xml,xsd,xsl,xslt,wsdl,wsdd,rmd,tld,xliff,xul,xhtml,mxml,jspx,xaml,css,cat,resx,msg,po,properties,rc,rc2,strings,json,ts,rxml,rjs,properties
 
 
#
 
# For the GitHub SonarQube Plugin
 
#
 
sonar.analysis.mode=preview
 
sonar.github.login=ursulaLingoport
 
sonar.github.oauth=d60bbe9d6d3c7caa995f5119ff2997b26d0ef191
 
sonar.github.repository=ursulaLingoport/indexing2
 
sonar.github.pullRequest=6
 
sonar.github.disableInlineComments=true
 
</pre>
 
 
===Sonar-Runner and GlobalyzerLite Paths/Aliases===
 
Instead of having to call the path of your respective program/script each time (I.E. - <code>java -jar "C:\lingoport\globalyzer-lite-4.8.5\globalyzer-lite.jar" GzProjectDefinition.xml</code>), you can create command-line aliases to run these commands quickly and more efficiently (I.E. - <code>lite GzProjectDefinition.xml</code>)
 
 
====Windows====
 
*Create an alias for GlobalyzerLite (running the doskey command is a quick way to create an alias on windows)
 
**<code>doskey lite=java -jar C:\lingoport\globalyzer-lite-4.8.5\globalyzer-lite.jar</code>
 
*Add sonar-runner to your path if you have not already, to allow the scanner to be run within any directory with the simple command <code>sonar-runner</code>
 
**<code>set PATH=%PATH%;C:\path\to\sonar-scanner-2.5.1</code>
 
 
====Unix====
 
*Create an alias for GlobalyzerLite (this command adds the alias to the end of your .bashrc file)
 
**<code>echo 'alias lite="java -jar /destination/to/globalyzer-lite-5.0/globalyzer-lite.jar" ' >> ~/.bashrc</code>
 
*Add sonar-runner to your path if you have not already, to allow the scanner to be run within any directory with the simple command <code>sonar-runner</code>
 
**<code>echo 'export PATH=$PATH:/etc/sonar-scanner-2.5.1/bin' >> ~/.bashrc</code>
 
 
 
===Globalyzer Lite Project Definition File===
 
In order to successfully scan the project directory for analysis, the <code>ProjectDefinition.xml</code> file needs to be correctly configured in your project's root directory.
 
:[https://www.globalyzer.com/gzserver/help/referenceLite/project-definition-file-overview.html For help on this setup, click here.]
 
 
== Running the Analysis ==
 
* Checkout code from a branch (not master) or create a new branch
 
**<code> git checkout <-b> workingBranchName </code> (use the -b flag to create a new branch)
 
* Modify the code in your working branch
 
* Add, commit & push the code to the branch (not master)
 
** <code>git add .</code>
 
** <code>git commit -am "Commit Message"</code>
 
** <code>git push origin workingBranchName </code>
 
* Create a pull request on GitHub.com (your working branch --> master)
 
* Make sure <kbd>sonar.github.pullRequest=PullRequest#</kbd> within the project's sonar-project.properties file.
 
* Run a couple of commands:
 
** <code>java -jar "c:\lingoport\globalyzer-lite-4.8.5\globalyzer-lite.jar" GzProjectDefinition.xml</code> or <code>lite GzProjectDefinition.xml</code> (if you set an alias)
 
** <code>C:\sonar\sonar-runner-2.4-local\bin\sonar-runner</code> or <code>sonar-runner</code> (if you added sonar-runner to the machines PATH)
 
* Check the pull request comments on GitHub: The i18n issues found on the code in the pull request will be shown.
 
 
:::<b>Note</b>: The command <code>java -jar "c:\lingoport\globalyzer-lite-4.8.5\globalyzer-lite.jar" GzProjectDefinition.xml</code> is used to create a Globalyzer report under the directory <code>GlobalyzerScans</code>. It could be any commands to create that Globalyzer report. The sonar-runner then needs to be executed from above the <code>GlobalyzerScans</code> directory.
 
 
== Additional Information ==
 
http://stackoverflow.com/questions/32047585/jenkins-sonar-github-integration
 

Latest revision as of 16:28, 12 December 2023

GitHub Pull Requests & Commits Globalyzer Analysis Webhook

Follow these steps to set up a Webhook to automate Globalyzer's analysis on your repository's pull requests and/or commits.

Jenkins Set Up

Install Globalyzer Lite

1. If not done so already, move the root Globalyzer Lite folder to your Jenkins file system. Create a `lingoport` directory at the root of your Jenkins system and place the downloaded globalyzer-lite folder there (e.g., /var/lib/jenkins/lingoport/globalyzer-lite). To avoid any errors, please do not modify or move the files in this folder unless specifically directed to do so.

2. Install Globalyzer Lite on your Jenkins system by following the instructions in the `README.txt` file (located in the globalyzer-lite folder).

Make sure to install Lite as the `jenkins` user!

Note: You should see the success message "Globalyzer Lite installation completed successfully" before proceeding.

3. You will need to download the Globalyzer.license file from here: https://www.globalyzer.com/gzserver/home/installclient (you may need to log in with your Globalyzer.com credentials). Once downloaded, place the Globalyzer.license file into the `lingoport` directory from step 1.

Set Up GitHub Credentials

4. In this folder, you will find the `github.properties` file. Your GitHub credentials will need to be set in this file before proceeding (failure to set these credentials will result in an error).

  • For `github.login`, enter your GitHub username. Note this is not your GitHub login email, it is the username as it appears after you sign in.
  • For `github.oauth`, you will need to enter a Personal Access Token created for your GitHub account. To generate an access token:
    • Sign in to GitHub.
    • Select your profile icon on the top right hand side and select "Settings".
    • Scroll down and select "Developer settings".
    • Select "Personal access tokens" > "Tokens (classic)".
    • Select "Generate new token" > "Generate new token (classic)".
    • Add a note and expiration date as desired.
    • Under "Select scopes", select "**repo**".
    • Select "Generate token".
    • Copy the generated token and paste it into the file. Note that the token will disappear after the window is closed/changed.

Create & Configure Jenkins Job

5. On Jenkins, navigate to "Manage Jenkins" > "Configure System" > "Global properties" > select the "Environment variables" box > select "Add".

Under name, enter `GITHUB_PRC_FILES` and for the value, enter the file path to the webhook file directory in the globalyzer-lite folder, for example: `$JENKINS_HOME/lingoport/globalyzer-lite-*.*.*_*.*/GitHub_PRC`

  • Note: Whenever a new version of Lite is downloaded, only this environment variable's value needs to be changed rather than having to reconfigure every Jenkins job.

6. On Jenkins, select "New Item" and create a new "Freestyle project". Give it a name, we recommend the format "Team.RepoName-GitHub-COMMAND" where COMMAND could be PullRequest, Commit, or PRC (for both PRs and commits) depending on what you would like to be scanned.

7. Configure the project:

  • General:
    • Check the "This project is parameterized" box. Two **string** parameters are needed for the job to run properly.
    • payload: The first parameter is named `payload` and the default value is `NOT_SET`. This parameter is NOT set by hand. GitHub's webhooks generate data based on the GitHub event and send the JSON data to Jenkins as the payload parameter. See the [GitHub Page](https://developer.github.com/webhooks/#payloads) for more details.
    • LITE_PROJECT_DEFINITION: The second parameter is named `LITE_PROJECT_DEFINITION` and it's default value is `DEFAULT`. This parameter is the location of the project definition file that will be used when scanning (such as: $JENKINS_HOME/jobs/$JOB_NAME/workspace/lingoport/LiteProjectDefinition.xml). If DEFAULT, it will use the lingoport/LiteProjectDefinition.xml file from the working branch of the pull request/commit. We highly recommend setting up concurrent build throttling as well to allow builds to process one at a time. Please refer to the [Throttle Concurrent Builds](#throttle-concurrent-builds) section at the bottom of the page for more details.
    • Source Code Management: You may leave this as "None". The job will locate the corresponding repo and branch through the webhook's payload data.
    • Build Triggers: Check the "Trigger builds remotely (e.g., from scripts)" box. Under "Authentication Token", enter a token to trigger the build remotely, we recommend `HOOK`. This token will be used to set up the webhook in the following section.

Note: If your job does not have the "Trigger builds remotely" option, it is likely that your Jenkins user does not have the necessary permissions.

    • Build: Select "Add build step" and select the "Execute shell" option. Under "Command", enter the code to execute the `job_lite_pr.sh` file and pass `pr_message.html` as an argument. If you haven't already, configure the `GITHUB_PRC_FILES` environment variable (refer to step 5) before proceeding.
       bash
       set +x
       $GITHUB_PRC_FILES/job_lite_pr.sh $GITHUB_PRC_FILES/pr_message.html
       

8. Select "Save" to save the project configuration. 9. Install jq on the system (if not already done): https://stedolan.github.io/jq/download/. jq is used to process the incoming webhook's JSON data.

Create The Github Webhook

10. Navigate to the repository where the webhook will be created. 11. Select "Settings" > "Webhooks" > "Add webhook".

  • Note: You will need admin access to the repository to be able to access the settings.

12. Under "Payload URL", use the following URL to trigger the build remotely: `https://JENKINS_URL/buildByToken/buildWithParameters/build?job=JOB_NAME&token=TOKEN`

  • `JENKINS_URL` is the URL to your Jenkins instance.
  • `JOB_NAME` is the Jenkins job name you entered in step 6.
  • `TOKEN` is the the token you entered in the "Build Triggers" subsection of step 7.

13. Under "Content type", select `application/x-www-form-urlencoded`. 14. Under "Which events would you like to trigger this webhook?", select "Let me select individual events" and then check "Pull Requests" and/or "Pushes" depending on if you want Globalyzer to scan pull requests, commits, or both. 15. Ensure that the webhook is set to "Active". 16. Select "Add webhook".

17. Navigate to the repository where the webhook was added. 18. Open/update a pull request or perform a commit depending on the option(s) you selected in step 14. 19. Once complete, navigate to the Jenkins job that was created in step 6. If all of the connections were completed correctly, a build should have automatically been started.

  • If the build did not start automatically, navigate to the Github repo and select "Settings" > "Webhooks". Locate and select the webhook created in the last section and click "Recent Deliveries". Check for any potential issues that may have prevented the webhook from delivering the payload before trying again. Once you are ready to try again, select "Redeliver" > "Yes, redeliver this payload".
    • If you see an error that says "We couldn’t deliver this payload: failed to connect to host", there may be a proxy or firewall issue that is blocking the webhook. See GitHub's guide to GitHub IP addresses here, and check with your IT and networking departments to set up access or forwarding of GitHub's connections before trying again.
    • If you see a 403 error ("No valid crumb was included in the request"), you may need to use a Jenkins API token as an added security measure. From your Jenkins home page, select the user on the top right hand side of the page > select "Configure". Under API Tokens, select "Add New Token" > enter a name > select "Generate". Copy the token (note that it will disappear once the page is changed/closed). Navigate to the webhook created in step 16 and modify the URL to include the Jenkins credentials like so: `http://JENKINS_USERNAME:JENKINS_TOKEN@JENKINS_URL/...` (Note the : and @ characters.)
      • `JENKINS_USERNAME`: The username used to log into Jenkins. Note that this is not the full name of the user.
      • `JENKINS_TOKEN`: The API token generated from Jenkins.
    • If you see a 404 error, please double check your URL from step 12. If everything looks correct, you may need to use this URL format instead for the GitHub webhook: `https://JENKINS_URL/job/JOB_NAME/buildWithParameters?token=TOKEN`.

20. If the build...

  • Finished Successfully: Navigate to the pull request/commit from step 18 and look at the comments.
    • If everything was set up correctly, you should see a new comment titled "Globalyzer Analysis Summary" with a concise summary of i18n issues detected by Globalyzer Lite. With that, you are done! Future pull requests and/or commits should automatically trigger the corresponding Jenkins job and run the Globalyzer analysis.
    • If a comment was not created on the corresponding pull request/commit, there may be an issue. Navigate to the Jenkins job > select the most recent build > select "Console Output" > locate and fix any errors that may have occurred. If there are no visible errors and the output "Globalyzer Analysis Summary added to {commit/pull-request}" is present but there is no comment, please contact support@lingoport.com for assistance.
  • Failed: Select the most recent build > select "Console Output". Locate and fix any errors that may have occurred before trying step 18 again.
    • If you see the error "Host key verification failed" or "git@github.com: Permission denied (publickey). fatal: Could not read from remote repository.", you will need to create an ssh key. If you do not already have a .ssh directory (it may be hidden) on the root of your Jenkins system, create the directory before proceeding.
      • If you already have a .ssh directory, check if you have a file named `id_rsa.pub`. If so, you may skip this step. If you do not, navigate to the .ssh directory and run the command `ssh-keygen`. It will ask where you want to save the key, it should default to the .ssh directory (enter the path to the Jenkins .ssh directory if not). Enter a passphrase if desired (it will ask you to enter it twice) to proceed. Once the key has been saved, navigate to the .ssh folder and you should see two new files: `id_rsa.pub` and `id_rsa`.
      • Next, sign in to GitHub > select your profile icon on the top right hand side and select "Settings" > select "SSH and GPG keys" > select "New SSH key". Enter a title and leave the "Key type" as Authentication Key. For the "Key" field, paste the contents of the `id_rsa.pub` file and select "Add SSH Key". It is also worth making sure that GitHub is on your list of known hosts. To check, navigate to the .ssh folder on your Jenkins system and view the `known_hosts` file. If github.com is not present in the file (or the file does not exist), you will need to add it either manually with the command: `ssh-keyscan -t rsa github.com >> <PATH_TO_SSH_FOLDER>/.ssh/known_hosts` or by letting ssh do it for you when you run a git command and entering "yes" when prompted. Once complete, try step 18 again.
    • If you see the error "Error: Lite failed, GlobalyzerLicense: Cannot find license file", refer to step 3. Ensure that the file is in the `lingoport` directory (the program will attempt to look here: `/var/lib/jenkins/lingoport` for the license file). If your license has expired, please download the license again.
    • If the build finished successfully but you see an error like so: "main ERROR Unable to create file /root/.globalyzer/log/globalyzer.log", you may have installed Globalyzer Lite as the root user instead of the jenkins user (refer to step 2). To fix this, navigate to your Jenkins file system > locate the Globalyzer Lite folder > locate the `log4j2.xml` file. Modify the LOG_DIR variable like so: `<Property name="LOG_DIR">/var/lib/jenkins/.globalyzer/log/globalyzer.log</Property>`.
      • If you are using an older version of Lite, you may have a `log4j.properties` file instead. If so, you want to modify the log4j.appender.FILE.File variable like so: `log4j.appender.FILE.File=/var/lib/jenkins/.globalyzer/log/globalyzer.log`.
      • If you see the error "File /var/lib/jenkins/.globalyzer/log/globalyzer.log exists and is not a directory. Unable to create directory" after this fix, you just need to delete the existing .globalyzer directory from your Jenkins system by running: `rm -rf /var/lib/jenkins/.globalyzer`.

Optional Additions

Throttle Concurrent Builds

- We highly recommend installing the "Throttle Concurrent Builds" plugin: https://plugins.jenkins.io/throttle-concurrents/throttling if not done so already. This allows multiple webhook calls to this job to execute one after the other, rather than all at the same time. - After installing the plugin, navigate to the Jenkins job created in step 6 and select "Configure". Select the "Throttle Concurrent Builds" checkbox. Select the "Throttle this project alone" option and configure the "Maximum Total Concurrent Builds" and "Maximum Concurrent Builds Per Node", we recommend leaving both values as 1 to allow the job to process each build one at a time.

Discard Old Builds

- We also recommend installing the "Discard Old Build" Jenkins plugin: https://plugins.jenkins.io/discard-old-build/ to automatically remove old builds. This plugin is highly configurable and helps to reduce build clutter. - After installing the plugin, navigate to the Jenkins job created in step 6 and select "Configure". Under "Post-build Actions", select "Add post-build action", and select "Discard Old Builds". You can customize how the builds are discarded here (i.e., over a set number of days, keep a max number of builds, etc.).