Pull Requests

Pull Requests, Merge Requests code changes can be analyzed using the Globalyzer Scanner.

GitHub Pull Request

See:

• https://vimeo.com/817611647

GitLab Pull Request

See:

• https://vimeo.com/817610534

Bitbucket Pull Request

See:

• https://vimeo.com/826230267

To Be Edited

• GitHub Pull Requests
• Distributed Jenkins

The Lingoport Suite offers two options for Pull Requests from your repository. The Pull Request can be done on your Lingoport Server or there is a distributed option that uses the Master/Agent capability of Jenkins to distribute the work among several nodes.

To use the Distributed Pull Request option, please install the Node Installer optional software.

GitHub and Git Pull Requests

GitHub is a Web-based Git repository hosting service. It offers all of the distributed revision control and source code management (SCM) functionality of Git as well as adding its own features.

Files can be committed in a Git branch or directly in the Master branch. When committing files in a branch, pull requests let you know what changes you've made to a repository before they are committed to the repository's master branch. Once a pull request is sent, interested parties can review the set of changes, discuss potential modifications, and even push follow-up commits if necessary.

Configuring Pull Requests

Verify the Lingoport Dashboard Plugin

The Pull Requests rely on the Dashboard GitHub plugin. If your system was installed using the Stack Installer, this plugin should have been included. Verify that it is installed.

1. Go to the Lingoport Dashboard and log in as Administrator.
2. Select the Administration tab at the top of the window.
3. Select the System pulldown and Update Center
4. With Installed selected, search for GitHub. You should find a plugin with the description: "Provide some integration between GitHub and SonarQube"

If you do not find the plugin installed, select Available, search for 'GitHub' again and Install it.

Set up Nodes on Jenkins

If you are using the Distributed Pull Requests option, the first thing that needs to be done is to set up nodes in Jenkins. If you installed the Node Installer and followed its setup instructions, there should be a master system (where Jenkins and Lingoport Dashboard are hosted) and at least one agent or node system. The master system should be able to ssh into the agents' centos user and the jenkins user. If this is not completed, make sure that is done before proceeding further.

If you are not using nodes and agents, skip to the PullRequests section.

Create Nodes in Jenkins

On the Master system, in Jenkins, select Manage Jenkins → Manage Nodes → New Node

Set up each agent node using this example:

Select Add to create the jenkins ssh credentials

Nothing needs to be done with the master node. Online the new nodes and the console output should look like:

[01/26/18 16:26:19] [SSH] Opening SSH connection to ec2-54-227-210-213.compute-1.amazonaws.com:22.
[01/26/18 16:26:19] [SSH] SSH host key matches key in Known Hosts file. Connection will be allowed.
[01/26/18 16:26:19] [SSH] Authentication successful.
[01/26/18 16:26:19] [SSH] The remote user's environment is:
BASH=/usr/bin/bash 
BASHOPTS=cmdhist:extquote:force_fignore:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath
....

Verify the Agent setup in Jenkins

To verify that the Agent setup is correct, use the Jenkins 'Debug' job.

Under General check Restrict where this project can be run and set it to lingoport.

In the Execute Shell leave only:

whoami
pwd
sleep 5

Run the job. The console output should show:

Started by user anonymous
Building remotely on Agent1 (lingoport) in workspace /var/lib/jenkins/jenkins_slave/workspace/Debug
[Debug] $ /bin/sh -xe /tmp/jenkins7953464360000549139.sh
+ whoami
jenkins
+ pwd
/var/lib/jenkins/jenkins_slave/workspace/Debug
+ sleep 5

Note that it shows that it ran on the Agent using the lingoport label.

Jenkins PullRequest and DistributedPullRequest jobs

There are two Jenkins job templates for Distributed Pull Requests.

• Lingoport.SampleLite-PullRequest
• Lingoport.SampleLite-DistributedPullRequest

The Distributed Pull Request makes use of the Jenkins master/agent system.

Create a new Jenkins job using the Lingoport.SampleLite-PullRequest or Lingoport.SampleLite-DistributedPullRequest template. This should be modeled on an existing Jenkins project.

For example, if you have a Jenkins project called Acme.BigProject and you want to use PullRequests with this project and repository, name the new project Acme.BigProject-PullRequest.

Configuring the PullRequest Jenkins job

Additional Configuration

• /var/lib/jenkins/Lingoport_Data/Dashboard/github.properties

This file tells what username/token in GitHub will be used to push the data over. It should have three key/value pairs:

 sonar.github.login=<myGitLogin>
 sonar.github.oauth=<created token>
 sonar.host.url=<http(s)://<dashboard url>/

• General: GitHub project / project Url

In the Source Code Management section of the Jenkins job configuration, the GitHub 'project url' text field needs a GitHub URL in the form

 https://github.com/<organization>/<project>

The Jenkins Job

Once you have created the new PullRequest project, some configurations need to be modified.

1. Make sure the name is <Group>.<Project>-PullRequest. The <Group>.<Project> Jenkins job should already exist and have been executed. Anything other than <Group>.<Project>-PullRequest may cause problems.
2. Leave the payload parameter alone. Do not modify it.
3. If nodes have been set up for other projects, select Restrict where this project can be run and set it to master. If nodes have NOT been set up, nothing needs to be done with this option.
4. Under Build Triggers, select Trigger builds remotely (e.g., from scripts) and set the value to HOOK.
5. Under Build, and Execute Shell, set the Project Definition File for <Group>.<Project>

   LITE_PROJECT_DEFINITION=/var/lib/jenkins/jobs/<Group>.<Project>/workspace/lingoport/LiteProjectDefinition.xml 
   

6. Save the configuration and Enable the project.

From the command line console, in /var/lib/jenkins/Lingoport_Data/Dashboard/Projects/GROUPNAME.PROJECTNAME-PullRequest/ (this may need to be created), make a sonar-project-pr.properties file that looks like:

# Make sure a Dashboard user is set up with the credential below
# otherwise the dashboard will not accept the pushed data
#
# -------Using $DASHBOARD_HOME/sonar-project-template.properties
#
sonar.login=admin
sonar.password=<admin password>
#
# Do not change the properties below
# the capital letter tokens will be replaced by
# build scripts
#
sonar.projectKey=PullRequest:scan
sonar.projectName=PullRequest
sonar.projectVersion=PR

sonar.language=lport
sonar.encoding=UTF-8

sonar.sources=/var/lib/jenkins/jobs/<Group>.<Project>-PullRequest/workspace
sonar.lingoport.project.root=/var/lib/jenkins/Lingoport_Data/Dashboard/Projects/<Group>.<Project>-PullRequest

#
# For the GitHub SonarQube Plugin to push the results to GitHub PR
# The same for all:
#
# -------Using $DASHBOARD_HOME/sonar-project-pr-template.properties
#

sonar.analysis.mode=preview
sonar.github.endpoint=https://api.github.com
sonar.github.disableInlineComments=true

# Specific for this install

# Specific to this GitHub Project
# If the github.properties file does not exist, the job will not push 
# the data to GitHub
#
# -------Using $LINGOPORT_DATA/Dashboard/github.properties
#
sonar.github.login=<myGitLogin>
sonar.github.oauth=<created token>
sonar.host.url=<http(s)://<dashboard url>/

# Generated from the Jenkins setting under 
# ------ General: GitHub project / project Url
sonar.github.repository=<Repo> #for example: NASA/nasa-latex-blocks
# Generated from the Webhook payload 
sonar.github.pullRequest=<pr number>

Configuring the DistributedPullRequest Jenkins job

Once you have created the new DistributedPullRequest project, some configurations need to be modified. The DistributedPullRequest should be based on a <Group>.<Project> Jenkins job that has been executed.

1. Leave the payload parameter alone
2. Set the GIT_REPO parameter. This should be set to the form Company/Project as seen on GitHub. For example, in GitHub, NASA has many projects so the GIT_REPO would be set to NASA/nasa-latex-blocks
3. Set the LITE_PROJECT_DEFINITION parameter to the value on the Agent. $JENKINS_HOME/jenkins_slave/workspace/<Group>.<Project>-DistributedPullRequest/lingoport/LiteProjectDefinition.xml
4. Select Restrict where this project can be run and set it to lingoport.
5. Under Build Triggers, select Trigger builds remotely (e.g., from scripts) and set the value to HOOK.
6. Save the configuration and Enable the project.

Configuring GitHub for PullRequests

Some configuration needs to be done on GitHub for the PullRequests to occur.

In the GitHub page for your Project:

1. Select Settings at the top of the window.
2. Select the Webhooks tab in the left menu.
3. Select the Add webhook button.

1. Set the Payload URL to

   http://<Jenkins IP>/buildByToken/buildWithParameters?job=<Group>.<Project>-PullRequest&token=HOOK

   for PullRequest jobs or

   http://<Jenkins IP>/buildByToken/buildWithParameters?job=<Group>.<Project>-DistributedPullRequest&token=HOOK

   for DistributedPullRequest jobs.
2. Make sure the Content Type is application/x-www-form-urlencoded
3. Leave Secret blank
4. Which events would you like to trigger this webhook? Select: Let me select individual events and check the Pull Request item.
5. Select Active.
6. Update or Add Webhook

If everything is correct, then after adding the Webhook, the Jenkins job <Group>.<Project>-PullRequest or <Group>.<Project>-DistributedPullRequest should be triggered. The job may FAIL, but at this point the verifying the communication between GitHub and the Jenkins job is a success.

Verify the PullRequest

To test that the Pull Request works,

1. Use or create a new branch in the Git Project with changes to internationalization issues.
2. In GitHub, select the Pull Requests tab
3. Select the New Pull Request button
4. Select the two branches you wish to compare. You should see the differences on the screen between the two branches.
5. Select the Create New Pull Request button and enter a comment.
6. Select Create Pull Request

Return to the Jenkins Pull Request job. It should be triggered and running. If it is successful, return to your Pull Request tab in GitHub and you should see something like:

Pull Requests via Jenkins Running on Docker

THIS SECTION IS CURRENTLY A WORK IN PROGRESS AN NOT YET COMPLETE

Prerequisites

1. VM running Linux and Docker
2. JQ JSON Parser - Statically compiled binary version
   1. https://stedolan.github.io/jq/
3. Github PAT credentials to connect to GitHub
   1. The user for these PAT credentials has read access to the repository
4. Globalyzer.com credentials for logging in to globalyzer.com
   1. username
   2. password
   3. server
5. Optional:
   1. Apache or nginx could be installed and running on the VM itself
   2. Used as a reverse proxy to front the Jenkins instance

File System Configuration

If Apache is installed as per normal for many Lingoport systems, the following is required to configure Apache to permit proper connection to the Jenkins Docker container

1. Copy the tar file to the home directory of the user that will be running docker and unzip
2. Identify the port that will be used to connect to the Jenkins instance in the container. It could be something like 8085 for example. That is what will be used in this instruction set.
3. Obtain the UID/GID for the user that will be running docker. Note this user must have sudo privileges.
4. Create the directory structure for the volume storage used by jenkins. Be sure there is sufficient space. Base install will consume 650MB of space for the volume itself.
5. Place files as noted below before installing
6. Be sure that the jq binary is executable once copied
7. Create the .globalyzerrc file and populate it with the credentials for logging into globalyzer.com or the globalyzer server that will be connected to.

Here are the steps from the terminal.

 
   mkdir ~/docker/jenkins/lingoport/bin
   pp ~/PRC-Docker/Globalyzer.license ~/docker/jenkins/lingoport/
   cp ~/PRC-Docker/globalyzer-lite-VERSION.zip ~/docker/jenkins/lingoport/
   cp -p  ~/PRC-Docker/jq ~/docker/jenkins/lingoport/bin/
   cd ~/docker/jenkins/lingoport
   unzip globalyzer-lite-VERSION.zip
   rm /globalyzer-lite-VERSION.zip
   cd globalyzer-lite/GitHub_PRC
   cp -p job_lite_pr.sh job_lite_pr.sh_orig
   cp ~/PRC-Docker/job_lite_pr.sh.tar .
   tar -xf job_lite_pr.sh.tar
   rm job_lite_pr.sh.tar
   cd ~/docker/jenkins
   vi .globalyzerrc
 

Launching the Jenkins Docker Container

Given the file system has been created and UID/GID are known as well as the port that will be used for connecting to the Jenkins instance. Start the docker instance for Jenkins.

• The -d detaches the command being run from the terminal. Normally this is something that will only be used once everything is in place and confirmed to be working. For initial startup, its use can be ignored.
• The -u is for setting the uid/gid of the user that owns the volume storage identified earlier. This user should also be the user that is starting the docker instance.
• The -p 8085:8080 is the port external to internal mapping identified earlier. The 8080 would never change, only the external port is modifiable
• The -p 50000:50000 is for mapping between Jenkins instances only. Not required for standalone systems.
• The JENKINS_OPTS sets the baseurl for the application
• The -v mounts the prior locally created file system to the running docker instance as a volume and mas it.
• The jenkins/jenkins:lts-jdk11 is the long term support image for Jenkins. Currently, it is Jenkins 2.387.3

The below command string will create a new container instance that will restart the container unless specifically stopped. It will reuse the existing container on a restart of the OS. If the container is stopped before the reboot is performed, it will need to be started using sudo docker start "CONTAINER ID"

 
   sudo docker run -d -u 1000:1000 -p 8085:8080 -p 50000:50000 --env JENKINS_OPTS="--prefix=/jenkins" --restart=unless-stopped -v /USER_HOME_DIR/docker/jenkins:/var/jenkins_home jenkins/jenkins:lts-jdk11
 

This will download the required images and start the container. There will be a message about initial password, make note of it as it will be used to log into Jenkins

 
   *************************************************************
   *************************************************************
   *************************************************************
   Jenkins initial setup is required. An admin user has been created and a password generated.
   Please use the following password to proceed to installation:
                                                                
   xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
                                                                  
   This may also be found at: /var/jenkins_home/secrets/initialAdminPassword
                                                                
   *************************************************************
   *************************************************************
   *************************************************************
 

Browser Connected Configuration

Go to application using the browser and use the above password to log in initially

At this screen select Install Suggested Plugins

The plugins will begin to install, this will only be a few minutes.

Create the Admin account. This will be a new account outside of what was used to connect initially. It will be the administrator account.

This sets up the URL with a path to the instance. It will include /jenkins so it should be something like “http(s)://<YOUR FQDN/jenkins/

The base Jenkins system install is complete at this point.

Click Start using Jenkins and the Dashboard appears

The alert is caused by this running locally and not using nodes. Click on it and select “Dismiss”

If once this is cleared, this message appears. It is due to a reverse proxy being used that needs to have some additional configuration.

When using Apache as a reverse proxy, adding these two entries to the virtual host SSL configuration should correct the error.

RequestHeader set X-Forwarded-Proto "https"

RequestHeader set X-Forwarded-Port "443"

Add Configuration Items for Pull Requests and Commits

Install and additional plugin called “Discard Old Builds” by going to Manage Jenkins → Manage Plugins → Available Plugins and search for “Discard old builds”

Install an additional plugin called “Throttle Concurrent Builds” by going to Manage Jenkins → Manage Plugins → Available Plugins and search for “Throttle Concurrent Builds”

Once plugin is installed to go Manage Jenkins → Configure System → Throttle Concurrent Builds and select Add Category:

Create Category Name PRC_THROTTLE and set Maximum Total Concurrent Builds = 1 and Maximum Concurrent Builds Per Node = 1.

Select “Apply” to save

Set up the environment variables in Jenkins:

1. Select the Environment Variables checkbox
2. Select Add and add the following
   1. For a GitHub Installation
      1. GITHUB_PRC_FILES = $JENKINS_HOME/lingoport/<globalyzer-lite-dir>/GitHub_PRC
   2. PATH = $PATH:$JENKINS_HOME/lingoport/bin
   3. If https is to be used for connecting to the git repos set to true. If ssh is to be used to connect to git repos set to false.
      1. useHTTPS = true or false

Click Apply and Save

Add new API token to be used by GitHub Webhook

In Jenkins create a new API token for the user that will be running the build. This will be required to be added to the webhook to permit connection back to the Jenkins instance

The credentials used for GitHub will require read access to the git repository.

1. Select Dashboard → Manage Jenkins → Manage Users → Select the gear icon on the right side of the Admin user (Admin user created earlier)
2. In the API Token section, select Add new Token
3. Set the name of the new token to GitHub-PRC-TOKEN or something that is memorable and can identify with the PRC action.
   1. Be sure to copy the token value as it will never be displayed beyond this time.

Leave the GUI and go back to the file system

1. Edit the github.properties file:
   1. /HOME_DIR/docker/jenkins/lingoport/globalyzer-lite-VERSION/GitHub_PRC/github.properties.
   2. Update the github.login and github.oauth with the appropriate PAT values that were created ahead of time.
2. When using https there will need to be a .netrc file placed in the jenkins_home directory inside of the container. This can be placed in the volume directory at /HOME_DIR/docker/jenkins/.netrc
   1. login <GitHub login ID> (Not email)
   2. password PAT
3. When using ssh there will need to be the ssh public and private keys stored in the /HOME_DIR/docker/jenkins/.ssh directory. Also with ssh be sure to manually connect to the git repo prior to using the PRC scripts. The known_hosts file needs to be updated and that is a manual process that requires manually accepting the key. This should only be needed to be performed once.
4. Run the installer for Globalyzer-Lite. This must be done inside of the contains so everything is properly set up in that environment
   1. Obtain the container id
      1. sudo docker container ps
   2. sudo docker exec -it CONTAINER ID /bin/bash
   3. cd ~/lingoport/globalyzer-lite-VERSION
   4. ./install-lite.sh (Should return Globalyzer Lite installation completed successfully)

      Create Jenkins Build for Pull Requests and Commits via GitHub

      GitHub Configuration

      Install Validation