Pull Requests

Pull Requests, Merge Requests code changes can be analyzed using the Globalyzer Scanner.

Video Introduction to GitHub, GitLab, and Bitbucket

Below are short introductions of the functionality around Globalyzer scanning for Git repository:

• GitHub https://vimeo.com/817611647

• GitLab https://vimeo.com/817610534

• Bitbucket https://vimeo.com/826230267

Pull Requests via Jenkins Running on Docker

THIS SECTION IS CURRENTLY A WORK IN PROGRESS AN NOT YET COMPLETE

Prerequisites

1. VM running Linux and Docker
2. JQ JSON Parser - Statically compiled binary version
   1. https://stedolan.github.io/jq/
3. GitHub PAT credentials to connect to GitHub
   1. The user for these PAT credentials has read access to the repository
4. A copy of your Globalyzer.license from globalyzer.com or your local globalyzer server instance.
5. Globalyzer.com credentials for logging in to globalyzer.com
   1. username
   2. password
   3. server
6. Optional:
   1. Apache or nginx could be installed and running on the VM itself
   2. Used as a reverse proxy to front the Jenkins instance

File System Configuration

If Apache is installed as per normal for many Lingoport systems, the following is required to configure Apache to permit proper connection to the Jenkins Docker container

1. Create the directory structure for the volume storage used by jenkins. Be sure there is sufficient space. Base install will consume 650MB of space for the volume itself.

      mkdir ~/docker/jenkins/lingoport/bin 

2. Place the Globalyzer License into the lingoport directory created

      cp ~/PRC-Docker/Globalyzer.license ~/docker/jenkins/lingoport/ 

3. Copy the downloaded copy of Globalyzer Lite into the lingoport directory

      cp ~/PRC-Docker/globalyzer-lite-VERSION.zip ~/docker/jenkins/lingoport/ 

4. Copy the jq binary into the bin directory and set the permissions

      
       cp -p  ~/PRC-Docker/jq ~/docker/jenkins/lingoport/bin/
       chmod +x ~/docker/jenkins/lingoport/bin/jq
     

5. Unzip the Globalyzer Lite zip file in the lingoport directory and remove the zip file once complete

     
       cd ~/docker/jenkins/lingoport
       unzip globalyzer-lite-VERSION.zip
       rm ./globalyzer-lite-VERSION.zip
     

6. Create the .globalyzerrc file in the jenkins directory, add the proper credentials and set the permissions

     
       vi .globalyzerrc
       chmod 600 .globalyzerrc
     

7. Confirm the contents of the .globalyzerrc file are entered like this: (Note no equals or colon, just whitespace between key and value)

     
       username <GIT LOGIN USERNAME>
       password <GIT LOGIN PASSWORD>
       server <URL of Globalyzer Server> 
     

Launching the Jenkins Docker Container

Retrieve these two items as they are now required to proceed

1. Obtain the UID and GID for the user that Docker is being ran by.
2. Identify the the port that Jenkins should be listening on.
   1. In the example below it was determined to be 8085

The next command line is an example of what would be used to start the docker instance for Jenkins. Here is an explanation of the switches being used.

The command string will create a new container instance that will automatically restart the container unless specifically stopped. It will also reuse the existing container on a restart of the OS. If the container is stopped before the reboot is performed using sudo docker stop "CONTAINTER ID" , it will need to be started using sudo docker start "CONTAINER ID"

1. The -d detaches the command being run from the terminal. Normally this is something that will only be used once everything is in place and confirmed to be working. For initial startup, its use can be ignored.
2. The -u is for setting the uid/gid of the user that owns the volume storage identified earlier. This user should also be the user that is starting the docker instance.
3. The -p 8085:8080 is the port external to internal mapping identified earlier. The 8080 would never change, only the external port is modifiable
4. The -p 50000:50000 is for mapping between Jenkins instances only. Not required for standalone systems.
5. The JENKINS_OPTS sets the baseurl for the application
6. The -v mounts the prior locally created file system to the running docker instance as a volume and mas it.
7. The jenkins/jenkins:lts-jdk11 is the long term support image for Jenkins. Currently, it is Jenkins 2.387.3

 
   sudo docker run -d -u 1000:1000 -p 8085:8080 -p 50000:50000 --env JENKINS_OPTS="--prefix=/jenkins" --restart=unless-stopped -v /USER_HOME_DIR/docker/jenkins:/var/jenkins_home jenkins/jenkins:lts-jdk11
 

This will download the required images and start the container. There will be a message about initial password, make note of it as it will be used to log into Jenkins

 
   *************************************************************
   *************************************************************
   *************************************************************
   Jenkins initial setup is required. An admin user has been created and a password generated.
   Please use the following password to proceed to installation:
                                                                
   xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
                                                                  
   This may also be found at: /var/jenkins_home/secrets/initialAdminPassword
                                                                
   *************************************************************
   *************************************************************
   *************************************************************
 

Browser Connected Configuration

Go to application using the browser and use the above password to log in initially

At this screen select Install Suggested Plugins

The plugins will begin to install, this will only be a few minutes.

Create the Admin account. This will be a new account outside of what was used to connect initially. It will be the administrator account.

This sets up the URL with a path to the instance. It will include /jenkins so it should be something like “http(s)://<YOUR FQDN/jenkins/

The base Jenkins system install is complete at this point.

Click Start using Jenkins and the Dashboard appears

The alert is caused by this running locally and not using nodes. Click on it and select “Dismiss”

If once this is cleared, this message appears. It is due to a reverse proxy being used that needs to have some additional configuration.

When using Apache as a reverse proxy, adding these two entries to the virtual host SSL configuration should correct the error.

RequestHeader set X-Forwarded-Proto "https"

RequestHeader set X-Forwarded-Port "443"

Add Configuration Items for Pull Requests and Commits

Install and additional plugin called “Discard Old Builds” by going to Manage Jenkins → Manage Plugins → Available Plugins and search for “Discard old builds”

Install an additional plugin called “Throttle Concurrent Builds” by going to Manage Jenkins → Manage Plugins → Available Plugins and search for “Throttle Concurrent Builds”

Once plugin is installed to go Manage Jenkins → Configure System → Throttle Concurrent Builds and select Add Category:

Create Category Name PRC_THROTTLE and set Maximum Total Concurrent Builds = 1 and Maximum Concurrent Builds Per Node = 1.

Select “Apply” to save

Set up the environment variables in Jenkins:

1. Select the Environment Variables checkbox
2. Select Add and add the following
   1. For a GitHub Installation
      1. GITHUB_PRC_FILES = $JENKINS_HOME/lingoport/<globalyzer-lite-dir>/GitHub_PRC
   2. PATH = $PATH:$JENKINS_HOME/lingoport/bin
   3. If https is to be used for connecting to the git repos set to true. If ssh is to be used to connect to git repos set to false.
      1. useHTTPS = true or false

Click Apply and Save

Add new API token to be used by GitHub Webhook

In Jenkins create a new API token for the user that will be running the build. This will be required to be added to the webhook to permit connection back to the Jenkins instance

The credentials used for GitHub will require read access to the git repository.

1. Select Dashboard → Manage Jenkins → Manage Users → Select the gear icon on the right side of the Admin user (Admin user created earlier)
2. In the API Token section, select Add new Token
3. Set the name of the new token to GitHub-PRC-TOKEN or something that is memorable and can identify with the PRC action.
   1. Be sure to copy the token value as it will never be displayed beyond this time.

Leave the GUI and go back to the file system

1. Edit the github.properties file:
   1. /HOME_DIR/docker/jenkins/lingoport/globalyzer-lite-VERSION/GitHub_PRC/github.properties.
   2. Update the github.login and github.oauth with the appropriate PAT values that were created ahead of time.
2. When using https there will need to be a .netrc file placed in the jenkins_home directory inside of the container. This can be placed in the volume directory at /HOME_DIR/docker/jenkins/.netrc
   1. login <GitHub login ID> (Not email)
   2. password PAT
3. When using ssh there will need to be the ssh public and private keys stored in the /HOME_DIR/docker/jenkins/.ssh directory. Also with ssh be sure to manually connect to the git repo prior to using the PRC scripts. The known_hosts file needs to be updated and that is a manual process that requires manually accepting the key. This should only be needed to be performed once.
4. Run the installer for Globalyzer-Lite. This must be done inside of the contains so everything is properly set up in that environment
   1. Obtain the container id
      1. sudo docker container ps
   2. sudo docker exec -it CONTAINER ID /bin/bash
   3. cd ~/lingoport/globalyzer-lite-VERSION
   4. ./install-lite.sh (Should return Globalyzer Lite installation completed successfully)

Create Jenkins Build for Pull Requests and Commits via GitHub

1. Log into Jenkins and go to the Dashboard
2. Select New Item
3. Create an item name for the PRC build.
4. Use an item name that is memorable, something like GitHub-PRC is fine.
5. Select Freestyle Project
6. Select OK
7. Add a description (optional)
8. Select “This project is parameterized”
9. Add two string parameters
   1. payload = NOT_SET
   2. LITE_PROJECT_DEFINITION = DEFAULT
      1. DEFAULT looks for the project definition to exist in the lingoport directory of the working branch as such: lingoport/LiteProjectDefinition.xml
   3. Select “Throttle Concurrent Builds”
      1. Set Maximum Total Concurrent Builds to 1
      2. Set Maximum Concurrent Builds Per Node to 1
   4. In Build Triggers Section
      1. Select Trigger builds remotely (e.g., from scripts)
      2. In the Authentication Token place a token you wish to use. Something like HOOK for example.
   5. In Build Steps Section
      1. Select Execute Shell
      2. Place the following shell code
         • set +x
         • $GITHUB_PRC_FILES/job_lite_pr.sh $GITHUB_PRC_FILES/pr_message.html
      3. In the Post-build Actions
         1. Select Discard Old Builds
         2. Set Days to keep builds to 14
      4. Save the configuration

   GitHub Configuration

   Webhooks are calls made from the GitHub side down to the Jenkins instance. They provide information required by the Jenkins build to properly scan the files that were either committed or had a pull request performed on them. This is discussed in detail in the Videos found in earlier sections of this page.

   Below is a summary of the steps.

   1. Log into GitHub
   2. Navigate to the repository that will have pull requests and commits applied to it
   3. Click on Settings → Webhooks→ Add webhook
   4. Enter the following:
      1. Payload URL
         1. URL found in Build Triggers Section of the Jenkins Build
         2. It will likely require the API Token generated prior along with the username that the build in Jenkins was created
         3. https://USER:TOKEN@JENKINS_URL/jenkins/job/BUILD_NAME/buildWithParameters?token=BUILD_TOKEN
      2. Content Type application/x-www-form-urlencoded
      3. Radio button: Let me select individual events
         1. Pushes
         2. Pull Requests
         3. Pushes, Pull Requests
   5. Using the Recent Deliveries tab it's possible to check the status of a web hook call to confirm that it worked correctly.
      1. There should be a build triggered for each webhook called submitted
      2. If there are issues use the Redeliver button to resend

   Install Validation

   This completes the basic installation. To confirm that everything is working correctly.

   1. Log into GitHub and navigate to a repository
   2. Make a change directly to a file in the GitHub repository and “Commit” the change (assuming Commits are enabled).
   3. Switch to Jenkins and confirm that the build was triggered and successfully completed
   4. Switch back to GitHub and click on the commit id for the modification in and confirm that there is an associated summary.
   5. Clicking on the File/Line link in the summary should show the code line and modification made.

   
   

   This is an image of the Analysis Summary in GitHub for a commit

   

   
   

   This image shows the line of code that is presented when clicking on the link in the Analysis Summary

   

   Historical Version of the Pull Request/Commit Globalyzer Scanning

   Historical Pull Request