Pull Requests

From Lingoport Wiki
Revision as of 13:23, 1 May 2024 by Rwilliams (talk | contribs) (→‎Pull Requests via Jenkins Running on Docker)
Jump to: navigation, search

Introduction

For GitHub, GitLab and Bitbucket, commits, Pull Requests / Merge Requests code changes can be analyzed using the Globalyzer Scanner.

A Webhook lets a Jenkins job know to scan the code committed. Only the code modified is analyzed by the Globalyzer scanner. The result is a comment in GitHub, GitLab or Bitbucket showing issues detected.

In the example below, one line was modified and the Globalyzer scan shows an i18n issue. The developer can decide how to handle it.

PullRequestExample.png

This is unobtrusive and provides actionable i18n information to developers.


Video Introduction to GitHub, GitLab, and Bitbucket

Below are short introductions of the functionality around Globalyzer scanning for Git repository:

Pull Requests via Jenkins Running on Docker

Prerequisites

  1. VM running Linux and Docker
  2. JQ JSON Parser - Statically compiled binary version
    1. https://stedolan.github.io/jq/
  3. GitHub PAT credentials to connect to GitHub
    1. The user for these PAT credentials has read access to the repository
  4. Globalyzer.com credentials for logging in to globalyzer.com
    1. username
    2. password
  5. Obtain a copy of Globalyzer Lite from https://www.globalyzer.com/gzserver/home/installclient (requires Globalyzer.com credentials)
  6. A copy of your Globalyzer.license from globalyzer.com or your local globalyzer server instance.
  7. Jenkins will use the following URL to download all plugins required during the installation. This URL is required to be reachable from the VM Jenkins is being installed on prior to starting the install.
    1. https://updates.jenkins.io/update-center.json
  8. Optional:
    1. Apache or nginx could be installed and running on the VM itself as a reverse proxy to front the Jenkins instance

File System Configuration

If Apache is installed as per normal for many Lingoport systems, the following is required to configure Apache to permit proper connection to the Jenkins Docker container

  1. Create the directory structure for the volume storage used by jenkins. Be sure there is sufficient space. Base install will consume 650MB of space for the volume itself. 
    2.       mkdir ~/docker/jenkins/lingoport/bin
  2. Place the Globalyzer License into the lingoport directory created 
    3.       cp ~/PRC-Docker/Globalyzer.license ~/docker/jenkins/lingoport/
  3. Copy the downloaded copy of Globalyzer Lite into the lingoport directory 
    4.       cp ~/PRC-Docker/globalyzer-lite-VERSION.zip ~/docker/jenkins/lingoport/
  4. Copy the jq binary into the bin directory and set the permissions 
    5.       
       cp -p  ~/PRC-Docker/jq ~/docker/jenkins/lingoport/bin/
       chmod +x ~/docker/jenkins/lingoport/bin/jq
  5. Unzip the Globalyzer Lite zip file in the lingoport directory and remove the zip file once complete 
    6.      
       cd ~/docker/jenkins/lingoport
       unzip globalyzer-lite-VERSION.zip
       rm ./globalyzer-lite-VERSION.zip
  6. Create the .globalyzerrc file in the jenkins directory, add the proper credentials and set the permissions 
    7.      
       vi .globalyzerrc
       chmod 600 .globalyzerrc
  7. Confirm the contents of the .globalyzerrc file are entered like this: (Note no equals or colon, just whitespace between key and value) 
    8.      
       username <GIT LOGIN USERNAME>
       password <GIT LOGIN PASSWORD>
       server <URL of Globalyzer Server>

Launching the Jenkins Docker Container

Retrieve these two items as they are now required to proceed

  1. Obtain the UID and GID for the user that Docker is being ran by.
  2. Identify the the port that Jenkins should be listening on.
    1. In the example below it was determined to be 8085

The next command line is an example of what would be used to start the docker instance for Jenkins. Here is an explanation of the switches being used when no proxy is required for the container to reach required destinations.

The command string will create a new container instance that will automatically restart the container unless specifically stopped. It will also reuse the existing container on a restart of the OS. If the container is stopped before the reboot is performed using sudo docker stop "CONTAINTER ID" , it will need to be started using sudo docker start "CONTAINER ID"

  1. The -d detaches the command being run from the terminal. Normally this is something that will only be used once everything is in place and confirmed to be working. For initial startup, its use can be ignored.
  2. The -u is for setting the uid/gid of the user that owns the volume storage identified earlier. This user should also be the user that is starting the docker instance.
  3. The -p 8085:8080 is the port external to internal mapping identified earlier. The 8080 would never change, only the external port is modifiable
  4. The -p 50000:50000 is for mapping between Jenkins instances only. Not required for standalone systems.
  5. The JENKINS_OPTS sets the baseurl for the application
  6. The -v mounts the prior locally created file system to the running docker instance as a volume and mas it.
  7. The jenkins/jenkins:lts-jdk11 is the long term support image for Jenkins. Currently, it is Jenkins 2.387.3 
 
   sudo docker run -d -u 1000:1000 -p 8085:8080 -p 50000:50000 \
     --env JENKINS_OPTS="--prefix=/jenkins" --restart=unless-stopped \
     -v /USER_HOME_DIR/docker/jenkins:/var/jenkins_home jenkins/jenkins:lts-jdk11

If a proxy is required to be leveraged by the container to reach destinations the next command line would be used to start the docker instance for Jenkins instead of the previous one. Note the additional env variables that are added to support the proxy. 

 
   sudo docker run -d -u 1000:1000 -p 8085:8080 -p 50000:50000 \
     --env JENKINS_OPTS="--prefix=/jenkins" \
     --env http_proxy="http://<PROXY_URL>:<PORT>" \
     --env https_proxy="http://<PROXY_URL>:<PORT>" \
     --restart=unless-stopped \
     -v /home/centos/docker/jenkins:/var/jenkins_home jenkins/jenkins:lts-jdk11

This will download the required images and start the container. There will be a message about initial password, make note of it as it will be used to log into Jenkins 


 
   *************************************************************
   *************************************************************
   *************************************************************
   Jenkins initial setup is required. An admin user has been created and a password generated.
   Please use the following password to proceed to installation:
                                                                
   xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
                                                                  
   This may also be found at: /var/jenkins_home/secrets/initialAdminPassword
                                                                
   *************************************************************
   *************************************************************
   *************************************************************

Browser Connected Configuration

Go to application using the browser and use the above password to log in initially

noframe

At this screen select Install Suggested Plugins

noframe

The plugins will begin to install, this will only be a few minutes.

noframe

Create the Admin account. This will be a new account outside of what was used to connect initially. It will be the administrator account.

noframe

This sets up the URL with a path to the instance. It will include /jenkins so it should be something like “http(s)://<YOUR FQDN/jenkins/

noframe

The base Jenkins system install is complete at this point.

noframe

Click Start using Jenkins and the Dashboard appears

noframe

The alert is caused by this running locally and not using nodes. Click on it and select “Dismiss”

noframe

If once this is cleared, this message appears. It is due to a reverse proxy being used that needs to have some additional configuration.

noframe

When using Apache as a reverse proxy, adding these two entries to the virtual host SSL configuration should correct the error.

RequestHeader set X-Forwarded-Proto "https"

RequestHeader set X-Forwarded-Port "443"

Add GUI Configuration Items for Pull Requests and Commits

Install and additional plugin called “Discard Old Builds” by going to Manage Jenkins → Manage Plugins → Available Plugins and search for “Discard old builds”

noframe

Install an additional plugin called “Throttle Concurrent Builds” by going to Manage Jenkins → Manage Plugins → Available Plugins and search for “Throttle Concurrent Builds”

noframe

Once plugin is installed to go Manage Jenkins → Configure System → Throttle Concurrent Builds and select Add Category:

Create Category Name PRC_THROTTLE and set Maximum Total Concurrent Builds = 1 and Maximum Concurrent Builds Per Node = 1.

Select “Apply” to save

Set up the environment variables in Jenkins:

  1. Select the Environment Variables checkbox
  2. Select Add and add the following
    1. For a GitHub Installation
      1. GITHUB_PRC_FILES = $JENKINS_HOME/lingoport/<globalyzer-lite-dir>/GitHub_PRC
    2. PATH = $PATH:$JENKINS_HOME/lingoport/bin
    3. If https is to be used for connecting to the git repos set to true. If ssh is to be used to connect to git repos set to false.
      1. useHTTPS = true or false
    4. Optional - If Proxy is required:
      1. Add entry _JAVA_OPTIONS with proxy details as shown in screenshot below
      2. Dhttp.proxyHost=<FQDN> Dhttp.proxyPort=<PORT> Dhttps.proxyHost=<FQDN> Dhttps.proxyPort=<PORT>


noframe
noframe
noframe

Click Apply and Save


Add new API token to be used by GitHub Webhook

In Jenkins create a new API token for the user that will be running the build. This will be required to be added to the webhook to permit connection back to the Jenkins instance

The credentials used for GitHub will require read access to the git repository.

  1. Select Dashboard → Manage Jenkins → Manage Users → Select the gear icon on the right side of the Admin user (Admin user created earlier)
  2. In the API Token section, select Add new Token
  3. Set the name of the new token to GitHub-PRC-TOKEN or something that is memorable and can identify with the PRC action.
    1. Be sure to copy the token value as it will never be displayed beyond this time.

noframe

Add File System Configuration Items for Pull Requests and Commits

Leave the GUI and go back to the file system

  1. Edit the github.properties file:
    1. /HOME_DIR/docker/jenkins/lingoport/globalyzer-lite-VERSION/GitHub_PRC/github.properties.
    2. Update the github.login and github.oauth with the appropriate PAT values that were created ahead of time.
  2. When using https there will need to be a .netrc file placed in the jenkins_home directory inside of the container. This can be placed in the volume directory at /HOME_DIR/docker/jenkins/.netrc
    1. login <GitHub login ID> (Not email)
    2. password PAT
  3. When using ssh there will need to be the ssh public and private keys stored in the /HOME_DIR/docker/jenkins/.ssh directory. Also with ssh be sure to manually connect to the git repo prior to using the PRC scripts. The known_hosts file needs to be updated and that is a manual process that requires manually accepting the key. This should only be needed to be performed once.
  4. Run the installer for Globalyzer-Lite. This must be done inside of the contains so everything is properly set up in that environment
    1. Obtain the container id
      1. sudo docker container ps
    2. sudo docker exec -it CONTAINER ID /bin/bash
    3. cd ~/lingoport/globalyzer-lite-VERSION
    4. ./install-lite.sh (Should return Globalyzer Lite installation completed successfully)

Create Jenkins Build for Pull Requests and Commits via GitHub

  1. Log into Jenkins and go to the Dashboard
  2. Select New Item
  3. Create an item name for the PRC build.
  4. Use an item name that is memorable, something like GitHub-PRC is fine.
  5. Select Freestyle Project
  6. Select OK
  7. Add a description (optional)
  8. Select “This project is parameterized”
  9. Add two string parameters
    1. payload = NOT_SET
    2. LITE_PROJECT_DEFINITION = DEFAULT
      1. DEFAULT looks for the project definition to exist in the lingoport directory of the working branch as such: lingoport/LiteProjectDefinition.xml
    3. Select “Throttle Concurrent Builds”
      1. Set Maximum Total Concurrent Builds to 1
      2. Set Maximum Concurrent Builds Per Node to 1
    4. In Build Triggers Section
      1. Select Trigger builds remotely (e.g., from scripts)
      2. In the Authentication Token place a token you wish to use. Something like HOOK for example.
    5. In Build Steps Section
      1. Select Execute Shell
      2. Place the following shell code
        • set +x
        • $GITHUB_PRC_FILES/job_lite_pr.sh $GITHUB_PRC_FILES/pr_message.html
      3. In the Post-build Actions
        1. Select Discard Old Builds
        2. Set Days to keep builds to 14
      4. Save the configuration

    GitHub Configuration

    Webhooks are calls made from the GitHub side down to the Jenkins instance. They provide information required by the Jenkins build to properly scan the files that were either committed or had a pull request performed on them. This is discussed in detail in the Videos found in earlier sections of this page.

    Below is a summary of the steps.

    1. Log into GitHub
    2. Navigate to the repository that will have pull requests and commits applied to it
    3. Click on Settings → Webhooks→ Add webhook
    4. Enter the following:
      1. Payload URL
        1. URL found in Build Triggers Section of the Jenkins Build
        2. It will likely require the API Token generated prior along with the username that the build in Jenkins was created
        3. https://USER:TOKEN@JENKINS_URL/jenkins/job/BUILD_NAME/buildWithParameters?token=BUILD_TOKEN
      2. Content Type application/x-www-form-urlencoded
      3. Radio button: Let me select individual events
        1. Pushes
        2. Pull Requests
        3. Pushes, Pull Requests
    5. Using the Recent Deliveries tab it's possible to check the status of a web hook call to confirm that it worked correctly.
      1. There should be a build triggered for each webhook called submitted
      2. If there are issues use the Redeliver button to resend

    Install Validation

    This completes the basic installation. To confirm that everything is working correctly.

    1. Log into GitHub and navigate to a repository
    2. Make a change directly to a file in the GitHub repository and “Commit” the change (assuming Commits are enabled).
    3. Switch to Jenkins and confirm that the build was triggered and successfully completed
    4. Switch back to GitHub and click on the commit id for the modification in and confirm that there is an associated summary.
    5. Clicking on the File/Line link in the summary should show the code line and modification made.


    This is an image of the Analysis Summary in GitHub for a commit

    noframe


    This image shows the line of code that is presented when clicking on the link in the Analysis Summary

    noframe

    SAML Integration (Optional)

    To optionally leverage SAML in place of the local user login system, follow the instructions found on this page

    Docker Jenkins SAML Configuration

    Jenkins Upgrade when using Docker

    Upgrade Required Indication

    When logging into Jenkins, there will be a message on the Manage Jenkins page indicating that a new version is available.

    Before Upgrade.png

    Upgrade Procedure

    1. Log into the VM containing docker with a user that has sudo privileges.

    2. Stop the existing jenkins/jenkins:lts-jdk11 container 

      sudo docker stop <CONTAINER ID>

    3. Make a backup of the existing Jenkins directory that is mounted as a volume. All Jenkins files are contained on the file system, there is no associated DB. 

      $ cp -pr /home/<USER_DIR>/docker/jenkins /home/USER_DIR/docker/jenkins_<MMDDYYYY>

    4. Pull the latest version of the jenkins/jenkins:lts-jdk11 from docker.io. Note that it will indicate a newer image has been downloaded. 

           sudo docker pull jenkins/jenkins:lts-jdk11
       lts-jdk11: Pulling from jenkins/jenkins
       bd73737482dd: Pull complete
       747b9186aa97: Pull complete
       77e4b3fa9f98: Pull complete
       4185da018337: Pull complete
       fbfe8a536a91: Pull complete
       c28d8031aee7: Pull complete
       d90093f5fc84: Pull complete
       b7bf9221df23: Pull complete
       22a3392c99b4: Pull complete
       bbe8cad20a08: Pull complete
       c81331c127c6: Pull complete
       c125b4d0161b: Pull complete
       da080ad7c1d0: Pull complete
       Digest: sha256:90f7f78a7e114516216b6c0a06e00c597d6490597caec9d7bd7a95ce5c7dada0
       Status: Downloaded newer image for jenkins/jenkins:lts-jdk11
       docker.io/jenkins/jenkins:lts-jdk11

    5. Start the new container with the updated image using one of the two options below depending upon whether a proxy is required to be leveraged from inside of the container or not.

    Option #1 - Run the newly updated container to start Jenkins with the latest version (Command is the same as the one used initially to run) 

       $ sudo docker run -d -u 1000:1000 -p 8085:8080 -p 50000:50000 \
     --env JENKINS_OPTS="--prefix=/jenkins" \
     --restart=unless-stopped \
     -v /home/centos/docker/jenkins:/var/jenkins_home jenkins/jenkins:lts-jdk11
   $
   $ sudo docker container ls
   CONTAINER ID   IMAGE                       COMMAND                  CREATED         STATUS         PORTS                                                                                      NAMES
   1924e0a6bf0d   jenkins/jenkins:lts-jdk11   "/usr/bin/tini -- /u…"   2 minutes ago   Up 2 minutes   0.0.0.0:50000->50000/tcp, :::50000->50000/tcp, 0.0.0.0:8085->8080/tcp, :::8085->8080/tcp   peaceful_snyder

    Option #2 - Run the newly updated container to start Jenkins with the latest version of Jenkins when a proxy is needed for outgoing communications version (Command is the same as the one used initially to run) 

       $ sudo docker run -d -u 1000:1000 -p 8085:8080 -p 50000:50000 \
     --env JENKINS_OPTS="--prefix=/jenkins" \
     --env http_proxy="http://<PROXY_URL>:<PORT>" \
     --env https_proxy="http://<PROXY_URL>:<PORT>" \
     --restart=unless-stopped \
     -v /home/centos/docker/jenkins:/var/jenkins_home jenkins/jenkins:lts-jdk11
   $
   $ sudo docker container ls
   CONTAINER ID   IMAGE                       COMMAND                  CREATED          STATUS          PORTS                                                                                      NAMES
   9e2650b5ec1c   jenkins/jenkins:lts-jdk11   "/usr/bin/tini -- /u…"   33 seconds ago   Up 32 seconds   0.0.0.0:50000->50000/tcp, :::50000->50000/tcp, 0.0.0.0:8085->8080/tcp, :::8085->8080/tcp   elegant_engelbart


    6. Confirm that the new version is running properly by logging in the web interface and going to the manage jenkins section. There should not be mention of a new version being available.

    After Upgrade.png

    7. At the bottom of this page, the current version will be displayed in the footer.

    8. The old container can be removed if desired at this point. 

           sudo docker rm <CONTAINER ID>

    9. The old image can also be removed if desired at this point. 

           sudo docker rmi <IMAGE ID>

    10. The system will now be running the latest version of the Jenkins docker image and both the old images and containers will have been removed.

    Historical Version of the Pull Request/Commit Globalyzer Scanning

    Historical Pull Request

Retrieved from "http://wiki.lingoport.com/index.php?title=Pull_Requests&oldid=98710"