Distributed Jenkins

Notes on the Jenkins Deployment for Scalability around Pull Requests from GitHub.

On-boarding a Distributed GitHub Pull Request Jenkins Job

Once Jenkins is setup in Master/Slave mode (See below for setup), the following instructions will help to on-board a GitHub Pull Request job which will be initiated by a Webhook triggering a Master Jenkins job which in turns will execute the job on a slave node.

On Jenkins Master Node

On the Master Node, copy the Lingoport.SampleLite-DistributedPullRequest and modify the Build Parameters:

• GIT_REPO: in the form Lingoport/sandbox
• LITE_PROJECT_DEFINITION: we recommend having a top level directory named 'lingoport' with a Lite project definition file in the repo itself.

Note: Do not modify the payload build parameter.

On GitHub

Add a Pull Request Web Hook on the Git Repository, for example:

 http://JENKINS_MASTER_IP_ADDRESS/buildByToken/buildWithParameters?job=Lingoport.SampleLite-DistributedPullRequest&token=HOOK

The job name is the Master's job name for your project (not Lingoport.SampleLite-DistributedPullRequest)

Check the On-Boarding

Test the system by creating a test pull request, for instance with an Embedded String commit.

On-boarding a Distributed BitBucket Pull Request Jenkins Job

Once Jenkins is setup in Master/Slave mode (See below for setup), the following instructions will help to on-board a BitBucket Pull Request job which will be initiated by a Webhook triggering a Master Jenkins job which in turns will execute the job on a slave node.

Note: Make sure the BitBucket 'Branches to Build' is set to 'any' (**)

On BitBucket

Add a Web Hook on the Git Repository in the following form:

 http://MASTERIPADDRESS:<MASTERPORT>/bitbucket-hook/

for example:

 http://i18n.jenkins.company/bitbucket-hook/

This hook will have a BITBUCKET_PAYLOAD JSON file and will trigger the execution of any job on-boarded with that BitBucket Git repository. For more information, see https://wiki.jenkins-ci.org/display/JENKINS/BitBucket+Plugin.

Master/Slave Jenkins Setup

Close to: http://devopscube.com/setup-slaves-on-jenkins-2/

Master Setup

The following is done once.

Generate an id_rsa key, for example:

 ssh-keygen -t rsa -C "The access key for Jenkins slaves" 

make sure the chmod's are right

Credentials for Jenkins based on SSH username with Private Key: jenkins dashboard –> credentials –> Global credentials –> add credentials

Choose from file with the default id_rsa / id_rsa.pub

copy the public key

Important: This is all based on the key under /var/lib/jenkins/.ssh/ being with the following owner and access permissions:

  -rw-------.  1 jenkins jenkins 1675 Dec 21 21:20 id_rsa
  -rw-r--r--.  1 jenkins jenkins  419 Dec 21 21:20 id_rsa.pub

(see credentials below based on file)

Slave Node Setup

The following is done once per slave node.

Jenkins User

The 'jenkins' user must be created with the same home directory as the master 'jenkins' user, i.e. /var/lib/jenkins.

Node Installer

Make sure the following are installed with the correct version of Java for the given release:

• Java
• git
• Globalyzer Lite under ~jenkins/lingoport/globalyzer-lite-x.y
• .globalyzerrc
• sonar-scanner under jenkins/lingoport/sonar-scanner-x.y/
  • the configuration file must point back to the Dashboard Server!
• jq (see for example https://www.linkedin.com/pulse/how-install-jq-centos-7-artur-todeschini)
  • yum install epel-release
  • yum install jq
• The remote scripts, distributed_lite_ghpr.sh ...)
• A template pullrequest-sonar-project.properties under /var/lib/jenkins/lingoport/templates
  • A number of items need to be set in the templates itself, including GitHub's username/OAuth Token, Dashboard Username / Password for all the project to be handled.

The directory structure should look like after install on the slave node:

 [jenkins@ip-172-31-45-177 jenkins]$ pwd
 /var/lib/jenkins
 [jenkins@ip-172-31-45-177 jenkins]$ ls -l
 total 4
 drwxrwxr-x. 4 jenkins jenkins   61 Dec 29 10:02 jenkins_slave
 drwxrwxr-x. 6 jenkins jenkins   82 Dec 27 23:29 lingoport

 [jenkins@ip-172-31-45-177 lingoport]$ pwd
 /var/lib/jenkins/lingoport
 [jenkins@ip-172-31-45-177 lingoport]$ ls -l
 total 4
 drwxrwxr-x. 2 jenkins jenkins   35 Dec 28 00:07 bin
 drwxr-xr-x. 5 jenkins jenkins 4096 Dec 22 21:24 globalyzer-lite-5.2
 drwxr-xr-x. 5 jenkins jenkins   37 Sep 20 09:57 sonar-scanner-2.8
 drwxrwxr-x. 2 jenkins jenkins   49 Dec 28 00:19 templates

SSH Key

copy the public key (id_rsa.pub) into

.ssh/authorized_keys

make sure the chmod is 600, as in:

-rw-------. 1 jenkins jenkins  831 Dec 21 23:05 authorized_keys

Note: Make sure in the copy you have one line for the copy. Sometimes, copying may break the key into multiple lines.

Ports

The Slave Node Installer cannot setup the ports. This needs to be done by IT.

On the master and slave nodes:

 Open the 22 ports to and from the two systems to allow jobs to run from Master to Slaves/Agents/Nodes
 Make sure the Outgoing ports and the Incoming ports for the Master and Slaves are allowed for Dashboard upload.

• Whitelists for events going to the Continuous G11n System:
  • BitBucket: 104.192.143.0/24
  • GitHub: 192.30.252.0/22

Adding a Slave Node to the Master Node

On your master machine go to Manage Jenkins > Manage Nodes. Use the specific label 'distributed' for the slave nodes, with 1 executor. All the 'lingoport' nodes can then be used to run jobs remotely.

When click OK. The Node should go on-line. The top of the log should look like:

 [12/22/16 15:47:13] [SSH] Opening SSH connection to 54.89.231.113:22.
 [12/22/16 15:47:13] [SSH] Authentication successful.
 [12/22/16 15:47:13] [SSH] The remote users environment is: