Difference between revisions of "Distributed Jenkins"
(45 intermediate revisions by 3 users not shown) | |||
Line 1: | Line 1: | ||
Notes on the Jenkins Deployment for Scalability around Pull Requests from GitHub. |
Notes on the Jenkins Deployment for Scalability around Pull Requests from GitHub. |
||
+ | == On-boarding a Distributed GitHub Pull Request Jenkins Job== |
||
+ | Once Jenkins is setup in Master/Slave mode (See below for setup), the following instructions will help to on-board a GitHub Pull Request job which will be initiated by a Webhook triggering a Master Jenkins job which in turns will execute the job on a slave node. |
||
+ | |||
+ | === On Jenkins Master Node === |
||
+ | On the Master Node, copy the <code>Lingoport.SampleLite-DistributedPullRequest</code> and modify the Build Parameters: |
||
+ | * <code>GIT_REPO</code>: in the form <code>Lingoport/sandbox</code> |
||
+ | * <code>LITE_PROJECT_DEFINITION</code>: we recommend having a top level directory named 'lingoport' with a Lite project definition file in the repo itself. |
||
+ | |||
+ | <b>Note</b>: Do not modify the <code>payload</code> build parameter. |
||
+ | |||
+ | === On GitHub === |
||
+ | Add a Pull Request Web Hook on the Git Repository, for example: |
||
+ | http://JENKINS_MASTER_IP_ADDRESS/buildByToken/buildWithParameters?job=Lingoport.SampleLite-DistributedPullRequest&token=HOOK |
||
+ | |||
+ | The job name is the Master's job name for your project (not Lingoport.SampleLite-DistributedPullRequest) |
||
+ | |||
+ | === Check the On-Boarding === |
||
+ | Test the system by creating a test pull request, for instance with an Embedded String commit. |
||
+ | |||
+ | == On-boarding a Distributed BitBucket Pull Request Jenkins Job== |
||
+ | Once Jenkins is setup in Master/Slave mode (See below for setup), the following instructions will help to on-board a BitBucket Pull Request job which will be initiated by a Webhook triggering a Master Jenkins job which in turns will execute the job on a slave node. |
||
+ | |||
+ | <b>Note:</b> Make sure the BitBucket 'Branches to Build' is set to 'any' (**) |
||
+ | |||
+ | === On BitBucket === |
||
+ | Add a Web Hook on the Git Repository in the following form: |
||
+ | http://MASTERIPADDRESS:<MASTERPORT>/bitbucket-hook/ |
||
+ | for example: |
||
+ | http://i18n.jenkins.company/bitbucket-hook/ |
||
+ | |||
+ | This hook will have a BITBUCKET_PAYLOAD JSON file and will trigger the execution of any job on-boarded with that BitBucket Git repository. For more information, see https://wiki.jenkins-ci.org/display/JENKINS/BitBucket+Plugin. |
||
+ | |||
+ | == Master/Slave Jenkins Setup == |
||
Close to: http://devopscube.com/setup-slaves-on-jenkins-2/ |
Close to: http://devopscube.com/setup-slaves-on-jenkins-2/ |
||
+ | ===Master Setup=== |
||
− | # on master: |
||
+ | The following is done once. |
||
+ | |||
Generate an id_rsa key, for example: |
Generate an id_rsa key, for example: |
||
ssh-keygen -t rsa -C "The access key for Jenkins slaves" |
ssh-keygen -t rsa -C "The access key for Jenkins slaves" |
||
make sure the chmod's are right |
make sure the chmod's are right |
||
+ | |||
+ | Credentials for Jenkins based on SSH username with Private Key: |
||
+ | jenkins dashboard –> credentials –> Global credentials –> add credentials |
||
+ | |||
+ | Choose from file with the default id_rsa / id_rsa.pub |
||
+ | |||
copy the public key |
copy the public key |
||
Line 15: | Line 56: | ||
(see credentials below based on file) |
(see credentials below based on file) |
||
+ | == Slave Node Setup== |
||
− | # on slave |
||
+ | The following is done once per slave node. |
||
+ | |||
+ | === Jenkins User === |
||
+ | The 'jenkins' user must be created with the same home directory as the master 'jenkins' user, i.e. <code>/var/lib/jenkins</code>. |
||
+ | |||
+ | === Node Installer === |
||
Make sure the following are installed with the correct version of Java for the given release: |
Make sure the following are installed with the correct version of Java for the given release: |
||
* Java |
* Java |
||
* git |
* git |
||
− | * Globalyzer Lite |
+ | * Globalyzer Lite under ~jenkins/lingoport/globalyzer-lite-x.y |
* .globalyzerrc |
* .globalyzerrc |
||
+ | * sonar-scanner under jenkins/lingoport/sonar-scanner-x.y/ |
||
+ | ** the configuration file must point back to the Dashboard Server! |
||
+ | * jq (see for example <del>https://www.linkedin.com/pulse/how-install-jq-centos-7-artur-todeschini</del>) |
||
+ | ** yum install epel-release |
||
+ | ** yum install jq |
||
+ | * The remote scripts, <code>distributed_lite_ghpr.sh</code> ...) |
||
+ | * A template <code>pullrequest-sonar-project.properties</code> under /var/lib/jenkins/lingoport/templates |
||
+ | ** A number of items need to be set in the templates itself, including GitHub's username/OAuth Token, Dashboard Username / Password for all the project to be handled. |
||
+ | The directory structure should look like after install on the slave node: |
||
+ | |||
+ | [jenkins@ip-172-31-45-177 jenkins]$ pwd |
||
+ | /var/lib/jenkins |
||
+ | [jenkins@ip-172-31-45-177 jenkins]$ ls -l |
||
+ | total 4 |
||
+ | drwxrwxr-x. 4 jenkins jenkins 61 Dec 29 10:02 jenkins_slave |
||
+ | drwxrwxr-x. 6 jenkins jenkins 82 Dec 27 23:29 lingoport |
||
+ | |||
+ | |||
+ | [jenkins@ip-172-31-45-177 lingoport]$ pwd |
||
+ | /var/lib/jenkins/lingoport |
||
+ | [jenkins@ip-172-31-45-177 lingoport]$ ls -l |
||
+ | total 4 |
||
+ | drwxrwxr-x. 2 jenkins jenkins 35 Dec 28 00:07 bin |
||
+ | drwxr-xr-x. 5 jenkins jenkins 4096 Dec 22 21:24 globalyzer-lite-5.2 |
||
+ | drwxr-xr-x. 5 jenkins jenkins 37 Sep 20 09:57 sonar-scanner-2.8 |
||
+ | drwxrwxr-x. 2 jenkins jenkins 49 Dec 28 00:19 templates |
||
+ | |||
+ | === SSH Key === |
||
copy the public key (id_rsa.pub) into |
copy the public key (id_rsa.pub) into |
||
.ssh/authorized_keys |
.ssh/authorized_keys |
||
Line 29: | Line 104: | ||
'''Note''': Make sure in the copy you have one line for the copy. Sometimes, copying may break the key into multiple lines. |
'''Note''': Make sure in the copy you have one line for the copy. Sometimes, copying may break the key into multiple lines. |
||
+ | ===Ports=== |
||
− | # In AWS console / the master and slave nodes: |
||
+ | The Slave Node Installer cannot setup the ports. This needs to be done by IT. |
||
− | Open the 22 ports to and from the two systems. |
||
− | + | On the master and slave nodes: |
|
+ | Open the 22 ports to and from the two systems to allow jobs to run from Master to Slaves/Agents/Nodes |
||
− | |||
+ | Make sure the Outgoing ports and the Incoming ports for the Master and Slaves are allowed for Dashboard upload. |
||
− | # On Master: |
||
+ | |||
+ | *Whitelists for events going to the Continuous G11n System: |
||
+ | ** BitBucket: 104.192.143.0/24 |
||
+ | ** GitHub: 192.30.252.0/22 |
||
+ | |||
+ | ===Adding a Slave Node to the Master Node === |
||
− | When click OK. The Node should go on-line. The top of the log should look like: |
||
− | [12/22/16 15:47:13] [SSH] Opening SSH connection to 54.89.231.113:22. |
||
− | [12/22/16 15:47:13] [SSH] Authentication successful. |
||
− | [12/22/16 15:47:13] [SSH] The remote users environment is: |
||
On your master machine go to Manage Jenkins > Manage Nodes. |
On your master machine go to Manage Jenkins > Manage Nodes. |
||
+ | Use the specific label 'distributed' for the slave nodes, with 1 executor. All the 'lingoport' nodes can then be used to run jobs remotely. |
||
+ | |||
+ | When click OK. The Node should go on-line. The top of the log should look like: |
||
+ | [12/22/16 15:47:13] [SSH] Opening SSH connection to 54.89.231.113:22. |
||
+ | [12/22/16 15:47:13] [SSH] Authentication successful. |
||
+ | [12/22/16 15:47:13] [SSH] The remote users environment is: |
Latest revision as of 20:31, 11 July 2018
Notes on the Jenkins Deployment for Scalability around Pull Requests from GitHub.
Contents
On-boarding a Distributed GitHub Pull Request Jenkins Job
Once Jenkins is setup in Master/Slave mode (See below for setup), the following instructions will help to on-board a GitHub Pull Request job which will be initiated by a Webhook triggering a Master Jenkins job which in turns will execute the job on a slave node.
On Jenkins Master Node
On the Master Node, copy the Lingoport.SampleLite-DistributedPullRequest
and modify the Build Parameters:
GIT_REPO
: in the formLingoport/sandbox
LITE_PROJECT_DEFINITION
: we recommend having a top level directory named 'lingoport' with a Lite project definition file in the repo itself.
Note: Do not modify the payload
build parameter.
On GitHub
Add a Pull Request Web Hook on the Git Repository, for example:
http://JENKINS_MASTER_IP_ADDRESS/buildByToken/buildWithParameters?job=Lingoport.SampleLite-DistributedPullRequest&token=HOOK
The job name is the Master's job name for your project (not Lingoport.SampleLite-DistributedPullRequest)
Check the On-Boarding
Test the system by creating a test pull request, for instance with an Embedded String commit.
On-boarding a Distributed BitBucket Pull Request Jenkins Job
Once Jenkins is setup in Master/Slave mode (See below for setup), the following instructions will help to on-board a BitBucket Pull Request job which will be initiated by a Webhook triggering a Master Jenkins job which in turns will execute the job on a slave node.
Note: Make sure the BitBucket 'Branches to Build' is set to 'any' (**)
On BitBucket
Add a Web Hook on the Git Repository in the following form:
http://MASTERIPADDRESS:<MASTERPORT>/bitbucket-hook/
for example:
http://i18n.jenkins.company/bitbucket-hook/
This hook will have a BITBUCKET_PAYLOAD JSON file and will trigger the execution of any job on-boarded with that BitBucket Git repository. For more information, see https://wiki.jenkins-ci.org/display/JENKINS/BitBucket+Plugin.
Master/Slave Jenkins Setup
Close to: http://devopscube.com/setup-slaves-on-jenkins-2/
Master Setup
The following is done once.
Generate an id_rsa key, for example:
ssh-keygen -t rsa -C "The access key for Jenkins slaves"
make sure the chmod's are right
Credentials for Jenkins based on SSH username with Private Key: jenkins dashboard –> credentials –> Global credentials –> add credentials
Choose from file with the default id_rsa / id_rsa.pub
copy the public key
Important: This is all based on the key under /var/lib/jenkins/.ssh/ being with the following owner and access permissions:
-rw-------. 1 jenkins jenkins 1675 Dec 21 21:20 id_rsa -rw-r--r--. 1 jenkins jenkins 419 Dec 21 21:20 id_rsa.pub
(see credentials below based on file)
Slave Node Setup
The following is done once per slave node.
Jenkins User
The 'jenkins' user must be created with the same home directory as the master 'jenkins' user, i.e. /var/lib/jenkins
.
Node Installer
Make sure the following are installed with the correct version of Java for the given release:
- Java
- git
- Globalyzer Lite under ~jenkins/lingoport/globalyzer-lite-x.y
- .globalyzerrc
- sonar-scanner under jenkins/lingoport/sonar-scanner-x.y/
- the configuration file must point back to the Dashboard Server!
- jq (see for example
https://www.linkedin.com/pulse/how-install-jq-centos-7-artur-todeschini)- yum install epel-release
- yum install jq
- The remote scripts,
distributed_lite_ghpr.sh
...) - A template
pullrequest-sonar-project.properties
under /var/lib/jenkins/lingoport/templates- A number of items need to be set in the templates itself, including GitHub's username/OAuth Token, Dashboard Username / Password for all the project to be handled.
The directory structure should look like after install on the slave node:
[jenkins@ip-172-31-45-177 jenkins]$ pwd /var/lib/jenkins [jenkins@ip-172-31-45-177 jenkins]$ ls -l total 4 drwxrwxr-x. 4 jenkins jenkins 61 Dec 29 10:02 jenkins_slave drwxrwxr-x. 6 jenkins jenkins 82 Dec 27 23:29 lingoport
[jenkins@ip-172-31-45-177 lingoport]$ pwd /var/lib/jenkins/lingoport [jenkins@ip-172-31-45-177 lingoport]$ ls -l total 4 drwxrwxr-x. 2 jenkins jenkins 35 Dec 28 00:07 bin drwxr-xr-x. 5 jenkins jenkins 4096 Dec 22 21:24 globalyzer-lite-5.2 drwxr-xr-x. 5 jenkins jenkins 37 Sep 20 09:57 sonar-scanner-2.8 drwxrwxr-x. 2 jenkins jenkins 49 Dec 28 00:19 templates
SSH Key
copy the public key (id_rsa.pub) into
.ssh/authorized_keys
make sure the chmod is 600, as in:
-rw-------. 1 jenkins jenkins 831 Dec 21 23:05 authorized_keys
Note: Make sure in the copy you have one line for the copy. Sometimes, copying may break the key into multiple lines.
Ports
The Slave Node Installer cannot setup the ports. This needs to be done by IT.
On the master and slave nodes:
Open the 22 ports to and from the two systems to allow jobs to run from Master to Slaves/Agents/Nodes Make sure the Outgoing ports and the Incoming ports for the Master and Slaves are allowed for Dashboard upload.
- Whitelists for events going to the Continuous G11n System:
- BitBucket: 104.192.143.0/24
- GitHub: 192.30.252.0/22
Adding a Slave Node to the Master Node
On your master machine go to Manage Jenkins > Manage Nodes. Use the specific label 'distributed' for the slave nodes, with 1 executor. All the 'lingoport' nodes can then be used to run jobs remotely.
When click OK. The Node should go on-line. The top of the log should look like:
[12/22/16 15:47:13] [SSH] Opening SSH connection to 54.89.231.113:22. [12/22/16 15:47:13] [SSH] Authentication successful. [12/22/16 15:47:13] [SSH] The remote users environment is: