Node Installer

Rationale

Jenkins is setup by default on one system and executes all the jobs locally to that system. The Jenkins platform has the ability to distribute jobs to remote systems. This is called a Master Slave configuration in Jenkins parlance. See https://wiki.jenkins-ci.org/display/JENKINS/Distributed+builds

To start with this model, Lingoport is putting together a Node Installer for slaves which relies on a Master node installed with the Stack Installer; The first remote job is a Pull Request (PR) analysis for GitHub, so that analyzing many PR's can scale up.

Slave Node Installation

On the machines for slave nodes

• Setup the Linux CentOS systems to be the slave nodes.
• In the sudo user of the slave nodes, add the id_rsa.pub from Master to ~/.ssh/authorized_keys (See ssh-keygen -t rsa). This will allow Master to ssh into the Slaves, which is a pre-requisites before the node installer can be run. The following is on a CentOS AWS sytem, where 'centos' is a user with 'sudo' privileges:

 [centos@ip-172-31-37-148 .ssh]$ pwd
 /home/centos/.ssh
 [centos@ip-172-31-37-148 .ssh]$ ls -l authorized_keys
 -rw-------. 1 centos centos 814 Jan 12 22:28 authorized_keys

On the Master nodes

• Download the Lingoport-node-installer-x.y.z.zip, where x.y.z is the version number on a system which has access to the slave nodes and the master node; Typically, download Lingoport-node-installer-x.y.z.zip on the Master node.
• Unzip the zip file (on the Master node most likely)
• If ansible is not installed, run install_ansible.sh
• Go through the README.md file to configure the setup and run the ansible playbook. Make sure to have the right .ssh/authorized_keys even in Master as the [master] host needs to be filled in the hosts configuration file. So in effect, Master will ssh into Master; you can append the /home/centos/.ssh/rsa_id.pub into /home/centos/.ssh/authorized_keys to do so.
  • The following checks for the communication between Master and Slave(s)

 [centos@ip-172-31-60-142 node-installer]$ ansible-playbook -i hosts pre-allow-master-agent-ssh.yml
 PLAY [setup master system so it is ready to ssh to nodes] **********************
 TASK [setup] 
 [...]

• • The following sets up the slave node using the master's node configuration and Lingoport installed components:

 [centos@ip-172-31-60-142 node-installer]$  ansible-playbook -i hosts setup-agents.yml
 PLAY [install pre-requisite programs to nodes]
 [...]

Subsequent Slave Node Installation from Master

Once the installation has been run once, you can repeat the installation (from Master) by simply changing the hosts file to point to other slave nodes to add. They must have been configured as above.

Adding the new node(s) to Master

At this point, the newly setup nodes should be available for Jenkins.

• Navigate to Jenkins Manage Node
• Add New Node and Launch the New Node. You should see something like:

 [01/12/17 23:35:23] [SSH] Opening SSH connection to 172.31.37.148:22.
 [01/12/17 23:35:23] [SSH] Authentication successful.
 [01/12/17 23:35:23] [SSH] The remote users environment is:
 BASH=/usr/bin/bash
 BASHOPTS=cmdhist:extquote:force_fignore:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath
 [...]
 <===[JENKINS REMOTING CAPACITY]===>channel started
 Slave.jar version: 2.62.3
 This is a Unix agent
 Evacuated stdout
 Agent successfully connected and online

• Check a job can be sent to the nodes: For instance, run a Debug job configured with some simple command like pwd and env.
  • The job must be using the remote label "Restrict where this project can be run". The remote label, for instance lingoport, was set up when adding the new node.
• In the Jenkins UI, the Build Execution Status widget should show "Master" and a number of slave nodes based on the nodes added so far.

The Pull Request Distributed Jobs

The first Lingoport job to be enabled is the GitHub Pull Request job. To configure it, copy the Lingoport.SampleLite-GitHubPullRequest job to your specific job, for instance Acme.FunWidgets-GitHubPullRequest. Make sure it is restricted to run on slave nodes identified by the label you created for those nodes, for instance lingoport.