Node Installer

Rationale

Jenkins is setup by default on one system and executes all the jobs locally to that system. The Jenkins platform has the ability to distribute jobs to remote systems. This is called a master/agent configuration in Jenkins parlance. See https://wiki.jenkins-ci.org/display/JENKINS/Distributed+builds

To start with this model, Lingoport is putting together a Node Installer for agents which relies on a Master system installed with the Stack Installer; The first remote job is performs Pull Request (PR) analysis for GitHub. A large number of GitHub pull requests may be performed concurrently, so the ability to scale is important. Distributing the PR job between multiple systems is a means to scale it.

Agent Installation

On the machines for agent nodes

• Setup the Linux CentOS systems to be the agent nodes.
• In the sudo user of the agent nodes, add the id_rsa.pub from Master to ~/.ssh/authorized_keys (See ssh-keygen -t rsa). This will allow Master to ssh into the Agents, which is a pre-requisites before the node installer can be run. The following is on a CentOS AWS sytem, where 'centos' is a user with 'sudo' privileges:

 [centos@ip-172-31-37-148 .ssh]$ pwd
 /home/centos/.ssh
 [centos@ip-172-31-37-148 .ssh]$ ls -l authorized_keys
 -rw-------. 1 centos centos 814 Jan 12 22:28 authorized_keys

On the Master system

• Download the Lingoport-node-installer-x.y.z.zip, where x.y.z is the version number on a system which has access to the agent nodes and the master node; Typically, download Lingoport-node-installer-x.y.z.zip on the Master node.
• Unzip the zip file (on the Master node most likely)
• If ansible is not installed, run install_ansible.sh
• Go through the README.md file to configure the setup and run the ansible playbook. Make sure to have the right .ssh/authorized_keys even in Master as the [master] host needs to be filled in the hosts configuration file. So in effect, Master will ssh into Master; you can append the /home/centos/.ssh/rsa_id.pub into /home/centos/.ssh/authorized_keys to do so.
  • The following checks for the communication between Master and Agent(s)

 [centos@ip-172-31-60-142 node-installer]$ ansible-playbook -i hosts pre-allow-master-agent-ssh.yml
 PLAY [setup master system so it is ready to ssh to nodes] **********************
 TASK [setup] 
 [...]

• • The following sets up the agent node using the master's node configuration and Lingoport installed components:

 [centos@ip-172-31-60-142 node-installer]$  ansible-playbook -i hosts setup-agents.yml
 PLAY [install pre-requisite programs to nodes]
 [...]

Subsequent Slave Node Installation from Master

Once the installation has been run once, you can repeat the installation (from Master) by simply changing the hosts file to point to other slave nodes to add. They must have been configured as above.

Adding the new node(s) to Master

At this point, the newly setup nodes should be available for Jenkins.

• Navigate to Jenkins Manage Node
• Add New Node and Launch the New Node. You should see something like:

 [01/12/17 23:35:23] [SSH] Opening SSH connection to 172.31.37.148:22.
 [01/12/17 23:35:23] [SSH] Authentication successful.
 [01/12/17 23:35:23] [SSH] The remote users environment is:
 BASH=/usr/bin/bash
 BASHOPTS=cmdhist:extquote:force_fignore:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath
 [...]
 <===[JENKINS REMOTING CAPACITY]===>channel started
 Slave.jar version: 2.62.3
 This is a Unix agent
 Evacuated stdout
 Agent successfully connected and online

• Check a job can be sent to the nodes: For instance, run a Debug job configured with some simple command like pwd and env.
  • The job must be using the remote label "Restrict where this project can be run". The remote label, for instance lingoport, was set up when adding the new node.
• In the Jenkins UI, the Build Execution Status widget should show "Master" and a number of slave nodes based on the nodes added so far.

The Pull Request Distributed Jobs

The first Lingoport job to be enabled is the GitHub Pull Request job. To configure it, copy the Lingoport.SampleLite-GitHubPullRequest job to your specific job, for instance Acme.FunWidgets-GitHubPullRequest. Make sure it is restricted to run on slave nodes identified by the label you created for those nodes, for instance lingoport.