Difference between revisions of "Transform Framework"

From Lingoport Wiki
Jump to: navigation, search
(Use the transform framework)
(Use the transform framework)
 
(18 intermediate revisions by the same user not shown)
Line 1: Line 1:
 
LRM supports a number of file types out of the box (See [[Supported Resource Bundles]]). However, other file types may represent user facing strings to be translated. In that case, some customization is required to on-board those projects.
 
LRM supports a number of file types out of the box (See [[Supported Resource Bundles]]). However, other file types may represent user facing strings to be translated. In that case, some customization is required to on-board those projects.
The bash script transform framework facilitates the customization.
+
The '''bash script transform framework''' facilitates the customization.
   
 
= Analyze the file types =
 
= Analyze the file types =
Line 7: Line 7:
 
= Use the transform framework =
 
= Use the transform framework =
 
The transform framework needs '''three scripts''' in order to fit in with LRM. The three scripts need to be under the <code>$JENKINS_HOME/lingoport/transform/<nameoftransform>/</code> directory.
 
The transform framework needs '''three scripts''' in order to fit in with LRM. The three scripts need to be under the <code>$JENKINS_HOME/lingoport/transform/<nameoftransform>/</code> directory.
  +
   
 
The <nameoftransform> must be indicative of the type of transformation to apply. For instance, it could be <code>loc</code> to handle .loc files (see below). In that case, three scripts will need to be under <code>/var/lib/jenkins/lingoport/transform/loc</code> for a typical installation where the <code>jenkins</code> user is under <code>/var/lib/jenkins</code>.
 
The <nameoftransform> must be indicative of the type of transformation to apply. For instance, it could be <code>loc</code> to handle .loc files (see below). In that case, three scripts will need to be under <code>/var/lib/jenkins/lingoport/transform/loc</code> for a typical installation where the <code>jenkins</code> user is under <code>/var/lib/jenkins</code>.
  +
   
 
The three scripts to write are:
 
The three scripts to write are:
Line 15: Line 17:
 
* '''transform_files_list.sh''': How to transform the file names from the LRM supported file naming into the repository file naming
 
* '''transform_files_list.sh''': How to transform the file names from the LRM supported file naming into the repository file naming
   
  +
''Note'': The scripts must be executable. For example you can use "chmod +x *.sh" to make sure the bash scripts are all executable under that directory.
When those scripts are written, the transformation is defined in the config directory of the on-boarded project with the <code>transform.properties</code>. This file contains one properties, 'transform'. For instance, if <code>loc</code> is the directory with those three scripts under <code>$JENKINS_HOME/lingoport/transform/</code> for a <PROJECT> under a <GROUP>, the file will be:
 
  +
  +
From Jenkins, to specify which transform directory to use, navigate to the project configuration advanced 'transform' text field and enter the directory name, for instance <code>loc</code> :
  +
[[File:Transform Jenkins.JPG]]
  +
  +
  +
In this example, the transformation scripts need to be located under <code>$JENKINS_HOME/lingoport/transform/loc</code>.
  +
  +
Note: A configuration file on disk will be generated when running the Automation Jenkins job for that project:
   
 
<code>$JENKINS_HOME/Lingoport_Data/L10nStreamlining/<GROUP>/projects/<PROJECT>/config/transform.properties</code>
 
<code>$JENKINS_HOME/Lingoport_Data/L10nStreamlining/<GROUP>/projects/<PROJECT>/config/transform.properties</code>
 
<pre>
 
<pre>
transform=loc
+
l10n.transform=loc
 
</pre>
 
</pre>
   
  +
== Bash Variables ==
  +
A few Bash variables are available when called from the Lingoport Jenkins jobs that use the transform framework.
  +
They are set before calling the transform framework.
  +
  +
* '''CLIENT_SOURCE_DIR''' : For an LRM project such as CET.json, the CLIENT_SOURCE_DIR would typically be ~jenkins/jobs/CET.json/workspace. Note: This is not necessarily the WORKSPACE of the running Jenkins job from which the transform is called (Dashboard Update for instance).
  +
* '''LRM_GROUP_NAME''' : The name of the LRM Group Name (e.g. 'CET' )
  +
* '''LRM_PROJECT_NAME''' : The name of the LRM Project Name (e.g. 'json' )
  +
* '''TRANSFORM_DIR''' : The transform scripts directory (e.g. 'loc' )
   
 
== Example: .loc files ==
 
== Example: .loc files ==
Line 44: Line 62:
 
<pre>
 
<pre>
 
#!/bin/bash
 
#!/bin/bash
  +
echo "Transform loc files into .properties files"
 
# Find all the files ending in the 'input suffix' (e.g. resx)
+
# Find all the files ending in 'loc'
 
find $CLIENT_SOURCE_DIR -name "*loc" > ~/tmp/input_files.txt
 
find $CLIENT_SOURCE_DIR -name "*loc" > ~/tmp/input_files.txt
   
Line 57: Line 75:
 
ROOTNAME=${FILEPATH%$SUFFIX}
 
ROOTNAME=${FILEPATH%$SUFFIX}
 
TARGET="${ROOTNAME}.properties"
 
TARGET="${ROOTNAME}.properties"
iconv -f UTF-16 -t UTF-8 -c "$FILEPATH" > "$TARGET"
+
echo "[${FILENAME}] -> [${TARGET}]"
  +
cp "$FILEPATH" "$TARGET"
 
sed -i 's/^#/# #/' "$TARGET"
 
sed -i 's/^#/# #/' "$TARGET"
 
sed -i 's/^;/# ;/' "$TARGET"
 
sed -i 's/^;/# ;/' "$TARGET"
Line 70: Line 89:
 
#!/bin/bash
 
#!/bin/bash
   
# Find all the files ending in the 'input suffix' (e.g. resx)
+
# Find all the files ending in .properties
 
find $CLIENT_SOURCE_DIR -name "*.properties" > ~/tmp/input_files.txt
 
find $CLIENT_SOURCE_DIR -name "*.properties" > ~/tmp/input_files.txt
   
 
#
 
#
# Transform each .properties into a .loc
+
echo "Transform .properties files into .loc files"
 
#
 
#
 
cat ~/tmp/input_files.txt | while read -r FILEPATH
 
cat ~/tmp/input_files.txt | while read -r FILEPATH
Line 84: Line 103:
 
ROOTNAME=${FILEPATH%$SUFFIX}
 
ROOTNAME=${FILEPATH%$SUFFIX}
 
TARGET="${ROOTNAME}.loc"
 
TARGET="${ROOTNAME}.loc"
  +
echo "[${FILENAME}] -> [${TARGET}]"
 
cp "$FILEPATH" "$TARGET"
 
cp "$FILEPATH" "$TARGET"
 
sed -i 's/^#=#/#/' "$TARGET"
 
sed -i 's/^#=#/#/' "$TARGET"
Line 89: Line 109:
 
sed -i -e "s/^#\([[:alnum:]]*\)/;\1/" "$TARGET"
 
sed -i -e "s/^#\([[:alnum:]]*\)/;\1/" "$TARGET"
 
sed -i -e "s/\([[:alnum:]]*\)=/\1\t/" "$TARGET"
 
sed -i -e "s/\([[:alnum:]]*\)=/\1\t/" "$TARGET"
iconv -f UTF-8 -t UTF-16 -c "$TARGET" > tmp.tmp
 
mv tmp.tmp "$TARGET"
 
 
done
 
done
 
</pre>
 
</pre>

Latest revision as of 20:14, 22 September 2021

LRM supports a number of file types out of the box (See Supported Resource Bundles). However, other file types may represent user facing strings to be translated. In that case, some customization is required to on-board those projects. The bash script transform framework facilitates the customization.

Analyze the file types

If the file types fall into a category not supported by LRM out of the box, the first thing to do is to see what is the closest file types supported by LRM.

Use the transform framework

The transform framework needs three scripts in order to fit in with LRM. The three scripts need to be under the $JENKINS_HOME/lingoport/transform/<nameoftransform>/ directory.


The <nameoftransform> must be indicative of the type of transformation to apply. For instance, it could be loc to handle .loc files (see below). In that case, three scripts will need to be under /var/lib/jenkins/lingoport/transform/loc for a typical installation where the jenkins user is under /var/lib/jenkins.


The three scripts to write are:

  • transform_from_repo.sh: How to transform the files from the repository so they fit into an LRM supported file type
  • transform_to_repo.sh: How to transform translated/pseudo-localized files in an LRM supported file type into the repository file type
  • transform_files_list.sh: How to transform the file names from the LRM supported file naming into the repository file naming

Note: The scripts must be executable. For example you can use "chmod +x *.sh" to make sure the bash scripts are all executable under that directory.

From Jenkins, to specify which transform directory to use, navigate to the project configuration advanced 'transform' text field and enter the directory name, for instance loc : Transform Jenkins.JPG


In this example, the transformation scripts need to be located under $JENKINS_HOME/lingoport/transform/loc.

Note: A configuration file on disk will be generated when running the Automation Jenkins job for that project:

$JENKINS_HOME/Lingoport_Data/L10nStreamlining/<GROUP>/projects/<PROJECT>/config/transform.properties

l10n.transform=loc

Bash Variables

A few Bash variables are available when called from the Lingoport Jenkins jobs that use the transform framework. They are set before calling the transform framework.

  • CLIENT_SOURCE_DIR : For an LRM project such as CET.json, the CLIENT_SOURCE_DIR would typically be ~jenkins/jobs/CET.json/workspace. Note: This is not necessarily the WORKSPACE of the running Jenkins job from which the transform is called (Dashboard Update for instance).
  • LRM_GROUP_NAME : The name of the LRM Group Name (e.g. 'CET' )
  • LRM_PROJECT_NAME : The name of the LRM Project Name (e.g. 'json' )
  • TRANSFORM_DIR : The transform scripts directory (e.g. 'loc' )

Example: .loc files

Say the repository contains resource files like the following hmUiMessage.loc file:

;hmUiMessage.loc
;*********************************************************************
#include hmUiMain.loc
;*********************************************************************
message1	The first message
message2	The second message
message3	The third message
message4	The fourth message

The file may not be in ASCII or UTF-8 format; For instance this file is in UTF-16BE

A supported file format that is close to this one is properties.

transform_from_repo.sh

An example snippet of bash code for this type of file may be something like:

#!/bin/bash
echo "Transform loc files into .properties files"
# Find all the files ending in 'loc'
find $CLIENT_SOURCE_DIR -name "*loc" > ~/tmp/input_files.txt

# Transform each .loc file into a .properties file
cat ~/tmp/input_files.txt | while read -r FILEPATH
do
  FILENAME=`basename $FILEPATH`
  DIRNAME=`dirname $FILEPATH`
  file "$FILEPATH"
  SUFFIX=".loc"
  ROOTNAME=${FILEPATH%$SUFFIX}
  TARGET="${ROOTNAME}.properties"
  echo "[${FILENAME}] -> [${TARGET}]"
  cp "$FILEPATH" "$TARGET"
  sed -i 's/^#/# #/' "$TARGET"
  sed -i 's/^;/# ;/' "$TARGET"
  sed -i -e "s/[[:space:]]\+/=/" "$TARGET"
  sed -i -e "s/^=$//" "$TARGET"
done

transform_to_repo.sh

An example snippet of bash code for this type of file may be something like:

#!/bin/bash

# Find all the files ending in .properties
find $CLIENT_SOURCE_DIR -name "*.properties" > ~/tmp/input_files.txt

#
echo "Transform .properties files into .loc files"
#
cat ~/tmp/input_files.txt | while read -r FILEPATH
do
  FILENAME=`basename $FILEPATH`
  DIRNAME=`dirname $FILEPATH`
  ls -l "$FILEPATH"
  SUFFIX=".properties"
  ROOTNAME=${FILEPATH%$SUFFIX}
  TARGET="${ROOTNAME}.loc"
  echo "[${FILENAME}] -> [${TARGET}]"
  cp "$FILEPATH" "$TARGET"
  sed -i 's/^#=#/#/' "$TARGET"
  sed -i 's/^#=;/;/' "$TARGET"
  sed -i -e  "s/^#\([[:alnum:]]*\)/;\1/" "$TARGET"
  sed -i -e  "s/\([[:alnum:]]*\)=/\1\t/" "$TARGET"
done

transform_files_list.sh

An example snippet of bash code for this type of file may be something like:

#!/bin/bash
# Check if there is a parameter
if [ -z "$1" ]
  then
    echo "Error: Missing the argument like /<path>/pseudo_files.txt"
    exit 1
fi

# If the file exists then do something, otherwise exit
if [ -f "$1" ]; then
    echo " File to rewrite: $1"
else
    echo " $1 not found"
    exit 1
fi

# Rename .properties to .loc files inside the list of files passed as a parameter
sed -i 's/\.properties/.loc/' "$1"