Transform Framework

Localyzer supports a number of file types out of the box (See Supported Resource Bundles). However, other file types may represent user facing strings to be translated. In that case, some customization is required to on-board those projects. The bash script transform framework facilitates the customization.

Analyze the file types

If the file types fall into a category not supported by Localyzer out of the box, the first thing to do is to see what is the closest file types supported by Localyzer and TMSs/LSPs/MTs.

Use the transform framework

The transform framework needs three scripts. The three scripts need to be in a repository, under a directory whose name will be used to identify the transform.

The three scripts to write are:

• transform_from_repo.sh: How to transform the files from the repository so they fit into an LRM supported file type
• transform_to_repo.sh: How to transform translated/pseudo-localized files in an LRM supported file type into the repository file type
• transform_files_list.sh: How to transform the file names from the LRM supported file naming into the repository file naming

Each of these script can in turn call other scripts as required. Be sure that when calling other bash shell scripts to use "source" or "." to do so. Command Center will automatically set the above three mentioned scripts to executable when used but does not do so for other scripts that may be added as part of the transform.

if source "${TRANSFORM_DIR}/additional_code.sh" "${arg1}" ; then
    echo "Successfully executed ${TRANSFORM_DIR}/additional_code.sh"
else
    echo "Failed to execute ${TRANSFORM_DIR}/additional_code.sh"
fi

For example, under

• https://github.com/Lingoport/CommandCenterConfig, the main branch

a number of transform directories indicate what transform are available, in other words, what directories have those three scripts.

For example, for the txt2prop transform, the directory is located :

• https://github.com/Lingoport/CommandCenterConfig/tree/main/transforms/txt2prop

Bash Variables

A few Bash variables are available when called from the Lingoport Jenkins jobs that use the transform framework. They are set before calling the transform framework.

Transform from repository variables

CLIENT_SOURCE_DIR : The top level directory of the workspace for this project
TRANSFORM_DIR : The directory of the transform, the .sh are underneath this directory
PROJECT_TMP_DIR : A tmp directory

Transform to repository variables

CLIENT_SOURCE_DIR : The top level directory of the workspace for this project
TRANSFORM_DIR : The transform scripts directory where the scripts are executed from.
PROJECT_TMP_DIR : Where temporary files can be stashed for this project during the script execution
$1 : The first parameter is the list of all the files to be transformed from the translated files back to the repository

Example: txt2propfiles

For a simple example, say the repository contains resource files like the following strings.txt file:

# Strings for use in GUI
#################################################
PushIfCommits,Push if commits
CloseButtonText,Close
ApplyButtonText,Apply
CancelButtonText,Cancel

The file may not be in ASCII, UTF format, etc.; For instance this file is in UTF-8

A supported file format that is close to this one is properties. So the transform script will change the file name, extension, and content, so as to be a standard resource file:

strings.properties

# Strings for use in GUI
#################################################
PushIfCommits=Push if commits
CloseButtonText=Close
ApplyButtonText=Apply
CancelButtonText=Cancel

transform_from_repo.sh

An example snippet of bash code for this type of file may be something like:

#!/bin/bash
find . -name "string*\.txt" -type f > "${PROJECT_TMP_DIR}/input_files.txt"

cat "${PROJECT_TMP_DIR}/input_files.txt" | while read -r FILEPATH
do
  FILENAME=`basename $FILEPATH`
  DIRNAME=`dirname $FILEPATH`
  SUFFIX=".txt"
  ROOTNAME=${FILENAME%$SUFFIX}
  TARGET_NAME="${ROOTNAME//-/_}.properties"
  TARGET_PATH="${DIRNAME}/${TARGET_NAME}"
  echo "    Transform [$FILENAME] -> [$TARGET_NAME]"

  rm $TARGET_PATH 2> /dev/null
  touch $TARGET_PATH

  sed -i 's/,,/, ,/' $FILEPATH 
  cat $FILEPATH | while read -r LINE
  do
      IFS=',' tokens=( $LINE )

      if [ -z "${tokens[1]}" ]
      then
        echo "${LINE}" >> $TARGET_PATH
      else
        KEY=${tokens[0]}
        VALUE=${LINE#"${KEY},"}
        echo "${KEY}=${VALUE}" >> $TARGET_PATH
      fi
  done
IFS=' '

done 

transform_to_repo.sh

An example snippet of bash code for this type of file may be something like:

#!/bin/bash
find . -name "string*\.txt" -type f > "${PROJECT_TMP_DIR}/input_files.txt"

cat "${PROJECT_TMP_DIR}/input_files.txt" | while read -r FILEPATH
do
  FILENAME=`basename $FILEPATH`
  DIRNAME=`dirname $FILEPATH`
  SUFFIX=".txt"
  ROOTNAME=${FILENAME%$SUFFIX}
  TARGET_NAME="${ROOTNAME//-/_}.properties"
  TARGET_PATH="${DIRNAME}/${TARGET_NAME}"
  echo "    Transform [$FILENAME] -> [$TARGET_NAME]"

  rm $TARGET_PATH 2> /dev/null
  touch $TARGET_PATH

  sed -i 's/,,/, ,/' $FILEPATH 
  cat $FILEPATH | while read -r LINE
  do
      IFS=',' tokens=( $LINE )

      if [ -z "${tokens[1]}" ]
      then
        echo "${LINE}" >> $TARGET_PATH
      else
        KEY=${tokens[0]}
        VALUE=${LINE#"${KEY},"}
        echo "${KEY}=${VALUE}" >> $TARGET_PATH
      fi
  done
IFS=' '

done 

transform_files_list.sh

An example snippet of bash code for this type of file may be something like:

#!/bin/bash
if [ -z "$1" ]
  then
    echo "Error: Missing the argument like /<path>/pseudo_files.txt"
    exit 1
fi

if [ -f "$1" ]; then
    echo " File to rewrite: $1"
else
    echo " $1 not found"
    exit 1
fi


echo " "
echo " --------------------------------------------"
echo " Files to Modify:  $1"
echo " for repository formatted files, not LRM OOTB ones"

# strings<locale>.properties -> strings<locale>.txt
echo "   >>  strings<locale>.properties to strings<locale>.txt"
sed -i 's/\.properties/.txt/' "$1"
sed -i 's/strings_/strings-/' "$1"
sed -i 's/strings-zh_Hans/strings-zh-Hans/' "$1"
sed -i 's/strings-zh_Hant/strings-zh-Hant/' "$1"
sed -i 's/_/-/g' "$1"

echo " " 
ls -l "$1"
cat "$1"
echo " --------------------------------------------"

Command Center Settings

Transform System Files

The transform is retrieved like any system file from the repository and set as a system file.

• Put Command Center in Maintenance Mode
• Go to Settings
• Go to System Files
• Add the Transform from the repository

For example:

When the repository is updated and the transform scripts are modified, update from the Edit button above to get the latest. The commit hash indicates which commit was last.

Set the Transform for a Project

Now that the Transform is ready to be used, a project can be configured to run the transform. To do so, either during the creation or the edit of the project. 1. Make sure the project resource format is that of the transformed file, not the files in the repository. Here, that's .properties files. 2. Click the 'Use transform script' checkbox and select the transform of your choice, here 'txt2prop'

For example:

Now whenever the project is analyzed or files are sent to translation, the repository is cloned or pulled, the files identified with .txt are transformed into .properties, then analyzed or sent to translation as properties.

Files are received from translation as properties, are analyzed for correctness, etc. as properties, then if the validations are passed, the files are transformed back in to .txt files and pushed to the repository.