AI Quality Review
Introduction
Starting with Command Center 5.0, Localyzer can translate, review, and score strings using LLMs directly from Command Center. A Translation Management System (TMS) can be used but is no longer required to get a quality-reviewed translation into the repository: the LLM translates, a second LLM task evaluates the result, and people only need to look at the strings the scores point them to.
AI Quality Review works on both sides of a string:
- Source strings are scored for correctness before they are translated. This catches developer errors early, such as malformed ICU MessageFormat syntax, encoding problems, typos, or text that does not make sense.
- Target strings (translations) are scored for translation quality, for each locale.
Scores, reviews, edits, re-translations, and pushes back to the repository are all handled from String Manager.
- The score comes from the Quality LLM. 1 is worst, 5 is best. For instance,
- The source string "G2 Mechanic Droid" is scored 3. Expand the view to see why.
- The translated string "Corvette CR90" is scored 3 as well
- The review flags are set by a user: grey is not reviewed; yellow is for strings needing a review; green for reviewed/OK strings
- The source string "Sphyrna Corvette" has been OKed by a human reviewer
- The "Droïde neurochirurgien" needs attention, as per a human reviewer.
Sorting by score helps human reviewers focus on the most likely strings to modify. Filtering by review status helps reviewers only see those strings which have not been reviewed yet or need the most attention
How It Works
The overall workflow is:
- Developers create and maintain the source strings in the repository.
- Localyzer analyzes the project. The LLM configured for quality review scores the source strings.
- Strings are sent to the LLM configured for translation, with the prompts and context (images) associated with each string.
- The translations come back and the quality review LLM scores each target string.
- Reviewers filter for low scores or unreviewed strings, then for each string either:
- accept it as is and mark it reviewed,
- edit the translation directly in Command Center, or
- improve the context or prompt and resend it to translation. The new translation is scored again.
- When satisfied, the reviewer pushes the changes back to the repository.
Who Changes What
The existing responsibility split between developers and Localyzer is kept:
| String | Owner | Where it is changed |
|---|---|---|
| Source strings | Developers | In the repository. String Manager links to the key/value pair in GitHub, GitLab, Bitbucket, etc. A low source score is a signal to the development team. |
| Target strings | Translators / reviewers | In Command Center String Manager, then pushed to the repository by Localyzer. |
Developers should continue not to edit target files directly, to avoid merge conflicts with the Localyzer branch.
Configuration
LLM Connections
AI Quality Review uses the LLM connections defined in Translation Management System & Machine Translator Connections. See LLM for the supported providers, API endpoints, and model examples.
To add a connection, provide:
- the provider type (OpenAI-compatible, Claude, or Gemini)
- the API endpoint URL
- the API token
- the model
Separate LLMs for Translation and Quality Review
The translation task and the quality review task are configured independently. Each can use a different LLM connection, model, and prompt, and the choice can differ from one project to another.
This allows you to:
- use a different model to evaluate a translation than the one that produced it, which is recommended so the reviewer is not grading its own work,
- use a smaller, cheaper model for one task and a stronger model for the other, to reduce token usage and cost,
- pick the models that perform best for a given project or set of locales.
Quality Review Prompts
Like translation, quality review can be customized with prompts. The translation prompt and the quality prompt are set separately. See Quality Review Prompting.
Context images associated with a string are also available to the quality review, so the LLM can judge a translation against how the string appears in the UI. Make sure the selected model supports images.
Scores
Each scored string receives a value from 1 to 5, where 5 is the best:
| Score | Meaning |
|---|---|
| 5 | Excellent: no change needed |
| 4 | Good: minor or stylistic issues |
| 3 | Acceptable: review recommended |
| 2 | Poor: significant issues, review needed |
| 1 | Unusable: wrong meaning, broken syntax, or not a valid string |
Source or target strings can be sorted by score, allowing the user to focus on the most likely issues first. The Review flag (grey, yellow, green) can be used to indicate which strings have been reviewed, so as to only look at unreviewed strings, either from source or from translation.
Source scores
The source score indicates whether a source string is correct and ready to be translated. Bad grammar, spelling, punctuation, vulgarity, or casing are examples of issues that lead to a low score.
When on-boarding a project or adding a Quality LLM to a project, source strings which have been translated prior do not need to be scored. Localyzer sets the Reviewed flag to true. These strings may have existed for months or years and do not tend to need reviews are scoring.
Non translated strings will be scored. Modified strings will be scored. New strings will be scored. The Reviewed flag will be unset.
Target scores
The translation score indicates translation quality for each locale. Translated strings prior to on-boarding a project or adding a Quality LLM to a project, do not need a score and the Reviewed flag is set.
Scoring is done on translated strings are handled by Localyzer (new translation post Quality Review LLM setting).
Score Reason
To view the reason for a score less than 5, expand the string view. The reason and candidate remediations will be listed. More than one reason may less to a low score.
Reviewing Strings in String Manager
[Admin, Manager, Translator]
It is recommended to sort by score and first check the low score translations. Expand the string view (on the right for each string) to see the LLM notes on the score and/or possible other translations. Set 'Reviewed' (green), 'Needs attention' (yellow) or 'To be reviewed' (grey) accordingly.
Filtering
In addition to the existing String Manager filters, filter strings by:
- score, for instance to show only low scoring strings
- review status: reviewed, needs attention, or not reviewed
- modified, not yet pushed: target strings edited in Command Center that are not in the repository yet
Combining a low score filter with the not reviewed filter gives the list of strings that need attention, so reviewers can make quick decisions instead of reading every string.
Editing a Translation
Open the expanded view of a string to see its source, its translations, and their scores. Edit a target string directly and save it.
Edited strings are tracked in Command Center. The string shows who modified it and when, until it is pushed to the repository.
To push the modification to the data source (repository), click the button on the left panel. The number of changed strings are shown for this locale.
Resending to Translation
When a translation is not good enough, improve what the LLM has to work with and translate again:
- add or replace a context image for the string,
- add or update the string prompt (or the project prompt),
- select the string(s) with the multi string selector and resend them to translation for the locales concerned.
The new translations are scored again when they come back.
Pushing to the Repository
Once the reviewer is satisfied, click the Push button to commit the modified target strings back to the repository, on the Localyzer branch configured for the project.
Using AI Quality Review with a TMS
AI Quality Review does not replace TMS integrations. Projects that send strings to a TMS keep working as before. Source scoring is useful in all cases, as it catches problems before strings are sent to translation, whichever translation connection is used.
Token Usage
Scoring consumes tokens on the quality review LLM connection, in addition to translation tokens. Token counts are shown on the LLM connection and in the project logs, as described in Token Count.