Translating Engine, CLI and Backend

Table of contents

Overview

This document has instructions for translation in PO files and the Gettext system. It is used in Zonemaster-Engine, Zonemaster-CLI and Zonemaster-Backend. Zonemaster-GUI uses a different solution. See the Zonemaster GUI translation guide for instructions for that part.

PO files are found in the following directories:

  • share in Zonemaster-Engine
  • share in Zonemaster-CLI
  • share in Zonemaster-Backend

There is one PO file per language in each component. A PO file consists of multiple entries. An entry consists of three parts. First an optional header (lines starting with "#") which can have the "fuzzy" tag (more below), second the msgid which is the untranslated text (in English in the case of Zonemaster) that comes from the program code (Perl code in the case of Zonemaster), and the third part is the msgstr which is the translated text. The task of the translator is to create and update msgstr.

In many msgid there is one or several arguments within {...}, e.g. {ns_ip_list}. The same arguments must be found in msgid and msgstr. The value of the argument will be inserted at run time. The format of that value depends on the argument name, and a specification for arguments used, mainly in Zonemaster-Engine, can be found in Arguments for test case messages.

Software preparation

For the steps below you need to work on a computer with Git, Perl and Gettext. Select what OS you want to work on from the list below. Other OSs will also work, but you will have to find instructions elsewhere.

  • Rocky Linux

    Install the following:

    dnf install git make perl-App-cpanminus perl-Try-Tiny
cpanm Locale::PO

  • Debian

    Install the following:

    apt install gettext git liblocale-po-perl

  • FreeBSD

    Install the following:

    pkg install gettext-tools git-lite gmake p5-Locale-PO p5-Try-Tiny

  • Ubuntu

    Install the following:

    apt install gettext git make liblocale-po-perl

GitHub preparation

For full integration with Zonemaster translation you need a GitHub account and forks of the Zonemaster repositories. If you do not have one you can easily create an account at GitHub. If you are not willing to create one, contact the Zonemaster work group for instructions by sending an email to "zonemaster@zonemaster.net".

To create the forks of the Zonemaster repositories

  1. Go to GitHub and log in with your account.
  2. Go to the Zonemaster-Engine repository.
  3. Press "Fork" in the top right corner (see "Forking a repository").
  4. Repeat steps 2 and 3 for the the following repositories:

Make sure that your public ssh key is uploaded to GitHub (see "Adding a new SSH key to your GitHub account") and that its private key is available on the computer you are going to work from.

Tools

The PO file can be edited with a plain text editor, but then it is important to keep the database structure of the file. There are tools that makes editing of the PO files easier. When using those, the PO file is handled as a database instead of as a plain file.

  • There is Emacs PO-mode, available as a plug-in to the GNU Emacs file editor, which makes updating and searching in the ".po" file easier and more robust.
  • There is also "GNOME Translation Editor", a graphical PO editor available for at least Windows and Linux.
  • There is POEDIT in free version that Zonemaster translator has reported to work well.
  • There are more tools available, either cloud services or programs for download, and they could be found by searching for "po editor".

Clone preparation

You need a local clone of the repository to work in (see "Cloning a repository"). You need that for all repositories with messages to translate -- Zonemaster-Engine, Zonemaster-CLI and Zonemaster-Backend. (For Zonemaster-GUI see Zonemaster GUI translation guide.)

Here it is assumed that the three repositories are cloned from the same directory, i.e. all three clones are sub-directories to the same directory.

  • If you use "old" clones, instead of creating them afresh, then you must do the following (if there is no develop branch, checkout origin/develop and skip pull).

    git fetch --all
git checkout develop
git pull

  • Clone the three repositories:

    git clone https://github.com/zonemaster/zonemaster-engine
git clone https://github.com/zonemaster/zonemaster-cli
git clone https://github.com/zonemaster/zonemaster-backend

  • Check out origin/develop branch in the three repositories:

    git -C zonemaster-engine checkout origin/develop
git -C zonemaster-cli checkout origin/develop
git -C zonemaster-backend checkout origin/develop

You must always start from develop branch (or origin/develop branch) and any develop branch must be even with origin/develop.

Now it is time to connect your own forks created in GitHub preparation above to the created clones. We take Zonemaster-Engine as an example, but should be repeated for all clones.

  • You have a user name at GitHub. Here we use "xxxx" as your user name and also the name of the remote in clone on the local machine.

  • Enter the clone:

    cd zonemaster-engine

  • Create the connection to your fork by making it "remote" and synchronize to it:

    git remote add xxxx git@github.com:xxxx/zonemaster-engine.git
git fetch --all

Now you should have updated local clones of all three repositories.

Language Codes

Zonemaster uses ISO 639-1 two-letter language codes, in lower case. Zonemaster-GUI is currently available in the following languages:

  • da for Danish language
  • en for English language
  • es for Spanish language
  • fi for Finnish language
  • fr for French language
  • nb for Norwegian language
  • sl for Slovenian language
  • sv for Swedish language

If a language is added the correct two-letter code must be used.

In the steps below xx represent the language code that you should work with. Replace that with the correct code.

Translation steps

The steps in this section will cover most translation work. The steps are the same for all three repositories.

  • Make sure that the working tree is clean. Remove all other files and updates. If you have edited your PO file, then make a copy of it outside the repository.

    git clean -dfx
git reset --hard

  • The following command should report that the working tree is clean.

    git status --ignored

  • Check-out the develop branch and create a new branch to work in. You can call the new branch whatever you want, but here we use the name "translation-update". If that name is already taken, you have to give it a new name or remove the old branch.

    git checkout origin/develop
git checkout -b translation-update

  • Go to the share directory. Stay in the share directory for the remaining translation steps.

    cd share

  • If you have a partly edited PO file that you want to continue on, then it is time to copy it back into the share directory before the next step. Or use the PO file already in the share directory.

  • Run the update command for the PO file for the language you are going to work with. Replace "xx" with the language code in question. This will synchronize the PO file with the msgids in the Perl code.

    ./update-po xx.po

  • The PO file is updated with new msgids, if any, and now you can start working with it. Unless this is the first translation you only have to work with updated or untranslated items.

  • Update the PO file with the tool of your choice (see section "Tools" above). You can copy the PO file to another computer, edit it there, and then copy it back to your Zonemaster-Engine clone -- or Zonemaster-CLI clone or Zonemaster-Backend clone depending on what repository you do the translation for.

  • When doing the update, do not change the msgid, only the msgstr. The msgid cannot be updated in this process. They are the links between the Perl module and the PO file.

    • If you find a msgid that needs an update please see section Updates to msgids for how to report it.

  • If there is a {...} in the msgid then Arguments for test case messages will help with the type of data if not obvious.

  • Try to translate as close to the original as possible, but still make sure that the language is sound and valid.

  • Inspect every fuzzy entry (tagged with "fuzzy"). Update msgstr if needed and remove the "fuzzy" tag. The "fuzzy" tag must always be removed.

  • Search for untranslated entries (empty msgstr) and add a translation.

    • At the end of the file there could be obsolete entries (lines starting with "#~") and those could have matching translations, especially of the msgid has been changed.

  • Any remaining obsolete entries (lines at the end of the file starting with "#~") could be removed. They serve no purpose anymore.

  • With the following command, check that the messages arguments ({...}) in all msgstr strings match up with those in the msgid strings.

    ../../zonemaster-engine/util/check-msg-args xx.po

  • When the update of all msgstr is complete run the tidy-po command to create a consistent formatting of the PO file. Do this again if you do further updates.

    make POFILES=xx.po tidy-po

  • You can check all updates that have been done to the xx.po file. That includes both updates done by the update command and your updates.

    git diff xx.po

  • When the update is completed, it is time to commit the changes. You should only commit the "xx.po" file.

    git commit -m 'Write a description of the change' xx.po

  • There could be other files changed or added that should not be included. Run the status command to see them.

    git status

  • Other changed files could be reset by a git checkout. This could also be done before creating the commit.

    git checkout FILENAME

  • Files that were created but are not needed can just be removed. This could also be done before the commit.

    rm FILENAME

  • Now push the local branch you created to your fork at GitHub. "translation-update" is name of the branch you created above and have committed the updates to. Use your GitHub user name instead of "xxxx".

    git push -u xxxx translation-update

  • Go to your fork collection at GitHub, https://github.com/xxxx/ using your GitHub user name instead of "xxxx". There you can select the fork for this translation.

  • Select to create a new pull request. Here we use Zonemaster-Engine as an example.

    • The base repository should be zonemaster/zonemaster-engine.
    • The base branch should be develop (not master).
    • The "head" should be your fork.
    • The "compare" should be the same branch as you created above and pushed to your fork, e.g. "translation-update".

  • Inspect what GitHub says that will change by the pull request. It should only be the PO file that you have updated and nothing else. If additional files are listed, please correct or request for help.

    • If you by mistake have not started with an updated version of the develop branch you can get extra files listed.

  • If you want to correct something now before the pull request has been created, do that in your local clone and do a new commit and push that to your fork.

  • Press "create pull request", write a nice description and press "create" again.

  • If you go back to your own computer and just keep the clone as it is, you can easily update the pull request if needed with more changes to the same PO file. When the pull request has been merged by the Zonemaster work group, you can delete the local clone and on your GitHub fork you can remove the branch. Or keep them for next time.

Updates to msgids

The msgid cannot be updated in the translation process. If you find an msgid that needs an update you can provide that information to the project. The preferred way is to create an issue in the relevant repository.

  • Message (msgid) in Zonemaster-Engine: issue
  • Message (msgid) in Zonemaster-CLI: issue
  • Message (msgid) in Zonemaster-Backend: issue

If the message is in Zonemaster-Engine then include the message tag found in the header, e.g. "BASIC:NO_PARENT", in the issue description. Also include the msgid as it is now and a suggestion for new wording.

Adding a new language

If you want to add a new language, then follow steps in section "Translation steps" above with some modifications. Before you add a language contact the Zonemaster project to discuss timeplan and other aspects of the new language. Every language should be updated at every new release of Zonemaster if there are updated or added messages to be translated.

  • First, as in the translation steps, create the branch to work in. 
    git checkout origin/develop
git checkout -b translation-update

There is no PO file for new language, and it has to be created. The easiest way is to make a copy of an existing file.

  • Determine what language code to use for the new language (also see section "Language Codes").

  • It must be a code that is available in the locale system in the OSs that Zonemaster supports. Try the following commands to see if it is available. Replace "xx" with that code that you think it should be. Consult with Zonemaster Project if in doubt.

    locale -a | grep xx_      # Works in FreeBSD and Rocky Linux
grep xx_ /etc/locale.gen  # Works in Debian and Ubuntu

  • Go to the share directory and create a new blank PO file. Then "add" it to git before working on it.

    cd share
make POLANG=xx new-po
git add xx.po

  • Verify and update the file header if needed.

  • Now you go back to section "Translation steps" and continue in the same way as with an existing language.

Technical details

The first step in updating the translations is to generate a new template file ("Zonemaster-Engine.pot"). In practice you rarely need to think about generating it as it is generally performed as an implicit intermediate step. If you do want to generate it, the command is make extract-pot.

The translated strings are maintained in files named ".po". You will find the current PO files in the "share" directories listed in the "Overview" section (also see section "Language Codes").

The command ./update-po will update the PO file with new message ids (msgid) from the source code. Execute ./update-po xx.po to update the PO file for language "xx". This should only be strictly necessary to do when a module has been added, changed or removed, but it it recommended to do this step every time.

Execute make update-po to update all the PO files with new message ids from the source code.

By default the updated PO file will suggested translations for new message ids based on fuzzy matching of similar strings. This is not always desirable and you can disable fuzzy matching by executing one of the following commands instead:

make update-po MSGMERGE_OPTS=--no-fuzzy-matching POFILES=xx.po
make update-po MSGMERGE_OPTS=--no-fuzzy-matching