Managing translations with scripts/i18n tooling

-----------------------------------------------

The general idea with the ``scripts/i18n`` tooling is to keep changes in the

main repository focussed on actual and reviewable changes with minimal noise.

Noisy generated or redundant localization changes (that are useful when

translations) are contained in the ``kallithea-i18n`` repo on the ``i18n``

branch. The translation files in the main repository have no line numbers, no

untranslated entries, no fuzzy entries, no unused entries, and no constantly

changing records of "latest" this and that (name, date, version, etc).

The branches in the main repo (``default`` and ``stable``) will thus only have

stripped ``.pot`` and ``.po`` files: an (almost) empty

``kallithea/i18n/kallithea.pot`` file, and minimal ``.po`` files. There are no

binary ``.mo`` files in any repo - these are only generated when packaging for

release (or locally if installing from source).

Generally, ``kallithea/i18n/`` should not be changed on the ``default`` and

``stable`` branches at all. The ``i18n`` branch should *only* change

``kallithea/i18n/`` . If there are changesets with exceptions from that, these

changesets should probably be grafted/redone in the "right" place.

The basic flow is thus:

0. All weblate translation is done on the ``i18n`` branch which generally is

   based on the ``stable`` branch.

1. Graft the essential part of all new changes on the ``i18n`` branch to

   ``stable`` (while normalizing to current stripped state of stable).

2. Merge from ``stable`` to ``i18n`` (while normalizing to the resulting

   unstripped and fully ``msgmerge``'d state and ``.pot``-updating state).

3. Verify that the content of the ``i18n`` branch will give exactly the content

   of the ``stable`` branch after stripping. If there is a diff, something has

   to be fixed in one way or the other ... and the whole process should

   probably be redone.

Translate

^^^^^^^^^

First land full translation changes in the ``kallithea-i18n`` repo on the

``i18n`` branch. That can be done in pretty much any way you want. If changes

for some reason have to be grafted or merged, there might be odd conflicts due

to all the noise. Conflicts on the full ``i18n`` branch can perhaps be resolved

more easily using non-stripping normalization before merging::

  python3 setup.py extract_messages && cp kallithea/i18n/kallithea.pot full.pot && hg revert kallithea/i18n/kallithea.pot -r .

  hg resolve kallithea/i18n/ --tool X --config merge-tools.X.executable=python3 --config merge-tools.X.args='scripts/i18n normalized-merge --merge-pot-file full.pot $local $base $other $output'

Land in main repository - stripped

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

When the full i18n changes have landed on the ``i18n`` branch, prepare to land

them on ``stable``::

  hg up -cr stable

  python3 setup.py extract_messages && cp kallithea/i18n/kallithea.pot full.pot && hg revert kallithea/i18n/kallithea.pot

Consider all new ``i18n`` changes since last merge from ``stable``::

  hg log -G --style compact -r 'only("i18n", children(::stable))'

Graft them one by one (or in collapsed chunks) while normalizing.

If the graft has conflicts, use the ``scripts/i18n`` normalization tool to

apply ``msgmerge`` and strip before doing 3-way merge and resolving conflicts::

  hg resolve kallithea/i18n/ --tool X --config merge-tools.X.executable=python3 --config merge-tools.X.args='scripts/i18n normalized-merge --merge-pot-file full.pot --strip $local $base $other $output'

When all conflicts have been resolved, continue the graft::

  hg graft --continue

Then make sure any non-conflicting files are normalized and stripped too::

  scripts/i18n normalize-po-files --strip --merge-pot-file full.pot kallithea/i18n/*/LC_MESSAGES/kallithea.po

  hg ci --amend --config ui.editor=true

When things have been grafted to the ``stable`` branch, clean up history if

necessary: clean up the author and commit message when necessary, and perhaps

merge multiple changesets from same contributor.

Merge back to ``i18n``

^^^^^^^^^^^^^^^^^^^^^^

For any i18n changes that for some reason have been done on the ``stable``

branch, apply them manually on the ``i18n`` branch too - perhaps by grafting

and editing manually. The merge done in this step will `not` take care of it.

If the verification step done a bit later points out that something has been

missed, strip and go back to this point.

Then merge back to the ``i18n`` branch using normalization while keeping the

full ``.po`` files, and updating the full ``.pot`` and ``.po`` to current

state::

  hg up -cr i18n

  hg merge stable --tool internal:fail

  hg revert kallithea/i18n/*/LC_MESSAGES/*.po -r .

  hg resolve -m kallithea/i18n/*/LC_MESSAGES/*.po

  hg resolve -l  # verify all conflicts have been resolved

  python3 setup.py extract_messages && cp kallithea/i18n/kallithea.pot full.pot

  scripts/i18n normalize-po-files --merge-pot-file full.pot kallithea/i18n/*/LC_MESSAGES/kallithea.po

  hg commit  # "Merge from stable"

Note: ``normalize-po-files`` can also pretty much be done manually with::

  for po in kallithea/i18n/*/LC_MESSAGES/kallithea.po; do msgmerge --width=76 --backup=none --previous --update $po full.pot ; done

Note: Additional merges from ``stable`` to ``i18n`` can be done any time.

Verify

^^^^^^

Verify things are in sync between the full ``i18n`` branch and the stripped

``stable`` branch::

  hg up -cr stable

  hg revert -a -r i18n

  python3 setup.py extract_messages && cp kallithea/i18n/kallithea.pot full.pot && hg revert kallithea/i18n/kallithea.pot

  scripts/i18n normalize-po-files --strip --merge-pot-file full.pot kallithea/i18n/*/LC_MESSAGES/kallithea.po

  hg diff

If there is a diff, figure out where it came from, go back and fix the root

cause, and redo the graft/merge.

Push

^^^^

The changes on the ``stable`` branch should now be ready for pushing - verify

the actual changes with a thorough review of::

  hg out -pvr stable

When ``stable`` changes have been pushed, also push the ``i18n`` branch to the

``kallithea-i18n`` repo so Weblate can see it.