ini: explain how % must be escaped as %% (Issue #375)

Gearbox / paste would fail with:
configparser.InterpolationSyntaxError: '%' must be followed by '%' or '('

The special handling of % is standard configparser behaviour, unless explicitly
fetching raw settings.

db: drop double specification of unique constraint (Issue #374)

Consistently use "unique=True" for single column constraints. Only use
UniqueConstraint for composite constraints.

MySQL did not like this double specification, while PostgreSQL and SQLite
did not care.

ini: put kallithea-config version in the ini file it creates (Issue #373)

Help keeping track of how the .ini was created.

docs: clarify the installation steps and how things fit together

Hint that create-db user information is for creating the initial local user
and that the database connection (which also might use some kind of user) is
something else.

tests: coverage of Git repo feeds

Coverage of the case reported in Issue #372.

feeds: fix failure getting feed for Git repos (Issue #372)

GitChangeset.diff() did
return ''.join(self.repository.get_diff(...))
even though get_diff returned a string. It worked, but was unnecessary and
inefficient.

That fails in py3: get_diff returns bytes ... and iterating doesn't give
characters but integers and we would get:
TypeError: sequence item 0: expected a bytes-like object, int found

Fixed by dropping the unnecessary iteration and joining.

This function is only used for feeds statistics.

auth: simplify handling of Crowd json response (Issue #370)

Correct error where Crowd authentication didn't work due to urllib.readlines()
returning bytes and thus failing to be joined with a string.

json.loads is however happy to take bytes directly.

Fix error handling to also handle bytes without crashing.

contributors: map different names of J. Lavoie onto one

Assuming the last used format is the right one.

i18n: updated translation for German

Currently translated at 68.9% (750 of 1087 strings)

i18n: updated translation for French

Currently translated at 98.9% (1076 of 1087 strings)

i18n: updated translation for French

Currently translated at 98.9% (1076 of 1087 strings)

i18n: updated translation for Russian

Currently translated at 100.0% (1087 of 1087 strings)

i18n: updated translation for French

Currently translated at 98.1% (1067 of 1087 strings)

i18n: updated translation for French

Currently translated at 98.0% (1066 of 1087 strings)

i18n: updated translation for Greek

Currently translated at 94.8% (1031 of 1087 strings)

i18n: updated translation for Luxembourgish

Currently translated at 6.1% (67 of 1087 strings)

issue: log error if still using issue_pat_XXX configuration after it was removed in d24051ce961c

Show a helpful:
found unsupported issue_prefix_pr = 'PR' - use issue_sub_pr instead
before bailing out with:
skipping incomplete issue pattern 'issue_pat_pr': '(?:PR\\s*)(\\d+)' -> 'https://kallithea-scm.org/repos/kallithea/pull-request/{id}' None

scripts/i18n: update i18n howto with recipe for use of scripts/i18n tooling

For now, just adding a new section. It should be integrated with the existing
documentation. Some steps can perhaps be simplified or automated so we need
less words.

inifile: add doctest coverage of comment corner cases for trailing space and value of None

Shows room for improvement - doesn't show any bugs.

The value of None is undefined behaviour.

tests: fix TestPermissions.test_inactive_user_group_does_not_affect_repo_group_permissions_inverse

Handle L1_new leak from test_rename_top_level_group_in_nested_setup .

tests: fix test_auth_ldap dependency on test order to avoid failure on uniqueness constraint

It would perhaps be even better to have each test clean up ... but let's only
aim for only avoiding collision on leaked test data.

tests: fix test_admin_settings test dependencies

Failure could be reproduced with
py.test kallithea/tests/functional/test_admin_settings.py -k x

uwsgi: enable reload-on-exception in configuration template

In case of severe failures in the stack (not handled by TG), it will thus
report "monitored exception detected, respawning worker ...".

hg: support Mercurial 5.4

rc0 has no release notes yet, but it seems to work and the test suite pass ...

ini: tweak template configuration for TG's handling of application errors

Mention what was learned while debugging request processing at WSGI/TG level
and the complexity of the TG stack gets in the way.

inifile: new implementation of setting updates to optimize reuse of comments and append location

Code comments should explain the algorithm, and test coverage suggest it works
correctly.

inifile: add doctest coverage of comment corner cases

Shows room for improvement - doesn't show any bugs.

uwsgi: clarify that --ini-paste-logged currently requires pastescript

This dependency is removed with uWSGI commit
https://github.com/unbit/uwsgi/commit/0807b3881f701a6cc1cc95fe872bb7ec2426ef3f
and should normally be included in the next release after 2.0.18.

uwsgi: slim down and tweak the default '[uwsgi]' configuration section

The goal is to have a basic working setup that show how and why to use uWSGI.
System administrators should check uWSGI documentation for further information
and general advice about operations.

Based on uWSGI documentation, experiments, and
https://www.techatbloomberg.com/blog/configuring-uwsgi-production-deployment/ .

uwsgi: drop unnecessary dependency of http module - just use http-socket directly

The http plugin has more advanced http functionality like https and load
balancing, duplicating what in many setups is handled by separate front-end
servers. In this simple template, just use the basic http-socket.

db: fix bad characters in __repr__

Introduced in effd091203ae.

i18n: fix dead code in Accept-Language workaround from 7c7d6b5c07c7

AppConfig is just providing defaults, and the .ini file has not been read yet.
There is thus no point in checking if i18n.lang has been set, and the code
would thus *always* set i18n.lang=en ... but that was fine, as it just is a
default.

email templates: fix missing translation of titles and buttons

The buttons and titles of email templates were not correctly translated. The
corresponding strings were not part of the i18n files because they were not
recognized by the extraction logic.

Note: this will generate new translation strings.

Reported-by: Jan Heylen <jan.heylen@nokia.com>

db: drop Repository CacheInvalidation

The benefit of this functionality is questionable. Especially in bigger setups
with multiple front-end instances all serving the same multitude of
repositories, making the hit rate very low. And the overhead of storing cache
invalidation data *in* the database is non-trivial.

We preserve a small cache in Repository SA records, but should probably just in
general know what we are doing and not ask for the same information multiple
times in each request.

db: drop SA caching_query and FromCache, and thus sql_cache_short beaker cache

It is not a good idea to have dead ORM objects. If we want caching, we should
do it explicit.

It is unknown how much this cache helps, but we can profile and introduce
better caching of simple data where relevant.

scripts: properly check for errors in whitespacecleanup/run-all-cleanup

An error in whitespacecleanup, like a bug in isort that raises an exception,
should be treated as a problem instead of silently ignored.

setup: install pip in virtualenv to make sure we have the latest version

Older versions are good enough for bootstrapping (and might also be good for
everything else) but gives:

WARNING: You are using pip version 19.3.1; however, version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

The user might still get this warning initially, but then it goes away ...

middleware: verify as early as possible in app initialization that the necessary configuration is provided

In addition to verifying a random config setting, also verify that config has
been read as Paste does it, an invariant introduced in 488b52cad890.

i18n: better stripping of header comment for new translations

Also strip any header occurrence of:

# FIRST AUTHOR <EMAIL@ADDRESS>, 2019.
#

while preserving stripping of:

#, fuzzy

and leading empty comment lines.

db: consistently use base_path from config instead of repeatedly getting from the database

Avoid using the inefficient Repository.base_path ... and avoid even more the
misleading Repository.repo_path .

config: set base_path config in set_app_settings using Ui.get_repos_location() instead of in app_cfg using make_ui()

Only hit the database once (when starting the application) to get this.

It would perhaps be more elegant to set it directly, for example in
kallithea.base_path ... but it seems like a setting that really belongs in the
.ini file ...

auth: for default permissions, use existing explicit query result values instead of following dot references in ORM result objects

There has been reports of spurious crashes on resolving references like
.repository from Permissions:

File ".../kallithea/lib/auth.py", line 678, in __wrapper
if self.check_permissions(user):
File ".../kallithea/lib/auth.py", line 718, in check_permissions
return user.has_repository_permission_level(repo_name, self.required_perm)
File ".../kallithea/lib/auth.py", line 450, in has_repository_permission_level
actual_perm = self.permissions['repositories'].get(repo_name)
File ".../kallithea/lib/vcs/utils/lazy.py", line 41, in __get__
value = self._func(obj)
File ".../kallithea/lib/auth.py", line 442, in permissions
return self.__get_perms(user=self, cache=False)
File ".../kallithea/lib/auth.py", line 498, in __get_perms
return compute(user_id, user_is_admin)
File ".../kallithea/lib/auth.py", line 190, in _cached_perms_data
r_k = perm.UserRepoToPerm.repository.repo_name
File ".../sqlalchemy/orm/attributes.py", line 285, in __get__
return self.impl.get(instance_state(instance), dict_)
File ".../sqlalchemy/orm/attributes.py", line 721, in get
value = self.callable_(state, passive)
File ".../sqlalchemy/orm/strategies.py", line 710, in _load_for_state
% (orm_util.state_str(state), self.key)

sqlalchemy.orm.exc.DetachedInstanceError: Parent instance <UserRepoToPerm at ...> is not bound to a Session; lazy load operation of attribute 'repository' cannot proceed (Background on this error at: http://sqlalche.me/e/bhk3)

Permissions are cached between requests: SA result records are stored in in
beaker.cache.sql_cache_short and resued in following requests after the initial
session as been removed. References in Permission objects would usually give
lazy lookup ... but not outside the original session, where we would get an
error like this.

Permissions are indeed implemented/used incorrectly. That might explain a part
of the problem. Even if not fully explaining or fixing this problem, it is
still worth fixing:

Permissions are fetched from the database using Session().query with multiple
class/table names (joined together in way that happens to match the references
specified in the table definitions) - including Repository. The results are
thus "structs" with selected objects. If repositories always were retrieved
using this selected repository, everything would be fine. In some places, this
was what we did.

But in some places, the code happened to do what was more intuitive: just use
.repository and rely on "lazy" resolving. SA was not aware that this one
already was present in the result struct, and would try to fetch it again. Best
case, that could be inefficient. Worst case, it would fail as we see here.

Fix this by only querying from one table but use the "joinedload" option to
also fetch other referenced tables in the same select. (This might
inefficiently return the main record multiple times ... but that was already
the case with the previous approach.)

This change is thus doing multiple things with circular dependencies that can't
be split up in minor parts without taking detours:

The existing repository join like:
.join((Repository, UserGroupRepoToPerm.repository_id == Repository.repo_id))
is thus replaced by:
.options(joinedload(UserGroupRepoToPerm.repository))

Since we only are doing Session.query() on one table, the results will be of
that type instead of "structs" with multiple objects. If only querying for
UserRepoToPerm this means:
- perm.UserRepoToPerm.repository becomes perm.repository
- perm.Permission.permission_name looked at the explicitly queried Permission
in the result struct - instead it should look in the the dereferenced
repository as perm.permission.permission_name

docs: fix contributing.rst reference to Session.remove()

(Second to) last example of Session.remove was removed in 1a409593f352.

py3: fix display of default user permissions

Avoid http://localhost:5000/vcs_test_hg/settings/permissions with explicit user
permissions failing:

TypeError: '<' not supported between instances of 'str' and 'bool'

db: drop superfluous space in UserGroup relationship definition

It doesn't seem like it had any actual impact.

scripts/i18n: let 'normalized-merge' leave 3-way conflict markers in output file

3-way conflict markers in the output file can be convenient ... and they do no
harm when merge tools like kdiff3 will overwrite output anyway.

scripts/i18n: also normalize casing of UTF-8 in Content-Type

f626260a376c introduced invariant msgmerge casing. Do the same when normalizing
to ensure consistency also without msgmerge and to avoid unnecessary conflicts.

scripts/i18n: introduce --merge-pot-file to control normalization

There are actually *two* kinds of normalization:

- in main branches, where we just want the translations - not any trivially
derived information or temporary or unstructured data.
- in i18n branches, where we want the trivially derived information, and also
want to preserve any other information there might be in the .po files.

If no pot file is specifed, do it as on the main branches and strip everything
but actual translations. This mode will primarily be used when grafting or
rebasing changes from i18n branches.

When a pot file is specified, run GNU msgmerge with it on the po files. The pot
file should ideally be fully updated (as done by extract_messages). That will
establish a common baseline, leaving only the essential changes as needing merge.

If merging from default branches to 18n, it is better to skip .po and .pot in
first 'hg merge' pass, while resolving everything else. Then, with the
uncommitted merge, run 'extract_messages', and then merge the .po files using
--merge-pot-file kallithea/i18n/kallithea.pot .

(Actually, these two different modes could perhaps be auto detected ...)

scripts/i18n: add command 'normalized-merge' for use with Mercurial's 'merge-tool' option

Add a 'normalized-merge' command to scripts/i18n that will first normalize
the i18n files contributing to the merge, then perform a standard merge. If
that merge fails (e.g. due to real conflicts) the normalized files are left
behind, and the user needs to run another merge tool manually and resolve the
merge of these.

Use by putting following snippets in your .hgrc file:

[merge-tools]
i18n.executable = /path/to/scripts/i18n
i18n.args = normalized-merge $local $base $other $output

or

[merge-tools]
i18n.executable = python3
i18n.args = /path/to/scripts/i18n normalized-merge $local $base $other $output

and when i18n files conflict, get all 3 sides of the merge normalized before
merge by running:

hg resolve 'kallithea/i18n/*/LC_MESSAGES/kallithea.po' --tool i18n

scripts/i18n: add command 'normalized-diff'

Add a command 'normalized-diff' that takes two (po) files and diff normalized
copies of them.

scripts/i18n: add command 'normalize-po-files'

The translation files in the Kallithea repository contained references to
the location(s) of each string in the repository. This is useful to
translators, but is not needed for all other users.

The big problem with that information is that it changes very commonly as a
result of normal development in Kallithea, causing a lot of unimportant
delta in the Kallithea repository, thus causing unnecessary repository
growth.

In this commit, a basic version of the script is added, only containing the
code to normalize the translation files by removing generated and outdated
data.

This can be used to check or ensure internal consistency between code and
translations, by extracting and merging and then removing most of it again with
normalize-po-files:

./setup.py extract_messages

for po in kallithea/i18n/*/LC_MESSAGES/kallithea.po; do msgmerge --width=76 --backup=none --previous --update $po kallithea/i18n/kallithea.pot ; done

scripts/i18n normalize-po-files kallithea/i18n/kallithea.pot kallithea/i18n/*/LC_MESSAGES/kallithea.po

Includes contributions from Mads Kiilerich.

scripts/i18n: introduce new i18n maintenance script

The translation files in the Kallithea repository contained references to
the location(s) of each string in the repository. This is useful to
translators, but is not needed for all other users.

The big problem with that information is that it changes very commonly as a
result of normal development in Kallithea, causing a lot of unimportant
delta in the Kallithea repository, thus causing unnecessary repository
growth.

A script 'i18n' is added to help maintain the i18n files. Functionality will be
added later.

vcs: avoid node base class knowledge of sub classes

Pytype very reasonably got confused over the sub class property references in
the base class.

Fixed by moving base class specific parts of comparison to the sub classes and
calling the base class.

With .content only referenced in FileNode, there is even less need for an
explicitly failing .content getter for DirNode.

vcs: simplify nodes kind handling

Avoid pytype's very reasonable confusion over the _kind handling.

Node.kind is actually only ever set in __init__, so there is not much need for
a paranoid setter.

scripts: fix crash from comparing integer with empty list

Fixed by using year 0 as default for contributors without any years.

scripts: in regexps, use + when we actually want to make long matches instead of empty

Greedyness seems to have changed with Python 3 and it is less forgiving and
would happily make empty matches everywhere.

scripts: use plain import of contributor_data

Relative import failed with:

Traceback (most recent call last):
File "scripts/update-copyrights.py", line 45, in <module>
from . import contributor_data
ImportError: attempted relative import with no known parent package

Fixed by backing out changeset 2786730e56e0. The pytype problem can be solved
differently.