setup: make chardet a mandatory dependency

It was made mandatory in 9685f50a69d0. At that time it seemed like it always
were present due to indirect dependencies, but apparently that is no longer the
case.

hg: read hgrc, also in repo_scan

This might make a difference in some real world cases.

db: read hgrc in Repository.scm_instance_no_cache and Gist.scm_instance

This will make sure .hg/hgrc is read in some code paths where it was missing.
That might make a difference in some real world cases.

i18n: updated translation for Portuguese

Currently translated at 45.8% (498 of 1087 strings)

i18n: updated translation for Greek

Currently translated at 98.7% (1073 of 1087 strings)

i18n: updated translation for Greek

Currently translated at 97.6% (1061 of 1087 strings)

i18n: updated translation for Greek

Currently translated at 97.5% (1060 of 1087 strings)

i18n: updated translation for Russian

Currently translated at 100.0% (1087 of 1087 strings)

i18n: updated translation for Russian

Currently translated at 100.0% (1087 of 1087 strings)

i18n: updated translation for Greek

Currently translated at 94.9% (1032 of 1087 strings)

i18n: updated translation for Russian

Currently translated at 100.0% (1087 of 1087 strings)

i18n: updated translation for Polish

Currently translated at 64.1% (697 of 1087 strings)

i18n: updated translation for Greek

Currently translated at 94.9% (1032 of 1087 strings)

i18n: updated translation for German

Currently translated at 69.0% (751 of 1087 strings)

i18n: updated translation for French

Currently translated at 99.7% (1084 of 1087 strings)

i18n: updated translation for Portuguese

Currently translated at 45.8% (498 of 1087 strings)

i18n: updated translation for Russian

Currently translated at 99.3% (1080 of 1087 strings)

hg: restore make_ui reading of .hg/hgrc

This regressed with f83326e2e66c, released in 0.6.0 .

Fix problem reported by Felipe Cardoso Resende.

hg: support Mercurial 5.8

Some 'remote:' messages changed to appear on stderr.

pytype: drop pytype_requirements.txt appenlight dependency

appenlight support was dropped in 09826e98b064.

tests: address pedantic pytype 2021.4.1 warning

File "kallithea/tests/api/api_base.py", line 1315, in test_api_fork_repo_non_admin_no_permission_to_fork: Name 'fork_name' is not defined [name-error]

diffs: let DiffProcessor inject context_lineno in chunk changes

Make it possible to use these numbers reliably, also outside as_html.

diffs: drop pointless check for "\ No newline at end of file"

Such lines would have been caught on '\' as "unknown command" by the command
check a few lines above.

diffs: drop pointless check for adding context line

The length of the match was always 5.

comments: don't open comment form on page load for comments on line 0

Data attributes have magic parsing of strings so '0' becomes 0 which is false.

Further, 0 == '' in JavaScript, so we have to use !== '' to catch empty strings.

diffs: DiffParser should only HTML escape lines when generating HTML

Move DiffParser HTML escaping so it only is applied when generating HTML.

diffs: add test coverage of inline diff with other markup

Add trailing whitespaces ... but also whitespace before changes (which might
seem trailing but isn't).

mail: drop unicode filter from templates

Unicode is the default in py3, and we no longer pass any non-string arguments
that need stringification before rendering.

This is backing out a part of 4fbab9d5be4d.

pytype: introduce configuration for running pytype

Install pytype and dependencies on Python 3.8 with:
pip install --upgrade -r pytype_requirements.txt

Checks can then be run with:
pytype

pytype: add assertion to guide pytype through Whoosh usage

Mute pytype warnings:

File "kallithea/controllers/admin/admin.py", line 105, in _journal_filter: No attribute 'fieldname' on None [attribute-error]
In Optional[Any]
File "kallithea/controllers/admin/admin.py", line 106, in _journal_filter: No attribute 'text' on None [attribute-error]
In Optional[Any]
File "kallithea/controllers/admin/admin.py", line 107, in _journal_filter: No attribute 'startdate' on None [attribute-error]
In Optional[Any]
File "kallithea/controllers/admin/admin.py", line 107, in _journal_filter: No attribute 'enddate' on None [attribute-error]
In Optional[Any]

pytype: add some assertions to guide pytype through const dict with tricky typing structure

These assertions also make the code more explicit and slightly more readable.

Mute pytype warnings:

File "kallithea/tests/vcs/base.py", line 76, in setup_repo: No attribute 'path' on str [attribute-error]
In Union[kallithea.lib.vcs.nodes.FileNode, nothing, str]
File "kallithea/tests/vcs/base.py", line 76, in setup_repo: No attribute 'content' on str [attribute-error]
In Union[kallithea.lib.vcs.nodes.FileNode, nothing, str]

pytype: add Python type annotations where necessary to guide pytype

Mute pytype warnings:

File "kallithea/lib/auth.py", line 142, in _cached_perms_data: No attribute 'DEFAULT_USER_ID' on module 'kallithea' [module-attr]
File "kallithea/lib/vcs/backends/base.py", line 73, in ...: No attribute '...' on BaseRepository [attribute-error]
File "kallithea/lib/vcs/backends/base.py", line 405, in ...: No attribute '...' on BaseChangeset [attribute-error]
File "kallithea/tests/api/api_base.py", line 2397, in test_api_get_changeset: No attribute 'TEST_REVISION' on _BaseTestApi [attribute-error]
File "kallithea/tests/api/api_base.py", line 2445, in test_api_get_pullrequest: No attribute 'TEST_PR_DST' on _BaseTestApi [attribute-error]
File "kallithea/tests/api/api_base.py", line 2445, in test_api_get_pullrequest: No attribute 'TEST_PR_SRC' on _BaseTestApi [attribute-error]
File "kallithea/tests/api/api_base.py", line 2467, in test_api_get_pullrequest: No attribute 'TEST_PR_REVISIONS' on _BaseTestApi [attribute-error]
File "kallithea/tests/api/api_base.py", line 67, in api_call: No attribute 'app' on _BaseTestApi [attribute-error]
File "kallithea/tests/base.py", line 154, in log_user: No attribute 'app' on TestController [attribute-error]
File "kallithea/tests/base.py", line 169, in _get_logged_user: No attribute '_logged_username' on TestController [attribute-error]

pytype: mute errors from import of optional or platform specific modules

Mute pytype warnings:

File "kallithea/bin/kallithea_cli_iis.py", line 69, in iis_install: Can't find module 'isapi_wsgi'. [import-error]
File "kallithea/config/post_receive_tmpl.py", line 24, in <module>: No attribute 'setmode' on module 'msvcrt' [module-attr]
File "kallithea/config/pre_receive_tmpl.py", line 24, in <module>: No attribute 'setmode' on module 'msvcrt' [module-attr]
File "kallithea/lib/compat.py", line 59, in kill: No attribute 'windll' on module 'ctypes' [module-attr]
File "kallithea/lib/utils.py", line 242, in is_valid_repo_uri: Can't find module 'hgsubversion.svnrepo'. [import-error]
File "kallithea/tests/scripts/manual_test_concurrency.py", line 203, in <module>: No attribute '_RandomNameSequence' on module 'tempfile' [module-attr]

bin: add missing imports

Spotted by pytype --strict-import:
File "kallithea/bin/kallithea_cli_db.py", line 73, in db_create: No attribute 'utils' on module 'kallithea.lib' [module-attr]
File "kallithea/bin/kallithea_cli_db.py", line 73, in db_create: No attribute 'scm' on module 'kallithea.model' [module-attr]

It happened to work anyway due to the import chain ... but it is better to be
explicit.

ssh: handle case where hg serve_forever doesn't serve forever

serve_forever will no longer exit in Mercurial 5.7 .
This changed with https://mercurial-scm.org/repo/hg/rev/b7b8a1538161 .

hooks: on Git, invoke hooks/post-receive-custom from hooks/post-receive

Make it possible for admins to install all kinds of hooks.

Based on a patch by Tim Ooms.

A more generic solution would be nice, but for now we aim for this minimal
solution.

hooks: clarify in UI and documentation how the hook control essentially is Mercurial only

The Git hook model doesn't directly allow multiple hooks of the same kind.

hooks: when overwriting non-Kallithea hooks, move the old hook to .bak

Note: This will only happen when actually overwriting an unknown hook. Repeated
hook installation will thus not clobber the existing .bak file until a
non-Kallithea hook is found again.

hooks: be more consistent in only using active Ui entries

There is no UI to control or display the ui_active value for custom hooks, but
*if* they are inactive, they will be ignored in make_ui, and it will be
misleading and confusing to show them in the list of active custom hooks.

There *should* never be any inactive hooks entries, but let's be consistent in
handling the case *if* it should happen. (It happened for me while hacking
around.)

hooks: put repo_size and update hooks in kallithea namespace

Keep things separate.

Include missing migration steps for 642847355a10.

hooks: use internal Mercurial module for update after push

Don't rely on having a working 'hg' command in $PATH.

hooks: fix incorrect visual nesting in the admin UI

A form-group was missing.

celery: upgrade to Celery 5.0 ... and adjust for Click API

Celery 5 has apparently no relevant API or config changes.

Celery is however switching to click. run_from_argv goes away, and there is no
simple way to do as before and start the worker with our Celery app but still
use Celery's own command line parser.

Apply hacks to make sure it still is possible to run like:

kallithea-cli celery-run -c my.ini -- --help

kallithea-cli celery-run -c my.ini -- --loglevel=ERROR broker_url=amqp://u:p@localhost:5672/v

celery: move send_email task to a better home in notification model

Avoid bundling everything from many different layers in one big task library.

This is more feasible now when we don't need kallithea.CELERY_APP set at import
time.

celery: move repo create tasks to repo model - that's where it belongs

Avoid bundling everything from many different layers in one big task library.

This is more feasible now when we don't need kallithea.CELERY_APP set at import
time.

The whole thing about model classes and instantiation is just trouble - we will
get rid of that soon.

celery: move Whoosh indexing task to whoosh library where it belongs

Avoid bundling everything from many different layers in one big task library.

This is more feasible now when we don't need kallithea.CELERY_APP set at import
time.

celery: let async tasks choose at runtime if they should use immediate execution or dispatch to the Celery worker

Make it completely safe to use task annotation at import time, before global
config has been set.

celery: celery-run should only initialize app and sqlalchemy after workers have been forked

If app and SqlAlchemy were initialized before launching celery, the forked
workers would inherit the database connection ... and that doesn't work.

This could be handled by disposing the engine after forking the worker or
before each task ... but it remains unnecessary and wrong to initialize the
engine early when it isn't used, and then fork it.

celery: change celery-run to not take a whole app - take a config file, and then initialize app

A step towards a better future ...

celery: don't rely on CELERY_APP only being set (from app_cfg) when use_celery

We would rather always have CELERY_APP set.

Kind of backing out 894a662b12b3.

cli: in runtime_wrapper for config file handling, don't return unused command handler value

Return values are not used - we should use sys.exit or similar to set command
return codes.

celery: drop tracking of task_id - we use ignore_result=True and will never get anything back

There is thus no need for configuration of celery.result_backend .

The alternative would be to fix it. That could give better error reporting from
failing repo creations, but would require quite a bit of additional changes
before it actually works reliably.

celery: drop traces of support for task_always_eager

It was broken - retrieval of results didn't work.

Also, task_always_eager still require a working celery.broker_url even though
it is unused. It is thus much better to just set use_celery=false .

celery: always drop db session after running a task

Drop dbsession annotation - it could easily be forgotten. This seems simpler
and more reliable.

celery: use regular automatic session handling for tasks

There were no point in having a special celery get_session() .

celery: use explicit task names - avoid automatic naming with "kallithea.lib.celerylib." prefix

We wrap async functions in a local f_async wrapper, defined in
kallithea/lib/celerylib/__init__.py . For a function Foo.X, even though we
changed the wrapper's __name__ to X, the tasks would be named
kallithea.lib.celerylib.X , without using the actual module name of X for
namespacing.

Drop modifying __name__, and just specify the name explicitly, without trying
to namespace it.

TurboGears: drop workaround for < 2.4

Backout a38e05a0c79e and tweak.

api: fix update_repo check for permission to create top level repos

The hg.create.repository permission only apply to creation (or renaming) of top
level repos - it is not relevant for other kinds of renaming.

Moving or renaming repos in other locations is now covered by other checks.

api: fix repo group permission check for repo creation

hg.create.repository is only controlling whether all users have write access at
top level. For other repo locations, check for write permission to the repo
group.

Note: This also covers "repo creation" by forking or by moving another repo.

api: fix repo creation inside repo groups when using Celery

Repo creation would fail when Celery tried serializing a whole RepoGroup
object. It only worked when not using Celery.

model: simplify how get_commits_stats task group on author

Avoid using the caching h.person . We want to get rid of the model
dependency on helpers.

The stats are persisted, and any temporary incorrectness in the long term
cached h.person will thus remain forever. It is thus arguably better to avoid
using it in this place.

get_commits_stats is also a long running task, so speed is not *that* critical.
And generally, processing commits in order will have a lot of the same
committers, so a local cache will have a good hit rate.

(Alternatively, h.person could perhaps be in user model ... but that's not how
it is now.)

lib: move some template filter functions from utils2 to webutils

While quite Kallithea specific, we prefer to have these functions in webutils
where they soon can be exposed when templates don't need the whole helpers
module.

controllers: move controllers base class from lib/base to controllers

TG quickstart put it in lib/base.py , but it fits better on the controllers
layer as a base there.

The contributing docs were a bit ahead of time ... but with a typo.

indexers: drop sys.path mangling in daemon

The original comment in b153a51b1d3b was "to get the pylons_app import" ...
which seems like a bad workaround for not having installed the app properly
before running.

tests: drop vcs setup_package

Repos are always created before running vcs tests, and the SCMFetcher code was
thus effectively dead.

tests: always pass username as cur_user

Passing a whole User wouldn't work if actually using celery and thus
serializing the parameters. The test scenario was thus different from actual
runtime.

tests: fix create_fork to actually pass a repo id as fork_parent_id

Passing a whole repo wouldn't work if actually using celery and thus
serializing the parameters. The test scenario was thus different from actual
runtime.

api: don't pass owner to create_repo in form_data

create_repo gets owner as 'cur_user', but 'owner' in form_data was unused.

celery: use kallithea.CELERY_APP.AsyncResult instead of celery.result import

This is more spot-on and more correct.

celery: drop kallithea.CELERY_EAGER - it is more spot-on to look directly at task_always_eager

The global flag was set as a side effect in the make_app function. That's not
pretty.

celery: fix CELERY_APP creation after breakage in ea1c608efa3a

A part of change still under development had sneaked in.

vcs: fix subprocessio process termination

659ecd26002c had a cut'n'paste error so it dropped process termination in the
SubprocessIOChunker close method.