hooks: move the vcs hook entry points and setup code out of lib

Mercurial hooks are running in a process that already has been initialized, so
they invoke the hooks lib directly. Git hooks are binaries and need a lot of
initialization before they can do the same. Move this extra setup code
elsewhere.

Having this high level code in bin is perhaps also not ideal, but it also
doesn't seem that bad: that is where other command line entry points invoke
make_app.

(It seems like it could be adventageous to somehow use "real" bin commands for
hooks ... but for now we use the home-made templates.)

Note: As a side effect of this change, all git hooks *must* be re-installed
when upgrading.

imports: try to use global imports unless it is a layering violation

To minimize the impact, do imports that violate the layering at runtime instead
of at import time.

lib: move webhelpers2 and friends to webutils

Gives less of the unfortunate use of helpers - especially in low level libs.

imports: always import the whole kallithea module to use top level kallithea variables

This is slightly more lazy and might avoid some depeendency issues.

hooks: add extensible create-pullrequest hook

Add a hook that will be called when a new pull request is created, and which
can be implemented in the 'extensions' package.

rcextensions: drop unused (and thus misleading) return values

Note that Mercurial only will check the hook exit value for some kinds of
hooks.

config: only assign kallitha.CONFIG once and keep it as a plain dict

Mute pytype warnings and make code more readable for developers:

File "kallithea/lib/hooks.py", line 318, in _hook_environment: No attribute 'global_conf' on Dict[nothing, nothing] [attribute-error]
File "kallithea/lib/hooks.py", line 318, in _hook_environment: No attribute 'local_conf' on Dict[nothing, nothing] [attribute-error]
File "kallithea/bin/kallithea_cli_base.py", line 82, in runtime_wrapper: No attribute 'global_conf' on Dict[nothing, nothing] [attribute-error]
File "kallithea/bin/kallithea_cli_base.py", line 82, in runtime_wrapper: No attribute 'local_conf' on Dict[nothing, nothing] [attribute-error]

isort: upgrade to 5.1.2

The changes to non-top-level imports seem nice.

py3: remove safe_unicode in places where it no longer is needed because all strings (except bytes) already *are* unicode strings

(The remaining safe_unicode calls are still needed and can't just be removed,
generally because we in these cases still have to convert from bytes to unicode
strings.)

cache: drop setup_cache_regions - tg will already have done that and coerced the types correctly

The configuration and type fixing will be invoked from make_base_app, and we
will thus not have to do it:

File "kallithea/config/middleware.py", line 31, in make_app_without_logging
return make_base_app(global_conf, full_stack=full_stack, **app_conf)
File ".../python3.7/site-packages/tg/configuration/app_config.py", line 176, in make_base_app
wrap_app)
File ".../python3.7/site-packages/tg/configurator/application.py", line 112, in _make_app
app = TGApp(conf)
File ".../python3.7/site-packages/tg/wsgiapp.py", line 49, in __init__
app_wrapper = wrapper(self.wrapped_dispatch, self.config)
File ".../python3.7/site-packages/tg/appwrappers/caching.py", line 36, in __init__
self.options = parse_cache_config_options(config)
File ".../python3.7/site-packages/beaker/util.py", line 430, in parse_cache_config_options

This will fix a py3 problem where setup_cache_regions was run *after* beaker
had coerced types, thus introducing string types in the config where beaker
expected the integers it had put there.

git: initialize hook app environment with paste as we do in other places

We want to use the whole paste / tg stack because it will do more
initialization for us - consistently, and in the right order.

This will allow a next change to avoid a problem in our custom beaker
initialization.

Going through paste might be a bit more heavy and slower than before, but it is
only run once for each git push.

Ultimately, we should probably get rid of the special hook entry point, and
just use kallithea-cli as the only entry point.

vcs: drop the superfluous and leaky hgcompat "layer"

Explicit is better. And gives less pyflakes noise.

git: more elegant handling of installed pre/post-receive hook failing on direct repo access

The hook would fail with a long backtrace when get_hook_environment raise an error exception.

Instead, as first thing in the entry point from the hook, catch that situation
and report it nicely before "quietly" skipping the hook:

[mk@here myrepo]$ git push
Enumerating objects: 3, done.
Counting objects: 100% (3/3), done.
Writing objects: 100% (3/3), 204 bytes | 204.00 KiB/s, done.
Total 3 (delta 0), reused 0 (delta 0)
remote: Skipping Kallithea Git post-recieve hook 'hooks/post-receive'.
remote: Git was apparently not invoked by Kallithea: Environment variable KALLITHEA_EXTRAS not found
To /tmp/somerepo
* [new branch] master -> master
[mk@here myrepo]$

We could be paranoid and let it (and the pre hook) fail ... but that doesn't
seem helpful.

Reported by Edmund Wong at [1].

[1] https://lists.sfconservancy.org/pipermail/kallithea-general/2019q4/003071.html

py3: add b'' annotations in some places where they will be needed later

Mostly entirely trivial adding of b prefix that is a ignored for py2 ... and
also a bit of related trivial reformatting/refactorings.

vcs: refactor run_git_command to just return stdout as unicode string

This will be convenient for py3.

ssh: introduce 'kallithea-cli ssh-serve' command for providing actual protocol access over ssh

To be invoked from ~/.ssh/authorized_keys as
command=".../kallithea-cli ssh-serve -c .../my.ini 1007 7",... ssh-rsa AAA...=

The command is not supposed to be used directly by users, and is thus hidden.

Based on work by Ilya Beda <ir4y.ix@gmail.com> on
https://bitbucket.org/ir4y/rhodecode/commits/branch/ssh_server_support ,
also incorporating updates for gearbox by Anton Schur <tonich.sh@gmail.com>,
and further heavily refactored and rewritten by Mads Kiilerich.

hooks: move _handle_rc_scm_extras to utils2 as set_hook_environment and get_hook_environment

This is also a step away from using the name 'extras' ... but still a lot of
traces left behind ...

auth: global permissions given to the default user are the bare minimum and should apply to *all* other users too

Drop the "subtractive permission" config option "inherit_from_default" that
when set to false would give users less global permissions than the default
unauthenticated user.

Instead, think positive and merge all positive permissions.

At the end, filter the global permissions to make sure we for each kind of
permissions only keep the one with most weight.

hooks: refactor log_push_action

The core of the functionality is to process a list of "raw id"s, log them, and
update / invalidate caches.

handle_git_post_receive and scm _handle_push already provide that list
directly. Things get much simpler when introducing a new function
(process_pushed_raw_ids) just for processing pushed raw ids. That also makes it
clear that scm _handle_push doesn't need any repo.

log_push_action remains the native entry point for the Mercurial hook. It was
not entirely correct using 'node:tip' - after Mercurial 3.7 and d6d3cf5fda6f,
it should be 'node:node_last'.

After several trivial refactorings, it turns out that the logic for creating
the hash list for Mercurial actually is very simple ...

utils: refactor make_ui to always read from db, optionally also augmenting with hgrc content

simplehg was the only user of reading hgrc content ... and it would rather just
have everything at once.

The implementation in utils is moved around and re-indented, but without
significant changes.

hooks: make sure push and pull hooks always are enabled

Don't put things in the database when we pretty much assume they always have
exact content, without any reasonable use case for customization.

locking: drop the pull-to-lock / push-to-unlock functionality

The feature is not worth the maintenance cost. The locking is too coarse and
unflexible with insufficient UI and UX. The implementation is also quite
invasive in tricky areas of the code, and thus high maintenance.
Dropping this will enable other cleanup ... or at least make it easier.

git: fix push to empty repo (Issue 323)

Git would fail to log revisions when the list of heads to exclude included an
empty string (in place of the pushed ref).

To avoid that, skip the skipped revision instead of making it an empty string.
`git log --not` works fine without providing any revisions to "not".

Verify in test_push_new_repo_git that it actually logged the push.

hooks: slight cleanup of handling of scm_repo.run_git_command return values

It would still be nice to have better error handling, but this makes it
slightly more clear what is going on and what error handling we don't have..

cache: move cache invalidation from web handler to post push hook

We need the post push hook anyway ... and having the cache invalidation here
will also work for ssh pushes in the future.

The name log_push_action is thus no longer spot-on. That might change later,
but requires some care as it also is used directly as hook name.

Note that having cache invalidation in the hook will do that debug logging no
longer will appear in the server log.

Based on a patch by Dominik Ruf.

hooks: clarify return value from Git hooks

don't rely on the implicit None

hooks: use os.environ directly for KALLITHEA_EXTRAS instead of sometimes passing it as parameter

It is better to be consistent ... even if that means always using the global
environment that is hard to avoid.

hooks: empower the Git entry points and avoid the big handle_git_receive dispatch

Use the new _hook_environment and enjoy the small and clean hooks.

Just moving code - no change.

hooks: extract a separate _hook_environment from handle_git_receive

We want to split handle_git_receive up, and this self-contained chunk is what
will be shared.

Just moving code - not changing comments or anything.

hooks: rename hooks to reflect what they are doing

The names of our hooks should reflect what they are doing.
So, since pre_push and pre_pull only handle the locking, rename
pre_push->push_lock_handling and pre_pull->pull_lock_handling.

This imply a database migration step which must be run as described in the
upgrade documentation.

Also rename db.Ui class variables.

hooks: rename 'push_data' and 'revs' to git_stdin_lines

...to better reflect what it is.

codingstyle: trivial whitespace fixes

Reported by flake8.

lib: fix Git hook failure after TG2 port

This fixes manual_test_vcs_operations.py failures after e1ab82613133 dropped
test_env and test_index parameters to load_environment without fixing
handle_git_receive().

tests: use the DB set in TEST_DB for git hooks

Obviously the git hooks and the web app need to use the same database.
Therefore, if the web app uses the TEST_DB environment variable, the git hooks
needs to do the same.

model: move code from __init__.py to base.py

Having too much code, in particular too much imports, inside a package's
__init__.py is a recipe for circular imports, and considered bad practice in
Python [1]

Move out everything from kallithea/model/__init__.py to a new file
kallithea/model/base.py and adapt the existing imports.

[1] http://docs.python-guide.org/en/latest/writing/structure/#packages

db: PullRequest/Repository/RepoGroup/UserGroup: change 'user' to 'owner'

Rename the 'user' and 'user_id' fields on the four classes to something
more informative. The database column names remain unchanged for now;
a later Alembic script can fix the name of these and other columns to
match their Python name.

This might break rcextensions, though, and external scripts that use
the HTML form interface.

Turbogears2 migration: remove some references to Pylons in comments

In order to minimize the diff of the actual Turbogears2 migration, this
commit already removes certain unnecessary references to Pylons from the
Kallithea source base.
Places where the reference to Pylons is important are still kept for now, as
well as references in kallithea/config where many changes are made for
Turbogears2 anyway.

Turbogears2 migration: use sqlalchemy.url iso sqlalchemy.db1.url

In Turbogears2, much of the application initialization is handled by the
framework, whereas in Pylons the application was responsible for it.
Initializing SQLAlchemy is one such part of initialization which is handled
by Turbogears2.

Turbogears2 expects the configuration file to refer to the database using
'sqlalchemy.url' rather than the current 'sqlalchemy.db1.url'. While the
exact name is not really important, not following this approach means we'll
need to override the sqlalchemy initialization method.

Therefore, as a preparation to the Turbogears2 migration, already change the
database reference string under Pylons.
When upgrading to a version of Kallithea containing this commit, the .ini
file will manually need to be adapted to remove the .db1 strings.

hooks: always convert unicode to byte strings when passed to ui.status

Kallithea generally uses unicode strings internally, but ui.status follows the
Mercurial convention and expects a byte string. Strings passed to ui.status
should thus always by converted to byte strings. Do that explicitly with
safe_str. (The alternative of using more byte strings internally seems less
appealing.)

hooks: fix encoding problems of lock release ui messages

The vcs test test_push_unlocks_repository_hg would fail when waitress tried to
append a unicode chunk to the byte stream. The unicode string came from the
'Released lock on repo' message passed to ui.status from the push hook.

ui.status do however follow the Mercurial convention and expects a byte string.

We solve the problem by applying safe_str.

Patch modified by Mads Kiilerich.

hooks: parse incoming git refs correctly

Hooks receive a line of the following format on standard input:

<old-value> SP <new-value> SP <ref-name> LF

where <old-value> is the old object name stored in the ref,
<new-value> is the new object name to be stored in the ref
and <ref-name> is the full name of the ref.

This means, we have to strip at least the LF in order to have a
correct version of the ref name after the split. Also, when
parsing the ref name itself, use all components but first instead
of just second, as a ref name may have slashes in it.

Previously, failure to parse ref name correctly would lead to the
following behaviour. A newly created repository with no commits pushed
has HEAD set to refs/heads/master by default, even though there's no
such ref in the repository yet. Upon first push, Kallithea rewrites
this symbolic reference with a reference to a real branch.

However, due to a bug in ref name parsing, if a ref name had a slash,
Kallithea would update HEAD to an invalid reference:

git push origin feature/branch

would rewrite HEAD to refs/heads/feature. All future attempts to work
with this repository would fail because dulwich would complain it can't
read HEAD as it is a directory.

git: make sure repository path and name are Unicode strings

This is needed for supporting latest Dulwich version, but it also works with
out supported version.

tests: Mercurial hooks must use ui.status for messages sent to the client

Mercurial changed so sys.stdout and sys.stderr now are intercepted instead of
being sent to the client. That caused a regression that
manual_test_vcs_operations.py test_push_new_file_hg caught - we no longer
reported 'Repository size' to the user.

The issue was introduced by a Mercurial change, but the fix is backwards
compatible with Mercurial 2.9.

The fix is to use ui.status everywhere. ui is a Mercurial thing, but
handle_git_receive also creates a ui object for Git hooks so ui.status can be
used there too.

git: run external commands as list of strings so we really get correct quoting (Issue #135)

a6dfd14d4b20 from
https://bitbucket.org/conservancy/kallithea/pull-request/17/add-quotes-to-repo-urls-for-git-backend
fixed that issue but did not make it "safe". The vcs git backend still used
command strings but tried to quote them correctly ... but that approach is
almost impossible to get right.

Instead, pass a string list all the way to the subprocess module and let it do
the quoting. This also makes some of the code more simple.

Correct licensing information in individual files.

The top-level license file is now LICENSE.md.

Also, in various places where there should have been joint copyright holders
listed, a single copyright holder was listed. It does not appear easy to add
a link to a large list of copyright holders in these places, so it simply
refers to the fact that various authors hold copyright.

In future, if an easy method is discovered to link to a list from those
places, we should do so.

Finally, text is added to LICENSE.md to point to where the full list of
copyright holders is, and that Kallithea as a whole is GPLv3'd.

Second step in two-part process to rename directories.
This is the actual directory rename.