git: show warning when removing existing unused Kallithea hook

These hooks are no longer used and should be removed, but it is nice to have
explicit logging when it happens.

hg: read hgrc, also in repo_scan

This might make a difference in some real world cases.

hooks: when overwriting non-Kallithea hooks, move the old hook to .bak

Note: This will only happen when actually overwriting an unknown hook. Repeated
hook installation will thus not clobber the existing .bak file until a
non-Kallithea hook is found again.

cleanup: get rid of redundant assignments 'foo = foo'

Redundant assignments were searched with:

grep '^ *\(.*\) = \1$' `hg files`

Some of these cases have resulted from earlier cleanup, like:

foo = safe_unicode(foo)

and later batch removing safe_unicode usage. See e.g. commit
e35373106528b29461b7f5144780cb9ec79f765d .


Note: two false positives occur:

kallithea/tests/models/test_settings.py:39:
setting.app_settings_value = setting.app_settings_value
kallithea/tests/models/test_settings.py:42:
setting.app_settings_value = setting.app_settings_value

which is not a plain assignment but where 'app_settings_value' is a property
method with side effects.

hooks: import whole hooks module

Minimize impact if there should be cycles.

git: detect existing symlink hooks before overwriting - only update plain files

If the existing hook is a symlink, the hook is not what we put in place, and we
should stay away - no matter if it points at something that looks like a
Kallithea hook.

git: write Git hook files atomically

Make sure we don't follow symlinks or inherit permissions from previously
installed hook.

git: minor refactor and clean-up of hook installation code

Make "pre-receive" and "post-receive" more explicit and grepable.

scm: simplify InMemoryChangeset handling

Avoid conditional imports.

config: move various py templates to kallithea/templates/py/

For some reason, we had some python templates in kallithea/config .
kallithea.config is mainly the TG entry point, and thus a high level controller
thing - not a place to store templates.

Instead, use the templates directory and introduce a new py subdirectory.

With git hook templates in a templates directory, there is no need for tmpl in
the name.

imports: always import the whole kallithea module to use top level kallithea variables

This is slightly more lazy and might avoid some depeendency issues.

scm: make sure __get_repo always returns something

Avoid pytype complaint:
File ".../kallithea/model/scm.py", line 438, in get_nodes: No attribute 'scm_instance' on None [attribute-error]
In Optional[Any]

Avoid invoking __get_repo with None as parameter when creating repos.

scm: refactor install_git_hooks

Rename, simplify, and negate some logic to make the flow more readable to me
and give better logging.

For example, "force_create" were more about "force overwrite". Calling it
"force" is more precise.

py3: remove safe_unicode in places where it no longer is needed because all strings (except bytes) already *are* unicode strings

(The remaining safe_unicode calls are still needed and can't just be removed,
generally because we in these cases still have to convert from bytes to unicode
strings.)

py3: trivial renaming of .iteritems() to .items()

A bit like "2to3 -f dict", but we don't want list().

py3: switch to use Python 3 interpreter, temporarily leaving many things very broken until they have been migrated/fixed in a reviewable way

Bump Mercurial minimum version to 5.2 - the first version that claim stable py3
support.

py3: add b'' annotations in some places where they will be needed later

Mostly entirely trivial adding of b prefix that is a ignored for py2 ... and
also a bit of related trivial reformatting/refactorings.

cleanup: minor refactorings and simplification of dict usage

Makes it more py3 compatible.

py3: use explicit octal literals

From 2to3 numliterals.

model: use repo.set_invalidate() instead of accesssing CacheInvalidation directly

Trying to contain the use of CacheInvalidation ...

scm: select options should be tuples - not lists with 2 elements

We use tuples in all(?) other places, so we should do the same here.

Since lists can be unpacked as tuples, it currently works to use 2-lists. But
in a following change for "correct" handling of webhelpers.select when using
webhelpers2.select, we want to give special handling to tuples vs strings vs
Options. We don't want to add list to that list.

hooks: move _handle_rc_scm_extras to utils2 as set_hook_environment and get_hook_environment

This is also a step away from using the name 'extras' ... but still a lot of
traces left behind ...

scm: don't try to get IP address from web request in model

Remove a layering violation and make functions more reusable when they no
longer depend on global state.

At this level, the IP address (and information about the current user) is only
used for hooks logging push / pull operations. Arguably, IP address logging
only belongs in an HTTP access log, not in the log of push/pull operations. But
as long as we have IP addresses in the logs, we have to provide it. The (good?)
alternative would be to drop IP address from the push / pull logs ...

hooks: drop 'server_url' parameter

The parameter might be conceptually nice, but it was only available for 2
hooks. To be really useful, it should have been available everywhere. It also
only reported the URL from the web request that initiated the hook ... and as
such it does some layering violations. The user URL might be the address that
should be used internally from the hook. And it can conceptually not be made
available actions that doesn't originate from a user web request. It seems much
better that custom hooks configure what they want to do. Perhaps by reading the
.ini file and using canonical_url.

hooks: refactor log_push_action

The core of the functionality is to process a list of "raw id"s, log them, and
update / invalidate caches.

handle_git_post_receive and scm _handle_push already provide that list
directly. Things get much simpler when introducing a new function
(process_pushed_raw_ids) just for processing pushed raw ids. That also makes it
clear that scm _handle_push doesn't need any repo.

log_push_action remains the native entry point for the Mercurial hook. It was
not entirely correct using 'node:tip' - after Mercurial 3.7 and d6d3cf5fda6f,
it should be 'node:node_last'.

After several trivial refactorings, it turns out that the logic for creating
the hash list for Mercurial actually is very simple ...

utils: refactor make_ui to always read from db, optionally also augmenting with hgrc content

simplehg was the only user of reading hgrc content ... and it would rather just
have everything at once.

The implementation in utils is moved around and re-indented, but without
significant changes.

locking: drop the pull-to-lock / push-to-unlock functionality

The feature is not worth the maintenance cost. The locking is too coarse and
unflexible with insufficient UI and UX. The implementation is also quite
invasive in tricky areas of the code, and thus high maintenance.
Dropping this will enable other cleanup ... or at least make it easier.

hooks: make the Python interpreter for Git hooks configurable as 'git_hook_interpreter' (Issue #333)

Commit 5e501b6ee639 introduced the use of 'sys.executable' as interpreter
for git hooks instead of 'python2' with the following argument:

"Windows doesn't necessarily have "python2" available in $PATH, but we
still want to make sure we don't end up invoking a python3. Using the
absolute path seems more safe."

But, sys.executable does not necessarily point to Python. When Kallithea is
started under uWSGI, sys.executable points to the uwsgi executable. As a
result, the interpreter encoded in the git hooks on the server repositories
would be:

#!/path/to/uwsgi

And pushing to such repo would result in following client errors:

$ git push
Password for 'http://user@localhost:5050':
Enumerating objects: 3, done.
Counting objects: 100% (3/3), done.
Writing objects: 100% (3/3), 241 bytes | 241.00 KiB/s, done.
Total 3 (delta 0), reused 0 (delta 0)
remote: unable to load configuration from hooks/pre-receive
To http://localhost:5050/gitrepo-new
! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'http://user@localhost:5050/gitrepo-new'

Fix this problem by introducing a configuration setting 'git_hook_interpreter'
that allow administrators to specify which Python interpreter to use.

A subsequent commit will cause its value to be filled in automatically when
generating a new ini file, but an administrator can always override it.

hooks: use /usr/bin/env only when needed

The use of /usr/bin/env is only needed for relative arguments (or to pass
variables in the environment, which we don't do). It is thus not needed in case
the Python interpreter for Git hooks is known with absolute path, as introduced
in 5e501b6ee639.

hooks: add intermediate function _get_git_hook_interpreter

The logic to determine the right interpreter for Git hooks is about to
change and possibly become more complex.
Split it off in a separate function so such changes do not require extra
code duplication and preserve the readability of the code.

model/scm: fix incorrect reporting of pull from remote git repo

In issue #327 it was reported that the pull of a remote git repo causes the
following inaccurate journal entry:
"committed via Kallithea into repository"

The problem is caused by the following flow:

1. pull_changes calls _handle_push for git repos, passing
action='push_remote'.

2. _handle_push calls _handle_rc_scm_extras, but does not pass 'action'

3. _handle_rc_scm_extras uses a default value of 'push_local' for action.

4. The action string 'push_local' will result in the mentioned journal
entry, via action_map in action_parser() (kallithea/lib/helpers.py).


To get a more accurate log entry, _handle_push should pass the action that
it got from pull_changes. In that case, the log entry will become
"pulled from remote into repository"

api: allow pulling from a custom remote

The 'pull' API call would currently pull from the configured repository
remote or the fork origin in case of a fork.

This commit allows to specify an optional 'clone_uri' parameter to the API
call that will be used during the pull.

codingstyle: trivial whitespace fixes

Reported by flake8.

repo-scan: rewrite get_filesystem_repos to use os.walk instead of stupid recursion

I think this is more readable. It is also faster. Perhaps because the more
readable implementation makes it easier to optimize.

repositories: allow to pull changes from forked repo

It should be possible to pull changes not only from remote,
but also from locally forked repositories.

tg: minimize future diff by some mocking and replacing some pylons imports with tg

No actual tg dependency yet, just a temporary hack faking tg as an alias for
pylons.

Based on work by Alessandro Molina.

model: remove BaseModel class

The only remaining purpose of this class was to provide the "sa" field,
allowing a custom SQLAlchemy session to be used for model operations.
However, this never actually worked, nor was it used anywhere.

There's always a global SQLAlchemy session associated with the current
thread; using another session for a single function call does not make
any sense (as sessions cannot be mixed), unless the code works carefully
to ensure the two sessions (and all objects loaded from them) are kept
completely separate. Suffice to say that Kallithea does no such thing,
thus there's no need to pretend to support multiple concurrent sessions.

cleanup: drop superfluous Session.add calls

Session.add should only be called on newly created database objects.

Per the Kallithea contribution guidelines:

When getting an object from the session (via Session().query or
any of the utility functions that look up objects in the database),
it's already part of the session, and should not be added again.

auth: simplify user group permission checks

In practice, Kallithea has the 'usergroup.admin' permission imply the
'usergroup.write' permission, which again implies 'usergroup.read'.

This codifies this practice by replacing the HasUserGroupPermissionAny
"perm function" with the new HasUserGroupLevel function, reducing the
risk of errors and saving quite a lot of typing.

auth: simplify repository group permission checks

In practice, Kallithea has the 'group.admin' permission imply the
'group.write' permission, which again implies 'group.read'.

This codifies this practice by replacing HasRepoGroupPermissionAny
"perm function" with the new HasRepoGroupLevel function, reducing the
risk of errors and saving quite a lot of typing.

auth: simplify repository permission checks

In practice, Kallithea has the 'repository.admin' permission imply the
'repository.write' permission, which again implies 'repository.read'.

This codifies/enforces this practice by replacing HasRepoPermissionAny
"perm function" with the new HasRepositoryLevel function, reducing the
risk of errors and saving quite a lot of typing.

db: rename UserFollowing.follows_repo_id to follows_repository_id

Since the relationship is 'follows_repository', rename the column to be
'follows_repository_id', not 'follows_repo_id'. This also makes the
Python column name match the actual database column name.

(The inconsistency dates back to early RhodeCode days.)

db: rename RepoGroup.group_parent_id to parent_group_id

Also for consistency with the existing parent_group relationship.

model: move code from __init__.py to base.py

Having too much code, in particular too much imports, inside a package's
__init__.py is a recipe for circular imports, and considered bad practice in
Python [1]

Move out everything from kallithea/model/__init__.py to a new file
kallithea/model/base.py and adapt the existing imports.

[1] http://docs.python-guide.org/en/latest/writing/structure/#packages

auth: remove HasPermissionAll and variants

First, find all calls to HasPermissionAll with only a single permission
given, and convert to equivalent calls to HasPermissionAny.

Next, observe that it's hard to envision situations requiring multiple
permissions (of the same scope: global/repo/repo group) to be satisfied.
Sufficiently hard that there are actually no such examples in the code.

Finally, considering that (should it ever be needed) HasPermissionAll
can be trivially built as a conjunction of HasPermissionAny calls (the
decorators, too) with only a small performance impact, simply remove
HasPermissionAll and related classes and functions.

repo-scan: rewrite get_filesystem_repos to use os.walk instead of stupid recursion

I think this is more readable. It is also faster. Perhaps because the more
readable implementation makes it easier to optimize.

cleanup: get rid of jn as shortcut for os.path.join

It is not used in that many places so it is better to be explicit and use os.path.join .

Discussed on https://bitbucket.org/domruf/kallithea/commits/1da05f42bca3f7042c444fda13a1ae1f6b9c300f#comment-2948124 .

Modified by Mads Kiilerich.

hooks: if available, use sys.executable as executable for git hooks

Windows doesn't necessarily have "python2" available in $PATH, but we still
want to make sure we don't end up invoking a python3. Using the absolute path
seems more safe.

scm: simplify get_repos and get rid of RepoList overloads

The RepoList variants might have made sense before any repo info was stored in
the database. Now it has no value and doesn't help at all.

windows: node paths should always use posixpath - avoid \ in repo internal paths on windows

Using \ on Windows would result in a wrong node path which could not be checked
out with Git on Windows.

cache: when invalidating a cache, always just delete all 'live cache' records instead of marking them inactive

Keep it simple. Adding the record again might be slightly more expensive than
just updating the active flag but instead we get get a simpler model and
automatic cleanup without using the cache-keys paster command.

remove vestiges of Python 2.5 support

We only support Python 2.6 and 2.7; hence we do not need to import
with-statement support from __future__.

git: add option for forcing overwrite of Git hooks when remapping and rescanning the repositories. (Issue #153)

README file was updated to include some information on updating Git hooks after migrating to Kallithea.

cleanup: consistently use 'except ... as ...:'

Use the Python 2.6+ syntax instead of the old confusing 'except ..., ...'
syntax.

cleanup: check for None object identity in cases where that is what the 'contract' says

Avoid applying bool() on complex objects - it might return something unexpected
such as the key (which might be 0 and thus be false). Checking for None is more
safe and faster.

scm: introduce AvailableRepoGroupChoices

Extract reusable code to reduce code duplication.

Note: groups_choices from the db module is only used from this function and
should perhaps also be extracted ... but it is also closely related to other
stuff that is (mis)placed in the db module so perhaps not ...

Correct licensing information in individual files.

The top-level license file is now LICENSE.md.

Also, in various places where there should have been joint copyright holders
listed, a single copyright holder was listed. It does not appear easy to add
a link to a large list of copyright holders in these places, so it simply
refers to the fact that various authors hold copyright.

In future, if an easy method is discovered to link to a list from those
places, we should do so.

Finally, text is added to LICENSE.md to point to where the full list of
copyright holders is, and that Kallithea as a whole is GPLv3'd.

Second step in two-part process to rename directories.
This is the actual directory rename.