tests: skip reading Git system and global configuration in test_vcs_operations

The parent changeset reduced the dependency on global configuration and made it
possible to run tests without any global git configuration. But it is still
unfortunate to even look at the global configuration when running tests.

Global configuration is already disabled for Mercurial by setting HGRCPATH.

Now do something similar for Git. According to the git man page,
GIT_CONFIG_GLOBAL and GIT_CONFIG_SYSTEM set to /dev/null will make Git skip
reading the configuration files on all platforms.

Note that the GIT_CONFIG variables were introduced in Git 2.32.0, so this will
not work with all the Git versions supported by Kallithea.

tests: stabilize Git committer in test_vcs_operations

Git tries to find out name and email in this order:

1. The author can be set e.g. via the `--author` option of `git commit`.
2. If set, the environment variables GIT_AUTHOR_NAME, GIT_AUTHOR_EMAIL,
GIT_COMMITTER_NAME and GIT_COMMITTER_EMAIL are taken.
3. If set, various (global) config files are considered.
4. Unless disabled by the user.useconfigonly config, the names and emails are
inferred from various system sources such as various fields from /etc/passwd,
/etc/mailname and the environment variable EMAIL.

The author can be provided on the command line (1), but that is not possible
for the committer.

It is not an option to modify Git’s configuration files, so the result of (3)
depends on the system the tests run on, which should be avoided. A follow-up
patch will try to instruct Git to not read the system Git configuration files.

(4) is also system-dependent. On some systems, (4) is disabled in the Git
configuration. If enabled, Git will try to infer the committer name from the
gecko field in /etc/passwd, but will fail if it is empty. The previous code
passed the environment variable EMAIL to provide the corresponding email
address.

By passing the names and emails via (2), we can set the author and committer
name and email uniformly and prevent Git from using the system-dependent ways
(3) and (4). This will replace the use of of EMAIL. The environment variables
were introduced in 2005, so there should be no backwards compatibility
problems.

The tests will specify --author explicitly in the cases where the actual name
matters. We just need default values that can be used for committing when we
don't care.

We set it as static defaults to:
Author: test_regular <test_regular@example.com>
Commit: test_admin <test_admin@example.com>

Based on changes and research by Manuel Jacob <me@manueljacob.de>.

hg: support Mercurial 5.8

Some 'remote:' messages changed to appear on stderr.

mercurial: support Mercurial 5.5

No release notes yet, but testing shows it working with a few changes.

A few tests need updating because error messages from remote ssh like our
sys.stderr.write('abort: %s\n' % error)
like for
remote: abort: Access to %r denied
will now appear on stderr instead of stdout - a very reasonable bug fix.

tests: fix test_vcs_operations for MariaDB/MySQL

This is apparently only needed for MariaDB/MySQL where it will fix tests that
were failing due to missing action log entries.

db: drop Repository CacheInvalidation

The benefit of this functionality is questionable. Especially in bigger setups
with multiple front-end instances all serving the same multitude of
repositories, making the hit rate very low. And the overhead of storing cache
invalidation data *in* the database is non-trivial.

We preserve a small cache in Repository SA records, but should probably just in
general know what we are doing and not ask for the same information multiple
times in each request.

py3: migrate from urllib2 to urllib

Based on 2to3 urllib, but with unnecessary imports removed.

py3: use global next() function instead of .next() method

From 2to3 -f next.

ssh: ignore trailing '/' after repo name in URLs (Issue #352)

Make SSH URLs similar to how HTTP URLs are handled in simplehg.py / simplegit.py .

We will consistently use the stripped repo name, so there should be no security
or ambiguities or reliability concerns.

cleanup: get rid of most "import *"

Apply script generated with the following hack:
(
hg loc '*.py'|xargs pyflakes-2 | sed -rn "s/([^:]*):.*'(.*)' may be undefined, or defined from star imports.*/sed -ri 's,\\\\<\2\\\\>([^=]|$),XXXX.\2\\\\1,g' \1/gp" | sort -u
hg loc '*.py'|xargs pyflakes-2 | sed -rn "s/([^:]*):.* undefined name '(.*)'$/sed -ri 's,\\\\<\2\\\\>([^=]|$),XXXX.\2\\\\1,g' \1/gp" | sort -u
hg loc '*.py'|xargs pyflakes-2 | sed -rn "s/([^:]*):.*'(from .*)\.([^.]*) import \*' used.*/sed -ri 's,\\\\<XXXX\\\\.,\3.,g' \1/gp" | sort -u
hg loc '*.py'|xargs pyflakes-2 | sed -rn "s/([^:]*):.*'(from .*)\.([^.]*) import \*' used.*/sed -ri 's,\2\\\\.\3 .*,\2 import \3,g' \1/gp" | sort -u
) | grep -v kallithea/bin/kallithea_cli_ishell.py > fix2.sh

tests: run vcs tests on ssh too

Error reporting when using SSH is different from when using HTTP. SSH doesn't
use HTTP error codes but can write anything to stdout/stderr. Some tests thus
need to have their expectations tweaked.

Inspired by work by Dominik Ruf.

tests: push with force in test_vcs_operations.py _add_files_and_push

Some tests push a fixed branch name. If such tests are run twice, it will fail
second time when the other branch head already exists.

Work around that by using force and thus replace the old head.

tests: refactor test_vcs_operations _add_files - make clone_url mandatory

It was only really used in one place.

tests: run the same vcs tests both for hg and git using http ... and soon also ssh

Parametrize vcs tests and remove duplicate code.

Refactored by Mads Kiilerich.

locking: drop the pull-to-lock / push-to-unlock functionality

The feature is not worth the maintenance cost. The locking is too coarse and
unflexible with insufficient UI and UX. The implementation is also quite
invasive in tricky areas of the code, and thus high maintenance.
Dropping this will enable other cleanup ... or at least make it easier.

git: fix push to empty repo (Issue 323)

Git would fail to log revisions when the list of heads to exclude included an
empty string (in place of the pushed ref).

To avoid that, skip the skipped revision instead of making it an empty string.
`git log --not` works fine without providing any revisions to "not".

Verify in test_push_new_repo_git that it actually logged the push.

git: introduce test_push_new_repo_git to reproduce a hook crash when pushing to empty repos

The crash will be fixed by a later changeset.

Reported on https://bitbucket.org/conservancy/kallithea/issues/323/git-hook-error-on-push-of-first-commit

remote: Traceback (most recent call last):
remote: File "hooks/post-receive", line 36, in <module>
remote: main()
remote: File "hooks/post-receive", line 32, in main
remote: sys.exit(kallithea.lib.hooks.handle_git_post_receive(repo_path, git_stdin_lines))
remote: File "kallithea/lib/hooks.py", line 453, in handle_git_post_receive
remote: git_revs += scm_repo.run_git_command(cmd)[0].splitlines()
remote: File "kallithea/lib/vcs/backends/git/repository.py", line 164, in run_git_command
remote: return self._run_git_command(cmd, **opts)
remote: File "kallithea/lib/vcs/backends/git/repository.py", line 151, in _run_git_command
remote: raise RepositoryError(tb_err)
remote: kallithea.lib.vcs.exceptions.RepositoryError: Couldn't run git command (['git', '-c', 'core.quotepath=false', 'log', 'b991c8d9ae7e66e165fc5eeb297c6843d21915e0', '--reverse', '--pretty=format:%H', '--not', '']).
remote: Original error was:Subprocess exited due to an error:
remote: fatal: ambiguous argument '': unknown revision or path not in the working tree.
remote:
To http://127.0.0.1:44433/new_git_DHAsQQ
* [new branch] master -> master

tests: fix caching issue in test_ip_restriction_git

Following test failure is observed in
TestVCSOperations.test_ip_restriction_git:

―――――――――――――― TestVCSOperations.test_ip_restriction_git ――――――――――――――――――
kallithea/tests/other/test_vcs_operations.py:584: in test_ip_restriction_git
assert re.search(r'\b403\b', stderr)
E assert None
E + where None = <function search at 0x7fb9772da578>('\\b403\\b', "Cloning into '/tmp/kallithea-test-SZhXDz/vcs_operations-krPNvZ'...\n")
E + where <function search at 0x7fb9772da578> = re.search
------------------------- Captured stdout call ----------------------------
*** CMD git clone http://test_admin:test12@127.0.0.1:45291/vcs_test_git /tmp/kallithea-test-SZhXDz/vcs_operations-krPNvZ ***
stderr: "Cloning into '/tmp/kallithea-test-SZhXDz/vcs_operations-krPNvZ'...\n"



The test is setting up IP restrictions, verifying that access is no longer
possible, then clears the restriction. There already were sleeps after
clearing the restrictions, in order for the cache to expire and have the
setting take effect.
But there was no sleep on the enabling of the IP restriction, allowing
situations where the code would still run without restriction, and thus
allow the access, failing the test.

The failure has only been observed on test_ip_restriction_git, but the
change is also made for test_ip_restriction_hg.

The existing sleeps after restriction clearing are moved up to the 'finally'
clause to make it clear to which code they belong.

tests: set EMAIL on git commit

If git user email is not set yet, git requires the EMAIL environment variable
to be set. If it wasn't set in the test environment, test_add_submodule would
fail with something like:

fatal: unable to auto-detect email address

tests: use testfork fixture for test_vcs_operations tests

This way only one fork needs to be created.

git: open submodule links in a new tab

Since you are leaving the project context (maybe even kallithea), sub module
links should open in a new tab.

git: fix links to nodes that are submodule links

Without this changeset, a link pointing to ./<submodulename> would generate a
'500 Internal Server Error'.

Instead, redirect requests to submodule paths to the target URL.

git: fix URL for submodules - make it link to the external URL

Without this changeset, the link on the files page would not point to the
submodule target. Instead it would simple point to ./<submodulename>, which when
clicked leads to a '500 Internal Server Error'.

git: let links to git submodules be real links that can open in new tabs - not just in-page navigation

Submodules should not be loaded with 'ypjax-link' because they point to a place
outside the repository or maybe even outside the kallithea instance.

tests: change vcs repo name strings to unicode

This fixes sqlalchemy warnings like
SAWarning: Unicode type received non-unicode bind param value 'vcs_test_git_forkSeWYRg'.

make-release: cleanup and fix bitrot

Try to make the build process more stable and maintainable.

So far, this leaves a problem with node_modules/bootstrap/ and
kallithea/public/css/style.css missing from the release .tar.gz .

tests: add custom hook tests

Git testing is omitted for now - errors are not reported as they are for Hg.

tests: split _add_files_and_push in two

In some cases it is usefull to push the data separately.
For example if a test tries to push multiple times.

tests: slight cleanup of test_vcs_operations _add_files_and_push

Get rid of odd "cwd = os.path.join(dest_dir)".

codingstyle: replace upper-case variable names with lower-case ones

PEP8 issues reported by flake8.

codingstyle: trivial whitespace fixes

Reported by flake8.

tests: vcs: remove influence of user's hg settings

Certain settings in the user's hgrc file can cause test failures. For
example, enabling a non-existent extension prints stderr messages like

*** failed to import extension journal: No module named journal

Tests that check stderr output for emptiness, like
test_clone_hg_repo_by_admin, then fail.

Instead, avoid all influence of the user's settings, by setting HGRCPATH and
HGPLAIN to empty.

In versions of Mercurial before 4.2, setting an empty HGRCPATH was generally
enough to get expected behavior, but since 4.2 the behavior of some commands
has changed, and a pager is now default. Setting HGPLAIN is a simple way of
neutralizing even that.

tests: put all temporary files in one top directory

Use KALLITHEA_TESTS_TMP_PATH or a create a temporary /tmp/kallithea-test-*
directory.

Some temporary files might still be placed in /tmp ... but nothing should be
left behind.

Correct licensing information in individual files.

The top-level license file is now LICENSE.md.

Also, in various places where there should have been joint copyright holders
listed, a single copyright holder was listed. It does not appear easy to add
a link to a large list of copyright holders in these places, so it simply
refers to the fact that various authors hold copyright.

In future, if an easy method is discovered to link to a list from those
places, we should do so.

Finally, text is added to LICENSE.md to point to where the full list of
copyright holders is, and that Kallithea as a whole is GPLv3'd.

Second step in two-part process to rename directories.
This is the actual directory rename.