hg: use mercurial.node.nullid directly - needed for 5.9

mercurial.commands.nullid was removed in d55b71393907.

vcs: drop repo contact ... and the odd uses of it

A half-baked and essentially unused hgweb feature.

The reference in get_commits_stats doesn't seem reachable in any relevant use
cases.

hg: make sure _check_url only accept the protocols accepted by is_valid_repo_uri

Avoid unnecessary flexibility, ambiguity, and complexity.

This is similar to what 3ea3d3a2b3e3 did for Git.

vcs: use urlparse in _check_url

This makes URL checking more standard and more explicit.

vcs: always pass str to _check_url

hg would sometimes pass bytes.

vcs: slight clean up of _check_url

Make static void function static and without return value.

imports: try to use global imports unless it is a layering violation

To minimize the impact, do imports that violate the layering at runtime instead
of at import time.

vcs: move changeset diff from controller to vcs

Remove unfortunate model dependency on controller ... and put the VCS details
where they belong, with less VCS specific knowledge in the controllers.

vcs: fix cloning remote repository with HTTP authentication (Issue #379)

Using a remote clone URI of
http://user:pass@host/...
triggered an exception:

...
E File ".../kallithea/lib/utils.py", line 256, in is_valid_repo_uri
E GitRepository._check_url(url)
E File ".../kallithea/lib/vcs/backends/git/repository.py", line 183, in _check_url
E passmgr.add_password(*authinfo)
E File "/usr/lib/python3.7/urllib/request.py", line 848, in add_password
E self.reduce_uri(u, default_port) for u in uri)
E File "/usr/lib/python3.7/urllib/request.py", line 848, in <genexpr>
E self.reduce_uri(u, default_port) for u in uri)
E File "/usr/lib/python3.7/urllib/request.py", line 875, in reduce_uri
E host, port = splitport(authority)
E File "/usr/lib/python3.7/urllib/parse.py", line 1022, in splitport
E match = _portprog.fullmatch(host)
E TypeError: cannot use a string pattern on a bytes-like object


The authinfo tuple is obtained via mercurial.util.url, which unfortunately
returns a tuple of bytes whereas urllib expects strings.
It seems that mercurial internally has some more hacking around urllib as
urllibcompat.py, which we don't use.

Therefore, transform the bytes into strings before passing authinfo to
urllib. As the realm can be None, we need to check it specifically otherwise
safe_str would return a string 'None'.

A basic test that catches the mentioned problem is added, even though it
does not actually test that cloning with auth info will actually work (it
only tests that it fails cleanly if the URI is not reachable).

Additionally, one use of 'test_uri' in hg/repository.py still needed to be
transformed from bytes to string. For git this was already ok.

vcs: extract get_urllib_request_handlers out of {hg,git}/repository.py

Avoid some duplication between hg and git backends.
A subsequent commit will need bugfixes in this area.

diff: fix per file display of "Raw diff" and others

With py3, the matcher must be a list of byte strings.

hg: fix crash on displaying bookmarks - add lost hex conversion when computing bookmark property

The crash was seen for example when using "Switch To" with bookmarks.

The problem was introduced with py3 conversion in e7dbe089e10d.

scripts: drop isort --wrap-length 160 - it is broken with py3 and not really necessary

Under Python 3, isort 4.3.21 failed with
https://github.com/timothycrosley/isort/issues/741 on
kallithea/controllers/api/api.py :

Traceback (most recent call last):
File "data/env/bin/isort", line 10, in <module>
sys.exit(main())
File ".../env/lib64/python3.7/site-packages/isort/main.py", line 379, in main
for sort_attempt in attempt_iterator:
File ".../env/lib64/python3.7/site-packages/isort/main.py", line 377, in <genexpr>
attempt_iterator = (sort_imports(file_name, **arguments) for file_name in file_names)
File ".../env/lib64/python3.7/site-packages/isort/main.py", line 88, in sort_imports
result = SortImports(file_name, **arguments)
File ".../env/lib64/python3.7/site-packages/isort/isort.py", line 207, in __init__
self._add_formatted_imports()
File ".../env/lib64/python3.7/site-packages/isort/isort.py", line 606, in _add_formatted_imports
self._add_from_imports(from_modules, section, section_output, sort_ignore_case)
File ".../env/lib64/python3.7/site-packages/isort/isort.py", line 526, in _add_from_imports
import_statement = self._multi_line_reformat(import_start, from_import_section, comments)
File ".../env/lib64/python3.7/site-packages/isort/isort.py", line 552, in _multi_line_reformat
dynamic_indent, indent, line_length, comments)
File ".../env/lib64/python3.7/site-packages/isort/isort.py", line 705, in _output_grid
if len(next_statement.split(self.line_separator)[-1]) + 1 > line_length:
TypeError: '>' not supported between instances of 'int' and 'str'

vcs: fix repo creation with a remote clone uri under Python 3

When trying to add a new repository based on a remote clone, it fails in
Python 3 as follows, for git:

ERROR kallithea.model.validators:validators.py:413 URL validation failed
Traceback (most recent call last):
File ".../kallithea/model/validators.py", line 411, in _validate_python
is_valid_repo_uri(repo_type, url, make_ui())
File ".../kallithea/lib/utils.py", line 250, in is_valid_repo_uri
GitRepository._check_url(url)
File ".../kallithea/lib/vcs/backends/git/repository.py", line 174, in _check_url
if not test_uri.endswith('info/refs'):
TypeError: endswith first arg must be bytes or a tuple of bytes, not str

and for hg, the other way around:

ERROR kallithea.model.validators:validators.py:413 URL validation failed
Traceback (most recent call last):
File ".../kallithea/model/validators.py", line 411, in _validate_python
is_valid_repo_uri(repo_type, url, make_ui())
File ".../kallithea/lib/utils.py", line 233, in is_valid_repo_uri
MercurialRepository._check_url(url, ui)
File ".../kallithea/lib/vcs/backends/hg/repository.py", line 297, in _check_url
if os.path.isdir(url) or url.startswith(b'file:'):
TypeError: startswith first arg must be str or a tuple of str, not bytes

In both cases, the code seems to actually expect a bytes url.

py3: migrate from urllib2 to urllib

Based on 2to3 urllib, but with unnecessary imports removed.

py3: use exception .args instead of .message

Args seems slightly more fragile and *could* introduce problems for trivial use
if args is empty. But .message is gone.

vcs: fix missing import of mercurial.localrepo

(localrepo might already always be available in the mercurial namespace due to
side effects from other imports, but it is still better to do it explicit ...
and also to please pytype.)

vcs: drop unused get_hook_location (Issue #353)

Also, the hg implementation was odd - the '.hgrc' directory name looks very
similar to 'hgrc' but could be anything, such as 'hooks'.

vcs: tweak how revisions and repo names are shown in error messages

Decode bytes to str, and show repo name instead of repr or full server file
system path. In some places, it will only report the "basename" of the
repository, without any "group names" that also would be nice to have. The easy
alternative would be to show the full file system path ... but it would be
unfortunate to leak absolute server side paths to end users.

vcs: minor cleanup

Make it more friendly to humans and py3.

vcs: drop the superfluous and leaky hgcompat "layer"

Explicit is better. And gives less pyflakes noise.

py3: add b'' annotations in some places where they will be needed later

Mostly entirely trivial adding of b prefix that is a ignored for py2 ... and
also a bit of related trivial reformatting/refactorings.

vcs: make hg get_changesets compatible with py3

If start was 0, we would end up with start_pos None and getting a compare with
None which would fail in py3.

Instead, make sure start_pos is None iff start is None.

Also, make the actual check more robust.

vcs: refactor and simplify _get_revision

This will be more thorough checking that revisions actually exist, and will in
some cases give better error messages.

But mainly, prepare for py3 where this was easier to make work correctly.

hg: bump minimum version to 5.1

We will soon move to Python 3 which only will support 5.1 or later.

Remove old hacks and tech debt.

Also avoids future warning:
DeprecationWarning: inspect.getargspec() is deprecated since Python 3.0, use inspect.signature() or inspect.getfullargspec()

cleanup: minor refactorings and simplification of dict usage

Makes it more py3 compatible.

vcs: fix get_changesets to .reverse() list of revision hashes in place instead of using reversed()

This is slightly more performant, and will also make CollectionGenerator work
with reverse=True. Before, CollectionGenerator was used on get_changesets, but
never with reverse option. Trying to use reverse, it would fail when applying
len():

TypeError: object of type 'listreverseiterator' has no len()

hg: prepare for Mercurial 5.0 changing "exact" arguments

In the backward compat wrapper, we use root=None. That might seem a bit risky.
But it seems to work for the single use case we have, and the changeset dropped
it in Mercurial https://www.mercurial-scm.org/repo/hg/rev/0531dff73d0b hint
that this parameter really is unused.

hg: fix URL cloning with Mercurial 4.6 and later

In 03dfcbe52906, I didn't notice that the API also changed ... and it was not
sufficiently tested.

Now, instead of using the peer classes directly, use the instance wrapper.

There is no automated testing of this, but it was tested manually to also work
in the oldest supported Mercurial version.

hg: use localrepo.instance instead of invoking localrepository directly

The instance method has been available for very long time and does almost the
same thing.

Also, for Mercurial 4.8, the create functionality is only available on the
instance factory.

tests: fix vcs Hg helper function get_config_value to not modify global state when using a custom config file

This leak was what made test_get_user_name and test_get_user_email pass when
test_get_config_value had been run first.

tests: fix vcs Hg helper functions get_user_name and get_user_email to actually use provided config_file

Running just test_get_user_name and test_get_user_email would fail when run as:
py.test kallithea/tests/vcs/test_repository.py -k test_get_user

The config file was not used at all, and the tests would only pass when
test_get_config_value had been run first and as an unfortunate side-effect had
read the same file.

hg: be more strict about which kinds of revision lookup we want from hg

Comply with
https://www.mercurial-scm.org/repo/hg/file/4.6/mercurial/context.py#l380
to support recent Mercurial versions.

hg: support more strict branch name checking in Mercurial 4.6 - be more careful about not passing unicode branch names

TestHgApi.test_api_get_changesets_with_branch would fail.

hg: use localrepo indexing for revision number lookup instead of the lookup function

Mercurial 4.6 (0a1fb171dc1d) changed lookup and would fail if given a revision number.

vcs: Fix internal server error when trying to get diff from Mercurial for paths that include globbing patterns (Issue #308):

- Treat passed-in paths as exact matches, dissallowing any type of
globbing/regex/pattern matching.
- Added accompanying tests.

api: add max_revisions option to get_changesets

The returning JSON can become pretty big and hard to parse.
Therefore add an option that allows a client to request the changesets in
smaller chuncks.

vcs: fix get_changesets filtering on hg repo to AND the criteria instead of OR

Mercurial scmutil.revrange takes a list of filters ... and OR them.
But when for example a user uses the api and sets branch name and date, he would
expect to only get revisions from the provided branch.
So we need to use AND when filtering.

When using AND, the special handling of start_date and end_date is no longer necessary.

Also add a test to check for this use case.

vcs: sanitize diff context values (Issue #306)

- Updated Git repository implementation to ensure context falls within
0 to 2**31-1 range (inclusive) when fetching a diff.
- Added tests for Git repositories for checking passed-in negative and
overflowing contexts (for the --unified option).
- Updated Mercurial repository implementation to ensure context is not
negative when fetching a diff.
- Added tests for Mercurial repositories for checking passed-in
negative context (for the --unified option).

codingstyle: replace "not ... in ..." with "... not in ..."

Reported by flake8.

codingstyle: trivial whitespace fixes

Reported by flake8.

hg: fix except clauses order

pylint found 'Bad except clauses order (LookupError is an ancestor class of
IndexError)'. This fix will in some cases give better error messages.

cleanup: remove unused imports

Found with pyflakes.

cleanup: consistently use 'except ... as ...:'

Use the Python 2.6+ syntax instead of the old confusing 'except ..., ...'
syntax.

hg: show code function in diffs

Just like already done for git diffs, show the function containing the
change in the hunk header of a diff, if available:

@@ -19,7 +19,7 @@ int func(void)

instead of

@@ -19,7 +19,7 @@

hg: adapt for changes in 3.2 and 3.3

Mercurial 3.2:

- moved localrepo.pull to exchange.pull
- removed __getitem__ from lazysets

remotes: add support to clone from Mercurial repositories over ssh

This commit adds support to clone a remote Mercurial repository over ssh.

Interactive password authentication is not implemented, nor is support for
pbulic key authentication with passphrases; the repository should be
accessible using bare ssh key authentication.
For this reason, the ssh options -oBatchMode=yes and -oIdentitiesOnly=yes
are added to the ui.ssh setting of Mercurial.

cleanup: avoid some 'except Exception' catching - catch specific exceptions or log it and show what happened

This has a risk of introducing regressions ... but we want to get rid of all
exception muting and make the whole system less fragile and easier to debug.

hg: reimplement branch listings more efficiently

Gives a 10 x speedup - which is noticable on big slow repos where the old
implementation took several seconds.

hg: drop support for hg web.hidden

Kallithea is not hgweb and should not try to support hgweb config options.

If someone wants to hide repos, they should just remove all access to them.

(_If_ Kallithea should support some kind of more thorough repo hiding, it
should be implemented differently: The hidden repos should not be in the
database and should thus instead be excluded when scanning for repos.)

hg: make get_ref_revision('rev', '00000000000000') give the null revision

It gave tip revision and caused weird pull request diffs and lots of grief when
(accidentally) creating a pull request from the null revision.

hg: optimize get_ref_revision

Avoid expensive branch(x) iterating over whole repo just to take max.

Second step in two-part process to rename directories.
This is the actual directory rename.