git: fix whitespace in previous commit

scripts/whitespacecleanup.sh will remove tabs introduced in f0fbb0fe4462 ...
but that is fine; it was just an example.

git: update check for invalid URL characters to work with Python versions that include an attempt at fixing the very same problem

With changes like
https://github.com/python/cpython/commit/76cd81d60310d65d01f9d7b48a8985d8ab89c8b4
making it to Python 3.10 and being backported to previous Python versions, the
approach in a8a51a3bdb61 no longer works when combined with
urllib.parse.urlparse in d2f59de17bef: path will never contain the invalid
characters.

To catch this case anyway, add a new check to verify that the parsed URL can
roundtrip back to the original representation with urllib.parse.urlunparse .

The actual exception might vary, but one of them should always fire.

There is a risk that the new check will reject some URLs that somehow isn't
normalized. No such cases have been found yet.

vcs: drop repo contact ... and the odd uses of it

A half-baked and essentially unused hgweb feature.

The reference in get_commits_stats doesn't seem reachable in any relevant use
cases.

vcs: use urlparse in _check_url

This makes URL checking more standard and more explicit.

vcs: slight clean up of _check_url

Make static void function static and without return value.

git: disallow odd characters in path of git:// URLs

Mitigate https://blog.harold.kim/2020/11/invalid-url-on-git-clone-leading-to-ssrf
until the problem is fixed properly in Git.

The checks might be more strict than necessary but should not have any impact
on real world use cases.

Thanks to stypr of Flatt Security for raising this.

git: make sure _check_url only accept the protocols accepted by is_valid_repo_uri

Avoid unnecessary flexibility, ambiguity, and complexity.

The file protocol was never used. But when cloning existing managed repos,
is_valid_repo_url would be skipped and _check_url would be called with absolute
paths.

imports: try to use global imports unless it is a layering violation

To minimize the impact, do imports that violate the layering at runtime instead
of at import time.

vcs: move changeset diff from controller to vcs

Remove unfortunate model dependency on controller ... and put the VCS details
where they belong, with less VCS specific knowledge in the controllers.

vcs: extract get_urllib_request_handlers out of {hg,git}/repository.py

Avoid some duplication between hg and git backends.
A subsequent commit will need bugfixes in this area.

vcs: fix validation of Git https URLs (Issue #365)

Fix py3 migration bug.

scripts: drop isort --wrap-length 160 - it is broken with py3 and not really necessary

Under Python 3, isort 4.3.21 failed with
https://github.com/timothycrosley/isort/issues/741 on
kallithea/controllers/api/api.py :

Traceback (most recent call last):
File "data/env/bin/isort", line 10, in <module>
sys.exit(main())
File ".../env/lib64/python3.7/site-packages/isort/main.py", line 379, in main
for sort_attempt in attempt_iterator:
File ".../env/lib64/python3.7/site-packages/isort/main.py", line 377, in <genexpr>
attempt_iterator = (sort_imports(file_name, **arguments) for file_name in file_names)
File ".../env/lib64/python3.7/site-packages/isort/main.py", line 88, in sort_imports
result = SortImports(file_name, **arguments)
File ".../env/lib64/python3.7/site-packages/isort/isort.py", line 207, in __init__
self._add_formatted_imports()
File ".../env/lib64/python3.7/site-packages/isort/isort.py", line 606, in _add_formatted_imports
self._add_from_imports(from_modules, section, section_output, sort_ignore_case)
File ".../env/lib64/python3.7/site-packages/isort/isort.py", line 526, in _add_from_imports
import_statement = self._multi_line_reformat(import_start, from_import_section, comments)
File ".../env/lib64/python3.7/site-packages/isort/isort.py", line 552, in _multi_line_reformat
dynamic_indent, indent, line_length, comments)
File ".../env/lib64/python3.7/site-packages/isort/isort.py", line 705, in _output_grid
if len(next_statement.split(self.line_separator)[-1]) + 1 > line_length:
TypeError: '>' not supported between instances of 'int' and 'str'

vcs: fix repo creation with a remote clone uri under Python 3

When trying to add a new repository based on a remote clone, it fails in
Python 3 as follows, for git:

ERROR kallithea.model.validators:validators.py:413 URL validation failed
Traceback (most recent call last):
File ".../kallithea/model/validators.py", line 411, in _validate_python
is_valid_repo_uri(repo_type, url, make_ui())
File ".../kallithea/lib/utils.py", line 250, in is_valid_repo_uri
GitRepository._check_url(url)
File ".../kallithea/lib/vcs/backends/git/repository.py", line 174, in _check_url
if not test_uri.endswith('info/refs'):
TypeError: endswith first arg must be bytes or a tuple of bytes, not str

and for hg, the other way around:

ERROR kallithea.model.validators:validators.py:413 URL validation failed
Traceback (most recent call last):
File ".../kallithea/model/validators.py", line 411, in _validate_python
is_valid_repo_uri(repo_type, url, make_ui())
File ".../kallithea/lib/utils.py", line 233, in is_valid_repo_uri
MercurialRepository._check_url(url, ui)
File ".../kallithea/lib/vcs/backends/hg/repository.py", line 297, in _check_url
if os.path.isdir(url) or url.startswith(b'file:'):
TypeError: startswith first arg must be str or a tuple of str, not bytes

In both cases, the code seems to actually expect a bytes url.

cleanup: minimize use of lambda expressions - we have 'def' for that purpose

Fix some flake8 warnings "E731 do not assign a lambda expression, use a def".

py3: remove safe_unicode in places where it no longer is needed because all strings (except bytes) already *are* unicode strings

(The remaining safe_unicode calls are still needed and can't just be removed,
generally because we in these cases still have to convert from bytes to unicode
strings.)

py3: migrate from urllib2 to urllib

Based on 2to3 urllib, but with unnecessary imports removed.

py3: trivial renaming of .iteritems() to .items()

A bit like "2to3 -f dict", but we don't want list().

vcs: drop unused get_hook_location (Issue #353)

Also, the hg implementation was odd - the '.hgrc' directory name looks very
similar to 'hgrc' but could be anything, such as 'hooks'.

vcs: tweak how revisions and repo names are shown in error messages

Decode bytes to str, and show repo name instead of repr or full server file
system path. In some places, it will only report the "basename" of the
repository, without any "group names" that also would be nice to have. The easy
alternative would be to show the full file system path ... but it would be
unfortunate to leak absolute server side paths to end users.

vcs: minor cleanup

Make it more friendly to humans and py3.

vcs: drop the superfluous and leaky hgcompat "layer"

Explicit is better. And gives less pyflakes noise.

py3: add b'' annotations in some places where they will be needed later

Mostly entirely trivial adding of b prefix that is a ignored for py2 ... and
also a bit of related trivial reformatting/refactorings.

vcs: refactor and simplify _get_revision

This will be more thorough checking that revisions actually exist, and will in
some cases give better error messages.

But mainly, prepare for py3 where this was easier to make work correctly.

vcs: refactor run_git_command to just return stdout as unicode string

This will be convenient for py3.

vcs: change get_diff to use _run_git_command directly

This will allow run_git_command to return strings while get_diff still return
raw bytes.

vcs: fix get_changesets to .reverse() list of revision hashes in place instead of using reversed()

This is slightly more performant, and will also make CollectionGenerator work
with reverse=True. Before, CollectionGenerator was used on get_changesets, but
never with reverse option. Trying to use reverse, it would fail when applying
len():

TypeError: object of type 'listreverseiterator' has no len()

vcs: close SubprocessIOChunker after use - it has a close method, so let's make sure we use it

It might not be strictly necessary, but it might free some resources earlier.

api: add max_revisions option to get_changesets

The returning JSON can become pretty big and hard to parse.
Therefore add an option that allows a client to request the changesets in
smaller chuncks.

vcs: sanitize diff context values (Issue #306)

- Updated Git repository implementation to ensure context falls within
0 to 2**31-1 range (inclusive) when fetching a diff.
- Added tests for Git repositories for checking passed-in negative and
overflowing contexts (for the --unified option).
- Updated Mercurial repository implementation to ensure context is not
negative when fetching a diff.
- Added tests for Mercurial repositories for checking passed-in
negative context (for the --unified option).

codingstyle: replace "not ... in ..." with "... not in ..."

Reported by flake8.

codingstyle: trivial whitespace fixes

Reported by flake8.

git: improve performance working with git changesets

GitRepository._repo instantiates a new dulwich.repo.Repo on every usage,
rather than once at initialization time of GitRepository. As this involves a
lot of filesystem access, this is a costly operation.

Instead, let GitRepository.__init__ instantiate a dulwich.repo.Repo once,
and let GitRepository._repo just return it.

This improves performance significantly.
On test_graphmod_git, performance improves from 6.29 seconds median to 3.06
seconds median.

[Thomas De Schampheleire: extend improvement to _all_ usage of
GitRepository._repo instead of only some. To limit the delta, retain the
_repo property but simply return self.repo.]

git: make sure repository path and name are Unicode strings

This is needed for supporting latest Dulwich version, but it also works with
out supported version.

cleanup: remove unused imports

Found with pyflakes.

cleanup: consistently use 'except ... as ...:'

Use the Python 2.6+ syntax instead of the old confusing 'except ..., ...'
syntax.

git: run external commands as list of strings so we really get correct quoting (Issue #135)

a6dfd14d4b20 from
https://bitbucket.org/conservancy/kallithea/pull-request/17/add-quotes-to-repo-urls-for-git-backend
fixed that issue but did not make it "safe". The vcs git backend still used
command strings but tried to quote them correctly ... but that approach is
almost impossible to get right.

Instead, pass a string list all the way to the subprocess module and let it do
the quoting. This also makes some of the code more simple.

vcs: added proper windows quoting to git commands (Issue #135)

Git repositories could not be forked on windows.

git: avoid OSError when encountering invalid lock file

This avoids crashing when repo scanning makes dulwich look at its own pip
install directory.

git: preserve line endings when calling out to git

\r line endings were replaced with \n, and some diffs would thus fail to parse.
That would be the cases similar to what was addressed in fbb992c719aa.

The leading text would be ignored anyway in diffs.py so there is no need to
strip it here. The stripping do also not seem very reliable. But for now we
will keep it.

cleanup: avoid some 'except Exception' catching - catch specific exceptions or log it and show what happened

This has a risk of introducing regressions ... but we want to get rid of all
exception muting and make the whole system less fragile and easier to debug.

Escape shell command parts for Git backend

Python 2.x uses pipes.quote and Python 3.3+ uses shlex.quote in
GitRepository._run_git_command()

Second step in two-part process to rename directories.
This is the actual directory rename.