diffs: add test coverage of inline diff with other markup

Add trailing whitespaces ... but also whitespace before changes (which might
seem trailing but isn't).

diff: handle some escaped characters in Git diffs

There are some odd characters (like \r and \n) that the Kallithea UI doesn't
allow in filenames in repos. Kallithea (through the routes module) will fail to
generate URLs when browsing Files. That is a known limitation with minimal
real-world impact, non-trivial to work around or fix.

There are very few relevant use cases for tracking files with odd filenames. \t
is valid but is hard to render in a meaningful way in the UI. And ASCII
characters like \ and " are not usable on Windows and should just be avoided.

Kallithea would parse Git diffs with odd characers incorrectly or fail, even
before hitting the known limitation. With this change, Kallithea will parse
diffs with odd filenames correctly (and then hit the limitation).

Git will quote odd filenames and escape the odd characters when emitting diffs.
(Mercurial does by design not allow \r and \n , and Mercurial will thus never
have to quote file names in diffs.)

Quotes are already handled (and ignored). With this change, Kallithea will
handle \ unescaping of \\ and \", the usual letters like \r and \n and \t, and
octal numbers like \033 (for ESC) .

Filenames with \ and " will work perfectly (when not on Windows).

Filenames with \t and ESC will work fine, but without helpful display in the
UI.

Filenames with \r and \n will still make the UI fail when trying to generate
URLs.

Thanks to stypr of Flatt Security for raising this.

diff: improved handling of Git diffs with " quoting

Kallithea would intentionally and explicitly fail with an ugly exception when
trying to parse Git diffs with quoted filenames.

Improve this by parsing quotes ... and ignore them, as long as they are
matching. The content inside the quotes might be \-escaped ... but that could
potentially also be the case without quoting. We will fix that later.

Adding some test cases that before would have failed to parse and raised an
exception.

Thanks to stypr of Flatt Security for raising this.

lib/diffs: make sure that trailing tabs are indicated

Backport of 53142fd5af4e - the problem also existed before f79c40759d6f.

Eat trailing newline after trailing tab as we already do for trailing spaces.

The change diff mainly shows re group numbers shifting.

lib/diffs: make sure that trailing tabs are indicated

Between the initial submission and final version of commit f79c40759d6f,
changes were made that turn out to be incorrect. The changes assume that the
later match on trailing tabs will 'win' from the plain 'tab' match. However,
Python 're' documentation says:

As the target string is scanned, REs separated by '|' are tried from
left to right. When one pattern completely matches, that branch is
accepted. This means that once A matches, B will not be tested further,
even if it would produce a longer overall match. In other words, the '|'
operator is never greedy.
https://docs.python.org/3.8/library/re.html

As a result, a trailing tab is seen as a plain tab and not highlighted in a
special way.

Unify the tab handling to make it unambiguous how they should be parsed.

The change diff mainly shows re group numbers shifting.

diff: refactor to avoid complexity of passing unnecessary trailing newline around

_escaper tried to handle these optional newlines ... but did not always preserve
them. Keep it simple.

In normal operations, all diff lines end with \n . We only saw un-terminated
lines in a couple of old test cases.

diff: drop internal inconsistent use of '...' as context line numbers

as_html() did in some cases not check for '...' and would thus emit unnecessary
html.

cleanup: get rid of most "import *"

Apply script generated with the following hack:
(
hg loc '*.py'|xargs pyflakes-2 | sed -rn "s/([^:]*):.*'(.*)' may be undefined, or defined from star imports.*/sed -ri 's,\\\\<\2\\\\>([^=]|$),XXXX.\2\\\\1,g' \1/gp" | sort -u
hg loc '*.py'|xargs pyflakes-2 | sed -rn "s/([^:]*):.* undefined name '(.*)'$/sed -ri 's,\\\\<\2\\\\>([^=]|$),XXXX.\2\\\\1,g' \1/gp" | sort -u
hg loc '*.py'|xargs pyflakes-2 | sed -rn "s/([^:]*):.*'(from .*)\.([^.]*) import \*' used.*/sed -ri 's,\\\\<XXXX\\\\.,\3.,g' \1/gp" | sort -u
hg loc '*.py'|xargs pyflakes-2 | sed -rn "s/([^:]*):.*'(from .*)\.([^.]*) import \*' used.*/sed -ri 's,\2\\\\.\3 .*,\2 import \3,g' \1/gp" | sort -u
) | grep -v kallithea/bin/kallithea_cli_ishell.py > fix2.sh

tests: avoid unicode u'' repr in test_diff_markup output

It will go away with py3 anyway.

diff: use fontello icon-diff-* names in generated html

This way we need less style code ... but generate slightly bigger html.

Also move the style code to kallithea-diff.less and remove the now obsolete
comment.

diffs: inline prepare() into __init__ and make the result available as .parsed

Make it more clear what the DiffProcessor is: Something that works on a raw
diff as input, mainly compute when initialized, and returns an object where the
result is available in different ways.

tests: add as little code as possible in __init__.py

kallithea/tests/__init__.py contained quite a lot of code, including the test
base class TestController. This in itself may be considered bad practice.

Specifically, this poses a problem when using pytest 3.0+, in which asserts
in some files are not automatically rewritten to give improved assert
output. That problem can be fixed by explicitly registering such files for
assertion rewriting, but that register call should be executed _before_ said
files are imported. I.e. if the register call is in
kallithea/tests/__init__.py, assert calls in __init__.py itself can not be
rewritten.

Since the TestController base class does effectively contain asserts, and we
do not want to execute the register call from somewhere outside the
kallithea/tests directory, we need to move the TestController class to
another file (kallithea/tests/base.py) so we can have a register call in
__init__.py before loading base.py.

While not strictly necessary to fix the mentioned pytest problem, we take
the opportunity to fully clean __init__.py and move everything to
the new kallithea/tests/base.py. While doing so, unnecessary imports are
removed, and imports are ordered alphabetically. Explicit imports of symbols
from modules that were already imported as a whole, are removed in favor of
fully qualifying the references (e.g. tempfile._RandomNameSequence).

diff: only highlight of difference between del and add line for one-liners

Comparing a single del line with the first of several add lines would often be
quite arbitrary and misleading.

Note: the non-gitdiff code paths in diffs.py are incomplete and unused and
unusable.

pytest migration: rename TestControllerPytest back to TestController

The name TestControllerPytest was introduced to allow a temporary situation
where nose/unittest and pytest-based tests could coexist.
This situation is now over, so the base test class can be renamed again.

pytest migration: models: switch to standard assert statements

Use unittest2pytest to replace unittest-style assert statements (e.g.
assertEqual) with standard Python assert statements to benefit from pytest's
improved reporting on assert failures.

The conversion by unittest2pytest was correct except for:
- 'import pytest' is not automatically added when needed
- line wrapping in string formatting caused a syntax error in the
transformed code. Reported upstream at
https://github.com/pytest-dev/unittest2pytest/issues/3 .
- in assertRaises with a lambda, the lambda needs to be removed

pytest migration: model: convert all tests to TestControllerPytest

The model tests were based on BaseTestCase which does not exist currently
for pytest-style tests. Nevertheless, there seems to be no advantage of
directly subclassing BaseTestCase over subclassing TestControllerPytest.
Thus, keep things simple and use TestControllerPytest.

git: fix parsing of git mode change + rename diffs

Assume mode lines come first, like for hg diffs.

remove vestiges of Python 2.5 support

We only support Python 2.6 and 2.7; hence we do not need to import
with-statement support from __future__.

diffs: avoid conflicts between inline diff mechanism and special markup

It would sometimes emit markup like
<pre><ins><u</ins> <ins>class</ins><ins>=</ins><ins>"cr</ins><ins>"></u></ins></pre>
instead of
<pre><ins><u class="cr"></u></ins></pre>

diff: handle GIT delta binary patches

It seems like they only occur rarely; recent git versions do apparently not create them.

Using a test case from https://secure.phabricator.com/T6157 .

diff: don't split lines on bare CR as python splitlines do

The line count from the @@ lines and the actual count would get out of sync and
an incomplete diff would be shown.

This is slower than native splitlines ... but that is what it takes to handle
it correctly. (Except that we perhaps could use .split('\n') and patch the
result ...)

diff: fix diff of renamed files with whitespace in their names

Before, header parsing would fail.

diff: don't group diffs based on the kind of change

We do no longer show the content of deleted files so there is no longer any
reason to put them at the end. Keeping all changes sorted by file name makes it
easier to review.

Second step in two-part process to rename directories.
This is the actual directory rename.