diffs: let DiffProcessor inject context_lineno in chunk changes

Make it possible to use these numbers reliably, also outside as_html.

diffs: drop pointless check for "\ No newline at end of file"

Such lines would have been caught on '\' as "unknown command" by the command
check a few lines above.

diffs: drop pointless check for adding context line

The length of the match was always 5.

diffs: DiffParser should only HTML escape lines when generating HTML

Move DiffParser HTML escaping so it only is applied when generating HTML.

diff: handle some escaped characters in Git diffs

There are some odd characters (like \r and \n) that the Kallithea UI doesn't
allow in filenames in repos. Kallithea (through the routes module) will fail to
generate URLs when browsing Files. That is a known limitation with minimal
real-world impact, non-trivial to work around or fix.

There are very few relevant use cases for tracking files with odd filenames. \t
is valid but is hard to render in a meaningful way in the UI. And ASCII
characters like \ and " are not usable on Windows and should just be avoided.

Kallithea would parse Git diffs with odd characers incorrectly or fail, even
before hitting the known limitation. With this change, Kallithea will parse
diffs with odd filenames correctly (and then hit the limitation).

Git will quote odd filenames and escape the odd characters when emitting diffs.
(Mercurial does by design not allow \r and \n , and Mercurial will thus never
have to quote file names in diffs.)

Quotes are already handled (and ignored). With this change, Kallithea will
handle \ unescaping of \\ and \", the usual letters like \r and \n and \t, and
octal numbers like \033 (for ESC) .

Filenames with \ and " will work perfectly (when not on Windows).

Filenames with \t and ESC will work fine, but without helpful display in the
UI.

Filenames with \r and \n will still make the UI fail when trying to generate
URLs.

Thanks to stypr of Flatt Security for raising this.

diff: improved handling of Git diffs with " quoting

Kallithea would intentionally and explicitly fail with an ugly exception when
trying to parse Git diffs with quoted filenames.

Improve this by parsing quotes ... and ignore them, as long as they are
matching. The content inside the quotes might be \-escaped ... but that could
potentially also be the case without quoting. We will fix that later.

Adding some test cases that before would have failed to parse and raised an
exception.

Thanks to stypr of Flatt Security for raising this.

lib/diffs: make sure that trailing tabs are indicated

Backport of 53142fd5af4e - the problem also existed before f79c40759d6f.

Eat trailing newline after trailing tab as we already do for trailing spaces.

The change diff mainly shows re group numbers shifting.

lib: move webhelpers2 and friends to webutils

Gives less of the unfortunate use of helpers - especially in low level libs.

diffs: remove unused argument enable_comments and class no-comment

enable_comments was only used to set/not-set the 'no-comment' CSS class.
While this class was emitted, no CSS rule nor any JavaScript logic was
actually using it. Last real usage of that class was removed with commit
e87baa8f1c5bd2488aefc23b95c0db3a04bc8431.

Cleanup the code by not emitting 'no-comment' and remove the
'enable_comments' flag.

lib/diffs: make sure that trailing tabs are indicated

Between the initial submission and final version of commit f79c40759d6f,
changes were made that turn out to be incorrect. The changes assume that the
later match on trailing tabs will 'win' from the plain 'tab' match. However,
Python 're' documentation says:

As the target string is scanned, REs separated by '|' are tried from
left to right. When one pattern completely matches, that branch is
accepted. This means that once A matches, B will not be tested further,
even if it would produce a longer overall match. In other words, the '|'
operator is never greedy.
https://docs.python.org/3.8/library/re.html

As a result, a trailing tab is seen as a plain tab and not highlighted in a
special way.

Unify the tab handling to make it unambiguous how they should be parsed.

The change diff mainly shows re group numbers shifting.

diff: refactor to avoid complexity of passing unnecessary trailing newline around

_escaper tried to handle these optional newlines ... but did not always preserve
them. Keep it simple.

In normal operations, all diff lines end with \n . We only saw un-terminated
lines in a couple of old test cases.

diffs: add doctests for _escaper, also showing incorrect behavior

Add some doctests for the _escaper function.
Note: some tests now show incorrect behavior that will be fixed soon.

diff: allow commenting on meta lines (Issue #329)

There is no good reason it only should be possible to comment on content lines.
Other lines might not have an obvious locator, but we can live with that as
long as each comment only apply in one place.

With this, we actually want the comment bubble on all lines with bubble markup,
so we can loosen the css selector.

diff: refactor as_html with separate code paths for diff lines with and without line number

Just inlining and factoring and moving things around.

diff: drop internal inconsistent use of '...' as context line numbers

as_html() did in some cases not check for '...' and would thus emit unnecessary
html.

lib/diffs: mark trailing tabs similar to trailing spaces

So far, spaces have not been marked up, but trailing spaces are followed by
'<i></i>' (newline symbol). Tabs have been marked up as '<u>\t</u>' (underlined
with icon), but trailing tabs didn't get the same "trailing whitespace" markup
as trailing spaces got.

Fix this unfairness by handling trailing tabs explicitly as '<u>\t</u><i></i>'
so they get both kinds of markup.

py3: trivial renaming of .iteritems() to .items()

A bit like "2to3 -f dict", but we don't want list().

py3: use global next() function instead of .next() method

From 2to3 -f next.

cleanup: minor correctness fixes

"Trivial" potential problems spotted with pytype.

py3: add b'' annotations in some places where they will be needed later

Mostly entirely trivial adding of b prefix that is a ignored for py2 ... and
also a bit of related trivial reformatting/refactorings.

vcs: always return bytes from node.content

We will rather have the unicode conversions explicit.

Note: Py3 bytes doesn't have .startswith - replace that with a regexp.

py3: use comprehensions and generators instead of filters - it is more explicit, and sometimes shorter

From 2to3 -f filter.

css: use pseudo-content trick to prevent diff line numbers from being pasted to text

When copy-pasting a diff from Chrome to a text editor, line numbers (on
separate lines) would be pasted as well. Even though 'user-select: none'
prevents text from being visually selected, in Chrome, the text still gets
copied to the clipboard when the user for example presses ctrl-c. (It worked in
Firefox.)

Instead, don't put the line numbers directly in the DOM, but put them in a data
attribute and render them as :before. That will give the same rendering as
before but prevent it from being copied.

(Firefox will however still add empty lines - that is how <pre> is hardcoded to
be rendered when pasting to text.)

diff: use fontello icon-diff-* names in generated html

This way we need less style code ... but generate slightly bigger html.

Also move the style code to kallithea-diff.less and remove the now obsolete
comment.

vcs: catch MemoryErrors when calling Git diff

Binary diffs can make the diffs VERY big and cause MemoryError exceptions.

Before giving MemoryError, the system might start swapping, any process might
fail when allocating memory, random processes might get killed, and our process
might fail in other places. The proper fix would be to avoid the problem by not
trying to process more data than we can handle - for example by not processing
more than a certain amount of Git output.

Before, memory errors were shown to the user as a 500 Internal Server Error
page.

Now, as long as we have no better/safer way get the diff, catch the MemoryError
and show the page with a flash error message and no diff.

The error handling is placed in the diffs module to avoid leaking flash
messages into the vcs lib.

diffs: wrap vcs repo get_diff

Refactor to get a single place for diff error handling outside the vcs lib.

diffs: move _escaper to a pure function

It used to count diff size - now it is a clean function.

diffs: move _highlight_inline_diff to a pure function

Make it clear that it doesn't modify any state.

diffs: move as_html and _safe_id from method to a pure function - avoid calling the method as function

as_html was sometimes used in a way where we actually don't want to use the
whole DiffProcessor - we just created a dummy instance and passed custom input
as parameter to the instance method.

Instead, make it the function we apparently want.

Make it clear that as_html not just returns a "diff" but that it is a html diff.

diffs: do check of cut_off_limit on the raw diff before any parsing and allocation

This redefines the exact meaning of cut_off_limit but it will still be
approximately the same thing. It makes it more a limitation of what amount of
work should be done, more than how much html should be outputted.

It could make sense to push the limit further back to vcs to also prevent
computing or allocating memory for huge diffs.

diffs: drop the DiffLimitExceeded container - just make it a flag available as property

Keep it simple.

diffs: inline prepare() into __init__ and make the result available as .parsed

Make it more clear what the DiffProcessor is: Something that works on a raw
diff as input, mainly compute when initialized, and returns an object where the
result is available in different ways.

diffs: avoid extra copy of diff when trying to create a lazy diff line iterator

Make the generator actually be lazy and avoid creating an extra full copy of
all the lines in a file diff at once.

And consistently call it diff_lines.

diffs: avoid extra copy of diff when splitting into files

Instead of allocating memory for an extra copy of the whole raw diff, just use
buffer() to give read-only string views of the relevant sections of the big
diff string given as input.

diffs: cleanup of variable naming around cut_off_limit

A brief summary of this area:

The base controller sets self.cut_off_limit from config and is used for diffs,
unless controllers are given a fulldiff query parameter. In a few cases, these
are passed to templates as c.cut_off_limit or c.fulldiff . Also, if the diff
function returns a LimitedDiffContainer, c.limited_diff is set so the UI can
report the data set is partial.

codingstyle: trivial whitespace fixes

Reported by flake8.

tg: minimize future diff by some mocking and replacing some pylons imports with tg

No actual tg dependency yet, just a temporary hack faking tg as an alias for
pylons.

Based on work by Alessandro Molina.

diff: don't make '...' a broken link - it doesn't do anything so don't make it a link

It would be nice if it could expand it and increase the context size like
bitbucket does ... but that would be a bigger change.

diff: correct handling of links to old filename in renames

There were links to the file at the parent revision ... but if the file had
been renamed, it used the wrong name.

diff: show correct operation for file diffs instead of '???'

Fix issue seen on the url $repo/diff/$filename?diff2=hash2&diff1=hash2 .

Drop returning unused size from diffs.wrapped_diff and return the operation
instead.

diff: only highlight of difference between del and add line for one-liners

Comparing a single del line with the first of several add lines would often be
quite arbitrary and misleading.

Note: the non-gitdiff code paths in diffs.py are incomplete and unused and
unusable.

git: fix parsing of git mode change + rename diffs

Assume mode lines come first, like for hg diffs.

diffs: avoid conflicts between inline diff mechanism and special markup

It would sometimes emit markup like
<pre><ins><u</ins> <ins>class</ins><ins>=</ins><ins>"cr</ins><ins>"></u></ins></pre>
instead of
<pre><ins><u class="cr"></u></ins></pre>

diff view: improve appearance of line numbers

* display the ellipsis just once and centered
instead of showing in twice in both - and + columns
* expand column width as needed to accomodate longer numbers
(previously, the column width was fixed)
* enable box-sizing: border-box for the number links to
make it easier to set paddings

diff: handle GIT delta binary patches

It seems like they only occur rarely; recent git versions do apparently not create them.

Using a test case from https://secure.phabricator.com/T6157 .

diff: don't split lines on bare CR as python splitlines do

The line count from the @@ lines and the actual count would get out of sync and
an incomplete diff would be shown.

This is slower than native splitlines ... but that is what it takes to handle
it correctly. (Except that we perhaps could use .split('\n') and patch the
result ...)

diff: eat trailing tabs in ---/+++ lines of diffs

There is usually no date in the diffs we parse but there might still be the tab
that was used as separator between filename and date.

The a_file and b_file entries are usually not used ... but nice to have them
right.

diff: fix diff of renamed files with whitespace in their names

Before, header parsing would fail.

diff: preserve leading space on empty lines - don't match in re if we don't plan to replace

Fix invisible bug from 7c094db329b3, exposed by future exact parsing of diffs.

diff: don't group diffs based on the kind of change

We do no longer show the content of deleted files so there is no longer any
reason to put them at the end. Keeping all changes sorted by file name makes it
easier to review.

diff: show whitespace

It could be considered to use the VisibleWhitespaceFilter instead ...

Correct licensing information in individual files.

The top-level license file is now LICENSE.md.

Also, in various places where there should have been joint copyright holders
listed, a single copyright holder was listed. It does not appear easy to add
a link to a large list of copyright holders in these places, so it simply
refers to the fact that various authors hold copyright.

In future, if an easy method is discovered to link to a list from those
places, we should do so.

Finally, text is added to LICENSE.md to point to where the full list of
copyright holders is, and that Kallithea as a whole is GPLv3'd.

Second step in two-part process to rename directories.
This is the actual directory rename.