lib: move some template filter functions from utils2 to webutils

While quite Kallithea specific, we prefer to have these functions in webutils
where they soon can be exposed when templates don't need the whole helpers
module.

lib: clean up pointless use of h.person on db.User just to get .username

After cleaning up, h.person will never be invoked with a db.User .

lib: move urlification to webutils

Less use of helpers in model.

lib: drop is_hg and is_git

It is just as simple to be explicit.

lib: move js escaping from helpers to webutils

It was no real problem it lived in helpers: it was only used from templates
anyway.

But for completeness and correctness, move all web utils to webutils.

imports: try to use global imports unless it is a layering violation

To minimize the impact, do imports that violate the layering at runtime instead
of at import time.

lib: move link_to_ref from helpers to utils2

Less reason to import the messy helpers.

lib: move webhelpers2 and friends to webutils

Gives less of the unfortunate use of helpers - especially in low level libs.

lib: move canonical_url & co to webutils

This gives less of the unfortunate use of helpers - especially in model.

lib: drop sparsely used h.not_mapped_error

It might be a good idea, but then we should use it much more consistently ...
and it should probably be done differently. Let's keep it simple and be
consistent.

routing: separate url handling from routing - move it to webutils

This is a helper method relying on the thread local tg.request. We didn't have
a good place to put it. Now we do.

This (re)moves unfortunate dependencies to the routing module (which almost is
a controller).

db: move URL_SEP constant from db to top level kallithea module

URL_SEP is used in several places - having it in the db module is too high
level.

__init__ might not be the best place for this, but it does no harm to the
dependencies graph to place constants there.

helpers: simplify age

It is just re-export utils2.age .

tests: simplify test_lib mocking of routing.url

Mock internals of the url generator to avoid import tricks.

isort: upgrade to 5.1.2

The changes to non-top-level imports seem nice.

diff: drop per file ignore-whitespace and context - it didn't work and had conceptual issue (Issue #344)

Diffs are currently generated at the low level as one big diff between two vcs
resisions, provided global values for diff context size and flag for ignoring
whitespace. All files use the same flags. There is no way to actually compute
the full diff using these use per file flags, and no simple and efficient way
to add it.

The best option is thus to drop the failed attempt at making it per file, and
just rely on the simple global flags in the URL.

The links for changing whitespace and context is sometimes shown for the whole
"page", and sometimes next to the diff for one file. For now, keep showing the
link in these places, but make sure it navigates back to the FID of the section
where the link was clicked.

The implementation is completely rewritten and moved to a more appropriate
location in helpers.

With a more clean implementation, we also consistently use the simple getters
to extract values from the URL.

lib: drop own asbool implementation and consistently use tg.support.converters as utils2.asbool

str2bool never reported error on odd input such as '' or '-1', but the tg
asbool behaviour of raising ValueError("String is not true/false: %r" % obj) in
that case seems fine.

issue: log error if still using issue_pat_XXX configuration after it was removed in d24051ce961c

Show a helpful:
found unsupported issue_prefix_pr = 'PR' - use issue_sub_pr instead
before bailing out with:
skipping incomplete issue pattern 'issue_pat_pr': '(?:PR\\s*)(\\d+)' -> 'https://kallithea-scm.org/repos/kallithea/pull-request/{id}' None

db: drop SA caching_query and FromCache, and thus sql_cache_short beaker cache

It is not a good idea to have dead ORM objects. If we want caching, we should
do it explicit.

It is unknown how much this cache helps, but we can profile and introduce
better caching of simple data where relevant.

cleanup: minimize use of lambda expressions - we have 'def' for that purpose

Fix some flake8 warnings "E731 do not assign a lambda expression, use a def".

py3: remove safe_unicode in places where it no longer is needed because all strings (except bytes) already *are* unicode strings

(The remaining safe_unicode calls are still needed and can't just be removed,
generally because we in these cases still have to convert from bytes to unicode
strings.)

py3: support new stdlib module names configparser, urllib and http lib

From "2to3 -f imports".

py3: use global next() function instead of .next() method

From 2to3 -f next.

lib: de-obfuscate nested functions in pygmentize_annotation and how they always pass an annotate_from_changeset_func to AnnotateHtmlFormatter

These functions contain the only call to annotate_highlight, which contains
the only call to AnnotateHtmlFormatter.

annotate_highlight is thus clearly always passed an
annotate_from_changeset_func.

lib: handle both HTML, unsafe strings, and exceptions passed to helpers.flash()

Before, h.flash would trust any input to contain html ... and callers would
convert exceptions to string, often with a simple str() or unicode() ... which
really didn't deserve to be trusted.

Instead, only trust messages that have a __html__ and escape anything else ...
but also apply str/unicode on the parameter so the caller doesn't have to but
*can* pass an exception directly.

vcs: always return bytes from node.content

We will rather have the unicode conversions explicit.

Note: Py3 bytes doesn't have .startswith - replace that with a regexp.

cleanup: minor refactorings and simplification of dict usage

Makes it more py3 compatible.

py3: use comprehensions and generators instead of filters - it is more explicit, and sometimes shorter

From 2to3 -f filter.

py3: cleanup map usage and avoid py3 ambiguity

Based on 2to3 -f map ... but replace map with something more explicit (unless
born and raised in a lisp world) (but sometimes slightly more verbose).

helpers: refactor user_or_none to keep things simple and make sure we only cache simple values - no SA objects

This could potentially improve "unbounded memory usage" ... or least clarify
which objects are kept alive.

helpers: replace webhelpers.flash with own implementation

webhelpers is dead.

One small function implements pretty much the same functionality, using the
same session key so tests still pass, but also very simple and without external
dependencies.

It could be implemented with a class and different methods for adding, getting
and clearing. But internally, it would probably have pretty much the same
helper function has here. So let's just avoid the unnecessary complexity and
keep it simple.

helpers: refactor flash - drop unnecessary subclassing, and just wrap using helper functions

Mainly dedenting.

helpers: handle webhelpers2 select with option groups

b077cf7e7f90 missed that we have those - for example for PR creation.

helpers: refactor select to build Options without temporary list

The API seems slightly more suited for that.

helpers: fix bad handling of select values with length 2 - 'hg' showed up as 'g' in repo types list

Strings with length 2 are not tuples ...

helpers: use WebHelpers2 as much as possible - it supports Python3, and WebHelpers is dead

The remaining uses of WebHelpers have to be replaced too - see
https://webhelpers2.readthedocs.io/en/latest/migrate.html .

helpers: handle CSRF protection directly, without using webhelpers, pylonslib and secure_form

Based on webhelpers/pylonslib/secure_form.py .

compare: correct display of special branch names in initial placeholder

When a branch name contains special characters like '<' or '>', and a
'compare' operation is performed with such branch as one of the two compare
sides, then the special branch name will be part of the URL, e.g.

http://localhost:5000/myrepo/compare/branch@master...branch@%3Cscript%3Eblabla%3C/script%3E?other_repo=myrepo

The encoded branch name is then used at page load as placeholders for the
branch selection dropdowns. But, the special characters, were escaped too
much, causing '<' to become &lt; in the display of the dropdown.

The placeholder was escaped via the default mako escape filter, before being
passed to make_revision_dropdown, thus too early. We want the raw value.
h.js() (copied from the default branch) gives us that, while still
formatting and escaping the string so it is safe inside the script tag.

urls: allow canonical_url to contain more than just a hostname

Although the .ini file gives the example:

canonical_url = https://kallithea.example.com/repos

it does not actually work. The '/repos' part is stripped off by the
canonical_url method.

The 'host' entry in the arguments passed to routes.url does not strictly
need to be a pure hostname. At least, the implementation does no validation
of this fact, it is concatenated verbatim between the protocol and the rest
of the URL.

As mapping Kallithea to a subpath of a base hostname is a valid
implementation, the canonical_url feature should allow it.

issues: support generic regex replacements in issue_url and issue_prefix

Issue reference linking is pretty limited:
- the issue_url is a literal with only three special tokens {id},
{repo} and {repo_name}. There is no way to let the URL be dependent on
other elements of the input issue reference.
- The value for {id} is somewhat oddly determined by the concatenation of
all parenthesized groups in the issue_pat regular expression
- the link text of the resulting link is limited to the contents of the
literal issue_prefix with the determined {id}. It is not possible to
retain the input issue reference verbatim, nor to let the link text be
dependent on other elements of the input issue reference.

This commit makes the issue reference linking more flexible:

- issue_prefix is replaced by the more generic issue_sub(stitution), which
is a string that may contain backreferences to regex groups specified in
issue_pat. This string, with backreferences resolved, is used as the
link text of urlified issue references.
- if issue_sub is empty, the entire text matched by issue_pat is used as
the link text.
- like issue_sub, also issue_url can contain backreferences to regex groups.
- {id} is no longer treated as a special token, as it can be solved by
generic backreferences ('\g<id>' assuming issue pattern contains something
like '(P<id>\d+)'. {repo} and {repo_name} are still supported, because
their value is provided externally and not normally part of the
issue pattern.

Documentation and ini file template is updated as well.

issues: backout special whitespace handling

This is essentially a backout of commit 32e1e0745d3c.

That commit checked for whitespace at the beginning of the matched issue
reference, and explicitly retained it in the resulting link text.

The way this was handled is not only suboptimal, e.g. a set of 4 spaces
would still be reduced to 1, but is also not actually necessary: if
whitespace before the issue reference is not required, then it does not need
to be specified in the issue pattern, and if it _is_ required, then a
positive lookbehind assertion can be used instead.

issues: gracefully handle invalid issue patterns

issue_pat is provided by the admin and can be invalid due to bad syntax.
In this case, a page load by a user could cause 500 Internal Server Error.

Instead, add a try..except clause around the compilation of issue_pat, and
skip invalid patterns.

lib: fix detection of &#39; as issue reference

Commit 494c793cc160 changed HTML escaping to please HTML 4 email readers.
The HTML entity '&apos;' was replaced by '&#39;'.
Unfortunately, the pound character '#' is often used to mark issue
references, like 'bug #56'. While this depends on the issue patterns
actually configured, this pattern is so common that we cannot expect users
to set their issue_pat regular expressions such that '&#123;' is not
matched.

Instead, keep the original '&apos;' replacement at first in method html_escape,
but introduce a final step that just replaces '&apos;' with '&#39;'.

The order of replacement in urlify_text then changes from:
html_escape (to HTML4)
urlify_issues
to
html_escape (to HTML5)
urlify_issues
make HTML5 more like HTML4

Test coverage show the problem case being solved.

issues: make issue_prefix optional again

Commit 39a59e6915bb398b42c3c2a63c48a950e9d63b55 (helpers: refactor and
optimize urlify_issues) made issue_prefix mandatory, while previously it
could be empty. An empty issue_prefix is useful when the entire issue
pattern needs to be used in the created link.

For example, consider a pattern 'PR123' that needs to be translated into:
http://example.com/pullrequests/PR123.

This could be configured with:
issue_pat = (PR\d+)
issue_server_link = http://example.com/pullrequests/{id}
issue_prefix =

We still refuse the issue pattern when issue_prefix is not present at all.

lib: change &apos; to &#39; to satisfy Outlook HTML rendering

The HTML entity &apos; (') did not exist in the HTML 4 spec [1] and was only
added later.
As Outlook (and Thunderbird) uses an old engine to render HTML and CSS, it does
not recognize this entity and treats it as a literal string.

Therefore, use the equivalent &#39; code which should be recognized by all
browsers, even those restricted to HTML 4.

[1] https://www.w3.org/TR/html4/sgml/entities.html

less: use .label and .label-* style instead of custom .*tag

This reduces the code and makes it more consistent.

Visualy this results in bold texts (which in case of small fonts is easier to read)
and a bit more padding left and right.

For a proper spacing, this requires to remove whitespaces before and after
the texts inside the labels.

Since the name changed from tag to label, also rename the variables and functions.

templates: remove redundant pull-left and pull-right for changeset progress bars

"float: left" and "float: right" for these elements are already set in
kallithea-diff.less.

Aligns it with compare_diff.html which already didn't use "pull" on "node" and
"changes".

gravatar: use icon-gravatar for default gravatar sizing and styling instead of icon-empty

We may want to use a different style for gravatar then for icon-empty,
so we should give it its own class, but still consider it an icon, like the
default icon-user is.

gravatar: use background-image style instead of img element for gravatar

When having several 1000 users, chrome will fail to load all the gravatar img
elements on the _admin/users page (ERR_INSUFFICIENT_RESOURCES).

If instead we use a background-image style, chrome will only load the images of
the elements that are actually displayed.

codingstyle: trivial whitespace fixes

Reported by flake8.

style: consistently use 'changeset_hash' class for monospace hashes

Get rid of 'revision-link' and some use of 'hash'.

There is still some use of 'hash' for columns, but the content there already
has 'changeset_hash'.

tg: minimize future diff by some mocking and replacing some pylons imports with tg

No actual tg dependency yet, just a temporary hack faking tg as an alias for
pylons.

Based on work by Alessandro Molina.

templates: properly escape inline JavaScript values

TLDR: Kallithea has issues with escaping values for use in inline JS.
Despite judicious poking of the code, no actual security vulnerabilities
have been found, just lots of corner-case bugs. This patch fixes those,
and hardens the code against actual security issues.

The long version:

To embed a Python value (typically a 'unicode' plain-text value) in a
larger file, it must be escaped in a context specific manner. Example:

>>> s = u'<script>alert("It\'s a trap!");</script>'

1) Escaped for insertion into HTML element context

>>> print cgi.escape(s)
&lt;script&gt;alert("It's a trap!");&lt;/script&gt;

2) Escaped for insertion into HTML element or attribute context

>>> print h.escape(s)
&lt;script&gt;alert(&#34;It&#39;s a trap!&#34;);&lt;/script&gt;

This is the default Mako escaping, as usually used by Kallithea.

3) Encoded as JSON

>>> print json.dumps(s)
"<script>alert(\"It's a trap!\");</script>"

4) Escaped for insertion into a JavaScript file

>>> print '(' + json.dumps(s) + ')'
("<script>alert(\"It's a trap!\");</script>")

The parentheses are not actually required for strings, but may be needed
to avoid syntax errors if the value is a number or dict (object).

5) Escaped for insertion into a HTML inline <script> element

>>> print h.js(s)
("\x3cscript\x3ealert(\"It's a trap!\");\x3c/script\x3e")

Here, we need to combine JS and HTML escaping, further complicated by
the fact that "<script>" tag contents can either be parsed in XHTML mode
(in which case '<', '>' and '&' must additionally be XML escaped) or
HTML mode (in which case '</script>' must be escaped, but not using HTML
escaping, which is not available in HTML "<script>" tags). Therefore,
the XML special characters (which can only occur in string literals) are
escaped using JavaScript string literal escape sequences.

(This, incidentally, is why modern web security best practices ban all
use of inline JavaScript...)

Unsurprisingly, Kallithea does not do (5) correctly. In most cases,
Kallithea might slap a pair of single quotes around the HTML escaped
Python value. A typical benign example:

$('#child_link').html('${_('No revisions')}');

This works in English, but if a localized version of the string contains
an apostrophe, the result will be broken JavaScript. In the more severe
cases, where the text is user controllable, it leaves the door open to
injections. In this example, the script inserts the string as HTML, so
Mako's implicit HTML escaping makes sense; but in many other cases, HTML
escaping is actually an error, because the value is not used by the
script in an HTML context.

The good news is that the HTML escaping thwarts attempts at XSS, since
it's impossible to inject syntactically valid JavaScript of any useful
complexity. It does allow JavaScript errors and gibberish to appear on
the page, though.

In these cases, the escaping has been fixed to use either the new 'h.js'
helper, which does JavaScript escaping (but not HTML escaping), OR the
new 'h.jshtml' helper (which does both), in those cases where it was
unclear if the value might be used (by the script) in an HTML context.
Some of these can probably be "relaxed" from h.jshtml to h.js later, but
for now, using h.jshtml fixes escaping and doesn't introduce new errors.

In a few places, Kallithea JSON encodes values in the controller, then
inserts the JSON (without any further escaping) into <script> tags. This
is also wrong, and carries actual risk of XSS vulnerabilities. However,
in all cases, security vulnerabilities were narrowly avoided due to other
filtering in Kallithea. (E.g. many special characters are banned from
appearing in usernames.) In these cases, the escaping has been fixed
and moved to the template, making it immediately visible that proper
escaping has been performed.

Mini-FAQ (frequently anticipated questions):

Q: Why do everything in one big, hard to review patch?
Q: Why add escaping in specific case FOO, it doesn't seem needed?

Because the goal here is to have "escape everywhere" as the default
policy, rather than identifying individual bugs and fixing them one
by one by adding escaping where needed. As such, this patch surely
introduces a lot of needless escaping. This is no different from
how Mako/Pylons HTML escape everything by default, even when not
needed: it's errs on the side of needless work, to prevent erring
on the side of skipping required (and security critical) work.

As for reviewability, the most important thing to notice is not where
escaping has been introduced, but any places where it might have been
missed (or where h.jshtml is needed, but h.js is used).

Q: The added escaping is kinda verbose/ugly.

That is not a question, but yes, I agree. Hopefully it'll encourage us
to move away from inline JavaScript altogether. That's a significantly
larger job, though; with luck this patch will keep us safe and secure
until such a time as we can implement the real fix.

Q: Why not use Mako filter syntax ("${val|h.js}")?

Because of long-standing Mako bug #140, preventing use of 'h' in
filters.

Q: Why not work around bug #140, or even use straight "${val|js}"?

Because Mako still applies the default h.escape filter before the
explicitly specified filters.

Q: Where do we go from here?

Longer term, we should stop doing variable expansions in script blocks,
and instead pass data to JS via e.g. data attributes, or asynchronously
using AJAX calls. Once we've done that, we can remove inline JavaScript
altogether in favor of separate script files, and set a strict Content
Security Policy explicitly blocking inline scripting, and thus also the
most common kind of cross-site scripting attack.

auth: simplify repository group permission checks

In practice, Kallithea has the 'group.admin' permission imply the
'group.write' permission, which again implies 'group.read'.

This codifies this practice by replacing HasRepoGroupPermissionAny
"perm function" with the new HasRepoGroupLevel function, reducing the
risk of errors and saving quite a lot of typing.

auth: simplify repository permission checks

In practice, Kallithea has the 'repository.admin' permission imply the
'repository.write' permission, which again implies 'repository.read'.

This codifies/enforces this practice by replacing HasRepoPermissionAny
"perm function" with the new HasRepositoryLevel function, reducing the
risk of errors and saving quite a lot of typing.

template: use Bootstrap tooltips and popover instead of handmade tooltips

Based on work from Andrew Shadura <andrew@shadura.me>.

Further modified by Mads Kiilerich.

show_changeset_tooltip is merged into tooltip_activate.

templates: use bootstrap for progress bars

Based on work form Andrew Shadura <andrew@shadura.me>.

Further modified by Mads Kiilerich.

templates: replace invalid metatag attributes with data attributes

(We could also use plain classes ... but stick to the plan ...)