lib: move some template filter functions from utils2 to webutils

While quite Kallithea specific, we prefer to have these functions in webutils
where they soon can be exposed when templates don't need the whole helpers
module.

lib: move urlification to webutils

Less use of helpers in model.

imports: try to use global imports unless it is a layering violation

To minimize the impact, do imports that violate the layering at runtime instead
of at import time.

lib: consistently import helpers the same way

Make it easier to grep for any remaining potential layering-violating use of
helpers.

lib: move canonical_url & co to webutils

This gives less of the unfortunate use of helpers - especially in model.

routing: separate url handling from routing - move it to webutils

This is a helper method relying on the thread local tg.request. We didn't have
a good place to put it. Now we do.

This (re)moves unfortunate dependencies to the routing module (which almost is
a controller).

tests: simplify test_lib mocking of routing.url

Mock internals of the url generator to avoid import tricks.

isort: upgrade to 5.1.2

The changes to non-top-level imports seem nice.

lib: drop own asbool implementation and consistently use tg.support.converters as utils2.asbool

str2bool never reported error on odd input such as '' or '-1', but the tg
asbool behaviour of raising ValueError("String is not true/false: %r" % obj) in
that case seems fine.

cleanup: get rid of most "import *"

Apply script generated with the following hack:
(
hg loc '*.py'|xargs pyflakes-2 | sed -rn "s/([^:]*):.*'(.*)' may be undefined, or defined from star imports.*/sed -ri 's,\\\\<\2\\\\>([^=]|$),XXXX.\2\\\\1,g' \1/gp" | sort -u
hg loc '*.py'|xargs pyflakes-2 | sed -rn "s/([^:]*):.* undefined name '(.*)'$/sed -ri 's,\\\\<\2\\\\>([^=]|$),XXXX.\2\\\\1,g' \1/gp" | sort -u
hg loc '*.py'|xargs pyflakes-2 | sed -rn "s/([^:]*):.*'(from .*)\.([^.]*) import \*' used.*/sed -ri 's,\\\\<XXXX\\\\.,\3.,g' \1/gp" | sort -u
hg loc '*.py'|xargs pyflakes-2 | sed -rn "s/([^:]*):.*'(from .*)\.([^.]*) import \*' used.*/sed -ri 's,\2\\\\.\3 .*,\2 import \3,g' \1/gp" | sort -u
) | grep -v kallithea/bin/kallithea_cli_ishell.py > fix2.sh

clone_url: clarify variable naming

Minor nits:
user -> username (temporarily more complex)
uri_tmpl -> clone_uri_tmpl
qualified_home_url -> prefix_url
decoded_path -> prefix

middleware: move handling of permanent repo URLs to separate middleware

This is about the handling of repo URLs like '_123' for the repo with repo_id
123. The de-mangling of such URLs was spread out across multiple layers. It
fits much more nicely as a middleware layer. The code in routing and simplehg /
simplegit can thus be removed.

The base _get_by_id function was confusing - fix it by removing it. To do that,
refactor utils introducing fix_repo_id_name to replace get_repo_by_id.

We now assume in the application that we never have any extra leading '/' in
URL paths.

And while trailing extra '/' might be fine in actual URLs, they must be handled
at the routing level, not propagated through all layers. This changeset is not
really changing that.

urls: allow canonical_url to contain more than just a hostname

Although the .ini file gives the example:

canonical_url = https://kallithea.example.com/repos

it does not actually work. The '/repos' part is stripped off by the
canonical_url method.

The 'host' entry in the arguments passed to routes.url does not strictly
need to be a pure hostname. At least, the implementation does no validation
of this fact, it is concatenated verbatim between the protocol and the rest
of the URL.

As mapping Kallithea to a subpath of a base hostname is a valid
implementation, the canonical_url feature should allow it.

issues: support generic regex replacements in issue_url and issue_prefix

Issue reference linking is pretty limited:
- the issue_url is a literal with only three special tokens {id},
{repo} and {repo_name}. There is no way to let the URL be dependent on
other elements of the input issue reference.
- The value for {id} is somewhat oddly determined by the concatenation of
all parenthesized groups in the issue_pat regular expression
- the link text of the resulting link is limited to the contents of the
literal issue_prefix with the determined {id}. It is not possible to
retain the input issue reference verbatim, nor to let the link text be
dependent on other elements of the input issue reference.

This commit makes the issue reference linking more flexible:

- issue_prefix is replaced by the more generic issue_sub(stitution), which
is a string that may contain backreferences to regex groups specified in
issue_pat. This string, with backreferences resolved, is used as the
link text of urlified issue references.
- if issue_sub is empty, the entire text matched by issue_pat is used as
the link text.
- like issue_sub, also issue_url can contain backreferences to regex groups.
- {id} is no longer treated as a special token, as it can be solved by
generic backreferences ('\g<id>' assuming issue pattern contains something
like '(P<id>\d+)'. {repo} and {repo_name} are still supported, because
their value is provided externally and not normally part of the
issue pattern.

Documentation and ini file template is updated as well.

issues: backout special whitespace handling

This is essentially a backout of commit 32e1e0745d3c.

That commit checked for whitespace at the beginning of the matched issue
reference, and explicitly retained it in the resulting link text.

The way this was handled is not only suboptimal, e.g. a set of 4 spaces
would still be reduced to 1, but is also not actually necessary: if
whitespace before the issue reference is not required, then it does not need
to be specified in the issue pattern, and if it _is_ required, then a
positive lookbehind assertion can be used instead.

issues: gracefully handle invalid issue patterns

issue_pat is provided by the admin and can be invalid due to bad syntax.
In this case, a page load by a user could cause 500 Internal Server Error.

Instead, add a try..except clause around the compilation of issue_pat, and
skip invalid patterns.

issues: add more tests and restructure

Add two test cases using the default issue pattern from the ini file
template prior to commit 9cef5615da7b, and some more cases, some of which
verify the behavior regarding test cases with mandatory/optional leading
whitespace.

Increase readability by grouping tests with the same issue pattern, server
and prefix, and place the urlified result on a separate line from the input,
wrapped in case there are multiple issues in the input.

For the result, """ string delimiters are used instead of ' as used in the
input, to visually distinguish them better.

lib: fix detection of &#39; as issue reference

Commit 494c793cc160 changed HTML escaping to please HTML 4 email readers.
The HTML entity '&apos;' was replaced by '&#39;'.
Unfortunately, the pound character '#' is often used to mark issue
references, like 'bug #56'. While this depends on the issue patterns
actually configured, this pattern is so common that we cannot expect users
to set their issue_pat regular expressions such that '&#123;' is not
matched.

Instead, keep the original '&apos;' replacement at first in method html_escape,
but introduce a final step that just replaces '&apos;' with '&#39;'.

The order of replacement in urlify_text then changes from:
html_escape (to HTML4)
urlify_issues
to
html_escape (to HTML5)
urlify_issues
make HTML5 more like HTML4

Test coverage show the problem case being solved.

tests: issues: use urlify_text wrapper rather than urlify_issues directly

urlify_text is a wrapper around different kinds of urlification methods.
The other tests for urlification use that wrapper, while the new
urlify_issues tests introduced in commit b43e77fa50dd used urlify_issues
directly.

By itself this is not a real problem, but it does not allow to detect the
interplay between different urlification issues. For example, an issue
pattern normally detecting #123 conflicts with the escaping of an apostrophe
' into &#39; . Some new tests exposing this problem are added.

issues: make issue_prefix optional again

Commit 39a59e6915bb398b42c3c2a63c48a950e9d63b55 (helpers: refactor and
optimize urlify_issues) made issue_prefix mandatory, while previously it
could be empty. An empty issue_prefix is useful when the entire issue
pattern needs to be used in the created link.

For example, consider a pattern 'PR123' that needs to be translated into:
http://example.com/pullrequests/PR123.

This could be configured with:
issue_pat = (PR\d+)
issue_server_link = http://example.com/pullrequests/{id}
issue_prefix =

We still refuse the issue pattern when issue_prefix is not present at all.

tests: add basic tests for urlify_issues

Add some tests to cover the existing behavior of urlify_issues.

less: use .label and .label-* style instead of custom .*tag

This reduces the code and makes it more consistent.

Visualy this results in bold texts (which in case of small fonts is easier to read)
and a bit more padding left and right.

For a proper spacing, this requires to remove whitespaces before and after
the texts inside the labels.

Since the name changed from tag to label, also rename the variables and functions.

codingstyle: replace upper-case variable names with lower-case ones

PEP8 issues reported by flake8.

codingstyle: trivial whitespace fixes

Reported by flake8.

style: consistently use 'changeset_hash' class for monospace hashes

Get rid of 'revision-link' and some use of 'hash'.

There is still some use of 'hash' for columns, but the content there already
has 'changeset_hash'.

backend: replace Pylons with TurboGears2

Replace the no-longer-supported Pylons application framework by TurboGears2
which is largely compatible/similar to Pylons.
Some interesting history is described at:
https://en.wikipedia.org/wiki/TurboGears

Changes by Dominik Ruf:
- fix sql config in test.ini

Changes by Thomas De Schampheleire:
- set-up of test suite
- tests: 'fix' repo archival test failure
Between Pylons and TurboGears2, there seems to be a small difference in the
headers sent for repository archive files, related to character encoding.
It is assumed that this difference is not important, and that the test
should just align with reality.
- remove need to import helpers/app_globals in lib
TurboGears2 by default expects helpers and app_globals to be available
in lib. For this reason kallithea/lib/__init__.py was originally changed
to include those files. However, this triggered several types of
circular import problems. If module A imported something from lib (e.g.
lib.annotate), and lib.helpers imported (possibly indirectly) module A,
then there was a circular import. Fix this by overruling the relevant
method of tg AppConfig, which is also hinted in the TurboGears2 code.
Hereby, the include of something from lib does not automatically import
helpers, greatly reducing the chances of circular import problems.
- make sure HTTP error '400' uses the custom error pages
TurboGears2 does not by default handle HTTP status code
'400 (Bad Request)' via the custom error page handling, causing a
standard non-styled error page.
- disable transaction manager
Kallithea currently handles its own transactions and does not need the
TurboGears2 transaction manager. However, TurboGears2 tries to enable it
by default and fails, throwing an error during application initialization.
The error itself seemed to be harmless for normal application functioning,
but was nevertheless confusing.
- add backlash as required dependency: backlash is meant as the WebError
replacement in TurboGears2 (originally WebError is part of Pylons). When
debug==true, it provides an interactive debugger in the browser. When
debug==false, backlash is necessary to show backtraces on the console.
- misc fixes

templates: replace invalid metatag attributes with data attributes

(We could also use plain classes ... but stick to the plan ...)

tests: use test_context for tests needing internationalization

Instead of relying on the top-level handling of the translator, use the
newly introduced test_context.

Turbogears2 migration: replace pylons.url by kallithea.config.routing.url

In preparation for the migration to Turbogears2, introduce a
kallithea.config.routing.url to replace pylons.url. The implementation is
basically the same: wrap around routes.url().

This change involves:
- a number of import statement changes
- fixing some tests in test_libs.py; to avoid duplication, the different
implementations of fake_url were grouped in one place.

This change was first proposed by Alessandro Molina in his initial
port. Following changes were made afterwards:
- move UrlGenerator from kallithea.lib.utils to kallithea.config.routing
- add documentation to UrlGenerator
- kallithea/lib/auth.py used url_for instead of url, for no apparent reason
so this was changed.
- fix libs tests
- rebase onto Pylons-based Kallithea first

tests: add as little code as possible in __init__.py

kallithea/tests/__init__.py contained quite a lot of code, including the test
base class TestController. This in itself may be considered bad practice.

Specifically, this poses a problem when using pytest 3.0+, in which asserts
in some files are not automatically rewritten to give improved assert
output. That problem can be fixed by explicitly registering such files for
assertion rewriting, but that register call should be executed _before_ said
files are imported. I.e. if the register call is in
kallithea/tests/__init__.py, assert calls in __init__.py itself can not be
rewritten.

Since the TestController base class does effectively contain asserts, and we
do not want to execute the register call from somewhere outside the
kallithea/tests directory, we need to move the TestController class to
another file (kallithea/tests/base.py) so we can have a register call in
__init__.py before loading base.py.

While not strictly necessary to fix the mentioned pytest problem, we take
the opportunity to fully clean __init__.py and move everything to
the new kallithea/tests/base.py. While doing so, unnecessary imports are
removed, and imports are ordered alphabetically. Explicit imports of symbols
from modules that were already imported as a whole, are removed in favor of
fully qualifying the references (e.g. tempfile._RandomNameSequence).

helpers: introduce ascii preserving visual markup of *bold*

No matter what kind of text (code and markup) is parsed, it will at most have
slightly odd styling. Nothing will be lost and it can be copy-pasted correctly.

Other kinds of markup (like _underline_ and /italic/) has also been considered
... but they will too often interfere with valid code and command snippets.

helpers: in urlify_text, use <br> for newlines in pre-formatted text so it can be cut'n'pasted correctly

Before, when copying the preformatted nice looking ASCII-artsy text in a PR or
changeset description from a browser to a text editor, it would be pasted as
text without newline. Simply put, copy/paste ignores that it is inside a <pre>
/ white-space:pre-wrap section.

Instead, translate newlines to <br> which always translates to a newline when
pasted.

helpers: merge urlify_commit into urlify_text

More reuse - they are already almost doing the same.

test_urlify_test is updated to use the new repo_name parameter and urlify
hashes.

pytest migration: rename TestControllerPytest back to TestController

The name TestControllerPytest was introduced to allow a temporary situation
where nose/unittest and pytest-based tests could coexist.
This situation is now over, so the base test class can be renamed again.

pytest migration: other: switch to standard assert statements

Use unittest2pytest to replace unittest-style assert statements (e.g.
assertEqual) with standard Python assert statements to benefit from pytest's
improved reporting on assert failures.

The conversion by unittest2pytest was correct except for:
- 'import pytest' is not automatically added when needed
- line wrapping in string formatting caused a syntax error in the
transformed code. Reported upstream at
https://github.com/pytest-dev/unittest2pytest/issues/3 .

utils: improve extract_mentioned_users usefulness

Previously, extract_mentioned_users performed a regex search, converted
to a set to remove duplicates, converted that set back into a list, and
sorted the list on username, ignoring case. Every single caller of the
function then promptly took the carefully pruned and sorted result and
put it back into a set, rendering the entire exercise pointless.

In addition, every caller also resolved the usernames to database User
objects.

In this changeset, extract_mentioned_users is changed to return a set
of database User objects directly. A new extract_mentioned_usernames
function, which does only username extraction, is kept for the sole
purpose of providing a more testable interface (no database needed).

Bonus feature: The new extract_mentioned_users will prune non-existent
users, as well as the DEFAULT user. This means it is no longer possible
to @mention (and send notifications to) the DEFAULT user.

cleanup: use example.com for tests and examples

example.com is explicitly reserved for this purpose. Using that means
we won't accidentally hammer a real server or real email address if an
example value escapes into the wild, e.g. in an automated test.

The domain "kallithea.example.com" has been used throughout to refer to the
example Kallithea server.

remove vestiges of Python 2.5 support

We only support Python 2.6 and 2.7; hence we do not need to import
with-statement support from __future__.

cleanup: make module self-naming consistent

The self naming is quite pointless ... but at least make it right.

helpers: make desc_stylize work when given html escaped strings

The function returns strings with html markup. The result can thus not be
escaped and we must assume that the input already has been escaped. That may or
may not the case yet.

utils: better display of ages >1 year when in short form

* 1..2 years: display as months instead
* 2+ years: round

mention: improve pattern for matching multiple mentions

It would (silently as always) fail to recognize some @mentions - especially
when only separated by a single whitespace char.

Correct licensing information in individual files.

The top-level license file is now LICENSE.md.

Also, in various places where there should have been joint copyright holders
listed, a single copyright holder was listed. It does not appear easy to add
a link to a large list of copyright holders in these places, so it simply
refers to the fact that various authors hold copyright.

In future, if an easy method is discovered to link to a list from those
places, we should do so.

Finally, text is added to LICENSE.md to point to where the full list of
copyright holders is, and that Kallithea as a whole is GPLv3'd.

Second step in two-part process to rename directories.
This is the actual directory rename.