lib: move some template filter functions from utils2 to webutils

While quite Kallithea specific, we prefer to have these functions in webutils
where they soon can be exposed when templates don't need the whole helpers
module.

imports: try to use global imports unless it is a layering violation

To minimize the impact, do imports that violate the layering at runtime instead
of at import time.

lib: move link_to_ref from helpers to utils2

Less reason to import the messy helpers.

lib: move extract_mentioned_users out of utils2

Avoid model.db reference to keep utils2.py independent of Kallithea classes.

imports: always import the whole kallithea module to use top level kallithea variables

This is slightly more lazy and might avoid some depeendency issues.

lib: remove now unused class Optional

Last usage was removed in previous commit.

lib: remove now unused class Oattr/OptionalAttr

Last usage was removed in previous commit.

isort: upgrade to 5.1.2

The changes to non-top-level imports seem nice.

lib: drop own asbool implementation and consistently use tg.support.converters as utils2.asbool

str2bool never reported error on odd input such as '' or '-1', but the tg
asbool behaviour of raising ValueError("String is not true/false: %r" % obj) in
that case seems fine.

py3: remove safe_unicode in places where it no longer is needed because all strings (except bytes) already *are* unicode strings

(The remaining safe_unicode calls are still needed and can't just be removed,
generally because we in these cases still have to convert from bytes to unicode
strings.)

py3: migrate from urllib2 to urllib

Based on 2to3 urllib, but with unnecessary imports removed.

py3: make get_current_authuser handle missing tg context consistently and explicitly

tg context handling ends up using
tg.support.registry.StackedObjectProxy._current_obj for attribute access ...
which if no context has been pushed will end up in:
raise TypeError(
'No object (name: %s) has been registered for this '
'thread' % self.____name__)

utils2.get_current_authuser used code like:
if hasattr(tg.tmpl_context, 'authuser'):

Python 2 hasattr will call __getattr__ and return False if it throws any
exception. (It would thus catch the TypeError and silently fall through to use
the default user None.) This hasattr behavior is confusing and hard to use
correctly. Here, it was used incorrectly. It has been common practice to work
around by using something like:
getattr(x, y, None) is not None

Python 3 hasattr fixed this flaw and only catches AttributeError. The TypeError
would thus (rightfully) be propagated. That is a change that must be handled
when introducing py3 support.

The get_current_authuser code could more clearly and simple and py3-compatible
be written as:
return getattr(tmpl_context, 'authuser', None)
- but then we also have to handle the TypeError explicitly ... which we are
happy to do.

cleanup: minor correctness fixes

"Trivial" potential problems spotted with pytype.

lib: clean up ext_json and how it is used - avoid monkey patching

Note that py3 json.dumps will return ASCII (with all unicode escaped) encoded
as str. But we generally want JSON as bytes (which json.loads also can read),
so also wrap the result with ascii_bytes in many places.

git: more elegant handling of installed pre/post-receive hook failing on direct repo access

The hook would fail with a long backtrace when get_hook_environment raise an error exception.

Instead, as first thing in the entry point from the hook, catch that situation
and report it nicely before "quietly" skipping the hook:

[mk@here myrepo]$ git push
Enumerating objects: 3, done.
Counting objects: 100% (3/3), done.
Writing objects: 100% (3/3), 204 bytes | 204.00 KiB/s, done.
Total 3 (delta 0), reused 0 (delta 0)
remote: Skipping Kallithea Git post-recieve hook 'hooks/post-receive'.
remote: Git was apparently not invoked by Kallithea: Environment variable KALLITHEA_EXTRAS not found
To /tmp/somerepo
* [new branch] master -> master
[mk@here myrepo]$

We could be paranoid and let it (and the pre hook) fail ... but that doesn't
seem helpful.

Reported by Edmund Wong at [1].

[1] https://lists.sfconservancy.org/pipermail/kallithea-general/2019q4/003071.html

lib: avoid import string.replace

Odd way to do it ... and gone in py3.

lib: introduce string conversion functions for ASCII without further encoding concerns

Avoid the trial-and-error and vagueness of the "safe" functions.

This should replace some use of safe_unicode and safe_str. It will mostly be a
noop in py2 but will be crucial in py3.

lib: establish py3 compatible strategy for string handling: introducing safe_bytes and deprecating safe_str

The meaning of safe_str will change when moving to py3. All use of safe_str is
thus tech debt that we have to chop off, mostly by moving to either
safe_unicode or safe_bytes ... or dropping because we know what we are doing
and rely on the improved type safety in py3.

lib: only maintain one copy of safe_str / safe_unicode

The standalone-ish nature of vcs gets a bit in the way. It already depends on
some very generic Kallithea functionality. But for now, avoid code duplication,
and let Kallithea use vcs functionality instead of duplicating it.

py3: use comprehensions and generators instead of filters - it is more explicit, and sometimes shorter

From 2to3 -f filter.

pylint: fix W0715(raising-format-tuple) Exception arguments suggest string formatting might be intended

kallithea/lib/utils2.py:453: W0715(raising-format-tuple) in get_changeset_safe: Exception arguments suggest string formatting might be intended
kallithea/lib/vcs/backends/git/changeset.py:360: W0715(raising-format-tuple) in GitChangeset.fill_archive: Exception arguments suggest string formatting might be intended
kallithea/lib/vcs/backends/hg/changeset.py:322: W0715(raising-format-tuple) in MercurialChangeset.fill_archive: Exception arguments suggest string formatting might be intended

helpers: use WebHelpers2 as much as possible - it supports Python3, and WebHelpers is dead

The remaining uses of WebHelpers have to be replaced too - see
https://webhelpers2.readthedocs.io/en/latest/migrate.html .

clone_url: use regexp and single pass for variable expansion - avoid nested expansion in case a variable value would contain a variable reference

It would probably never be a problem in real life, but just avoid the tech debt
of doing it wrong ...

clone_url: clarify variable naming

Minor nits:
user -> username (temporarily more complex)
uri_tmpl -> clone_uri_tmpl
qualified_home_url -> prefix_url
decoded_path -> prefix

hooks: move _handle_rc_scm_extras to utils2 as set_hook_environment and get_hook_environment

This is also a step away from using the name 'extras' ... but still a lot of
traces left behind ...

hooks: drop 'server_url' parameter

The parameter might be conceptually nice, but it was only available for 2
hooks. To be really useful, it should have been available everywhere. It also
only reported the URL from the web request that initiated the hook ... and as
such it does some layering violations. The user URL might be the address that
should be used internally from the hook. And it can conceptually not be made
available actions that doesn't originate from a user web request. It seems much
better that custom hooks configure what they want to do. Perhaps by reading the
.ini file and using canonical_url.

middleware: drop fix_PATH

Back out ccbdff90e5a0. That seemed like an odd hack. In order to work properly,
it should not only be applied for protocol access middleware, but also for web
UI and for commands. So evidently, it is not really necessary.

The problem it describes is fixed much better in 5e501b6ee639 by setting the
right python executable in the hook scripts, further improved in 1bafb2d07709
and 6df08d78f8e7 to *actually* use the right python executable.

locking: drop the pull-to-lock / push-to-unlock functionality

The feature is not worth the maintenance cost. The locking is too coarse and
unflexible with insufficient UI and UX. The implementation is also quite
invasive in tricky areas of the code, and thus high maintenance.
Dropping this will enable other cleanup ... or at least make it easier.

tests: fix utils2 extract_mentioned_usernames doctest

The extra matched output seems ok.

utils2: Move ask_ok out of paster_commands

The ask_ok function is still a mess and should be cleaned up ...

hooks: use os.environ directly for KALLITHEA_EXTRAS instead of sometimes passing it as parameter

It is better to be consistent ... even if that means always using the global
environment that is hard to avoid.

utils: move repo_name_slug to utils2 to prevent import cycle on setup_db

After commit 57a733313e4f, 'gearbox setup-db -c my.ini' fails with an import
cycle as follows:

Traceback (most recent call last):
File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/bin/gearbox", line 11, in <module>
sys.exit(main())
File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/gearbox/main.py", line 199, in main
return gearbox.run(args)
File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/gearbox/main.py", line 145, in run
return self._run_subcommand(remainder)
File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/gearbox/main.py", line 149, in _run_subcommand
subcommand = self.command_manager.find_command(argv)
File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/gearbox/commandmanager.py", line 78, in find_command
cmd_factory = cmd_ep.resolve()
File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2324, in resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
File "/home/tdescham/repo/contrib/kallithea/kallithea-release/kallithea/lib/paster_commands/setup_db.py", line 27, in <module>
from kallithea.lib.db_manage import DbManage
File "/home/tdescham/repo/contrib/kallithea/kallithea-release/kallithea/lib/db_manage.py", line 47, in <module>
from kallithea.model.repo_group import RepoGroupModel
File "/home/tdescham/repo/contrib/kallithea/kallithea-release/kallithea/model/repo_group.py", line 35, in <module>
import kallithea.lib.utils
File "/home/tdescham/repo/contrib/kallithea/kallithea-release/kallithea/lib/utils.py", line 48, in <module>
from kallithea.model.repo_group import RepoGroupModel
ImportError: cannot import name RepoGroupModel


i.e. kallithea.model.repo_group wants to import kallithea.lib.utils which
in turn wants to import kallithea.model.repo_group.

In fact there exists kallithea.lib.utils and kallithea.lib.utils2.
The current split is that 'utils2' contains 'simple' utilities, none of which
depend on kallithea models, controllers, ... In contrast, 'utils' does rely
on such kallithea classes.

As kallithea.model.repo_group was only include kallithea.lib.utils for its
repo_name_slug method, which has no dependency on other kallithea classes,
move that method (and its dependent recursive_replace) to
kallithea.lib.utils2 instead. This fixes the import cycle.

commands: remove custom (re-)initializing of database model

The model has already beeen initialized through TurboGears after
make_app_without_logging has been called.

Since kallithea.lib.utils2.engine_from_config no longer is used, remove it.
All remaining references to engine_from_config are directly to the one from
sqlalchemy.

codingstyle: trivial whitespace fixes

Reported by flake8.

tg: minimize future diff by some mocking and replacing some pylons imports with tg

No actual tg dependency yet, just a temporary hack faking tg as an alias for
pylons.

Based on work by Alessandro Molina.

cleanup: refer less to User.DEFAULT_USER

Down the road we might want to identify the default user in another
way than by username.

model: greatly simplify generation of gist IDs

Holy overcomplication, Batman!

auth: add support for "Bearer" auth scheme (API key variant)

This allows the API key to be passed in a header instead of the query
string, reducing the risk of accidental API key leaks:

Authorization: Bearer <api key>

The Bearer authorization scheme is standardized in RFC 6750, though
used here outside the full OAuth 2.0 authorization framework. (Full
OAuth can still be added later without breaking existing users.)

pygmentsutils: separate from util2

__get_lem and __get_index_filenames has nothing in common with the rest
of util2, neither with respects to implementation nor usage, and belongs
in a dedicated module. Also, they are clearly not actually private, so
shouldn't be named with leading underscores.

utils: improve extract_mentioned_users usefulness

Previously, extract_mentioned_users performed a regex search, converted
to a set to remove duplicates, converted that set back into a list, and
sorted the list on username, ignoring case. Every single caller of the
function then promptly took the carefully pruned and sorted result and
put it back into a set, rendering the entire exercise pointless.

In addition, every caller also resolved the usernames to database User
objects.

In this changeset, extract_mentioned_users is changed to return a set
of database User objects directly. A new extract_mentioned_usernames
function, which does only username extraction, is kept for the sole
purpose of providing a more testable interface (no database needed).

Bonus feature: The new extract_mentioned_users will prune non-existent
users, as well as the DEFAULT user. This means it is no longer possible
to @mention (and send notifications to) the DEFAULT user.

cleanup: make module self-naming consistent

The self naming is quite pointless ... but at least make it right.

cleanup: consistently use 'except ... as ...:'

Use the Python 2.6+ syntax instead of the old confusing 'except ..., ...'
syntax.

i18n: use plain strings on the english lookup side of translations

Unicode strings are unnecessarily complex and do not in any of our cases add
any value. It is thus better to consistently use plain strings.

utils: make API key generator more random

The API key generator abused temporary filenames in what seems to be an attempt
of creating keys that unambiguously specified the user and thus were unique
across users. A final hashing did however remove that property.

More importantly, tempfile is not documented to use secure random numbers ...
and it only uses 6 characters, giving approximately 36 bits of entropy.

Instead, use the cryptographically secure os.urandom directly to generate keys
with the same length but with the full 160 bits of entropy.

Reported and fixed by Andrew Bartlett.

comments: don't eat character before @mention when rendering to rst

The character before @ should not only be ungrouped it should also not be eaten
so re.sub still works as expected.

utils: better display of ages >1 year when in short form

* 1..2 years: display as months instead
* 2+ years: round

cleanup: avoid some 'except Exception' catching - catch specific exceptions or log it and show what happened

This has a risk of introducing regressions ... but we want to get rid of all
exception muting and make the whole system less fragile and easier to debug.

mention: improve pattern for matching multiple mentions

It would (silently as always) fail to recognize some @mentions - especially
when only separated by a single whitespace char.

Correct licensing information in individual files.

The top-level license file is now LICENSE.md.

Also, in various places where there should have been joint copyright holders
listed, a single copyright holder was listed. It does not appear easy to add
a link to a large list of copyright holders in these places, so it simply
refers to the fact that various authors hold copyright.

In future, if an easy method is discovered to link to a list from those
places, we should do so.

Finally, text is added to LICENSE.md to point to where the full list of
copyright holders is, and that Kallithea as a whole is GPLv3'd.