imports: try to use global imports unless it is a layering violation

To minimize the impact, do imports that violate the layering at runtime instead
of at import time.

routing: move ADMIN_PREFIX constant from routing to top level kallithea module

Perhaps not the ideal place, but that is how we handle somewhat similar data.
This avoids an unfortunate dependency from the model to routing.

db: move URL_SEP constant from db to top level kallithea module

URL_SEP is used in several places - having it in the db module is too high
level.

__init__ might not be the best place for this, but it does no harm to the
dependencies graph to place constants there.

permissions: drop hg.create.write_on_repogroup "Repository creation with group write access" setting

Simplify permissions system and get rid of some confusing tech debt.

Before, the global 'write_on_repogroup' setting controlled what write
permission on a repo group meant.

With this change, users can create repositories in a repo group if and only if
they have write access. Write access to a repo group will now mean the
permission to create repositories in it.

Write access to repo groups must be granted explicitly. There should not be any
other reason to grant write access than to allow users to create repos. There
is thus no upgrade concerns for this change.

An admin that doesn't want users to create repos in a repogroup should just not
give them write access.

These global settings might still exist in the database, but is ignored and no
longer used and do no harm.

lib: drop own asbool implementation and consistently use tg.support.converters as utils2.asbool

str2bool never reported error on odd input such as '' or '-1', but the tg
asbool behaviour of raising ValueError("String is not true/false: %r" % obj) in
that case seems fine.

validators: don't catch all Exceptions as invalid clone URIs, be specific

When adding a new repository with a remote clone URI, the URI will be
validated in some way. Several exceptions could occur during that
validation.

Previously, the code would catch based on 'Exception', which means that
_any_ exception would cause the URI to be found invalid. This means that
errors in the code (e.g. related to Python 3 conversion) were also
categorized as 'invalid clone URI'. And thus, the tests that test an
actually invalid URI would pass, even though there was a bug.

Now, things have been refactored so it only is relevant to catch
InvalidCloneUriException. Any other exception will now yield a 500 Internal
Server Error, as expected.

cleanup: improve exception logging from validators

There were no point in logging the whole raw exception with stacktrace - it is
more relevant to log something helpful

Catching of Exception is however bad.

Also, it seems odd what ValidPerms is doing with "update permissions" in the
validator.

cleanup: minimize use of lambda expressions - we have 'def' for that purpose

Fix some flake8 warnings "E731 do not assign a lambda expression, use a def".

py3: trivial renaming of .iteritems() to .items()

A bit like "2to3 -f dict", but we don't want list().

validator: fix ASCII password check to verify if it can be *encoded* in ascii

In Python 2, unicode strings have a .decode method (which really doesn't make
sense). Python 3 has more strict typing by design, and unicode strings don't
have a .decode method.

A Unicode string "is ASCII" if it can be encoded as ASCII. The check should
thus *encode* to ASCII - not decode.

cleanup: minor refactorings and simplification of dict usage

Makes it more py3 compatible.

py3: use comprehensions and generators instead of filters - it is more explicit, and sometimes shorter

From 2to3 -f filter.

utils: drop make_ui parameter clear_session - this utility function should never mess with sessions

clear_session seems to have been based on the idea that utility functions that
happened to create a session also should clean up. But instead, we now
consistently take care of session removal at a high level.

flake8: fix F632 use ==/!= to compare str, bytes, and int literals

- even if the values probably would have been interned ...

setup: bump formencode minimum version to 1.3.0

The formencode version range included both 1.2.x and 1.3.x releases.
However, since 1.3.0, _to_python and validate_python are deprecated and
renamed to _convert_to_python and _validate_python, respectively.

With current pytest, these (long) deprecation warnings are shown in the test
logs.

There are two options:
- restrict maximum version to 1.2.x
- bump minimum version to 1.3.x

In this commit we choose the latter approach, going towards the future
rather than the past.

utils: refactor make_ui to always read from db, optionally also augmenting with hgrc content

simplehg was the only user of reading hgrc content ... and it would rather just
have everything at once.

The implementation in utils is moved around and re-indented, but without
significant changes.

locking: drop the pull-to-lock / push-to-unlock functionality

The feature is not worth the maintenance cost. The locking is too coarse and
unflexible with insufficient UI and UX. The implementation is also quite
invasive in tricky areas of the code, and thus high maintenance.
Dropping this will enable other cleanup ... or at least make it easier.

utils: move repo_name_slug to utils2 to prevent import cycle on setup_db

After commit 57a733313e4f, 'gearbox setup-db -c my.ini' fails with an import
cycle as follows:

Traceback (most recent call last):
File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/bin/gearbox", line 11, in <module>
sys.exit(main())
File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/gearbox/main.py", line 199, in main
return gearbox.run(args)
File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/gearbox/main.py", line 145, in run
return self._run_subcommand(remainder)
File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/gearbox/main.py", line 149, in _run_subcommand
subcommand = self.command_manager.find_command(argv)
File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/gearbox/commandmanager.py", line 78, in find_command
cmd_factory = cmd_ep.resolve()
File "/home/tdescham/repo/contrib/kallithea/venv/kallithea-release/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2324, in resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
File "/home/tdescham/repo/contrib/kallithea/kallithea-release/kallithea/lib/paster_commands/setup_db.py", line 27, in <module>
from kallithea.lib.db_manage import DbManage
File "/home/tdescham/repo/contrib/kallithea/kallithea-release/kallithea/lib/db_manage.py", line 47, in <module>
from kallithea.model.repo_group import RepoGroupModel
File "/home/tdescham/repo/contrib/kallithea/kallithea-release/kallithea/model/repo_group.py", line 35, in <module>
import kallithea.lib.utils
File "/home/tdescham/repo/contrib/kallithea/kallithea-release/kallithea/lib/utils.py", line 48, in <module>
from kallithea.model.repo_group import RepoGroupModel
ImportError: cannot import name RepoGroupModel


i.e. kallithea.model.repo_group wants to import kallithea.lib.utils which
in turn wants to import kallithea.model.repo_group.

In fact there exists kallithea.lib.utils and kallithea.lib.utils2.
The current split is that 'utils2' contains 'simple' utilities, none of which
depend on kallithea models, controllers, ... In contrast, 'utils' does rely
on such kallithea classes.

As kallithea.model.repo_group was only include kallithea.lib.utils for its
repo_name_slug method, which has no dependency on other kallithea classes,
move that method (and its dependent recursive_replace) to
kallithea.lib.utils2 instead. This fixes the import cycle.

utils: move clone URI validator function to more generic utils.is_valid_repo_uri

No changes to the functionality, even though the API and implementation could
use some clean-up ...

utils: move clone URI validator function to more generic utils.is_valid_repo_uri

No changes to the functionality, even though the API and implementation could
use some clean-up ...

codingstyle: trivial whitespace fixes

Reported by flake8.

validators: wrap hgsubversion import with try/except block like in settings.py

Found by pylint.

repositories: make sure repositories not only differ in casing

Repositories only differing in case cause problems:
* it can't be stored on case insensitive filesystems (Windows and MacOS)
* some databases can't easily handle case sensitive queries
* users will most certainly be confused by names that only differ in case

We will keep trying to be case sensitive on systems that can ... but on some
systems wrong casings might work. We don't care.

The validators are changed to prevent mixed case repo and repo group names.

Repository sensitivity tests are removed, and insensitivity tests are added
instead.

tg: minimize future diff by some mocking and replacing some pylons imports with tg

No actual tg dependency yet, just a temporary hack faking tg as an alias for
pylons.

Based on work by Alessandro Molina.

auth: simplify repository group permission checks

In practice, Kallithea has the 'group.admin' permission imply the
'group.write' permission, which again implies 'group.read'.

This codifies this practice by replacing HasRepoGroupPermissionAny
"perm function" with the new HasRepoGroupLevel function, reducing the
risk of errors and saving quite a lot of typing.

validators: remove unneeded M/StateObj classes

The M class is actually a wrapper around formencode.api.Validator.message,
inserting a translator into the 'state' object. Setting a translator into
the state object is indeed what is mentioned in the formencode docs.

But, if you work this way, the custom state should be set both for custom
validators, as well as for validators that are simple wrappers around
formencode's default validators (for example wrappers that just set a custom
message string). And the latter is what Kallithea is currently _not_ doing.

Also, when using formencode.api.Validator.message correctly, you should not
use the translator function _ on your validator strings manually.

Remove the inconsistency in Kallithea validators as follows:
- remove M and StateObj classes
- replace the usage of M by direct calls to self.message (which is now no
longer in charge of translating but simply of mapping a keyword onto the
right message)
- translation will be done by explicit _ calls at Kallithea side, so that
formencode directly receives translated strings.

db: rename RepoGroup.group_parent_id to parent_group_id

Also for consistency with the existing parent_group relationship.

auth: allow web login with email addresses

Let users log in using their email addresses instead of their user names.
This only applies to the web login, not git+http or hg+http protocols.

db: always match user emails case insensitively

This commit removes case-sensitive email matching.
It also adds a couple of tests which fail, to demonstrate a defect in the
current implementation (using ILIKE matching instead of case-insensitive
equality comparison).

spelling: use "email" consistently

The common English spelling is "email", not "e-mail" (and was indeed
also the most common, but not only, variant in the Kallithea UI).

http://grammarist.com/style/e-mail-email/

cleanup: remove unused imports

Found with pyflakes.

cleanup: consistently use 'except ... as ...:'

Use the Python 2.6+ syntax instead of the old confusing 'except ..., ...'
syntax.

privacy: don't tell users what is the reason for a failed login

Makes it harder for strangers to probe the instance for presence of
certain users. This can make it harder to break in, as it is now
harder to tell is a username or a password are wrong, so bruteforcing
should probably take a bit longer if you don't know what exactly are
you doing.

cleanup: check for None object identity in cases where that is what the 'contract' says

Avoid applying bool() on complex objects - it might return something unexpected
such as the key (which might be 0 and thus be false). Checking for None is more
safe and faster.

validators: cleanup of message wording

NotReviewedRevisions was unused and could thus got the big cleanup ...

i18n: use plain strings on the english lookup side of translations

Unicode strings are unnecessarily complex and do not in any of our cases add
any value. It is thus better to consistently use plain strings.

remote: simplify clone_uri UI

The UI was a bit weird ... probably in an attempt of making it editable while
hiding passwords. Instead, just show the URL with password hidden, and only
save it back if it changed.

The UI only contains the clone_uri with passwords hidden. It will thus only be
saved when the form result is different from the value that was shown to the
user.

spelling: consistent capitalization of URL

Change Url / url into URL.
Additionally, convert the sole use of 'Uri' to URL.

remotes: add support to clone from Mercurial repositories over ssh

This commit adds support to clone a remote Mercurial repository over ssh.

Interactive password authentication is not implemented, nor is support for
pbulic key authentication with passphrases; the repository should be
accessible using bare ssh key authentication.
For this reason, the ssh options -oBatchMode=yes and -oIdentitiesOnly=yes
are added to the ui.ssh setting of Mercurial.

spelling: fix various typos

This commit fixes various typos or basic English grammar mistakes found by
reviewing the kallithea.pot file.

Full correction of sentences that are not very well formulated, like missing
articles, is out of scope for this commit. Likewise for inconsistent
capitalization of strings like 'Repository group'/'Repository Group'.

cleanup: avoid some 'except Exception' catching - catch specific exceptions or log it and show what happened

This has a risk of introducing regressions ... but we want to get rid of all
exception muting and make the whole system less fragile and easier to debug.

Second step in two-part process to rename directories.
This is the actual directory rename.