lib: move webhelpers2 and friends to webutils

Gives less of the unfortunate use of helpers - especially in low level libs.

ssh: use fingerprint when deleting public keys

Avoid relying on a database index of the full public key string.

cleanup: get rid of most "import *"

Apply script generated with the following hack:
(
hg loc '*.py'|xargs pyflakes-2 | sed -rn "s/([^:]*):.*'(.*)' may be undefined, or defined from star imports.*/sed -ri 's,\\\\<\2\\\\>([^=]|$),XXXX.\2\\\\1,g' \1/gp" | sort -u
hg loc '*.py'|xargs pyflakes-2 | sed -rn "s/([^:]*):.* undefined name '(.*)'$/sed -ri 's,\\\\<\2\\\\>([^=]|$),XXXX.\2\\\\1,g' \1/gp" | sort -u
hg loc '*.py'|xargs pyflakes-2 | sed -rn "s/([^:]*):.*'(from .*)\.([^.]*) import \*' used.*/sed -ri 's,\\\\<XXXX\\\\.,\3.,g' \1/gp" | sort -u
hg loc '*.py'|xargs pyflakes-2 | sed -rn "s/([^:]*):.*'(from .*)\.([^.]*) import \*' used.*/sed -ri 's,\2\\\\.\3 .*,\2 import \3,g' \1/gp" | sort -u
) | grep -v kallithea/bin/kallithea_cli_ishell.py > fix2.sh

lib: handle both HTML, unsafe strings, and exceptions passed to helpers.flash()

Before, h.flash would trust any input to contain html ... and callers would
convert exceptions to string, often with a simple str() or unicode() ... which
really didn't deserve to be trusted.

Instead, only trust messages that have a __html__ and escape anything else ...
but also apply str/unicode on the parameter so the caller doesn't have to but
*can* pass an exception directly.

ssh: when adding a new public key, if no description is provided and the SSH key itself has a comment, use that as description

Based on work by Ilya Beda <ir4y.ix@gmail.com> on
https://bitbucket.org/ir4y/rhodecode/commits/branch/ssh_server_support , also
heavily modified by Mads Kiilerich.

ssh: admin management of ssh keys

Based on work by Ilya Beda <ir4y.ix@gmail.com> on
https://bitbucket.org/ir4y/rhodecode/commits/branch/ssh_server_support .
Bootstrap support, updates for POST methods, and tests by Anton Schur
<tonich.sh@gmail.com>.
Additional Bootstrap fixes by Dominik Ruf.
Also heavily modified by Mads Kiilerich.

tests: make test_admin_users user_and_repo_group_fail() fixture more stable

When adding authentication_token() to log_user(), database session lifetime
will in some cases change:

test_admin_users test_delete_repo_group_err() use the
user_and_repo_group_fail() fixture.

Before, it got ObjectDeletedError when trying to delete a deleted RepoGroup and
moved on.

After changing log_user(), py.test would emit a warning:

kallithea/tests/functional/test_admin_users.py::TestAdminUsersController::()::test_delete_repo_group_err
.../site-packages/sqlalchemy/orm/persistence.py:1340: SAWarning: DELETE statement on table 'groups' expected to delete 1 row(s); 0 were matched. Please set confirm_deleted_rows=False within the mapper configuration to prevent this warning.
% (table.description, expected, rows_matched)

Instead, use RepoGroup.get_by_group_name to verify the group exists before
trying to delete it.

codingstyle: trivial whitespace fixes

Reported by flake8.

backend: replace Pylons with TurboGears2

Replace the no-longer-supported Pylons application framework by TurboGears2
which is largely compatible/similar to Pylons.
Some interesting history is described at:
https://en.wikipedia.org/wiki/TurboGears

Changes by Dominik Ruf:
- fix sql config in test.ini

Changes by Thomas De Schampheleire:
- set-up of test suite
- tests: 'fix' repo archival test failure
Between Pylons and TurboGears2, there seems to be a small difference in the
headers sent for repository archive files, related to character encoding.
It is assumed that this difference is not important, and that the test
should just align with reality.
- remove need to import helpers/app_globals in lib
TurboGears2 by default expects helpers and app_globals to be available
in lib. For this reason kallithea/lib/__init__.py was originally changed
to include those files. However, this triggered several types of
circular import problems. If module A imported something from lib (e.g.
lib.annotate), and lib.helpers imported (possibly indirectly) module A,
then there was a circular import. Fix this by overruling the relevant
method of tg AppConfig, which is also hinted in the TurboGears2 code.
Hereby, the include of something from lib does not automatically import
helpers, greatly reducing the chances of circular import problems.
- make sure HTTP error '400' uses the custom error pages
TurboGears2 does not by default handle HTTP status code
'400 (Bad Request)' via the custom error page handling, causing a
standard non-styled error page.
- disable transaction manager
Kallithea currently handles its own transactions and does not need the
TurboGears2 transaction manager. However, TurboGears2 tries to enable it
by default and fails, throwing an error during application initialization.
The error itself seemed to be harmless for normal application functioning,
but was nevertheless confusing.
- add backlash as required dependency: backlash is meant as the WebError
replacement in TurboGears2 (originally WebError is part of Pylons). When
debug==true, it provides an interactive debugger in the browser. When
debug==false, backlash is necessary to show backtraces on the console.
- misc fixes

tests: use test_context for tests needing internationalization (bis)

Commit 8e3137064ab6 already introduced the use of test_context to cover
internationalization in the test suite, instead of setting it up globally.

When making changes related to formencode internationalization, a new batch
of internationalization errors popped up and a commit was made to fix them.
However, after some later refactoring, it looked as if the commit was not
needed anymore.

In Turbogears context, it was indeed not necessary as long as we still had
some places that used the dummy formencode.api._ rather than a real version
of _ (ugettext).
After cleaning up that forgotten import, the test internationalization
errors popped up again. Hence, we need to reapply the earlier commit (with
some changes).

tests: replace yield_fixtures with plain fixtures

yield_fixtures and fixtures are the same since pytest-3.0, but the name
yield_fixture is now deprecated.

tests: use test_context for tests needing internationalization

Instead of relying on the top-level handling of the translator, use the
newly introduced test_context.

users: use the form URL for POST so the htmlfill error page URL will be the same as the initial form URL.

This also reveals that a test used a GET route for POST - they just happened to
be the same.

These problems can probably be found in many places - there should be a general
cleanup.

tests: add as little code as possible in __init__.py

kallithea/tests/__init__.py contained quite a lot of code, including the test
base class TestController. This in itself may be considered bad practice.

Specifically, this poses a problem when using pytest 3.0+, in which asserts
in some files are not automatically rewritten to give improved assert
output. That problem can be fixed by explicitly registering such files for
assertion rewriting, but that register call should be executed _before_ said
files are imported. I.e. if the register call is in
kallithea/tests/__init__.py, assert calls in __init__.py itself can not be
rewritten.

Since the TestController base class does effectively contain asserts, and we
do not want to execute the register call from somewhere outside the
kallithea/tests directory, we need to move the TestController class to
another file (kallithea/tests/base.py) so we can have a register call in
__init__.py before loading base.py.

While not strictly necessary to fix the mentioned pytest problem, we take
the opportunity to fully clean __init__.py and move everything to
the new kallithea/tests/base.py. While doing so, unnecessary imports are
removed, and imports are ordered alphabetically. Explicit imports of symbols
from modules that were already imported as a whole, are removed in favor of
fully qualifying the references (e.g. tempfile._RandomNameSequence).

admin: don't redirect back to the list of users after creating a user

Staying on the user page gives a more natural flow, instead of having to click
the link in the 'Created' flash message to get back to the user.

tests: admin_users: make sure repo group is deleted

test_delete_repo_group_err creates then deletes a repository group. However,
if the delete fails the repository group remains. This later causes problems
in the model tests.

Introduce a pytest yield fixture to handle the creation _and_ deletion of
the repository group (suggested by Søren Løvborg).
The creation of the user needs to happen _before_ that of the user group,
and we cannot share data between two pytest fixtures, so the user is created
in the fixture as well.

tests: admin_users: make sure all custom IP permissions are cleared

test_delete_ip changes IP permissions and at the end tries to clean up by
deleting it again. When the delete fails, there is still a restricted IP
permission configuration, causing other tests to fail.
Use the recently added pytest fixture auto_clear_ip_permissions to fix this.
The fixture is extended to not only clear IP permissions for the default
user, but also for the 'regular' test user.

Similar cleanup code in test_add_ip is deleted because it serves the same
purpose, but would fail to execute if something went wrong earlier in the
test method.

This commit is very similar to an earlier commit that covers similar
add/delete IP functionality for the default user, in test_permissions.py.

pytest migration: rename TestControllerPytest back to TestController

The name TestControllerPytest was introduced to allow a temporary situation
where nose/unittest and pytest-based tests could coexist.
This situation is now over, so the base test class can be renamed again.

pytest migration: functional: switch to standard assert statements

Use unittest2pytest to replace unittest-style assert statements (e.g.
assertEqual) with standard Python assert statements to benefit from pytest's
improved reporting on assert failures.

The conversion by unittest2pytest was correct, except for line wrapping
problems.

tests: users: enable commented UsersController test

Previously, a test was written but could not be enabled until pytest
was fully available. Now that it's the case, enable the test.

pytest migration: convert simple functional tests to TestControllerPytest

Replace usage of TestController with TestControllerPytest for those files in
tests/functional where there is no setUp/tearDown method (the pytest
equivalent to be investigated) and that do not use test parametrization.

tests: more consistently use unicode where unicode is expected

Nothing but extra u annotation to turn str constants into unicode.

This has been verified by hacking sqlalchemy to fail if wrong string types are passed.

cleanup: use example.com for tests and examples

example.com is explicitly reserved for this purpose. Using that means
we won't accidentally hammer a real server or real email address if an
example value escapes into the wild, e.g. in an automated test.

The domain "kallithea.example.com" has been used throughout to refer to the
example Kallithea server.

security: apply CSRF check to all non-GET requests

The automatic CSRF protection was broken for POST requests with no
request payload parameters (but possibly containing request URI
parameters); a security hole was narrowly avoided because the code
base quite consistently checks the request method in the same way,
and because of browser protection against PUT/DELETE CSRF attacks.

Since explicit is better than implicit, the better way of checking
the HTTP request method is to simply check request.method, instead
of checking if request.POST is non-empty, which is subtly different
(it doesn't catch POST requests if all parameters are in the query
string) and non-obvious (because it also applies to PUT requests).

The commit also fixes some tests which relied on the CSRF protection
being broken. It does not fix all the controllers that still does
the misleading request.POST check, but since the CSRF check has now
been tightened, those are no longer a potential security issue.

spelling: use "email" consistently

The common English spelling is "email", not "e-mail" (and was indeed
also the most common, but not only, variant in the Kallithea UI).

http://grammarist.com/style/e-mail-email/

cleanup: remove unused imports

Found with pyflakes.

repos: fix abuse of dead 'repo' routing

The repo "show" controller didn't do anything and was unused. There was a
routing GET entry for it but it was only used for generating URLs for DELETE
and PUT operations that have separate controllers that happen to have the same
URL.

Use the right routing entries when generating URLs and drop the dead code.

users: add extra checks on editing the default user

There is no need to be able to edit e-mails or permissions of the default
user, so add the same checks as present in many other methods in the users
controller.

admin: users: factorize check for default user

Note that one specific unittest has been commented because it relies on
pytest features (monkeypatch). When pytest is the default test runner, the
test should be uncommented.

spelling: fix title casing on various translated strings

Primarily captions on other not-just-text and data.

tests: provide _authentication_token when POSTing

So far not used, just preparing for the the time when the actual checking is
introduced ...

This change is very verbose. self.app.post should perhaps just at this value
automagically ...

[security fix] api: don't send internal data unless asked for it

This changeset fixes CVE-2015-0260.
See <https://kallithea-scm.org/security/cve-2015-0260.html>; for
more details.

Second step in two-part process to rename directories.
This is the actual directory rename.