imports: try to use global imports unless it is a layering violation

To minimize the impact, do imports that violate the layering at runtime instead
of at import time.

tests: introduce REUSE_TEST_DB as an alternative to TEST_DB

With TEST_DB, the specified database would be dropped (if existing) and created
from scratch. That would fail if the user only had been granted access to that
database without general database creation permissions.

With REUSE_TEST_DB, the database must exist, and the tables in the existing
database will be deleted before creating the new ones.

This is for example useful for testing on PostgreSQL without full database
privileges. (With MariaDB/MySQL, it is possible to grant create permissions to
a database before it exists, so it is easier to use TEST_DB.)

db: drop SA caching_query and FromCache, and thus sql_cache_short beaker cache

It is not a good idea to have dead ORM objects. If we want caching, we should
do it explicit.

It is unknown how much this cache helps, but we can profile and introduce
better caching of simple data where relevant.

ssh: introduce 'kallithea-cli ssh-update-authorized-keys' command for updating authorized_keys file

Based on work by Ilya Beda <ir4y.ix@gmail.com> on
https://bitbucket.org/ir4y/rhodecode/commits/branch/ssh_server_support ,
incorporating gearbox support by Anton Schur <tonich.sh@gmail.com> and also
heavily modified by Mads Kiilerich.

This commit also incorporates a fix for Windows by Dominik Ruf,
and better handling of the case where the parent dir of 'authorized_keys'
does not exist or is not writable, by Bradley M. Kuhn <bkuhn@ebb.org>.

ssh: introduce ini setting 'ssh_enabled', disabled by default

Administrators should control the use of SSH and may want to disable SSH
access, temporarily or permanently.

An explicit setting ssh_enabled is better than e.g. checking for a valid
ssh_authorized_keys setting, to allow such trivial temporary disabling.

To keep the controllers simple, introduce a decorator IfSshEnabled instead
of repeating the same config checks in every method.

config: rename .ini settings to avoid beaker-session deprecation warnings

Gets rid of:

data/env/lib/python2.7/site-packages/tg/wsgiapp.py:68
data/env/lib/python2.7/site-packages/tg/wsgiapp.py:68: DeprecationWarning: Session options should start with session. instead of baker.session.
app_wrapper = wrapper(self.wrapped_dispatch, self.config)

Old .ini files will still work, but one day upstream will drop the deprecated functionality ...

tests: make a commit in auto_clear_ip_permissions

Sometimes, test_delete_ip would fail because UserIpMap entries left behind. It
was perhaps because a commit was missing and sessions thus sometimes were leaked?

docs: update i18n doc after TG migration changed lang to i18n.lang and test.ini is generated

These descriptions are not elegant, but now we make them less wrong.

tests: generate test.ini directly from template

Get rid of the committed generated test.ini

codingstyle: trivial whitespace fixes

Reported by flake8.

tests: use temporary copy of test.ini, possibly customized for TEST_DB

It was an undocumented feature that if setting the environment variable
TEST_DB, that would be used for tests instead of the default
kallithea/tests/test.ini sqlalchemy.url . That did however not work for Git
hooks and tests would fail.

Instead, create a copy of test.ini in the temp folder and use that for testing.
If TEST_DB is set, edit the file so the specified DB URL is used. This fixes
Git hook related tests if TEST_DB is used.

Since this also changes the path of %(here)s to the temporary location, just
use the default paths.

This also has the advantage that the data folders are now in the temp folder
as well. Therefore a broken data folder, from a past run, can no longer
influence a test.

tests: fix failures due to environment LANG settings (Issue #286)

When the environment in which the tests are run dictates a non-English
language (e.g. via LANG=de_DE.UTF-8), certain tests started to fail because
formencode validation messages where translated in that language.

A test failing this way is for example
test_create_in_group_without_needed_permissions in
kallithea/tests/functional/test_admin_repos.py .

This is a regression of commit 3b29103657df (i18n: remove explicit
formencode language setting), but in fact reverting it is not the right
solution as there is no problem except during test execution.

Instead, force the formencode language to 'empty' (meaning English) for test
execution only.

See also http://www.formencode.org/en/latest/i18n.html

tests: drop 'environ' as global variable - just pass it to URLGenerator as temporary variable

HTTP_HOST is mandatory - specify it as 'example.com'.

tests: move time zone setup to pytest_configure

Test setup stuff belongs in pytest_configure.

tests: use webserver fixture for running vcs tests, removing need for serving manually

So far a bit hack-ish.

tests: introduce webserver fixture

To be used for tests that actually access a running web server - especially to
make the manual vcs tests less manual.

config: initialize routes directly from RootController

Let RootController directly initialize routes instead of app_cfg injecting
the mapper.

For test initialization, we also need a handle to the mapper. We could
either recreate it (but fragile if the real mapper initialization is changed
later) or obtain it from a fresh RootController. This commit opts for the
latter.

backend: replace Pylons with TurboGears2

Replace the no-longer-supported Pylons application framework by TurboGears2
which is largely compatible/similar to Pylons.
Some interesting history is described at:
https://en.wikipedia.org/wiki/TurboGears

Changes by Dominik Ruf:
- fix sql config in test.ini

Changes by Thomas De Schampheleire:
- set-up of test suite
- tests: 'fix' repo archival test failure
Between Pylons and TurboGears2, there seems to be a small difference in the
headers sent for repository archive files, related to character encoding.
It is assumed that this difference is not important, and that the test
should just align with reality.
- remove need to import helpers/app_globals in lib
TurboGears2 by default expects helpers and app_globals to be available
in lib. For this reason kallithea/lib/__init__.py was originally changed
to include those files. However, this triggered several types of
circular import problems. If module A imported something from lib (e.g.
lib.annotate), and lib.helpers imported (possibly indirectly) module A,
then there was a circular import. Fix this by overruling the relevant
method of tg AppConfig, which is also hinted in the TurboGears2 code.
Hereby, the include of something from lib does not automatically import
helpers, greatly reducing the chances of circular import problems.
- make sure HTTP error '400' uses the custom error pages
TurboGears2 does not by default handle HTTP status code
'400 (Bad Request)' via the custom error page handling, causing a
standard non-styled error page.
- disable transaction manager
Kallithea currently handles its own transactions and does not need the
TurboGears2 transaction manager. However, TurboGears2 tries to enable it
by default and fails, throwing an error during application initialization.
The error itself seemed to be harmless for normal application functioning,
but was nevertheless confusing.
- add backlash as required dependency: backlash is meant as the WebError
replacement in TurboGears2 (originally WebError is part of Pylons). When
debug==true, it provides an interactive debugger in the browser. When
debug==false, backlash is necessary to show backtraces on the console.
- misc fixes

tests: add global test_context_fixture

Move the existing app_test_context_fixture from test_pullrequests.py to
conftest.py to make it available to all test modules.

It is useful in two cases:

1. there is test setup code (xUnit style) that needs to execute in the same
test context as the actual test.

2. even without test setup code, an entire test needs to be executed in a
test context. In this case, the fixture just reduces code complexity by not
requiring changes in the test code (compared to standard 'with
test_context').

It is possible to apply this (or any) fixture to an entire test class using
the class decorator
@pytest.mark.usefixtures("...")
This is similar to 'autouse=True' but can be used even if the fixture is
defined elsewhere.

tests: replace yield_fixtures with plain fixtures

yield_fixtures and fixtures are the same since pytest-3.0, but the name
yield_fixture is now deprecated.

tests: remove unused code from conftest.py

conftest.py does not contain an import of pylons (only of pylons.test) so
usage of e.g. pylons.config can never work, so it cannot be used.
Since the tests run fine regardless, we can get rid of this code.

db: always add to session in Setting.create_or_update

There's no use case for not adding the newly created Setting to the
SQLAlchemy session (and thus, once we commit, the database).

With this change, all the various "create" model methods add their
result to the session.

tests: add as little code as possible in __init__.py

kallithea/tests/__init__.py contained quite a lot of code, including the test
base class TestController. This in itself may be considered bad practice.

Specifically, this poses a problem when using pytest 3.0+, in which asserts
in some files are not automatically rewritten to give improved assert
output. That problem can be fixed by explicitly registering such files for
assertion rewriting, but that register call should be executed _before_ said
files are imported. I.e. if the register call is in
kallithea/tests/__init__.py, assert calls in __init__.py itself can not be
rewritten.

Since the TestController base class does effectively contain asserts, and we
do not want to execute the register call from somewhere outside the
kallithea/tests directory, we need to move the TestController class to
another file (kallithea/tests/base.py) so we can have a register call in
__init__.py before loading base.py.

While not strictly necessary to fix the mentioned pytest problem, we take
the opportunity to fully clean __init__.py and move everything to
the new kallithea/tests/base.py. While doing so, unnecessary imports are
removed, and imports are ordered alphabetically. Explicit imports of symbols
from modules that were already imported as a whole, are removed in favor of
fully qualifying the references (e.g. tempfile._RandomNameSequence).

tests: admin_users: make sure all custom IP permissions are cleared

test_delete_ip changes IP permissions and at the end tries to clean up by
deleting it again. When the delete fails, there is still a restricted IP
permission configuration, causing other tests to fail.
Use the recently added pytest fixture auto_clear_ip_permissions to fix this.
The fixture is extended to not only clear IP permissions for the default
user, but also for the 'regular' test user.

Similar cleanup code in test_add_ip is deleted because it serves the same
purpose, but would fail to execute if something went wrong earlier in the
test method.

This commit is very similar to an earlier commit that covers similar
add/delete IP functionality for the default user, in test_permissions.py.

tests: admin_permissions: make sure all custom IP permissions are cleared

test_add_ips changes IP permissions and at the end tries to clean up by
deleting it again. When the delete fails, there is still a restricted IP
permission configuration, causing other tests to fail.

Attempt a direct delete to the database without relying on the application
post/get interface.

tests: fix issue with set_test_settings fixture not committing changes

Flushing session is not enough to make setting changes visible in controller
tests.

tests: add set_test_settings fixture

Uses https://pytest.org/latest/yieldfixture.html#yieldfixture to restore
settings after tests. Takes into account current specifics of list-valued
settings.

tests: add create_test_user fixture

Uses https://pytest.org/latest/yieldfixture.html#yieldfixture to provide
auto-deleting users to tests.

tests: hide database setup queries with pytest

Unlike nosetest, pytest would show the database setup queries at the
beginning of the test run, which don't bring added value.
Hide them by disabling logging during this time.

Note that this does not provide an easy method of enabling the logging on
demand (what could be done with the -s switch in nosetest).

tests: set app_globals and translater in conftest.py for py.test

Copy over code from the pylons nose plugin for setting app_globals and for
setting a translator object.

This allows more of the functional tests to pass; particularly the ones related
to i18n.

tests: Add conftest.py for pytest

This adds a conftest.py for pytest that initializes the Pylons app so that the
SQLAlchemy session is bound and such.

This is sort of the Python analogue for the nosetest plugin that Pylons
includes: https://github.com/Pylons/pylons/blob/master/pylons/test.py