git: fix pull request deletion - don't crash on deleting refs to PR heads

The refs name was passed as unicode string, and that would cause failure like:
File ".../site-packages/dulwich/repo.py", line 720, in __delitem__
if name.startswith(b"refs/") or name == b"HEAD":
TypeError: startswith first arg must be str or a tuple of str, not bytes

Fixed by correctly passing the ref name as bytes, as we do when creating the PR
refs.

Tests added by Mads Kiilerich.

cleanup: get rid of most "import *"

Apply script generated with the following hack:
(
hg loc '*.py'|xargs pyflakes-2 | sed -rn "s/([^:]*):.*'(.*)' may be undefined, or defined from star imports.*/sed -ri 's,\\\\<\2\\\\>([^=]|$),XXXX.\2\\\\1,g' \1/gp" | sort -u
hg loc '*.py'|xargs pyflakes-2 | sed -rn "s/([^:]*):.* undefined name '(.*)'$/sed -ri 's,\\\\<\2\\\\>([^=]|$),XXXX.\2\\\\1,g' \1/gp" | sort -u
hg loc '*.py'|xargs pyflakes-2 | sed -rn "s/([^:]*):.*'(from .*)\.([^.]*) import \*' used.*/sed -ri 's,\\\\<XXXX\\\\.,\3.,g' \1/gp" | sort -u
hg loc '*.py'|xargs pyflakes-2 | sed -rn "s/([^:]*):.*'(from .*)\.([^.]*) import \*' used.*/sed -ri 's,\2\\\\.\3 .*,\2 import \3,g' \1/gp" | sort -u
) | grep -v kallithea/bin/kallithea_cli_ishell.py > fix2.sh

lib: handle both HTML, unsafe strings, and exceptions passed to helpers.flash()

Before, h.flash would trust any input to contain html ... and callers would
convert exceptions to string, often with a simple str() or unicode() ... which
really didn't deserve to be trusted.

Instead, only trust messages that have a __html__ and escape anything else ...
but also apply str/unicode on the parameter so the caller doesn't have to but
*can* pass an exception directly.

codingstyle: replace comparison for equality against None with "is" expression

Both style and correctness.

Reported by flake8.

codingstyle: trivial whitespace fixes

Reported by flake8.

pullrequests: fix broken "new PR iteration" handling of ancestor changes

An earlier refactor (5d60c9a391cd) flubbed this code and accidentally
assumed the most recent common ancestor would not change when iterating,
which is of course only the case when there are no merges from 'other' to
'org' among the new revisions.

This was not only not caught during manual testing (nor review), but
neither did the test suite test this. That has now also been rectified.

backend: replace Pylons with TurboGears2

Replace the no-longer-supported Pylons application framework by TurboGears2
which is largely compatible/similar to Pylons.
Some interesting history is described at:
https://en.wikipedia.org/wiki/TurboGears

Changes by Dominik Ruf:
- fix sql config in test.ini

Changes by Thomas De Schampheleire:
- set-up of test suite
- tests: 'fix' repo archival test failure
Between Pylons and TurboGears2, there seems to be a small difference in the
headers sent for repository archive files, related to character encoding.
It is assumed that this difference is not important, and that the test
should just align with reality.
- remove need to import helpers/app_globals in lib
TurboGears2 by default expects helpers and app_globals to be available
in lib. For this reason kallithea/lib/__init__.py was originally changed
to include those files. However, this triggered several types of
circular import problems. If module A imported something from lib (e.g.
lib.annotate), and lib.helpers imported (possibly indirectly) module A,
then there was a circular import. Fix this by overruling the relevant
method of tg AppConfig, which is also hinted in the TurboGears2 code.
Hereby, the include of something from lib does not automatically import
helpers, greatly reducing the chances of circular import problems.
- make sure HTTP error '400' uses the custom error pages
TurboGears2 does not by default handle HTTP status code
'400 (Bad Request)' via the custom error page handling, causing a
standard non-styled error page.
- disable transaction manager
Kallithea currently handles its own transactions and does not need the
TurboGears2 transaction manager. However, TurboGears2 tries to enable it
by default and fails, throwing an error during application initialization.
The error itself seemed to be harmless for normal application functioning,
but was nevertheless confusing.
- add backlash as required dependency: backlash is meant as the WebError
replacement in TurboGears2 (originally WebError is part of Pylons). When
debug==true, it provides an interactive debugger in the browser. When
debug==false, backlash is necessary to show backtraces on the console.
- misc fixes

tests: add global test_context_fixture

Move the existing app_test_context_fixture from test_pullrequests.py to
conftest.py to make it available to all test modules.

It is useful in two cases:

1. there is test setup code (xUnit style) that needs to execute in the same
test context as the actual test.

2. even without test setup code, an entire test needs to be executed in a
test context. In this case, the fixture just reduces code complexity by not
requiring changes in the test code (compared to standard 'with
test_context').

It is possible to apply this (or any) fixture to an entire test class using
the class decorator
@pytest.mark.usefixtures("...")
This is similar to 'autouse=True' but can be used even if the fixture is
defined elsewhere.

tests: give test_pullrequests a test_context in addition to the usual app_fixture

test_pullrequests has unit-ish tests that invoke library functions directly ...
but these happen to require localization initialized. For TG2 we only have
localization initialized when we have a test_context. Thus, additionally wrap
these tests in a test_context.

tests: avoid test_pullrequests keeping dead sqlalchemy repository instance around outside transaction

The previous behaviour happened to work but is fragile when things change -
such as migrating to TG2.

tests: fix pull request tests - review_members is not a string but a list of integers (as strings)

Also sneaking in a bit of response status cleanup ...

db: remove superfluous Session.add calls

Don't re-add objects to the SQLAlchemy Session just because they were
modified. Session.add is only for freshly constructed objects that
SQLAlchemy doesn't know about yet.

The rules are quite simple:

When creating a database object by calling the constructor directly, it
must explicitly be added to the session.

When creating an object using a factory function (like "create_repo"),
the returned object has already (by convention) been added to the
session, and should not be added again.

When getting an object from the session (via Session.query or any of the
utility functions that look up objects in the database), it's already
added, and should not be added again. SQLAlchemy notices attribute
modifications automatically for all objects it knows about.

tests: add as little code as possible in __init__.py

kallithea/tests/__init__.py contained quite a lot of code, including the test
base class TestController. This in itself may be considered bad practice.

Specifically, this poses a problem when using pytest 3.0+, in which asserts
in some files are not automatically rewritten to give improved assert
output. That problem can be fixed by explicitly registering such files for
assertion rewriting, but that register call should be executed _before_ said
files are imported. I.e. if the register call is in
kallithea/tests/__init__.py, assert calls in __init__.py itself can not be
rewritten.

Since the TestController base class does effectively contain asserts, and we
do not want to execute the register call from somewhere outside the
kallithea/tests directory, we need to move the TestController class to
another file (kallithea/tests/base.py) so we can have a register call in
__init__.py before loading base.py.

While not strictly necessary to fix the mentioned pytest problem, we take
the opportunity to fully clean __init__.py and move everything to
the new kallithea/tests/base.py. While doing so, unnecessary imports are
removed, and imports are ordered alphabetically. Explicit imports of symbols
from modules that were already imported as a whole, are removed in favor of
fully qualifying the references (e.g. tempfile._RandomNameSequence).

pytest migration: rename TestControllerPytest back to TestController

The name TestControllerPytest was introduced to allow a temporary situation
where nose/unittest and pytest-based tests could coexist.
This situation is now over, so the base test class can be renamed again.

pytest migration: functional: switch to standard assert statements

Use unittest2pytest to replace unittest-style assert statements (e.g.
assertEqual) with standard Python assert statements to benefit from pytest's
improved reporting on assert failures.

The conversion by unittest2pytest was correct, except for line wrapping
problems.

pytest migration: convert functional tests with setup/teardown methods

pytest also supports setup/teardown methods like unittest (even though
pytest fixtures are more powerful and should be the end goal).
Only difference is the naming and signature of setUp (setup_method) and
tearDown (teardown_method).

pytest migration: convert simple functional tests to TestControllerPytest

Replace usage of TestController with TestControllerPytest for those files in
tests/functional where there is no setUp/tearDown method (the pytest
equivalent to be investigated) and that do not use test parametrization.

tests: more consistently use unicode where unicode is expected

Nothing but extra u annotation to turn str constants into unicode.

This has been verified by hacking sqlalchemy to fail if wrong string types are passed.

cleanup: remove unused imports

Found with pyflakes.

pullrequests: detect invalid reviewers and raise HTTPBadRequest

Normally, the creation of a pullrequest with invalid reviewers is not
possible because the list of reviewers is populated from a form element
that only shows valid reviewers.
However, if creating a pullrequest through an API call, invalid reviewers
can be specified but would not be detected. The reviewer would be encoded
in the database as 'NULL'/None, and opening such a pull request would cause
a server error.

Instead, detect invalid reviewers at pullrequest creation/update time and
raise HTTPBadRequest.

Second step in two-part process to rename directories.
This is the actual directory rename.