backend: replace Pylons with TurboGears2

Replace the no-longer-supported Pylons application framework by TurboGears2
which is largely compatible/similar to Pylons.
Some interesting history is described at:
https://en.wikipedia.org/wiki/TurboGears

Changes by Dominik Ruf:
- fix sql config in test.ini

Changes by Thomas De Schampheleire:
- set-up of test suite
- tests: 'fix' repo archival test failure
Between Pylons and TurboGears2, there seems to be a small difference in the
headers sent for repository archive files, related to character encoding.
It is assumed that this difference is not important, and that the test
should just align with reality.
- remove need to import helpers/app_globals in lib
TurboGears2 by default expects helpers and app_globals to be available
in lib. For this reason kallithea/lib/__init__.py was originally changed
to include those files. However, this triggered several types of
circular import problems. If module A imported something from lib (e.g.
lib.annotate), and lib.helpers imported (possibly indirectly) module A,
then there was a circular import. Fix this by overruling the relevant
method of tg AppConfig, which is also hinted in the TurboGears2 code.
Hereby, the include of something from lib does not automatically import
helpers, greatly reducing the chances of circular import problems.
- make sure HTTP error '400' uses the custom error pages
TurboGears2 does not by default handle HTTP status code
'400 (Bad Request)' via the custom error page handling, causing a
standard non-styled error page.
- disable transaction manager
Kallithea currently handles its own transactions and does not need the
TurboGears2 transaction manager. However, TurboGears2 tries to enable it
by default and fails, throwing an error during application initialization.
The error itself seemed to be harmless for normal application functioning,
but was nevertheless confusing.
- add backlash as required dependency: backlash is meant as the WebError
replacement in TurboGears2 (originally WebError is part of Pylons). When
debug==true, it provides an interactive debugger in the browser. When
debug==false, backlash is necessary to show backtraces on the console.
- misc fixes

gearbox: make a make-config sub-command available again

Drop the old kallithea-config command line tool and rework it to a replacement
for the make-config paster command. Make-config was experimental in paste, it
doesn't exist with gearbox, it was tied to pylons, and we already have the
luxury problem of having two ways of doing almost the same thing. This is a
good opportunity to get rid of one of them.

This replacement tool is Kallithea specific and doesn't need to be told that it
is Kallithea it has to create a config file for. The command should thus
perhaps be given a Kallithea specific name instead of the very generic name ...

gearbox: replace paster with something TurboGears2-ish that still works with the Pylons stack

This is a step towards moving away from the Pylons stack to TurboGears2, but
still independent of it.


Some notes from the porting - it could perhaps be the missing(?) documentation
for migrating from paster to gearbox:

Note: 'gearbox' without parameters will crash - specify '-h' to get started
testing.

Replace paster
summary = 'yada yada'
with the first line of the docstring of the Command class ... or override
get_description.

Note: All newlines in the docstring will be collapsed and mangle the long help
text.

Grouping of commands is not possible. Standard commands (for development) can't
be customized under the same name or hidden. (Like for paster, the conceptual
model also assumes that the sub-command naming is namespaced so commands from
other packages won't conflict.)

The usage help is fully automated from the declared options.

For all deprecated Commands, replace paster
hidden = True
with gearbox
deprecated = True

Note: config_file, takes_config_file, min_args and max_args are not available /
relevant.

The gearbox parser is customized by overriding get_parser - there is nothing
like paster update_parser.

Gearbox is using argparse instead of optparse ... but argparse add_argument is
mostly backwards compatible with optparse add_option.

Instead of overriding command or run as in paster, override take_action in
gearbox. The parsed arguments are passed to take_action, not available on the
command instance.

Paster BadCommand is not available and must be handled manually, terminating
with sys.exit(1).

There is no standard make-config command in gearbox.

Paster appinstall has been replaced by the somewhat different setup_app module
in gearbox. There is still no clean way to pass parameters to SetupAppCommand
and it relies on websetup and other apparently unnecessary complexity. Instead,
implement setup-db from scratch.


Minor change by Thomas De Schampheleire: add gearbox logging configuration.
Because we use logging.config.fileConfig(.inifile) during gearbox command
execution, the logging settings need to be correct and contain a block for
gearbox logging itself. Otherwise, errors in command processing are not even
visible and the command exits silently.

setup: bump WebOb to >=1.7

TurboGears will soon require WebOb>=1.7 and that cause an innocent test change.

Older WebOb versions also required older WebTest versions but the WebTest
dependency from Pylons will now automatically pick compatible versions.

setup: bump whoosh and bcrypt minimum versions

Some changes had sneaked in that crashed with the oldest package versions.

setup: really support latest Beaker version (1.8.1) after 5a5bab2b773c

Also, bump the minimum version to 1.7.0 to fix a failure with older Beaker versions;
8377f2bfb88f assumes https://github.com/bbangert/beaker/commit/5bf56a5cbcbe61454cfcfd6d80f05949c74177ad
so the expire value is converted to a string as the documentation suggests.

setup: bump babel version to <2.4

Babel versions up to 2.3.* provide lots of bugfixes and maintain
backward compatibility.

setup: don't constrain docutils version

Docutils maintains stable interface, and seems to be developed
at slow enough pace, so we can drop the upped version restriction.

hg: support Mercurial 4.1

There doesn't see to be any significant API changes.

celery: upgrade to Celery 3

Celery 3 is more than 2 and is the future and should be more stable than Celery 2.

There might be problems like
PRECONDITION_FAILED - cannot redeclare exchange 'celeryresults' in vhost 'kallithea' with different type, durable, internal or autodelete value
when celery is upgraded and it tries to upgrade existing tasks.

One way to avoid that problem:
- make sure celery has run to completion and there are no important pending tasks,
- delete the old vhost,
- upgrade celery,
- create new vhost
- set permissions for vhost
- profit

celeryd: move the celeryd paster command to the place where paster commands live

Also, partial implementations of other Celery functionality is dropped.

setup: limit WebTest version to <2.0

Running
pip install -e .
in a clean environment (or --upgrade), would cause:
"ContextualVersionConflict: (WebOb 1.1.1 (.../lib/python2.7/site-packages), Requirement.parse('WebOb>=1.2'), set(['WebTest']))"

setup.py required webob>=1.0.8,<=1.1.1 which pip resolved as WebOb==1.1.1.
setup.py required Pylons which requires WebTest>=1.3.1. Pip would thus install
the latest WebTest-2.0.23 ... which however has a WebOb>=1.2 dependency.
Pip only makes one pass at resolving dependencies and it would not resolve or
notice this inconsistency.

This problem came with dd676fdeda0f which moved pinning of WebTest 1.4.3 out of
setup.py.

Since Pylons require webtest anyway, there is no point in having it as a dev
requirement only - put it back in setup.py.

This might just be a temporary workaround until the WebOb version is updated.

Original patch by Dominik has been rewritten by Mads.

setup: support later waitress versions

Latest version (0.9.0) seems to work and has a nice-to-have fix for coverage.py
( https://github.com/Pylons/waitress/issues/107 ).

docs: drop empty Changelog in the documentation

Changelog.rst is empty and just refer to the Mercurial logs. More readable
release notes and other meta information is also available on the web site and
in announcement mails.

setup: drop ez_setup

It should no longer be necessary - everybody should have setuptools and use
pip. If not, just use ez to install setuptools and pip.

setup: move test dependencies to dev_requirements.txt to make them optional

Remove the need for having test tools on production systems. Installing test
dependencies is made an extra explicit step.

pip is the future, but doesn't have the same tests_require features as
setuptools kind of has.

I don't like this way of handling it without setup.py support and with explicit
naming of the ugly dev_requirements.txt ... but that seems to be the way to do
it.

db: use consistent key/index/constraint names when creating database

This simplifies future Alembic upgrades/downgrades.
http://alembic.zzzcomputing.com/en/latest/naming.html

A future changeset will update old databases to match this naming
convention.

Bump SQLAlchemy minimum version to 1.0, required for naming_convention
support.

db: add Alembic "migration environment"

Add in kallithea/alembic an unmodified Alembic "migration environment",
as generated by 'alembic init':

Copyright (C) 2009-2016 by Michael Bayer.
Alembic is a trademark of Michael Bayer.

Also update LICENSE, MANIFEST and setup metadata.

paster: add informational/dummy "upgrade-db" command

The command is placed kallithea.lib.dbmigrate:UpgradeDb (which was the
location of the old command, too), to ensure that "paster upgrade-db"
continues to work, even if Kallithea is installed in "editable" mode
(setup.py develop/pip install -e) and package metadata has not been
updated (and also to prevent issues caused by stale .pyc files).

db: remove SQLAlchemy Migrate support

The DbMigrateVersion table will be removed in a later changeset.

setup: bump dulwich dependency to 0.14.1

This version has been tested and reviewed and we believe all relevant issues
from the introduction of Python 3 support in 1.0 has been fixed.

Supporting older dulwich versions is not feasible/relevant.

We assume all future versions will be backwards compatible.

setup: use modern bcrypt implementation instead of unsupported old one

py-bcrypt has been deprecated by bcrypt, and is no longer developed
or supported.

bcrypt requires bytestrings instead of strings, use safe_str to ensure
they're encoded before they're passed to bcrypt. Also, use check_pw
to minimise the number of manual conversions and comparisons.

Installation of bcrypt will probably compile a C extension and require
libffi-dev.

setup: upgrade SQLAlchemy to 1.0.x series

Tested and reviewed with 1.0.13.

setup.py: enable pytest-sugar plugin

pytest-sugar enables a nice visual representation of test progress and
results.
Minimum version is set to 0.7.0 as it fixes a problem where skipped tests
would not be shown in the test summary.
The upper version 1.0.0 is arbitrarily chosen and could be increased as seen
fit.

setup.py: remove unused package_data

Per the setuptools documentation, package_data is ignored if
include_package_data is True, and indeed, we instead use MANIFEST.in.

hg: support Mercurial 3.7.x

Testing and upgrade notes doesn't indicate problems.

pytest migration: switch to pytest; remove nose support

Make pytest the default test runner and remove support for nose.
Tests can be run using:
- py.test
- python setup.py test

The pytest configuration needs to move from setup.cfg to pytest.ini to support
this - see https://github.com/pytest-dev/pytest/issues/567 and
https://bitbucket.org/pytest-dev/pytest-runner/issues/7/support-all-pytest-commands .

setup: support the Mercurial 3.6 series

No significant API changes and testing doesn't show any problems.

This upgrade enables use of bundle2 with generaldelta.

setup: reintroduce nosetests support that was removed in Pylons after 1.0.0

Use workaround from https://github.com/Pylons/pylons/issues/13 .

setup: drop unused simplejson dependency

c796b6bdd532 removed the use of it but didn't drop the dependency.

cleanup: consistently use 'except ... as ...:'

Use the Python 2.6+ syntax instead of the old confusing 'except ..., ...'
syntax.

mercurial: support Mercurial 3.5

test_archive failed without the hgcompat workaround.

pylons: support Pylons 1.0.2

Pylons 1.0.2 requires webob>=1.1.1 and Pylons 1.0.0 seemed to have problems
with later versions so stick the minimum version for now.

setup: bump mercurial requirement to 2.9

Commit 1cd9bdf1362d started using iterbranches() method of branchmap

setup: constrain dulwich to version 0.9.9

65c5e70a1d0c bumped the version to 0.10 but there seems to be some unicode
compatibility issues there. So for now, just stay on 0.9.9 which both works and
has the fix for CVE-2015-0838.

cleanup-repos: reintroduce paster command

The command was removed in e84116ab7d87 "Removed clenup repos command will be
part of new tools release".

This version is based on the last revision before that, c0bc961ec5d9 "back to
gpl license".

paster: add install-iis command to automate IIS handler generation

A new paster command, install-iis, is added that automates generating
the ISAPI-WSGI file that allows IIS to serve up Kallithea's WSGI application
using IIS' ISAPI filters.

The paster command's output also describes the final steps necessary to complete
IIS installation.

Use Conservancy's Kallithea Committee address as author_email.

the setup.py needs an author_email setting. Since this is a fork, the
author_email should be the individuals responsible for the fork. This
address reaches the Kallithea Committee at Conservancy.

First step in two-part process to rename directories to kallithea.
This first step is to change all references in the files where they refer
to the old directory name.

Imported some of the GPLv3'd changes from RhodeCode v2.2.5.

This imports changes between changesets 21af6c4eab3d and 6177597791c2 in
RhodeCode's original repository, including only changes to Python files and HTML.

RhodeCode clearly licensed its changes to these files under GPLv3
in their /LICENSE file, which states the following:
The Python code and integrated HTML are licensed under the GPLv3 license.

(See:
https://code.rhodecode.com/rhodecode/files/v2.2.5/LICENSE
or
http://web.archive.org/web/20140512193334/https://code.rhodecode.com/rhodecode/files/f3b123159901f15426d18e3dc395e8369f70ebe0/LICENSE
for an online copy of that LICENSE file)

Conservancy reviewed these changes and confirmed that they can be licensed as
a whole to the Kallithea project under GPLv3-only.

While some of the contents committed herein are clearly licensed
GPLv3-or-later, on the whole we must assume the are GPLv3-only, since the
statement above from RhodeCode indicates that they intend GPLv3-only as their
license, per GPLv3§14 and other relevant sections of GPLv3.

Added ini config maker with dynamic variable replacements.
Used internally for creating dynamic configs in automated enviroments.

Later this will replace old make-config from paster

moved make-index command to paster_commands module

- optimized imports and code

Implemented simple gist functionality ref #530.

- creation of public/private gists with given lifetime
- rhodecode-gist CLI for quick gist creation

unified RhodeCode paster commands
- moved them to commont paster_commands package
- re-use sqlalchemy session initializaiton
- some docs updates