setup: bump Sphinx version

Sphinx 3.0.4 doesn't version its dependencies correctly, and it fails with latest jinja 3.1.2:
ImportError: cannot import name 'environmentfilter' from 'jinja2' (.../site-packages/jinja2/__init__.py)

Latest Sphinx version supports latest Jinja version and seems to render the
documentation correctly.

setup: support Pytest 6.2 for Python 3.10 support

https://github.com/pytest-dev/pytest/issues/8539

isort: upgrade to 5.1.2

The changes to non-top-level imports seem nice.

setup: bump version numbers to current max

Based on 'pip freeze' before/after:

sed -i -e '/>=[^#]*$/s/, < [0-9.]\+//g' -e 's, == , >= ,g' setup.py dev_requirements.txt

Markdown 3.3 is a few months old could be bumped too ... but it change test
output due to
https://python-markdown.github.io/change_log/release-3.2/#codehilite-now-always-wraps-with-ltcodegt-tags
and can wait.

This is landing early in the development phase and will thus get good test
coverage.

cleanup: run pyflakes as a part of scripts/run-all-cleanup

pyflakes has no usable configuration options, so create a small wrapper script.
Instead of having two wrapper scripts (with one being almost nothing and the
other containing configuration), just keep it simple and use one combined.

setup: bump version numbers to current max

This will most likely give best py3 support, and we need thorough testing
anyway.

Based on 'pip freeze' before/after:

sed -i '/>=[^#]*$/s/, < [0-9.]\+//g' setup.py dev_requirements.txt

setup: bump webtest minimum version to 2.0.6 to get py3 stdlib support

/usr/lib64/python3.7/http/cookiejar.py:723: in is_third_party
if not domain_match(req_host, reach(request.origin_req_host)):
E AttributeError: '_RequestCookieAdapter' object has no attribute 'origin_req_host'

setup: set explicit minimum version for all dependencies

Kind of arbitrarily chosen, but it seems like reasonable not-too-recent
versions that still provide a consistent baseline.

dev_requirements.txt: bump minimum pytest versions to a working set

Not all combinations of pytest-related packages seem to work.
Trial-and-error lead to these combinations of minimum versions.

Add a minimum bound to pytest-benchmark and pytest-localserver to help pip
in figuring out a suitable combination with pytest, and also to allow
testing minimum combinations in the future.

Drop pytest-runner, which should not be needed unless when running tests
directly from setup.py.

setup: bump some dev_requirements.txt versions

Upgrade notes for these libraries have not been investigated thoroughly, but
testing seems to show that it works.

setup: bump py.test version

Note: latest pytest versions has an undeclared dependency / incompatibility
with pytest-benchmark, which thus has to be bumped at the same time:

INTERNALERROR> Traceback (most recent call last):
INTERNALERROR> File "data/env/lib/python2.7/site-packages/_pytest/main.py", line 202, in wrap_session
INTERNALERROR> config._do_configure()
INTERNALERROR> File "data/env/lib/python2.7/site-packages/_pytest/config/__init__.py", line 671, in _do_configure
INTERNALERROR> self.hook.pytest_configure.call_historic(kwargs=dict(config=self))
INTERNALERROR> File "data/env/lib/python2.7/site-packages/pluggy/hooks.py", line 311, in call_historic
INTERNALERROR> res = self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR> File "data/env/lib/python2.7/site-packages/pluggy/manager.py", line 87, in _hookexec
INTERNALERROR> return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR> File "data/env/lib/python2.7/site-packages/pluggy/manager.py", line 81, in <lambda>
INTERNALERROR> firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
INTERNALERROR> File "data/env/lib/python2.7/site-packages/pluggy/callers.py", line 208, in _multicall
INTERNALERROR> return outcome.get_result()
INTERNALERROR> File "data/env/lib/python2.7/site-packages/pluggy/callers.py", line 81, in get_result
INTERNALERROR> _reraise(*ex) # noqa
INTERNALERROR> File "data/env/lib/python2.7/site-packages/pluggy/callers.py", line 187, in _multicall
INTERNALERROR> res = hook_impl.function(*args)
INTERNALERROR> File "data/env/lib/python2.7/site-packages/pytest_benchmark/plugin.py", line 427, in pytest_configure
INTERNALERROR> bs = config._benchmarksession = BenchmarkSession(config)
INTERNALERROR> File "data/env/lib/python2.7/site-packages/pytest_benchmark/session.py", line 31, in __init__
INTERNALERROR> self.logger = Logger(self.verbose, config)
INTERNALERROR> File "data/env/lib/python2.7/site-packages/pytest_benchmark/logger.py", line 15, in __init__
INTERNALERROR> self.pytest_warn = config.warn
INTERNALERROR> AttributeError: 'Config' object has no attribute 'warn'


The new py.test will show deprecation warnings from other libraries and how we use them:

kallithea/tests/__init__.py:28
kallithea/tests/__init__.py:28: PytestAssertRewriteWarning: Module already imported so cannot be rewritten: kallithea.tests
pytest.register_assert_rewrite('kallithea.tests')

data/env/lib/python2.7/site-packages/pkg_resources/__init__.py:1145
kallithea/tests/api/test_api_git.py::TestGitApi::test_api_wrong_key
data/env/lib/python2.7/site-packages/pkg_resources/__init__.py:1145: DeprecationWarning: Use of .. or absolute path in a resource path is not allowed and will raise exceptions in a future release.
self, resource_name

<string>:2
<string>:2: SADeprecationWarning: Mapper.order_by is deprecated.Use Query.order_by() in order to affect the ordering of ORM result sets.

data/env/lib/python2.7/site-packages/tg/configuration/app_config.py:473
data/env/lib/python2.7/site-packages/tg/configuration/app_config.py:473: DeprecationWarning: direct usage of error tracing options has been deprecated, please specify them as trace_errors.option_name instad of directly setting option_name. EXAMPLE: trace_errors.error_email
"setting option_name. EXAMPLE: trace_errors.error_email", DeprecationWarning)

data/env/lib/python2.7/site-packages/tg/wsgiapp.py:68
data/env/lib/python2.7/site-packages/tg/wsgiapp.py:68: DeprecationWarning: Session options should start with session. instead of baker.session.
app_wrapper = wrapper(self.wrapped_dispatch, self.config)

...
kallithea/model/validators.py:279: DeprecationWarning: validate_python is deprecated; use _validate_python instead
class _validator(formencode.validators.FancyValidator):

...
kallithea/model/validators.py:793: DeprecationWarning: _to_python is deprecated; use _convert_to_python instead
class _validator(formencode.validators.FancyValidator):

...

kallithea/tests/other/test_doctest.py::test_doctests[kallithea.lib.markup_renderer]
data/env/lib/python2.7/site-packages/markdown/__init__.py:259: DeprecationWarning: Using short names for Markdown's builtin extensions is deprecated. Use the full path to the extension with Python's dot notation (eg: "markdown.extensions.codehilite" instead of "codehilite"). The current behavior will raise an error in version 2.7. See the Release Notes for Python-Markdown version 2.6 for more info.
DeprecationWarning)

kallithea/tests/other/test_doctest.py::test_doctests[kallithea.lib.markup_renderer]
data/env/lib/python2.7/site-packages/markdown/__init__.py:259: DeprecationWarning: Using short names for Markdown's builtin extensions is deprecated. Use the full path to the extension with Python's dot notation (eg: "markdown.extensions.extra" instead of "extra"). The current behavior will raise an error in version 2.7. See the Release Notes for Python-Markdown version 2.6 for more info.
DeprecationWarning)

dependencies: remove duplicate dependencies in dev_requirements.txt

Babel and WebOb were present both in setup.py and dev_requirements.txt.
When explicitly setting minimum dependencies in setup.py and running:

pip install --upgrade -e . -r dev_requirements.txt python-ldap python-pam

this duplication replaced useful diagnostic message from pip with less
useful ones.
For example, the following message was displayed when the Babel dependency
duplication is present:

kallithea 0.4.0rc1 has requirement Babel<2.7,==0.9.6, but you'll have babel 2.6.0 which is incompatible.

When removing the duplication in dev_requirements.txt, this becomes:

sphinx 1.7.9 has requirement babel!=2.0,>=1.3, but you'll have babel 0.9.6 which is incompatible.

which makes it clear that to solve this problem, we need to bump the minimum
dependency for Babel in setup.py from 0.9.6 to 1.3.

setup: bump all upper pip dependency versions to minor updates of what currently is available and testable on pypi

Based on manual editing after:
pip freeze | sed -n 's/==/ /gp' | while read p v; do sed -i -e "/\<$p[ \"]/s/\(\",\|$\)/, < $v\1/gi" setup.py dev_requirements.txt; done

These updates have been tested with automated tests and some amount of manual
testing. Remaining problems will hopefully be caught by additional testing
before the branch is declared stable.

setup: normalize casing of dependencies and formatting of constraints

The package name casing might seem random, but it is more correct than our own random casing.

These changes mainly come from:
pip freeze | sed -n '/==/s/==.*//gp' | while read a; do sed -i "s/$a/$a/i" setup.py dev_requirements.txt ; done
sed -i -e 's/>=/ >= /' -e 's/ >=/ >=/' -e 's/>= />= /g' -e 's/,</, </' -e 's/</< /' -e 's/< /< /g' -e 's/< =/<= /g' setup.py dev_requirements.txt

tests: pytest-catchlog is now integrated in pytest 3.3+

See: https://docs.pytest.org/en/latest/changelog.html?highlight=catchlog#id56

This removes the deprecation warning at the end of the test run:
pytest-catchlog plugin has been merged into the core, please remove it
from your requirements.

tests: use pytest-benchmark for performance tests

pytest-benchmark is a benchmarking suite for individual functions. It
presents itself as a pytest fixture, i.e. you accept it as a test function
argument, and then can use it to benchmark a function.

Output is presented in the following form:

----------------------------------------- benchmark: 1 tests ----------------------------------------
Name (time in s) Min Max Mean StdDev Median IQR Outliers(*) Rounds Iterations
-----------------------------------------------------------------------------------------------------
test_graphmod_git 3.1122 3.1682 3.1295 0.0236 3.1196 0.0313 1;0 5 1
-----------------------------------------------------------------------------------------------------

tests: introduce webserver fixture

To be used for tests that actually access a running web server - especially to
make the manual vcs tests less manual.

backend: replace Pylons with TurboGears2

Replace the no-longer-supported Pylons application framework by TurboGears2
which is largely compatible/similar to Pylons.
Some interesting history is described at:
https://en.wikipedia.org/wiki/TurboGears

Changes by Dominik Ruf:
- fix sql config in test.ini

Changes by Thomas De Schampheleire:
- set-up of test suite
- tests: 'fix' repo archival test failure
Between Pylons and TurboGears2, there seems to be a small difference in the
headers sent for repository archive files, related to character encoding.
It is assumed that this difference is not important, and that the test
should just align with reality.
- remove need to import helpers/app_globals in lib
TurboGears2 by default expects helpers and app_globals to be available
in lib. For this reason kallithea/lib/__init__.py was originally changed
to include those files. However, this triggered several types of
circular import problems. If module A imported something from lib (e.g.
lib.annotate), and lib.helpers imported (possibly indirectly) module A,
then there was a circular import. Fix this by overruling the relevant
method of tg AppConfig, which is also hinted in the TurboGears2 code.
Hereby, the include of something from lib does not automatically import
helpers, greatly reducing the chances of circular import problems.
- make sure HTTP error '400' uses the custom error pages
TurboGears2 does not by default handle HTTP status code
'400 (Bad Request)' via the custom error page handling, causing a
standard non-styled error page.
- disable transaction manager
Kallithea currently handles its own transactions and does not need the
TurboGears2 transaction manager. However, TurboGears2 tries to enable it
by default and fails, throwing an error during application initialization.
The error itself seemed to be harmless for normal application functioning,
but was nevertheless confusing.
- add backlash as required dependency: backlash is meant as the WebError
replacement in TurboGears2 (originally WebError is part of Pylons). When
debug==true, it provides an interactive debugger in the browser. When
debug==false, backlash is necessary to show backtraces on the console.
- misc fixes

setup: bump WebOb to >=1.7

TurboGears will soon require WebOb>=1.7 and that cause an innocent test change.

Older WebOb versions also required older WebTest versions but the WebTest
dependency from Pylons will now automatically pick compatible versions.

test: bump pytest minimum version

Recent test case changes broke compatibility with pytest < 3.0.

FYI, the ~= version spec is a shorthand (PEP 440) used here to select
"3.0 or any later 3.* release". We could probably use this more often.

tests: drop upper bound to pytest-sugar version

With some other dev-requirements having no version specification, there is
no real reason to keep an upper bound to pytest-sugar either.
The lower bound is still relevant as that fixed a bug.

tests: bump pytest version requirement to allow 3.x

pytest is an active project, since our previous requirement setting there
have been many releases, with new features and bug fixes.

One of the improvements in pytest-3.0 is that 'yield_fixture' (a fixture
that can embed cleanup code after a yield) is no longer different than
'fixture', i.e. a standard fixture supports the yield cleanup mechanism. We
will switch to use that when we make 3.0 mandatory.

setup: limit WebTest version to <2.0

Running
pip install -e .
in a clean environment (or --upgrade), would cause:
"ContextualVersionConflict: (WebOb 1.1.1 (.../lib/python2.7/site-packages), Requirement.parse('WebOb>=1.2'), set(['WebTest']))"

setup.py required webob>=1.0.8,<=1.1.1 which pip resolved as WebOb==1.1.1.
setup.py required Pylons which requires WebTest>=1.3.1. Pip would thus install
the latest WebTest-2.0.23 ... which however has a WebOb>=1.2 dependency.
Pip only makes one pass at resolving dependencies and it would not resolve or
notice this inconsistency.

This problem came with dd676fdeda0f which moved pinning of WebTest 1.4.3 out of
setup.py.

Since Pylons require webtest anyway, there is no point in having it as a dev
requirement only - put it back in setup.py.

This might just be a temporary workaround until the WebOb version is updated.

Original patch by Dominik has been rewritten by Mads.

setup: move test dependencies to dev_requirements.txt to make them optional

Remove the need for having test tools on production systems. Installing test
dependencies is made an extra explicit step.

pip is the future, but doesn't have the same tests_require features as
setuptools kind of has.

I don't like this way of handling it without setup.py support and with explicit
naming of the ugly dev_requirements.txt ... but that seems to be the way to do
it.