ini: set translate_forwarded_server = False when using paste prefix middleware

Paste Deploy PrefixMiddleware will be default trust HTTP headers that a proxy
server might have set. That can be a problem if there is no proxy server or if
the proxy just pass these headers through.

Change the configuration example to disable this translation.

auth: only use X- headers instead of wsgi.url_scheme if explicitly told so in url_scheme_header - drop https_fixup setting

Before, several X- headers would be trusted to overrule the actual connection
protocol (http or https) seen by the Kallithea WSGI server. That was mainly
when https_fixup were set, but it incorrectly also kicked in if https_fixup or
use_htsts were configured. The ambiguity of which headers were used also made
it less reliable. The proxy server not only had to be configured to set one of
the headers correctly, it also had to make sure other headers were not passed
on from the client. It would thus in some cases be possible for clients to fake
the connection scheme, and thus potentially be possible to bypass restrictions
configured in Kallithea.

Fixed by making it configurable which WSGI environment variable to use for the
protocol. Users can configure url_scheme_header to for example
HTTP_X_FORWARDED_PROTO instead of using the default wsgi.url_scheme .

This change is a bit similar to what is going on in the https_fixup middleware,
but is doing a bit more of what for example is happening in similar code in
werkzeug/middleware/proxy_fix.py .

The semantics of the old https_fixup were unsafe, so it has been dropped.
Admins that are upgrading must change their configuration to use the new
url_scheme_header option.

auth: only use X- headers instead of REMOTE_ADDR if explicitly told so in remote_addr_header

Before, X-Forwarded-For (and others) headers would *always* be trusted blindly,
also in setups without a proxy server. It would thus in some cases be
possible for users to fake their IP, and thus potentially be possible to bypass
IP restrictions configured in Kallithea.

Fixed by making it configurable which WSGI environment variable to use for the
remote address. Users can configure remote_addr_header to for example
HTTP_X_FORWARDED_FOR instead of using the default REMOTE_ADDR.

This change is a bit similar to what is going on in the https_fixup middleware,
but is doing a bit more of what for example is happening in similar code in
werkzeug/middleware/proxy_fix.py .

celery: drop tracking of task_id - we use ignore_result=True and will never get anything back

There is thus no need for configuration of celery.result_backend .

The alternative would be to fix it. That could give better error reporting from
failing repo creations, but would require quite a bit of additional changes
before it actually works reliably.

celery: drop traces of support for task_always_eager

It was broken - retrieval of results didn't work.

Also, task_always_eager still require a working celery.broker_url even though
it is unused. It is thus much better to just set use_celery=false .

celery: change template to use celery 4 setting name celery.result_backend

Missed in e1d4a0d8520f ...

middleware: drop sentry support

Our sentry support didn't seem to work with py3. It doesn't seem to ever have
been tested on py3 - it failed already when installing.

It seems like we used the 'legacy client':
https://docs.sentry.io/clients/python/
and nowadays one is expected to use:
https://docs.sentry.io/platforms/python/
which has integrations for Pyramid and other frameworks (but not TG2). There
are Celery and SQLAlchemy integrations too. See also the migration guide:
https://docs.sentry.io/platforms/python/migration/ .

Also, there is not much essential code in the old sentry support, and it seems
like it would be easier to reimplement from scratch. There is thus not much
lost by dropping it.

ini: move archive_cache_dir to the data directory

Give less data locations to care about.

mysql: bump sqlalchemy.url MariaDB/MySQL charset to to 'utf8mb4' to get full UTF-8 support

The change in 210e76d69b62 only changed character_set_database, as shown by output after:

--- a/kallithea/model/base.py
+++ b/kallithea/model/base.py
@@ -46,3 +46,8 @@ def init_model(engine):
engine_str = obfuscate_url_pw(str(engine.url))
log.info("initializing db for %s", engine_str)
meta.Base.metadata.bind = engine
+
+ meta.Session.configure(bind=engine)
+ for a, b in meta.Session().execute('''show variables''').fetchall():
+ if 'character_set_' in a:
+ print(a, repr(b))

Before, with charset=utf8, the utf8mb3 charset was used all the way through the stack:

[kallithea.model.base] initializing db for mysql://kallithea-test:XXXXX@localhost/kallithea-test?charset=utf8
character_set_client 'utf8'
character_set_connection 'utf8'
character_set_database 'utf8mb4'
character_set_filesystem 'binary'
character_set_results 'utf8'
character_set_server 'latin1'
character_set_system 'utf8'

With explicit charset=utf8mb4:

[kallithea.model.base] initializing db for mysql://kallithea-test:XXXXX@localhost/kallithea-test?charset=utf8mb4
character_set_client 'utf8mb4'
character_set_connection 'utf8mb4'
character_set_database 'utf8mb4'
character_set_filesystem 'binary'
character_set_results 'utf8mb4'
character_set_server 'latin1'
character_set_system 'utf8'

docs: augment setup description with more details of http server and database

Dive into more details than in overview.rst .

Databases should perhaps have more mentioning in overview.rst , but there is
nothing but details and thus not much to say in the overview ...

ini: change .ini template to include sqlalchemy.url examples for all database types

Rely less on users creating their initial .ini "correctly" with the right
database choice.

cleanup: drop last traces of kallithea-config

The kallithea-config sources were removed in commit
213085032127e941a3bd93d0e510828a9d87bf32 but an entry point was still
created by setup.py.
Moreover, the ini file still referenced this, instead of kallithea-cli
(config-create).

ini: explain how % must be escaped as %% (Issue #375)

Gearbox / paste would fail with:
configparser.InterpolationSyntaxError: '%' must be followed by '%' or '('

The special handling of % is standard configparser behaviour, unless explicitly
fetching raw settings.

ini: tweak template configuration for TG's handling of application errors

Mention what was learned while debugging request processing at WSGI/TG level
and the complexity of the TG stack gets in the way.

inifile: new implementation of setting updates to optimize reuse of comments and append location

Code comments should explain the algorithm, and test coverage suggest it works
correctly.

db: drop SA caching_query and FromCache, and thus sql_cache_short beaker cache

It is not a good idea to have dead ORM objects. If we want caching, we should
do it explicit.

It is unknown how much this cache helps, but we can profile and introduce
better caching of simple data where relevant.

celery: upgrade to Celery 4

Celery 3 doesn't support Python 3.7 or later. This upgrade is thus essential
for full Python 3 support. But note that
https://docs.celeryproject.org/en/4.4.0/faq.html#does-celery-support-windows
says "No".

The names of config settings changed in Celery 3 to 4, as described on
https://docs.celeryproject.org/en/3.0/whatsnew-4.0.html#lowercase-setting-names .

Celery 4 config settings can now be specified in Kallithea .ini files by prefixing
with `celery.` - for example as `celery.broker_url`.

Remain backwards compatible for the usual settings, and map old names to the
new names.

celery: drop config settings not supported in Celery 4

CELERY_SEND_TASK_ERROR_EMAILS has been removed, according to
https://docs.celeryproject.org/en/3.0/whatsnew-4.0.html#features-removed-for-simplicity :
"Tasks no longer sends error emails. This also removes support for
app.mail_admins, and any functionality related to sending emails."

The "ADMINS" prefix might be a left-over from the long removed
celery.conf.ADMINS - last seen on
https://docs.celeryproject.org/en/2.1-archived/reference/celery.conf.html#celery.conf.ADMINS .

The "ROUTES" prefix might refer to "CELERY_ROUTES" ... but it doesn't take a
simple string list anyway, so there is no point in treating it as a list value.

celery: change ini template to use sqlite for results

'amqp://' might be good - also for results, but seems to need additional
non-trivial setup. And according to
https://docs.celeryproject.org/en/3.0/whatsnew-4.0.html#features-removed-for-lack-of-funding
it is deprecated.

Kallithea only uses Celery results when repos are created or forked and user
browsers are reloading pages to poll for completion. amqp seems like
unnecessary complexity for that use case.

Sqlite does however seem like a minimal but fine solution for the Kallithea use
case in most setups.

celery: set default config values in code and remove them from the generated .ini

It is hard to imagine any reason the user should change celery.imports . And if
it ever should change, we want it controlled in code - not left stale in user
controlled config files.

Everybody sould just use .json and there is no reason anybody should specify
that in the .ini ... and it will be the default in Celery 4.

celery: update ini template to use json for tasks and results

Json seems to be more reliable than pickle, and seems to be the future.

ini: drop deprecated CELERY_RESULT_DBURI

https://docs.celeryproject.org/en/3.1/configuration.html#celery-result-dburi says:

This setting is no longer used as it’s now possible to specify the database
URL directly in the CELERY_RESULT_BACKEND setting.

ini: fix spelling of the CELERY_TASK_SERIALIZER

It has "always" been wrong, and we have thus been using the default of "pickle"
- not "json" as we thought.

I doubt this change has any immediate visible impact. I guess it only means
that it will use json for results instead of pickle. That might be more stable
and debuggable.

Note: celery_config will uppercase the config settings and replace '.' with
'_', so 'celery.result.serializer' turns into 'CELERY_TASK_SERIALIZER'.

CELERY_TASK_SERIALIZER is described on
https://docs.celeryproject.org/en/3.1/configuration.html#celery-result-serializer
https://docs.celeryproject.org/en/3.1/userguide/calling.html#serializers

mails: make error reporting by mail work with secure mail servers

Even with Kallithea mails working, TurboGears / backlash error reporting would
fail like:

Error while reporting exception with <backlash.tracing.reporters.mail.EmailReporter object at 0x7f8f986f8710>
Traceback (most recent call last):
File ".../env/lib/python3.7/site-packages/backlash/tracing/reporters/mail.py", line 49, in report
result = server.sendmail(self.from_address, self.error_email, msg.as_string())
File "/usr/lib64/python3.7/smtplib.py", line 867, in sendmail
raise SMTPSenderRefused(code, resp, from_addr)
smtplib.SMTPSenderRefused: (530, b'5.7.0 Must issue a STARTTLS command first.', 'kallithea@example.com')

Extend the .ini template to forward all the mail settings described on
https://turbogears.readthedocs.io/en/latest/reference/config-options.html#error-reporting

py3: switch to use Python 3 interpreter, temporarily leaving many things very broken until they have been migrated/fixed in a reviewable way

Bump Mercurial minimum version to 5.2 - the first version that claim stable py3
support.

i18n: make sure 'en' in Accept-Language is recognized as having 100% coverage - i18n.lang is for source language

https://github.com/cdent/paste/blob/3.2.3/paste/wsgiwrappers.py#L89 describes
how paste solved the problem. TG solves the same problem (with less
explanation) in
https://github.com/TurboGears/tg2/blob/tg2.4.2/tg/request_local.py#L36 with
fallback language specified in i18n.lang .

Thus, clarify the use of i18n.lang (refining f2f7a8c1281e and 8931078f70db) and
set 'en' as default value on app startup.

TurboGears requires an (empty) translation for the source language which is
default for i18n.lang . The empty .mo for en is created as the 4 magic .mo
bytes followed by lengths of 0:
printf '\x95\x04\x12\xde\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0' > kallithea/i18n/en/LC_MESSAGES/kallithea.mo

Based on report and analysis by Wolfgang Scherer. Also discussed upstream on
https://github.com/TurboGears/tg2/pull/115 .

kallithea-cli: use null handler to mute all console logging for ssh-serve

Augment logger_root to use the null handler when running ssh-serve. Set the log
level to CRITICAL to filter early and minimize the amount of logging sent to
the null handler.

It is however recommended to configure the system to use some logging facility
that can handle logging for SSH access from many different processes.

ssh: set a valid locale in the ssh-serve process

In the SSH client configuration, the setting 'SendEnv' could contain variables
like 'LANG', 'LC_ALL', and others. This causes these environment variables
(with their values at the client-side) to be set in the server. However, not
every locale setting valid in the client, is also valid on the server.

This could lead to the error:
'locale.Error: unsupported locale setting'
when 'from mercurial import archival, merge as hg_merge, patch, ui' is
called.

Fix this problem by providing an ini setting 'ssh_locale' that the user can
set correctly, and which will be used to set LC_ALL and LANGUAGE in the
'kallithea-cli ssh-serve' process.

If an environment variable LC_ALL is set, it takes precedence over all other
'LC_xxx' variables, as well as over LANG. So, setting LC_ALL ensures that no
user setting of 'LC_xxx' or 'LANG' could influence ssh-serve badly.

There is one environment variable that might overrule LC_ALL, specifically
for showing messages: 'LANGUAGE'. GNU gettext lets it take precedence over
LC_ALL [1]:
"GNU gettext gives preference to LANGUAGE over LC_ALL and LANG for the
purpose of message handling"

So, also set LANGUAGE to the same value as we set LC_ALL to.


The principle of setting a specific locale in the server process to fix this
error, was first proposed by Dominik Ruf.

[1] https://www.gnu.org/software/gettext/manual/html_node/The-LANGUAGE-variable.html#The-LANGUAGE-variable

ssh: introduce 'kallithea-cli ssh-update-authorized-keys' command for updating authorized_keys file

Based on work by Ilya Beda <ir4y.ix@gmail.com> on
https://bitbucket.org/ir4y/rhodecode/commits/branch/ssh_server_support ,
incorporating gearbox support by Anton Schur <tonich.sh@gmail.com> and also
heavily modified by Mads Kiilerich.

This commit also incorporates a fix for Windows by Dominik Ruf,
and better handling of the case where the parent dir of 'authorized_keys'
does not exist or is not writable, by Bradley M. Kuhn <bkuhn@ebb.org>.

ssh: introduce ini setting 'ssh_enabled', disabled by default

Administrators should control the use of SSH and may want to disable SSH
access, temporarily or permanently.

An explicit setting ssh_enabled is better than e.g. checking for a valid
ssh_authorized_keys setting, to allow such trivial temporary disabling.

To keep the controllers simple, introduce a decorator IfSshEnabled instead
of repeating the same config checks in every method.

ini: create separate log handlers for color and null, and add comments hinting how they can be used

Let development.ini use color for the root logger as before. The special effect
of color_sql was not visible with the default sqlalchemy log level of WARN, so
just use color there as well.

config: rename .ini settings to avoid beaker-session deprecation warnings

Gets rid of:

data/env/lib/python2.7/site-packages/tg/wsgiapp.py:68
data/env/lib/python2.7/site-packages/tg/wsgiapp.py:68: DeprecationWarning: Session options should start with session. instead of baker.session.
app_wrapper = wrapper(self.wrapped_dispatch, self.config)

Old .ini files will still work, but one day upstream will drop the deprecated functionality ...

config: change default .ini to always include trace_errors settings and thus avoid deprecation warnings

Gets rid of:

data/env/lib/python2.7/site-packages/tg/configuration/app_config.py:473
data/env/lib/python2.7/site-packages/tg/configuration/app_config.py:473: DeprecationWarning: direct usage of error tracing options has been deprecated, please specify them as trace_errors.option_name instad of directly setting option_name. EXAMPLE: trace_errors.error_email
"setting option_name. EXAMPLE: trace_errors.error_email", DeprecationWarning)

auth: drop api_access_controllers_whitelist and give API key auth same access as other kinds of auth

All authentication methods are created equal. There is no point in
discriminating api key authentication and limit it to few APIs.

locking: drop the pull-to-lock / push-to-unlock functionality

The feature is not worth the maintenance cost. The locking is too coarse and
unflexible with insufficient UI and UX. The implementation is also quite
invasive in tricky areas of the code, and thus high maintenance.
Dropping this will enable other cleanup ... or at least make it easier.

hooks: make the Python interpreter for Git hooks configurable as 'git_hook_interpreter' (Issue #333)

Commit 5e501b6ee639 introduced the use of 'sys.executable' as interpreter
for git hooks instead of 'python2' with the following argument:

"Windows doesn't necessarily have "python2" available in $PATH, but we
still want to make sure we don't end up invoking a python3. Using the
absolute path seems more safe."

But, sys.executable does not necessarily point to Python. When Kallithea is
started under uWSGI, sys.executable points to the uwsgi executable. As a
result, the interpreter encoded in the git hooks on the server repositories
would be:

#!/path/to/uwsgi

And pushing to such repo would result in following client errors:

$ git push
Password for 'http://user@localhost:5050&apos;:
Enumerating objects: 3, done.
Counting objects: 100% (3/3), done.
Writing objects: 100% (3/3), 241 bytes | 241.00 KiB/s, done.
Total 3 (delta 0), reused 0 (delta 0)
remote: unable to load configuration from hooks/pre-receive
To http://localhost:5050/gitrepo-new
! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'http://user@localhost:5050/gitrepo-new&apos;

Fix this problem by introducing a configuration setting 'git_hook_interpreter'
that allow administrators to specify which Python interpreter to use.

A subsequent commit will cause its value to be filled in automatically when
generating a new ini file, but an administrator can always override it.

ini: fix typo i18n.enable -> i18n.enabled (Issue #339)

Commit f2f7a8c1281e changed the i18n-related ini settings to match
TurboGears2. Even though the commit message correctly refers to
'i18n.enabled', the code incorrectly used 'i18n.enable'.

hg: set encoding to utf-8 by default to always show unicode characters correctly

Unicode characters would be shown as '?' if Kallithea was launched in a LANG=C
environment (or similar).

The problem could be solved by setting HGENCODING before launching Kallithea or
before importing Mercurial. These are often not good solutions.

Instead, introduce a hgencoding config setting that triggers monkey patching of
Mercurial.

ini: drop setting initial_repo_scan

The ini setting 'initial_repo_scan' caused a repository scan on each startup
of Kallithea. The accompanying comment already warns that the feature should
be disabled after the first run to improve startup time.

Now that setup-db is performing the initial repository scan, and considering
that administrators can always request a new scan using 'gearbox repo-scan'
or via the web interface, the 'initial_repo_scan' feature is no longer
considered useful and is removed with this commit.

issues: support generic regex replacements in issue_url and issue_prefix

Issue reference linking is pretty limited:
- the issue_url is a literal with only three special tokens {id},
{repo} and {repo_name}. There is no way to let the URL be dependent on
other elements of the input issue reference.
- The value for {id} is somewhat oddly determined by the concatenation of
all parenthesized groups in the issue_pat regular expression
- the link text of the resulting link is limited to the contents of the
literal issue_prefix with the determined {id}. It is not possible to
retain the input issue reference verbatim, nor to let the link text be
dependent on other elements of the input issue reference.

This commit makes the issue reference linking more flexible:

- issue_prefix is replaced by the more generic issue_sub(stitution), which
is a string that may contain backreferences to regex groups specified in
issue_pat. This string, with backreferences resolved, is used as the
link text of urlified issue references.
- if issue_sub is empty, the entire text matched by issue_pat is used as
the link text.
- like issue_sub, also issue_url can contain backreferences to regex groups.
- {id} is no longer treated as a special token, as it can be solved by
generic backreferences ('\g<id>' assuming issue pattern contains something
like '(P<id>\d+)'. {repo} and {repo_name} are still supported, because
their value is provided externally and not normally part of the
issue pattern.

Documentation and ini file template is updated as well.

ini: set default log level to WARN

DEBUG and INFO are not good choices for the default log levels.
With DEBUG and INFO most of the time you can't see the wood for the trees.
It is too easy to overlook critical errors or warnings if the log levels
DEBUG and INFO are enabled.

issues: simplify example in ini file

The example setting from the default ini file is:
issue_pat = (?:\s*#)(\d+)
issue_server_link = https://issues.example.com/{repo}/issue/{id}
issue_prefix = #

in which the clause '(?:\s*#)' is overly complex:
1. '\s*' means optional whitespace, which means there can be whitespace or
no whitespace, which means that you could equally write:
(?:#)(\d+)
(If any leading whitespace was eaten by the regexp, 32e1e0745d3c would take
care to restore a part of it.)
2. '(?:xxx)' means a non-capturing set of parentheses. In this case this is
equal to just mentioning 'xxx'. So the simplified pattern becomes:
#(\d+)

If instead of _optional_ whitespace, _mandatory_ whitespace was intended,
then the pattern should be different. But this would also mean that
patterns like PR#123 would not be matched anymore.

ini: error email settings are comma-separated under backlash

Document how it email_to works under backlash.

When specifying multiple email addresses to receive error emails, they
should be comma-separated under backlash, rather than space-separated
previously. Extra spaces should not be a problem but is not
"supported"/"recommended".

config: tweak template http_server conditionals - don't leave an empty section when using UWSGI

The web servers run by 'gearbox serve' share the same basic configuration:

[server:main]
host = ...
port = ...
use = egg:...
...

UWSGI doesn't use gearbox and doesn't use a [server:main] section but will read
it's own section:

[uwsgi]
http = ...:...
...

Before, 'make-config my.ini http_server=uwsgi' would create an empty
[server:main] section, and only after 6a5fb5070765 it stopped putting unused
host/port information there.

Now, let everything but UWSGI share the [server:main] template section and put
host/port first. Everything UWSGI is handled in its own conditional template
section.

ini: drop insertion of header comments in generated ini files

The header comments were kind of redundant and could easily get out of sync.
Also, we are moving towards just generating files and don't need this and don't
want to maintain it.

ini: simplify preservation of old settings - just keep the old line as comment

The thing about skipping mako function references is not important.

config: clarify that we only recommend and support single threaded operation

Sad, but true. Especially because we reuse Repository instances between
threads.

i18n: use TG2 compatible settings names i18n.lang and i18n.enabled

Following changes are made:
- Remove list of available translations: severely outdated and hard to
maintain.
- Replace lang with i18n.lang, reasons:
- Follow TurboGears2 conventions
- Align with 'i18n.enabled' which cannot be shortened
- Scoped settings provide grouping and are more clear
The old 'lang' is still recognized though, so no upgrade issue.
- Introduce/document i18n.enabled

Comments are intentionally kept concise in favor of the documentation.

logging: don't change the log level in the handler config

I find it confusing that even though the log level for sqlalchemy.engine was
INFO, because the log level of handler_console_sql was WARN, there were no
INFO output from sqlalchemy.engine.
Therefore remove the level setting for the handlers in the default settings.
I think this makes it (a bit) more intuitive.

sqlalchemy: remove echo setting from .ini files and create_engine

Instead of 'echo' the normal logging configration should be used.

Follow the advice from
https://docs.sqlalchemy.org/en/rel_1_1/core/engines.html#configuring-logging
which says "when using Python logging, ensure all echo flags are set to False
at all times".

TurboGears2 migration: update ini files for error email settings

Error emails are now handled by backlash, which is configured through
TurboGears2's ErrorReporter. ErrorReporter expects different configuration
key names than Pylons did, moreover under a new 'trace_errors' namespace.

Since some of the email-related settings are shared between application and
error emails, we cannot just rename the existing settings (it would be very
odd to have application settings under a 'trace_errors' namespace).
Requiring the user to duplicate its settings is also undesirable.

Instead, use 'get' to populate the trace_errors namespace based on the
existing settings we already had. Unfortunately, 'get' expects the setting
to be actually present, so we need to provide an out-of-the-box value for
the error-related email settings or there will be an error at startup. We
use empty values because there is no realistic default value we can provide.

ini: set 'debug' only in one place

In Pylons-based Kallithea, the 'debug' option was set first in [DEFAULTS]
and then overridden with 'set debug = X' in [app:main]. Even when the value
under [DEFAULTS] was commented out, the use of the 'set' keyword (providing
override semantics) was required, because a default value for the global
'debug' was provided in the framework.

The presence of these two debug assignments is confusing.
Moreover, TurboGears2 makes the situation more easy and does not expect
'debug' to be in [DEFAULTS] nor does it provide a default at that level.
As a result, we can simple use 'debug = X' under [app:main].

Additionally, clarify the Big Fat Warning: the wording 'the line below' is
ambiguous and could be misunderstood in an ini file that had previously been
changed.

ini: remove 'pdebug' setting (profiling)

The mysterious 'pdebug' was used in Pylons-based Kallithea to enable
ProfilingMiddleware. Profiling should now be handled by TurboGears2.

In any case, the pdebug variable is no longer checked anywhere, so can be
removed.

backend: replace Pylons with TurboGears2

Replace the no-longer-supported Pylons application framework by TurboGears2
which is largely compatible/similar to Pylons.
Some interesting history is described at:
https://en.wikipedia.org/wiki/TurboGears

Changes by Dominik Ruf:
- fix sql config in test.ini

Changes by Thomas De Schampheleire:
- set-up of test suite
- tests: 'fix' repo archival test failure
Between Pylons and TurboGears2, there seems to be a small difference in the
headers sent for repository archive files, related to character encoding.
It is assumed that this difference is not important, and that the test
should just align with reality.
- remove need to import helpers/app_globals in lib
TurboGears2 by default expects helpers and app_globals to be available
in lib. For this reason kallithea/lib/__init__.py was originally changed
to include those files. However, this triggered several types of
circular import problems. If module A imported something from lib (e.g.
lib.annotate), and lib.helpers imported (possibly indirectly) module A,
then there was a circular import. Fix this by overruling the relevant
method of tg AppConfig, which is also hinted in the TurboGears2 code.
Hereby, the include of something from lib does not automatically import
helpers, greatly reducing the chances of circular import problems.
- make sure HTTP error '400' uses the custom error pages
TurboGears2 does not by default handle HTTP status code
'400 (Bad Request)' via the custom error page handling, causing a
standard non-styled error page.
- disable transaction manager
Kallithea currently handles its own transactions and does not need the
TurboGears2 transaction manager. However, TurboGears2 tries to enable it
by default and fails, throwing an error during application initialization.
The error itself seemed to be harmless for normal application functioning,
but was nevertheless confusing.
- add backlash as required dependency: backlash is meant as the WebError
replacement in TurboGears2 (originally WebError is part of Pylons). When
debug==true, it provides an interactive debugger in the browser. When
debug==false, backlash is necessary to show backtraces on the console.
- misc fixes

gearbox: replace paster with something TurboGears2-ish that still works with the Pylons stack

This is a step towards moving away from the Pylons stack to TurboGears2, but
still independent of it.


Some notes from the porting - it could perhaps be the missing(?) documentation
for migrating from paster to gearbox:

Note: 'gearbox' without parameters will crash - specify '-h' to get started
testing.

Replace paster
summary = 'yada yada'
with the first line of the docstring of the Command class ... or override
get_description.

Note: All newlines in the docstring will be collapsed and mangle the long help
text.

Grouping of commands is not possible. Standard commands (for development) can't
be customized under the same name or hidden. (Like for paster, the conceptual
model also assumes that the sub-command naming is namespaced so commands from
other packages won't conflict.)

The usage help is fully automated from the declared options.

For all deprecated Commands, replace paster
hidden = True
with gearbox
deprecated = True

Note: config_file, takes_config_file, min_args and max_args are not available /
relevant.

The gearbox parser is customized by overriding get_parser - there is nothing
like paster update_parser.

Gearbox is using argparse instead of optparse ... but argparse add_argument is
mostly backwards compatible with optparse add_option.

Instead of overriding command or run as in paster, override take_action in
gearbox. The parsed arguments are passed to take_action, not available on the
command instance.

Paster BadCommand is not available and must be handled manually, terminating
with sys.exit(1).

There is no standard make-config command in gearbox.

Paster appinstall has been replaced by the somewhat different setup_app module
in gearbox. There is still no clean way to pass parameters to SetupAppCommand
and it relies on websetup and other apparently unnecessary complexity. Instead,
implement setup-db from scratch.


Minor change by Thomas De Schampheleire: add gearbox logging configuration.
Because we use logging.config.fileConfig(.inifile) during gearbox command
execution, the logging settings need to be correct and contain a block for
gearbox logging itself. Otherwise, errors in command processing are not even
visible and the command exits silently.

middleware: replace references to Errormator with AppEnlight.

Errormator has been rebranded as AppEnlight a while back.

Errormator is no longer available. This is just a trivial rename that can't
make things worse.

Turbogears2 migration: use sqlalchemy.url iso sqlalchemy.db1.url

In Turbogears2, much of the application initialization is handled by the
framework, whereas in Pylons the application was responsible for it.
Initializing SQLAlchemy is one such part of initialization which is handled
by Turbogears2.

Turbogears2 expects the configuration file to refer to the database using
'sqlalchemy.url' rather than the current 'sqlalchemy.db1.url'. While the
exact name is not really important, not following this approach means we'll
need to override the sqlalchemy initialization method.

Therefore, as a preparation to the Turbogears2 migration, already change the
database reference string under Pylons.
When upgrading to a version of Kallithea containing this commit, the .ini
file will manually need to be adapted to remove the .db1 strings.

celery: use Celery 3 config settings instead of deprecated

As warned by:
The 'CELERYD_LOG_LEVEL' setting is scheduled for deprecation in version 2.4 and removal in version v4.0. Use the --loglevel argument instead
remove celeryd.log.file and celeryd.log.level from the ini file.
Instead, use:
paster celeryd my.ini --loglevel=DEBUG --logfile=my.log
or, in the future:
gearbox celeryd -c my.ini -- --loglevel=DEBUG --logfile=my.log

As warned by:
The 'BROKER_VHOST' setting is scheduled for deprecation in version 2.5 and removal in version v4.0. Use the BROKER_URL setting instead
The 'BROKER_HOST' setting is scheduled for deprecation in version 2.5 and removal in version v4.0. Use the BROKER_URL setting instead
The 'BROKER_USER' setting is scheduled for deprecation in version 2.5 and removal in version v4.0. Use the BROKER_URL setting instead
The 'BROKER_PASSWORD' setting is scheduled for deprecation in version 2.5 and removal in version v4.0. Use the BROKER_URL setting instead
The 'BROKER_PORT' setting is scheduled for deprecation in version 2.5 and removal in version v4.0. Use the BROKER_URL setting instead
change the .ini template to use:
broker.url = amqp://rabbitmq:qewqew@localhost:5672/rabbitmqhost

As warned by:
Starting from version 3.2 Celery will refuse to accept pickle by default.

The pickle serializer is a security concern as it may give attackers
the ability to execute any command. It's important to secure
your broker from unauthorized access when using pickle, so we think
that enabling pickle should require a deliberate action and not be
the default choice.

If you depend on pickle then you should set a setting to disable this
warning and to be sure that everything will continue working
when you upgrade to Celery 3.2::

CELERY_ACCEPT_CONTENT = ['pickle', 'json', 'msgpack', 'yaml']

You must only enable the serializers that you will actually use.
change the .ini template to use:
celery.accept.content = pickle

(Note: The warning is there for a reason. It would probably be nice to change
from pickle to something like json. That is left as an exercise.)

config: clarify that we only recommend and support single threaded operation

Sad, but true. Especially because we reuse Repository instances between
threads.

db: enable use of main Kallithea config as Alembic config

Newly generated Kallithea config .ini files will be valid Alembic
config files, eliminating the need for a separate alembic.ini config
redundantly specifying the database connection string.

We reference the Alembic migration environment using kallithea:alembic,
which should work independently of how Kallithea is installed.

We also configure a default 'alembic' log level of WARNING, to reduce
the amount of clutter in the config file, reduce the changes needed
to upgrade existing config files for use with Alembic, and allowing us
to change the default Alembic log level for all users down the road.

(It makes sense to define Alembic logging in code, while all other
loggers are configured in the configuration file, because Alembic
is special: it runs on the command line, not as part of the web app.)

docs: tweak email documentation - add gmail example

smtp_auth is too obscure to have in the .ini template.

db: get rid of vcs_full_cache - it should always be used

It might make tests less deterministic, but it makes sure that we test what we actually use.

db: stop messing with convert_unicode in .ini and db scheme

convert_unicode=False is the default for String fields ... except when the .ini file contains sets it to True.

String fields are however no longer used for unicode and we can drop convert_unicode everywhere.

See
http://docs.sqlalchemy.org/en/latest/core/type_basics.html#sqlalchemy.types.String.params.convert_unicode
for the advice of never setting convert_unicode.

ini: specify utf8 for sample MySQL connection strings

By default, the MySQL stack will store unicode as UTF-8 encoded data in string
fields, thus without using any unicode capabilities in the database.

As described in
http://docs.sqlalchemy.org/en/latest/dialects/mysql.html#mysql-unicode , set
charset=utf8 to actually put unicode in the database.

Existing databases that already store utf8 in the database should keep using
the old url.

This will only support 16 bit code points, but utf8mb4 will double the key size
and make them too big for MySQL.

ini: add scripts/generate-ini.py for generating all .ini files from template.ini.mako

Based on kallithea/bin/template.ini.mako, generate
kallithea/config/deployment.ini_tmpl
development.ini
kallithea/tests/test.ini

The script might not be pretty, but it works for the given input, and all
changes in output will be reviewed thoroughly anyway ...

cache: make instance_id = * the default and deprecate it

Auto assigning instance_id's works perfectly fine now when the clean-up issue
has been resolved.

cleanup: use example.com for tests and examples

example.com is explicitly reserved for this purpose. Using that means
we won't accidentally hammer a real server or real email address if an
example value escapes into the wild, e.g. in an automated test.

The domain "kallithea.example.com" has been used throughout to refer to the
example Kallithea server.

spelling: use "email" consistently

The common English spelling is "email", not "e-mail" (and was indeed
also the most common, but not only, variant in the Kallithea UI).

http://grammarist.com/style/e-mail-email/

docs: improve documentation of beaker session configuration

beaker.session.auto is dropped; it defaults to false and there is no reason to
ever set it true for Kallithea.

beaker.session.cookie_path and secure are dropped; like cookie_domain, they
should automatically be set to the right value.
* * *
beaker.session.cookie_expires MUST have the default value of True to provide the default value of 'browser session lifetime' when not enabling 'remember' in the login box. The cookie life is hardcoded to 365 days when remember is selected.

e-mail: add documentation about configuration settings

Document e-mail related configuration settings in the documentation
(overview) and the .ini file itself (technical details and example
settings).

e-mail: add some default values and some examples to .ini file as comments

This includes a replacement of @localhost examples to @example.com, to
indicate that real addresses should normally be used.

e-mail: clarify that smtp_use_ssl defaults to false

Commented settings in configuration files conventionally indicate the
default value.

e-mail: remove unused setting error_message from ini files

The ini files and templates contain a commented setting of 'error_message'
which does not seem to be used. It is referring to the error_message
variable in Paste, which has as description (Paste:ErrorMiddleware):

When debug mode is off, the error message to show to users.

However, setting this value apparently made no effect at all in Kallithea.