auth: only use X- headers instead of wsgi.url_scheme if explicitly told so in url_scheme_header - drop https_fixup setting

Before, several X- headers would be trusted to overrule the actual connection
protocol (http or https) seen by the Kallithea WSGI server. That was mainly
when https_fixup were set, but it incorrectly also kicked in if https_fixup or
use_htsts were configured. The ambiguity of which headers were used also made
it less reliable. The proxy server not only had to be configured to set one of
the headers correctly, it also had to make sure other headers were not passed
on from the client. It would thus in some cases be possible for clients to fake
the connection scheme, and thus potentially be possible to bypass restrictions
configured in Kallithea.

Fixed by making it configurable which WSGI environment variable to use for the
protocol. Users can configure url_scheme_header to for example
HTTP_X_FORWARDED_PROTO instead of using the default wsgi.url_scheme .

This change is a bit similar to what is going on in the https_fixup middleware,
but is doing a bit more of what for example is happening in similar code in
werkzeug/middleware/proxy_fix.py .

The semantics of the old https_fixup were unsafe, so it has been dropped.
Admins that are upgrading must change their configuration to use the new
url_scheme_header option.

auth: only use X- headers instead of REMOTE_ADDR if explicitly told so in remote_addr_header

Before, X-Forwarded-For (and others) headers would *always* be trusted blindly,
also in setups without a proxy server. It would thus in some cases be
possible for users to fake their IP, and thus potentially be possible to bypass
IP restrictions configured in Kallithea.

Fixed by making it configurable which WSGI environment variable to use for the
remote address. Users can configure remote_addr_header to for example
HTTP_X_FORWARDED_FOR instead of using the default REMOTE_ADDR.

This change is a bit similar to what is going on in the https_fixup middleware,
but is doing a bit more of what for example is happening in similar code in
werkzeug/middleware/proxy_fix.py .

hooks: on Git, invoke hooks/post-receive-custom from hooks/post-receive

Make it possible for admins to install all kinds of hooks.

Based on a patch by Tim Ooms.

A more generic solution would be nice, but for now we aim for this minimal
solution.

hooks: clarify in UI and documentation how the hook control essentially is Mercurial only

The Git hook model doesn't directly allow multiple hooks of the same kind.

docs: give "File system location" overview

Also make some other adjustments for consistency.

docs: clarify how to activate virtualenv from mod_wsgi with py3

activate_this.py is not available with the venv module in py3. According to
https://modwsgi.readthedocs.io/en/develop/user-guides/virtual-environments.html
we just need to set python-home - that is the way to do proper virtualenv
activation, and there is no need for hacks in the wrapper script.

docs: drop changing cwd in WSGI wrapper script

It was introduced in 5a31d387f347 and has never been in the virtualenv version
of the WSGI script, which "proves" it really is unnecessary.

docs: clean up readthedocs URLs

It will redirect to the canonical URL https://*.readthedocs.io/en/latest , and
there is no point in linking to /en/latest .

db: introduce db-create --reuse option

Support use of an existing database so the Kallithea database user doesn't have
to be granted permissions to create databases.

The existing database must of course have been created "correctly", for example
using the right charset and collation on MariaDB/MySQL. Creating the database
manually also provides a way to avoid the hardcoded defaults.

docs: augment setup description with more details of http server and database

Dive into more details than in overview.rst .

Databases should perhaps have more mentioning in overview.rst , but there is
nothing but details and thus not much to say in the overview ...

docs: drop odd verbose note on writeable root path

It was introduced back in a60cd29ba7e2.

celery: upgrade to Celery 4

Celery 3 doesn't support Python 3.7 or later. This upgrade is thus essential
for full Python 3 support. But note that
https://docs.celeryproject.org/en/4.4.0/faq.html#does-celery-support-windows
says "No".

The names of config settings changed in Celery 3 to 4, as described on
https://docs.celeryproject.org/en/3.0/whatsnew-4.0.html#lowercase-setting-names .

Celery 4 config settings can now be specified in Kallithea .ini files by prefixing
with `celery.` - for example as `celery.broker_url`.

Remain backwards compatible for the usual settings, and map old names to the
new names.

py3: officially support Python 3

All tests pass and no known regressions.

i18n: make sure 'en' in Accept-Language is recognized as having 100% coverage - i18n.lang is for source language

https://github.com/cdent/paste/blob/3.2.3/paste/wsgiwrappers.py#L89 describes
how paste solved the problem. TG solves the same problem (with less
explanation) in
https://github.com/TurboGears/tg2/blob/tg2.4.2/tg/request_local.py#L36 with
fallback language specified in i18n.lang .

Thus, clarify the use of i18n.lang (refining f2f7a8c1281e and 8931078f70db) and
set 'en' as default value on app startup.

TurboGears requires an (empty) translation for the source language which is
default for i18n.lang . The empty .mo for en is created as the 4 magic .mo
bytes followed by lengths of 0:
printf '\x95\x04\x12\xde\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0' > kallithea/i18n/en/LC_MESSAGES/kallithea.mo

Based on report and analysis by Wolfgang Scherer. Also discussed upstream on
https://github.com/TurboGears/tg2/pull/115 .

logging: always invoke fileConfig with '__file__' and 'here'

WSGI servers tend to provide '__file__' and 'here' as 'defaults' when invoking
fileConfig, so '%(here)s' string interpolation also can be used in logging
configuration.

Make sure we do the same when we initialize logging without using a WSGI server.

It is annoying to have to do this, and it will only in rare cases make any
difference ... but it seems like the best option.

Patch also modified by Mads Kiilerich.

docs: add high-level description of SSH repository access

In addition to the existing technical documentation about SSH repository
access, add some high-level info about what this means.

(some editing by Thomas De Schampheleire)

ssh: add documentation of SSH support

Based on work by Ilya Beda <ir4y.ix@gmail.com> on
https://bitbucket.org/ir4y/rhodecode/commits/branch/ssh_server_support , also
heavily modified by Mads Kiilerich and Thomas De Schampheleire.

Text related to the case where the parent directory of 'authorized_keys'
does not exist or is not writable, was contributed by Bradley M. Kuhn
<bkuhn@ebb.org>.

docs: outline the challenges of specifying a locale for services (Issue #340)

Trying to embed the most essential parts of
http://blog.dscpl.com.au/2014/09/setting-lang-and-lcall-when-using.html and
https://modwsgi.readthedocs.io/en/develop/configuration-directives/WSGIDaemonProcess.html
.

(some rewording by Thomas De Schampheleire)

docs: tweak documentation of Apache+mod_wsgi further

Make the last bullet not only about WSGIScriptAlias but about the entire set
of WSGI* directives in the Apache Virtual Host configuration file.
This fits better with the already existing explanation of changing
user/group settings, and with the upcoming hints about locale settings.

docs: move Apache+mod_wsgi example code to the corresponding bullets

The documentation about Apache+mod_wsgi has bullet points with inline
snippets, yet the example WSGI dispatch script is placed at the bottom of
the section instead of near its corresponding bullet.

It seems more readable and more according to the logical setup flow to move
the code next to its bullet.

Due to the additional indentation required to 'attach' the code to the
bullet, this commit is best viewed with the 'ignore whitespace changes'
setting.

install-iis: use logging.config.fileConfig instead of paste.script copy

paste.script.util.logging_config is an old copy of logging.config from
Python 2.5.1 (see [1]). There are no paste-specific details in it.

In other places, we are always using logging.config directly, there is no
reason not to do it in install-iis.


[1] https://bitbucket.org/wilig/pastescript/commits/04d0db6b2f5ab360bdaa5b10511d969a24fde0ec

cli: add command 'kallithea-cli front-end-build'

Kallithea is under the GPL license, and we can thus only distribute any
generated code if we also ship the corresponding source.

We are moving towards a web front-end that use npm to download and compile
various open source components. The components might not be GPL, but if we
distribute any parts of their code (compiled or converted to other
representation), then we also must distribute the corresponding source under
the GPL.

It doesn't seem feasible for us to distribute the source of everything that
npm downloads and includes when we are building. It thus also doesn't seem
feasible for us to build and ship the compiled (possibly minified) front-end
code. Instead, we have to make it as smooth as possible for our users to
get up and running.

It doesn't seem feasible for us to ship or install npm. We must assume it is
available. That requirement must be documented clearly, and we must recommend
how to install npm for the most common platforms.

We could perhaps just document what manual steps to run. Kallithea doesn't
work out of the box anyway - it has to be configured and initialized. Extra
steps might not be a big problem.

Another approach is to call out to npm while pip is installing Kallithea and
download the requirements and build the files. It can be done by customizing
setuptools commands in setup.py. But: Python packaging is fragile. Even
though we only support pip, it really isn't built for things like this.
Custom output is muted and buffered and only shown if running with -v or the
command fails. And pip and setup.py can be used to build and install in so
many ways that we probably can't make it work reliably with all ways of
installing Kallithea.

The approach implemented by this commit is to add a custom cli command
'front-end-build' to run the required commands. This single user-facing
command can internally run various steps as needed. The only current
requirement is the presence of npm and an internet connection.

For now, this will just create/update style.css ... but currently probably
without any actual changes. The files created by npm (and the node_modules
directory) must *not* be a part of the release package made with 'setup.py
sdist'.

(Commit message is mostly written by Mads Kiilerich)

cli: convert 'gearbox celeryd' into 'kallithea-cli celery-run'

Note:
- '--' is never explicitly present in the arguments when using Click.
The click parser will take care of '--' as separator between
dash-dash-arguments and positional arguments, following standard UNIX
conventions.

cli: convert 'gearbox make-config' into 'kallithea-cli config-create'

... and update documentation.

docs: move authentication info to separate file

Goal is to reduce the gigantic 'setup' page in size, and make the
information more understandable. Currently, the different topics do not
really belong together.
This is one small step towards that goal.

remove references to pythonhosted.org (issue #293)

The pythonhosted service is no longer supported.
Update links to point to docs.kallithea-scm.org (which redirects to
kallithea.readthedocs.io)

While at it, fix the whoosh pythonhosted.org reference as well.

remove references to pythonhosted.org (issue #293)

The pythonhosted service is no longer supported.
Update links to point to docs.kallithea-scm.org (which redirects to
kallithea.readthedocs.io)

While at it, fix the whoosh pythonhosted.org reference as well.

hg: set encoding to utf-8 by default to always show unicode characters correctly

Unicode characters would be shown as '?' if Kallithea was launched in a LANG=C
environment (or similar).

The problem could be solved by setting HGENCODING before launching Kallithea or
before importing Mercurial. These are often not good solutions.

Instead, introduce a hgencoding config setting that triggers monkey patching of
Mercurial.

issues: support generic regex replacements in issue_url and issue_prefix

Issue reference linking is pretty limited:
- the issue_url is a literal with only three special tokens {id},
{repo} and {repo_name}. There is no way to let the URL be dependent on
other elements of the input issue reference.
- The value for {id} is somewhat oddly determined by the concatenation of
all parenthesized groups in the issue_pat regular expression
- the link text of the resulting link is limited to the contents of the
literal issue_prefix with the determined {id}. It is not possible to
retain the input issue reference verbatim, nor to let the link text be
dependent on other elements of the input issue reference.

This commit makes the issue reference linking more flexible:

- issue_prefix is replaced by the more generic issue_sub(stitution), which
is a string that may contain backreferences to regex groups specified in
issue_pat. This string, with backreferences resolved, is used as the
link text of urlified issue references.
- if issue_sub is empty, the entire text matched by issue_pat is used as
the link text.
- like issue_sub, also issue_url can contain backreferences to regex groups.
- {id} is no longer treated as a special token, as it can be solved by
generic backreferences ('\g<id>' assuming issue pattern contains something
like '(P<id>\d+)'. {repo} and {repo_name} are still supported, because
their value is provided externally and not normally part of the
issue pattern.

Documentation and ini file template is updated as well.

move package.json to root directory

In the future we'll probably use it to manage more then just less/css stuff.
So the less directory is the wrong place.
The most common place is the root directory, so lets put it there.
Also, this way the --prefix parameter for npm is no longer required.

less: don't distribute the generated style.css file - for now, it must be built with npm after installing Kallithea

Generated files should not be included in the repository. Especially not when
using GPL with the requirement of distributing corresponding source.

Also add the two commands to generate the css file to the docs.

Modified by Mads Kiilerich.

make-config: allow configuration of any ini value

Custom values can now be specified like:

gearbox make-config my.ini host=8.8.8.8 '[handler_console]' formatter=color_formatter '[app:main]' i18n.lang=zu

docs: fix some sphinx warnings

RST code blocks must have valid syntax, and <someprefix> could thus not be used
as placeholder.

docs/setup: heading whitespace cleanup

Make sure 'run-all-cleanup' renders zero differences after 8931078f70db.

config: clarify that we only recommend and support single threaded operation

Sad, but true. Especially because we reuse Repository instances between
threads.

docs: add documentation about internationalization from a user perspective

While there is already documentation about creating and updating
translations (developer perspective), there was nothing for the user/admin
of Kallithea.
Since the handling in TurboGears2 is different, this is a good time to add
it.

gearbox: make a make-config sub-command available again

Drop the old kallithea-config command line tool and rework it to a replacement
for the make-config paster command. Make-config was experimental in paste, it
doesn't exist with gearbox, it was tied to pylons, and we already have the
luxury problem of having two ways of doing almost the same thing. This is a
good opportunity to get rid of one of them.

This replacement tool is Kallithea specific and doesn't need to be told that it
is Kallithea it has to create a config file for. The command should thus
perhaps be given a Kallithea specific name instead of the very generic name ...

gearbox: replace paster with something TurboGears2-ish that still works with the Pylons stack

This is a step towards moving away from the Pylons stack to TurboGears2, but
still independent of it.


Some notes from the porting - it could perhaps be the missing(?) documentation
for migrating from paster to gearbox:

Note: 'gearbox' without parameters will crash - specify '-h' to get started
testing.

Replace paster
summary = 'yada yada'
with the first line of the docstring of the Command class ... or override
get_description.

Note: All newlines in the docstring will be collapsed and mangle the long help
text.

Grouping of commands is not possible. Standard commands (for development) can't
be customized under the same name or hidden. (Like for paster, the conceptual
model also assumes that the sub-command naming is namespaced so commands from
other packages won't conflict.)

The usage help is fully automated from the declared options.

For all deprecated Commands, replace paster
hidden = True
with gearbox
deprecated = True

Note: config_file, takes_config_file, min_args and max_args are not available /
relevant.

The gearbox parser is customized by overriding get_parser - there is nothing
like paster update_parser.

Gearbox is using argparse instead of optparse ... but argparse add_argument is
mostly backwards compatible with optparse add_option.

Instead of overriding command or run as in paster, override take_action in
gearbox. The parsed arguments are passed to take_action, not available on the
command instance.

Paster BadCommand is not available and must be handled manually, terminating
with sys.exit(1).

There is no standard make-config command in gearbox.

Paster appinstall has been replaced by the somewhat different setup_app module
in gearbox. There is still no clean way to pass parameters to SetupAppCommand
and it relies on websetup and other apparently unnecessary complexity. Instead,
implement setup-db from scratch.


Minor change by Thomas De Schampheleire: add gearbox logging configuration.
Because we use logging.config.fileConfig(.inifile) during gearbox command
execution, the logging settings need to be correct and contain a block for
gearbox logging itself. Otherwise, errors in command processing are not even
visible and the command exits silently.

auth: change default LDAP to LDAPS on port 636 - insecure authentication is kind of pointless

This will only change the default value for new systems. Existing installations
will keep using whatever value they have in the database.

docs: remove some references to Pylons

These references are either not used, or do not provide value.
The Windows installation and email pages still references documents from the
Pylons website -- alternative pages from Turbogears2 need to be found.

docs: update links to Mercurial's website and wiki

Also Mercurial itself is not a coding standard, so add possessive suffix.

auth: make ldap OPT_X_TLS_CACERTDIR configurable

A location was hardcoded. The location was wrong for many systems and prevented
actual TLS from working. Also, it should not be necessary with modern Pythons.

For some reason, instead of removing it, we now decide to expose it to the
user. Choice FTW!

config: clarify that we only recommend and support single threaded operation

Sad, but true. Especially because we reuse Repository instances between
threads.

docs: improve mod_wsgi documentation (Issue #203)

Based on feedback from Graham Dumpleton.

* Recommend setting global Apache config WSGIRestrictEmbedded On
* Don't set the default value for processes=1 - it has side effects
* Use python-home instead of python-path
* Fix confusing mentioning of root and clarify the use of specifying user and group
* Include WSGIProcessGroup in configuration excerpt

docs: add notes about IIS, Windows Authentication and Mercurial

(The original patch from Konstantin has been heavily copyedited and modified by
Mads Kiilerich but is still [based on] Konstantin's feedback and contribution.)

docs: document Kallithea customization options

Create basic documentation for the various customization options in
Kallithea. Move the rcextensions section to this new page and reword.

cache: make instance_id = * the default and deprecate it

Auto assigning instance_id's works perfectly fine now when the clean-up issue
has been resolved.

cleanup: use example.com for tests and examples

example.com is explicitly reserved for this purpose. Using that means
we won't accidentally hammer a real server or real email address if an
example value escapes into the wild, e.g. in an automated test.

The domain "kallithea.example.com" has been used throughout to refer to the
example Kallithea server.

docs: update example output and example server configs

kallithea-api example output was not up-to-date, and the text was a
little vague on whether you specify a hostname or a URL.

The Nginx example config has been updated to assume a Kallithea backend
server on localhost:5000, like the Apache example.

The redundant ServerAlias option was removed from the Apache example.

docs: consistent spacing around headings

Start out with 2 empty lines before/after for top level, decrease for deeper
levels.

docs: update menu navigation notation to use *Menu > Menu Item*

Follow Apple's notation for 'Menus' from
http://help.apple.com/asg/mac/2013/#apsg1f285825 .

docs: cleanup of casing, markup and spacing of headings

Mostly stuff found and fixed by Søren, extracted here to separate things.

Other uses of title casing might be debatable, but here it was just a few
documentation headings that clearly were inconsistent with the majority.

spelling: use "email" consistently

The common English spelling is "email", not "e-mail" (and was indeed
also the most common, but not only, variant in the Kallithea UI).

http://grammarist.com/style/e-mail-email/

rcextensions: cleanup of code and documentation

More cleanup is needed - this is just to clarify what it currently does.

docs: move all instructions on Celery to Setup

Instead of having some info on Celery in both Installation and Setup, move
everything to Setup and do some rewrite.

Additionally, update some outdated URLs and remove unused link targets.

docs: rework stuff

The existing docs were far from how we wanted it to be. There was so much to do
and it is not feasible to do that cleanup it in clean patches.

Instead, I took a sweep through the docs and changed what I thought could
benefit from a change: structure, examples, advices, language, markup, content,
etc.

https: introduce https_fixup config setting to enable the special https hacks

Without https_fixup, correctly configured WSGI systems work correctly.

The https_fixup middleware will only be loaded when enabled in the configuration.

First step in two-part process to rename directories to kallithea.
This first step is to change all references in the files where they refer
to the old directory name.

Added two headers into example nginx proxy conf that allows container auth
to work properly