setup: avoid setuptools 67 - it can't handle celery's broken pytz dependency

With setuptools 67 or later, launching Kallithea fails as:

$ gearbox serve -c my.ini --reload
15:56:54,111 ERROR [gearbox] Expected closing RIGHT_PARENTHESIS
pytz (>dev)
~^

The `packaging` vendored in setuptools cannot handle the broken syntax
`Requires-Dist: pytz (>dev)` in
venv/lib/python3.11/site-packages/celery-5.0.5.dist-info/METADATA .

The old celery version currently used by Kallithea is wrong, and setuptools has
moved on after a reasonable grace period. We thus have to work around and avoid
latest setuptools.

See https://github.com/pypa/setuptools/issues/3889 .

controllers: move controllers base class from lib/base to controllers

TG quickstart put it in lib/base.py , but it fits better on the controllers
layer as a base there.

The contributing docs were a bit ahead of time ... but with a typo.

docs: describe, visualize, and verify internal code structure and layering

Try to describe something that isn't entirely there yet.

deps.py will help track and minimize violations through checks and
visualization in deps.svg .

docs/contributing: wrap long line

Forgotten review comment.

docs: when using venv with a clone, recommend using 'venv' as location

Avoid having the odd inconsistent location ../kallithea-venv to care about.

For good or bad, this might change how it is feasible to grep. I usually make
sure to only search tracked files:
grep Foo `hg mani`

docs: contributing: use '.' rather than 'source' to align with 'installation'

See commit 855ba1f07aeb437483138d48d7929f0ff8c38b67 .

mysql: bump sqlalchemy.url MariaDB/MySQL charset to to 'utf8mb4' to get full UTF-8 support

The change in 210e76d69b62 only changed character_set_database, as shown by output after:

--- a/kallithea/model/base.py
+++ b/kallithea/model/base.py
@@ -46,3 +46,8 @@ def init_model(engine):
engine_str = obfuscate_url_pw(str(engine.url))
log.info("initializing db for %s", engine_str)
meta.Base.metadata.bind = engine
+
+ meta.Session.configure(bind=engine)
+ for a, b in meta.Session().execute('''show variables''').fetchall():
+ if 'character_set_' in a:
+ print(a, repr(b))

Before, with charset=utf8, the utf8mb3 charset was used all the way through the stack:

[kallithea.model.base] initializing db for mysql://kallithea-test:XXXXX@localhost/kallithea-test?charset=utf8
character_set_client 'utf8'
character_set_connection 'utf8'
character_set_database 'utf8mb4'
character_set_filesystem 'binary'
character_set_results 'utf8'
character_set_server 'latin1'
character_set_system 'utf8'

With explicit charset=utf8mb4:

[kallithea.model.base] initializing db for mysql://kallithea-test:XXXXX@localhost/kallithea-test?charset=utf8mb4
character_set_client 'utf8mb4'
character_set_connection 'utf8mb4'
character_set_database 'utf8mb4'
character_set_filesystem 'binary'
character_set_results 'utf8mb4'
character_set_server 'latin1'
character_set_system 'utf8'

tests: introduce REUSE_TEST_DB as an alternative to TEST_DB

With TEST_DB, the specified database would be dropped (if existing) and created
from scratch. That would fail if the user only had been granted access to that
database without general database creation permissions.

With REUSE_TEST_DB, the database must exist, and the tables in the existing
database will be deleted before creating the new ones.

This is for example useful for testing on PostgreSQL without full database
privileges. (With MariaDB/MySQL, it is possible to grant create permissions to
a database before it exists, so it is easier to use TEST_DB.)

ini: tweak template configuration for TG's handling of application errors

Mention what was learned while debugging request processing at WSGI/TG level
and the complexity of the TG stack gets in the way.

docs: fix contributing.rst reference to Session.remove()

(Second to) last example of Session.remove was removed in 1a409593f352.

py3: officially support Python 3

All tests pass and no known regressions.

docs: clean up installation of optional dependencies

Install them together with Kallithea, to make things simpler, and make sure all
constraints are handled correctly.

docs: always use dev_requirements.txt together with main version constraints in setup.py

pip isn't good at considering old constraints when installing or updating
packages later. Instead, install everything at once.

Note: pip and setuptools are usually installed automatically and could perhaps
be removed from documentation. But for now, let's keep it.

setup: drop support for Python 2.6

It would be a a stepping stone for the migration to Python 3 to only support
Python 2.7. Even though we don't make any big changes now, it might allow us to
remove some workarounds or use some new forward-compatible features.

Mercurial dropped support for Python 2.6 2 years ago.

cli: add command 'kallithea-cli front-end-build'

Kallithea is under the GPL license, and we can thus only distribute any
generated code if we also ship the corresponding source.

We are moving towards a web front-end that use npm to download and compile
various open source components. The components might not be GPL, but if we
distribute any parts of their code (compiled or converted to other
representation), then we also must distribute the corresponding source under
the GPL.

It doesn't seem feasible for us to distribute the source of everything that
npm downloads and includes when we are building. It thus also doesn't seem
feasible for us to build and ship the compiled (possibly minified) front-end
code. Instead, we have to make it as smooth as possible for our users to
get up and running.

It doesn't seem feasible for us to ship or install npm. We must assume it is
available. That requirement must be documented clearly, and we must recommend
how to install npm for the most common platforms.

We could perhaps just document what manual steps to run. Kallithea doesn't
work out of the box anyway - it has to be configured and initialized. Extra
steps might not be a big problem.

Another approach is to call out to npm while pip is installing Kallithea and
download the requirements and build the files. It can be done by customizing
setuptools commands in setup.py. But: Python packaging is fragile. Even
though we only support pip, it really isn't built for things like this.
Custom output is muted and buffered and only shown if running with -v or the
command fails. And pip and setup.py can be used to build and install in so
many ways that we probably can't make it work reliably with all ways of
installing Kallithea.

The approach implemented by this commit is to add a custom cli command
'front-end-build' to run the required commands. This single user-facing
command can internally run various steps as needed. The only current
requirement is the presence of npm and an internet connection.

For now, this will just create/update style.css ... but currently probably
without any actual changes. The files created by npm (and the node_modules
directory) must *not* be a part of the release package made with 'setup.py
sdist'.

(Commit message is mostly written by Mads Kiilerich)

cli: convert 'gearbox make-config' into 'kallithea-cli config-create'

... and update documentation.

docs: update i18n doc after TG migration changed lang to i18n.lang and test.ini is generated

These descriptions are not elegant, but now we make them less wrong.

move package.json to root directory

In the future we'll probably use it to manage more then just less/css stuff.
So the less directory is the wrong place.
The most common place is the root directory, so lets put it there.
Also, this way the --prefix parameter for npm is no longer required.

less: don't distribute the generated style.css file - for now, it must be built with npm after installing Kallithea

Generated files should not be included in the repository. Especially not when
using GPL with the requirement of distributing corresponding source.

Also add the two commands to generate the css file to the docs.

Modified by Mads Kiilerich.

docs: recommend --upgrade for all pip installations in a virtualenv

As suggested by Mads Kiilerich.

Don't recommend this practice when no virtualenv is used, i.e. root
installation or --user installation: in this case upgrading existing packages
may be too disruptive. The user can still add it manually if he so chooses.

docs/contributing: move installation of dev_requirements to default instructions

When intending to contribute to Kallithea, you are expected to run all
tests, so installation of the dev_requirements are actually mandatory.
Hence, update the instructions.

docs/contributing: make contribution information more accessible

Try to make the contribution information more accessible to new users, by
giving a bit more detail about the different steps in making contributions.

Some rework of the original patch done by Thomas De Schampheleire:
- don't give the impression that Bitbucket is mandatory
- don't recommend creating feature branches for regular development
- reword/cross-reference section on contribution flow

docs: mention pytest-profiling

Could be useful for analysis of performance problems.

tests: add simple (skipped) unit tests for graph_data that may be used to measure performance

Modified by Thomas De Schampheleire to have performance tests in a separate
directory so it is easier to run them, and also added some documentation.

docs: clarify that Session usually should be called - methods should not be used directly

Documentation based on clarification by Søren Løvborg:

Session is the factory/singleton manager, which tracks the current session (per
thread). To end the current session entirely and destroy the Session object, we
call remove on the manager (Session.remove()). (A new session will be created
on-demand.)

Session() returns the current session for the active thread (or creates a new
session, if there's none). commit is a method of the SQLAlchemy Session class,
thus called as Session().commit() ... it's a method call on the current Session
object, not the session factory/manager.

SQLAlchemy may have some hackery to allow Session.commit() to be called, and
the call automatically redirect to the actual Session object... but that's a
hack and should be avoided.

TL;DR: for remove, call it on Session; for everything else, call it on
Session().

setup: don't use setuptools 34 - it has indirect conflicts with the celery version supported on the stable branch (Issue #266)

Setuptools==34 requires packaging>=16.8 which has an unconstrained requirement
of pyparsing ... but actually it doesn't work with pyparsing==1.5.7 ... which
is required by celery<2.3 ... which this version of Kallithea requires.

Celery has been upgraded on the development branch but we don't want to do that
on the stable branch.

backend: add TurboGears2 DebugBar support

DebugBar is a toolbar overlayed over the Kallithea web interface, allowing
you to see:

* timing information of the current request, including profiling information
* request data, including GET data, POST data, cookies, headers and
environment variables
* a list of executed database queries, including timing and result values

Enable it automatically when in debug mode and all requirements are met.
Add documentation.

gearbox: make a make-config sub-command available again

Drop the old kallithea-config command line tool and rework it to a replacement
for the make-config paster command. Make-config was experimental in paste, it
doesn't exist with gearbox, it was tied to pylons, and we already have the
luxury problem of having two ways of doing almost the same thing. This is a
good opportunity to get rid of one of them.

This replacement tool is Kallithea specific and doesn't need to be told that it
is Kallithea it has to create a config file for. The command should thus
perhaps be given a Kallithea specific name instead of the very generic name ...

gearbox: replace paster with something TurboGears2-ish that still works with the Pylons stack

This is a step towards moving away from the Pylons stack to TurboGears2, but
still independent of it.


Some notes from the porting - it could perhaps be the missing(?) documentation
for migrating from paster to gearbox:

Note: 'gearbox' without parameters will crash - specify '-h' to get started
testing.

Replace paster
summary = 'yada yada'
with the first line of the docstring of the Command class ... or override
get_description.

Note: All newlines in the docstring will be collapsed and mangle the long help
text.

Grouping of commands is not possible. Standard commands (for development) can't
be customized under the same name or hidden. (Like for paster, the conceptual
model also assumes that the sub-command naming is namespaced so commands from
other packages won't conflict.)

The usage help is fully automated from the declared options.

For all deprecated Commands, replace paster
hidden = True
with gearbox
deprecated = True

Note: config_file, takes_config_file, min_args and max_args are not available /
relevant.

The gearbox parser is customized by overriding get_parser - there is nothing
like paster update_parser.

Gearbox is using argparse instead of optparse ... but argparse add_argument is
mostly backwards compatible with optparse add_option.

Instead of overriding command or run as in paster, override take_action in
gearbox. The parsed arguments are passed to take_action, not available on the
command instance.

Paster BadCommand is not available and must be handled manually, terminating
with sys.exit(1).

There is no standard make-config command in gearbox.

Paster appinstall has been replaced by the somewhat different setup_app module
in gearbox. There is still no clean way to pass parameters to SetupAppCommand
and it relies on websetup and other apparently unnecessary complexity. Instead,
implement setup-db from scratch.


Minor change by Thomas De Schampheleire: add gearbox logging configuration.
Because we use logging.config.fileConfig(.inifile) during gearbox command
execution, the logging settings need to be correct and contain a block for
gearbox logging itself. Otherwise, errors in command processing are not even
visible and the command exits silently.

docs: update links to Mercurial's website and wiki

Also Mercurial itself is not a coding standard, so add possessive suffix.

docs: separate coding/contribution guidelines

Try to separate "process" guidelines from "content" guidelines.
(Just moving stuff around here, not changing the actual guidelines.)

setup: move test dependencies to dev_requirements.txt to make them optional

Remove the need for having test tools on production systems. Installing test
dependencies is made an extra explicit step.

pip is the future, but doesn't have the same tests_require features as
setuptools kind of has.

I don't like this way of handling it without setup.py support and with explicit
naming of the ugly dev_requirements.txt ... but that seems to be the way to do
it.

docs: move translation to a separate page

Start new tradition of putting it under docs/dev/

docs: make the default method for running tests more visible

Sometimes, users found the convoluted and risky method of testing without
recreating the database before spotting the simple method.

pytest migration: update documentation

Update documentation and comments to refer to pytest instead of nose.

docs: add advice of upgrading pip and setuptools in new virtualenvs

Withtout this, the new virtualenv might have setuptools version 12 while the
URLObject dependency mock fails because it requires setuptools>=17.1 .

docs: consistent spacing around headings

Start out with 2 empty lines before/after for top level, decrease for deeper
levels.

spelling: use "email" consistently

The common English spelling is "email", not "e-mail" (and was indeed
also the most common, but not only, variant in the Kallithea UI).

http://grammarist.com/style/e-mail-email/

docs/contributing: move 'roadmap' to the wiki

As the 'roadmap' is typically changed often, it should not be in the
documentation but rather on the wiki. The wiki has been updated to reflect
the latest version removed here.

docs: rework stuff

The existing docs were far from how we wanted it to be. There was so much to do
and it is not feasible to do that cleanup it in clean patches.

Instead, I took a sweep through the docs and changed what I thought could
benefit from a change: structure, examples, advices, language, markup, content,
etc.

First step in two-part process to rename directories to kallithea.
This first step is to change all references in the files where they refer
to the old directory name.