hooks: use internal Mercurial module for update after push

Don't rely on having a working 'hg' command in $PATH.

db_manage: drop special DbManage mode for tests

There is no need for a special mode - just use force_ask to answer yes to
"destroy old database".

db_manage: clarify that the purpose of admin_prompt is to create the admin user

Use cli_args to pass the test admin user information so it can be created the
usual way. This removes an unfortunate lib dependency on tests.

isort: upgrade to 5.1.2

The changes to non-top-level imports seem nice.

db: introduce db-create --reuse option

Support use of an existing database so the Kallithea database user doesn't have
to be granted permissions to create databases.

The existing database must of course have been created "correctly", for example
using the right charset and collation on MariaDB/MySQL. Creating the database
manually also provides a way to avoid the hardcoded defaults.

db: refactor to clarify that we always invoke SA create_all with checkfirst=False

create_all(checkfirst=True) would skip creating tables that already exist,
potentially leaving them with wrong schema.

We are strict and want a successful create_all to leave all the tables in a
fully known state. We thus want it to fail if any tables already are present.
Existing tables should not silently be accepted.

mysql: create database with explicit UTF-8 character set and collation

A spin-off from Issue #378.

In MySQL, the character sets for server, database, tables, and connection are
set independently. Ideally, they should all use UTF-8, but systems tend to use
latin1 as default encoding, for example:

character_set_server = latin1
collation_server = latin1_swedish_ci

Databases would thus by default be created as:

character_set_database = latin1
collation_database = latin1_swedish_ci

To make things work consistently anyway, we have so far specified the utf8mb4
charset explicitly when creating tables, but there is no corresponding simple
option for specifying the collation for tables. We need a better solution.

If necessary and possible, the system charset and collation should be set to
UTF-8. Some systems already have these defaults default - see
https://mariadb.com/kb/en/differences-in-mariadb-in-debian-and-ubuntu/ .
The defaults can be changed as described on
https://mariadb.com/kb/en/setting-character-sets-and-collations/#example-changing-the-default-character-set-to-utf-8
to give something like:

character_set_server = utf8mb4
collation_server = utf8mb4_unicode_ci

Databases will then by default be created as:

character_set_database = utf8mb4
collation_database = utf8mb4_unicode_ci

and there is thus no longer any need for specifying the charset when creating
tables.

To be reasonably resilient across all systems without relying on system
defaults, we will now start specifying the charset and collation when creating
the database, but drop the specification of charset when creating tables.

For existing databases, it is recommended to change encoding (and collation) by
altering the database and each of the tables inside it as described on
https://stackoverflow.com/questions/6115612/how-to-convert-an-entire-mysql-database-characterset-and-collation-to-utf-8 .

Note the use of utf8mb4_unicode_ci instead of utf8mb4_general_ci - see
https://stackoverflow.com/questions/766809/whats-the-difference-between-utf8-general-ci-and-utf8-unicode-ci .

For investigation of these issues, consider the output from:
show variables like '%char%';
show variables like '%collation%';
show create database `KALLITHEA_DB_NAME`;
SELECT * FROM information_schema.SCHEMATA WHERE schema_name = "KALLITHEA_DB_NAME";
SELECT * FROM information_schema.TABLES T, information_schema.COLLATION_CHARACTER_SET_APPLICABILITY CCSA WHERE CCSA.collation_name = T.table_collation AND T.table_schema = "KALLITHEA_DB_NAME";

db: better support for databases with "odd" characters in the name, such as "-"

Add missing quoting, using the quoting flavour preferred by the DB.

Tested with

echo "CREATE USER 'kallithea-test'@'localhost' IDENTIFIED BY 'password'"|sudo -u mysql mysql
echo "GRANT ALL PRIVILEGES ON \`kallithea-test\`.* TO 'kallithea-test'@'localhost'" | sudo -u mysql mysql
TEST_DB='mysql://kallithea-test:password@localhost/kallithea-test?charset=utf8mb4' py.test

sudo -u postgres createuser 'kallithea-test' --pwprompt --createdb
TEST_DB='postgresql://kallithea-test:password@localhost/kallithea-test' py.test

db_manage: drop unused fix functions

The functions might be ok, but we don't know when or why to call them ... and
apparently we don't need them.

admin: drop dead code for system update notifications

If we want it, we probably want to do it in a different way.

ssh: show ssh URL on summary page

Original change has been heavily modified by Mads Kiilerich.

hooks: make sure push and pull hooks always are enabled

Don't put things in the database when we pretty much assume they always have
exact content, without any reasonable use case for customization.

locking: drop the pull-to-lock / push-to-unlock functionality

The feature is not worth the maintenance cost. The locking is too coarse and
unflexible with insufficient UI and UX. The implementation is also quite
invasive in tricky areas of the code, and thus high maintenance.
Dropping this will enable other cleanup ... or at least make it easier.

lib: remove unused method 'notify' from DbManage

This method is no longer used.
Last usage was removed in 46db3368c2ae and previously c7ef77ab2f95.

utils2: Move ask_ok out of paster_commands

The ask_ok function is still a mess and should be cleaned up ...

db_manage: remove create_ui_settings method

We already have a loop for adding Ui entries, so why not handle the hook entries
there as well?!
This is also much easier to read.
create_settings is only called from setup_db so querying for existing settings
makes no sense.

hooks: rename hooks to reflect what they are doing

The names of our hooks should reflect what they are doing.
So, since pre_push and pre_pull only handle the locking, rename
pre_push->push_lock_handling and pre_pull->pull_lock_handling.

This imply a database migration step which must be run as described in the
upgrade documentation.

Also rename db.Ui class variables.

less: use .label and .label-* style instead of custom .*tag

This reduces the code and makes it more consistent.

Visualy this results in bold texts (which in case of small fonts is easier to read)
and a bit more padding left and right.

For a proper spacing, this requires to remove whitespaces before and after
the texts inside the labels.

Since the name changed from tag to label, also rename the variables and functions.

codingstyle: replace comparison for equality against None with "is" expression

Both style and correctness.

Reported by flake8.

codingstyle: trivial whitespace fixes

Reported by flake8.

sqlalchemy: remove echo setting from .ini files and create_engine

Instead of 'echo' the normal logging configration should be used.

Follow the advice from
https://docs.sqlalchemy.org/en/rel_1_1/core/engines.html#configuring-logging
which says "when using Python logging, ensure all echo flags are set to False
at all times".

db: drop and re-create old schemas instead of Base.metadata.drop_all()

Sadly, Base.metadata.drop_all() sometimes doesn't work because of foreign key
constraints. We only know this is the case on PostgreSQL and MySQL - use
workarounds for these.

lib: reduce impact of bad import cycle

Follow-up to e0f31c7d0f5e.

It is unfortunate that generic lib code depends on
kallithea.lib.paster_commands.common - it would make more sense if it was the
other way around. We thus move the import down to inside the function that
needs it.

The import of kallithea.lib.paster_commands.common in db_manage triggered its
import of logging.config . middleware.py thus also got logging.config when it
just imported logging and things worked. Without this import of
paster_commands, we also have to fix the import in db_manage.

model: remove BaseModel class

The only remaining purpose of this class was to provide the "sa" field,
allowing a custom SQLAlchemy session to be used for model operations.
However, this never actually worked, nor was it used anywhere.

There's always a global SQLAlchemy session associated with the current
thread; using another session for a single function call does not make
any sense (as sessions cannot be mixed), unless the code works carefully
to ensure the two sessions (and all objects loaded from them) are kept
completely separate. Suffice to say that Kallithea does no such thing,
thus there's no need to pretend to support multiple concurrent sessions.

cleanup: drop superfluous Session.add calls

Session.add should only be called on newly created database objects.

Per the Kallithea contribution guidelines:

When getting an object from the session (via Session().query or
any of the utility functions that look up objects in the database),
it's already part of the session, and should not be added again.

cleanup: do Session.add directly inside _create_default_perms

Let the various _create_default_perms functions add the created object
to the SQLAlchemy database session, instead of having the caller do it.

This is in accordance with the Kallithea contribution guidelines:

When creating an object using a factory function (like create_repo),
the returned object has already (by convention) been added to the
session, and should not be added again.

cleanup: refer less to User.DEFAULT_USER

Down the road we might want to identify the default user in another
way than by username.

model: move code from __init__.py to base.py

Having too much code, in particular too much imports, inside a package's
__init__.py is a recipe for circular imports, and considered bad practice in
Python [1]

Move out everything from kallithea/model/__init__.py to a new file
kallithea/model/base.py and adapt the existing imports.

[1] http://docs.python-guide.org/en/latest/writing/structure/#packages

db: remove superfluous Session.add calls

Don't re-add objects to the SQLAlchemy Session just because they were
modified. Session.add is only for freshly constructed objects that
SQLAlchemy doesn't know about yet.

The rules are quite simple:

When creating a database object by calling the constructor directly, it
must explicitly be added to the session.

When creating an object using a factory function (like "create_repo"),
the returned object has already (by convention) been added to the
session, and should not be added again.

When getting an object from the session (via Session.query or any of the
utility functions that look up objects in the database), it's already
added, and should not be added again. SQLAlchemy notices attribute
modifications automatically for all objects it knows about.

repos: inline RepoModel.update_repoinfo

It had only two invocations, each triggering separate code paths, all
to do a simple loop.

tests: add as little code as possible in __init__.py

kallithea/tests/__init__.py contained quite a lot of code, including the test
base class TestController. This in itself may be considered bad practice.

Specifically, this poses a problem when using pytest 3.0+, in which asserts
in some files are not automatically rewritten to give improved assert
output. That problem can be fixed by explicitly registering such files for
assertion rewriting, but that register call should be executed _before_ said
files are imported. I.e. if the register call is in
kallithea/tests/__init__.py, assert calls in __init__.py itself can not be
rewritten.

Since the TestController base class does effectively contain asserts, and we
do not want to execute the register call from somewhere outside the
kallithea/tests directory, we need to move the TestController class to
another file (kallithea/tests/base.py) so we can have a register call in
__init__.py before loading base.py.

While not strictly necessary to fix the mentioned pytest problem, we take
the opportunity to fully clean __init__.py and move everything to
the new kallithea/tests/base.py. While doing so, unnecessary imports are
removed, and imports are ordered alphabetically. Explicit imports of symbols
from modules that were already imported as a whole, are removed in favor of
fully qualifying the references (e.g. tempfile._RandomNameSequence).

auth: cleanup of EXTERN_TYPE_INTERNAL

Don't set it in top level namespace - it is a weak link between the database
and the actual implementation. Don't make it more than that.

Don't hardcode in that many places that 'internal' is the default - just call
it DEFAULT_AUTH_TYPE.

Don't use it for extern_name - it is only intended for use as extern_type.

Remove unused uses.

db_manage: cleanup and minor refactorings

- because we like it better that way.

hg: drop pointless push_ssl configuration setting - if there is a risk push can be compromised, credentials can also easily be stolen for pull

Everybody should have a ssl-only setup now. Alternatively, there is a use case
for 'only anonymous traffic on ssl - all authentication and authenticated
traffic must be on ssl'. That can be done with proper web server configuration.

paster: split paster specifics out of kallithea.lib.utils

BasePasterCommand and ask_ok are only useful in a Paster/command-line
context, and can thus be removed from the already overly cluttered main
utils module.

(The new common.py has been added to Mercurial as a copy of utils.py,
preserving its file history, but creating a somewhat bewildering diff.)

db: stamp Alembic version into database

This ensures that a freshly created database will have the correct
Alembic version.

db: remove SQLAlchemy Migrate support

The DbMigrateVersion table will be removed in a later changeset.

cleanup: get rid of dn as shortcut for os.path.dirname

We keep 'dirname' as shortcut despite having removed the 'join' shortcut:
* the name is less ambiguous than 'join'.
* dirname is often applied multiple times - spelling it out would be too
verbose.

cleanup: get rid of jn as shortcut for os.path.join

It is not used in that many places so it is better to be explicit and use os.path.join .

Discussed on https://bitbucket.org/domruf/kallithea/commits/1da05f42bca3f7042c444fda13a1ae1f6b9c300f#comment-2948124 .

Modified by Mads Kiilerich.

cleanup: remove unused imports

Found with pyflakes.

cleanup: avoid some 'except Exception' catching - catch specific exceptions or log it and show what happened

This has a risk of introducing regressions ... but we want to get rid of all
exception muting and make the whole system less fragile and easier to debug.

db_manage: don't ask for confirmation before using symlink - just warn

Mostly a backout of bbaf0b86a1fe ... but we still use the normalized path.

Correct licensing information in individual files.

The top-level license file is now LICENSE.md.

Also, in various places where there should have been joint copyright holders
listed, a single copyright holder was listed. It does not appear easy to add
a link to a large list of copyright holders in these places, so it simply
refers to the fact that various authors hold copyright.

In future, if an easy method is discovered to link to a list from those
places, we should do so.

Finally, text is added to LICENSE.md to point to where the full list of
copyright holders is, and that Kallithea as a whole is GPLv3'd.

Second step in two-part process to rename directories.
This is the actual directory rename.