db: read hgrc in Repository.scm_instance_no_cache and Gist.scm_instance

This will make sure .hg/hgrc is read in some code paths where it was missing.
That might make a difference in some real world cases.

hooks: be more consistent in only using active Ui entries

There is no UI to control or display the ui_active value for custom hooks, but
*if* they are inactive, they will be ignored in make_ui, and it will be
misleading and confusing to show them in the list of active custom hooks.

There *should* never be any inactive hooks entries, but let's be consistent in
handling the case *if* it should happen. (It happened for me while hacking
around.)

hooks: put repo_size and update hooks in kallithea namespace

Keep things separate.

Include missing migration steps for 642847355a10.

repo_groups: extra escape of names when used in select drop-downs

The lack of escaping could be a problem *if* it was possible to create repo
groups with dangerous names.

This was seen for example when specifying parent group of repos and repo
groups.

We want to keep groups_choices as HTML literals so paths can use » as
separator.

imports: try to use global imports unless it is a layering violation

To minimize the impact, do imports that violate the layering at runtime instead
of at import time.

lib: move webhelpers2 and friends to webutils

Gives less of the unfortunate use of helpers - especially in low level libs.

lib: move canonical_url & co to webutils

This gives less of the unfortunate use of helpers - especially in model.

lib: consistently use webutils.url instead of h.url

Templates are still the valid use case for accessing through h.

Simplify mocking of url() function.

db: consistently use the name settings for the return value from get_app_settings

Get rid of yet another old rc reference.

(The "settings" in auth is slightly different, but pretty much the same.)

db: move URL_SEP constant from db to top level kallithea module

URL_SEP is used in several places - having it in the db module is too high
level.

__init__ might not be the best place for this, but it does no harm to the
dependencies graph to place constants there.

model: stop using 'Optional' class in Setting.create_or_update()

The 'Optional' class can help to identify whether a value is set by the
caller or a default value.

But if you can make the assumption that the caller will never specify a
'None' value as 'real' value, then the Optional class is not really
necessary.

As this usage in the Setting class is the only remaining usage of
'Optional', it does not seem worth it to keep it.

db: introduce constraint ensuring no duplicate (reviewer, pullrequest) combinations

A reviewer should only be added once to a review.

Previously, this was not ensured by the database itself, although that the
controller would try to not add duplicate reviewers. But there was no hard
guarantee: e.g. simultaneous adding of the same reviewer to the same review
by a review owner and admin, a framework bug that sends the same request
twice, ... could still trigger duplicate addition. Additionally, code
changes (e.g. a new API) could introduce bugs at the controller level.

Existing production databases were found to contain such duplicate entries.
Nevertheless, as the code displaying reviewers in a pull request filtered
out duplicates, this never showed in the UI, and never was a 'real' problem.

Add a UniqueConstraint in the database to prevent such entries, with a
database migration step that will first find and remove existing duplicates.

api: extend get_pullrequest data with created_on and updated_on fields

The get_pullrequest API call returned most attributes of a pullrequest, but
not created_on and updated_on.

Set microseconds to 0, as done for modified_at in ChangesetStatus. See
commit 71033bd37b4c.

db: drop unused Setting.DEFAULT_UPDATE_URL

References were removed in 62822054a1e2.

ssh: drop simple key parser for fingerprint calculation in public_key setter - use ssh.parse_pub_key

Avoid having multiple slightly different parsers.

isort: upgrade to 5.1.2

The changes to non-top-level imports seem nice.

mysql: create database with explicit UTF-8 character set and collation

A spin-off from Issue #378.

In MySQL, the character sets for server, database, tables, and connection are
set independently. Ideally, they should all use UTF-8, but systems tend to use
latin1 as default encoding, for example:

character_set_server = latin1
collation_server = latin1_swedish_ci

Databases would thus by default be created as:

character_set_database = latin1
collation_database = latin1_swedish_ci

To make things work consistently anyway, we have so far specified the utf8mb4
charset explicitly when creating tables, but there is no corresponding simple
option for specifying the collation for tables. We need a better solution.

If necessary and possible, the system charset and collation should be set to
UTF-8. Some systems already have these defaults default - see
https://mariadb.com/kb/en/differences-in-mariadb-in-debian-and-ubuntu/ .
The defaults can be changed as described on
https://mariadb.com/kb/en/setting-character-sets-and-collations/#example-changing-the-default-character-set-to-utf-8
to give something like:

character_set_server = utf8mb4
collation_server = utf8mb4_unicode_ci

Databases will then by default be created as:

character_set_database = utf8mb4
collation_database = utf8mb4_unicode_ci

and there is thus no longer any need for specifying the charset when creating
tables.

To be reasonably resilient across all systems without relying on system
defaults, we will now start specifying the charset and collation when creating
the database, but drop the specification of charset when creating tables.

For existing databases, it is recommended to change encoding (and collation) by
altering the database and each of the tables inside it as described on
https://stackoverflow.com/questions/6115612/how-to-convert-an-entire-mysql-database-characterset-and-collation-to-utf-8 .

Note the use of utf8mb4_unicode_ci instead of utf8mb4_general_ci - see
https://stackoverflow.com/questions/766809/whats-the-difference-between-utf8-general-ci-and-utf8-unicode-ci .

For investigation of these issues, consider the output from:
show variables like '%char%';
show variables like '%collation%';
show create database `KALLITHEA_DB_NAME`;
SELECT * FROM information_schema.SCHEMATA WHERE schema_name = "KALLITHEA_DB_NAME";
SELECT * FROM information_schema.TABLES T, information_schema.COLLATION_CHARACTER_SET_APPLICABILITY CCSA WHERE CCSA.collation_name = T.table_collation AND T.table_schema = "KALLITHEA_DB_NAME";

mysql: bump charset to to 'utf8mb4' to get full UTF-8 support

We used to use 'utf8', but in MySQL, this is just an alias for 'utf8mb3' which
isn't full unicode. 'utf8mb4' has less surprises (especially if used with the
'utf8mb4_unicode_ci' collation).

MySQL character sets for server, database, tables, and connection are set
independently. Until now, we have specified 'utf8' when creating tables to
overrule the database charset and in the default MySQL connection URL.

For existing databases, it is recommended to change encoding (and collation) by
altering the each of the tables inside it as described on
https://stackoverflow.com/questions/6115612/how-to-convert-an-entire-mysql-database-characterset-and-collation-to-utf-8 .

db: drop unused _hash_key

A bit less md5 ...

permissions: drop hg.create.write_on_repogroup "Repository creation with group write access" setting

Simplify permissions system and get rid of some confusing tech debt.

Before, the global 'write_on_repogroup' setting controlled what write
permission on a repo group meant.

With this change, users can create repositories in a repo group if and only if
they have write access. Write access to a repo group will now mean the
permission to create repositories in it.

Write access to repo groups must be granted explicitly. There should not be any
other reason to grant write access than to allow users to create repos. There
is thus no upgrade concerns for this change.

An admin that doesn't want users to create repos in a repogroup should just not
give them write access.

These global settings might still exist in the database, but is ignored and no
longer used and do no harm.

lib: drop own asbool implementation and consistently use tg.support.converters as utils2.asbool

str2bool never reported error on odd input such as '' or '-1', but the tg
asbool behaviour of raising ValueError("String is not true/false: %r" % obj) in
that case seems fine.

db: drop double specification of unique constraint (Issue #374)

Consistently use "unique=True" for single column constraints. Only use
UniqueConstraint for composite constraints.

MySQL did not like this double specification, while PostgreSQL and SQLite
did not care.

db: fix bad characters in __repr__

Introduced in effd091203ae.

db: drop Repository CacheInvalidation

The benefit of this functionality is questionable. Especially in bigger setups
with multiple front-end instances all serving the same multitude of
repositories, making the hit rate very low. And the overhead of storing cache
invalidation data *in* the database is non-trivial.

We preserve a small cache in Repository SA records, but should probably just in
general know what we are doing and not ask for the same information multiple
times in each request.

db: drop SA caching_query and FromCache, and thus sql_cache_short beaker cache

It is not a good idea to have dead ORM objects. If we want caching, we should
do it explicit.

It is unknown how much this cache helps, but we can profile and introduce
better caching of simple data where relevant.

db: consistently use base_path from config instead of repeatedly getting from the database

Avoid using the inefficient Repository.base_path ... and avoid even more the
misleading Repository.repo_path .

auth: for default permissions, use existing explicit query result values instead of following dot references in ORM result objects

There has been reports of spurious crashes on resolving references like
.repository from Permissions:

File ".../kallithea/lib/auth.py", line 678, in __wrapper
if self.check_permissions(user):
File ".../kallithea/lib/auth.py", line 718, in check_permissions
return user.has_repository_permission_level(repo_name, self.required_perm)
File ".../kallithea/lib/auth.py", line 450, in has_repository_permission_level
actual_perm = self.permissions['repositories'].get(repo_name)
File ".../kallithea/lib/vcs/utils/lazy.py", line 41, in __get__
value = self._func(obj)
File ".../kallithea/lib/auth.py", line 442, in permissions
return self.__get_perms(user=self, cache=False)
File ".../kallithea/lib/auth.py", line 498, in __get_perms
return compute(user_id, user_is_admin)
File ".../kallithea/lib/auth.py", line 190, in _cached_perms_data
r_k = perm.UserRepoToPerm.repository.repo_name
File ".../sqlalchemy/orm/attributes.py", line 285, in __get__
return self.impl.get(instance_state(instance), dict_)
File ".../sqlalchemy/orm/attributes.py", line 721, in get
value = self.callable_(state, passive)
File ".../sqlalchemy/orm/strategies.py", line 710, in _load_for_state
% (orm_util.state_str(state), self.key)

sqlalchemy.orm.exc.DetachedInstanceError: Parent instance <UserRepoToPerm at ...> is not bound to a Session; lazy load operation of attribute 'repository' cannot proceed (Background on this error at: http://sqlalche.me/e/bhk3)

Permissions are cached between requests: SA result records are stored in in
beaker.cache.sql_cache_short and resued in following requests after the initial
session as been removed. References in Permission objects would usually give
lazy lookup ... but not outside the original session, where we would get an
error like this.

Permissions are indeed implemented/used incorrectly. That might explain a part
of the problem. Even if not fully explaining or fixing this problem, it is
still worth fixing:

Permissions are fetched from the database using Session().query with multiple
class/table names (joined together in way that happens to match the references
specified in the table definitions) - including Repository. The results are
thus "structs" with selected objects. If repositories always were retrieved
using this selected repository, everything would be fine. In some places, this
was what we did.

But in some places, the code happened to do what was more intuitive: just use
.repository and rely on "lazy" resolving. SA was not aware that this one
already was present in the result struct, and would try to fetch it again. Best
case, that could be inefficient. Worst case, it would fail as we see here.

Fix this by only querying from one table but use the "joinedload" option to
also fetch other referenced tables in the same select. (This might
inefficiently return the main record multiple times ... but that was already
the case with the previous approach.)

This change is thus doing multiple things with circular dependencies that can't
be split up in minor parts without taking detours:

The existing repository join like:
.join((Repository, UserGroupRepoToPerm.repository_id == Repository.repo_id))
is thus replaced by:
.options(joinedload(UserGroupRepoToPerm.repository))

Since we only are doing Session.query() on one table, the results will be of
that type instead of "structs" with multiple objects. If only querying for
UserRepoToPerm this means:
- perm.UserRepoToPerm.repository becomes perm.repository
- perm.Permission.permission_name looked at the explicitly queried Permission
in the result struct - instead it should look in the the dereferenced
repository as perm.permission.permission_name

db: drop superfluous space in UserGroup relationship definition

It doesn't seem like it had any actual impact.

user: make get_by_username_or_email default to treat username case insensitive

The get_by_username_or_email is a flexible function, intended to find users in
multiple ways, suitable for login prompts. The function was sometimes used
with case sensitive user lookup, sometimes without. Instead, be consistent and
just default to be insensitive.

hg: always show and run Mercurial hooks in alphabetical order (Issue #246)

Mercurial will generally run hooks in the order they are found in the
configuration. For entries found in the database, there is no such order.
Instead, always use alphabetical order for these.

Since we now want to order things explicitly in the db query, we want an index
with a composite key. We do that even though we don't really need it for the
few entries in this table, and even though it might/could use the same index as
the existing unique constraint. This composite UniqueConstraint was added in
b9f4b444a172 where it replaced a wrong UniqueConstraint that could/should have
been removed in c25191aadf92. Fix that while touching this area and running a
migration script.

scripts: drop isort --wrap-length 160 - it is broken with py3 and not really necessary

Under Python 3, isort 4.3.21 failed with
https://github.com/timothycrosley/isort/issues/741 on
kallithea/controllers/api/api.py :

Traceback (most recent call last):
File "data/env/bin/isort", line 10, in <module>
sys.exit(main())
File ".../env/lib64/python3.7/site-packages/isort/main.py", line 379, in main
for sort_attempt in attempt_iterator:
File ".../env/lib64/python3.7/site-packages/isort/main.py", line 377, in <genexpr>
attempt_iterator = (sort_imports(file_name, **arguments) for file_name in file_names)
File ".../env/lib64/python3.7/site-packages/isort/main.py", line 88, in sort_imports
result = SortImports(file_name, **arguments)
File ".../env/lib64/python3.7/site-packages/isort/isort.py", line 207, in __init__
self._add_formatted_imports()
File ".../env/lib64/python3.7/site-packages/isort/isort.py", line 606, in _add_formatted_imports
self._add_from_imports(from_modules, section, section_output, sort_ignore_case)
File ".../env/lib64/python3.7/site-packages/isort/isort.py", line 526, in _add_from_imports
import_statement = self._multi_line_reformat(import_start, from_import_section, comments)
File ".../env/lib64/python3.7/site-packages/isort/isort.py", line 552, in _multi_line_reformat
dynamic_indent, indent, line_length, comments)
File ".../env/lib64/python3.7/site-packages/isort/isort.py", line 705, in _output_grid
if len(next_statement.split(self.line_separator)[-1]) + 1 > line_length:
TypeError: '>' not supported between instances of 'int' and 'str'

py3: remove safe_unicode in places where it no longer is needed because all strings (except bytes) already *are* unicode strings

(The remaining safe_unicode calls are still needed and can't just be removed,
generally because we in these cases still have to convert from bytes to unicode
strings.)

py3: trivial renaming of .iteritems() to .items()

A bit like "2to3 -f dict", but we don't want list().

py3: drop support for long - just use int instead

From 2to3 long.

lib: clean up ext_json and how it is used - avoid monkey patching

Note that py3 json.dumps will return ASCII (with all unicode escaped) encoded
as str. But we generally want JSON as bytes (which json.loads also can read),
so also wrap the result with ascii_bytes in many places.

ssh: drop usk_public_key_idx again

Essentially a backout of d2a97f73fa1f and the
4851d15bc437_db_migration_step_after_95c01895c006_ alembic step.

We can't reliably have full index on fields with unbounded length. The
upgrade step has been reported to fail on MySQL [1]:

sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError)
(1170, "BLOB/TEXT column 'public_key' used in key specification without
a key length") [SQL: u'CREATE INDEX usk_public_key_idx ON user_ssh_keys
(public_key)'] (Background on this error at: http://sqlalche.me/e/e3q8)

And we really don't need this index ... especially now when we use
fingerprints for key deletion instead of looking up by the full public key.


[1] https://lists.sfconservancy.org/pipermail/kallithea-general/2019q4/003068.html

cleanup: fix db.py "import *"

pyflakes rightfully complains about wildcard imports. But we like to keep the
short names, so let's just be explicit about which names we import.

py3: drop __unicode__ ... and generally move towards just providing a good __repr__

We should never show str() (or similar) to end users.

py3: drop .keys when we don't need them

In python 3 they will be iterators and mostly useless ... but they are already
mostly redundant in py2.

cleanup: minor refactorings and simplification of dict usage

Makes it more py3 compatible.

py3: cleanup map usage and avoid py3 ambiguity

Based on 2to3 -f map ... but replace map with something more explicit (unless
born and raised in a lisp world) (but sometimes slightly more verbose).

cleanup: use isinstance instead of comparing types

From 2to3 idioms.

utils: drop make_ui parameter clear_session - this utility function should never mess with sessions

clear_session seems to have been based on the idea that utility functions that
happened to create a session also should clean up. But instead, we now
consistently take care of session removal at a high level.

helpers: use WebHelpers2 as much as possible - it supports Python3, and WebHelpers is dead

The remaining uses of WebHelpers have to be replaced too - see
https://webhelpers2.readthedocs.io/en/latest/migrate.html .

ssh: show ssh URL on summary page

Original change has been heavily modified by Mads Kiilerich.

ssh: db models for ssh key management

Add database components for SSH based access. Actual use of this will be added
soon.

The work in this commit is based heavily off of the existing API key
code for the sake of consistency.

The original code has been heavily modified by Mads Kiilerich.

Updates to use User.guess_instance by Anton Schur <tonich.sh@gmail.com>.

clone_url: simplify the logic - move summary handling of different URLs with/without id to db

Drop the half-baked concept of a separate "clone_uri_by_id" - just always use a
single template and change back and forth between {repo} and _{repoid} as
necessary.

Make it clear that clone_uri_tmpl always is passed as argument.

clone_url: always pass a clone_uri_tmpl, with Repository.DEFAULT_CLONE_URI as last resort

clone_url() had a layering violation of using c.clone_uri_tmpl . This
refactoring now makes it clear that this only was used from
PullRequest.__json__(), so move the hack there and simplify it.

clone_url: clarify variable naming

Minor nits:
user -> username (temporarily more complex)
uri_tmpl -> clone_uri_tmpl
qualified_home_url -> prefix_url
decoded_path -> prefix

db: drop __mapper_args__ that cause SQLAlchemy deprecation warnings

We don't really need them. There were only few uses left, and we seem to be
using .order_by() in all places where we really care about the ordering.

Gets rid of

<string>:2
<string>:2: SADeprecationWarning: Mapper.order_by is deprecated.Use Query.order_by() in order to affect the ordering of ORM result sets.

auth: make User.get_by_api_key more strict about only returning active non-default users

Thus drop some extra checks against default user.

auth: global permissions given to the default user are the bare minimum and should apply to *all* other users too

Drop the "subtractive permission" config option "inherit_from_default" that
when set to false would give users less global permissions than the default
unauthenticated user.

Instead, think positive and merge all positive permissions.

At the end, filter the global permissions to make sure we for each kind of
permissions only keep the one with most weight.

auth: minor code improvements around global permission

Use immutable lists for constants ... just because we can.

Fix trailing dot in check for hg.create.write_on_repogroup.

Add missing (but so far unused) global permissions to PERM_WEIGHTS.

utils: refactor make_ui to always read from db, optionally also augmenting with hgrc content

simplehg was the only user of reading hgrc content ... and it would rather just
have everything at once.

The implementation in utils is moved around and re-indented, but without
significant changes.

hooks: make sure push and pull hooks always are enabled

Don't put things in the database when we pretty much assume they always have
exact content, without any reasonable use case for customization.

locking: drop the pull-to-lock / push-to-unlock functionality

The feature is not worth the maintenance cost. The locking is too coarse and
unflexible with insufficient UI and UX. The implementation is also quite
invasive in tricky areas of the code, and thus high maintenance.
Dropping this will enable other cleanup ... or at least make it easier.

auth: drop confusing and layering-violating User.AuthUser property

Keep it simple and just explicitly create an AuthUser if "needed". That can
perhaps be simplified further later on.

model: move notification types from Notification to NotificationModel

This commit is part of the removal of the UI notification feature from
Kallithea, which is not deemed useful in its current form. Only email
notifications are preserved.

As there is no database storage of notifications anymore, the Notification
class will be removed. However, the notification type definitions are still
used for email notifications, and need to live somewhere. As creating
notifications is always passing via NotificationModel, it makes sense to
move the types there.

model: remove UI notification feature

This commit is part of the removal of the UI notification feature from
Kallithea, which is not deemed useful in its current form. Only email
notifications are preserved.

db: drop constraint that started failing with MariaDB 10.2 / MySQL 5.7 (Issue #324)

The constraint was to prevent simple recursive parent references, but it only
checked direct parents. We thus have to rely on the high level application
maintaining the invariant anyway.

hooks: rename hooks to reflect what they are doing

The names of our hooks should reflect what they are doing.
So, since pre_push and pre_pull only handle the locking, rename
pre_push->push_lock_handling and pre_pull->pull_lock_handling.

This imply a database migration step which must be run as described in the
upgrade documentation.

Also rename db.Ui class variables.

codingstyle: trivial whitespace fixes

Reported by flake8.

api: change precision of ChangesetStatus.modified_at to seconds

Per default MySQL only uses seconds. So in order to be consistent on all
databases, this is the easiest solution. I don't think the microseconds are
necessary. And AFAICS mercurial only uses seconds for the changeset
modification time as well. So why should we use microseconds for ChangesetStatus.

Without this, for example test_api_get_changeset_with_reviews fails, because of
datetime mismatch if MySQL is used.

api: add get_pullrequest and comment_pullrequest methods

Modified by Mads Kiilerich, mainly to let the test helper function
create_pullrequest use model directly.

api: add optional branches, tags and bookmarks information to get_repo API data

Modified by Mads Kiilerich to make these revision names optional - there can be
a lot of them.

Add JSON-RPC API for reading changeset status

Allows reading a changeset status as json. This is useful for e.g. reporting,
extracting review status for all changesets between two releases.