auth: extract private parts from get_user_permissions

Prepare for splitting it up.

auth: compute AuthUser.permissions lazily

Prepare for computing each kind of permissions separately so they can be
skipped in the common case.

This will undo a part of 1ecd6c0e2787 where we took the detour of assigning
permissions in __init__ - now we move it back to a LazyProperty, but in a
better way.

config: only assign kallitha.CONFIG once and keep it as a plain dict

Mute pytype warnings and make code more readable for developers:

File "kallithea/lib/hooks.py", line 318, in _hook_environment: No attribute 'global_conf' on Dict[nothing, nothing] [attribute-error]
File "kallithea/lib/hooks.py", line 318, in _hook_environment: No attribute 'local_conf' on Dict[nothing, nothing] [attribute-error]
File "kallithea/bin/kallithea_cli_base.py", line 82, in runtime_wrapper: No attribute 'global_conf' on Dict[nothing, nothing] [attribute-error]
File "kallithea/bin/kallithea_cli_base.py", line 82, in runtime_wrapper: No attribute 'local_conf' on Dict[nothing, nothing] [attribute-error]

cli: refactor db_create to avoid app initialization in multiple places

Make sure kallithea.config.application.make_app only is invoked at a high level
... and avoid references to kallithea.CONFIG.global_conf right before
kallithea.CONFIG is set to a plain dict again.

Click commands that have both needs_config_file=True and
config_file_initialize_app=True will now be called twice - first time before
setting kallithea.CONFIG but with the config dict as parameter.

That seems kind of intuitive and will simplify the code and allow cleanup of
config handling.

Mute pytype warning:

File "kallithea/bin/kallithea_cli_db.py", line 73, in db_create: No attribute 'global_conf' on Dict[nothing, nothing] [attribute-error]
File "kallithea/bin/kallithea_cli_db.py", line 73, in db_create: No attribute 'local_conf' on Dict[nothing, nothing] [attribute-error]

ssh: update authorized_keys after deleting a user with ssh keys

This fixes a minor problem with minimal impact: The authorized_keys entries
would fail anyway when the referenced user wasn't found ... and the entries
would go away next time authorized_keys for some reason were updated.

Reported by Louis Bertrand.

docs: reduce double nesting level in performance.rst

Rendered text was too far to the right.

docs: fix display of 'note' in installation

Due to insufficient indentation on some lines, the note was displayed
incorrectly.

auth: refactor permissions

Avoid using complex vague typing in dict-of-dicts.

vcs: fix subprocessio handling of hack for handling end-of-stream

Address pytype warning:
File "kallithea/lib/vcs/subprocessio.py", line 383, in __next__: No attribute 'stop' on listiterator[bytes] [attribute-error]

ini: move archive_cache_dir to the data directory

Give less data locations to care about.

docs: clarify how to activate virtualenv from mod_wsgi with py3

activate_this.py is not available with the venv module in py3. According to
https://modwsgi.readthedocs.io/en/develop/user-guides/virtual-environments.html
we just need to set python-home - that is the way to do proper virtualenv
activation, and there is no need for hacks in the wrapper script.

docs: contributing: use '.' rather than 'source' to align with 'installation'

See commit 855ba1f07aeb437483138d48d7929f0ff8c38b67 .

docs: drop changing cwd in WSGI wrapper script

It was introduced in 5a31d387f347 and has never been in the virtualenv version
of the WSGI script, which "proves" it really is unnecessary.

docs: clean up readthedocs URLs

It will redirect to the canonical URL https://*.readthedocs.io/en/latest , and
there is no point in linking to /en/latest .

tg: fix launching with kajiki after moving to FullStackApplicationConfigurator 27d9ca0c8381

Application start failed with:
TypeError: 'FullStackApplicationConfigurator' object is not subscriptable

py3: fix rc_types error

Spotted by pytypes:

File "kallithea/bin/base.py", line 133, in make_config: Function BinaryIO.write was called with the wrong arguments [wrong-arg-types]
Expected: (self, s: Union[bytearray, bytes, memoryview])
Actually passed: (self, s: str)

vcs: fix cloning remote repository with HTTP authentication (Issue #379)

Using a remote clone URI of
http://user:pass@host/...
triggered an exception:

...
E File ".../kallithea/lib/utils.py", line 256, in is_valid_repo_uri
E GitRepository._check_url(url)
E File ".../kallithea/lib/vcs/backends/git/repository.py", line 183, in _check_url
E passmgr.add_password(*authinfo)
E File "/usr/lib/python3.7/urllib/request.py", line 848, in add_password
E self.reduce_uri(u, default_port) for u in uri)
E File "/usr/lib/python3.7/urllib/request.py", line 848, in <genexpr>
E self.reduce_uri(u, default_port) for u in uri)
E File "/usr/lib/python3.7/urllib/request.py", line 875, in reduce_uri
E host, port = splitport(authority)
E File "/usr/lib/python3.7/urllib/parse.py", line 1022, in splitport
E match = _portprog.fullmatch(host)
E TypeError: cannot use a string pattern on a bytes-like object


The authinfo tuple is obtained via mercurial.util.url, which unfortunately
returns a tuple of bytes whereas urllib expects strings.
It seems that mercurial internally has some more hacking around urllib as
urllibcompat.py, which we don't use.

Therefore, transform the bytes into strings before passing authinfo to
urllib. As the realm can be None, we need to check it specifically otherwise
safe_str would return a string 'None'.

A basic test that catches the mentioned problem is added, even though it
does not actually test that cloning with auth info will actually work (it
only tests that it fails cleanly if the URI is not reachable).

Additionally, one use of 'test_uri' in hg/repository.py still needed to be
transformed from bytes to string. For git this was already ok.

vcs: extract get_urllib_request_handlers out of {hg,git}/repository.py

Avoid some duplication between hg and git backends.
A subsequent commit will need bugfixes in this area.

mercurial: support Mercurial 5.5

No release notes yet, but testing shows it working with a few changes.

A few tests need updating because error messages from remote ssh like our
sys.stderr.write('abort: %s\n' % error)
like for
remote: abort: Access to %r denied
will now appear on stderr instead of stdout - a very reasonable bug fix.

isort: upgrade to 5.1.2

The changes to non-top-level imports seem nice.

scm: refactor install_git_hooks

Rename, simplify, and negate some logic to make the flow more readable to me
and give better logging.

For example, "force_create" were more about "force overwrite". Calling it
"force" is more precise.

mysql: bump sqlalchemy.url MariaDB/MySQL charset to to 'utf8mb4' to get full UTF-8 support

The change in 210e76d69b62 only changed character_set_database, as shown by output after:

--- a/kallithea/model/base.py
+++ b/kallithea/model/base.py
@@ -46,3 +46,8 @@ def init_model(engine):
engine_str = obfuscate_url_pw(str(engine.url))
log.info("initializing db for %s", engine_str)
meta.Base.metadata.bind = engine
+
+ meta.Session.configure(bind=engine)
+ for a, b in meta.Session().execute('''show variables''').fetchall():
+ if 'character_set_' in a:
+ print(a, repr(b))

Before, with charset=utf8, the utf8mb3 charset was used all the way through the stack:

[kallithea.model.base] initializing db for mysql://kallithea-test:XXXXX@localhost/kallithea-test?charset=utf8
character_set_client 'utf8'
character_set_connection 'utf8'
character_set_database 'utf8mb4'
character_set_filesystem 'binary'
character_set_results 'utf8'
character_set_server 'latin1'
character_set_system 'utf8'

With explicit charset=utf8mb4:

[kallithea.model.base] initializing db for mysql://kallithea-test:XXXXX@localhost/kallithea-test?charset=utf8mb4
character_set_client 'utf8mb4'
character_set_connection 'utf8mb4'
character_set_database 'utf8mb4'
character_set_filesystem 'binary'
character_set_results 'utf8mb4'
character_set_server 'latin1'
character_set_system 'utf8'

tests: introduce REUSE_TEST_DB as an alternative to TEST_DB

With TEST_DB, the specified database would be dropped (if existing) and created
from scratch. That would fail if the user only had been granted access to that
database without general database creation permissions.

With REUSE_TEST_DB, the database must exist, and the tables in the existing
database will be deleted before creating the new ones.

This is for example useful for testing on PostgreSQL without full database
privileges. (With MariaDB/MySQL, it is possible to grant create permissions to
a database before it exists, so it is easier to use TEST_DB.)

db: introduce db-create --reuse option

Support use of an existing database so the Kallithea database user doesn't have
to be granted permissions to create databases.

The existing database must of course have been created "correctly", for example
using the right charset and collation on MariaDB/MySQL. Creating the database
manually also provides a way to avoid the hardcoded defaults.

db: refactor to clarify that we always invoke SA create_all with checkfirst=False

create_all(checkfirst=True) would skip creating tables that already exist,
potentially leaving them with wrong schema.

We are strict and want a successful create_all to leave all the tables in a
fully known state. We thus want it to fail if any tables already are present.
Existing tables should not silently be accepted.

mysql: create database with explicit UTF-8 character set and collation

A spin-off from Issue #378.

In MySQL, the character sets for server, database, tables, and connection are
set independently. Ideally, they should all use UTF-8, but systems tend to use
latin1 as default encoding, for example:

character_set_server = latin1
collation_server = latin1_swedish_ci

Databases would thus by default be created as:

character_set_database = latin1
collation_database = latin1_swedish_ci

To make things work consistently anyway, we have so far specified the utf8mb4
charset explicitly when creating tables, but there is no corresponding simple
option for specifying the collation for tables. We need a better solution.

If necessary and possible, the system charset and collation should be set to
UTF-8. Some systems already have these defaults default - see
https://mariadb.com/kb/en/differences-in-mariadb-in-debian-and-ubuntu/ .
The defaults can be changed as described on
https://mariadb.com/kb/en/setting-character-sets-and-collations/#example-changing-the-default-character-set-to-utf-8
to give something like:

character_set_server = utf8mb4
collation_server = utf8mb4_unicode_ci

Databases will then by default be created as:

character_set_database = utf8mb4
collation_database = utf8mb4_unicode_ci

and there is thus no longer any need for specifying the charset when creating
tables.

To be reasonably resilient across all systems without relying on system
defaults, we will now start specifying the charset and collation when creating
the database, but drop the specification of charset when creating tables.

For existing databases, it is recommended to change encoding (and collation) by
altering the database and each of the tables inside it as described on
https://stackoverflow.com/questions/6115612/how-to-convert-an-entire-mysql-database-characterset-and-collation-to-utf-8 .

Note the use of utf8mb4_unicode_ci instead of utf8mb4_general_ci - see
https://stackoverflow.com/questions/766809/whats-the-difference-between-utf8-general-ci-and-utf8-unicode-ci .

For investigation of these issues, consider the output from:
show variables like '%char%';
show variables like '%collation%';
show create database `KALLITHEA_DB_NAME`;
SELECT * FROM information_schema.SCHEMATA WHERE schema_name = "KALLITHEA_DB_NAME";
SELECT * FROM information_schema.TABLES T, information_schema.COLLATION_CHARACTER_SET_APPLICABILITY CCSA WHERE CCSA.collation_name = T.table_collation AND T.table_schema = "KALLITHEA_DB_NAME";

mysql: bump charset to to 'utf8mb4' to get full UTF-8 support

We used to use 'utf8', but in MySQL, this is just an alias for 'utf8mb3' which
isn't full unicode. 'utf8mb4' has less surprises (especially if used with the
'utf8mb4_unicode_ci' collation).

MySQL character sets for server, database, tables, and connection are set
independently. Until now, we have specified 'utf8' when creating tables to
overrule the database charset and in the default MySQL connection URL.

For existing databases, it is recommended to change encoding (and collation) by
altering the each of the tables inside it as described on
https://stackoverflow.com/questions/6115612/how-to-convert-an-entire-mysql-database-characterset-and-collation-to-utf-8 .

tests: fix test_vcs_operations for MariaDB/MySQL

This is apparently only needed for MariaDB/MySQL where it will fix tests that
were failing due to missing action log entries.

tests: consistent use of more helpful raise_exception "fixture"

"Total Crash" was too misleading when debugging problems.

diff: allow commenting on meta lines (Issue #329)

There is no good reason it only should be possible to comment on content lines.
Other lines might not have an obvious locator, but we can live with that as
long as each comment only apply in one place.

With this, we actually want the comment bubble on all lines with bubble markup,
so we can loosen the css selector.

diff: refactor as_html with separate code paths for diff lines with and without line number

Just inlining and factoring and moving things around.

diff: drop per file ignore-whitespace and context - it didn't work and had conceptual issue (Issue #344)

Diffs are currently generated at the low level as one big diff between two vcs
resisions, provided global values for diff context size and flag for ignoring
whitespace. All files use the same flags. There is no way to actually compute
the full diff using these use per file flags, and no simple and efficient way
to add it.

The best option is thus to drop the failed attempt at making it per file, and
just rely on the simple global flags in the URL.

The links for changing whitespace and context is sometimes shown for the whole
"page", and sometimes next to the diff for one file. For now, keep showing the
link in these places, but make sure it navigates back to the FID of the section
where the link was clicked.

The implementation is completely rewritten and moved to a more appropriate
location in helpers.

With a more clean implementation, we also consistently use the simple getters
to extract values from the URL.

diff: fix ignorews/context link to use the right target as anchor

The value in url_fid might not be a valid anchor.

For changesets, url_fid would be like 'C--9c390eb52cd6' even though the actual
target included the changeset hash and were like 'C-1536d03b4869-9c390eb52cd6'.

For pullrequests and compare, it wouldn't link to anything at all, even though
there was a target like 'C--56535da5df40'.

Instead, pass id_fid as anchor value as a separate argument. That one is a
valid anchor.

diff: fix "Show inline comments"

jQuery has weak "friendly" semantics for data attributes and do for example
interpret the data attribute name 'id_for' as 'idFor'. We could thus not
retrieve it as 'id_for'.

Instead, avoid the special semantics of '_' by just naming the data attribute
'for'.

docs: augment setup description with more details of http server and database

Dive into more details than in overview.rst .

Databases should perhaps have more mentioning in overview.rst , but there is
nothing but details and thus not much to say in the overview ...

docs: drop odd verbose note on writeable root path

It was introduced back in a60cd29ba7e2.

db: better support for databases with "odd" characters in the name, such as "-"

Add missing quoting, using the quoting flavour preferred by the DB.

Tested with

echo "CREATE USER 'kallithea-test'@'localhost' IDENTIFIED BY 'password'"|sudo -u mysql mysql
echo "GRANT ALL PRIVILEGES ON \`kallithea-test\`.* TO 'kallithea-test'@'localhost'" | sudo -u mysql mysql
TEST_DB='mysql://kallithea-test:password@localhost/kallithea-test?charset=utf8mb4' py.test

sudo -u postgres createuser 'kallithea-test' --pwprompt --createdb
TEST_DB='postgresql://kallithea-test:password@localhost/kallithea-test' py.test

diff: fix width of line number columns

Column widths did in some cases break with 32757d5e9d0b. The markup was too
magic to find and fix the exact root cause, but we fix it by refactoring to a
slightly different approach:

Use fixed table-layout to be able to control columns width from the first row,
without too much auto sizing.

Instead of using padding, put the line number in a centered block-inline with
right-align and min-width. The numbers will thus generally be right aligned,
but will expand to use less margin for big line numbers.

diff: refactor table styling to clarify how classes are applied to tr or td

The styling is so specific that the generic selectors are misleading.

The font styling will now also be put on the <td> instead of just on the <a> it
might contain.

lib: use sha1 instead of md5 in a couple of places

md5 is dead and should be avoided. In the places changed here, we want to keep
using hashes without trivial collisions, but do not expect strong crypto
security. sha1 seems like a trivial step up from md5 and without obvious
alternatives. It is more expensive than md5, but we can live with that in these
places.

The remaining few uses of md5() cannot be changed without breaking backwards
compatibility or external API.

docs: mention the need for worker process restarts to avoid "memory leaks" (Issue #346)

It might not be ideal to mention such details in the overview, but that seems
like the best place for generic advice.

(Kallithea should still aim for keeping memory growth under control.)

db: drop unused _hash_key

A bit less md5 ...

diff: drop internal inconsistent use of '...' as context line numbers

as_html() did in some cases not check for '...' and would thus emit unnecessary
html.

diff: fix per file display of "Raw diff" and others

With py3, the matcher must be a list of byte strings.

ini: change .ini template to include sqlalchemy.url examples for all database types

Rely less on users creating their initial .ini "correctly" with the right
database choice.

hg: fix crash on displaying bookmarks - add lost hex conversion when computing bookmark property

The crash was seen for example when using "Switch To" with bookmarks.

The problem was introduced with py3 conversion in e7dbe089e10d.

comments: add changeset author in mail notification subject

In analogy to the handling of comments on pull requests, as changed in
commit d4061c6cc0e2.

lib/diffs: mark trailing tabs similar to trailing spaces

So far, spaces have not been marked up, but trailing spaces are followed by
'<i></i>' (newline symbol). Tabs have been marked up as '<u>\t</u>' (underlined
with icon), but trailing tabs didn't get the same "trailing whitespace" markup
as trailing spaces got.

Fix this unfairness by handling trailing tabs explicitly as '<u>\t</u><i></i>'
so they get both kinds of markup.

notifications: explicitly add author of pull request in invitation mail

When an author of a pull request does not get feedback from any of the
reviewers, he is left wondering whether they got the invitation mail at all.
By explicitly sending the invitation also to the author, he can at least know
if e-mail works at the server side and how reviewers see it.
The mail can also be forwarded as a 'reminder'.

logging: try to avoid using ANSI color codes when not logging to a terminal

Color on the console is nice, but it is annoying to get these codes when for
example redirecting to a file.

The formatter doesn't know exactly where the message will be used - it is fully
configurable. In the default configuration, the messages are passed to a
StreamHandler to sys.stderr .

As an approximate optmization, hardcode the color formatters to only emit color
formatting if logging to something that isatty.

setup: bump version numbers to current max

Based on 'pip freeze' before/after:

sed -i -e '/>=[^#]*$/s/, < [0-9.]\+//g' -e 's, == , >= ,g' setup.py dev_requirements.txt

Markdown 3.3 is a few months old could be bumped too ... but it change test
output due to
https://python-markdown.github.io/change_log/release-3.2/#codehilite-now-always-wraps-with-ltcodegt-tags
and can wait.

This is landing early in the development phase and will thus get good test
coverage.

permissions: drop hg.create.write_on_repogroup "Repository creation with group write access" setting

Simplify permissions system and get rid of some confusing tech debt.

Before, the global 'write_on_repogroup' setting controlled what write
permission on a repo group meant.

With this change, users can create repositories in a repo group if and only if
they have write access. Write access to a repo group will now mean the
permission to create repositories in it.

Write access to repo groups must be granted explicitly. There should not be any
other reason to grant write access than to allow users to create repos. There
is thus no upgrade concerns for this change.

An admin that doesn't want users to create repos in a repogroup should just not
give them write access.

These global settings might still exist in the database, but is ignored and no
longer used and do no harm.

db_manage: drop unused fix functions

The functions might be ok, but we don't know when or why to call them ... and
apparently we don't need them.

inifile: make it possible for expand() to comment out settings without assigning new value

For completeness, when we already can create and update values, also make it
possible to delete. This fits nicely into the implementation. Potentiallly
"nice to have" but not used yet.

The ini file is a text file and it only really makes sense to set string
values. Intuitively, it makes sense that setting something to None means that
the old value should be commented out but no new value should be set. If we
want to set a value of "None", then specify "None" - not None.

tg: include the Kallithea middleware wrapping of the TG WSGI application in application.py

This seems to make the architecture more clear than using hooks.

tg: refactor app_cfg blueprint configuration to align more with TG quickstart conventions

Drop prefer_toscawidgets2 and use_toscawidgets - they are not important and
already default to False.

lib: drop own asbool implementation and consistently use tg.support.converters as utils2.asbool

str2bool never reported error on odd input such as '' or '-1', but the tg
asbool behaviour of raising ValueError("String is not true/false: %r" % obj) in
that case seems fine.

setup: exclude celery 4.4.4 which is broken due to unexpressed dependency

Celery 4.4.4 introduced the use of the 'future' package but forgot to
express it in its dependencies.

We could add the missing dependency on 'future' in Kallithea, but since the
problem is already fixed upstream shortly after 4.4.4 was released [1], we can
be sure that the next release (presumably 4.4.5) will contain the fix.

[1] https://github.com/celery/celery/pull/6146

cleanup: drop last traces of kallithea-config

The kallithea-config sources were removed in commit
213085032127e941a3bd93d0e510828a9d87bf32 but an entry point was still
created by setup.py.
Moreover, the ini file still referenced this, instead of kallithea-cli
(config-create).

html: fix invalid html / mako structure

Fix bad nesting of tags and mako conditional and drop superfluous '}' outside
the mako block.

tests: actually test something useful in test_edit for gists (Issue #376)

Even though there was a test for editing gists, it did not catch the basic
loading problem reported in issue #376. In fact, the test just loaded the
edit page, but since no user was actually logged in, it just loaded the
login screen. As a result, no real gist editing code was tested at all.

Instead, explicitly check the redirection to a login screen, then proceed
with logging in and check that the edit page can be loaded.

Additionally, don't rely on the magic gist id '1' but create an actual gist
first.

Note: the test does not yet really edit a gist.