mail: migrate to modern email.message instead of using old compat32 functionality

Before, the HTML part could be detected as pure 7-bit and sent as:
Content-Type: text/html; charset="us-ascii"
Content-Transfer-Encoding: 7bit

Now, it will use:
Content-Type: text/html; charset="utf-8"
Content-Transfer-Encoding: quoted-printable

That should work fine too.

mail: fix duplicate "From" headers

Problem introduced in 9a0c41175e66: When iterating the headers dict and setting
"msg[key] = value", it wasn't replacing the header but performing add_header so
we sometimes ended up with two From headers.

It is also a general problem that while the headers dict only can contain each
key once, it can contain entries that only differ in casing and thus will fold
to the same message header, making it possible to end up adding duplicate
headers.

"msg.replace_header(key, value)" is not a simple solution to the problem: it
will raise KeyError if no such previous key exists.

Now, make the problem more clear by explicitly using add_header.

Avoid the duplication problem by deleting the key (no matter which casing)
before invoking add_header. Delete promises that "No exception is raised if the
named field isn’t present in the headers".

hooks: import whole hooks module

Minimize impact if there should be cycles.

lib: consistently import helpers the same way

Make it easier to grep for any remaining potential layering-violating use of
helpers.

mail: use plain standard library for sending mails instead of rcmail

Avoid a lot of custom complexity that adds no value.

A few trivial lines from rcmail are inlined.

mail: simplify celery task for sending mail

Name variables after their config setting, and drop undocumented debug setting.

config: move config.conf to kallithea.lib

kallithea.config is mainly the WSGI entry point with TG stuff, and thus a high
level controller thing - not a place to store configuration.

Note: The content of conf.py is kind of similar to what we have in
kallithea/__init__.py . These two should perhaps be merged somehow.

isort: upgrade to 5.1.2

The changes to non-top-level imports seem nice.

lib: drop own asbool implementation and consistently use tg.support.converters as utils2.asbool

str2bool never reported error on odd input such as '' or '-1', but the tg
asbool behaviour of raising ValueError("String is not true/false: %r" % obj) in
that case seems fine.

db: consistently use base_path from config instead of repeatedly getting from the database

Avoid using the inefficient Repository.base_path ... and avoid even more the
misleading Repository.repo_path .

celery: fix send_email to work with JSON encoding (Issue #363)

Long time ago, c935bcaf7086 introduced an optional User object parameter to the
send_email task and used the computed full_name_or_username property. Due to the
magic of pickle, that also worked when using Celery to run the task async.

Now, Celery 4 changed the default encoding from Pickle to JSON, which we
anticipated in e539db6cc0da. That broke send_email in some cases, for example
when a user comments on another user's changeset.

Fixed by passing the "From" name as string instead of passing the whole User
object.

Thanks to vyom for reporting.

cleanup: minimize use of lambda expressions - we have 'def' for that purpose

Fix some flake8 warnings "E731 do not assign a lambda expression, use a def".

celery: fix logging from inside tasks

Celery will hijack the global logging configuration and disable the usual
Kallithea logging (unless CELERYD_HIJACK_ROOT_LOGGER is set False).

Fixed partially by using the special celery loggers inside the tasks. Logging
from modules will still not work.

celery: refactor initialization - replace global CELERY_ON flag with CELERY_APP with the actual celery app that it indicates

Prepare for fixing how 193138922d56 broke celery due to magic dependencies on
initialization of global state at import time.

lib: clean up ext_json and how it is used - avoid monkey patching

Note that py3 json.dumps will return ASCII (with all unicode escaped) encoded
as str. But we generally want JSON as bytes (which json.loads also can read),
so also wrap the result with ascii_bytes in many places.

lib: avoid comparing ordering with None

Fails in py3.

lib: mktime needs a tuple - not a list

Py3 is more strict.

cleanup: minor refactorings and simplification of dict usage

Makes it more py3 compatible.

tg2: handle cache_dir configuration jumping around between TurboGears versions

Tweak 87672c1916f8 - the use of 'tg.cache_dir' turns out to not be a good idea.

TurboGears 2.4 moved 'cache_dir' configuration to the expected 'cache_dir' key,
and no longer has 'tg.cache_dir' ... and also not the old
'app_conf'.'cache_dir' .

Just try both locations.

config: stop using the app_conf sub key - TurboGears2 2.4 will drop it

All [app:main] settings *except* cache_dir are also available at the top level. cache_dir can be found as tg.cache_dir .

refactor: use string methods instead of class functions and map

Python 3 has no lower or strip functions - only string methods.

locking: drop the pull-to-lock / push-to-unlock functionality

The feature is not worth the maintenance cost. The locking is too coarse and
unflexible with insufficient UI and UX. The implementation is also quite
invasive in tricky areas of the code, and thus high maintenance.
Dropping this will enable other cleanup ... or at least make it easier.

email: fix fallback sending to multiple admins

If no recipients are set, emails are sent to all admins and the address
specified in the config file setting 'email_to'.

However, the existing code did not cater for the fact that multiple
addresses could be set there: the setting was treated as a single string and
passed along as-is.

Adapt the code to split the email_to contents on a comma, as this is the
separator expected by backlash.

codingstyle: trivial whitespace fixes

Reported by flake8.

celeryd: let the gearbox command use db session as most other gearbox commands do

304aae43194c changed the common gearbox wrapper so make_app_without_logging
only was run for commands tagged as requires_db_session. That broke celeryd -
even a plain 'gearbox celeryd -c my.ini' would fail on the safety check in
celerypylons asserting on tg.config having 'celery.imports' configuration.

The gearbox celeryd command did not really require a db session - it just
required app configuration so it could create db sessions on the fly.

To to get the missing make_app_without_logging invocation back, set
requires_db_session (the default for our gearbox commands).

requires_db_session not only calls make_app_without_logging (which undo the
effect from 304aae43194c), it also calls setup_cache_regions,
engine_from_config, and init_model. These were also invoked explicitly in
celeryd code - these double invocations are dropped too.

Also, make_app_without_logging will call into tg and thus invoke the
setup_configuration hook which will set kallithea.CELERY_ON and call
load_rcextensions. The celeryd code for doing that is thus dropped.

tests: clarify that default parameters are for form - direct model access requires different types

_get_repo_group_create_params parent_group_id is thus set to the form value
'-1' instead of None. It worked before anyway because the model failed to find
the repo '-1' and thus got pretty much the same as None.

tg: minimize future diff by some mocking and replacing some pylons imports with tg

No actual tg dependency yet, just a temporary hack faking tg as an alias for
pylons.

Based on work by Alessandro Molina.

model: remove BaseModel class

The only remaining purpose of this class was to provide the "sa" field,
allowing a custom SQLAlchemy session to be used for model operations.
However, this never actually worked, nor was it used anywhere.

There's always a global SQLAlchemy session associated with the current
thread; using another session for a single function call does not make
any sense (as sessions cannot be mixed), unless the code works carefully
to ensure the two sessions (and all objects loaded from them) are kept
completely separate. Suffice to say that Kallithea does no such thing,
thus there's no need to pretend to support multiple concurrent sessions.

cleanup: remove SQLAlchemy session argument to action_logger

There's always a global SQLAlchemy session associated with the current
thread; using another session for a single function call does not make
any sense (as sessions cannot be mixed), unless the code works carefully
to ensure the two sessions (and all objects loaded from them) are kept
completely separate. Suffice to say that Kallithea does no such thing,
thus there's no need to pretend to support multiple concurrent sessions.

celery: import the whole celerylib - no cherry picking in tasks

No big deal, but arguably(?!) slightly prettier.

The database session handling should probably also be cleaned up - that is not
touched here but left as an exercise.

celeryd: annotate tasks so they can be run directly without run_task

This also makes the system less forgiving about celery configuration problems
and thus easier to debug. I like that.

celery: drop pointless get_logger

It didn't give meaningful names anyway ... and using the tasks module name is
fine.

utils: rename add_cache to setup_cache_regions

Rename method 'add_cache' to something that better describes what it
actually does.

cleanup: get rid of jn as shortcut for os.path.join

It is not used in that many places so it is better to be explicit and use os.path.join .

Discussed on https://bitbucket.org/domruf/kallithea/commits/1da05f42bca3f7042c444fda13a1ae1f6b9c300f#comment-2948124 .

Modified by Mads Kiilerich.

celery: stop using deprecated celery.decorators

We don't use the old magic keyword arguments and can just use celery.task
instead - see
http://docs.celeryproject.org/en/latest/history/changelog-2.2.html#version-2-2-0

email: send comment and pullrequest mails with the author's name in 'From'

When emails are sent for comments and pullrequest invitations, set the From
header to:
Author's Name (no-reply) <generic email address>

Using the name of the person that causes the email, makes the emails more
useful and interpretable for the recipient of the emails.
To avoid replies directly to the author, triggering an 'offline' email
discussion that is not visible in the Kallithea interface, a generic
'no-reply' email address is used instead of the author's email
address.
This approach is assumed to be accepted by spam filters, as several other
web services are using the same approach.

The sender used for other email types, e.g. password reset mails, is
untouched and remains the value configured in app_email_from.

The sender used for the SMTP envelope is untouched as well.

Based on code by Cedric De Herdt.

email: better logging of misconfiguration and better debug logging

Always add details about emails being sent (or attempted to be sent).
When a mail server is configured, log as debug, otherwise as warning.

This also adds logging of recipients and headers.

cleanup: consistently use 'except ... as ...:'

Use the Python 2.6+ syntax instead of the old confusing 'except ..., ...'
syntax.

e-mail: properly handle no recipients when there is no email_to set

When the configuration file does not contain a value for email_to, and no
recipients are specified in a call to send_email, recipients would be set to
[None, admins]
which causes an error when logging this list as ' '.join(recipients).

e-mail: revive dead code that checks for unspecified recipients

Commit 609e06b6c52f6a8ea9581372805c4bbb60db81a1 introduced dead code:
in the beginning of send_email an assert verifies that recipients is of type
'list', so checking it for None later is useless and that branch can never
be reached.

Instead of removing the dead code, revive it and add a test case.

spelling: fix various typos

This commit fixes various typos or basic English grammar mistakes found by
reviewing the kallithea.pot file.

Full correction of sentences that are not very well formulated, like missing
articles, is out of scope for this commit. Likewise for inconsistent
capitalization of strings like 'Repository group'/'Repository Group'.

cleanup: avoid some 'except Exception' catching - catch specific exceptions or log it and show what happened

This has a risk of introducing regressions ... but we want to get rid of all
exception muting and make the whole system less fragile and easier to debug.

notifications: insert 'References' mail headers to help MUA threading

gmail is however fubar.

Correct licensing information in individual files.

The top-level license file is now LICENSE.md.

Also, in various places where there should have been joint copyright holders
listed, a single copyright holder was listed. It does not appear easy to add
a link to a large list of copyright holders in these places, so it simply
refers to the fact that various authors hold copyright.

In future, if an easy method is discovered to link to a list from those
places, we should do so.

Finally, text is added to LICENSE.md to point to where the full list of
copyright holders is, and that Kallithea as a whole is GPLv3'd.

Second step in two-part process to rename directories.
This is the actual directory rename.