celery: let async tasks choose at runtime if they should use immediate execution or dispatch to the Celery worker

Make it completely safe to use task annotation at import time, before global
config has been set.

celery: don't rely on CELERY_APP only being set (from app_cfg) when use_celery

We would rather always have CELERY_APP set.

Kind of backing out 894a662b12b3.

celery: drop tracking of task_id - we use ignore_result=True and will never get anything back

There is thus no need for configuration of celery.result_backend .

The alternative would be to fix it. That could give better error reporting from
failing repo creations, but would require quite a bit of additional changes
before it actually works reliably.

celery: always drop db session after running a task

Drop dbsession annotation - it could easily be forgotten. This seems simpler
and more reliable.

celery: use regular automatic session handling for tasks

There were no point in having a special celery get_session() .

celery: use explicit task names - avoid automatic naming with "kallithea.lib.celerylib." prefix

We wrap async functions in a local f_async wrapper, defined in
kallithea/lib/celerylib/__init__.py . For a function Foo.X, even though we
changed the wrapper's __name__ to X, the tasks would be named
kallithea.lib.celerylib.X , without using the actual module name of X for
namespacing.

Drop modifying __name__, and just specify the name explicitly, without trying
to namespace it.

TurboGears: drop workaround for < 2.4

Backout a38e05a0c79e and tweak.

celery: drop kallithea.CELERY_EAGER - it is more spot-on to look directly at task_always_eager

The global flag was set as a side effect in the make_app function. That's not
pretty.

lib: use sha1 instead of md5 in a couple of places

md5 is dead and should be avoided. In the places changed here, we want to keep
using hashes without trivial collisions, but do not expect strong crypto
security. sha1 seems like a trivial step up from md5 and without obvious
alternatives. It is more expensive than md5, but we can live with that in these
places.

The remaining few uses of md5() cannot be changed without breaking backwards
compatibility or external API.

celery: use the proper configured global app for scheduling and retrieving tasks

193138922d56 broke celery, due to magic dependencies on initialization and
setting global configuration at import time.

Instead, always use the correctly configured global Celery app, both when
creating tasks and checking result status.

This has been tested to work on Python 3.6 - for example for sending mails and
forking repos.

Celery has however been found to not work on Python 3.7, due to Celery 3.x
using the new reserved keyword 'async'.

celery: refactor initialization - replace global CELERY_ON flag with CELERY_APP with the actual celery app that it indicates

Prepare for fixing how 193138922d56 broke celery due to magic dependencies on
initialization of global state at import time.

tg2: handle cache_dir configuration jumping around between TurboGears versions

Tweak 87672c1916f8 - the use of 'tg.cache_dir' turns out to not be a good idea.

TurboGears 2.4 moved 'cache_dir' configuration to the expected 'cache_dir' key,
and no longer has 'tg.cache_dir' ... and also not the old
'app_conf'.'cache_dir' .

Just try both locations.

config: stop using the app_conf sub key - TurboGears2 2.4 will drop it

All [app:main] settings *except* cache_dir are also available at the top level. cache_dir can be found as tg.cache_dir .

commands: remove custom (re-)initializing of database model

The model has already beeen initialized through TurboGears after
make_app_without_logging has been called.

Since kallithea.lib.utils2.engine_from_config no longer is used, remove it.
All remaining references to engine_from_config are directly to the one from
sqlalchemy.

codingstyle: trivial whitespace fixes

Reported by flake8.

celeryd: let the gearbox command use db session as most other gearbox commands do

304aae43194c changed the common gearbox wrapper so make_app_without_logging
only was run for commands tagged as requires_db_session. That broke celeryd -
even a plain 'gearbox celeryd -c my.ini' would fail on the safety check in
celerypylons asserting on tg.config having 'celery.imports' configuration.

The gearbox celeryd command did not really require a db session - it just
required app configuration so it could create db sessions on the fly.

To to get the missing make_app_without_logging invocation back, set
requires_db_session (the default for our gearbox commands).

requires_db_session not only calls make_app_without_logging (which undo the
effect from 304aae43194c), it also calls setup_cache_regions,
engine_from_config, and init_model. These were also invoked explicitly in
celeryd code - these double invocations are dropped too.

Also, make_app_without_logging will call into tg and thus invoke the
setup_configuration hook which will set kallithea.CELERY_ON and call
load_rcextensions. The celeryd code for doing that is thus dropped.

tg: minimize future diff by some mocking and replacing some pylons imports with tg

No actual tg dependency yet, just a temporary hack faking tg as an alias for
pylons.

Based on work by Alessandro Molina.

model: move code from __init__.py to base.py

Having too much code, in particular too much imports, inside a package's
__init__.py is a recipe for circular imports, and considered bad practice in
Python [1]

Move out everything from kallithea/model/__init__.py to a new file
kallithea/model/base.py and adapt the existing imports.

[1] http://docs.python-guide.org/en/latest/writing/structure/#packages

Turbogears2 migration: use sqlalchemy.url iso sqlalchemy.db1.url

In Turbogears2, much of the application initialization is handled by the
framework, whereas in Pylons the application was responsible for it.
Initializing SQLAlchemy is one such part of initialization which is handled
by Turbogears2.

Turbogears2 expects the configuration file to refer to the database using
'sqlalchemy.url' rather than the current 'sqlalchemy.db1.url'. While the
exact name is not really important, not following this approach means we'll
need to override the sqlalchemy initialization method.

Therefore, as a preparation to the Turbogears2 migration, already change the
database reference string under Pylons.
When upgrading to a version of Kallithea containing this commit, the .ini
file will manually need to be adapted to remove the .db1 strings.

celerypylons: wrap celery import so we always get the right environment variables set and check configuration

Get rid of magic ... or at least document and encapsulate it.

Before, celerypylons would set the environment variables that made it possible
to import Celery after Pylons had been configured.

Now, the module will import Celery ... and verify that Pylons has been
configured.

(A next stop could be to move things around so this got tied closely to
initializing Pylons. Or something.)

celery: import the whole celerylib - no cherry picking in tasks

No big deal, but arguably(?!) slightly prettier.

The database session handling should probably also be cleaned up - that is not
touched here but left as an exercise.

celeryd: annotate tasks so they can be run directly without run_task

This also makes the system less forgiving about celery configuration problems
and thus easier to debug. I like that.

celerylib: improve handling of sync results and get rid of BaseAsyncResult handling

A better wrapper of sync results simplifies the code.

Note: Results are currently not really used.

cleanup: get rid of jn as shortcut for os.path.join

It is not used in that many places so it is better to be explicit and use os.path.join .

Discussed on https://bitbucket.org/domruf/kallithea/commits/1da05f42bca3f7042c444fda13a1ae1f6b9c300f#comment-2948124 .

Modified by Mads Kiilerich.

cleanup: make module self-naming consistent

The self naming is quite pointless ... but at least make it right.

cleanup: consistently use 'except ... as ...:'

Use the Python 2.6+ syntax instead of the old confusing 'except ..., ...'
syntax.

Correct licensing information in individual files.

The top-level license file is now LICENSE.md.

Also, in various places where there should have been joint copyright holders
listed, a single copyright holder was listed. It does not appear easy to add
a link to a large list of copyright holders in these places, so it simply
refers to the fact that various authors hold copyright.

In future, if an easy method is discovered to link to a list from those
places, we should do so.

Finally, text is added to LICENSE.md to point to where the full list of
copyright holders is, and that Kallithea as a whole is GPLv3'd.

Second step in two-part process to rename directories.
This is the actual directory rename.