kallithea Changeset - b9853a3cc254

Changeset - b9853a3cc254

Parent rev.

Child rev.

[Not reviewed]

default

0 2 0

domruf - 8 years ago 2017-06-11 16:13:09
dominikruf@gmail.com

celery: simplify internal configuration and app creation

We used celery.app.app_or_default() which creates a "global fallback app
instance" which relies on the CELERY_LOADER environment variable to load the
configuration. That worked but was messy.

Instead, do something more like described in
http://docs.celeryproject.org/en/3.1/userguide/application.html where the app
is a celery.Celery() instance and configuration is loaded explicitly by its
config_from_object method. Using config_from_object we don't need explicit
invocation of import_default_modules and can take pass PylonsSettingsProxy
directly, leaving PylonsLoader unused and removed.

Modified by Mads Kiilerich.

2 files changed with 8 insertions and 30 deletions:

kallithea/lib/celerypylons/__init__.py

kallithea/lib/celerypylons/loader.py

0 comments (0 inline, 0 general)

kallithea/lib/celerypylons/__init__.py

➞

Show inline comments

 # -*- coding: utf-8 -*-
 """
 Kallithea wrapper of Celery
 The Celery configuration is in the normal Pylons ini file. We thus have to set
 the `CELERY_LOADER` environment variable to point at a custom "loader" that can
 read it. That environment variable must be set *before* importing celery. To
 ensure that, we wrap celery in this module.
 The Celery configuration is in the ini file. To read the settings and translate
 to a Celery format we use PylonsSettingsProxy.
 We read the configuration from tg.config, thus it must be initialized before
 loading this module. To make sure that really is the case and give an early
 warning, we check one of the mandatory settings.
 This module must thus not be imported in global scope but must be imported on
 demand in function scope after tg.config has been initialized.
 """
 import os
 import warnings
 import celery.app
 import celery
 from kallithea.lib.celerypylons.loader import PylonsSettingsProxy
 # Verify Pylons configuration has been loaded
 from tg import config
 assert config['celery.imports'] == 'kallithea.lib.celerylib.tasks', 'Kallithea Celery configuration has not been loaded'
 # Prepare environment to point at Kallithea Pylons loader
 CELERYPYLONS_LOADER = 'kallithea.lib.celerypylons.loader.PylonsLoader'
 if os.environ.get('CELERY_LOADER', CELERYPYLONS_LOADER) != CELERYPYLONS_LOADER:
     warnings.warn("'CELERY_LOADER' environment variable will be overridden by celery-pylons.")
 os.environ['CELERY_LOADER'] = CELERYPYLONS_LOADER
 # Create celery app, thus immediately triggering use of the custom Pylons loader
 app = celery.app.app_or_default()
 # Create celery app from the TurboGears configuration file
 app = celery.Celery()
 app.config_from_object(PylonsSettingsProxy())
 import celery.result as result
 from celery.task import task
 from celery.bin import worker

kallithea/lib/celerypylons/loader.py

➞

Show inline comments

 # -*- coding: utf-8 -*-
 from celery.loaders.base import BaseLoader
 from tg import config
 # TODO: drop this mangling and just use a separate celery config section
 to_pylons = lambda x: x.replace('_', '.').lower()
 to_celery = lambda x: x.replace('.', '_').upper()
@@ @@ -56,22 +55,6 @@ class PylonsSettingsProxy(object): @@
             return int(value)
         #cast to bool
         if value.lower() in ['true', 'false']:
             return value.lower() == 'true'
         return value
 class PylonsLoader(BaseLoader):
     """Pylons celery loader
     Maps the celery config onto pylons.config
     """
     def read_configuration(self):
         self.configured = True
         return PylonsSettingsProxy()
     def on_worker_init(self):
         """
         Import task modules.
         """
         self.import_default_modules()

0 comments (0 inline, 0 general)