kallithea Changeset - 3fb0ce6de10d

Changeset - 3fb0ce6de10d

Parent rev.

Child rev.

[Not reviewed]

default

1 1 0

Mads Kiilerich - 8 years ago 2017-08-13 01:59:19
mads@kiilerich.com

celery: replace loader.PylonsSettingsProxy with a simple configuration object

The proxy was quite complex and did all the conversion at runtime, making it
hard to figure out exactly what it was doing.

Instead, just create a simple configuration object as described on
http://docs.celeryproject.org/en/3.1/userguide/application.html and convert all
relevant config settings up front. We don't convert all config settings, but
only the ones Celery seems to be interested in. The value conversion heuristics
are the same as before and still seem quite fragile.

2 files changed with 38 insertions and 72 deletions:

kallithea/lib/celerypylons/__init__.py

kallithea/lib/celerypylons/loader.py

0 comments (0 inline, 0 general)

kallithea/lib/celerypylons/__init__.py

➞

Show inline comments

@@ @@ -3,31 +3,57 @@ @@
 """
 Kallithea wrapper of Celery
 The Celery configuration is in the ini file. To read the settings and translate
 to a Celery format we use PylonsSettingsProxy.
 The Celery configuration is in the Kallithea ini file but must be converted to an
 entirely different format before Celery can use it.
 We read the configuration from tg.config, thus it must be initialized before
 loading this module. To make sure that really is the case and give an early
 warning, we check one of the mandatory settings.
 We read the configuration from tg.config at module import time. This module can
 thus not be imported in global scope but must be imported on demand in function
 scope after tg.config has been initialized.
 This module must thus not be imported in global scope but must be imported on
 demand in function scope after tg.config has been initialized.
 To make sure that the config really has been initialized, we check one of the
 mandatory settings.
 """
 import os
 import warnings
 import tg
 import celery
 from kallithea.lib.celerypylons.loader import PylonsSettingsProxy
 def celery_config(config):
     """Return Celery config object populated from relevant settings in a config dict, such as tg.config"""
     # Verify .ini file configuration has been loaded
     assert config['celery.imports'] == 'kallithea.lib.celerylib.tasks', 'Kallithea Celery configuration has not been loaded'
     class CeleryConfig(object): pass
     celery_config = CeleryConfig()
     PREFIXES = """ADMINS BROKER CASSANDRA CELERYBEAT CELERYD CELERYMON CELERY EMAIL SERVER""".split()
     LIST_PARAMS = """CELERY_IMPORTS ADMINS ROUTES CELERY_ACCEPT_CONTENT""".split()
 # Verify Pylons configuration has been loaded
 from tg import config
 assert config['celery.imports'] == 'kallithea.lib.celerylib.tasks', 'Kallithea Celery configuration has not been loaded'
     for config_key, config_value in sorted(config.items()):
         celery_key = config_key.replace('.', '_').upper()
         if celery_key.split('_', 1)[0] not in PREFIXES:
             continue
         if not isinstance(config_value, basestring):
             continue
         if celery_key in LIST_PARAMS:
             celery_value = config_value.split()
         elif config_value.isdigit():
             celery_value = int(config_value)
         elif config_value.lower() in ['true', 'false']:
             celery_value = config_value.lower() == 'true'
         else:
             celery_value = config_value
         setattr(celery_config, celery_key, celery_value)
     return celery_config
 # Create celery app from the TurboGears configuration file
 app = celery.Celery()
-app.config_from_object(PylonsSettingsProxy())
+app.config_from_object(celery_config(tg.config))
 import celery.result as result
 from celery.task import task

kallithea/lib/celerypylons/loader.py

➞

Show inline comments

deleted file

0 comments (0 inline, 0 general)