Changeset - 55ccfc66479d
[Not reviewed]
default
0 7 0
Mads Kiilerich - 10 years ago 2015-10-08 23:21:58
madski@unity3d.com
cache: make instance_id = * the default and deprecate it

Auto assigning instance_id's works perfectly fine now when the clean-up issue
has been resolved.
7 files changed with 1 insertions and 31 deletions:
0 comments (0 inline, 0 general)
development.ini
Show inline comments
 
@@ -259,54 +259,48 @@ default_encoding = utf8
 
## {id} will be all groups matched from this pattern
 

	
 
issue_pat = (?:\s*#)(\d+)
 

	
 
## server url to the issue, each {id} will be replaced with match
 
## fetched from the regex and {repo} is replaced with full repository name
 
## including groups {repo_name} is replaced with just name of repo
 

	
 
issue_server_link = https://issues.example.com/{repo}/issue/{id}
 

	
 
## prefix to add to link to indicate it's an url
 
## #314 will be replaced by <issue_prefix><id>
 

	
 
issue_prefix = #
 

	
 
## issue_pat, issue_server_link, issue_prefix can have suffixes to specify
 
## multiple patterns, to other issues server, wiki or others
 
## below an example how to create a wiki pattern
 
# wiki-some-id -> https://wiki.example.com/some-id
 

	
 
#issue_pat_wiki = (?:wiki-)(.+)
 
#issue_server_link_wiki = https://wiki.example.com/{id}
 
#issue_prefix_wiki = WIKI-
 

	
 
## instance-id prefix
 
## a prefix key for this instance used for cache invalidation when running
 
## multiple instances of kallithea, make sure it's globally unique for
 
## all running kallithea instances. Leave empty if you don't use it
 
instance_id =
 

	
 
## alternative return HTTP header for failed authentication. Default HTTP
 
## response is 401 HTTPUnauthorized. Currently Mercurial clients have trouble with
 
## handling that. Set this variable to 403 to return HTTPForbidden
 
auth_ret_code =
 

	
 
## locking return code. When repository is locked return this HTTP code. 2XX
 
## codes don't break the transactions while 4XX codes do
 
lock_ret_code = 423
 

	
 
## allows to change the repository location in settings page
 
allow_repo_location_change = True
 

	
 
## allows to setup custom hooks in settings page
 
allow_custom_hooks_settings = True
 

	
 
####################################
 
###        CELERY CONFIG        ####
 
####################################
 

	
 
use_celery = false
 
broker.host = localhost
 
broker.vhost = rabbitmqhost
 
broker.port = 5672
 
broker.user = rabbitmq
docs/setup.rst
Show inline comments
 
@@ -725,53 +725,48 @@ that, you'll need to:
 
  correctly specified.
 

	
 
Here is a sample excerpt from an Apache Virtual Host configuration file:
 

	
 
.. code-block:: apache
 

	
 
    WSGIDaemonProcess kallithea \
 
        processes=1 threads=4 \
 
        python-path=/srv/kallithea/pyenv/lib/python2.7/site-packages
 
    WSGIScriptAlias / /srv/kallithea/dispatch.wsgi
 
    WSGIPassAuthorization On
 

	
 
Or if using a dispatcher WSGI script with proper virtualenv activation:
 

	
 
.. code-block:: apache
 

	
 
    WSGIDaemonProcess kallithea processes=1 threads=4
 
    WSGIScriptAlias / /srv/kallithea/dispatch.wsgi
 
    WSGIPassAuthorization On
 

	
 
.. note::
 
   When running apache as root, please make sure it doesn't run Kallithea as
 
   root, for examply by adding: ``user=www-data group=www-data`` to the configuration.
 

	
 
.. note::
 
   If running Kallithea in multiprocess mode,
 
   make sure you set ``instance_id = *`` in the configuration so each process
 
   gets it's own cache invalidation key.
 

	
 
Example WSGI dispatch script:
 

	
 
.. code-block:: python
 

	
 
    import os
 
    os.environ["HGENCODING"] = "UTF-8"
 
    os.environ['PYTHON_EGG_CACHE'] = '/srv/kallithea/.egg-cache'
 

	
 
    # sometimes it's needed to set the curent dir
 
    os.chdir('/srv/kallithea/')
 

	
 
    import site
 
    site.addsitedir("/srv/kallithea/pyenv/lib/python2.7/site-packages")
 

	
 
    from paste.deploy import loadapp
 
    from paste.script.util.logging_config import fileConfig
 

	
 
    fileConfig('/srv/kallithea/my.ini')
 
    application = loadapp('config:/srv/kallithea/my.ini')
 

	
 
Or using proper virtualenv activation:
 

	
 
.. code-block:: python
 

	
docs/usage/performance.rst
Show inline comments
 
@@ -17,40 +17,39 @@ the best performance.
 
  option and ``vcs_full_cache`` setting in .ini file.
 

	
 
Follow these few steps to improve performance of Kallithea system.
 

	
 
1. Increase cache
 

	
 
    Tweak beaker cache settings in the ini file. The actual effect of that
 
    is questionable.
 

	
 
2. Switch from SQLite to PostgreSQL or MySQL
 

	
 
    SQLite is a good option when having a small load on the system. But due to
 
    locking issues with SQLite, it is not recommended to use it for larger
 
    deployments. Switching to MySQL or PostgreSQL will result in an immediate
 
    performance increase. A tool like SQLAlchemyGrate_ can be used for
 
    migrating to another database platform.
 

	
 
3. Scale Kallithea horizontally
 

	
 
    Scaling horizontally can give huge performance benefits when dealing with
 
    large amounts of traffic (many users, CI servers, etc.). Kallithea can be
 
    scaled horizontally on one (recommended) or multiple machines. In order
 
    to scale horizontally you need to do the following:
 

	
 
    - Each instance needs its own .ini file and unique ``instance_id`` set.
 
    - Each instance's ``data`` storage needs to be configured to be stored on a
 
      shared disk storage, preferably together with repositories. This ``data``
 
      dir contains template caches, sessions, whoosh index and is used for
 
      task locking (so it is safe across multiple instances). Set the
 
      ``cache_dir``, ``index_dir``, ``beaker.cache.data_dir``, ``beaker.cache.lock_dir``
 
      variables in each .ini file to a shared location across Kallithea instances
 
    - If celery is used each instance should run a separate Celery instance, but
 
      the message broker should be common to all of them (e.g.,  one
 
      shared RabbitMQ server)
 
    - Load balance using round robin or IP hash, recommended is writing LB rules
 
      that will separate regular user traffic from automated processes like CI
 
      servers or build bots.
 

	
 

	
 
.. _SQLAlchemyGrate: https://github.com/shazow/sqlalchemygrate
kallithea/bin/template.ini.mako
Show inline comments
 
@@ -257,54 +257,48 @@ default_encoding = utf8
 
<%text>## {id} will be all groups matched from this pattern</%text>
 

	
 
issue_pat = (?:\s*#)(\d+)
 

	
 
<%text>## server url to the issue, each {id} will be replaced with match</%text>
 
<%text>## fetched from the regex and {repo} is replaced with full repository name</%text>
 
<%text>## including groups {repo_name} is replaced with just name of repo</%text>
 

	
 
issue_server_link = https://issues.example.com/{repo}/issue/{id}
 

	
 
<%text>## prefix to add to link to indicate it's an url</%text>
 
<%text>## #314 will be replaced by <issue_prefix><id></%text>
 

	
 
issue_prefix = #
 

	
 
<%text>## issue_pat, issue_server_link, issue_prefix can have suffixes to specify</%text>
 
<%text>## multiple patterns, to other issues server, wiki or others</%text>
 
<%text>## below an example how to create a wiki pattern</%text>
 
# wiki-some-id -> https://wiki.example.com/some-id
 

	
 
#issue_pat_wiki = (?:wiki-)(.+)
 
#issue_server_link_wiki = https://wiki.example.com/{id}
 
#issue_prefix_wiki = WIKI-
 

	
 
<%text>## instance-id prefix</%text>
 
<%text>## a prefix key for this instance used for cache invalidation when running</%text>
 
<%text>## multiple instances of kallithea, make sure it's globally unique for</%text>
 
<%text>## all running kallithea instances. Leave empty if you don't use it</%text>
 
instance_id =
 

	
 
<%text>## alternative return HTTP header for failed authentication. Default HTTP</%text>
 
<%text>## response is 401 HTTPUnauthorized. Currently Mercurial clients have trouble with</%text>
 
<%text>## handling that. Set this variable to 403 to return HTTPForbidden</%text>
 
auth_ret_code =
 

	
 
<%text>## locking return code. When repository is locked return this HTTP code. 2XX</%text>
 
<%text>## codes don't break the transactions while 4XX codes do</%text>
 
lock_ret_code = 423
 

	
 
<%text>## allows to change the repository location in settings page</%text>
 
allow_repo_location_change = True
 

	
 
<%text>## allows to setup custom hooks in settings page</%text>
 
allow_custom_hooks_settings = True
 

	
 
<%text>####################################</%text>
 
<%text>###        CELERY CONFIG        ####</%text>
 
<%text>####################################</%text>
 

	
 
use_celery = false
 
broker.host = localhost
 
broker.vhost = rabbitmqhost
 
broker.port = 5672
 
broker.user = rabbitmq
kallithea/config/deployment.ini_tmpl
Show inline comments
 
@@ -253,54 +253,48 @@ default_encoding = utf8
 
## {id} will be all groups matched from this pattern
 

	
 
issue_pat = (?:\s*#)(\d+)
 

	
 
## server url to the issue, each {id} will be replaced with match
 
## fetched from the regex and {repo} is replaced with full repository name
 
## including groups {repo_name} is replaced with just name of repo
 

	
 
issue_server_link = https://issues.example.com/{repo}/issue/{id}
 

	
 
## prefix to add to link to indicate it's an url
 
## #314 will be replaced by <issue_prefix><id>
 

	
 
issue_prefix = #
 

	
 
## issue_pat, issue_server_link, issue_prefix can have suffixes to specify
 
## multiple patterns, to other issues server, wiki or others
 
## below an example how to create a wiki pattern
 
# wiki-some-id -> https://wiki.example.com/some-id
 

	
 
#issue_pat_wiki = (?:wiki-)(.+)
 
#issue_server_link_wiki = https://wiki.example.com/{id}
 
#issue_prefix_wiki = WIKI-
 

	
 
## instance-id prefix
 
## a prefix key for this instance used for cache invalidation when running
 
## multiple instances of kallithea, make sure it's globally unique for
 
## all running kallithea instances. Leave empty if you don't use it
 
instance_id =
 

	
 
## alternative return HTTP header for failed authentication. Default HTTP
 
## response is 401 HTTPUnauthorized. Currently Mercurial clients have trouble with
 
## handling that. Set this variable to 403 to return HTTPForbidden
 
auth_ret_code =
 

	
 
## locking return code. When repository is locked return this HTTP code. 2XX
 
## codes don't break the transactions while 4XX codes do
 
lock_ret_code = 423
 

	
 
## allows to change the repository location in settings page
 
allow_repo_location_change = True
 

	
 
## allows to setup custom hooks in settings page
 
allow_custom_hooks_settings = True
 

	
 
####################################
 
###        CELERY CONFIG        ####
 
####################################
 

	
 
use_celery = false
 
broker.host = localhost
 
broker.vhost = rabbitmqhost
 
broker.port = 5672
 
broker.user = rabbitmq
kallithea/config/environment.py
Show inline comments
 
@@ -97,44 +97,44 @@ def load_environment(global_conf, app_co
 
            # swap config if we pass enviroment variable
 
            config['sqlalchemy.db1.url'] = os.environ.get('TEST_DB')
 

	
 
        from kallithea.lib.utils import create_test_env, create_test_index
 
        from kallithea.tests import TESTS_TMP_PATH
 
        #set KALLITHEA_NO_TMP_PATH=1 to disable re-creating the database and
 
        #test repos
 
        if test_env:
 
            create_test_env(TESTS_TMP_PATH, config)
 
        #set KALLITHEA_WHOOSH_TEST_DISABLE=1 to disable whoosh index during tests
 
        if test_index:
 
            create_test_index(TESTS_TMP_PATH, config, True)
 

	
 
    DbManage.check_waitress()
 
    # MULTIPLE DB configs
 
    # Setup the SQLAlchemy database engine
 
    sa_engine_db1 = engine_from_config(config, 'sqlalchemy.db1.')
 
    init_model(sa_engine_db1)
 

	
 
    set_available_permissions(config)
 
    repos_path = make_ui('db').configitems('paths')[0][1]
 
    config['base_path'] = repos_path
 
    set_app_settings(config)
 

	
 
    instance_id = kallithea.CONFIG.get('instance_id')
 
    instance_id = kallithea.CONFIG.get('instance_id', '*')
 
    if instance_id == '*':
 
        instance_id = '%s-%s' % (platform.uname()[1], os.getpid())
 
        kallithea.CONFIG['instance_id'] = instance_id
 

	
 
    # CONFIGURATION OPTIONS HERE (note: all config options will override
 
    # any Pylons config options)
 

	
 
    # store config reference into our module to skip import magic of
 
    # pylons
 
    kallithea.CONFIG.update(config)
 
    set_vcs_config(kallithea.CONFIG)
 

	
 
    #check git version
 
    check_git_version()
 

	
 
    if str2bool(config.get('initial_repo_scan', True)):
 
        repo2db_mapper(ScmModel().repo_scan(repos_path),
 
                       remove_obsolete=False, install_git_hooks=False)
 
    return config
kallithea/tests/test.ini
Show inline comments
 
@@ -259,54 +259,48 @@ default_encoding = utf8
 
## {id} will be all groups matched from this pattern
 

	
 
issue_pat = (?:\s*#)(\d+)
 

	
 
## server url to the issue, each {id} will be replaced with match
 
## fetched from the regex and {repo} is replaced with full repository name
 
## including groups {repo_name} is replaced with just name of repo
 

	
 
issue_server_link = https://issues.example.com/{repo}/issue/{id}
 

	
 
## prefix to add to link to indicate it's an url
 
## #314 will be replaced by <issue_prefix><id>
 

	
 
issue_prefix = #
 

	
 
## issue_pat, issue_server_link, issue_prefix can have suffixes to specify
 
## multiple patterns, to other issues server, wiki or others
 
## below an example how to create a wiki pattern
 
# wiki-some-id -> https://wiki.example.com/some-id
 

	
 
#issue_pat_wiki = (?:wiki-)(.+)
 
#issue_server_link_wiki = https://wiki.example.com/{id}
 
#issue_prefix_wiki = WIKI-
 

	
 
## instance-id prefix
 
## a prefix key for this instance used for cache invalidation when running
 
## multiple instances of kallithea, make sure it's globally unique for
 
## all running kallithea instances. Leave empty if you don't use it
 
instance_id =
 

	
 
## alternative return HTTP header for failed authentication. Default HTTP
 
## response is 401 HTTPUnauthorized. Currently Mercurial clients have trouble with
 
## handling that. Set this variable to 403 to return HTTPForbidden
 
auth_ret_code =
 

	
 
## locking return code. When repository is locked return this HTTP code. 2XX
 
## codes don't break the transactions while 4XX codes do
 
lock_ret_code = 423
 

	
 
## allows to change the repository location in settings page
 
allow_repo_location_change = True
 

	
 
## allows to setup custom hooks in settings page
 
allow_custom_hooks_settings = True
 

	
 
####################################
 
###        CELERY CONFIG        ####
 
####################################
 

	
 
use_celery = false
 
broker.host = localhost
 
broker.vhost = rabbitmqhost
 
broker.port = 5672
 
broker.user = rabbitmq
0 comments (0 inline, 0 general)