Changeset - 6afa528ee30e
[Not reviewed]
default
0 7 0
Mads Kiilerich - 10 years ago 2016-03-14 16:17:46
madski@unity3d.com
db: get rid of vcs_full_cache - it should always be used

It might make tests less deterministic, but it makes sure that we test what we actually use.
7 files changed with 6 insertions and 30 deletions:
0 comments (0 inline, 0 general)
development.ini
Show inline comments
 
@@ -190,27 +190,24 @@ index_dir = %(here)s/data/index
 
#initial_repo_scan = false
 
initial_repo_scan = true
 

	
 
## uncomment and set this path to use archive download cache
 
archive_cache_dir = %(here)s/tarballcache
 

	
 
## change this to unique ID for security
 
app_instance_uuid = development-not-secret
 

	
 
## cut off limit for large diffs (size in bytes)
 
cut_off_limit = 256000
 

	
 
## use cache version of scm repo everywhere
 
vcs_full_cache = true
 

	
 
## force https in Kallithea, fixes https redirects, assumes it's always https
 
force_https = false
 

	
 
## use Strict-Transport-Security headers
 
use_htsts = false
 

	
 
## number of commits stats will parse on each iteration
 
commit_parse_limit = 25
 

	
 
## path to git executable
 
git_path = git
 

	
docs/usage/performance.rst
Show inline comments
 
.. _performance:
 

	
 
================================
 
Optimizing Kallithea performance
 
================================
 

	
 
When serving a large amount of big repositories, Kallithea can start
 
performing slower than expected. Because of the demanding nature of handling large
 
amounts of data from version control systems, here are some tips on how to get
 
the best performance.
 

	
 
* Kallithea is often I/O bound, and hence a fast disk (SSD/SAN) is
 
Follow these few steps to improve performance of Kallithea system.
 

	
 
1.  Kallithea is often I/O bound, and hence a fast disk (SSD/SAN) is
 
  usually more important than a fast CPU.
 

	
 
* Sluggish loading of the front page can easily be fixed by grouping repositories or by
 
  increasing cache size (see below). This includes using the lightweight dashboard
 
  option and ``vcs_full_cache`` setting in .ini file.
 

	
 
Follow these few steps to improve performance of Kallithea system.
 

	
 
1. Increase cache
 
2. Increase cache
 

	
 
    Tweak beaker cache settings in the ini file. The actual effect of that
 
    is questionable.
 

	
 
2. Switch from SQLite to PostgreSQL or MySQL
 
3. Switch from SQLite to PostgreSQL or MySQL
 

	
 
    SQLite is a good option when having a small load on the system. But due to
 
    locking issues with SQLite, it is not recommended to use it for larger
 
    deployments. Switching to MySQL or PostgreSQL will result in an immediate
 
    performance increase. A tool like SQLAlchemyGrate_ can be used for
 
    migrating to another database platform.
 

	
 
3. Scale Kallithea horizontally
 
4. Scale Kallithea horizontally
 

	
 
    Scaling horizontally can give huge performance benefits when dealing with
 
    large amounts of traffic (many users, CI servers, etc.). Kallithea can be
 
    scaled horizontally on one (recommended) or multiple machines. In order
 
    to scale horizontally you need to do the following:
 

	
 
    - Each instance's ``data`` storage needs to be configured to be stored on a
 
      shared disk storage, preferably together with repositories. This ``data``
 
      dir contains template caches, sessions, whoosh index and is used for
 
      task locking (so it is safe across multiple instances). Set the
 
      ``cache_dir``, ``index_dir``, ``beaker.cache.data_dir``, ``beaker.cache.lock_dir``
 
      variables in each .ini file to a shared location across Kallithea instances
kallithea/bin/template.ini.mako
Show inline comments
 
@@ -187,27 +187,24 @@ index_dir = ${here}/data/index
 
<%text>## set to false after first startup, to allow faster server restarts.</%text>
 
initial_repo_scan = false
 

	
 
<%text>## uncomment and set this path to use archive download cache</%text>
 
archive_cache_dir = ${here}/tarballcache
 

	
 
<%text>## change this to unique ID for security</%text>
 
app_instance_uuid = ${uuid()}
 

	
 
<%text>## cut off limit for large diffs (size in bytes)</%text>
 
cut_off_limit = 256000
 

	
 
<%text>## use cache version of scm repo everywhere</%text>
 
vcs_full_cache = true
 

	
 
<%text>## force https in Kallithea, fixes https redirects, assumes it's always https</%text>
 
force_https = false
 

	
 
<%text>## use Strict-Transport-Security headers</%text>
 
use_htsts = false
 

	
 
<%text>## number of commits stats will parse on each iteration</%text>
 
commit_parse_limit = 25
 

	
 
<%text>## path to git executable</%text>
 
git_path = git
 

	
kallithea/config/deployment.ini_tmpl
Show inline comments
 
@@ -183,27 +183,24 @@ index_dir = %(here)s/data/index
 
## set to false after first startup, to allow faster server restarts.
 
initial_repo_scan = false
 

	
 
## uncomment and set this path to use archive download cache
 
archive_cache_dir = %(here)s/tarballcache
 

	
 
## change this to unique ID for security
 
app_instance_uuid = ${app_instance_uuid}
 

	
 
## cut off limit for large diffs (size in bytes)
 
cut_off_limit = 256000
 

	
 
## use cache version of scm repo everywhere
 
vcs_full_cache = true
 

	
 
## force https in Kallithea, fixes https redirects, assumes it's always https
 
force_https = false
 

	
 
## use Strict-Transport-Security headers
 
use_htsts = false
 

	
 
## number of commits stats will parse on each iteration
 
commit_parse_limit = 25
 

	
 
## path to git executable
 
git_path = git
 

	
kallithea/model/db.py
Show inline comments
 
@@ -1430,29 +1430,25 @@ class Repository(Base, BaseModel):
 
    #==========================================================================
 
    # SCM CACHE INSTANCE
 
    #==========================================================================
 

	
 
    def set_invalidate(self):
 
        """
 
        Mark caches of this repo as invalid.
 
        """
 
        CacheInvalidation.set_invalidate(self.repo_name)
 

	
 
    @property
 
    def scm_instance(self):
 
        import kallithea
 
        full_cache = str2bool(kallithea.CONFIG.get('vcs_full_cache'))
 
        if full_cache:
 
            return self.scm_instance_cached()
 
        return self.__get_instance()
 

	
 
    def scm_instance_cached(self, valid_cache_keys=None):
 
        @cache_region('long_term', 'scm_instance_cached')
 
        def _c(repo_name): # repo_name is just for the cache key
 
            log.debug('Creating new %s scm_instance and populating cache', repo_name)
 
            return self.scm_instance_no_cache()
 
        rn = self.repo_name
 

	
 
        valid = CacheInvalidation.test_and_set_valid(rn, None, valid_cache_keys=valid_cache_keys)
 
        if not valid:
 
            log.debug('Cache for %s invalidated, getting new object', rn)
 
            region_invalidate(_c, None, 'scm_instance_cached', rn)
kallithea/tests/test.ini
Show inline comments
 
################################################################################
 
################################################################################
 
# Kallithea - config for tests:                                                #
 
# initial_repo_scan = true                                                     #
 
# vcs_full_cache = false                                                       #
 
# sqlalchemy and kallithea_test.sqlite                                         #
 
# custom logging                                                               #
 
#                                                                              #
 
# The %(here)s variable will be replaced with the parent directory of this file#
 
################################################################################
 
################################################################################
 

	
 
[DEFAULT]
 
debug = true
 
pdebug = false
 

	
 
################################################################################
 
@@ -189,28 +188,24 @@ index_dir = %(here)s/data/index
 
#initial_repo_scan = false
 
initial_repo_scan = true
 

	
 
## uncomment and set this path to use archive download cache
 
archive_cache_dir = %(here)s/tarballcache
 

	
 
## change this to unique ID for security
 
app_instance_uuid = test
 

	
 
## cut off limit for large diffs (size in bytes)
 
cut_off_limit = 256000
 

	
 
## use cache version of scm repo everywhere
 
#vcs_full_cache = true
 
vcs_full_cache = false
 

	
 
## force https in Kallithea, fixes https redirects, assumes it's always https
 
force_https = false
 

	
 
## use Strict-Transport-Security headers
 
use_htsts = false
 

	
 
## number of commits stats will parse on each iteration
 
commit_parse_limit = 25
 

	
 
## path to git executable
 
git_path = git
 

	
scripts/generate-ini.py
Show inline comments
 
@@ -31,38 +31,36 @@ ini_files = [
 
    ('kallithea/config/deployment.ini_tmpl',
 
        '''
 
        Kallithea - Example config
 

	
 
        The %(here)s variable will be replaced with the parent directory of this file
 
        ''',
 
        {}, # exactly the same settings as template.ini.mako
 
    ),
 
    ('kallithea/tests/test.ini',
 
        '''
 
        Kallithea - config for tests:
 
        initial_repo_scan = true
 
        vcs_full_cache = false
 
        sqlalchemy and kallithea_test.sqlite
 
        custom logging
 

	
 
        The %(here)s variable will be replaced with the parent directory of this file
 
        ''',
 
        {
 
            '[server:main]': {
 
                'port': '4999',
 
            },
 
            '[app:main]': {
 
                'initial_repo_scan': 'true',
 
                'app_instance_uuid': 'test',
 
                'vcs_full_cache': 'false',
 
                'show_revision_number': 'true',
 
                'beaker.cache.sql_cache_short.expire': '1',
 
                'beaker.session.secret': '{74e0cd75-b339-478b-b129-07dd221def1f}',
 
                'sqlalchemy.db1.url': 'sqlite:///%(here)s/kallithea_test.sqlite',
 
            },
 
            '[logger_root]': {
 
                'level': 'DEBUG',
 
            },
 
            '[logger_sqlalchemy]': {
 
                'level': 'ERROR',
 
                'handlers': 'console',
 
            },
0 comments (0 inline, 0 general)