Changeset - aac24db58ce8
[Not reviewed]
beta
0 4 0
Marcin Kuzminski - 15 years ago 2010-11-27 01:43:04
marcin@python-works.com
fixed cache problem,
updated docs
4 files changed with 46 insertions and 4 deletions:
0 comments (0 inline, 0 general)
docs/changelog.rst
Show inline comments
 
@@ -3,49 +3,49 @@
 
Changelog
 
=========
 

	
 
1.1.0 (**2010-XX-XX**)
 
----------------------
 

	
 
:status: in-progress
 
:branch: beta
 

	
 
news
 
++++
 

	
 
- rewrite of internals for vcs >=0.1.10
 
- anonymous access, authentication via ldap
 
- performance upgrade for cached repos list - each repository has it's own 
 
  cache that's invalidated when needed.
 
- main page quick filter for filtering repositories
 
- user dashboards with ability to follow chosen repositories actions
 
- sends email to admin on new user registration
 
- added cache/statistics reset options into repository settings
 
- more detailed action logger (based on hooks) with pushed changesets lists
 
  and options to disable those hooks from admin panel
 
- introduced new enhanced changelog for merges that shows more accurate results
 
- gui optimizations, fixed application width to 1024px
 
- whoosh,celeryd,upgrade moved to paster command
 
- whoosh, celeryd, upgrade moved to paster command
 

	
 
fixes
 
+++++
 

	
 
- fixes #61 forked repo was showing only after cache expired
 
- fixes #76 no confirmation on user deletes
 
- fixes #66 Name field misspelled
 
- fixes #72 block user removal when he owns repositories
 
- fixes #69 added password confirmation fields
 
- numerous small bugfixes
 
- a lot of fixes and tweaks for file browser
 
- fixed detached session issues
 

	
 
(special thanks for TkSoh for detailed feedback)
 

	
 

	
 
1.0.2 (**2010-11-12**)
 
----------------------
 

	
 
news
 
++++
 

	
 
- tested under python2.7
 
- bumped sqlalchemy and celery versions
docs/setup.rst
Show inline comments
 
@@ -110,48 +110,64 @@ Here's a typical ldap setup::
 
 Password     = <password>              #password for ldap server(if required)
 
 Base DN      = uid=%(user)s,CN=users,DC=host,DC=domain,DC=org
 
 
 

	
 
`Account` and `Password` are optional, and used for two-phase ldap 
 
authentication so those are credentials to access Your ldap, if it doesn't 
 
support anonymous search/user lookups. 
 

	
 
Base DN must have %(user)s template inside, it's a placer where Your uid used
 
to login would go, it allows admins to specify not standard schema for uid 
 
variable
 

	
 
If all data are entered correctly, and `python-ldap` is properly installed
 
Users should be granted to access RhodeCode wit ldap accounts. When 
 
logging at the first time an special ldap account is created inside RhodeCode, 
 
so You can control over permissions even on ldap users. If such user exists 
 
already in RhodeCode database ldap user with the same username would be not 
 
able to access RhodeCode.
 

	
 
If You have problems with ldap access and believe You entered correct 
 
information check out the RhodeCode logs,any error messages sent from 
 
ldap will be saved there.
 

	
 

	
 

	
 
Setting Up Celery
 
-----------------
 

	
 
Since version 1.1 celery is configured by the rhodecode ini configuration files
 
simply set use_celery=true in the ini file then add / change the configuration 
 
variables inside the ini file.
 

	
 
Remember that the ini files uses format with '.' not with '_' like celery
 
so for example setting `BROKER_HOST` in celery means setting `broker.host` in
 
the config file.
 

	
 
In order to make start using celery run::
 
 paster celeryd <configfile.ini>
 

	
 

	
 
Nginx virtual host example
 
--------------------------
 

	
 
Sample config for nginx using proxy::
 

	
 
 server {
 
    listen          80;
 
    server_name     hg.myserver.com;
 
    access_log      /var/log/nginx/rhodecode.access.log;
 
    error_log       /var/log/nginx/rhodecode.error.log;
 
    location / {
 
            root /var/www/rhodecode/rhodecode/public/;
 
            if (!-f $request_filename){
 
                proxy_pass      http://127.0.0.1:5000;
 
            }
 
            #this is important for https !!!
 
            proxy_set_header X-Url-Scheme $scheme;
 
            include         /etc/nginx/proxy.conf;  
 
    }
 
 }  
 
  
 
Here's the proxy.conf. It's tuned so it'll not timeout on long
 
pushes and also on large pushes::
 

	
rhodecode/lib/celerylib/tasks.py
Show inline comments
 
from celery.decorators import task
 

	
 
import os
 
import traceback
 
import beaker
 
from time import mktime
 
from operator import itemgetter
 

	
 
from pylons import config
 
from pylons.i18n.translation import _
 

	
 
from rhodecode.lib.celerylib import run_task, locked_task, str2bool
 
from rhodecode.lib.helpers import person
 
from rhodecode.lib.smtp_mailer import SmtpMailer
 
from rhodecode.lib.utils import OrderedDict
 
from rhodecode.model import init_model
 
from rhodecode.model import meta
 
from rhodecode.model.db import RhodeCodeUi
 

	
 
from vcs.backends import get_repo
 

	
 
from sqlalchemy import engine_from_config
 

	
 
#set cache regions for beaker so celery can utilise it
 
def add_cache(settings):
 
    cache_settings = {'regions':None}
 
    for key in settings.keys():
 
        for prefix in ['beaker.cache.', 'cache.']:
 
            if key.startswith(prefix):
 
                name = key.split(prefix)[1].strip()
 
                cache_settings[name] = settings[key].strip()
 
    if cache_settings['regions']:
 
        for region in cache_settings['regions'].split(','):
 
            region = region.strip()
 
            region_settings = {}
 
            for key, value in cache_settings.items():
 
                if key.startswith(region):
 
                    region_settings[key.split('.')[1]] = value
 
            region_settings['expire'] = int(region_settings.get('expire',
 
                                                                60))
 
            region_settings.setdefault('lock_dir',
 
                                       cache_settings.get('lock_dir'))
 
            if 'type' not in region_settings:
 
                region_settings['type'] = cache_settings.get('type',
 
                                                             'memory')
 
            beaker.cache.cache_regions[region] = region_settings
 
add_cache(config)
 

	
 
try:
 
    import json
 
except ImportError:
 
    #python 2.5 compatibility
 
    import simplejson as json
 

	
 
__all__ = ['whoosh_index', 'get_commits_stats',
 
           'reset_user_password', 'send_email']
 

	
 
CELERY_ON = str2bool(config['app_conf'].get('use_celery'))
 

	
 
def get_session():
 
    if CELERY_ON:
 
        engine = engine_from_config(config, 'sqlalchemy.db1.')
 
        init_model(engine)
 
    sa = meta.Session()
 
    return sa
 

	
 
def get_repos_path():
 
    sa = get_session()
 
    q = sa.query(RhodeCodeUi).filter(RhodeCodeUi.ui_key == '/').one()
 
    return q.ui_value
 

	
 
@task
 
@locked_task
 
def whoosh_index(repo_location, full_index):
 
    log = whoosh_index.get_logger()
 
    from rhodecode.lib.indexers.daemon import WhooshIndexingDaemon
 
    index_location = config['index_dir']
 
    WhooshIndexingDaemon(index_location=index_location,
 
                         repo_location=repo_location).run(full_index=full_index)
 
                         repo_location=repo_location, sa=get_session())\
 
                         .run(full_index=full_index)
 

	
 
@task
 
@locked_task
 
def get_commits_stats(repo_name, ts_min_y, ts_max_y):
 
    from rhodecode.model.db import Statistics, Repository
 
    log = get_commits_stats.get_logger()
 

	
 
    #for js data compatibilty
 
    author_key_cleaner = lambda k: person(k).replace('"', "")
 

	
 
    commits_by_day_author_aggregate = {}
 
    commits_by_day_aggregate = {}
 
    repos_path = get_repos_path()
 
    p = os.path.join(repos_path, repo_name)
 
    repo = get_repo(p)
 

	
 
    skip_date_limit = True
 
    parse_limit = 250 #limit for single task changeset parsing optimal for
 
    last_rev = 0
 
    last_cs = None
 
    timegetter = itemgetter('time')
 

	
 
    sa = get_session()
 

	
rhodecode/lib/indexers/daemon.py
Show inline comments
 
@@ -46,60 +46,60 @@ import logging
 

	
 
log = logging.getLogger('whooshIndexer')
 
# create logger
 
log.setLevel(logging.DEBUG)
 
log.propagate = False
 
# create console handler and set level to debug
 
ch = logging.StreamHandler()
 
ch.setLevel(logging.DEBUG)
 

	
 
# create formatter
 
formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
 

	
 
# add formatter to ch
 
ch.setFormatter(formatter)
 

	
 
# add ch to logger
 
log.addHandler(ch)
 

	
 
class WhooshIndexingDaemon(object):
 
    """
 
    Deamon for atomic jobs
 
    """
 

	
 
    def __init__(self, indexname='HG_INDEX', index_location=None,
 
                 repo_location=None):
 
                 repo_location=None, sa=None):
 
        self.indexname = indexname
 

	
 
        self.index_location = index_location
 
        if not index_location:
 
            raise Exception('You have to provide index location')
 

	
 
        self.repo_location = repo_location
 
        if not repo_location:
 
            raise Exception('You have to provide repositories location')
 

	
 
        self.repo_paths = ScmModel().repo_scan(self.repo_location, None)
 
        self.repo_paths = ScmModel(sa).repo_scan(self.repo_location, None)
 
        self.initial = False
 
        if not os.path.isdir(self.index_location):
 
            os.makedirs(self.index_location)
 
            log.info('Cannot run incremental index since it does not'
 
                     ' yet exist running full build')
 
            self.initial = True
 

	
 
    def get_paths(self, repo):
 
        """recursive walk in root dir and return a set of all path in that dir
 
        based on repository walk function
 
        """
 
        index_paths_ = set()
 
        try:
 
            for topnode, dirs, files in repo.walk('/', 'tip'):
 
                for f in files:
 
                    index_paths_.add(jn(repo.path, f.path))
 
                for dir in dirs:
 
                    for f in files:
 
                        index_paths_.add(jn(repo.path, f.path))
 

	
 
        except RepositoryError:
 
            pass
 
        return index_paths_
 

	
0 comments (0 inline, 0 general)