Changeset - 341beaa9edba
[Not reviewed]
beta
0 10 0
Marcin Kuzminski - 15 years ago 2010-11-13 02:29:46
marcin@python-works.com
Implemented whoosh index building as paster command.
docs update
fixed manifest.in for missing yui file.
Fixed setup to beta
added base for paster upgrade-db command
10 files changed with 57 insertions and 35 deletions:
0 comments (0 inline, 0 general)
MANIFEST.in
Show inline comments
 
include rhodecode/config/deployment.ini_tmpl
 

	
 
include README.rst
 
recursive-include rhodecode/i18n/ *
 

	
 
#images
 
recursive-include rhodecode/public/css *
 
recursive-include rhodecode/public/images *
 
#js
 
include rhodecode/public/js/yui2.js
 
include rhodecode/public/js/yui2a.js
 
include rhodecode/public/js/excanvas.min.js
 
include rhodecode/public/js/yui.flot.js
 
include rhodecode/public/js/graph.js
 
#templates
 
recursive-include rhodecode/templates *
development.ini
Show inline comments
 
@@ -34,24 +34,25 @@ threadpool_max_requests = 6
 
use_threadpool = false
 

	
 
use = egg:Paste#http
 
host = 127.0.0.1
 
port = 5000
 

	
 
[app:main]
 
use = egg:rhodecode
 
full_stack = true
 
static_files = true
 
lang=en
 
cache_dir = %(here)s/data
 
index_dir = %(here)s/data/index
 

	
 
####################################
 
###         BEAKER CACHE        ####
 
####################################
 
beaker.cache.data_dir=/%(here)s/data/cache/data
 
beaker.cache.lock_dir=/%(here)s/data/cache/lock
 
beaker.cache.regions=super_short_term,short_term,long_term,sql_cache_short,sql_cache_med,sql_cache_long
 

	
 
beaker.cache.super_short_term.type=memory
 
beaker.cache.super_short_term.expire=10
 

	
 
beaker.cache.short_term.type=memory
docs/changelog.rst
Show inline comments
 
@@ -8,24 +8,25 @@ Changelog
 
- git support with push/pull via RhodeCode web interface
 
- rewrite of internals for vcs >=0.1.9
 
- anonymous access
 
- performance upgrade for cached repos list - each repository has it's own 
 
  cache that's invalidated when needed.
 
- main page quick filter for filtering repositories
 
- more detailed action logger (based on hooks) with pushed changesets lists
 
  and options to disable those hooks from admin panel
 
- a lot of fixes and tweaks for file browser
 
- introduced new enhanced changelog for merges that shows more accurate results
 
- gui optimizations, fixed application width to 1024px
 
- numerous small bugfixes
 
- whoosh index moved to paster command
 

	
 
1.0.2 (**2010-11-12**)
 
----------------------
 

	
 
- fixed #59 missing graph.js
 
- fixed repo_size crash when repository had broken symlinks
 
- fixed python2.5 crashes.
 
- tested under python2.7
 
- bumped sqlalchemy and celery versions
 

	
 
1.0.1 (**2010-11-10**)
 
----------------------
docs/setup.rst
Show inline comments
 
@@ -32,39 +32,58 @@ Setting up the application
 
::
 
 
 
 paster serve production.ini
 
 
 
- This command runs the rhodecode server the app should be available at the 
 
  127.0.0.1:5000. This ip and port is configurable via the production.ini 
 
  file  created in previous step
 
- Use admin account you created to login.
 
- Default permissions on each repository is read, and owner is admin. So 
 
  remember to update these if needed.
 
  
 
    
 
Setting up Whoosh
 
-----------------
 
Setting up Whoosh full text search
 
----------------------------------
 

	
 
Index for whoosh can be build starting from version 1.1 using paster command
 
passing repo locations to index, as well as Your config file that stores
 
whoosh index files locations. There is possible to pass `-f` to the options
 
to enable full index rebuild. Without that indexing will run always in in
 
incremental mode.
 

	
 
::
 
 paster make-index --repo-location=<location for repos> production.ini  
 

	
 
for full index rebuild You can use
 

	
 
::
 
 paster make-index -f --repo-location=<location for repos> production.ini
 

	
 
- For full text search You can either put crontab entry for
 

	
 
This command can be run even from crontab in order to do periodical 
 
index builds and keep Your index always up to date. An example entry might 
 
look like this
 

	
 
::
 
 
 
 python /var/www/rhodecode/<rhodecode_installation_path>/lib/indexers/daemon.py incremental <put_here_path_to_repos>
 
 /path/to/python/bin/paster --repo-location=<location for repos> /path/to/rhodecode/production.ini
 
  
 
When using incremental mode whoosh will check last modification date of each file
 
and add it to reindex if newer file is available. Also indexing daemon checks
 
for removed files and removes them from index. Sometime You might want to rebuild
 
index from scrach, in admin pannel You can check `build from scratch` flag
 
and in standalone daemon You can pass `full` instead on incremental to build
 
remove previos index and build new one.
 
When using incremental(default) mode whoosh will check last modification date 
 
of each file and add it to reindex if newer file is available. Also indexing 
 
daemon checks for removed files and removes them from index. 
 

	
 
Sometime You might want to rebuild index from scratch. You can do that using 
 
the `-f` flag passed to paster command or, in admin panel You can check 
 
`build from scratch` flag.
 

	
 
Nginx virtual host example
 
--------------------------
 

	
 
Sample config for nginx::
 

	
 
 server {
 
    listen          80;
 
    server_name     hg.myserver.com;
 
    access_log      /var/log/nginx/rhodecode.access.log;
 
    error_log       /var/log/nginx/rhodecode.error.log;
 
    location / {
production.ini
Show inline comments
 
@@ -34,24 +34,25 @@ threadpool_max_requests = 2
 
use_threadpool = true
 

	
 
use = egg:Paste#http
 
host = 127.0.0.1
 
port = 8001
 

	
 
[app:main]
 
use = egg:rhodecode
 
full_stack = true
 
static_files = false
 
lang=en
 
cache_dir = %(here)s/data
 
index_dir = %(here)s/data/index
 

	
 
####################################
 
###         BEAKER CACHE        ####
 
####################################
 
beaker.cache.data_dir=/%(here)s/data/cache/data
 
beaker.cache.lock_dir=/%(here)s/data/cache/lock
 
beaker.cache.regions=super_short_term,short_term,long_term,sql_cache_short,sql_cache_med,sql_cache_long
 

	
 
beaker.cache.super_short_term.type=memory
 
beaker.cache.super_short_term.expire=10
 

	
 
beaker.cache.short_term.type=memory
rhodecode/config/deployment.ini_tmpl
Show inline comments
 
@@ -34,24 +34,25 @@ threadpool_max_requests = 10
 
use_threadpool = true
 

	
 
use = egg:Paste#http
 
host = 127.0.0.1
 
port = 5000
 

	
 
[app:main]
 
use = egg:rhodecode
 
full_stack = true
 
static_files = true
 
lang=en
 
cache_dir = %(here)s/data
 
index_dir = %(here)s/data/index
 
app_instance_uuid = ${app_instance_uuid}
 

	
 
####################################
 
###         BEAKER CACHE        ####
 
####################################
 
beaker.cache.data_dir=/%(here)s/data/cache/data
 
beaker.cache.lock_dir=/%(here)s/data/cache/lock
 
beaker.cache.regions=super_short_term,short_term,long_term,sql_cache_short,sql_cache_med,sql_cache_long
 

	
 
beaker.cache.super_short_term.type=memory
 
beaker.cache.super_short_term.expire=10
 

	
rhodecode/lib/indexers/__init__.py
Show inline comments
 
@@ -50,61 +50,57 @@ FORMATTER = HtmlFormatter('span', betwee
 
FRAGMENTER = SimpleFragmenter(200)
 

	
 
from paste.script import command
 
import ConfigParser
 

	
 
class MakeIndex(command.Command):
 

	
 
    max_args = 1
 
    min_args = 1
 

	
 
    usage = "CONFIG_FILE"
 
    summary = "Creates index for full text search given configuration file"
 
    group_name = "Whoosh indexing"
 

	
 
    group_name = "RhodeCode"
 
    takes_config_file = -1
 
    parser = command.Command.standard_parser(verbose=True)
 
#    parser.add_option('--repo-location',
 
#                      action='store',
 
#                      dest='repo_location',
 
#                      help="Specifies repositories location to index",
 
#                      )
 
    parser.add_option('--repo-location',
 
                      action='store',
 
                      dest='repo_location',
 
                      help="Specifies repositories location to index REQUIRED",
 
                      )
 
    parser.add_option('-f',
 
                      action='store_true',
 
                      dest='full_index',
 
                      help="Specifies that index should be made full i.e"
 
                            " destroy old and build from scratch",
 
                      default=False)
 
    def command(self):
 
        config_name = self.args[0]
 

	
 
        p = config_name.split('/')
 
        if len(p) == 1:
 
            root = '.'
 
        else:
 
            root = '/'.join(p[:-1])
 
        print root
 
        root = '.' if len(p) == 1 else '/'.join(p[:-1])
 
        config = ConfigParser.ConfigParser({'here':root})
 
        config.read(config_name)
 
        print dict(config.items('app:main'))['index_dir']
 

	
 
        index_location = dict(config.items('app:main'))['index_dir']
 
        #return
 
        repo_location = self.options.repo_location
 

	
 
        #=======================================================================
 
        #======================================================================
 
        # WHOOSH DAEMON
 
        #=======================================================================
 
        #======================================================================
 
        from rhodecode.lib.pidlock import LockHeld, DaemonLock
 
        from rhodecode.lib.indexers.daemon import WhooshIndexingDaemon
 
        try:
 
            l = DaemonLock()
 
            WhooshIndexingDaemon(index_location=index_location)\
 
            WhooshIndexingDaemon(index_location=index_location,
 
                                 repo_location=repo_location)\
 
                .run(full_index=self.options.full_index)
 
            l.release()
 
        except LockHeld:
 
            sys.exit(1)
 

	
 

	
 
class ResultWrapper(object):
 
    def __init__(self, search_type, searcher, matcher, highlight_items):
 
        self.search_type = search_type
 
        self.searcher = searcher
 
        self.matcher = matcher
 
        self.highlight_items = highlight_items
rhodecode/lib/indexers/daemon.py
Show inline comments
 
@@ -69,62 +69,60 @@ class WhooshIndexingDaemon(object):
 
    def __init__(self, indexname='HG_INDEX', index_location=None,
 
                 repo_location=None):
 
        self.indexname = indexname
 

	
 
        self.index_location = index_location
 
        if not index_location:
 
            raise Exception('You have to provide index location')
 

	
 
        self.repo_location = repo_location
 
        if not repo_location:
 
            raise Exception('You have to provide repositories location')
 

	
 

	
 

	
 
        self.repo_paths = HgModel.repo_scan('/', self.repo_location, None, True)
 
        self.repo_paths = HgModel().repo_scan(self.repo_location, None, True)
 
        self.initial = False
 
        if not os.path.isdir(self.index_location):
 
            os.mkdir(self.index_location)
 
            log.info('Cannot run incremental index since it does not'
 
                     ' yet exist running full build')
 
            self.initial = True
 

	
 
    def get_paths(self, repo):
 
        """
 
        recursive walk in root dir and return a set of all path in that dir
 
        """recursive walk in root dir and return a set of all path in that dir
 
        based on repository walk function
 
        """
 
        index_paths_ = set()
 
        try:
 
            for topnode, dirs, files in repo.walk('/', 'tip'):
 
                for f in files:
 
                    index_paths_.add(jn(repo.path, f.path))
 
                for dir in dirs:
 
                    for f in files:
 
                        index_paths_.add(jn(repo.path, f.path))
 

	
 
        except RepositoryError:
 
            pass
 
        return index_paths_
 

	
 
    def get_node(self, repo, path):
 
        n_path = path[len(repo.path) + 1:]
 
        node = repo.get_changeset().get_node(n_path)
 
        return node
 

	
 
    def get_node_mtime(self, node):
 
        return mktime(node.last_changeset.date.timetuple())
 

	
 
    def add_doc(self, writer, path, repo):
 
        """Adding doc to writer"""
 
        """Adding doc to writer this function itself fetches data from
 
        the instance of vcs backend"""
 
        node = self.get_node(repo, path)
 

	
 
        #we just index the content of chosen files
 
        if node.extension in INDEX_EXTENSIONS:
 
            log.debug('    >> %s [WITH CONTENT]' % path)
 
            u_content = node.content
 
        else:
 
            log.debug('    >> %s' % path)
 
            #just index file name without it's content
 
            u_content = u''
 

	
 
        writer.add_document(owner=unicode(repo.contact),
setup.cfg
Show inline comments
 
[egg_info]
 
tag_build = rc4
 
tag_build = beta
 
tag_svn_revision = true
 

	
 
[easy_install]
 
find_links = http://www.pylonshq.com/download/
 

	
 
[nosetests]
 
verbose=True
 
verbosity=2
 
with-pylons=test.ini
 
detailed-errors=1
 

	
 
# Babel configuration
setup.py
Show inline comments
 
@@ -80,14 +80,19 @@ setup(
 
    message_extractors={'rhodecode': [
 
            ('**.py', 'python', None),
 
            ('templates/**.mako', 'mako', {'input_encoding': 'utf-8'}),
 
            ('public/**', 'ignore', None)]},
 
    zip_safe=False,
 
    paster_plugins=['PasteScript', 'Pylons'],
 
    entry_points="""
 
    [paste.app_factory]
 
    main = rhodecode.config.middleware:make_app
 

	
 
    [paste.app_install]
 
    main = pylons.util:PylonsInstaller
 

	
 
    [paste.global_paster_command]
 
    make-index = rhodecode.lib.indexers:MakeIndex
 
    upgrade-db = rhodecode.lib.utils:UpgradeDb
 
        
 
    """,
 
)
0 comments (0 inline, 0 general)