.. _changelog:

=========

Changelog

=========

1.3.4 (**2012-XX-XX**)

----------------------

:status: in-progress

:branch: beta

news

++++

- Whoosh logging is now controlled by the .ini files logging setup

- added clone-url into edit form on /settings page

- added help text into repo add/edit forms

fixes

+++++

- fixed #390 cache invalidation problems on repos inside group

- fixed #385 clone by ID url was loosing proxy prefix in URL

- fixed some unicode problems with waitress

1.3.3 (**2012-03-02**)

----------------------

news

++++

fixes

+++++

- fixed some python2.5 compatibility issues 

- fixed issues with removed repos was accidentally added as groups, after

  full rescan of paths

- fixes #376 Cannot edit user (using container auth)

- fixes #378 Invalid image urls on changeset screen with proxy-prefix 

  configuration

.. _setup:

=====

Setup

=====

Setting up RhodeCode

--------------------

First, you will need to create a RhodeCode configuration file. Run the 

following command to do this::

    paster make-config RhodeCode production.ini

- This will create the file `production.ini` in the current directory. This

  configuration file contains the various settings for RhodeCode, e.g proxy 

  port, email settings, usage of static files, cache, celery settings and 

  logging.

Next, you need to create the databases used by RhodeCode. I recommend that you

default ensure you properly adjust the db url in your production.ini

the following command::

    paster setup-app production.ini

This will prompt you for a "root" path. This "root" path is the location where

RhodeCode will store all of its repositories on the current machine. After

entering this "root" path ``setup-app`` will also prompt you for a username 

and password for the initial admin account which ``setup-app`` sets up for you.

- The ``setup-app`` command will create all of the needed tables and an admin

  account. When choosing a root path you can either use a new empty location, 

  or a location which already contains existing repositories. If you choose a 

  location which contains existing repositories RhodeCode will simply add all 

  of the repositories at the chosen location to it's database. (Note: make 

  sure you specify the correct path to the root).

- Note: the given path for mercurial_ repositories **must** be write accessible

  for the application. It's very important since the RhodeCode web interface 

  will work without write access, but when trying to do a push it will 

  eventually fail with permission denied errors unless it has write access.

You are now ready to use RhodeCode, to run it simply execute::

    paster serve production.ini

- This command runs the RhodeCode server. The web app should be available at the 

  127.0.0.1:5000. This ip and port is configurable via the production.ini 

  file created in previous step

- Use the admin account you created above when running ``setup-app`` to login 

  to the web app.

- The default permissions on each repository is read, and the owner is admin. 

  Remember to update these if needed.

- In the admin panel you can toggle ldap, anonymous, permissions settings. As

  well as edit more advanced options on users and repositories

Using RhodeCode with SSH

------------------------

RhodeCode currently only hosts repositories using http and https. (The addition

of ssh hosting is a planned future feature.) However you can easily use ssh in

parallel with RhodeCode. (Repository access via ssh is a standard "out of

the box" feature of mercurial_ and you can use this to access any of the

repositories that RhodeCode is hosting. See PublishingRepositories_)

RhodeCode repository structures are kept in directories with the same name 

as the project. When using repository groups, each group is a subdirectory.

This allows you to easily use ssh for accessing repositories.

In order to use ssh you need to make sure that your web-server and the users 

login accounts have the correct permissions set on the appropriate directories.

(Note that these permissions are independent of any permissions you have set up

using the RhodeCode web interface.)

If your main directory (the same as set in RhodeCode settings) is for example

set to **/home/hg** and the repository you are using is named `rhodecode`, then

to clone via ssh you should run::

    hg clone ssh://user@server.com/home/hg/rhodecode

  # after insertion to test_group group the url will be

  http://server.com/test_group/repo_name

This can be an issue for build systems and any other hardcoded scripts, moving

repository to a group leads to a need for changing external systems. To 

overcome this RhodeCode introduces a non changable replacement url. It's 

simply an repository ID prefixed with `_` above urls are also accessible as::

  http://server.com/_<ID>

Since ID are always the same moving the repository will not affect such url.

the _<ID> syntax can be used anywhere in the system so urls with repo_name 

for changelogs, files and other can be exchanged with _<ID> syntax.

Mailing

-------

When administrator will fill up the mailing settings in .ini files

RhodeCode will send mails on user registration, or when RhodeCode errors occur

on errors the mails will have a detailed traceback of error.

Trending source files

---------------------

Trending source files are calculated based on pre defined dict of known

types and extensions. If You miss some extension or Would like to scan some

custom files it's possible to add new types in `LANGUAGES_EXTENSIONS_MAP` dict

located in `/rhodecode/lib/celerylib/tasks.py`

    from rhodecode.lib import get_current_revision

    _rev = get_current_revision()

except ImportError:

    # this is needed when doing some setup.py operations

    _rev = False

if len(VERSION) > 3 and _rev:

    __version__ += ' [rev:%s]' % _rev[0]

def get_version():

    """Returns shorter version (digit parts only) as string."""

    return '.'.join((str(each) for each in VERSION[:3]))

BACKENDS = {

    'hg': 'Mercurial repository',

    'git': 'Git repository',

}

CELERY_ON = False

# link to config for pylons

CONFIG = {}

"""Pylons environment configuration"""

import os

import logging

from mako.lookup import TemplateLookup

from pylons.configuration import PylonsConfig

from pylons.error import handle_mako_error

import rhodecode

import rhodecode.lib.app_globals as app_globals

import rhodecode.lib.helpers

from rhodecode.config.routing import make_map

# don't remove this import it does magic for celery

from rhodecode.lib import celerypylons, str2bool

from rhodecode.lib import engine_from_config

from rhodecode.lib.auth import set_available_permissions

from rhodecode.lib.utils import repo2db_mapper, make_ui, set_rhodecode_config

from rhodecode.model import init_model

from rhodecode.model.scm import ScmModel

log = logging.getLogger(__name__)

def load_environment(global_conf, app_conf, initial=False):

    """Configure the Pylons environment via the ``pylons.config``

    object

    """

    config = PylonsConfig()

    # Pylons paths

    root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))

    paths = dict(root=root,

                 controllers=os.path.join(root, 'controllers'),

                 static_files=os.path.join(root, 'public'),

                 templates=[os.path.join(root, 'templates')])

    # Initialize config with the basic options

    config.init_app(global_conf, app_conf, package='rhodecode', paths=paths)

    # store some globals into rhodecode

    rhodecode.CELERY_ON = str2bool(config['app_conf'].get('use_celery'))

    config['routes.map'] = make_map(config)

    config['pylons.app_globals'] = app_globals.Globals(config)

    config['pylons.h'] = rhodecode.lib.helpers

    rhodecode.CONFIG = config

    # Setup cache object as early as possible

    import pylons

    pylons.cache._push_object(config['pylons.app_globals'].cache)

    # Create the Mako TemplateLookup, with the default auto-escaping

    config['pylons.app_globals'].mako_lookup = TemplateLookup(

        directories=paths['templates'],

        error_handler=handle_mako_error,

        module_directory=os.path.join(app_conf['cache_dir'], 'templates'),

        input_encoding='utf-8', default_filters=['escape'],

        imports=['from webhelpers.html import escape'])

    # sets the c attribute access when don't existing attribute are accessed

    config['pylons.strict_tmpl_context'] = True

    test = os.path.split(config['__file__'])[-1] == 'test.ini'

    if test:

        from rhodecode.lib.utils import create_test_env, create_test_index

        from rhodecode.tests import  TESTS_TMP_PATH

        create_test_env(TESTS_TMP_PATH, config)

        create_test_index(TESTS_TMP_PATH, config, True)

    # MULTIPLE DB configs

    # Setup the SQLAlchemy database engine

    sa_engine_db1 = engine_from_config(config, 'sqlalchemy.db1.')

    ~~~~~~~~~~~~~~~~~~~~~~~

    Some simple helper functions

    :created_on: Jan 5, 2011

    :author: marcink

    :copyright: (C) 2011-2012 Marcin Kuzminski <marcin@python-works.com>

    :license: GPLv3, see COPYING for more details.

"""

# This program is free software: you can redistribute it and/or modify

# it under the terms of the GNU General Public License as published by

# the Free Software Foundation, either version 3 of the License, or

# (at your option) any later version.

#

# This program is distributed in the hope that it will be useful,

# but WITHOUT ANY WARRANTY; without even the implied warranty of

# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the

# GNU General Public License for more details.

#

# You should have received a copy of the GNU General Public License

# along with this program.  If not, see <http://www.gnu.org/licenses/>.

import os

import re

from rhodecode.lib.vcs.utils.lazy import LazyProperty

def __get_lem():

    from pygments import lexers

    from string import lower

    from collections import defaultdict

    d = defaultdict(lambda: [])

    def __clean(s):

        s = s.lstrip('*')

        s = s.lstrip('.')

        if s.find('[') != -1:

            exts = []

            start, stop = s.find('['), s.find(']')

            for suffix in s[start + 1:stop]:

                exts.append(s[:s.find('[')] + suffix)

            return map(lower, exts)

        else:

            return map(lower, [s])

    for lx, t in sorted(lexers.LEXERS.items()):

        m = map(__clean, t[-2])

        if m:

            m = reduce(lambda x, y: x + y, m)

            for ext in m:

                desc = lx.replace('Lexer', '')

                d[ext].append(desc)

    return dict(d)

# language map is also used by whoosh indexer, which for those specified

# extensions will index it's content

LANGUAGES_EXTENSIONS_MAP = __get_lem()

# Additional mappings that are not present in the pygments lexers

# list of readme files to search in file tree and display in summary

# attached weights defines the search  order lower is first

ALL_READMES = [

    ('readme', 0), ('README', 0), ('Readme', 0),

    ('doc/readme', 1), ('doc/README', 1), ('doc/Readme', 1),

    ('Docs/readme', 2), ('Docs/README', 2), ('Docs/Readme', 2),

    ('DOCS/readme', 2), ('DOCS/README', 2), ('DOCS/Readme', 2),

    ('docs/readme', 2), ('docs/README', 2), ('docs/Readme', 2),

]

# extension together with weights to search lower is first

RST_EXTS = [

    ('', 0), ('.rst', 1), ('.rest', 1),

    ('.RST', 2), ('.REST', 2),

    ('.txt', 3), ('.TXT', 3)

]

MARKDOWN_EXTS = [

    ('.md', 1), ('.MD', 1),

    ('.mkdn', 2), ('.MKDN', 2),

    ('.mdown', 3), ('.MDOWN', 3),

    ('.markdown', 4), ('.MARKDOWN', 4)

]

    def check_permissions(self):

        group_name = get_repos_group_slug(request)

        try:

            user_perms = set([self.user_perms['repositories_groups'][group_name]])

        except KeyError:

            return False

        if self.required_perms.intersection(user_perms):

            return True

        return False

#==============================================================================

# CHECK FUNCTIONS

#==============================================================================

class PermsFunction(object):

    """Base function for other check functions"""

    def __init__(self, *perms):

        available_perms = config['available_permissions']

        for perm in perms:

            if perm not in available_perms:

        self.required_perms = set(perms)

        self.user_perms = None

        self.granted_for = ''

        self.repo_name = None

    def __call__(self, check_Location=''):

        user = request.user

        log.debug('checking %s %s %s', self.__class__.__name__,

                  self.required_perms, user)

        if not user:

            log.debug('Empty request user')

            return False

        self.user_perms = user.permissions

        self.granted_for = user

        if self.check_permissions():

            log.debug('Permission granted %s @ %s', self.granted_for,

                      check_Location or 'unspecified location')

            return True

        else:

            log.debug('Permission denied for %s @ %s', self.granted_for,

                        check_Location or 'unspecified location')

            return False

    Hooks runned by rhodecode

    :created_on: Aug 6, 2010

    :author: marcink

    :copyright: (C) 2010-2012 Marcin Kuzminski <marcin@python-works.com>

    :license: GPLv3, see COPYING for more details.

"""

# This program is free software: you can redistribute it and/or modify

# it under the terms of the GNU General Public License as published by

# the Free Software Foundation, either version 3 of the License, or

# (at your option) any later version.

#

# This program is distributed in the hope that it will be useful,

# but WITHOUT ANY WARRANTY; without even the implied warranty of

# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the

# GNU General Public License for more details.

#

# You should have received a copy of the GNU General Public License

# along with this program.  If not, see <http://www.gnu.org/licenses/>.

import os

import sys

from mercurial.scmutil import revrange

from mercurial.node import nullrev

from rhodecode.lib import helpers as h

from rhodecode.lib.utils import action_logger

def repo_size(ui, repo, hooktype=None, **kwargs):

    """

    Presents size of repository after push

    :param ui:

    :param repo:

    :param hooktype:

    """

    size_hg, size_root = 0, 0

    for path, dirs, files in os.walk(repo.root):

        if path.find('.hg') != -1:

            for f in files:

                try:

                    size_hg += os.path.getsize(os.path.join(path, f))

                except OSError:

                    pass

        else:

            for f in files:

                try:

                    size_root += os.path.getsize(os.path.join(path, f))

                except OSError:

                    pass

    size_hg_f = h.format_byte_size(size_hg)

    size_root_f = h.format_byte_size(size_root)

    size_total_f = h.format_byte_size(size_root + size_hg)

    last_cs = repo[len(repo) - 1]

    msg = ('Repository size .hg:%s repo:%s total:%s\n'

           'Last revision is now r%s:%s\n') % (

        size_hg_f, size_root_f, size_total_f, last_cs.rev(), last_cs.hex()[:12]

    )

    sys.stdout.write(msg)

def log_pull_action(ui, repo, **kwargs):

    """

    Logs user last pull action

    :param ui:

    :param repo:

    """

    action = 'pull'

    return 0

def log_push_action(ui, repo, **kwargs):

    """

    Maps user last push action to new changeset id, from mercurial

    :param ui:

    :param repo:

    """

    node = kwargs['node']

    def get_revs(repo, rev_opt):

        if rev_opt:

            revs = revrange(repo, rev_opt)

            if len(revs) == 0:

                return (nullrev, nullrev)

            return (max(revs), min(revs))

        else:

            return (len(repo) - 1, 0)

    stop, start = get_revs(repo, [node + ':'])

    revs = (str(repo[r]) for r in xrange(start, stop + 1))

    action = action % ','.join(revs)

    return 0

def log_create_repository(repository_dict, created_by, **kwargs):

    """

    Post create repository Hook. This is a dummy function for admins to re-use

    :param repository: dict dump of repository object

    :param created_by: username who created repository

    :param created_date: date of creation

    available keys of repository_dict:

     'repo_type',

     'description',

     'private',

     'created_on',

     'enable_downloads',

     'repo_id',

     'user_id',

     'enable_statistics',

     'clone_uri',

     'fork_id',

     'group_id',

     'repo_name'

    """

    return 0

import sys

import traceback

import logging

from os.path import dirname as dn, join as jn

#to get the rhodecode import

sys.path.append(dn(dn(dn(os.path.realpath(__file__)))))

from string import strip

from shutil import rmtree

from whoosh.analysis import RegexTokenizer, LowercaseFilter, StopFilter

from whoosh.fields import TEXT, ID, STORED, Schema, FieldType

from whoosh.index import create_in, open_dir

from whoosh.formats import Characters

from whoosh.highlight import highlight, HtmlFormatter, ContextFragmenter

from webhelpers.html.builder import escape

from sqlalchemy import engine_from_config

from rhodecode.model import init_model

from rhodecode.model.scm import ScmModel

from rhodecode.model.repo import RepoModel

from rhodecode.config.environment import load_environment

from rhodecode.lib.utils import BasePasterCommand, Command, add_cache