.. _changelog:

=========

Changelog

=========

----------------------

news

++++

- chinese traditional translation

fixes

+++++

- fixed no scm found warning

- fixed __future__ import error on rcextensions

- made simplejson required lib for speedup on JSON encoding

- fixes #449 bad regex could get more than revisions from parsing history

1.3.5 (**2012-05-10**)

----------------------

news

++++

- use ext_json for json module

- unified annotation view with file source view

- notification improvements, better inbox + css

- #419 don't strip passwords for login forms, make rhodecode 

  more compatible with LDAP servers

- Added HTTP_X_FORWARDED_FOR as another method of extracting 

  IP for pull/push logs. - moved all to base controller  

- #415: Adding comment to changeset causes reload. 

  Comments are now added via ajax and doesn't reload the page

- #374 LDAP config is discarded when LDAP can't be activated

- limited push/pull operations are now logged for git in the journal

- bumped mercurial to 2.2.X series

- added support for displaying submodules in file-browser

- #421 added bookmarks in changelog view

fixes

+++++

- fixed dev-version marker for stable when served from source codes

- fixed missing permission checks on show forks page

- #418 cast to unicode fixes in notification objects

- #426 fixed mention extracting regex

- fixed remote-pulling for git remotes remopositories

- fixed #434: Error when accessing files or changesets of a git repository 

  with submodules

- fixed issue with empty APIKEYS for users after registration ref. #438

- fixed issue with getting README files from git repositories

1.3.4 (**2012-03-28**)

----------------------

news

++++

- Whoosh logging is now controlled by the .ini files logging setup

- added clone-url into edit form on /settings page

- added help text into repo add/edit forms

- created rcextensions module with additional mappings (ref #322) and

  post push/pull/create repo hooks callbacks

- implemented #377 Users view for his own permissions on account page

- #399 added inheritance of permissions for users group on repos groups

- #401 repository group is automatically pre-selected when adding repos 

  inside a repository group

- added alternative HTTP 403 response when client failed to authenticate. Helps 

  solving issues with Mercurial and LDAP

- #402 removed group prefix from repository name when listing repositories 

  inside a group

- added gravatars into permission view and permissions autocomplete

- #347 when running multiple RhodeCode instances, properly invalidates cache 

  for all registered servers

fixes

+++++

- fixed #390 cache invalidation problems on repos inside group

- fixed #385 clone by ID url was loosing proxy prefix in URL

- fixed some unicode problems with waitress

- fixed issue with escaping < and > in changeset commits

- fixed error occurring during recursive group creation in API 

  create_repo function

- fixed #393 py2.5 fixes for routes url generator

- fixed #397 Private repository groups shows up before login

- fixed #396 fixed problems with revoking users in nested groups

- fixed mysql unicode issues + specified InnoDB as default engine with 

  utf8 charset

- #406 trim long branch/tag names in changelog to not break UI

1.3.3 (**2012-03-02**)

----------------------

news

++++

fixes

+++++

- fixed some python2.5 compatibility issues 

- fixed issues with removed repos was accidentally added as groups, after

  full rescan of paths

- fixes #376 Cannot edit user (using container auth)

.. _setup:

=====

Setup

=====

Setting up RhodeCode

--------------------

First, you will need to create a RhodeCode configuration file. Run the 

following command to do this::

    paster make-config RhodeCode production.ini

- This will create the file `production.ini` in the current directory. This

  configuration file contains the various settings for RhodeCode, e.g proxy 

  port, email settings, usage of static files, cache, celery settings and 

  logging.

Next, you need to create the databases used by RhodeCode. I recommend that you

use postgresql or sqlite (default). If you choose a database other than the

default ensure you properly adjust the db url in your production.ini

configuration file to use this other database. RhodeCode currently supports

postgresql, sqlite and mysql databases. Create the database by running

the following command::

This will prompt you for a "root" path. This "root" path is the location where

RhodeCode will store all of its repositories on the current machine. After

- Note: the given path for mercurial_ repositories **must** be write accessible

  for the application. It's very important since the RhodeCode web interface 

  will work without write access, but when trying to do a push it will 

  eventually fail with permission denied errors unless it has write access.

You are now ready to use RhodeCode, to run it simply execute::

    paster serve production.ini

- This command runs the RhodeCode server. The web app should be available at the 

  127.0.0.1:5000. This ip and port is configurable via the production.ini 

  file created in previous step

- The default permissions on each repository is read, and the owner is admin. 

  Remember to update these if needed.

- In the admin panel you can toggle ldap, anonymous, permissions settings. As

  well as edit more advanced options on users and repositories

Optionally users can create `rcextensions` package that extends RhodeCode

functionality. To do this simply execute::

    paster make-rcext production.ini

This will create `rcextensions` package in the same place that your `ini` file

lives. With `rcextensions` it's possible to add additional mapping for whoosh, 

stats and add additional code into the push/pull/create repo hooks. For example

for sending signals to build-bots such as jenkins.

Please see the `__init__.py` file inside `rcextensions` package 

for more details.

Using RhodeCode with SSH

------------------------

RhodeCode currently only hosts repositories using http and https. (The addition

of ssh hosting is a planned future feature.) However you can easily use ssh in

parallel with RhodeCode. (Repository access via ssh is a standard "out of

the box" feature of mercurial_ and you can use this to access any of the

repositories that RhodeCode is hosting. See PublishingRepositories_)

RhodeCode repository structures are kept in directories with the same name 

as the project. When using repository groups, each group is a subdirectory.

This allows you to easily use ssh for accessing repositories.

In order to use ssh you need to make sure that your web-server and the users 

login accounts have the correct permissions set on the appropriate directories.

(Note that these permissions are independent of any permissions you have set up

using the RhodeCode web interface.)

If your main directory (the same as set in RhodeCode settings) is for example

set to **/home/hg** and the repository you are using is named `rhodecode`, then

to clone via ssh you should run::

    hg clone ssh://user@server.com/home/hg/rhodecode

Using other external tools such as mercurial-server_ or using ssh key based

authentication is fully supported.

Note: In an advanced setup, in order for your ssh access to use the same

permissions as set up via the RhodeCode web interface, you can create an

authentication hook to connect to the rhodecode db and runs check functions for

permissions against that.

Setting up Whoosh full text search

----------------------------------

Starting from version 1.1 the whoosh index can be build by using the paster

command ``make-index``. To use ``make-index`` you must specify the configuration

file that stores the location of the index. You may specify the location of the 

repositories (`--repo-location`).  If not specified, this value is retrieved 

from the RhodeCode database.  This was required prior to 1.2.  Starting from 

version 1.2 it is also possible to specify a comma separated list of 

repositories (`--index-only`) to build index only on chooses repositories 

skipping any other found in repos location

You may optionally pass the option `-f` to enable a full index rebuild. Without

the `-f` option, indexing will run always in "incremental" mode.

For an incremental index build use::

	paster make-index production.ini 

For a full index rebuild use::

	paster make-index production.ini -f 

building index just for chosen repositories is possible with such command::

 paster make-index production.ini --index-only=vcs,rhodecode

In order to do periodical index builds and keep your index always up to date.

It's recommended to do a crontab entry for incremental indexing. 

An example entry might look like this::

    /path/to/python/bin/paster make-index /path/to/rhodecode/production.ini 

When using incremental mode (the default) whoosh will check the last

modification date of each file and add it to be reindexed if a newer file is

available. The indexing daemon checks for any removed files and removes them

from index.

If you want to rebuild index from scratch, you can use the `-f` flag as above,

or in the admin panel you can check `build from scratch` flag.

Setting up LDAP support

-----------------------

                self.klass = klass

            def step_0(self):

                # step 0 is the schema upgrade, and than follow proper upgrades

                print ('attempting to do database upgrade to version %s' \

                                % __dbversion__)

                api.upgrade(db_uri, repository_path, __dbversion__)

                print ('Schema upgrade completed')

            def step_1(self):

                pass

            def step_2(self):

                print ('Patching repo paths for newer version of RhodeCode')

                self.klass.fix_repo_paths()

                print ('Patching default user of RhodeCode')

                self.klass.fix_default_user()

                log.info('Changing ui settings')

                self.klass.create_ui_settings()

            def step_3(self):

                print ('Adding additional settings into RhodeCode db')

                self.klass.fix_settings()

                print ('Adding ldap defaults')

                self.klass.create_ldap_options(skip_existing=True)

            def step_4(self):

                print ('create permissions and fix groups')

                self.klass.create_permissions()

                self.klass.fixup_groups()

            def step_5(self):

                pass

        upgrade_steps = [0] + range(curr_version + 1, __dbversion__ + 1)

        # CALL THE PROPER ORDER OF STEPS TO PERFORM FULL UPGRADE

        for step in upgrade_steps:

            print ('performing upgrade step %s' % step)

            getattr(UpgradeSteps(self), 'step_%s' % step)()

            self.sa.commit()

    def fix_repo_paths(self):

        """

        Fixes a old rhodecode version path into new one without a '*'

        """

        paths = self.sa.query(RhodeCodeUi)\

                .filter(RhodeCodeUi.ui_key == '/')\

                .scalar()

        paths.ui_value = paths.ui_value.replace('*', '')

        try:

            self.sa.add(paths)

            self.sa.commit()

        except:

            self.sa.rollback()

            raise

    def fix_default_user(self):

        """

        Fixes a old default user with some 'nicer' default values,

        used mostly for anonymous access

        """

        def_user = self.sa.query(User)\

                .filter(User.username == 'default')\

                .one()

        def_user.name = 'Anonymous'

        def_user.lastname = 'User'

        def_user.email = 'anonymous@rhodecode.org'

        try:

            self.sa.add(def_user)

            self.sa.commit()

        except:

            self.sa.rollback()

            raise

    def fix_settings(self):

        """

        Fixes rhodecode settings adds ga_code key for google analytics

        """

        hgsettings3 = RhodeCodeSetting('ga_code', '')

        try:

            self.sa.add(hgsettings3)

            self.sa.commit()

        except:

            self.sa.rollback()

            raise

        if not self.tests:

            import getpass

            def get_password():

                password = getpass.getpass('Specify admin password '

                                           '(min 6 chars):')

                confirm = getpass.getpass('Confirm password:')

                if password != confirm:

                    log.error('passwords mismatch')

                    return False

                if len(password) < 6:

                    log.error('password is to short use at least 6 characters')

                    return False

                return password

                password = get_password()

                if not password:

            self.create_user(username, password, email, True)

        else:

            log.info('creating admin and regular test users')

            from rhodecode.tests import TEST_USER_ADMIN_LOGIN,\

            TEST_USER_ADMIN_PASS, TEST_USER_ADMIN_EMAIL,\

            TEST_USER_REGULAR_LOGIN, TEST_USER_REGULAR_PASS,\

            TEST_USER_REGULAR_EMAIL, TEST_USER_REGULAR2_LOGIN, \

            TEST_USER_REGULAR2_PASS, TEST_USER_REGULAR2_EMAIL

            self.create_user(TEST_USER_ADMIN_LOGIN, TEST_USER_ADMIN_PASS,

                             TEST_USER_ADMIN_EMAIL, True)

            self.create_user(TEST_USER_REGULAR_LOGIN, TEST_USER_REGULAR_PASS,

                             TEST_USER_REGULAR_EMAIL, False)

            self.create_user(TEST_USER_REGULAR2_LOGIN, TEST_USER_REGULAR2_PASS,

                             TEST_USER_REGULAR2_EMAIL, False)

    def create_ui_settings(self):

        """

        Creates ui settings, fills out hooks

        and disables dotencode

        """

        #HOOKS

        hooks1_key = RhodeCodeUi.HOOK_UPDATE

        hooks1_ = self.sa.query(RhodeCodeUi)\

            .filter(RhodeCodeUi.ui_key == hooks1_key).scalar()

        hooks1 = RhodeCodeUi() if hooks1_ is None else hooks1_

        hooks1.ui_section = 'hooks'

        hooks1.ui_key = hooks1_key

        hooks1.ui_value = 'hg update >&2'

        hooks1.ui_active = False

        hooks2_key = RhodeCodeUi.HOOK_REPO_SIZE

        hooks2_ = self.sa.query(RhodeCodeUi)\

            .filter(RhodeCodeUi.ui_key == hooks2_key).scalar()

        hooks2 = RhodeCodeUi() if hooks2_ is None else hooks2_

        hooks2.ui_section = 'hooks'

        hooks2.ui_key = hooks2_key

        hooks2.ui_value = 'python:rhodecode.lib.hooks.repo_size'

        hooks3 = RhodeCodeUi()

        hooks3.ui_section = 'hooks'

        hooks3.ui_key = RhodeCodeUi.HOOK_PUSH

        hooks3.ui_value = 'python:rhodecode.lib.hooks.log_push_action'

        hooks4 = RhodeCodeUi()

        hooks4.ui_section = 'hooks'

        hooks4.ui_key = RhodeCodeUi.HOOK_PULL

        hooks4.ui_value = 'python:rhodecode.lib.hooks.log_pull_action'

        # For mercurial 1.7 set backward comapatibility with format

        dotencode_disable = RhodeCodeUi()

        dotencode_disable.ui_section = 'format'

        dotencode_disable.ui_key = 'dotencode'

        dotencode_disable.ui_value = 'false'

        # enable largefiles

        largefiles = RhodeCodeUi()

        largefiles.ui_section = 'extensions'

        largefiles.ui_key = 'largefiles'

        largefiles.ui_value = ''

        self.sa.add(hooks1)

        self.sa.add(hooks2)

        self.sa.add(hooks3)

        self.sa.add(hooks4)

        self.sa.add(largefiles)

    def create_ldap_options(self, skip_existing=False):

        """Creates ldap settings"""

        for k, v in [('ldap_active', 'false'), ('ldap_host', ''),

                    ('ldap_port', '389'), ('ldap_tls_kind', 'PLAIN'),

                    ('ldap_tls_reqcert', ''), ('ldap_dn_user', ''),

                    ('ldap_dn_pass', ''), ('ldap_base_dn', ''),

                    ('ldap_filter', ''), ('ldap_search_scope', ''),

                    ('ldap_attr_login', ''), ('ldap_attr_firstname', ''),

                    ('ldap_attr_lastname', ''), ('ldap_attr_email', '')]:

            if skip_existing and RhodeCodeSetting.get_by_name(k) != None:

                log.debug('Skipping option %s' % k)

                continue

            setting = RhodeCodeSetting(k, v)

            self.sa.add(setting)

    def fixup_groups(self):

        def_usr = User.get_by_username('default')

        for g in RepoGroup.query().all():

            g.group_name = g.get_new_name(g.name)

            self.sa.add(g)

            # get default perm

            default = UserRepoGroupToPerm.query()\

                .filter(UserRepoGroupToPerm.group == g)\

                .filter(UserRepoGroupToPerm.user == def_usr)\

                .scalar()

            if default is None:

                log.debug('missing default permission for group %s adding' % g)

                ReposGroupModel()._create_default_perms(g)

        if retries == 3:

            log.info('Setting up repositories config')

            path = raw_input(

                 'Enter a valid absolute path to store repositories. '

                 'All repositories in that path will be added automatically:'

            )

        else:

            path = test_repo_path

        path_ok = True

        # check proper dir

        if not os.path.isdir(path):

            path_ok = False

            log.error('Given path %s is not a valid directory' % path)

        elif not os.path.isabs(path):

            path_ok = False

            log.error('Given path %s is not an absolute path' % path)

        # check write access

        elif not os.access(path, os.W_OK) and path_ok:

            path_ok = False

            log.error('No write permission to given path %s' % path)

        if retries == 0:

            sys.exit('max retries reached')

        if path_ok is False:

            retries -= 1

            return self.config_prompt(test_repo_path, retries)

        return path

    def create_settings(self, path):

        self.create_ui_settings()

        #HG UI OPTIONS

        web1 = RhodeCodeUi()

        web1.ui_section = 'web'

        web1.ui_key = 'push_ssl'

        web1.ui_value = 'false'

        web2 = RhodeCodeUi()

        web2.ui_section = 'web'

        web2.ui_key = 'allow_archive'

        web2.ui_value = 'gz zip bz2'

        web3 = RhodeCodeUi()

        web3.ui_section = 'web'

        web3.ui_key = 'allow_push'

        web3.ui_value = '*'

        web4 = RhodeCodeUi()

        web4.ui_section = 'web'

        web4.ui_key = 'baseurl'

        web4.ui_value = '/'

        paths = RhodeCodeUi()

        paths.ui_section = 'paths'

        paths.ui_key = '/'

        paths.ui_value = path

        hgsettings1 = RhodeCodeSetting('realm', 'RhodeCode authentication')

        hgsettings2 = RhodeCodeSetting('title', 'RhodeCode')

        hgsettings3 = RhodeCodeSetting('ga_code', '')

        self.sa.add(web1)

        self.sa.add(web2)

        self.sa.add(web3)

        self.sa.add(web4)

        self.sa.add(paths)

        self.sa.add(hgsettings1)

        self.sa.add(hgsettings2)

        self.sa.add(hgsettings3)

        self.create_ldap_options()

        log.info('created ui config')

    def create_user(self, username, password, email='', admin=False):

        log.info('creating user %s' % username)

        UserModel().create_or_update(username, password, email,

                                     name='RhodeCode', lastname='Admin',

                                     active=True, admin=admin)

    def create_default_user(self):

        log.info('creating default user')

        # create default user for handling default permissions.

        UserModel().create_or_update(username='default',

                              password=str(uuid.uuid1())[:8],

                              email='anonymous@rhodecode.org',

                              name='Anonymous', lastname='User')

    def create_permissions(self):

        # module.(access|create|change|delete)_[name]

        # module.(none|read|write|admin)

        perms = [

         ('repository.none', 'Repository no access'),

import re

from itertools import chain

from dulwich import objects

from subprocess import Popen, PIPE

from rhodecode.lib.vcs.conf import settings

from rhodecode.lib.vcs.exceptions import RepositoryError

from rhodecode.lib.vcs.exceptions import ChangesetError

from rhodecode.lib.vcs.exceptions import NodeDoesNotExistError

from rhodecode.lib.vcs.exceptions import VCSError

from rhodecode.lib.vcs.exceptions import ChangesetDoesNotExistError

from rhodecode.lib.vcs.exceptions import ImproperArchiveTypeError

from rhodecode.lib.vcs.backends.base import BaseChangeset

from rhodecode.lib.vcs.nodes import FileNode, DirNode, NodeKind, RootNode, \

    RemovedFileNode, SubModuleNode

from rhodecode.lib.vcs.utils import safe_unicode

from rhodecode.lib.vcs.utils import date_fromtimestamp

from rhodecode.lib.vcs.utils.lazy import LazyProperty

class GitChangeset(BaseChangeset):

    """

    Represents state of the repository at single revision.

    """

    def __init__(self, repository, revision):

        self._stat_modes = {}

        self.repository = repository

        self.raw_id = revision

        self.revision = repository.revisions.index(revision)

        self.short_id = self.raw_id[:12]

        self.id = self.raw_id

        try:

            commit = self.repository._repo.get_object(self.raw_id)

        except KeyError:

            raise RepositoryError("Cannot get object with id %s" % self.raw_id)

        self._commit = commit

        self._tree_id = commit.tree

        try:

        except UnicodeDecodeError:

        #self.branch = None

        self.tags = []

        self.nodes = {}

        self._paths = {}

    @LazyProperty

    def author(self):

        return safe_unicode(self._commit.committer)

    @LazyProperty

    def date(self):

        return date_fromtimestamp(self._commit.commit_time,

                                  self._commit.commit_timezone)

    @LazyProperty

    def status(self):

        """

        Returns modified, added, removed, deleted files for current changeset

        """

        return self.changed, self.added, self.removed

    @LazyProperty

    def branch(self):

        heads = self.repository._heads(reverse=False)

        ref = heads.get(self.raw_id)

        if ref:

            return safe_unicode(ref)

    def _fix_path(self, path):

        """

        Paths are stored without trailing slash so we need to get rid off it if

        needed.

        """

        if path.endswith('/'):

            path = path.rstrip('/')

        return path

    def _get_id_for_path(self, path):

        # FIXME: Please, spare a couple of minutes and make those codes cleaner;

        if not path in self._paths:

            path = path.strip('/')

            # set root tree

            tree = self.repository._repo[self._commit.tree]

            if path == '':

                self._paths[''] = tree.id

                return tree.id

            splitted = path.split('/')

            dirs, name = splitted[:-1], splitted[-1]

            curdir = ''

            # initially extract things from root dir

            for item, stat, id in tree.iteritems():

                if curdir:

                    name = '/'.join((curdir, item))

                else:

                    name = item

                self._paths[name] = id

                self._stat_modes[name] = stat

            for dir in dirs:

                if curdir:

                    curdir = '/'.join((curdir, dir))

                else:

                    curdir = dir

                dir_id = None

                for item, stat, id in tree.iteritems():

                    if dir == item:

                        dir_id = id

                if dir_id:

                    # Update tree

                    tree = self.repository._repo[dir_id]

                    if not isinstance(tree, objects.Tree):

                        raise ChangesetError('%s is not a directory' % curdir)

                else:

                    raise ChangesetError('%s have not been found' % curdir)

                # cache all items from the given traversed tree

                for item, stat, id in tree.iteritems():

                    if curdir:

                        name = '/'.join((curdir, item))

                    else:

                        name = item

                    self._paths[name] = id

                    self._stat_modes[name] = stat

            if not path in self._paths:

                raise NodeDoesNotExistError("There is no file nor directory "

                    "at the given path %r at revision %r"

                    % (path, self.short_id))

        return self._paths[path]