.. _changelog:

=========

Changelog

=========

1.4.4 (**2012-XX-XX**)

----------------------

:status: in-progress

:branch: beta

news

++++

- obfuscate db password in logs for engine connection string

- #574 Show pull request status also in shortlog (if any)

- remember selected tab in my account page

- Bumped mercurial version to 2.3.2

fixes

+++++

- Add git version detection to warn users that Git used in system is to

  old. Ref #588 - also show git version in system details in settings page

- fixed files quick filter links

- #590 Add GET flag that controls the way the diff are generated, for pull

  requests we want to use non-bundle based diffs, That are far better for

  doing code reviews. The /compare url still uses bundle compare for full

  comparison including the incoming changesets

- Fixed #585, checks for status of revision where to strict, and made

  opening pull request with those revision impossible due to previously set

  status. Checks now are made also for the repository.

- fixes #591 git backend was causing encoding errors when handling binary

  files - added a test case for VCS lib tests

- fixed #597 commits in future get negative age.

- fixed #598 API docs methods had wrong members parameter as returned data

1.4.3 (**2012-09-28**)

----------------------

news

++++

- #558 Added config file to hooks extra data

- bumped mercurial version to 2.3.1

- #518 added possibility of specifying multiple patterns for issues

- update codemirror to latest version

fixes

+++++

- fixed #570 explicit users group permissions can overwrite owner permissions

- fixed #578 set proper PATH with current Python for Git

  hooks to execute within same Python as RhodeCode 

- fixed issue with Git bare repos that ends with .git in name

1.4.2 (**2012-09-12**)

----------------------

news

++++

- added option to menu to quick lock/unlock repository for users that have

  write access to

- Implemented permissions for writing to repo

  groups. Now only write access to group allows to create a repostiory

  within that group

- #565 Add support for {netloc} and {scheme} to alternative_gravatar_url

- updated translation for zh_CN 

fixes

+++++

- fixed visual permissions check on repos groups inside groups

- fixed issues with non-ascii search terms in search, and indexers

- fixed parsing of page number in GET parameters

- fixed issues with generating pull-request overview for repos with

  bookmarks and tags, also preview doesn't loose chosen revision from

  select dropdown

1.4.1 (**2012-09-07**)

----------------------

news

++++

- always put a comment about code-review status change even if user send

  empty data 

- modified_on column saves repository update and it's going to be used

  later for light version of main page ref #500

- pull request notifications send much nicer emails with details about pull

  request

- #551 show breadcrumbs in summary view for repositories inside a group

fixes

+++++

- fixed migrations of permissions that can lead to inconsistency.

  Some users sent feedback that after upgrading from older versions issues 

  with updating default permissions occurred. RhodeCode detects that now and

  resets default user permission to initial state if there is a need for that.

  Also forces users to set the default value for new forking permission. 

- #535 improved apache wsgi example configuration in docs

- fixes #550 mercurial repositories comparision failed when origin repo had

  additional not-common changesets

- fixed status of code-review in preview windows of pull request

- git forks were not initialized at bare repos

- fixes #555 fixes issues with comparing non-related repositories

- fixes #557 follower counter always counts up

- fixed issue #560 require push ssl checkbox wasn't shown when option was

  enabled

- fixed #559

- fixed issue #559 fixed bug in routing that mapped repo names with <name>_<num> in name as

  if it was a request to url by repository ID

# Additional mappings that are not present in the pygments lexers

# used for building stats

# format is {'ext':['Names']} eg. {'py':['Python']} note: there can be

# more than one name for extension

# NOTE: that this will overide any mappings in LANGUAGES_EXTENSIONS_MAP

# build by pygments

EXTRA_MAPPINGS = {}

#==============================================================================

# WHOOSH INDEX EXTENSIONS

#==============================================================================

# if INDEX_EXTENSIONS is [] it'll use pygments lexers extensions by default.

# To set your own just add to this list extensions to index with content

INDEX_EXTENSIONS = []

# additional extensions for indexing besides the default from pygments

# those get's added to INDEX_EXTENSIONS

EXTRA_INDEX_EXTENSIONS = []

#==============================================================================

# POST CREATE REPOSITORY HOOK

#==============================================================================

# this function will be executed after each repository is created

def _crhook(*args, **kwargs):

    """

    Post create repository HOOK

    kwargs available:

     :param repo_name:

     :param repo_type:

     :param description:

     :param private:

     :param created_on:

     :param enable_downloads:

     :param repo_id:

     :param user_id:

     :param enable_statistics:

     :param clone_uri:

     :param fork_id:

     :param group_id:

     :param created_by:

    """

    return 0

CREATE_REPO_HOOK = _crhook

#==============================================================================

# POST PUSH HOOK

#==============================================================================

# this function will be executed after each push it's runned after the build-in

# hook that rhodecode uses for logging pushes

def _pushhook(*args, **kwargs):

    """

    Post push hook

    kwargs available:

      :param username: name of user who pushed

      :param ip: ip of who pushed

      :param action: pull

      :param repository: repository name

      :param pushed_revs: generator of pushed revisions

    """

    return 0

PUSH_HOOK = _pushhook

#==============================================================================

# POST PULL HOOK

#==============================================================================

# this function will be executed after each push it's runned after the build-in

# hook that rhodecode uses for logging pushes

def _pullhook(*args, **kwargs):

    """

    Post pull hook

    kwargs available::

      :param username: name of user who pulled

      :param ip: ip of who pushed

      :param action: pull

      :param repository: repository name

    """

    return 0

PULL_HOOK = _pullhook

# -*- coding: utf-8 -*-

"""

    rhodecode.lib.hooks

    ~~~~~~~~~~~~~~~~~~~

    Hooks runned by rhodecode

    :created_on: Aug 6, 2010

    :author: marcink

    :copyright: (C) 2010-2012 Marcin Kuzminski <marcin@python-works.com>

    :license: GPLv3, see COPYING for more details.

"""

# This program is free software: you can redistribute it and/or modify

# it under the terms of the GNU General Public License as published by

# the Free Software Foundation, either version 3 of the License, or

# (at your option) any later version.

#

# This program is distributed in the hope that it will be useful,

# but WITHOUT ANY WARRANTY; without even the implied warranty of

# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the

# GNU General Public License for more details.

#

# You should have received a copy of the GNU General Public License

# along with this program.  If not, see <http://www.gnu.org/licenses/>.

import os

import sys

import binascii

from inspect import isfunction

from mercurial.scmutil import revrange

from mercurial.node import nullrev

from rhodecode.lib import helpers as h

from rhodecode.lib.utils import action_logger

from rhodecode.lib.vcs.backends.base import EmptyChangeset

from rhodecode.lib.compat import json

from rhodecode.lib.exceptions import HTTPLockedRC

from rhodecode.lib.utils2 import safe_str

from rhodecode.model.db import Repository, User

def _get_scm_size(alias, root_path):

    if not alias.startswith('.'):

        alias += '.'

    size_scm, size_root = 0, 0

    for path, dirs, files in os.walk(root_path):

        if path.find(alias) != -1:

            for f in files:

                try:

                    size_scm += os.path.getsize(os.path.join(path, f))

                except OSError:

                    pass

        else:

            for f in files:

                try:

                    size_root += os.path.getsize(os.path.join(path, f))

                except OSError:

                    pass

    size_scm_f = h.format_byte_size(size_scm)

    size_root_f = h.format_byte_size(size_root)

    size_total_f = h.format_byte_size(size_root + size_scm)

    return size_scm_f, size_root_f, size_total_f

def repo_size(ui, repo, hooktype=None, **kwargs):

    """

    Presents size of repository after push

    :param ui:

    :param repo:

    :param hooktype:

    """

    size_hg_f, size_root_f, size_total_f = _get_scm_size('.hg', repo.root)

    last_cs = repo[len(repo) - 1]

    msg = ('Repository size .hg:%s repo:%s total:%s\n'

           'Last revision is now r%s:%s\n') % (

        size_hg_f, size_root_f, size_total_f, last_cs.rev(), last_cs.hex()[:12]

    )

    sys.stdout.write(msg)

def pre_push(ui, repo, **kwargs):

    # pre push function, currently used to ban pushing when

    # repository is locked

    try:

        rc_extras = json.loads(os.environ.get('RC_SCM_DATA', "{}"))

    except:

        rc_extras = {}

    extras = dict(repo.ui.configitems('rhodecode_extras'))

    if 'username' in extras:

        username = extras['username']

        repository = extras['repository']

        scm = extras['scm']

        locked_by = extras['locked_by']

    elif 'username' in rc_extras:

        username = rc_extras['username']

        repository = rc_extras['repository']

        scm = rc_extras['scm']

        locked_by = rc_extras['locked_by']

    else:

        raise Exception('Missing data in repo.ui and os.environ')

    usr = User.get_by_username(username)

    if locked_by[0] and usr.user_id != int(locked_by[0]):

        locked_by = User.get(locked_by[0]).username

        raise HTTPLockedRC(repository, locked_by)

def pre_pull(ui, repo, **kwargs):

    # pre push function, currently used to ban pushing when

    # repository is locked

    try:

        rc_extras = json.loads(os.environ.get('RC_SCM_DATA', "{}"))

    except:

        rc_extras = {}

    extras = dict(repo.ui.configitems('rhodecode_extras'))

    if 'username' in extras:

        username = extras['username']

        repository = extras['repository']

        scm = extras['scm']

        locked_by = extras['locked_by']

    elif 'username' in rc_extras:

        username = rc_extras['username']

        repository = rc_extras['repository']

        scm = rc_extras['scm']

        locked_by = rc_extras['locked_by']

    else:

        raise Exception('Missing data in repo.ui and os.environ')

    action_logger(user, action, repository, extras['ip'], commit=True)

    # extension hook call

    from rhodecode import EXTENSIONS

    callback = getattr(EXTENSIONS, 'PULL_HOOK', None)

    if isfunction(callback):

        kw = {}

        kw.update(extras)

        callback(**kw)

    if make_lock is True:

        Repository.lock(Repository.get_by_repo_name(repository), user.user_id)

        #msg = 'Made lock on repo `%s`' % repository

        #sys.stdout.write(msg)

    return 0

def log_push_action(ui, repo, **kwargs):

    """

    Maps user last push action to new changeset id, from mercurial

    :param ui:

    :param repo: repo object containing the `ui` object

    """

    try:

        rc_extras = json.loads(os.environ.get('RC_SCM_DATA', "{}"))

    except:

        rc_extras = {}

    extras = dict(repo.ui.configitems('rhodecode_extras'))

    if 'username' in extras:

        username = extras['username']

        repository = extras['repository']

        scm = extras['scm']

        make_lock = extras['make_lock']

    elif 'username' in rc_extras:

        username = rc_extras['username']

        repository = rc_extras['repository']

        scm = rc_extras['scm']

        make_lock = rc_extras['make_lock']

    else:

        raise Exception('Missing data in repo.ui and os.environ')

    action = 'push' + ':%s'

    if scm == 'hg':

        node = kwargs['node']

        def get_revs(repo, rev_opt):

            if rev_opt:

                revs = revrange(repo, rev_opt)

                if len(revs) == 0:

                    return (nullrev, nullrev)

                return (max(revs), min(revs))

            else:

                return (len(repo) - 1, 0)

        stop, start = get_revs(repo, [node + ':'])

        h = binascii.hexlify

        revs = [h(repo[r].node()) for r in xrange(start, stop + 1)]

    elif scm == 'git':

        revs = kwargs.get('_git_revs', [])

        if '_git_revs' in kwargs:

            kwargs.pop('_git_revs')

    action = action % ','.join(revs)

    action_logger(username, action, repository, extras['ip'], commit=True)

    # extension hook call

    from rhodecode import EXTENSIONS

    callback = getattr(EXTENSIONS, 'PUSH_HOOK', None)

    if isfunction(callback):

        kw = {'pushed_revs': revs}

        kw.update(extras)

        callback(**kw)

    if make_lock is False:

        Repository.unlock(Repository.get_by_repo_name(repository))

        msg = 'Released lock on repo `%s`\n' % repository

        sys.stdout.write(msg)

    return 0

def log_create_repository(repository_dict, created_by, **kwargs):

    """

    Post create repository Hook. This is a dummy function for admins to re-use

    if needed. It's taken from rhodecode-extensions module and executed

    if present

    :param repository: dict dump of repository object

    :param created_by: username who created repository

    available keys of repository_dict:

     'repo_type',

     'description',

     'private',

     'created_on',

     'enable_downloads',

     'repo_id',

     'user_id',

     'enable_statistics',

     'clone_uri',

     'fork_id',

     'group_id',

     'repo_name'

    """

    from rhodecode import EXTENSIONS

    callback = getattr(EXTENSIONS, 'CREATE_REPO_HOOK', None)

    if isfunction(callback):

        kw = {}

        kw.update(repository_dict)

        kw.update({'created_by': created_by})

        kw.update(kwargs)

        return callback(**kw)

    return 0

handle_git_pre_receive = (lambda repo_path, revs, env:

    handle_git_receive(repo_path, revs, env, hook_type='pre'))

handle_git_post_receive = (lambda repo_path, revs, env:

    handle_git_receive(repo_path, revs, env, hook_type='post'))

def handle_git_receive(repo_path, revs, env, hook_type='post'):

    """

    A really hacky method that is runned by git post-receive hook and logs

    an push action together with pushed revisions. It's executed by subprocess

    thus needs all info to be able to create a on the fly pylons enviroment,

    connect to database and run the logging code. Hacky as sh*t but works.

    :param repo_path:

    :type repo_path:

    :param revs:

    :type revs:

    :param env:

    :type env:

    """

    from paste.deploy import appconfig

    from sqlalchemy import engine_from_config

    from rhodecode.config.environment import load_environment

    from rhodecode.model import init_model

    from rhodecode.model.db import RhodeCodeUi

    from rhodecode.lib.utils import make_ui

    extras = json.loads(env['RHODECODE_EXTRAS'])

    path, ini_name = os.path.split(extras['config'])

    conf = appconfig('config:%s' % ini_name, relative_to=path)

    load_environment(conf.global_conf, conf.local_conf)

    engine = engine_from_config(conf, 'sqlalchemy.db1.')

    init_model(engine)

    baseui = make_ui('db')

    # fix if it's not a bare repo

    if repo_path.endswith(os.sep + '.git'):

        repo_path = repo_path[:-5]

    repo = Repository.get_by_full_path(repo_path)

    if not repo:

        raise OSError('Repository %s not found in database'

                      % (safe_str(repo_path)))

    _hooks = dict(baseui.configitems('hooks')) or {}

    for k, v in extras.items():

        baseui.setconfig('rhodecode_extras', k, v)

    repo = repo.scm_instance

    repo.ui = baseui

    if hook_type == 'pre':

        pre_push(baseui, repo)

    # if push hook is enabled via web interface

    elif hook_type == 'post' and _hooks.get(RhodeCodeUi.HOOK_PUSH):

        rev_data = []

        for l in revs:

            old_rev, new_rev, ref = l.split(' ')

            _ref_data = ref.split('/')

            if _ref_data[1] in ['tags', 'heads']:

                rev_data.append({'old_rev': old_rev,

                                 'new_rev': new_rev,

                                 'ref': ref,

                                 'type': _ref_data[1],

                                 'name': _ref_data[2].strip()})

        git_revs = []

        for push_ref  in rev_data:

            _type = push_ref['type']

            if _type == 'heads':

                if push_ref['old_rev'] == EmptyChangeset().raw_id:

                    cmd = "for-each-ref --format='%(refname)' 'refs/heads/*'"

                    heads = repo.run_git_command(cmd)[0]

                    heads = heads.replace(push_ref['ref'], '')

                    heads = ' '.join(map(lambda c: c.strip('\n').strip(),

                                         heads.splitlines()))

                    cmd = (('log %(new_rev)s' % push_ref) +

                           ' --reverse --pretty=format:"%H" --not ' + heads)

                else:

                    cmd = (('log %(old_rev)s..%(new_rev)s' % push_ref) +

                           ' --reverse --pretty=format:"%H"')

                git_revs += repo.run_git_command(cmd)[0].splitlines()

            elif _type == 'tags':

                git_revs += [push_ref['name']]

        log_push_action(baseui, repo, _git_revs=git_revs)

# -*- coding: utf-8 -*-

"""

    rhodecode.model.repo

    ~~~~~~~~~~~~~~~~~~~~

    Repository model for rhodecode

    :created_on: Jun 5, 2010

    :author: marcink

    :copyright: (C) 2010-2012 Marcin Kuzminski <marcin@python-works.com>

    :license: GPLv3, see COPYING for more details.

"""

# This program is free software: you can redistribute it and/or modify

# it under the terms of the GNU General Public License as published by

# the Free Software Foundation, either version 3 of the License, or

# (at your option) any later version.

#

# This program is distributed in the hope that it will be useful,

# but WITHOUT ANY WARRANTY; without even the implied warranty of

# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the

# GNU General Public License for more details.

#

# You should have received a copy of the GNU General Public License

# along with this program.  If not, see <http://www.gnu.org/licenses/>.

from __future__ import with_statement

import os

import shutil

import logging

import traceback

from datetime import datetime

from rhodecode.lib.vcs.backends import get_backend

from rhodecode.lib.compat import json

from rhodecode.lib.utils2 import LazyProperty, safe_str, safe_unicode

from rhodecode.lib.caching_query import FromCache

from rhodecode.model import BaseModel

from rhodecode.model.db import Repository, UserRepoToPerm, User, Permission, \

    Statistics, UsersGroup, UsersGroupRepoToPerm, RhodeCodeUi, RepoGroup

from rhodecode.lib import helpers as h

log = logging.getLogger(__name__)

class RepoModel(BaseModel):

    cls = Repository

    URL_SEPARATOR = Repository.url_sep()

    def __get_users_group(self, users_group):

        return self._get_instance(UsersGroup, users_group,

                                  callback=UsersGroup.get_by_group_name)

    def _get_repos_group(self, repos_group):

        return self._get_instance(RepoGroup, repos_group,

                                  callback=RepoGroup.get_by_group_name)

    @LazyProperty

    def repos_path(self):

        """

        Get's the repositories root path from database

        """

        q = self.sa.query(RhodeCodeUi).filter(RhodeCodeUi.ui_key == '/').one()

        return q.ui_value

    def get(self, repo_id, cache=False):

        repo = self.sa.query(Repository)\

            .filter(Repository.repo_id == repo_id)

        if cache:

            repo = repo.options(FromCache("sql_cache_short",

                                          "get_repo_%s" % repo_id))

        return repo.scalar()

    def get_repo(self, repository):

        return self._get_repo(repository)

    def get_by_repo_name(self, repo_name, cache=False):

        repo = self.sa.query(Repository)\

            .filter(Repository.repo_name == repo_name)

        if cache:

            repo = repo.options(FromCache("sql_cache_short",

                                          "get_repo_%s" % repo_name))

        return repo.scalar()

    def get_users_js(self):

        users = self.sa.query(User).filter(User.active == True).all()

        return json.dumps([

            {

             'id': u.user_id,

             'fname': u.name,

             'lname': u.lastname,

             'nname': u.username,

             'gravatar_lnk': h.gravatar_url(u.email, 14)

            } for u in users]

        )

    def get_users_groups_js(self):

        users_groups = self.sa.query(UsersGroup)\

            .filter(UsersGroup.users_group_active == True).all()

        return json.dumps([

            {

             'id': gr.users_group_id,

             'grname': gr.users_group_name,

             'grmembers': len(gr.members),

            } for gr in users_groups]

        )

    def _get_defaults(self, repo_name):

        """

        Get's information about repository, and returns a dict for

        usage in forms

        :param repo_name:

        """

        repo_info = Repository.get_by_repo_name(repo_name)

        if repo_info is None:

            return None

        defaults = repo_info.get_dict()

        group, repo_name = repo_info.groups_and_repo

        defaults['repo_name'] = repo_name

        defaults['repo_group'] = getattr(group[-1] if group else None,

                                         'group_id', None)

            self.sa.add(new_repo)

            def _create_default_perms():

                # create default permission

                repo_to_perm = UserRepoToPerm()

                default = 'repository.read'

                for p in User.get_by_username('default').user_perms:

                    if p.permission.permission_name.startswith('repository.'):

                        default = p.permission.permission_name

                        break

                default_perm = 'repository.none' if private else default

                repo_to_perm.permission_id = self.sa.query(Permission)\

                        .filter(Permission.permission_name == default_perm)\

                        .one().permission_id

                repo_to_perm.repository = new_repo

                repo_to_perm.user_id = User.get_by_username('default').user_id

                self.sa.add(repo_to_perm)

            if fork_of:

                if copy_fork_permissions:

                    repo = fork_of

                    user_perms = UserRepoToPerm.query()\

                        .filter(UserRepoToPerm.repository == repo).all()

                    group_perms = UsersGroupRepoToPerm.query()\

                        .filter(UsersGroupRepoToPerm.repository == repo).all()

                    for perm in user_perms:

                        UserRepoToPerm.create(perm.user, new_repo,

                                              perm.permission)

                    for perm in group_perms:

                        UsersGroupRepoToPerm.create(perm.users_group, new_repo,

                                                    perm.permission)

                else:

                    _create_default_perms()

            else:

                _create_default_perms()

            if not just_db:

                self.__create_repo(repo_name, repo_type,

                                   repos_group,

                                   clone_uri)

                log_create_repository(new_repo.get_dict(),

                                      created_by=owner.username)

            # now automatically start following this repository as owner

            ScmModel(self.sa).toggle_following_repo(new_repo.repo_id,

                                                    owner.user_id)

            return new_repo

        except:

            log.error(traceback.format_exc())

            raise

    def create(self, form_data, cur_user, just_db=False, fork=None):

        """

        Backward compatibility function, just a wrapper on top of create_repo

        :param form_data:

        :param cur_user:

        :param just_db:

        :param fork:

        """

        repo_name = form_data['repo_name_full']

        repo_type = form_data['repo_type']

        description = form_data['description']

        owner = cur_user

        private = form_data['private']

        clone_uri = form_data.get('clone_uri')

        repos_group = form_data['repo_group']

        landing_rev = form_data['landing_rev']

        copy_fork_permissions = form_data.get('copy_permissions')

        fork_of = form_data.get('fork_parent_id')

        return self.create_repo(

            repo_name, repo_type, description, owner, private, clone_uri,

            repos_group, landing_rev, just_db, fork_of, copy_fork_permissions

        )

    def create_fork(self, form_data, cur_user):

        """

        Simple wrapper into executing celery task for fork creation

        :param form_data:

        :param cur_user:

        """

        from rhodecode.lib.celerylib import tasks, run_task

        run_task(tasks.create_repo_fork, form_data, cur_user)

    def delete(self, repo):

        repo = self._get_repo(repo)

        if repo:

            try:

                self.sa.delete(repo)

                self.__delete_repo(repo)

            except:

                log.error(traceback.format_exc())

                raise

    def grant_user_permission(self, repo, user, perm):

        """

        Grant permission for user on given repository, or update existing one

        if found