.. _changelog:

=========

Changelog

=========

1.3.4 (**2012-XX-XX**)

----------------------

:status: in-progress

:branch: beta

news

++++

- Whoosh logging is now controlled by the .ini files logging setup

- added clone-url into edit form on /settings page

- added help text into repo add/edit forms

- created rcextensions module with additional mappings (ref #322) and

  post push/pull/create repo hooks callbacks

fixes

+++++

- fixed #390 cache invalidation problems on repos inside group

- fixed #385 clone by ID url was loosing proxy prefix in URL

- fixed some unicode problems with waitress

- fixed issue with escaping < and > in changeset commits

1.3.3 (**2012-03-02**)

----------------------

news

++++

fixes

+++++

- fixed some python2.5 compatibility issues 

- fixed issues with removed repos was accidentally added as groups, after

  full rescan of paths

- fixes #376 Cannot edit user (using container auth)

- fixes #378 Invalid image urls on changeset screen with proxy-prefix 

  configuration

- fixed initial sorting of repos inside repo group

- fixes issue when user tried to resubmit same permission into user/user_groups

- bumped beaker version that fixes #375 leap error bug

- fixed raw_changeset for git. It was generated with hg patch headers

- fixed vcs issue with last_changeset for filenodes

- fixed missing commit after hook delete

- fixed #372 issues with git operation detection that caused a security issue 

  for git repos

1.3.2 (**2012-02-28**)

----------------------

news

++++

fixes

+++++

- fixed git protocol issues with repos-groups

- fixed git remote repos validator that prevented from cloning remote git repos

- fixes #370 ending slashes fixes for repo and groups

- fixes #368 improved git-protocol detection to handle other clients

- fixes #366 When Setting Repository Group To Blank Repo Group Wont Be 

  Moved To Root

- fixes #371 fixed issues with beaker/sqlalchemy and non-ascii cache keys 

- fixed #373 missing cascade drop on user_group_to_perm table

1.3.1 (**2012-02-27**)

----------------------

news

++++

fixes

+++++

- redirection loop occurs when remember-me wasn't checked during login

- fixes issues with git blob history generation 

- don't fetch branch for git in file history dropdown. Causes unneeded slowness

1.3.0 (**2012-02-26**)

----------------------

news

++++

- code review, inspired by github code-comments 

- #215 rst and markdown README files support

- #252 Container-based and proxy pass-through authentication support

- #44 branch browser. Filtering of changelog by branches

- mercurial bookmarks support

- new hover top menu, optimized to add maximum size for important views

- configurable clone url template with possibility to specify  protocol like 

  ssh:// or http:// and also manually alter other parts of clone_url.

- enabled largefiles extension by default

- optimized summary file pages and saved a lot of unused space in them

- #239 option to manually mark repository as fork

- #320 mapping of commit authors to RhodeCode users

- #304 hashes are displayed using monospace font    

- diff configuration, toggle white lines and context lines

- #307 configurable diffs, whitespace toggle, increasing context lines

- sorting on branches, tags and bookmarks using YUI datatable

- improved file filter on files page

- implements #330 api method for listing nodes ar particular revision

- #73 added linking issues in commit messages to chosen issue tracker url

  based on user defined regular expression

- added linking of changesets in commit messages  

- new compact changelog with expandable commit messages

- firstname and lastname are optional in user creation

- #348 added post-create repository hook

- #212 global encoding settings is now configurable from .ini files 

- #227 added repository groups permissions

- markdown gets codehilite extensions

- new API methods, delete_repositories, grante/revoke permissions for groups 

  and repos

# -*- coding: utf-8 -*-

"""

    rhodecode.controllers.api

    ~~~~~~~~~~~~~~~~~~~~~~~~~

    API controller for RhodeCode

    :created_on: Aug 20, 2011

    :author: marcink

    :copyright: (C) 2011-2012 Marcin Kuzminski <marcin@python-works.com>

    :license: GPLv3, see COPYING for more details.

"""

# This program is free software; you can redistribute it and/or

# modify it under the terms of the GNU General Public License

# as published by the Free Software Foundation; version 2

# of the License or (at your opinion) any later version of the license.

#

# This program is distributed in the hope that it will be useful,

# but WITHOUT ANY WARRANTY; without even the implied warranty of

# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the

# GNU General Public License for more details.

#

# You should have received a copy of the GNU General Public License

# along with this program; if not, write to the Free Software

# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,

# MA  02110-1301, USA.

import traceback

import logging

from rhodecode.controllers.api import JSONRPCController, JSONRPCError

from rhodecode.lib.auth import HasPermissionAllDecorator, \

    HasPermissionAnyDecorator, PasswordGenerator

from rhodecode.model.meta import Session

from rhodecode.model.scm import ScmModel

from rhodecode.model.db import User, UsersGroup, RepoGroup, Repository

from rhodecode.model.repo import RepoModel

from rhodecode.model.user import UserModel

from rhodecode.model.users_group import UsersGroupModel

from rhodecode.model.repos_group import ReposGroupModel

log = logging.getLogger(__name__)

class ApiController(JSONRPCController):

    """

    API Controller

    Each method needs to have USER as argument this is then based on given

    API_KEY propagated as instance of user object

    Preferably this should be first argument also

    Each function should also **raise** JSONRPCError for any

    errors that happens

    """

    @HasPermissionAllDecorator('hg.admin')

    def pull(self, apiuser, repo_name):

        """

        Dispatch pull action on given repo

        :param user:

        :param repo_name:

        """

        if Repository.is_valid(repo_name) is False:

            raise JSONRPCError('Unknown repo "%s"' % repo_name)

        try:

            ScmModel().pull_changes(repo_name, self.rhodecode_user.username)

            return 'Pulled from %s' % repo_name

        except Exception:

            raise JSONRPCError('Unable to pull changes from "%s"' % repo_name)

    @HasPermissionAllDecorator('hg.admin')

    def get_user(self, apiuser, userid):

        """"

        Get a user by username

        :param apiuser:

        :param username:

        """

        user = UserModel().get_user(userid)

        if user is None:

            return user

        return dict(

            id=user.user_id,

            username=user.username,

            firstname=user.name,

            lastname=user.lastname,

            email=user.email,

            active=user.active,

            admin=user.admin,

            ldap_dn=user.ldap_dn

        )

    @HasPermissionAllDecorator('hg.admin')

    def get_users(self, apiuser):

        """"

        Get all users

        :param apiuser:

        """

        result = []

        for user in User.getAll():

            result.append(

                dict(

                    id=user.user_id,

                    username=user.username,

                    firstname=user.name,

                    lastname=user.lastname,

                    email=user.email,

                    active=user.active,

                    admin=user.admin,

                    ldap_dn=user.ldap_dn

                )

            )

        return result

    @HasPermissionAllDecorator('hg.admin')

    def create_user(self, apiuser, username, email, password, firstname=None,

                    lastname=None, active=True, admin=False, ldap_dn=None):

        """

        Create new user

        :param apiuser:

        :param username:

                    ldap=user.ldap_dn,

                    permission=perm

                )

            )

        for users_group in repo.users_group_to_perm:

            perm = users_group.permission.permission_name

            users_group = users_group.users_group

            members.append(

                dict(

                    type_="users_group",

                    id=users_group.users_group_id,

                    name=users_group.users_group_name,

                    active=users_group.users_group_active,

                    permission=perm

                )

            )

        return dict(

            id=repo.repo_id,

            repo_name=repo.repo_name,

            type=repo.repo_type,

            description=repo.description,

            members=members

        )

    @HasPermissionAnyDecorator('hg.admin')

    def get_repos(self, apiuser):

        """"

        Get all repositories

        :param apiuser:

        """

        result = []

        for repository in Repository.getAll():

            result.append(

                dict(

                    id=repository.repo_id,

                    repo_name=repository.repo_name,

                    type=repository.repo_type,

                    description=repository.description

                )

            )

        return result

    @HasPermissionAnyDecorator('hg.admin')

    def get_repo_nodes(self, apiuser, repo_name, revision, root_path,

                       ret_type='all'):

        """

        returns a list of nodes and it's children

        for a given path at given revision. It's possible to specify ret_type

        to show only files or dirs

        :param apiuser:

        :param repo_name: name of repository

        :param revision: revision for which listing should be done

        :param root_path: path from which start displaying

        :param ret_type: return type 'all|files|dirs' nodes

        """

        try:

            _d, _f = ScmModel().get_nodes(repo_name, revision, root_path,

                                          flat=False)

            _map = {

                'all': _d + _f,

                'files': _f,

                'dirs': _d,

            }

            return _map[ret_type]

        except KeyError:

            raise JSONRPCError('ret_type must be one of %s' % _map.keys())

        except Exception, e:

            raise JSONRPCError(e)

    @HasPermissionAnyDecorator('hg.admin', 'hg.create.repository')

    def create_repo(self, apiuser, repo_name, owner_name, description='',

                    repo_type='hg', private=False, clone_uri=None):

        """

        Create repository, if clone_url is given it makes a remote clone

        :param apiuser:

        :param repo_name:

        :param owner_name:

        :param description:

        :param repo_type:

        :param private:

        :param clone_uri:

        """

        try:

            owner = User.get_by_username(owner_name)

            if owner is None:

                raise JSONRPCError('unknown user %s' % owner_name)

            if Repository.get_by_repo_name(repo_name):

                raise JSONRPCError("repo %s already exist" % repo_name)

            real_name = groups[-1]

            repo = RepoModel().create(

                dict(

                    repo_name=real_name,

                    repo_name_full=repo_name,

                    description=description,

                    private=private,

                    repo_type=repo_type,

                    clone_uri=clone_uri

                ),

                owner

            )

            Session.commit()

            return dict(

                id=repo.repo_id,

                msg="Created new repository %s" % repo.repo_name

            )

        except Exception:

            log.error(traceback.format_exc())

            raise JSONRPCError('failed to create repository %s' % repo_name)

    @HasPermissionAnyDecorator('hg.admin')

    def delete_repo(self, apiuser, repo_name):

        """

        Deletes a given repository

        :param repo_name:

        """

        if not Repository.get_by_repo_name(repo_name):

            raise JSONRPCError("repo %s does not exist" % repo_name)

        try:

            RepoModel().delete(repo_name)

            Session.commit()

            return dict(

                msg='Deleted repository %s' % repo_name

            )

        except Exception:

            log.error(traceback.format_exc())

            raise JSONRPCError('failed to delete repository %s' % repo_name)

    @HasPermissionAnyDecorator('hg.admin')

    def grant_user_permission(self, apiuser, repo_name, username, perm):

        """

        Grant permission for user on given repository, or update existing one

        if found

        :param repo_name:

        :param username:

        :param perm:

        """

        try:

            repo = Repository.get_by_repo_name(repo_name)

            if repo is None:

                raise JSONRPCError('unknown repository %s' % repo)

            user = User.get_by_username(username)

            if user is None:

                raise JSONRPCError('unknown user %s' % username)

            RepoModel().grant_user_permission(repo=repo, user=user, perm=perm)

            Session.commit()

            return dict(

                msg='Granted perm: %s for user: %s in repo: %s' % (

                    perm, username, repo_name

                )

            )

        except Exception:

            log.error(traceback.format_exc())

            raise JSONRPCError(

                'failed to edit permission %(repo)s for %(user)s' % dict(

                    user=username, repo=repo_name

                )

            )

    @HasPermissionAnyDecorator('hg.admin')

    def revoke_user_permission(self, apiuser, repo_name, username):

        """

        Revoke permission for user on given repository

        :param repo_name:

        :param username:

        """

        try:

            repo = Repository.get_by_repo_name(repo_name)

            if repo is None:

                raise JSONRPCError('unknown repository %s' % repo)

            user = User.get_by_username(username)

            if user is None:

                raise JSONRPCError('unknown user %s' % username)

            RepoModel().revoke_user_permission(repo=repo_name, user=username)

            Session.commit()

            return dict(

                msg='Revoked perm for user: %s in repo: %s' % (

                    username, repo_name

                )

            )

            log.debug('hgrc file is not present at %s skipping...' % path)

            return False

        log.debug('reading hgrc from %s' % path)

        cfg = config.config()

        cfg.read(path)

        for section in ui_sections:

            for k, v in cfg.items(section):

                log.debug('settings ui from file[%s]%s:%s' % (section, k, v))

                baseui.setconfig(section, k, v)

    elif read_from == 'db':

        sa = meta.Session

        ret = sa.query(RhodeCodeUi)\

            .options(FromCache("sql_cache_short", "get_hg_ui_settings"))\

            .all()

        hg_ui = ret

        for ui_ in hg_ui:

            if ui_.ui_active:

                log.debug('settings ui from db[%s]%s:%s', ui_.ui_section,

                          ui_.ui_key, ui_.ui_value)

                baseui.setconfig(ui_.ui_section, ui_.ui_key, ui_.ui_value)

        meta.Session.remove()

    return baseui

def set_rhodecode_config(config):

    """

    Updates pylons config with new settings from database

    :param config:

    """

    hgsettings = RhodeCodeSetting.get_app_settings()

    for k, v in hgsettings.items():

        config[k] = v

def invalidate_cache(cache_key, *args):

    """

    Puts cache invalidation task into db for

    further global cache invalidation

    """

    from rhodecode.model.scm import ScmModel

    if cache_key.startswith('get_repo_cached_'):

        name = cache_key.split('get_repo_cached_')[-1]

        ScmModel().mark_for_invalidation(name)

class EmptyChangeset(BaseChangeset):

    """

    An dummy empty changeset. It's possible to pass hash when creating

    an EmptyChangeset

    """

    def __init__(self, cs='0' * 40, repo=None, requested_revision=None,

                 alias=None):

        self._empty_cs = cs

        self.revision = -1

        self.message = ''

        self.author = ''

        self.date = ''

        self.repository = repo

        self.requested_revision = requested_revision

        self.alias = alias

    @LazyProperty

    def raw_id(self):

        """

        Returns raw string identifying this changeset, useful for web

        representation.

        """

        return self._empty_cs

    @LazyProperty

    def branch(self):

        return get_backend(self.alias).DEFAULT_BRANCH_NAME

    @LazyProperty

    def short_id(self):

        return self.raw_id[:12]

    def get_file_changeset(self, path):

        return self

    def get_file_content(self, path):

        return u''

    def get_file_size(self, path):

        return 0

    """

    """

    sa = meta.Session

    parent = None

    group = None

    # last element is repo in nested groups structure

    groups = groups[:-1]

    rgm = ReposGroupModel(sa)

    for lvl, group_name in enumerate(groups):

        group_name = '/'.join(groups[:lvl] + [group_name])

        group = RepoGroup.get_by_group_name(group_name)

        desc = '%s group' % group_name

        # skip folders that are now removed repos

        if REMOVED_REPO_PAT.match(group_name):

            break

        if group is None:

            group = RepoGroup(group_name, parent)

            group.group_description = desc

            sa.add(group)

            rgm._create_default_perms(group)

        parent = group

    return group

def repo2db_mapper(initial_repo_list, remove_obsolete=False):

    """

    maps all repos given in initial_repo_list, non existing repositories

    are created, if remove_obsolete is True it also check for db entries

    that are not in initial_repo_list and removes them.

    :param initial_repo_list: list of repositories found by scanning methods

    :param remove_obsolete: check for obsolete entries in database

    """

    from rhodecode.model.repo import RepoModel

    sa = meta.Session

    rm = RepoModel()

    user = sa.query(User).filter(User.admin == True).first()

    if user is None:

        raise Exception('Missing administrative account !')

    added = []

    for name, repo in initial_repo_list.items():

        if not rm.get_by_repo_name(name, cache=False):

            log.info('repository %s not found creating default' % name)

            added.append(name)

            form_data = {

             'repo_name': name,

             'repo_name_full': name,

             'repo_type': repo.alias,

             'description': repo.description \

                if repo.description != 'unknown' else '%s repository' % name,

             'private': False,

             'group_id': getattr(group, 'group_id', None)

            }

            rm.create(form_data, user, just_db=True)

    sa.commit()

    removed = []

    if remove_obsolete:

        #remove from database those repositories that are not in the filesystem

        for repo in sa.query(Repository).all():

            if repo.repo_name not in initial_repo_list.keys():

                removed.append(repo.repo_name)

                sa.delete(repo)

                sa.commit()

    return added, removed

# set cache regions for beaker so celery can utilise it

def add_cache(settings):

    cache_settings = {'regions': None}

    for key in settings.keys():

        for prefix in ['beaker.cache.', 'cache.']:

            if key.startswith(prefix):

                name = key.split(prefix)[1].strip()

                cache_settings[name] = settings[key].strip()

    if cache_settings['regions']:

        for region in cache_settings['regions'].split(','):

            region = region.strip()

            region_settings = {}

            for key, value in cache_settings.items():

                if key.startswith(region):

                    region_settings[key.split('.')[1]] = value

            region_settings['expire'] = int(region_settings.get('expire',

                                                                60))

            region_settings.setdefault('lock_dir',

                                       cache_settings.get('lock_dir'))

            region_settings.setdefault('data_dir',

                                       cache_settings.get('data_dir'))

            if 'type' not in region_settings:

                region_settings['type'] = cache_settings.get('type',

                                                             'memory')

            beaker.cache.cache_regions[region] = region_settings

def load_rcextensions(root_path):

    import rhodecode

    from rhodecode.config import conf

    path = os.path.join(root_path, 'rcextensions', '__init__.py')

    if os.path.isfile(path):

        rcext = create_module('rc', path)

        EXT = rhodecode.EXTENSIONS = rcext

        log.debug('Found rcextensions now loading %s...' % rcext)

        # Additional mappings that are not present in the pygments lexers

        conf.LANGUAGES_EXTENSIONS_MAP.update(getattr(EXT, 'EXTRA_MAPPINGS', {}))

        #OVERRIDE OUR EXTENSIONS FROM RC-EXTENSIONS (if present)

        if getattr(EXT, 'INDEX_EXTENSIONS', []) != []:

            log.debug('settings custom INDEX_EXTENSIONS')

            conf.INDEX_EXTENSIONS = getattr(EXT, 'INDEX_EXTENSIONS', [])

        #ADDITIONAL MAPPINGS

        log.debug('adding extra into INDEX_EXTENSIONS')

        conf.INDEX_EXTENSIONS.extend(getattr(EXT, 'EXTRA_INDEX_EXTENSIONS', []))

#==============================================================================

# TEST FUNCTIONS AND CREATORS

#==============================================================================

def create_test_index(repo_location, config, full_index):

    """

    Makes default test index

    :param config: test config

    :param full_index:

    """

    from rhodecode.lib.indexers.daemon import WhooshIndexingDaemon

    from rhodecode.lib.pidlock import DaemonLock, LockHeld

    repo_location = repo_location

    index_location = os.path.join(config['app_conf']['index_dir'])

    if not os.path.exists(index_location):