Changeset - d5527cebf76a
[Not reviewed]
beta
0 3 0
Marcin Kuzminski - 14 years ago 2012-03-11 16:51:23
marcin@python-works.com
Resolve error occurring during recursive group creation in API create-repo function
3 files changed with 17 insertions and 22 deletions:
0 comments (0 inline, 0 general)
docs/changelog.rst
Show inline comments
 
.. _changelog:
 

	
 
=========
 
Changelog
 
=========
 

	
 

	
 
1.3.4 (**2012-XX-XX**)
 
----------------------
 

	
 
:status: in-progress
 
:branch: beta
 

	
 
news
 
++++
 

	
 
- Whoosh logging is now controlled by the .ini files logging setup
 
- added clone-url into edit form on /settings page
 
- added help text into repo add/edit forms
 
- created rcextensions module with additional mappings (ref #322) and
 
  post push/pull/create repo hooks callbacks
 

	
 
fixes
 
+++++
 

	
 
- fixed #390 cache invalidation problems on repos inside group
 
- fixed #385 clone by ID url was loosing proxy prefix in URL
 
- fixed some unicode problems with waitress
 
- fixed issue with escaping < and > in changeset commits
 
- fixed error occurring during recursive group creation in API 
 
  create_repo function
 

	
 
1.3.3 (**2012-03-02**)
 
----------------------
 

	
 
news
 
++++
 

	
 

	
 
fixes
 
+++++
 

	
 
- fixed some python2.5 compatibility issues 
 
- fixed issues with removed repos was accidentally added as groups, after
 
  full rescan of paths
 
- fixes #376 Cannot edit user (using container auth)
 
- fixes #378 Invalid image urls on changeset screen with proxy-prefix 
 
  configuration
 
- fixed initial sorting of repos inside repo group
 
- fixes issue when user tried to resubmit same permission into user/user_groups
 
- bumped beaker version that fixes #375 leap error bug
 
- fixed raw_changeset for git. It was generated with hg patch headers
 
- fixed vcs issue with last_changeset for filenodes
 
- fixed missing commit after hook delete
 
- fixed #372 issues with git operation detection that caused a security issue 
 
  for git repos
 

	
 
1.3.2 (**2012-02-28**)
 
----------------------
 

	
 
news
 
++++
 

	
 

	
 
fixes
 
+++++
 

	
 
- fixed git protocol issues with repos-groups
 
- fixed git remote repos validator that prevented from cloning remote git repos
 
- fixes #370 ending slashes fixes for repo and groups
 
- fixes #368 improved git-protocol detection to handle other clients
 
- fixes #366 When Setting Repository Group To Blank Repo Group Wont Be 
 
  Moved To Root
 
- fixes #371 fixed issues with beaker/sqlalchemy and non-ascii cache keys 
 
- fixed #373 missing cascade drop on user_group_to_perm table
 

	
 
1.3.1 (**2012-02-27**)
 
----------------------
 

	
 
news
 
++++
 

	
 

	
 
fixes
 
+++++
 

	
 
- redirection loop occurs when remember-me wasn't checked during login
 
- fixes issues with git blob history generation 
 
- don't fetch branch for git in file history dropdown. Causes unneeded slowness
 

	
 
1.3.0 (**2012-02-26**)
 
----------------------
 

	
 
news
 
++++
 

	
 
- code review, inspired by github code-comments 
 
- #215 rst and markdown README files support
 
- #252 Container-based and proxy pass-through authentication support
 
- #44 branch browser. Filtering of changelog by branches
 
- mercurial bookmarks support
 
- new hover top menu, optimized to add maximum size for important views
 
- configurable clone url template with possibility to specify  protocol like 
 
  ssh:// or http:// and also manually alter other parts of clone_url.
 
- enabled largefiles extension by default
 
- optimized summary file pages and saved a lot of unused space in them
 
- #239 option to manually mark repository as fork
 
- #320 mapping of commit authors to RhodeCode users
 
- #304 hashes are displayed using monospace font    
 
- diff configuration, toggle white lines and context lines
 
- #307 configurable diffs, whitespace toggle, increasing context lines
 
- sorting on branches, tags and bookmarks using YUI datatable
 
- improved file filter on files page
 
- implements #330 api method for listing nodes ar particular revision
 
- #73 added linking issues in commit messages to chosen issue tracker url
 
  based on user defined regular expression
 
- added linking of changesets in commit messages  
 
- new compact changelog with expandable commit messages
 
- firstname and lastname are optional in user creation
 
- #348 added post-create repository hook
 
- #212 global encoding settings is now configurable from .ini files 
 
- #227 added repository groups permissions
 
- markdown gets codehilite extensions
 
- new API methods, delete_repositories, grante/revoke permissions for groups 
 
  and repos
 
  
 
    
 
fixes
 
+++++
 

	
 
- rewrote dbsession management for atomic operations, and better error handling
 
- fixed sorting of repo tables
 
- #326 escape of special html entities in diffs
 
- normalized user_name => username in api attributes
 
- fixes #298 ldap created users with mixed case emails created conflicts 
 
  on saving a form
 
- fixes issue when owner of a repo couldn't revoke permissions for users 
 
  and groups
 
- fixes #271 rare JSON serialization problem with statistics
 
- fixes #337 missing validation check for conflicting names of a group with a
 
  repositories group
 
- #340 fixed session problem for mysql and celery tasks
 
- fixed #331 RhodeCode mangles repository names if the a repository group 
 
  contains the "full path" to the repositories
 
- #355 RhodeCode doesn't store encrypted LDAP passwords
 

	
 
1.2.5 (**2012-01-28**)
 
----------------------
 

	
 
news
 
++++
 

	
 
fixes
 
+++++
 

	
 
- #340 Celery complains about MySQL server gone away, added session cleanup
 
  for celery tasks
 
- #341 "scanning for repositories in None" log message during Rescan was missing
 
  a parameter
 
- fixed creating archives with subrepos. Some hooks were triggered during that
 
  operation leading to crash.
 
- fixed missing email in account page.
 
- Reverted Mercurial to 2.0.1 for windows due to bug in Mercurial that makes
 
  forking on windows impossible 
 

	
 
1.2.4 (**2012-01-19**)
 
----------------------
 

	
 
news
 
++++
 

	
 
- RhodeCode is bundled with mercurial series 2.0.X by default, with
 
  full support to largefiles extension. Enabled by default in new installations
 
- #329 Ability to Add/Remove Groups to/from a Repository via AP
 
- added requires.txt file with requirements
 
     
 
fixes
 
+++++
 

	
 
- fixes db session issues with celery when emailing admins
 
- #331 RhodeCode mangles repository names if the a repository group 
 
  contains the "full path" to the repositories
 
- #298 Conflicting e-mail addresses for LDAP and RhodeCode users
 
- DB session cleanup after hg protocol operations, fixes issues with
 
  `mysql has gone away` errors
 
- #333 doc fixes for get_repo api function
 
- #271 rare JSON serialization problem with statistics enabled
 
- #337 Fixes issues with validation of repository name conflicting with 
 
  a group name. A proper message is now displayed.
 
- #292 made ldap_dn in user edit readonly, to get rid of confusion that field
 
  doesn't work   
 
- #316 fixes issues with web description in hgrc files 
 

	
 
1.2.3 (**2011-11-02**)
 
----------------------
 

	
 
news
 
++++
 

	
 
- added option to manage repos group for non admin users
 
- added following API methods for get_users, create_user, get_users_groups, 
 
  get_users_group, create_users_group, add_user_to_users_groups, get_repos, 
 
  get_repo, create_repo, add_user_to_repo
 
- implements #237 added password confirmation for my account 
 
  and admin edit user.
 
- implements #291 email notification for global events are now sent to all
 
  administrator users, and global config email.
 
     
 
fixes
 
+++++
 

	
 
- added option for passing auth method for smtp mailer
 
- #276 issue with adding a single user with id>10 to usergroups
 
- #277 fixes windows LDAP settings in which missing values breaks the ldap auth 
 
- #288 fixes managing of repos in a group for non admin user
 

	
 
1.2.2 (**2011-10-17**)
 
----------------------
 

	
 
news
 
++++
 

	
 
- #226 repo groups are available by path instead of numerical id
 
 
 
fixes
 
+++++
 

	
 
- #259 Groups with the same name but with different parent group
 
- #260 Put repo in group, then move group to another group -> repo becomes unavailable
 
- #258 RhodeCode 1.2 assumes egg folder is writable (lockfiles problems)
 
- #265 ldap save fails sometimes on converting attributes to booleans, 
 
  added getter and setter into model that will prevent from this on db model level
 
- fixed problems with timestamps issues #251 and #213
 
- fixes #266 RhodeCode allows to create repo with the same name and in 
 
  the same parent as group
 
- fixes #245 Rescan of the repositories on Windows
 
- fixes #248 cannot edit repos inside a group on windows
 
- fixes #219 forking problems on windows
 

	
 
1.2.1 (**2011-10-08**)
 
----------------------
 

	
 
news
 
++++
 

	
 

	
 
fixes
 
+++++
 

	
 
- fixed problems with basic auth and push problems 
 
- gui fixes
 
- fixed logger
 

	
 
1.2.0 (**2011-10-07**)
 
----------------------
 

	
 
news
 
++++
 

	
 
- implemented #47 repository groups
 
- implemented #89 Can setup google analytics code from settings menu
 
- implemented #91 added nicer looking archive urls with more download options
 
  like tags, branches
 
- implemented #44 into file browsing, and added follow branch option
 
- implemented #84 downloads can be enabled/disabled for each repository
 
- anonymous repository can be cloned without having to pass default:default
 
  into clone url
 
- fixed #90 whoosh indexer can index chooses repositories passed in command 
 
  line
 
- extended journal with day aggregates and paging
 
- implemented #107 source code lines highlight ranges
 
- implemented #93 customizable changelog on combined revision ranges - 
 
  equivalent of githubs compare view 
 
- implemented #108 extended and more powerful LDAP configuration
 
- implemented #56 users groups
 
- major code rewrites optimized codes for speed and memory usage
 
- raw and diff downloads are now in git format
 
- setup command checks for write access to given path
 
- fixed many issues with international characters and unicode. It uses utf8
 
  decode with replace to provide less errors even with non utf8 encoded strings
 
- #125 added API KEY access to feeds
 
- #109 Repository can be created from external Mercurial link (aka. remote 
 
  repository, and manually updated (via pull) from admin panel
 
- beta git support - push/pull server + basic view for git repos
 
- added followers page and forks page
 
- server side file creation (with binary file upload interface) 
 
  and edition with commits powered by codemirror 
 
- #111 file browser file finder, quick lookup files on whole file tree 
 
- added quick login sliding menu into main page
 
- changelog uses lazy loading of affected files details, in some scenarios 
 
  this can improve speed of changelog page dramatically especially for 
 
  larger repositories.
 
- implements #214 added support for downloading subrepos in download menu.
 
- Added basic API for direct operations on rhodecode via JSON
 
- Implemented advanced hook management
 

	
 
fixes
 
+++++
 

	
 
- fixed file browser bug, when switching into given form revision the url was 
 
  not changing
 
- fixed propagation to error controller on simplehg and simplegit middlewares
 
- fixed error when trying to make a download on empty repository
 
- fixed problem with '[' chars in commit messages in journal
 
- fixed #99 Unicode errors, on file node paths with non utf-8 characters
 
- journal fork fixes
 
- removed issue with space inside renamed repository after deletion
 
- fixed strange issue on formencode imports
 
- fixed #126 Deleting repository on Windows, rename used incompatible chars. 
 
- #150 fixes for errors on repositories mapped in db but corrupted in 
 
  filesystem
 
- fixed problem with ascendant characters in realm #181
 
- fixed problem with sqlite file based database connection pool
 
- whoosh indexer and code stats share the same dynamic extensions map
 
- fixes #188 - relationship delete of repo_to_perm entry on user removal
 
- fixes issue #189 Trending source files shows "show more" when no more exist
 
- fixes issue #197 Relative paths for pidlocks
 
- fixes issue #198 password will require only 3 chars now for login form
 
- fixes issue #199 wrong redirection for non admin users after creating a repository
 
- fixes issues #202, bad db constraint made impossible to attach same group 
 
  more than one time. Affects only mysql/postgres
 
- fixes #218 os.kill patch for windows was missing sig param
 
- improved rendering of dag (they are not trimmed anymore when number of 
 
  heads exceeds 5)
 
    
 
1.1.8 (**2011-04-12**)
 
----------------------
 

	
 
news
 
++++
 

	
 
- improved windows support
 

	
 
fixes
 
+++++
 

	
 
- fixed #140 freeze of python dateutil library, since new version is python2.x
 
  incompatible
 
- setup-app will check for write permission in given path
 
- cleaned up license info issue #149
 
- fixes for issues #137,#116 and problems with unicode and accented characters.
 
- fixes crashes on gravatar, when passed in email as unicode
 
- fixed tooltip flickering problems
 
- fixed came_from redirection on windows
 
- fixed logging modules, and sql formatters
 
- windows fixes for os.kill issue #133
 
- fixes path splitting for windows issues #148
 
- fixed issue #143 wrong import on migration to 1.1.X
 
- fixed problems with displaying binary files, thanks to Thomas Waldmann
 
- removed name from archive files since it's breaking ui for long repo names
 
- fixed issue with archive headers sent to browser, thanks to Thomas Waldmann
 
- fixed compatibility for 1024px displays, and larger dpi settings, thanks to 
 
  Thomas Waldmann
 
- fixed issue #166 summary pager was skipping 10 revisions on second page
 

	
 

	
 
1.1.7 (**2011-03-23**)
 
----------------------
 

	
 
news
 
++++
 

	
 
fixes
 
+++++
 

	
 
- fixed (again) #136 installation support for FreeBSD
 

	
 

	
 
1.1.6 (**2011-03-21**)
 
----------------------
 

	
 
news
 
++++
 

	
 
fixes
 
+++++
 

	
 
- fixed #136 installation support for FreeBSD
 
- RhodeCode will check for python version during installation
 

	
 
1.1.5 (**2011-03-17**)
 
----------------------
 

	
 
news
 
++++
 

	
 
- basic windows support, by exchanging pybcrypt into sha256 for windows only
 
  highly inspired by idea of mantis406
 

	
 
fixes
 
+++++
 

	
 
- fixed sorting by author in main page
 
- fixed crashes with diffs on binary files
 
- fixed #131 problem with boolean values for LDAP
 
- fixed #122 mysql problems thanks to striker69 
 
- fixed problem with errors on calling raw/raw_files/annotate functions 
 
  with unknown revisions
 
- fixed returned rawfiles attachment names with international character
 
- cleaned out docs, big thanks to Jason Harris
 

	
 
1.1.4 (**2011-02-19**)
 
----------------------
 

	
 
news
 
++++
 

	
 
fixes
 
+++++
 

	
 
- fixed formencode import problem on settings page, that caused server crash
 
  when that page was accessed as first after server start
 
- journal fixes
 
- fixed option to access repository just by entering http://server/<repo_name> 
 

	
rhodecode/controllers/api/api.py
Show inline comments
 
# -*- coding: utf-8 -*-
 
"""
 
    rhodecode.controllers.api
 
    ~~~~~~~~~~~~~~~~~~~~~~~~~
 

	
 
    API controller for RhodeCode
 

	
 
    :created_on: Aug 20, 2011
 
    :author: marcink
 
    :copyright: (C) 2011-2012 Marcin Kuzminski <marcin@python-works.com>
 
    :license: GPLv3, see COPYING for more details.
 
"""
 
# This program is free software; you can redistribute it and/or
 
# modify it under the terms of the GNU General Public License
 
# as published by the Free Software Foundation; version 2
 
# of the License or (at your opinion) any later version of the license.
 
#
 
# This program is distributed in the hope that it will be useful,
 
# but WITHOUT ANY WARRANTY; without even the implied warranty of
 
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 
# GNU General Public License for more details.
 
#
 
# You should have received a copy of the GNU General Public License
 
# along with this program; if not, write to the Free Software
 
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
 
# MA  02110-1301, USA.
 

	
 
import traceback
 
import logging
 

	
 
from rhodecode.controllers.api import JSONRPCController, JSONRPCError
 
from rhodecode.lib.auth import HasPermissionAllDecorator, \
 
    HasPermissionAnyDecorator, PasswordGenerator
 

	
 
from rhodecode.model.meta import Session
 
from rhodecode.model.scm import ScmModel
 
from rhodecode.model.db import User, UsersGroup, RepoGroup, Repository
 
from rhodecode.model.repo import RepoModel
 
from rhodecode.model.user import UserModel
 
from rhodecode.model.users_group import UsersGroupModel
 
from rhodecode.model.repos_group import ReposGroupModel
 
from rhodecode.lib.utils import map_groups
 

	
 

	
 
log = logging.getLogger(__name__)
 

	
 

	
 
class ApiController(JSONRPCController):
 
    """
 
    API Controller
 

	
 

	
 
    Each method needs to have USER as argument this is then based on given
 
    API_KEY propagated as instance of user object
 

	
 
    Preferably this should be first argument also
 

	
 

	
 
    Each function should also **raise** JSONRPCError for any
 
    errors that happens
 

	
 
    """
 

	
 
    @HasPermissionAllDecorator('hg.admin')
 
    def pull(self, apiuser, repo_name):
 
        """
 
        Dispatch pull action on given repo
 

	
 

	
 
        :param user:
 
        :param repo_name:
 
        """
 

	
 
        if Repository.is_valid(repo_name) is False:
 
            raise JSONRPCError('Unknown repo "%s"' % repo_name)
 

	
 
        try:
 
            ScmModel().pull_changes(repo_name, self.rhodecode_user.username)
 
            return 'Pulled from %s' % repo_name
 
        except Exception:
 
            raise JSONRPCError('Unable to pull changes from "%s"' % repo_name)
 

	
 
    @HasPermissionAllDecorator('hg.admin')
 
    def get_user(self, apiuser, userid):
 
        """"
 
        Get a user by username
 

	
 
        :param apiuser:
 
        :param username:
 
        """
 

	
 
        user = UserModel().get_user(userid)
 
        if user is None:
 
            return user
 

	
 
        return dict(
 
            id=user.user_id,
 
            username=user.username,
 
            firstname=user.name,
 
            lastname=user.lastname,
 
            email=user.email,
 
            active=user.active,
 
            admin=user.admin,
 
            ldap_dn=user.ldap_dn
 
        )
 

	
 
    @HasPermissionAllDecorator('hg.admin')
 
    def get_users(self, apiuser):
 
        """"
 
        Get all users
 

	
 
        :param apiuser:
 
        """
 

	
 
        result = []
 
        for user in User.getAll():
 
            result.append(
 
                dict(
 
                    id=user.user_id,
 
                    username=user.username,
 
                    firstname=user.name,
 
                    lastname=user.lastname,
 
                    email=user.email,
 
                    active=user.active,
 
                    admin=user.admin,
 
                    ldap_dn=user.ldap_dn
 
                )
 
            )
 
        return result
 

	
 
    @HasPermissionAllDecorator('hg.admin')
 
    def create_user(self, apiuser, username, email, password, firstname=None,
 
                    lastname=None, active=True, admin=False, ldap_dn=None):
 
        """
 
        Create new user
 

	
 
        :param apiuser:
 
        :param username:
 
        :param password:
 
        :param email:
 
        :param name:
 
        :param lastname:
 
        :param active:
 
        :param admin:
 
        :param ldap_dn:
 
        """
 
        if User.get_by_username(username):
 
            raise JSONRPCError("user %s already exist" % username)
 

	
 
        if User.get_by_email(email, case_insensitive=True):
 
            raise JSONRPCError("email %s already exist" % email)
 

	
 
        if ldap_dn:
 
            # generate temporary password if ldap_dn
 
            password = PasswordGenerator().gen_password(length=8)
 

	
 
        try:
 
            usr = UserModel().create_or_update(
 
                username, password, email, firstname,
 
                lastname, active, admin, ldap_dn
 
            )
 
            Session.commit()
 
            return dict(
 
                id=usr.user_id,
 
                msg='created new user %s' % username
 
            )
 
        except Exception:
 
            log.error(traceback.format_exc())
 
            raise JSONRPCError('failed to create user %s' % username)
 

	
 
    @HasPermissionAllDecorator('hg.admin')
 
    def update_user(self, apiuser, userid, username, password, email,
 
                    firstname, lastname, active, admin, ldap_dn):
 
        """
 
        Updates given user
 

	
 
        :param apiuser:
 
        :param username:
 
        :param password:
 
        :param email:
 
        :param name:
 
        :param lastname:
 
        :param active:
 
        :param admin:
 
        :param ldap_dn:
 
        """
 
        if not UserModel().get_user(userid):
 
            raise JSONRPCError("user %s does not exist" % username)
 

	
 
        try:
 
            usr = UserModel().create_or_update(
 
                username, password, email, firstname,
 
                lastname, active, admin, ldap_dn
 
            )
 
            Session.commit()
 
            return dict(
 
                id=usr.user_id,
 
                msg='updated user %s' % username
 
            )
 
        except Exception:
 
            log.error(traceback.format_exc())
 
            raise JSONRPCError('failed to update user %s' % username)
 

	
 
    @HasPermissionAllDecorator('hg.admin')
 
    def get_users_group(self, apiuser, group_name):
 
        """"
 
        Get users group by name
 

	
 
        :param apiuser:
 
        :param group_name:
 
        """
 

	
 
        users_group = UsersGroup.get_by_group_name(group_name)
 
        if not users_group:
 
            return None
 

	
 
        members = []
 
        for user in users_group.members:
 
            user = user.user
 
            members.append(dict(id=user.user_id,
 
                            username=user.username,
 
                            firstname=user.name,
 
                            lastname=user.lastname,
 
                            email=user.email,
 
                            active=user.active,
 
                            admin=user.admin,
 
                            ldap=user.ldap_dn))
 

	
 
        return dict(id=users_group.users_group_id,
 
                    group_name=users_group.users_group_name,
 
                    active=users_group.users_group_active,
 
                    members=members)
 

	
 
    @HasPermissionAllDecorator('hg.admin')
 
    def get_users_groups(self, apiuser):
 
        """"
 
        Get all users groups
 

	
 
        :param apiuser:
 
        """
 

	
 
        result = []
 
        for users_group in UsersGroup.getAll():
 
            members = []
 
            for user in users_group.members:
 
                user = user.user
 
                members.append(dict(id=user.user_id,
 
                                username=user.username,
 
                                firstname=user.name,
 
                                lastname=user.lastname,
 
                                email=user.email,
 
                                active=user.active,
 
                                admin=user.admin,
 
                                ldap=user.ldap_dn))
 

	
 
            result.append(dict(id=users_group.users_group_id,
 
                                group_name=users_group.users_group_name,
 
                                active=users_group.users_group_active,
 
                                members=members))
 
        return result
 

	
 
    @HasPermissionAllDecorator('hg.admin')
 
    def create_users_group(self, apiuser, group_name, active=True):
 
        """
 
        Creates an new usergroup
 

	
 
        :param group_name:
 
        :param active:
 
        """
 

	
 
        if self.get_users_group(apiuser, group_name):
 
            raise JSONRPCError("users group %s already exist" % group_name)
 

	
 
        try:
 
            ug = UsersGroupModel().create(name=group_name, active=active)
 
            Session.commit()
 
            return dict(id=ug.users_group_id,
 
                        msg='created new users group %s' % group_name)
 
        except Exception:
 
            log.error(traceback.format_exc())
 
            raise JSONRPCError('failed to create group %s' % group_name)
 

	
 
    @HasPermissionAllDecorator('hg.admin')
 
    def add_user_to_users_group(self, apiuser, group_name, username):
 
        """"
 
        Add a user to a group
 

	
 
        :param apiuser:
 
        :param group_name:
 
        :param username:
 
        """
 

	
 
        try:
 
            users_group = UsersGroup.get_by_group_name(group_name)
 
            if not users_group:
 
                raise JSONRPCError('unknown users group %s' % group_name)
 

	
 
            user = User.get_by_username(username)
 
            if user is None:
 
                raise JSONRPCError('unknown user %s' % username)
 

	
 
            ugm = UsersGroupModel().add_user_to_group(users_group, user)
 
            success = True if ugm != True else False
 
            msg = 'added member %s to users group %s' % (username, group_name)
 
            msg = msg if success else 'User is already in that group'
 
            Session.commit()
 

	
 
            return dict(
 
                id=ugm.users_group_member_id if ugm != True else None,
 
                success=success,
 
                msg=msg
 
            )
 
        except Exception:
 
            log.error(traceback.format_exc())
 
            raise JSONRPCError('failed to add users group member')
 

	
 
    @HasPermissionAllDecorator('hg.admin')
 
    def remove_user_from_users_group(self, apiuser, group_name, username):
 
        """
 
        Remove user from a group
 

	
 
        :param apiuser
 
        :param group_name
 
        :param username
 
        """
 

	
 
        try:
 
            users_group = UsersGroup.get_by_group_name(group_name)
 
            if not users_group:
 
                raise JSONRPCError('unknown users group %s' % group_name)
 

	
 
            user = User.get_by_username(username)
 
            if user is None:
 
                raise JSONRPCError('unknown user %s' % username)
 

	
 
            success = UsersGroupModel().remove_user_from_group(users_group, user)
 
            msg = 'removed member %s from users group %s' % (username, group_name)
 
            msg = msg if success else "User wasn't in group"
 
            Session.commit()
 
            return dict(success=success, msg=msg)
 
        except Exception:
 
            log.error(traceback.format_exc())
 
            raise JSONRPCError('failed to remove user from group')
 

	
 
    @HasPermissionAnyDecorator('hg.admin')
 
    def get_repo(self, apiuser, repoid):
 
        """"
 
        Get repository by name
 

	
 
        :param apiuser:
 
        :param repo_name:
 
        """
 

	
 
        repo = RepoModel().get_repo(repoid)
 
        if repo is None:
 
            raise JSONRPCError('unknown repository %s' % repo)
 

	
 
        members = []
 
        for user in repo.repo_to_perm:
 
            perm = user.permission.permission_name
 
            user = user.user
 
            members.append(
 
                dict(
 
                    type_="user",
 
                    id=user.user_id,
 
                    username=user.username,
 
                    firstname=user.name,
 
                    lastname=user.lastname,
 
                    email=user.email,
 
                    active=user.active,
 
                    admin=user.admin,
 
                    ldap=user.ldap_dn,
 
                    permission=perm
 
                )
 
            )
 
        for users_group in repo.users_group_to_perm:
 
            perm = users_group.permission.permission_name
 
            users_group = users_group.users_group
 
            members.append(
 
                dict(
 
                    type_="users_group",
 
                    id=users_group.users_group_id,
 
                    name=users_group.users_group_name,
 
                    active=users_group.users_group_active,
 
                    permission=perm
 
                )
 
            )
 

	
 
        return dict(
 
            id=repo.repo_id,
 
            repo_name=repo.repo_name,
 
            type=repo.repo_type,
 
            description=repo.description,
 
            members=members
 
        )
 

	
 
    @HasPermissionAnyDecorator('hg.admin')
 
    def get_repos(self, apiuser):
 
        """"
 
        Get all repositories
 

	
 
        :param apiuser:
 
        """
 

	
 
        result = []
 
        for repository in Repository.getAll():
 
            result.append(
 
                dict(
 
                    id=repository.repo_id,
 
                    repo_name=repository.repo_name,
 
                    type=repository.repo_type,
 
                    description=repository.description
 
                )
 
            )
 
        return result
 

	
 
    @HasPermissionAnyDecorator('hg.admin')
 
    def get_repo_nodes(self, apiuser, repo_name, revision, root_path,
 
                       ret_type='all'):
 
        """
 
        returns a list of nodes and it's children
 
        for a given path at given revision. It's possible to specify ret_type
 
        to show only files or dirs
 

	
 
        :param apiuser:
 
        :param repo_name: name of repository
 
        :param revision: revision for which listing should be done
 
        :param root_path: path from which start displaying
 
        :param ret_type: return type 'all|files|dirs' nodes
 
        """
 
        try:
 
            _d, _f = ScmModel().get_nodes(repo_name, revision, root_path,
 
                                          flat=False)
 
            _map = {
 
                'all': _d + _f,
 
                'files': _f,
 
                'dirs': _d,
 
            }
 
            return _map[ret_type]
 
        except KeyError:
 
            raise JSONRPCError('ret_type must be one of %s' % _map.keys())
 
        except Exception, e:
 
            raise JSONRPCError(e)
 

	
 
    @HasPermissionAnyDecorator('hg.admin', 'hg.create.repository')
 
    def create_repo(self, apiuser, repo_name, owner_name, description='',
 
                    repo_type='hg', private=False, clone_uri=None):
 
        """
 
        Create repository, if clone_url is given it makes a remote clone
 

	
 
        :param apiuser:
 
        :param repo_name:
 
        :param owner_name:
 
        :param description:
 
        :param repo_type:
 
        :param private:
 
        :param clone_uri:
 
        """
 

	
 
        try:
 
            owner = User.get_by_username(owner_name)
 
            if owner is None:
 
                raise JSONRPCError('unknown user %s' % owner_name)
 

	
 
            if Repository.get_by_repo_name(repo_name):
 
                raise JSONRPCError("repo %s already exist" % repo_name)
 

	
 
            groups = repo_name.split('/')
 
            groups = repo_name.split(Repository.url_sep())
 
            real_name = groups[-1]
 
            groups = groups[:-1]
 
            parent_id = None
 
            for g in groups:
 
                group = RepoGroup.get_by_group_name(g)
 
                if not group:
 
                    group = ReposGroupModel().create(g, '', parent_id)
 
                parent_id = group.group_id
 
            # create structure of groups
 
            group = map_groups(repo_name)
 

	
 
            repo = RepoModel().create(
 
                dict(
 
                    repo_name=real_name,
 
                    repo_name_full=repo_name,
 
                    description=description,
 
                    private=private,
 
                    repo_type=repo_type,
 
                    repo_group=parent_id,
 
                    repo_group=group.group_id if group else None,
 
                    clone_uri=clone_uri
 
                ),
 
                owner
 
            )
 
            Session.commit()
 

	
 
            return dict(
 
                id=repo.repo_id,
 
                msg="Created new repository %s" % repo.repo_name
 
            )
 

	
 
        except Exception:
 
            log.error(traceback.format_exc())
 
            raise JSONRPCError('failed to create repository %s' % repo_name)
 

	
 
    @HasPermissionAnyDecorator('hg.admin')
 
    def delete_repo(self, apiuser, repo_name):
 
        """
 
        Deletes a given repository
 

	
 
        :param repo_name:
 
        """
 
        if not Repository.get_by_repo_name(repo_name):
 
            raise JSONRPCError("repo %s does not exist" % repo_name)
 
        try:
 
            RepoModel().delete(repo_name)
 
            Session.commit()
 
            return dict(
 
                msg='Deleted repository %s' % repo_name
 
            )
 
        except Exception:
 
            log.error(traceback.format_exc())
 
            raise JSONRPCError('failed to delete repository %s' % repo_name)
 

	
 
    @HasPermissionAnyDecorator('hg.admin')
 
    def grant_user_permission(self, apiuser, repo_name, username, perm):
 
        """
 
        Grant permission for user on given repository, or update existing one
 
        if found
 

	
 
        :param repo_name:
 
        :param username:
 
        :param perm:
 
        """
 

	
 
        try:
 
            repo = Repository.get_by_repo_name(repo_name)
 
            if repo is None:
 
                raise JSONRPCError('unknown repository %s' % repo)
 

	
 
            user = User.get_by_username(username)
 
            if user is None:
 
                raise JSONRPCError('unknown user %s' % username)
 

	
 
            RepoModel().grant_user_permission(repo=repo, user=user, perm=perm)
 

	
 
            Session.commit()
 
            return dict(
 
                msg='Granted perm: %s for user: %s in repo: %s' % (
 
                    perm, username, repo_name
 
                )
 
            )
 
        except Exception:
 
            log.error(traceback.format_exc())
 
            raise JSONRPCError(
 
                'failed to edit permission %(repo)s for %(user)s' % dict(
 
                    user=username, repo=repo_name
 
                )
 
            )
 

	
 
    @HasPermissionAnyDecorator('hg.admin')
 
    def revoke_user_permission(self, apiuser, repo_name, username):
 
        """
 
        Revoke permission for user on given repository
 

	
 
        :param repo_name:
 
        :param username:
 
        """
 

	
 
        try:
 
            repo = Repository.get_by_repo_name(repo_name)
 
            if repo is None:
 
                raise JSONRPCError('unknown repository %s' % repo)
 

	
 
            user = User.get_by_username(username)
 
            if user is None:
 
                raise JSONRPCError('unknown user %s' % username)
 

	
 
            RepoModel().revoke_user_permission(repo=repo_name, user=username)
 

	
 
            Session.commit()
 
            return dict(
 
                msg='Revoked perm for user: %s in repo: %s' % (
 
                    username, repo_name
 
                )
 
            )
 
        except Exception:
 
            log.error(traceback.format_exc())
 
            raise JSONRPCError(
 
                'failed to edit permission %(repo)s for %(user)s' % dict(
 
                    user=username, repo=repo_name
 
                )
 
            )
 

	
 
    @HasPermissionAnyDecorator('hg.admin')
 
    def grant_users_group_permission(self, apiuser, repo_name, group_name, perm):
 
        """
 
        Grant permission for users group on given repository, or update
 
        existing one if found
 

	
 
        :param repo_name:
 
        :param group_name:
 
        :param perm:
 
        """
 

	
 
        try:
 
            repo = Repository.get_by_repo_name(repo_name)
 
            if repo is None:
 
                raise JSONRPCError('unknown repository %s' % repo)
 

	
 
            user_group = UsersGroup.get_by_group_name(group_name)
 
            if user_group is None:
 
                raise JSONRPCError('unknown users group %s' % user_group)
 

	
 
            RepoModel().grant_users_group_permission(repo=repo_name,
 
                                                     group_name=group_name,
 
                                                     perm=perm)
 

	
 
            Session.commit()
 
            return dict(
 
                msg='Granted perm: %s for group: %s in repo: %s' % (
 
                    perm, group_name, repo_name
 
                )
 
            )
 
        except Exception:
 
            log.error(traceback.format_exc())
 
            raise JSONRPCError(
 
                'failed to edit permission %(repo)s for %(usersgr)s' % dict(
 
                    usersgr=group_name, repo=repo_name
 
                )
 
            )
 

	
 
    @HasPermissionAnyDecorator('hg.admin')
 
    def revoke_users_group_permission(self, apiuser, repo_name, group_name):
 
        """
 
        Revoke permission for users group on given repository
 

	
 
        :param repo_name:
 
        :param group_name:
 
        """
 

	
 
        try:
 
            repo = Repository.get_by_repo_name(repo_name)
 
            if repo is None:
 
                raise JSONRPCError('unknown repository %s' % repo)
 

	
 
            user_group = UsersGroup.get_by_group_name(group_name)
 
            if user_group is None:
 
                raise JSONRPCError('unknown users group %s' % user_group)
 

	
 
            RepoModel().revoke_users_group_permission(repo=repo_name,
 
                                                      group_name=group_name)
 

	
 
            Session.commit()
 
            return dict(
 
                msg='Revoked perm for group: %s in repo: %s' % (
 
                    group_name, repo_name
 
                )
 
            )
 
        except Exception:
 
            log.error(traceback.format_exc())
 
            raise JSONRPCError(
 
                'failed to edit permission %(repo)s for %(usersgr)s' % dict(
 
                    usersgr=group_name, repo=repo_name
 
                )
 
            )
rhodecode/lib/utils.py
Show inline comments
 
# -*- coding: utf-8 -*-
 
"""
 
    rhodecode.lib.utils
 
    ~~~~~~~~~~~~~~~~~~~
 

	
 
    Utilities library for RhodeCode
 

	
 
    :created_on: Apr 18, 2010
 
    :author: marcink
 
    :copyright: (C) 2010-2012 Marcin Kuzminski <marcin@python-works.com>
 
    :license: GPLv3, see COPYING for more details.
 
"""
 
# This program is free software: you can redistribute it and/or modify
 
# it under the terms of the GNU General Public License as published by
 
# the Free Software Foundation, either version 3 of the License, or
 
# (at your option) any later version.
 
#
 
# This program is distributed in the hope that it will be useful,
 
# but WITHOUT ANY WARRANTY; without even the implied warranty of
 
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 
# GNU General Public License for more details.
 
#
 
# You should have received a copy of the GNU General Public License
 
# along with this program.  If not, see <http://www.gnu.org/licenses/>.
 

	
 
import os
 
import re
 
import logging
 
import datetime
 
import traceback
 
import paste
 
import beaker
 
import tarfile
 
import shutil
 
from os.path import abspath
 
from os.path import dirname as dn, join as jn
 

	
 
from paste.script.command import Command, BadCommand
 

	
 
from mercurial import ui, config
 

	
 
from webhelpers.text import collapse, remove_formatting, strip_tags
 

	
 
from rhodecode.lib.vcs import get_backend
 
from rhodecode.lib.vcs.backends.base import BaseChangeset
 
from rhodecode.lib.vcs.utils.lazy import LazyProperty
 
from rhodecode.lib.vcs.utils.helpers import get_scm
 
from rhodecode.lib.vcs.exceptions import VCSError
 

	
 
from rhodecode.lib.caching_query import FromCache
 

	
 
from rhodecode.model import meta
 
from rhodecode.model.db import Repository, User, RhodeCodeUi, \
 
    UserLog, RepoGroup, RhodeCodeSetting, UserRepoGroupToPerm
 
from rhodecode.model.meta import Session
 
from rhodecode.model.repos_group import ReposGroupModel
 
from rhodecode.lib.utils2 import safe_str, safe_unicode
 
from rhodecode.lib.vcs.utils.fakemod import create_module
 

	
 
log = logging.getLogger(__name__)
 

	
 
REMOVED_REPO_PAT = re.compile(r'rm__\d{8}_\d{6}_\d{6}__.*')
 

	
 

	
 
def recursive_replace(str_, replace=' '):
 
    """
 
    Recursive replace of given sign to just one instance
 

	
 
    :param str_: given string
 
    :param replace: char to find and replace multiple instances
 

	
 
    Examples::
 
    >>> recursive_replace("Mighty---Mighty-Bo--sstones",'-')
 
    'Mighty-Mighty-Bo-sstones'
 
    """
 

	
 
    if str_.find(replace * 2) == -1:
 
        return str_
 
    else:
 
        str_ = str_.replace(replace * 2, replace)
 
        return recursive_replace(str_, replace)
 

	
 

	
 
def repo_name_slug(value):
 
    """
 
    Return slug of name of repository
 
    This function is called on each creation/modification
 
    of repository to prevent bad names in repo
 
    """
 

	
 
    slug = remove_formatting(value)
 
    slug = strip_tags(slug)
 

	
 
    for c in """=[]\;'"<>,/~!@#$%^&*()+{}|: """:
 
        slug = slug.replace(c, '-')
 
    slug = recursive_replace(slug, '-')
 
    slug = collapse(slug, '-')
 
    return slug
 

	
 

	
 
def get_repo_slug(request):
 
    _repo = request.environ['pylons.routes_dict'].get('repo_name')
 
    if _repo:
 
        _repo = _repo.rstrip('/')
 
    return _repo
 

	
 

	
 
def get_repos_group_slug(request):
 
    _group = request.environ['pylons.routes_dict'].get('group_name')
 
    if _group:
 
        _group = _group.rstrip('/')
 
    return _group
 

	
 

	
 
def action_logger(user, action, repo, ipaddr='', sa=None, commit=False):
 
    """
 
    Action logger for various actions made by users
 

	
 
    :param user: user that made this action, can be a unique username string or
 
        object containing user_id attribute
 
    :param action: action to log, should be on of predefined unique actions for
 
        easy translations
 
    :param repo: string name of repository or object containing repo_id,
 
        that action was made on
 
    :param ipaddr: optional ip address from what the action was made
 
    :param sa: optional sqlalchemy session
 

	
 
    """
 

	
 
    if not sa:
 
        sa = meta.Session
 

	
 
    try:
 
        if hasattr(user, 'user_id'):
 
            user_obj = user
 
        elif isinstance(user, basestring):
 
            user_obj = User.get_by_username(user)
 
        else:
 
            raise Exception('You have to provide user object or username')
 

	
 
        if hasattr(repo, 'repo_id'):
 
            repo_obj = Repository.get(repo.repo_id)
 
            repo_name = repo_obj.repo_name
 
        elif  isinstance(repo, basestring):
 
            repo_name = repo.lstrip('/')
 
            repo_obj = Repository.get_by_repo_name(repo_name)
 
        else:
 
            raise Exception('You have to provide repository to action logger')
 

	
 
        user_log = UserLog()
 
        user_log.user_id = user_obj.user_id
 
        user_log.action = action
 

	
 
        user_log.repository_id = repo_obj.repo_id
 
        user_log.repository_name = repo_name
 

	
 
        user_log.action_date = datetime.datetime.now()
 
        user_log.user_ip = ipaddr
 
        sa.add(user_log)
 

	
 
        log.info(
 
            'Adding user %s, action %s on %s' % (user_obj, action,
 
                                                 safe_unicode(repo))
 
        )
 
        if commit:
 
            sa.commit()
 
    except:
 
        log.error(traceback.format_exc())
 
        raise
 

	
 

	
 
def get_repos(path, recursive=False):
 
    """
 
    Scans given path for repos and return (name,(type,path)) tuple
 

	
 
    :param path: path to scan for repositories
 
    :param recursive: recursive search and return names with subdirs in front
 
    """
 

	
 
    # remove ending slash for better results
 
    path = path.rstrip(os.sep)
 

	
 
    def _get_repos(p):
 
        if not os.access(p, os.W_OK):
 
            return
 
        for dirpath in os.listdir(p):
 
            if os.path.isfile(os.path.join(p, dirpath)):
 
                continue
 
            cur_path = os.path.join(p, dirpath)
 
            try:
 
                scm_info = get_scm(cur_path)
 
                yield scm_info[1].split(path, 1)[-1].lstrip(os.sep), scm_info
 
            except VCSError:
 
                if not recursive:
 
                    continue
 
                #check if this dir containts other repos for recursive scan
 
                rec_path = os.path.join(p, dirpath)
 
                if os.path.isdir(rec_path):
 
                    for inner_scm in _get_repos(rec_path):
 
                        yield inner_scm
 

	
 
    return _get_repos(path)
 

	
 

	
 
def is_valid_repo(repo_name, base_path):
 
    """
 
    Returns True if given path is a valid repository False otherwise
 

	
 
    :param repo_name:
 
    :param base_path:
 

	
 
    :return True: if given path is a valid repository
 
    """
 
    full_path = os.path.join(safe_str(base_path), safe_str(repo_name))
 

	
 
    try:
 
        get_scm(full_path)
 
        return True
 
    except VCSError:
 
        return False
 

	
 

	
 
def is_valid_repos_group(repos_group_name, base_path):
 
    """
 
    Returns True if given path is a repos group False otherwise
 

	
 
    :param repo_name:
 
    :param base_path:
 
    """
 
    full_path = os.path.join(safe_str(base_path), safe_str(repos_group_name))
 

	
 
    # check if it's not a repo
 
    if is_valid_repo(repos_group_name, base_path):
 
        return False
 

	
 
    # check if it's a valid path
 
    if os.path.isdir(full_path):
 
        return True
 

	
 
    return False
 

	
 

	
 
def ask_ok(prompt, retries=4, complaint='Yes or no, please!'):
 
    while True:
 
        ok = raw_input(prompt)
 
        if ok in ('y', 'ye', 'yes'):
 
            return True
 
        if ok in ('n', 'no', 'nop', 'nope'):
 
            return False
 
        retries = retries - 1
 
        if retries < 0:
 
            raise IOError
 
        print complaint
 

	
 
#propagated from mercurial documentation
 
ui_sections = ['alias', 'auth',
 
                'decode/encode', 'defaults',
 
                'diff', 'email',
 
                'extensions', 'format',
 
                'merge-patterns', 'merge-tools',
 
                'hooks', 'http_proxy',
 
                'smtp', 'patch',
 
                'paths', 'profiling',
 
                'server', 'trusted',
 
                'ui', 'web', ]
 

	
 

	
 
def make_ui(read_from='file', path=None, checkpaths=True):
 
    """
 
    A function that will read python rc files or database
 
    and make an mercurial ui object from read options
 

	
 
    :param path: path to mercurial config file
 
    :param checkpaths: check the path
 
    :param read_from: read from 'file' or 'db'
 
    """
 

	
 
    baseui = ui.ui()
 

	
 
    # clean the baseui object
 
    baseui._ocfg = config.config()
 
    baseui._ucfg = config.config()
 
    baseui._tcfg = config.config()
 

	
 
    if read_from == 'file':
 
        if not os.path.isfile(path):
 
            log.debug('hgrc file is not present at %s skipping...' % path)
 
            return False
 
        log.debug('reading hgrc from %s' % path)
 
        cfg = config.config()
 
        cfg.read(path)
 
        for section in ui_sections:
 
            for k, v in cfg.items(section):
 
                log.debug('settings ui from file[%s]%s:%s' % (section, k, v))
 
                baseui.setconfig(section, k, v)
 

	
 
    elif read_from == 'db':
 
        sa = meta.Session
 
        ret = sa.query(RhodeCodeUi)\
 
            .options(FromCache("sql_cache_short", "get_hg_ui_settings"))\
 
            .all()
 

	
 
        hg_ui = ret
 
        for ui_ in hg_ui:
 
            if ui_.ui_active:
 
                log.debug('settings ui from db[%s]%s:%s', ui_.ui_section,
 
                          ui_.ui_key, ui_.ui_value)
 
                baseui.setconfig(ui_.ui_section, ui_.ui_key, ui_.ui_value)
 

	
 
        meta.Session.remove()
 
    return baseui
 

	
 

	
 
def set_rhodecode_config(config):
 
    """
 
    Updates pylons config with new settings from database
 

	
 
    :param config:
 
    """
 
    hgsettings = RhodeCodeSetting.get_app_settings()
 

	
 
    for k, v in hgsettings.items():
 
        config[k] = v
 

	
 

	
 
def invalidate_cache(cache_key, *args):
 
    """
 
    Puts cache invalidation task into db for
 
    further global cache invalidation
 
    """
 

	
 
    from rhodecode.model.scm import ScmModel
 

	
 
    if cache_key.startswith('get_repo_cached_'):
 
        name = cache_key.split('get_repo_cached_')[-1]
 
        ScmModel().mark_for_invalidation(name)
 

	
 

	
 
class EmptyChangeset(BaseChangeset):
 
    """
 
    An dummy empty changeset. It's possible to pass hash when creating
 
    an EmptyChangeset
 
    """
 

	
 
    def __init__(self, cs='0' * 40, repo=None, requested_revision=None,
 
                 alias=None):
 
        self._empty_cs = cs
 
        self.revision = -1
 
        self.message = ''
 
        self.author = ''
 
        self.date = ''
 
        self.repository = repo
 
        self.requested_revision = requested_revision
 
        self.alias = alias
 

	
 
    @LazyProperty
 
    def raw_id(self):
 
        """
 
        Returns raw string identifying this changeset, useful for web
 
        representation.
 
        """
 

	
 
        return self._empty_cs
 

	
 
    @LazyProperty
 
    def branch(self):
 
        return get_backend(self.alias).DEFAULT_BRANCH_NAME
 

	
 
    @LazyProperty
 
    def short_id(self):
 
        return self.raw_id[:12]
 

	
 
    def get_file_changeset(self, path):
 
        return self
 

	
 
    def get_file_content(self, path):
 
        return u''
 

	
 
    def get_file_size(self, path):
 
        return 0
 

	
 

	
 
def map_groups(groups):
 
def map_groups(path):
 
    """
 
    Checks for groups existence, and creates groups structures.
 
    It returns last group in structure
 
    Given a full path to a repository, create all nested groups that this
 
    repo is inside. This function creates parent-child relationships between
 
    groups and creates default perms for all new groups.
 

	
 
    :param groups: list of groups structure
 
    :param paths: full path to repository
 
    """
 
    sa = meta.Session
 

	
 
    groups = path.split(Repository.url_sep())
 
    parent = None
 
    group = None
 

	
 
    # last element is repo in nested groups structure
 
    groups = groups[:-1]
 
    rgm = ReposGroupModel(sa)
 
    for lvl, group_name in enumerate(groups):
 
        group_name = '/'.join(groups[:lvl] + [group_name])
 
        group = RepoGroup.get_by_group_name(group_name)
 
        desc = '%s group' % group_name
 

	
 
#        # WTF that doesn't work !?
 
#        if group is None:
 
#            group = rgm.create(group_name, desc, parent, just_db=True)
 
#            sa.commit()
 

	
 
        # skip folders that are now removed repos
 
        if REMOVED_REPO_PAT.match(group_name):
 
            break
 

	
 
        if group is None:
 
            log.debug('creating group level: %s group_name: %s' % (lvl, group_name))
 
            log.debug('creating group level: %s group_name: %s' % (lvl,
 
                                                                   group_name))
 
            group = RepoGroup(group_name, parent)
 
            group.group_description = desc
 
            sa.add(group)
 
            rgm._create_default_perms(group)
 
            sa.commit()
 
            sa.flush()
 
        parent = group
 
    return group
 

	
 

	
 
def repo2db_mapper(initial_repo_list, remove_obsolete=False):
 
    """
 
    maps all repos given in initial_repo_list, non existing repositories
 
    are created, if remove_obsolete is True it also check for db entries
 
    that are not in initial_repo_list and removes them.
 

	
 
    :param initial_repo_list: list of repositories found by scanning methods
 
    :param remove_obsolete: check for obsolete entries in database
 
    """
 
    from rhodecode.model.repo import RepoModel
 
    sa = meta.Session
 
    rm = RepoModel()
 
    user = sa.query(User).filter(User.admin == True).first()
 
    if user is None:
 
        raise Exception('Missing administrative account !')
 
    added = []
 

	
 
    for name, repo in initial_repo_list.items():
 
        group = map_groups(name.split(Repository.url_sep()))
 
        group = map_groups(name)
 
        if not rm.get_by_repo_name(name, cache=False):
 
            log.info('repository %s not found creating default' % name)
 
            added.append(name)
 
            form_data = {
 
             'repo_name': name,
 
             'repo_name_full': name,
 
             'repo_type': repo.alias,
 
             'description': repo.description \
 
                if repo.description != 'unknown' else '%s repository' % name,
 
             'private': False,
 
             'group_id': getattr(group, 'group_id', None)
 
            }
 
            rm.create(form_data, user, just_db=True)
 
    sa.commit()
 
    removed = []
 
    if remove_obsolete:
 
        #remove from database those repositories that are not in the filesystem
 
        for repo in sa.query(Repository).all():
 
            if repo.repo_name not in initial_repo_list.keys():
 
                removed.append(repo.repo_name)
 
                sa.delete(repo)
 
                sa.commit()
 

	
 
    return added, removed
 

	
 

	
 
# set cache regions for beaker so celery can utilise it
 
def add_cache(settings):
 
    cache_settings = {'regions': None}
 
    for key in settings.keys():
 
        for prefix in ['beaker.cache.', 'cache.']:
 
            if key.startswith(prefix):
 
                name = key.split(prefix)[1].strip()
 
                cache_settings[name] = settings[key].strip()
 
    if cache_settings['regions']:
 
        for region in cache_settings['regions'].split(','):
 
            region = region.strip()
 
            region_settings = {}
 
            for key, value in cache_settings.items():
 
                if key.startswith(region):
 
                    region_settings[key.split('.')[1]] = value
 
            region_settings['expire'] = int(region_settings.get('expire',
 
                                                                60))
 
            region_settings.setdefault('lock_dir',
 
                                       cache_settings.get('lock_dir'))
 
            region_settings.setdefault('data_dir',
 
                                       cache_settings.get('data_dir'))
 

	
 
            if 'type' not in region_settings:
 
                region_settings['type'] = cache_settings.get('type',
 
                                                             'memory')
 
            beaker.cache.cache_regions[region] = region_settings
 

	
 

	
 
def load_rcextensions(root_path):
 
    import rhodecode
 
    from rhodecode.config import conf
 

	
 
    path = os.path.join(root_path, 'rcextensions', '__init__.py')
 
    if os.path.isfile(path):
 
        rcext = create_module('rc', path)
 
        EXT = rhodecode.EXTENSIONS = rcext
 
        log.debug('Found rcextensions now loading %s...' % rcext)
 

	
 
        # Additional mappings that are not present in the pygments lexers
 
        conf.LANGUAGES_EXTENSIONS_MAP.update(getattr(EXT, 'EXTRA_MAPPINGS', {}))
 

	
 
        #OVERRIDE OUR EXTENSIONS FROM RC-EXTENSIONS (if present)
 

	
 
        if getattr(EXT, 'INDEX_EXTENSIONS', []) != []:
 
            log.debug('settings custom INDEX_EXTENSIONS')
 
            conf.INDEX_EXTENSIONS = getattr(EXT, 'INDEX_EXTENSIONS', [])
 

	
 
        #ADDITIONAL MAPPINGS
 
        log.debug('adding extra into INDEX_EXTENSIONS')
 
        conf.INDEX_EXTENSIONS.extend(getattr(EXT, 'EXTRA_INDEX_EXTENSIONS', []))
 

	
 

	
 
#==============================================================================
 
# TEST FUNCTIONS AND CREATORS
 
#==============================================================================
 
def create_test_index(repo_location, config, full_index):
 
    """
 
    Makes default test index
 

	
 
    :param config: test config
 
    :param full_index:
 
    """
 

	
 
    from rhodecode.lib.indexers.daemon import WhooshIndexingDaemon
 
    from rhodecode.lib.pidlock import DaemonLock, LockHeld
 

	
 
    repo_location = repo_location
 

	
 
    index_location = os.path.join(config['app_conf']['index_dir'])
 
    if not os.path.exists(index_location):
 
        os.makedirs(index_location)
 

	
 
    try:
 
        l = DaemonLock(file_=jn(dn(index_location), 'make_index.lock'))
 
        WhooshIndexingDaemon(index_location=index_location,
 
                             repo_location=repo_location)\
 
            .run(full_index=full_index)
 
        l.release()
 
    except LockHeld:
 
        pass
 

	
 

	
 
def create_test_env(repos_test_path, config):
 
    """
 
    Makes a fresh database and
 
    install test repository into tmp dir
 
    """
 
    from rhodecode.lib.db_manage import DbManage
 
    from rhodecode.tests import HG_REPO, TESTS_TMP_PATH
 

	
 
    # PART ONE create db
 
    dbconf = config['sqlalchemy.db1.url']
 
    log.debug('making test db %s' % dbconf)
 

	
 
    # create test dir if it doesn't exist
 
    if not os.path.isdir(repos_test_path):
 
        log.debug('Creating testdir %s' % repos_test_path)
 
        os.makedirs(repos_test_path)
 

	
 
    dbmanage = DbManage(log_sql=True, dbconf=dbconf, root=config['here'],
 
                        tests=True)
 
    dbmanage.create_tables(override=True)
 
    dbmanage.create_settings(dbmanage.config_prompt(repos_test_path))
 
    dbmanage.create_default_user()
 
    dbmanage.admin_prompt()
 
    dbmanage.create_permissions()
 
    dbmanage.populate_default_permissions()
 
    Session.commit()
 
    # PART TWO make test repo
 
    log.debug('making test vcs repositories')
 

	
 
    idx_path = config['app_conf']['index_dir']
 
    data_path = config['app_conf']['cache_dir']
 

	
 
    #clean index and data
 
    if idx_path and os.path.exists(idx_path):
 
        log.debug('remove %s' % idx_path)
 
        shutil.rmtree(idx_path)
 

	
 
    if data_path and os.path.exists(data_path):
 
        log.debug('remove %s' % data_path)
 
        shutil.rmtree(data_path)
 

	
 
    #CREATE DEFAULT HG REPOSITORY
 
    cur_dir = dn(dn(abspath(__file__)))
 
    tar = tarfile.open(jn(cur_dir, 'tests', "vcs_test_hg.tar.gz"))
 
    tar.extractall(jn(TESTS_TMP_PATH, HG_REPO))
 
    tar.close()
 

	
 

	
 
#==============================================================================
 
# PASTER COMMANDS
 
#==============================================================================
 
class BasePasterCommand(Command):
 
    """
 
    Abstract Base Class for paster commands.
 

	
 
    The celery commands are somewhat aggressive about loading
 
    celery.conf, and since our module sets the `CELERY_LOADER`
 
    environment variable to our loader, we have to bootstrap a bit and
 
    make sure we've had a chance to load the pylons config off of the
 
    command line, otherwise everything fails.
 
    """
 
    min_args = 1
 
    min_args_error = "Please provide a paster config file as an argument."
 
    takes_config_file = 1
 
    requires_config_file = True
 

	
 
    def notify_msg(self, msg, log=False):
 
        """Make a notification to user, additionally if logger is passed
 
        it logs this action using given logger
 

	
 
        :param msg: message that will be printed to user
 
        :param log: logging instance, to use to additionally log this message
 

	
 
        """
 
        if log and isinstance(log, logging):
 
            log(msg)
 

	
 
    def run(self, args):
 
        """
 
        Overrides Command.run
 

	
 
        Checks for a config file argument and loads it.
 
        """
 
        if len(args) < self.min_args:
 
            raise BadCommand(
 
                self.min_args_error % {'min_args': self.min_args,
 
                                       'actual_args': len(args)})
 

	
 
        # Decrement because we're going to lob off the first argument.
 
        # @@ This is hacky
 
        self.min_args -= 1
 
        self.bootstrap_config(args[0])
 
        self.update_parser()
 
        return super(BasePasterCommand, self).run(args[1:])
 

	
 
    def update_parser(self):
 
        """
 
        Abstract method.  Allows for the class's parser to be updated
 
        before the superclass's `run` method is called.  Necessary to
 
        allow options/arguments to be passed through to the underlying
 
        celery command.
 
        """
 
        raise NotImplementedError("Abstract Method.")
 

	
 
    def bootstrap_config(self, conf):
 
        """
 
        Loads the pylons configuration.
 
        """
 
        from pylons import config as pylonsconfig
 

	
 
        self.path_to_ini_file = os.path.realpath(conf)
 
        conf = paste.deploy.appconfig('config:' + self.path_to_ini_file)
 
        pylonsconfig.init_app(conf.global_conf, conf.local_conf)
0 comments (0 inline, 0 general)