.. _changelog:

=========

Changelog

=========

1.4.0 (**2012-XX-XX**)

----------------------

:status: in-progress

:branch: beta

news

++++

 - new codereview system

 - changed setup-app into setup-rhodecode and added default options to it.

fixes

+++++

1.3.6 (**2012-05-17**)

----------------------

news

++++

- chinese traditional translation

- changed setup-app into setup-rhodecode and added arguments for auto-setup 

  mode that doesn't need user interaction 

fixes

+++++

- fixed no scm found warning

- fixed __future__ import error on rcextensions

- made simplejson required lib for speedup on JSON encoding

- fixes #449 bad regex could get more than revisions from parsing history

1.3.5 (**2012-05-10**)

----------------------

news

++++

- use ext_json for json module

- unified annotation view with file source view

- notification improvements, better inbox + css

- #419 don't strip passwords for login forms, make rhodecode 

  more compatible with LDAP servers

- Added HTTP_X_FORWARDED_FOR as another method of extracting 

  IP for pull/push logs. - moved all to base controller  

- #415: Adding comment to changeset causes reload. 

  Comments are now added via ajax and doesn't reload the page

- #374 LDAP config is discarded when LDAP can't be activated

- limited push/pull operations are now logged for git in the journal

- bumped mercurial to 2.2.X series

- added support for displaying submodules in file-browser

- #421 added bookmarks in changelog view

fixes

+++++

- fixed dev-version marker for stable when served from source codes

- fixed missing permission checks on show forks page

- #418 cast to unicode fixes in notification objects

- #426 fixed mention extracting regex

- fixed remote-pulling for git remotes remopositories

- fixed #434: Error when accessing files or changesets of a git repository 

  with submodules

- fixed issue with empty APIKEYS for users after registration ref. #438

- fixed issue with getting README files from git repositories

1.3.4 (**2012-03-28**)

----------------------

news

++++

- Whoosh logging is now controlled by the .ini files logging setup

- added clone-url into edit form on /settings page

- added help text into repo add/edit forms

- created rcextensions module with additional mappings (ref #322) and

  post push/pull/create repo hooks callbacks

- implemented #377 Users view for his own permissions on account page

- #399 added inheritance of permissions for users group on repos groups

- #401 repository group is automatically pre-selected when adding repos 

  inside a repository group

- added alternative HTTP 403 response when client failed to authenticate. Helps 

  solving issues with Mercurial and LDAP

- #402 removed group prefix from repository name when listing repositories 

  inside a group

- added gravatars into permission view and permissions autocomplete

- #347 when running multiple RhodeCode instances, properly invalidates cache 

  for all registered servers

fixes

+++++

- fixed #390 cache invalidation problems on repos inside group

- fixed #385 clone by ID url was loosing proxy prefix in URL

- fixed some unicode problems with waitress

- fixed issue with escaping < and > in changeset commits

- fixed error occurring during recursive group creation in API 

  create_repo function

- fixed #393 py2.5 fixes for routes url generator

- fixed #397 Private repository groups shows up before login

- fixed #396 fixed problems with revoking users in nested groups

- fixed mysql unicode issues + specified InnoDB as default engine with 

  utf8 charset

- #406 trim long branch/tag names in changelog to not break UI

1.3.3 (**2012-03-02**)

----------------------

news

++++

fixes

+++++

- fixed some python2.5 compatibility issues 

- fixed issues with removed repos was accidentally added as groups, after

  full rescan of paths

- fixes #376 Cannot edit user (using container auth)

- fixes #378 Invalid image urls on changeset screen with proxy-prefix 

  configuration

- fixed initial sorting of repos inside repo group

- fixes issue when user tried to resubmit same permission into user/user_groups

- bumped beaker version that fixes #375 leap error bug

- fixed raw_changeset for git. It was generated with hg patch headers

- fixed vcs issue with last_changeset for filenodes

- fixed missing commit after hook delete

- fixed #372 issues with git operation detection that caused a security issue 

  for git repos

1.3.2 (**2012-02-28**)

----------------------

news

++++

fixes

+++++

- fixed git protocol issues with repos-groups

- fixed git remote repos validator that prevented from cloning remote git repos

- fixes #370 ending slashes fixes for repo and groups

- fixes #368 improved git-protocol detection to handle other clients

- fixes #366 When Setting Repository Group To Blank Repo Group Wont Be 

  Moved To Root

- fixes #371 fixed issues with beaker/sqlalchemy and non-ascii cache keys 

- fixed #373 missing cascade drop on user_group_to_perm table

1.3.1 (**2012-02-27**)

----------------------

news

++++

fixes

+++++

- redirection loop occurs when remember-me wasn't checked during login

- fixes issues with git blob history generation 

- don't fetch branch for git in file history dropdown. Causes unneeded slowness

1.3.0 (**2012-02-26**)

----------------------

news

++++

- code review, inspired by github code-comments 

- #215 rst and markdown README files support

- #252 Container-based and proxy pass-through authentication support

- #44 branch browser. Filtering of changelog by branches

- mercurial bookmarks support

- new hover top menu, optimized to add maximum size for important views

- configurable clone url template with possibility to specify  protocol like 

  ssh:// or http:// and also manually alter other parts of clone_url.

- enabled largefiles extension by default

- optimized summary file pages and saved a lot of unused space in them

- #239 option to manually mark repository as fork

- #320 mapping of commit authors to RhodeCode users

- #304 hashes are displayed using monospace font    

- diff configuration, toggle white lines and context lines

- #307 configurable diffs, whitespace toggle, increasing context lines

- sorting on branches, tags and bookmarks using YUI datatable

- improved file filter on files page

- implements #330 api method for listing nodes ar particular revision

- #73 added linking issues in commit messages to chosen issue tracker url

  based on user defined regular expression

- added linking of changesets in commit messages  

- new compact changelog with expandable commit messages

- firstname and lastname are optional in user creation

- #348 added post-create repository hook

- #212 global encoding settings is now configurable from .ini files 

- #227 added repository groups permissions

- markdown gets codehilite extensions

- new API methods, delete_repositories, grante/revoke permissions for groups 

  and repos

fixes

+++++

- rewrote dbsession management for atomic operations, and better error handling

- fixed sorting of repo tables

- #326 escape of special html entities in diffs

- normalized user_name => username in api attributes

- fixes #298 ldap created users with mixed case emails created conflicts 

  on saving a form

- fixes issue when owner of a repo couldn't revoke permissions for users 

  and groups

- fixes #271 rare JSON serialization problem with statistics

- fixes #337 missing validation check for conflicting names of a group with a

  repositories group

- #340 fixed session problem for mysql and celery tasks

- fixed #331 RhodeCode mangles repository names if the a repository group 

  contains the "full path" to the repositories

- #355 RhodeCode doesn't store encrypted LDAP passwords

1.2.5 (**2012-01-28**)

----------------------

news

# -*- coding: utf-8 -*-

"""

    rhodecode.__init__

    ~~~~~~~~~~~~~~~~~~

    RhodeCode, a web based repository management based on pylons

    versioning implementation: http://www.python.org/dev/peps/pep-0386/

    :created_on: Apr 9, 2010

    :author: marcink

    :copyright: (C) 2010-2012 Marcin Kuzminski <marcin@python-works.com>

    :license: GPLv3, see COPYING for more details.

"""

# This program is free software: you can redistribute it and/or modify

# it under the terms of the GNU General Public License as published by

# the Free Software Foundation, either version 3 of the License, or

# (at your option) any later version.

#

# This program is distributed in the hope that it will be useful,

# but WITHOUT ANY WARRANTY; without even the implied warranty of

# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the

# GNU General Public License for more details.

#

# You should have received a copy of the GNU General Public License

# along with this program.  If not, see <http://www.gnu.org/licenses/>.

import sys

import platform

VERSION = (1, 4, 0, 'b')

try:

    from rhodecode.lib import get_current_revision

    _rev = get_current_revision()

    if _rev and len(VERSION) > 3:

        VERSION += ('dev%s' % _rev[0],)

except ImportError:

    pass

__version__ = ('.'.join((str(each) for each in VERSION[:3])) +

               '.'.join(VERSION[3:]))

__dbversion__ = 5  # defines current db version for migrations

__platform__ = platform.system()

__license__ = 'GPLv3'

__py_version__ = sys.version_info

PLATFORM_WIN = ('Windows')

PLATFORM_OTHERS = ('Linux', 'Darwin', 'FreeBSD', 'OpenBSD', 'SunOS')

is_windows = __platform__ in PLATFORM_WIN

is_unix = __platform__ in PLATFORM_OTHERS

requirements = [

    "Pylons==1.0.0",

    "Beaker==1.6.3",

    "WebHelpers==1.3",

    "formencode==1.2.4",

    "SQLAlchemy==0.7.6",

    "Mako==0.7.0",

    "pygments>=1.4",

    "whoosh>=2.4.0,<2.5",

    "celery>=2.2.5,<2.3",

    "babel",

    "python-dateutil>=1.5.0,<2.0.0",

    "dulwich>=0.8.5,<0.9.0",

    "webob==1.0.8",

    "markdown==2.1.1",

    "docutils==0.8.1",

    "simplejson==2.5.2",

]

if __py_version__ < (2, 6):

    requirements.append("pysqlite")

if is_windows:

    requirements.append("mercurial>=2.2.1,<2.3")

else:

    requirements.append("py-bcrypt")

    requirements.append("mercurial>=2.2.1,<2.3")

def get_version():

    """Returns shorter version (digit parts only) as string."""

    return '.'.join((str(each) for each in VERSION[:3]))

BACKENDS = {

    'hg': 'Mercurial repository',

    'git': 'Git repository',

}

CELERY_ON = False

# link to config for pylons

CONFIG = {}

# Linked module for extensions

EXTENSIONS = {}

"""Pylons environment configuration"""

import os

import logging

import rhodecode

from mako.lookup import TemplateLookup

from pylons.configuration import PylonsConfig

from pylons.error import handle_mako_error

# don't remove this import it does magic for celery

from rhodecode.lib import celerypylons

import rhodecode.lib.app_globals as app_globals

from rhodecode.config.routing import make_map

from rhodecode.lib import helpers

from rhodecode.lib.auth import set_available_permissions

from rhodecode.lib.utils import repo2db_mapper, make_ui, set_rhodecode_config,\

    load_rcextensions

from rhodecode.lib.utils2 import engine_from_config, str2bool

from rhodecode.model import init_model

from rhodecode.model.scm import ScmModel

log = logging.getLogger(__name__)

def load_environment(global_conf, app_conf, initial=False):

    """

    Configure the Pylons environment via the ``pylons.config``

    object

    """

    config = PylonsConfig()

    # Pylons paths

    root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))

    paths = dict(

        root=root,

        controllers=os.path.join(root, 'controllers'),

        static_files=os.path.join(root, 'public'),

        templates=[os.path.join(root, 'templates')]

    )

    # Initialize config with the basic options

    config.init_app(global_conf, app_conf, package='rhodecode', paths=paths)

    # store some globals into rhodecode

    rhodecode.CELERY_ON = str2bool(config['app_conf'].get('use_celery'))

    config['routes.map'] = make_map(config)

    config['pylons.app_globals'] = app_globals.Globals(config)

    config['pylons.h'] = helpers

    rhodecode.CONFIG = config

    load_rcextensions(root_path=config['here'])

    # Setup cache object as early as possible

    import pylons

    pylons.cache._push_object(config['pylons.app_globals'].cache)

    # Create the Mako TemplateLookup, with the default auto-escaping

    config['pylons.app_globals'].mako_lookup = TemplateLookup(

        directories=paths['templates'],

        error_handler=handle_mako_error,

        module_directory=os.path.join(app_conf['cache_dir'], 'templates'),

        input_encoding='utf-8', default_filters=['escape'],

        imports=['from webhelpers.html import escape'])

    # sets the c attribute access when don't existing attribute are accessed

    config['pylons.strict_tmpl_context'] = True

    test = os.path.split(config['__file__'])[-1] == 'test.ini'

    if test:

        from rhodecode.lib.utils import create_test_env, create_test_index

        from rhodecode.tests import  TESTS_TMP_PATH

        create_test_env(TESTS_TMP_PATH, config)

        create_test_index(TESTS_TMP_PATH, config, True)

    # MULTIPLE DB configs

    # Setup the SQLAlchemy database engine

    sa_engine_db1 = engine_from_config(config, 'sqlalchemy.db1.')

    init_model(sa_engine_db1)

    repos_path = make_ui('db').configitems('paths')[0][1]

    repo2db_mapper(ScmModel().repo_scan(repos_path))

    set_available_permissions(config)

    config['base_path'] = repos_path

    set_rhodecode_config(config)

    # CONFIGURATION OPTIONS HERE (note: all config options will override

    # any Pylons config options)

    # store config reference into our module to skip import magic of

    # pylons

    rhodecode.CONFIG.update(config)

    return config

# -*- coding: utf-8 -*-

"""

    rhodecode.lib.celerylib.__init__

    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    celery libs for RhodeCode

    :created_on: Nov 27, 2010

    :author: marcink

    :copyright: (C) 2010-2012 Marcin Kuzminski <marcin@python-works.com>

    :license: GPLv3, see COPYING for more details.

"""

# This program is free software: you can redistribute it and/or modify

# it under the terms of the GNU General Public License as published by

# the Free Software Foundation, either version 3 of the License, or

# (at your option) any later version.

#

# This program is distributed in the hope that it will be useful,

# but WITHOUT ANY WARRANTY; without even the implied warranty of

# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the

# GNU General Public License for more details.

#

# You should have received a copy of the GNU General Public License

# along with this program.  If not, see <http://www.gnu.org/licenses/>.

import os

import sys

import socket

import traceback

import logging

from os.path import dirname as dn, join as jn

from pylons import config

from hashlib import md5

from decorator import decorator

from rhodecode.lib.vcs.utils.lazy import LazyProperty

from rhodecode.lib.utils2 import str2bool, safe_str

from rhodecode.lib.pidlock import DaemonLock, LockHeld

from rhodecode.model import init_model

from rhodecode.model import meta

from rhodecode.model.db import Statistics, Repository, User

from sqlalchemy import engine_from_config

from celery.messaging import establish_connection

log = logging.getLogger(__name__)

class ResultWrapper(object):

    def __init__(self, task):

        self.task = task

    @LazyProperty

    def result(self):

        return self.task

def run_task(task, *args, **kwargs):

    if CELERY_ON:

        try:

            t = task.apply_async(args=args, kwargs=kwargs)

            log.info('running task %s:%s' % (t.task_id, task))

            return t

        except socket.error, e:

            if isinstance(e, IOError) and e.errno == 111:

                log.debug('Unable to connect to celeryd. Sync execution')

            else:

                log.error(traceback.format_exc())

        except KeyError, e:

                log.debug('Unable to connect to celeryd. Sync execution')

        except Exception, e:

            log.error(traceback.format_exc())

    log.debug('executing task %s in sync mode' % task)

    return ResultWrapper(task(*args, **kwargs))

def __get_lockkey(func, *fargs, **fkwargs):

    params = list(fargs)

    params.extend(['%s-%s' % ar for ar in fkwargs.items()])

    func_name = str(func.__name__) if hasattr(func, '__name__') else str(func)

    lockkey = 'task_%s.lock' % \

        md5(func_name + '-' + '-'.join(map(safe_str, params))).hexdigest()

    return lockkey

def locked_task(func):

    def __wrapper(func, *fargs, **fkwargs):

        lockkey = __get_lockkey(func, *fargs, **fkwargs)

        lockkey_path = config['here']

        log.info('running task with lockkey %s' % lockkey)

        try:

            l = DaemonLock(file_=jn(lockkey_path, lockkey))

            ret = func(*fargs, **fkwargs)

            l.release()

            return ret

        except LockHeld:

            log.info('LockHeld')

            return 'Task with key %s already running' % lockkey

    return decorator(__wrapper, func)

def get_session():

    if CELERY_ON:

        engine = engine_from_config(config, 'sqlalchemy.db1.')

        init_model(engine)

    sa = meta.Session

    return sa

def dbsession(func):

    def __wrapper(func, *fargs, **fkwargs):

        try:

            ret = func(*fargs, **fkwargs)

            return ret

        finally:

                meta.Session.remove()

    return decorator(__wrapper, func)

# -*- coding: utf-8 -*-

"""

    rhodecode.lib.celerylib.tasks

    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    RhodeCode task modules, containing all task that suppose to be run

    by celery daemon

    :created_on: Oct 6, 2010

    :author: marcink

    :copyright: (C) 2010-2012 Marcin Kuzminski <marcin@python-works.com>

    :license: GPLv3, see COPYING for more details.

"""

# This program is free software: you can redistribute it and/or modify

# it under the terms of the GNU General Public License as published by

# the Free Software Foundation, either version 3 of the License, or

# (at your option) any later version.

#

# This program is distributed in the hope that it will be useful,

# but WITHOUT ANY WARRANTY; without even the implied warranty of

# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the

# GNU General Public License for more details.

#

# You should have received a copy of the GNU General Public License

# along with this program.  If not, see <http://www.gnu.org/licenses/>.

from celery.decorators import task

import os

import traceback

import logging

from os.path import join as jn

from time import mktime

from operator import itemgetter

from string import lower

from pylons import config, url

from pylons.i18n.translation import _

from rhodecode.lib.vcs import get_backend

from rhodecode.lib.utils2 import safe_str

from rhodecode.lib.celerylib import run_task, locked_task, dbsession, \

    str2bool, __get_lockkey, LockHeld, DaemonLock, get_session

from rhodecode.lib.helpers import person

from rhodecode.lib.rcmail.smtp_mailer import SmtpMailer

from rhodecode.lib.utils import add_cache, action_logger

from rhodecode.lib.compat import json, OrderedDict

from rhodecode.lib.hooks import log_create_repository

from rhodecode.model.db import Statistics, Repository, User

add_cache(config)

__all__ = ['whoosh_index', 'get_commits_stats',

           'reset_user_password', 'send_email']

def get_logger(cls):

    if CELERY_ON:

        try:

            log = cls.get_logger()

        except:

            log = logging.getLogger(__name__)

    else:

        log = logging.getLogger(__name__)

    return log

@task(ignore_result=True)

@locked_task

@dbsession

def whoosh_index(repo_location, full_index):

    from rhodecode.lib.indexers.daemon import WhooshIndexingDaemon

    log = whoosh_index.get_logger(whoosh_index)

    DBS = get_session()

    index_location = config['index_dir']

    WhooshIndexingDaemon(index_location=index_location,

                         repo_location=repo_location, sa=DBS)\

                         .run(full_index=full_index)

@task(ignore_result=True)

@dbsession

def get_commits_stats(repo_name, ts_min_y, ts_max_y):

    log = get_logger(get_commits_stats)

    DBS = get_session()

    lockkey = __get_lockkey('get_commits_stats', repo_name, ts_min_y,

                            ts_max_y)

    lockkey_path = config['here']

    log.info('running task with lockkey %s' % lockkey)

    try:

        lock = l = DaemonLock(file_=jn(lockkey_path, lockkey))

        # for js data compatibility cleans the key for person from '

        akc = lambda k: person(k).replace('"', "")

        co_day_auth_aggr = {}

        commits_by_day_aggregate = {}

        repo = Repository.get_by_repo_name(repo_name)

        if repo is None:

            return True

        repo = repo.scm_instance

        repo_size = repo.count()

        # return if repo have no revisions

        if repo_size < 1:

            lock.release()

            return True

        skip_date_limit = True

        parse_limit = int(config['app_conf'].get('commit_parse_limit'))

        last_rev = None

        last_cs = None

        timegetter = itemgetter('time')

        dbrepo = DBS.query(Repository)\

            .filter(Repository.repo_name == repo_name).scalar()

        cur_stats = DBS.query(Statistics)\

            .filter(Statistics.repository == dbrepo).scalar()

        if cur_stats is not None:

            last_rev = cur_stats.stat_on_revision

        if last_rev == repo.get_changeset().revision and repo_size > 1:

            # pass silently without any work if we're not on first revision or

            # current state of parsing revision(from db marker) is the

            # last revision

            lock.release()

            return True

        if cur_stats:

            commits_by_day_aggregate = OrderedDict(json.loads(

                                        cur_stats.commit_activity_combined))

            co_day_auth_aggr = json.loads(cur_stats.commit_activity)

        log.debug('starting parsing %s' % parse_limit)

        lmktime = mktime

        last_rev = last_rev + 1 if last_rev >= 0 else 0

        log.debug('Getting revisions from %s to %s' % (

             last_rev, last_rev + parse_limit)

        )

        for cs in repo[last_rev:last_rev + parse_limit]:

            log.debug('parsing %s' % cs)

            last_cs = cs  # remember last parsed changeset

            k = lmktime([cs.date.timetuple()[0], cs.date.timetuple()[1],

                          cs.date.timetuple()[2], 0, 0, 0, 0, 0, 0])

            if akc(cs.author) in co_day_auth_aggr:

                try:

                    l = [timegetter(x) for x in

                         co_day_auth_aggr[akc(cs.author)]['data']]

                    time_pos = l.index(k)

                except ValueError:

                    time_pos = False

                if time_pos >= 0 and time_pos is not False:

                    datadict = \

                        co_day_auth_aggr[akc(cs.author)]['data'][time_pos]

                    datadict["commits"] += 1

                    datadict["added"] += len(cs.added)

                    datadict["changed"] += len(cs.changed)

                    datadict["removed"] += len(cs.removed)

                else:

                    if k >= ts_min_y and k <= ts_max_y or skip_date_limit:

                        datadict = {"time": k,

                                    "commits": 1,

                                    "added": len(cs.added),

                                    "changed": len(cs.changed),

                                    "removed": len(cs.removed),

                                   }

                        co_day_auth_aggr[akc(cs.author)]['data']\

                            .append(datadict)

            else:

                if k >= ts_min_y and k <= ts_max_y or skip_date_limit:

                    co_day_auth_aggr[akc(cs.author)] = {

                                        "label": akc(cs.author),

                                        "data": [{"time":k,

                                                 "commits":1,

                                                 "added":len(cs.added),

                                                 "changed":len(cs.changed),

                                                 "removed":len(cs.removed),

                                                 }],

                                        "schema": ["commits"],

                                        }

            #gather all data by day

            if k in commits_by_day_aggregate:

                commits_by_day_aggregate[k] += 1

            else:

                commits_by_day_aggregate[k] = 1

        overview_data = sorted(commits_by_day_aggregate.items(),

                               key=itemgetter(0))

        if not co_day_auth_aggr:

            co_day_auth_aggr[akc(repo.contact)] = {

                "label": akc(repo.contact),

                "data": [0, 1],

                "schema": ["commits"],

            }

        stats = cur_stats if cur_stats else Statistics()

        stats.commit_activity = json.dumps(co_day_auth_aggr)

        stats.commit_activity_combined = json.dumps(overview_data)

        log.debug('last revison %s' % last_rev)

        leftovers = len(repo.revisions[last_rev:])

        log.debug('revisions to parse %s' % leftovers)

        if last_rev == 0 or leftovers < parse_limit:

            log.debug('getting code trending stats')

            stats.languages = json.dumps(__get_codes_stats(repo_name))

        try:

            stats.repository = dbrepo

            stats.stat_on_revision = last_cs.revision if last_cs else 0

            DBS.add(stats)

            DBS.commit()

        except:

            log.error(traceback.format_exc())

            DBS.rollback()