Setting up Whoosh full text search

----------------------------------

Starting from version 1.1 the whoosh index can be build by using the paster

command ``make-index``. To use ``make-index`` you must specify the configuration

file that stores the location of the index, and the location of the repositories

(`--repo-location`).Starting from version 1.2 it is 

also possible to specify a comma separated list of repositories (`--index-only`)

to build index only on chooses repositories skipping any other found in repos

location

You may optionally pass the option `-f` to enable a full index rebuild. Without

the `-f` option, indexing will run always in "incremental" mode.

For an incremental index build use::

	paster make-index production.ini --repo-location=<location for repos> 

For a full index rebuild use::

	paster make-index production.ini -f --repo-location=<location for repos>

building index just for chosen repositories is possible with such command::

 paster make-index production.ini --repo-location=<location for repos> --index-only=vcs,rhodecode

In order to do periodical index builds and keep your index always up to date.

It's recommended to do a crontab entry for incremental indexing. 

An example entry might look like this::

    /path/to/python/bin/paster /path/to/rhodecode/production.ini --repo-location=<location for repos> 

When using incremental mode (the default) whoosh will check the last

modification date of each file and add it to be reindexed if a newer file is

available. The indexing daemon checks for any removed files and removes them

from index.

If you want to rebuild index from scratch, you can use the `-f` flag as above,

or in the admin panel you can check `build from scratch` flag.

Setting up LDAP support

-----------------------

RhodeCode starting from version 1.1 supports ldap authentication. In order

    easy_install python-ldap

    pip install python-ldap

.. note::

   python-ldap requires some certain libs on your system, so before installing 

   it check that you have at least `openldap`, and `sasl` libraries.

LDAP settings are located in admin->ldap section,

Here's a typical ldap setup::

 Connection settings

 Enable LDAP          = checked

 Host                 = host.example.org

 Port                 = 389

 Account              = <account>

 Password             = <password>

 Certificate Checks   = DEMAND

 Search settings

 Base DN              = CN=users,DC=host,DC=example,DC=org

 LDAP Filter          = (&(objectClass=user)(!(objectClass=computer)))

 LDAP Search Scope    = SUBTREE

 Attribute mappings

 Login Attribute      = uid

 First Name Attribute = firstName

 Last Name Attribute  = lastName

 E-mail Attribute     = mail

.. _enable_ldap:

Enable LDAP : required

    Whether to use LDAP for authenticating users.

.. _ldap_host:

Host : required

    LDAP server hostname or IP address.

.. _Port:

Port : required

    389 for un-encrypted LDAP, 636 for SSL-encrypted LDAP.

.. _ldap_account:

Account : optional

    Only required if the LDAP server does not allow anonymous browsing of

    records.  This should be a special account for record browsing.  This

    will require `LDAP Password`_ below.

.. _LDAP Password:

Password : optional

    Only required if the LDAP server does not allow anonymous browsing of

    records.

.. _Enable LDAPS:

.. _Certificate Checks:

Certificate Checks : optional

    How SSL certificates verification is handled - this is only useful when

    `Enable LDAPS`_ is enabled.  Only DEMAND or HARD offer full SSL security while

    the other options are susceptible to man-in-the-middle attacks.  SSL

    certificates can be installed to /etc/openldap/cacerts so that the

    DEMAND or HARD options can be used with self-signed certificates or

    certificates that do not have traceable certificates of authority.

    NEVER

        A serve certificate will never be requested or checked.

    ALLOW

        A server certificate is requested.  Failure to provide a

        certificate or providing a bad certificate will not terminate the

        session.

    TRY

        A server certificate is requested.  Failure to provide a

        certificate does not halt the session; providing a bad certificate

        halts the session.

    DEMAND

        A server certificate is requested and must be provided and

        authenticated for the session to proceed.

    HARD

        The same as DEMAND.

.. _Base DN:

Base DN : required

    The Distinguished Name (DN) where searches for users will be performed.

    Searches can be controlled by `LDAP Filter`_ and `LDAP Search Scope`_.

.. _LDAP Filter:

LDAP Filter : optional

    A LDAP filter defined by RFC 2254.  This is more useful when `LDAP

    Search Scope`_ is set to SUBTREE.  The filter is useful for limiting

    which LDAP objects are identified as representing Users for

    authentication.  The filter is augmented by `Login Attribute`_ below.

    This can commonly be left blank.

.. _LDAP Search Scope:

# -*- coding: utf-8 -*-

"""

    rhodecode.controllers.admin.ldap_settings

    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    ldap controller for RhodeCode

    :created_on: Nov 26, 2010

    :author: marcink

    :copyright: (C) 2009-2011 Marcin Kuzminski <marcin@python-works.com>

    :license: GPLv3, see COPYING for more details.

"""

# This program is free software: you can redistribute it and/or modify

# it under the terms of the GNU General Public License as published by

# the Free Software Foundation, either version 3 of the License, or

# (at your option) any later version.

#

# This program is distributed in the hope that it will be useful,

# but WITHOUT ANY WARRANTY; without even the implied warranty of

# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the

# GNU General Public License for more details.

#

# You should have received a copy of the GNU General Public License

# along with this program.  If not, see <http://www.gnu.org/licenses/>.

import logging

import formencode

import traceback

from formencode import htmlfill

from pylons import request, response, session, tmpl_context as c, url

from pylons.controllers.util import abort, redirect

from pylons.i18n.translation import _

from rhodecode.lib.base import BaseController, render

from rhodecode.lib import helpers as h

from rhodecode.lib.auth import LoginRequired, HasPermissionAllDecorator

from rhodecode.model.forms import LdapSettingsForm

log = logging.getLogger(__name__)

class LdapSettingsController(BaseController):

    search_scope_choices = [('BASE', _('BASE'),),

                            ('ONELEVEL', _('ONELEVEL'),),

                            ('SUBTREE', _('SUBTREE'),),

                            ]

    search_scope_default = 'SUBTREE'

    tls_reqcert_choices = [('NEVER', _('NEVER'),),

                           ('ALLOW', _('ALLOW'),),

                           ('TRY', _('TRY'),),

                           ('DEMAND', _('DEMAND'),),

                           ('HARD', _('HARD'),),

                           ]

    tls_reqcert_default = 'DEMAND'

    tls_kind_choices = [('PLAIN', _('No encryption'),),

                        ('LDAPS', _('LDAPS connection'),),

                        ('START_TLS', _('START_TLS on LDAP connection'),)

                        ]

    tls_kind_default = 'PLAIN'

    @LoginRequired()

    @HasPermissionAllDecorator('hg.admin')

    def __before__(self):

        c.admin_user = session.get('admin_user')

        c.admin_username = session.get('admin_username')

        c.search_scope_choices = self.search_scope_choices

        c.tls_reqcert_choices = self.tls_reqcert_choices

        c.tls_kind_choices = self.tls_kind_choices

        super(LdapSettingsController, self).__before__()

    def index(self):

        c.search_scope_cur = defaults.get('ldap_search_scope')

        c.tls_reqcert_cur = defaults.get('ldap_tls_reqcert')

        c.tls_kind_cur = defaults.get('ldap_tls_kind')

        return htmlfill.render(

                    render('admin/ldap/ldap.html'),

                    defaults=defaults,

                    encoding="UTF-8",

                    force_defaults=True,)

    def ldap_settings(self):

        """POST ldap create and store ldap settings"""

        _form = LdapSettingsForm([x[0] for x in self.tls_reqcert_choices],

                                 [x[0] for x in self.search_scope_choices],

                                 [x[0] for x in self.tls_kind_choices])()

        try:

            form_result = _form.to_python(dict(request.POST))

            try:

                for k, v in form_result.items():

                    if k.startswith('ldap_'):

                        setting.app_settings_value = v

                        self.sa.add(setting)

                self.sa.commit()

                h.flash(_('Ldap settings updated successfully'),

                    category='success')

            except (DatabaseError,):

                raise

        except LdapImportError:

            h.flash(_('Unable to activate ldap. The "python-ldap" library '

                      'is missing.'), category='warning')

        except formencode.Invalid, errors:

            return htmlfill.render(

                render('admin/ldap/ldap.html'),

                defaults=errors.value,

                prefix_error=False,

                encoding="UTF-8")

        except Exception:

            log.error(traceback.format_exc())

            h.flash(_('error occurred during update of ldap settings'),

                    category='error')

        return redirect(url('ldap_home'))

# -*- coding: utf-8 -*-

"""

    rhodecode.controllers.admin.settings

    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    settings controller for rhodecode admin

    :created_on: Jul 14, 2010

    :author: marcink

    :copyright: (C) 2009-2011 Marcin Kuzminski <marcin@python-works.com>

    :license: GPLv3, see COPYING for more details.

"""

# This program is free software: you can redistribute it and/or modify

# it under the terms of the GNU General Public License as published by

# the Free Software Foundation, either version 3 of the License, or

# (at your option) any later version.

#

# This program is distributed in the hope that it will be useful,

# but WITHOUT ANY WARRANTY; without even the implied warranty of

# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the

# GNU General Public License for more details.

#

# You should have received a copy of the GNU General Public License

# along with this program.  If not, see <http://www.gnu.org/licenses/>.

import logging

import traceback

import formencode

from sqlalchemy import func

from formencode import htmlfill

from pylons import request, session, tmpl_context as c, url, config

from pylons.controllers.util import abort, redirect

from pylons.i18n.translation import _

from rhodecode.lib import helpers as h

from rhodecode.lib.auth import LoginRequired, HasPermissionAllDecorator, \

    HasPermissionAnyDecorator, NotAnonymous

from rhodecode.lib.base import BaseController, render

from rhodecode.lib.celerylib import tasks, run_task

from rhodecode.lib.utils import repo2db_mapper, invalidate_cache, \

    set_rhodecode_config, repo_name_slug

from rhodecode.model.forms import UserForm, ApplicationSettingsForm, \

    ApplicationUiSettingsForm

from rhodecode.model.scm import ScmModel

from rhodecode.model.user import UserModel

log = logging.getLogger(__name__)

class SettingsController(BaseController):

    """REST Controller styled on the Atom Publishing Protocol"""

    # To properly map this controller, ensure your config/routing.py

    # file has a resource setup:

    #     map.resource('setting', 'settings', controller='admin/settings',

    #         path_prefix='/admin', name_prefix='admin_')

    @LoginRequired()

    def __before__(self):

        c.admin_user = session.get('admin_user')

        c.admin_username = session.get('admin_username')

        super(SettingsController, self).__before__()

    @HasPermissionAllDecorator('hg.admin')

    def index(self, format='html'):

        """GET /admin/settings: All items in the collection"""

        # url('admin_settings')

        defaults.update(self.get_hg_ui_settings())

        return htmlfill.render(

            render('admin/settings/settings.html'),

            defaults=defaults,

            encoding="UTF-8",

            force_defaults=False

        )

    @HasPermissionAllDecorator('hg.admin')

    def create(self):

        """POST /admin/settings: Create a new item"""

        # url('admin_settings')

    @HasPermissionAllDecorator('hg.admin')

    def new(self, format='html'):

        """GET /admin/settings/new: Form to create a new item"""

        # url('admin_new_setting')

    @HasPermissionAllDecorator('hg.admin')

    def update(self, setting_id):

        """PUT /admin/settings/setting_id: Update an existing item"""

        # Forms posted to this method should contain a hidden field:

        #    <input type="hidden" name="_method" value="PUT" />

        # Or using helpers:

        #    h.form(url('admin_setting', setting_id=ID),

        #           method='put')

        # url('admin_setting', setting_id=ID)

        if setting_id == 'mapping':

            rm_obsolete = request.POST.get('destroy', False)

            log.debug('Rescanning directories with destroy=%s', rm_obsolete)

            initial = ScmModel().repo_scan()

            log.debug('invalidating all repositories')

            for repo_name in initial.keys():

                invalidate_cache('get_repo_cached_%s' % repo_name)

            added, removed = repo2db_mapper(initial, rm_obsolete)

            h.flash(_('Repositories successfully'

                      ' rescanned added: %s,removed: %s') % (added, removed),

                    category='success')

        if setting_id == 'whoosh':

            repo_location = self.get_hg_ui_settings()['paths_root_path']

            full_index = request.POST.get('full_index', False)

            run_task(tasks.whoosh_index, repo_location, full_index)

            h.flash(_('Whoosh reindex task scheduled'), category='success')

        if setting_id == 'global':

            application_form = ApplicationSettingsForm()()

            try:

                form_result = application_form.to_python(dict(request.POST))

                try:

                    hgsettings1.app_settings_value = \

                        form_result['rhodecode_title']

                    hgsettings2.app_settings_value = \

                        form_result['rhodecode_realm']

                    hgsettings3.app_settings_value = \

                        form_result['rhodecode_ga_code']

                    self.sa.add(hgsettings1)

                    self.sa.add(hgsettings2)

                    self.sa.add(hgsettings3)

                    self.sa.commit()

                    set_rhodecode_config(config)

                    h.flash(_('Updated application settings'),

                            category='success')

                except Exception:

                    log.error(traceback.format_exc())

                    h.flash(_('error occurred during updating '

                              'application settings'),

                            category='error')

                    self.sa.rollback()

            except formencode.Invalid, errors:

                return htmlfill.render(

                     render('admin/settings/settings.html'),

                     defaults=errors.value,

                     errors=errors.error_dict or {},

                     prefix_error=False,

                     encoding="UTF-8")

        if setting_id == 'mercurial':

            application_form = ApplicationUiSettingsForm()()

            try:

                form_result = application_form.to_python(dict(request.POST))

                try:

                    hgsettings1 = self.sa.query(RhodeCodeUi)\

                    .filter(RhodeCodeUi.ui_key == 'push_ssl').one()

                    hgsettings1.ui_value = form_result['web_push_ssl']

                    hgsettings2 = self.sa.query(RhodeCodeUi)\

                    .filter(RhodeCodeUi.ui_key == '/').one()

                    hgsettings2.ui_value = form_result['paths_root_path']

                    #HOOKS

                    hgsettings3 = self.sa.query(RhodeCodeUi)\

                    .filter(RhodeCodeUi.ui_key == 'changegroup.update').one()

                    hgsettings3.ui_active = \

                        bool(form_result['hooks_changegroup_update'])

    rhodecode.lib.auth

    ~~~~~~~~~~~~~~~~~~

    authentication and permission libraries

    :created_on: Apr 4, 2010

    :copyright: (c) 2010 by marcink.

    :license: LICENSE_NAME, see LICENSE_FILE for more details.

"""

# This program is free software: you can redistribute it and/or modify

# it under the terms of the GNU General Public License as published by

# the Free Software Foundation, either version 3 of the License, or

# (at your option) any later version.

#

# This program is distributed in the hope that it will be useful,

# but WITHOUT ANY WARRANTY; without even the implied warranty of

# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the

# GNU General Public License for more details.

#

# You should have received a copy of the GNU General Public License

# along with this program.  If not, see <http://www.gnu.org/licenses/>.

import random

import logging

import traceback

import hashlib

from tempfile import _RandomNameSequence

from decorator import decorator

from pylons import config, session, url, request

from pylons.controllers.util import abort, redirect

from pylons.i18n.translation import _

from rhodecode import __platform__, PLATFORM_WIN, PLATFORM_OTHERS

if __platform__ in PLATFORM_WIN:

    from hashlib import sha256

if __platform__ in PLATFORM_OTHERS:

    import bcrypt

from rhodecode.lib import str2bool

from rhodecode.lib.exceptions import LdapPasswordError, LdapUsernameError

from rhodecode.lib.utils import get_repo_slug

from rhodecode.lib.auth_ldap import AuthLdap

from rhodecode.model import meta

from rhodecode.model.user import UserModel

log = logging.getLogger(__name__)

class PasswordGenerator(object):

    """This is a simple class for generating password from

        different sets of characters

        usage:

        passwd_gen = PasswordGenerator()

        #print 8-letter password containing only big and small letters

            of alphabet

        print passwd_gen.gen_password(8, passwd_gen.ALPHABETS_BIG_SMALL)

    """

    ALPHABETS_NUM = r'''1234567890'''

    ALPHABETS_SMALL = r'''qwertyuiopasdfghjklzxcvbnm'''

    ALPHABETS_BIG = r'''QWERTYUIOPASDFGHJKLZXCVBNM'''

    ALPHABETS_SPECIAL = r'''`-=[]\;',./~!@#$%^&*()_+{}|:"<>?'''

    ALPHABETS_FULL = ALPHABETS_BIG + ALPHABETS_SMALL \

        + ALPHABETS_NUM + ALPHABETS_SPECIAL

    ALPHABETS_ALPHANUM = ALPHABETS_BIG + ALPHABETS_SMALL + ALPHABETS_NUM

    ALPHABETS_BIG_SMALL = ALPHABETS_BIG + ALPHABETS_SMALL

    ALPHABETS_ALPHANUM_BIG = ALPHABETS_BIG + ALPHABETS_NUM

    ALPHABETS_ALPHANUM_SMALL = ALPHABETS_SMALL + ALPHABETS_NUM

    def __init__(self, passwd=''):

        self.passwd = passwd

    def gen_password(self, len, type):

        self.passwd = ''.join([random.choice(type) for _ in xrange(len)])

        return self.passwd

class RhodeCodeCrypto(object):

    @classmethod

    def hash_string(cls, str_):

        """

        Cryptographic function used for password hashing based on pybcrypt

        or pycrypto in windows

        :param password: password to hash

        """

        if __platform__ in PLATFORM_WIN:

            return sha256(str_).hexdigest()

        elif __platform__ in PLATFORM_OTHERS:

            return bcrypt.hashpw(str_, bcrypt.gensalt(10))

        else:

            raise Exception('Unknown or unsupported platform %s' \

        """

        Checks matching password with it's hashed value, runs different

        implementation based on platform it runs on

        :param password: password

        :param hashed: password in hashed form

        """

        if __platform__ in PLATFORM_WIN:

            return sha256(password).hexdigest() == hashed

        elif __platform__ in PLATFORM_OTHERS:

            return bcrypt.hashpw(password, hashed) == hashed

        else:

            raise Exception('Unknown or unsupported platform %s' \

                            % __platform__)

def get_crypt_password(password):

    return RhodeCodeCrypto.hash_string(password)

def check_password(password, hashed):

    return RhodeCodeCrypto.hash_check(password, hashed)

def generate_api_key(username, salt=None):

    if salt is None:

        salt = _RandomNameSequence().next()

    return hashlib.sha1(username + salt).hexdigest()

def authfunc(environ, username, password):

    """Dummy authentication function used in Mercurial/Git/ and access control,

    :param environ: needed only for using in Basic auth

    """

    return authenticate(username, password)

def authenticate(username, password):

    """Authentication function used for access control,

    firstly checks for db authentication then if ldap is enabled for ldap

    authentication, also creates ldap user if not in database

    :param username: username

    :param password: password

    """

    user_model = UserModel()

    user = user_model.get_by_username(username, cache=False)

    log.debug('Authenticating user using RhodeCode account')

    if user is not None and not user.ldap_dn:

        if user.active:

            if user.username == 'default' and user.active:

                log.info('user %s authenticated correctly as anonymous user',

                         username)

                return True

            elif user.username == username and check_password(password,

                                                              user.password):

                log.info('user %s authenticated correctly', username)

                return True

        else:

            log.warning('user %s is disabled', username)

    else:

        log.debug('Regular authentication failed')

        user_obj = user_model.get_by_username(username, cache=False,

                                            case_insensitive=True)

        if user_obj is not None and not user_obj.ldap_dn:

            log.debug('this user already exists as non ldap')

            return False

        #======================================================================

        # FALLBACK TO LDAP AUTH IF ENABLE

        #======================================================================

        if str2bool(ldap_settings.get('ldap_active')):

            log.debug("Authenticating user using ldap")

            kwargs = {

                  'server': ldap_settings.get('ldap_host', ''),

                  'base_dn': ldap_settings.get('ldap_base_dn', ''),

                  'port': ldap_settings.get('ldap_port'),

                  'bind_dn': ldap_settings.get('ldap_dn_user'),

                  'bind_pass': ldap_settings.get('ldap_dn_pass'),

                  'tls_kind': ldap_settings.get('ldap_tls_kind'),

                  'tls_reqcert': ldap_settings.get('ldap_tls_reqcert'),

                  'ldap_filter': ldap_settings.get('ldap_filter'),

                  'search_scope': ldap_settings.get('ldap_search_scope'),

                  'attr_login': ldap_settings.get('ldap_attr_login'),

                  'ldap_version': 3,

                  }

            log.debug('Checking for ldap authentication')

            try:

                aldap = AuthLdap(**kwargs)

                (user_dn, ldap_attrs) = aldap.authenticate_ldap(username,

                                                                password)

                log.debug('Got ldap DN response %s', user_dn)

                user_attrs = {

                    }

                if user_model.create_ldap(username, password, user_dn,

                                          user_attrs):

                    log.info('created new ldap user %s', username)

                return True

            except (LdapUsernameError, LdapPasswordError,):

                pass

            except (Exception,):

                log.error(traceback.format_exc())

                pass

    return False

class  AuthUser(object):

    """

    A simple object that handles all attributes of user in RhodeCode

    It does lookup based on API key,given user, or user present in session

    Then it fills all required information for such user. It also checks if

    anonymous access is enabled and if so, it returns default user as logged

    in

    """

    def __init__(self, user_id=None, api_key=None):

        self.user_id = user_id

        self.api_key = None

        self.username = 'None'

        self.name = ''

        self.lastname = ''

        self.email = ''

        self.is_authenticated = False

        self.admin = False

        self.permissions = {}

        self._api_key = api_key

        self.propagate_data()

    def propagate_data(self):

        user_model = UserModel()

        self.anonymous_user = user_model.get_by_username('default', cache=True)

        if self._api_key and self._api_key != self.anonymous_user.api_key:

            #try go get user by api key

            log.debug('Auth User lookup by API KEY %s', self._api_key)

            user_model.fill_data(self, api_key=self._api_key)

        else:

# This program is free software: you can redistribute it and/or modify

# it under the terms of the GNU General Public License as published by

# the Free Software Foundation, either version 3 of the License, or

# (at your option) any later version.

#

# This program is distributed in the hope that it will be useful,

# but WITHOUT ANY WARRANTY; without even the implied warranty of

# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the

# GNU General Public License for more details.

#

# You should have received a copy of the GNU General Public License

# along with this program.  If not, see <http://www.gnu.org/licenses/>.

import logging

log = logging.getLogger(__name__)

try:

    import ldap

except ImportError:

    pass

class AuthLdap(object):

    def __init__(self, server, base_dn, port=389, bind_dn='', bind_pass='',

                 tls_kind = 'PLAIN', tls_reqcert='DEMAND', ldap_version=3,

                 ldap_filter='(&(objectClass=user)(!(objectClass=computer)))',

                 search_scope='SUBTREE',

                 attr_login='uid'):

        self.ldap_version = ldap_version

        ldap_server_type = 'ldap'