.. _setup:

Setup

=====

Setting up RhodeCode

--------------------------

First, you will need to create a RhodeCode configuration file. Run the following

command to do this::

    paster make-config RhodeCode production.ini

- This will create the file `production.ini` in the current directory. This

  configuration file contains the various settings for RhodeCode, e.g proxy port,

  email settings, usage of static files, cache, celery settings and logging.

Next, you need to create the databases used by RhodeCode. I recommend that you

use sqlite (default) or postgresql. If you choose a database other than the

default ensure you properly adjust the db url in your production.ini

configuration file to use this other database. Create the databases by running

the following command::

    paster setup-app production.ini

This will prompt you for a "root" path. This "root" path is the location where

RhodeCode will store all of its repositories on the current machine. After

entering this "root" path ``setup-app`` will also prompt you for a username and password

for the initial admin account which ``setup-app`` sets up for you.

- The ``setup-app`` command will create all of the needed tables and an admin

  account. When choosing a root path you can either use a new empty location, or a

  location which already contains existing repositories. If you choose a location

  which contains existing repositories RhodeCode will simply add all of the

  repositories at the chosen location to it's database. (Note: make sure you

  specify the correct path to the root).

- Note: the given path for mercurial_ repositories **must** be write accessible

  for the application. It's very important since the RhodeCode web interface will

  work without write access, but when trying to do a push it will eventually fail

  with permission denied errors unless it has write access.

You are now ready to use RhodeCode, to run it simply execute::

    paster serve production.ini

- This command runs the RhodeCode server. The web app should be available at the 

  127.0.0.1:5000. This ip and port is configurable via the production.ini 

  file created in previous step

- Use the admin account you created above when running ``setup-app`` to login to the web app.

- The default permissions on each repository is read, and the owner is admin. 

  Remember to update these if needed.

- In the admin panel you can toggle ldap, anonymous, permissions settings. As

  well as edit more advanced options on users and repositories

Try copying your own mercurial repository into the "root" directory you are

using, then from within the RhodeCode web application choose Admin >

repositories. Then choose Add New Repository. Add the repository you copied into

the root. Test that you can browse your repository from within RhodeCode and then

try cloning your repository from RhodeCode with::

    hg clone http://127.0.0.1:5000/<repository name>

where *repository name* is replaced by the name of your repository.

Using RhodeCode with SSH

------------------------

RhodeCode currently only hosts repositories using http and https. (The addition of

ssh hosting is a planned future feature.) However you can easily use ssh in

parallel with RhodeCode. (Repository access via ssh is a standard "out of

the box" feature of mercurial_ and you can use this to access any of the

repositories that RhodeCode is hosting. See PublishingRepositories_)

RhodeCode repository structures are kept in directories with the same name 

as the project. When using repository groups, each group is a subdirectory.

This allows you to easily use ssh for accessing repositories.

In order to use ssh you need to make sure that your web-server and the users login

accounts have the correct permissions set on the appropriate directories. (Note

that these permissions are independent of any permissions you have set up using

the RhodeCode web interface.)

If your main directory (the same as set in RhodeCode settings) is for example

set to **/home/hg** and the repository you are using is named `rhodecode`, then

to clone via ssh you should run::

    hg clone ssh://user@server.com/home/hg/rhodecode

Using other external tools such as mercurial-server_ or using ssh key based

authentication is fully supported.

Note: In an advanced setup, in order for your ssh access to use the same

permissions as set up via the RhodeCode web interface, you can create an

authentication hook to connect to the rhodecode db and runs check functions for

permissions against that.

Setting up Whoosh full text search

----------------------------------

Starting from version 1.1 the whoosh index can be build by using the paster

command ``make-index``. To use ``make-index`` you must specify the configuration

file that stores the location of the index, and the location of the repositories

(`--repo-location`).Starting from version 1.2 it is 

also possible to specify a comma separated list of repositories (`--index-only`)

to build index only on chooses repositories skipping any other found in repos

location

You may optionally pass the option `-f` to enable a full index rebuild. Without

the `-f` option, indexing will run always in "incremental" mode.

For an incremental index build use::

	paster make-index production.ini --repo-location=<location for repos> 

For a full index rebuild use::

	paster make-index production.ini -f --repo-location=<location for repos>

building index just for chosen repositories is possible with such command::

 paster make-index production.ini --repo-location=<location for repos> --index-only=vcs,rhodecode

In order to do periodical index builds and keep your index always up to date.

It's recommended to do a crontab entry for incremental indexing. 

An example entry might look like this::

    /path/to/python/bin/paster /path/to/rhodecode/production.ini --repo-location=<location for repos> 

When using incremental mode (the default) whoosh will check the last

modification date of each file and add it to be reindexed if a newer file is

available. The indexing daemon checks for any removed files and removes them

from index.

If you want to rebuild index from scratch, you can use the `-f` flag as above,

or in the admin panel you can check `build from scratch` flag.

Setting up LDAP support

-----------------------

RhodeCode starting from version 1.1 supports ldap authentication. In order

    easy_install python-ldap

    pip install python-ldap

.. note::

   python-ldap requires some certain libs on your system, so before installing 

   it check that you have at least `openldap`, and `sasl` libraries.

LDAP settings are located in admin->ldap section,

Here's a typical ldap setup::

 Connection settings

 Enable LDAP          = checked

 Host                 = host.example.org

 Port                 = 389

 Account              = <account>

 Password             = <password>

 Certificate Checks   = DEMAND

 Search settings

 Base DN              = CN=users,DC=host,DC=example,DC=org

 LDAP Filter          = (&(objectClass=user)(!(objectClass=computer)))

 LDAP Search Scope    = SUBTREE

 Attribute mappings

 Login Attribute      = uid

 First Name Attribute = firstName

 Last Name Attribute  = lastName

 E-mail Attribute     = mail

.. _enable_ldap:

Enable LDAP : required

    Whether to use LDAP for authenticating users.

.. _ldap_host:

Host : required

    LDAP server hostname or IP address.

.. _Port:

Port : required

    389 for un-encrypted LDAP, 636 for SSL-encrypted LDAP.

.. _ldap_account:

Account : optional

    Only required if the LDAP server does not allow anonymous browsing of

    records.  This should be a special account for record browsing.  This

    will require `LDAP Password`_ below.

.. _LDAP Password:

Password : optional

    Only required if the LDAP server does not allow anonymous browsing of

    records.

.. _Enable LDAPS:

.. _Certificate Checks:

Certificate Checks : optional

    How SSL certificates verification is handled - this is only useful when

    `Enable LDAPS`_ is enabled.  Only DEMAND or HARD offer full SSL security while

    the other options are susceptible to man-in-the-middle attacks.  SSL

    certificates can be installed to /etc/openldap/cacerts so that the

    DEMAND or HARD options can be used with self-signed certificates or

    certificates that do not have traceable certificates of authority.

    NEVER

        A serve certificate will never be requested or checked.

    ALLOW

        A server certificate is requested.  Failure to provide a

        certificate or providing a bad certificate will not terminate the

        session.

    TRY

        A server certificate is requested.  Failure to provide a

        certificate does not halt the session; providing a bad certificate

        halts the session.

    DEMAND

        A server certificate is requested and must be provided and

        authenticated for the session to proceed.

    HARD

        The same as DEMAND.

.. _Base DN:

Base DN : required

    The Distinguished Name (DN) where searches for users will be performed.

    Searches can be controlled by `LDAP Filter`_ and `LDAP Search Scope`_.

.. _LDAP Filter:

LDAP Filter : optional

    A LDAP filter defined by RFC 2254.  This is more useful when `LDAP

    Search Scope`_ is set to SUBTREE.  The filter is useful for limiting

    which LDAP objects are identified as representing Users for

    authentication.  The filter is augmented by `Login Attribute`_ below.

    This can commonly be left blank.

.. _LDAP Search Scope:

LDAP Search Scope : required

    This limits how far LDAP will search for a matching object.

    BASE

        Only allows searching of `Base DN`_ and is usually not what you

        want.

    ONELEVEL

        Searches all entries under `Base DN`_, but not Base DN itself.

    SUBTREE

        Searches all entries below `Base DN`_, but not Base DN itself.

        When using SUBTREE `LDAP Filter`_ is useful to limit object

        location.

.. _Login Attribute:

Login Attribute : required        

    The LDAP record attribute that will be matched as the USERNAME or

    ACCOUNT used to connect to RhodeCode.  This will be added to `LDAP

    Filter`_ for locating the User object.  If `LDAP Filter`_ is specified as

    "LDAPFILTER", `Login Attribute`_ is specified as "uid" and the user has

    connected as "jsmith" then the `LDAP Filter`_ will be augmented as below

    ::

        (&(LDAPFILTER)(uid=jsmith))

.. _ldap_attr_firstname:

First Name Attribute : required

    The LDAP record attribute which represents the user's first name.

.. _ldap_attr_lastname:

Last Name Attribute : required

    The LDAP record attribute which represents the user's last name.

.. _ldap_attr_email:

Email Attribute : required

    The LDAP record attribute which represents the user's email address.

If all data are entered correctly, and python-ldap_ is properly installed

users should be granted access to RhodeCode with ldap accounts.  At this

time user information is copied from LDAP into the RhodeCode user database.

This means that updates of an LDAP user object may not be reflected as a

user update in RhodeCode.

If You have problems with LDAP access and believe You entered correct

information check out the RhodeCode logs, any error messages sent from LDAP

will be saved there.

Active Directory

''''''''''''''''

RhodeCode can use Microsoft Active Directory for user authentication.  This

is done through an LDAP or LDAPS connection to Active Directory.  The

following LDAP configuration settings are typical for using Active

Directory ::

 Base DN              = OU=SBSUsers,OU=Users,OU=MyBusiness,DC=v3sys,DC=local

 Login Attribute      = sAMAccountName

 First Name Attribute = givenName

 Last Name Attribute  = sn

 E-mail Attribute     = mail

All other LDAP settings will likely be site-specific and should be

appropriately configured.

Setting Up Celery

-----------------

Since version 1.1 celery is configured by the rhodecode ini configuration files.

Simply set use_celery=true in the ini file then add / change the configuration 

variables inside the ini file.

Remember that the ini files use the format with '.' not with '_' like celery.

So for example setting `BROKER_HOST` in celery means setting `broker.host` in

the config file.

In order to start using celery run::

 paster celeryd <configfile.ini>

.. note::

   Make sure you run this command from the same virtualenv, and with the same user

   that rhodecode runs.

HTTPS support

-------------

There are two ways to enable https:

- Set HTTP_X_URL_SCHEME in your http server headers, than rhodecode will

  recognize this headers and make proper https redirections

- Alternatively, set `force_https = true` in the ini configuration to force using

  https, no headers are needed than to enable https

Nginx virtual host example

--------------------------

Sample config for nginx using proxy::

    server {

       listen          80;

       server_name     hg.myserver.com;

       access_log      /var/log/nginx/rhodecode.access.log;

       error_log       /var/log/nginx/rhodecode.error.log;

       location / {

               root /var/www/rhodecode/rhodecode/public/;

               if (!-f $request_filename){

                   proxy_pass      http://127.0.0.1:5000;

               }

               #this is important if you want to use https !!!

               proxy_set_header X-Url-Scheme $scheme;

               include         /etc/nginx/proxy.conf;  

       }

    }  

Here's the proxy.conf. It's tuned so it will not timeout on long

pushes or large pushes::

    proxy_redirect              off;

    proxy_set_header            Host $host;

    proxy_set_header            X-Host $http_host;

    proxy_set_header            X-Real-IP $remote_addr;

    proxy_set_header            X-Forwarded-For $proxy_add_x_forwarded_for;

    proxy_set_header            Proxy-host $proxy_host;

    client_max_body_size        400m;

    client_body_buffer_size     128k;

    proxy_buffering             off;

    proxy_connect_timeout       3600;

    proxy_send_timeout          3600;

    proxy_read_timeout          3600;

    proxy_buffer_size           16k;

    proxy_buffers               4 16k;

    proxy_busy_buffers_size     64k;

    proxy_temp_file_write_size  64k;

Also, when using root path with nginx you might set the static files to false

in the production.ini file::

    [app:main]

      use = egg:rhodecode

      full_stack = true

      static_files = false

      lang=en

      cache_dir = %(here)s/data

In order to not have the statics served by the application. This improves speed.

Apache virtual host example

---------------------------

Here is a sample configuration file for apache using proxy::

    <VirtualHost *:80>

            ServerName hg.myserver.com

            ServerAlias hg.myserver.com

            <Proxy *>

              Order allow,deny

              Allow from all

            </Proxy>

            #important !

            #Directive to properly generate url (clone url) for pylons

            ProxyPreserveHost On

            #rhodecode instance

            ProxyPass / http://127.0.0.1:5000/

            ProxyPassReverse / http://127.0.0.1:5000/

            #to enable https use line below

            #SetEnvIf X-Url-Scheme https HTTPS=1

    </VirtualHost> 

Additional tutorial

http://wiki.pylonshq.com/display/pylonscookbook/Apache+as+a+reverse+proxy+for+Pylons

Apache as subdirectory

----------------------

Apache subdirectory part::

    <Location /<someprefix> >

      ProxyPass http://127.0.0.1:5000/<someprefix>

      ProxyPassReverse http://127.0.0.1:5000/<someprefix>

      SetEnvIf X-Url-Scheme https HTTPS=1

    </Location> 

Besides the regular apache setup you will need to add the following to your .ini file::

    filter-with = proxy-prefix

Add the following at the end of the .ini file::

    [filter:proxy-prefix]

    use = egg:PasteDeploy#prefix

    prefix = /<someprefix> 

then change <someprefix> into your choosen prefix

Apache's example FCGI config

----------------------------

TODO !

Other configuration files

-------------------------

Some example init.d scripts can be found here, for debian and gentoo:

https://rhodecode.org/rhodecode/files/tip/init.d

Troubleshooting

---------------

:Q: **Missing static files?**

:A: Make sure either to set the `static_files = true` in the .ini file or

   double check the root path for your http setup. It should point to 

   for example:

   /home/my-virtual-python/lib/python2.6/site-packages/rhodecode/public

| 

:Q: **Can't install celery/rabbitmq**

:A: Don't worry RhodeCode works without them too. No extra setup is required.

|

:Q: **Long lasting push timeouts?**

:A: Make sure you set a longer timeouts in your proxy/fcgi settings, timeouts

    are caused by https server and not RhodeCode.

| 

:Q: **Large pushes timeouts?**

:A: Make sure you set a proper max_body_size for the http server.

|

:Q: **Apache doesn't pass basicAuth on pull/push?**

:A: Make sure you added `WSGIPassAuthorization true`.

For further questions search the `Issues tracker`_, or post a message in the `google group rhodecode`_

.. _virtualenv: http://pypi.python.org/pypi/virtualenv

.. _python: http://www.python.org/

.. _mercurial: http://mercurial.selenic.com/

.. _celery: http://celeryproject.org/

.. _rabbitmq: http://www.rabbitmq.com/

.. _python-ldap: http://www.python-ldap.org/

.. _mercurial-server: http://www.lshift.net/mercurial-server.html

.. _PublishingRepositories: http://mercurial.selenic.com/wiki/PublishingRepositories

.. _Issues tracker: https://bitbucket.org/marcinkuzminski/rhodecode/issues

.. _google group rhodecode: http://groups.google.com/group/rhodecode

# -*- coding: utf-8 -*-

"""

    rhodecode.controllers.admin.ldap_settings

    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    ldap controller for RhodeCode

    :created_on: Nov 26, 2010

    :author: marcink

    :copyright: (C) 2009-2011 Marcin Kuzminski <marcin@python-works.com>

    :license: GPLv3, see COPYING for more details.

"""

# This program is free software: you can redistribute it and/or modify

# it under the terms of the GNU General Public License as published by

# the Free Software Foundation, either version 3 of the License, or

# (at your option) any later version.

#

# This program is distributed in the hope that it will be useful,

# but WITHOUT ANY WARRANTY; without even the implied warranty of

# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the

# GNU General Public License for more details.

#

# You should have received a copy of the GNU General Public License

# along with this program.  If not, see <http://www.gnu.org/licenses/>.

import logging

import formencode

import traceback

from formencode import htmlfill

from pylons import request, response, session, tmpl_context as c, url

from pylons.controllers.util import abort, redirect

from pylons.i18n.translation import _

from rhodecode.lib.base import BaseController, render

from rhodecode.lib import helpers as h

from rhodecode.lib.auth import LoginRequired, HasPermissionAllDecorator

from rhodecode.model.forms import LdapSettingsForm

log = logging.getLogger(__name__)

class LdapSettingsController(BaseController):

    search_scope_choices = [('BASE', _('BASE'),),

                            ('ONELEVEL', _('ONELEVEL'),),

                            ('SUBTREE', _('SUBTREE'),),

                            ]

    search_scope_default = 'SUBTREE'

    tls_reqcert_choices = [('NEVER', _('NEVER'),),

                           ('ALLOW', _('ALLOW'),),

                           ('TRY', _('TRY'),),

                           ('DEMAND', _('DEMAND'),),

                           ('HARD', _('HARD'),),

                           ]

    tls_reqcert_default = 'DEMAND'

    tls_kind_choices = [('PLAIN', _('No encryption'),),

                        ('LDAPS', _('LDAPS connection'),),

                        ('START_TLS', _('START_TLS on LDAP connection'),)

                        ]

    tls_kind_default = 'PLAIN'

    @LoginRequired()

    @HasPermissionAllDecorator('hg.admin')

    def __before__(self):

        c.admin_user = session.get('admin_user')

        c.admin_username = session.get('admin_username')

        c.search_scope_choices = self.search_scope_choices

        c.tls_reqcert_choices = self.tls_reqcert_choices

        c.tls_kind_choices = self.tls_kind_choices

        super(LdapSettingsController, self).__before__()

    def index(self):

        c.search_scope_cur = defaults.get('ldap_search_scope')

        c.tls_reqcert_cur = defaults.get('ldap_tls_reqcert')

        c.tls_kind_cur = defaults.get('ldap_tls_kind')

        return htmlfill.render(

                    render('admin/ldap/ldap.html'),

                    defaults=defaults,

                    encoding="UTF-8",

                    force_defaults=True,)

    def ldap_settings(self):

        """POST ldap create and store ldap settings"""

        _form = LdapSettingsForm([x[0] for x in self.tls_reqcert_choices],

                                 [x[0] for x in self.search_scope_choices],

                                 [x[0] for x in self.tls_kind_choices])()

        try:

            form_result = _form.to_python(dict(request.POST))

            try:

                for k, v in form_result.items():

                    if k.startswith('ldap_'):

                        setting.app_settings_value = v

                        self.sa.add(setting)

                self.sa.commit()

                h.flash(_('Ldap settings updated successfully'),

                    category='success')

            except (DatabaseError,):

                raise

        except LdapImportError:

            h.flash(_('Unable to activate ldap. The "python-ldap" library '

                      'is missing.'), category='warning')

        except formencode.Invalid, errors:

            return htmlfill.render(

                render('admin/ldap/ldap.html'),

                defaults=errors.value,

                prefix_error=False,

                encoding="UTF-8")

        except Exception:

            log.error(traceback.format_exc())

            h.flash(_('error occurred during update of ldap settings'),

                    category='error')

        return redirect(url('ldap_home'))

# -*- coding: utf-8 -*-

"""

    rhodecode.controllers.admin.settings

    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    settings controller for rhodecode admin

    :created_on: Jul 14, 2010

    :author: marcink

    :copyright: (C) 2009-2011 Marcin Kuzminski <marcin@python-works.com>

    :license: GPLv3, see COPYING for more details.

"""

# This program is free software: you can redistribute it and/or modify

# it under the terms of the GNU General Public License as published by

# the Free Software Foundation, either version 3 of the License, or

# (at your option) any later version.

#

# This program is distributed in the hope that it will be useful,

# but WITHOUT ANY WARRANTY; without even the implied warranty of

# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the

# GNU General Public License for more details.

#

# You should have received a copy of the GNU General Public License

# along with this program.  If not, see <http://www.gnu.org/licenses/>.

import logging

import traceback

import formencode

from sqlalchemy import func

from formencode import htmlfill

from pylons import request, session, tmpl_context as c, url, config

from pylons.controllers.util import abort, redirect

from pylons.i18n.translation import _

from rhodecode.lib import helpers as h

from rhodecode.lib.auth import LoginRequired, HasPermissionAllDecorator, \

    HasPermissionAnyDecorator, NotAnonymous

from rhodecode.lib.base import BaseController, render

from rhodecode.lib.celerylib import tasks, run_task

from rhodecode.lib.utils import repo2db_mapper, invalidate_cache, \

    set_rhodecode_config, repo_name_slug

from rhodecode.model.forms import UserForm, ApplicationSettingsForm, \

    ApplicationUiSettingsForm

from rhodecode.model.scm import ScmModel

from rhodecode.model.user import UserModel

log = logging.getLogger(__name__)

class SettingsController(BaseController):

    """REST Controller styled on the Atom Publishing Protocol"""

    # To properly map this controller, ensure your config/routing.py

    # file has a resource setup:

    #     map.resource('setting', 'settings', controller='admin/settings',

    #         path_prefix='/admin', name_prefix='admin_')

    @LoginRequired()

    def __before__(self):

        c.admin_user = session.get('admin_user')

        c.admin_username = session.get('admin_username')

        super(SettingsController, self).__before__()

    @HasPermissionAllDecorator('hg.admin')

    def index(self, format='html'):

        """GET /admin/settings: All items in the collection"""

        # url('admin_settings')

        defaults.update(self.get_hg_ui_settings())

        return htmlfill.render(

            render('admin/settings/settings.html'),

            defaults=defaults,

            encoding="UTF-8",

            force_defaults=False

        )

    @HasPermissionAllDecorator('hg.admin')

    def create(self):

        """POST /admin/settings: Create a new item"""

        # url('admin_settings')

    @HasPermissionAllDecorator('hg.admin')

    def new(self, format='html'):

        """GET /admin/settings/new: Form to create a new item"""

        # url('admin_new_setting')

    @HasPermissionAllDecorator('hg.admin')

    def update(self, setting_id):

        """PUT /admin/settings/setting_id: Update an existing item"""

        # Forms posted to this method should contain a hidden field:

        #    <input type="hidden" name="_method" value="PUT" />

        # Or using helpers:

        #    h.form(url('admin_setting', setting_id=ID),

        #           method='put')

        # url('admin_setting', setting_id=ID)

        if setting_id == 'mapping':

            rm_obsolete = request.POST.get('destroy', False)

            log.debug('Rescanning directories with destroy=%s', rm_obsolete)

            initial = ScmModel().repo_scan()

            log.debug('invalidating all repositories')

            for repo_name in initial.keys():

                invalidate_cache('get_repo_cached_%s' % repo_name)

            added, removed = repo2db_mapper(initial, rm_obsolete)

            h.flash(_('Repositories successfully'

                      ' rescanned added: %s,removed: %s') % (added, removed),

                    category='success')

        if setting_id == 'whoosh':

            repo_location = self.get_hg_ui_settings()['paths_root_path']

            full_index = request.POST.get('full_index', False)

            run_task(tasks.whoosh_index, repo_location, full_index)

            h.flash(_('Whoosh reindex task scheduled'), category='success')

        if setting_id == 'global':

            application_form = ApplicationSettingsForm()()

            try:

                form_result = application_form.to_python(dict(request.POST))

                try:

                    hgsettings1.app_settings_value = \

                        form_result['rhodecode_title']

                    hgsettings2.app_settings_value = \

                        form_result['rhodecode_realm']

                    hgsettings3.app_settings_value = \

                        form_result['rhodecode_ga_code']

                    self.sa.add(hgsettings1)

                    self.sa.add(hgsettings2)

                    self.sa.add(hgsettings3)

                    self.sa.commit()

                    set_rhodecode_config(config)

                    h.flash(_('Updated application settings'),

                            category='success')

                except Exception:

                    log.error(traceback.format_exc())

                    h.flash(_('error occurred during updating '

                              'application settings'),

                            category='error')