.. _contributing:

=========================

Contributing to Kallithea

=========================

Kallithea is developed and maintained by its users. Please join us and scratch

your own itch.

Infrastructure

--------------

The main repository is hosted on Our Own Kallithea (aka OOK) at

https://kallithea-scm.org/repos/kallithea/, our self-hosted instance

of Kallithea.

For now, we use Bitbucket_ for `pull requests`_ and `issue tracking`_. The

issue tracker is for tracking bugs, not for support, discussion, or ideas --

please use the `mailing list`_ or :ref:`IRC <readme>` to reach the community.

We use Weblate_ to translate the user interface messages into languages other

than English. Join our project on `Hosted Weblate`_ to help us.

To register, you can use your Bitbucket or GitHub account. See :ref:`translations`

for more details.

Getting started

---------------

To get started with development::

        hg clone https://kallithea-scm.org/repos/kallithea

        cd kallithea

        virtualenv ../kallithea-venv

        source ../kallithea-venv/bin/activate

        pip install --upgrade pip setuptools

        pip install -e .

        paster make-config Kallithea my.ini

        paster setup-db my.ini --user=user --email=user@example.com --password=password --repos=/tmp

        paster serve my.ini --reload &

        firefox http://127.0.0.1:5000/

You can also start out by forking https://bitbucket.org/conservancy/kallithea

on Bitbucket_ and create a local clone of your own fork.

Running tests

-------------

After finishing your changes make sure all tests pass cleanly. Install the test

dependencies, then run the testsuite by invoking ``py.test`` from the

project root::

    pip install -r dev_requirements.txt

    py.test

Note that testing on Python 2.6 also requires ``unittest2``.

You can also use ``tox`` to run the tests with all supported Python versions

(currently Python 2.6--2.7).

When running tests, Kallithea uses `kallithea/tests/test.ini` and populates the

SQLite database specified there.

It is possible to avoid recreating the full test database on each invocation of

the tests, thus eliminating the initial delay. To achieve this, run the tests as::

    paster serve kallithea/tests/test.ini --pid-file=test.pid --daemon

    KALLITHEA_WHOOSH_TEST_DISABLE=1 KALLITHEA_NO_TMP_PATH=1 py.test

    kill -9 $(cat test.pid)

In these commands, the following variables are used::

    KALLITHEA_WHOOSH_TEST_DISABLE=1 - skip whoosh index building and tests

    KALLITHEA_NO_TMP_PATH=1 - disable new temp path for tests, used mostly for testing_vcs_operations

You can run individual tests by specifying their path as argument to py.test.

py.test also has many more options, see `py.test -h`. Some useful options

are::

    -k EXPRESSION         only run tests which match the given substring

                          expression. An expression is a python evaluable

                          expression where all names are substring-matched

                          against test names and their parent classes. Example:

    -x, --exitfirst       exit instantly on first error or failed test.

    --lf                  rerun only the tests that failed at the last run (or

                          all if none failed)

    --ff                  run all tests but run the last failures first. This

                          may re-order tests and thus lead to repeated fixture

                          setup/teardown

    --pdb                 start the interactive Python debugger on errors.

    -s, --capture=no      don't capture stdout (any stdout output will be

                          printed immediately)

Contribution guidelines

-----------------------

Kallithea is GPLv3 and we assume all contributions are made by the

committer/contributor and under GPLv3 unless explicitly stated. We do care a

lot about preservation of copyright and license information for existing code

that is brought into the project.

Contributions will be accepted in most formats -- such as pull requests on

Bitbucket, something hosted on your own Kallithea instance, or patches sent by

email to the `kallithea-general`_ mailing list.

When contributing via Bitbucket, please make your fork of

https://bitbucket.org/conservancy/kallithea/ `non-publishing`_ -- it is one of

the settings on "Repository details" page. This ensures your commits are in

"draft" phase and makes it easier for you to address feedback and for project

maintainers to integrate your changes.

.. _non-publishing: https://www.mercurial-scm.org/wiki/Phases#Publishing_Repository

Make sure to test your changes both manually and with the automatic tests

before posting.

We care about quality and review and keeping a clean repository history. We

might give feedback that requests polishing contributions until they are

"perfect". We might also rebase and collapse and make minor adjustments to your

changes when we apply them.

We try to make sure we have consensus on the direction the project is taking.

Everything non-sensitive should be discussed in public -- preferably on the

mailing list.  We aim at having all non-trivial changes reviewed by at least

one other core developer before pushing. Obvious non-controversial changes will

be handled more casually.

For now we just have one official branch ("default") and will keep it so stable

that it can be (and is) used in production. Experimental changes should live

elsewhere (for example in a pull request) until they are ready.

Coding guidelines

-----------------

We don't have a formal coding/formatting standard. We are currently using a mix

consistency with existing code. Run ``scripts/run-all-cleanup`` before

committing to ensure some basic code formatting consistency.

We support both Python 2.6.x and 2.7.x and nothing else. For now we don't care

about Python 3 compatibility.

We try to support the most common modern web browsers. IE9 is still supported

to the extent it is feasible, IE8 is not.

We primarily support Linux and OS X on the server side but Windows should also work.

HTML templates should use 2 spaces for indentation ... but be pragmatic. We

should use templates cleverly and avoid duplication. We should use reasonable

semantic markup with element classes and IDs that can be used for styling and testing.

We should only use inline styles in places where it really is semantic (such as

``display: none``).

JavaScript must use ``;`` between/after statements. Indentation 4 spaces. Inline

multiline functions should be indented two levels -- one for the ``()`` and one for

``{}``.

Variables holding jQuery objects should be named with a leading ``$``.

Commit messages should have a leading short line summarizing the changes. For

bug fixes, put ``(Issue #123)`` at the end of this line.

Use American English grammar and spelling overall. Use `English title case`_ for

page titles, button labels, headers, and 'labels' for fields in forms.

.. _English title case: https://en.wikipedia.org/wiki/Capitalization#Title_case

Template helpers (that is, everything in ``kallithea.lib.helpers``)

should only be referenced from templates. If you need to call a

helper from the Python code, consider moving the function somewhere

else (e.g. to the model).

Notes on the SQLAlchemy session

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Each HTTP request runs inside an independent SQLAlchemy session (as well

as in an independent database transaction). Database model objects

(almost) always belong to a particular SQLAlchemy session, which means

that SQLAlchemy will ensure that they're kept in sync with the database

(but also means that they cannot be shared across requests).

Objects can be added to the session using ``Session().add``, but this is

rarely needed:

* When creating a database object by calling the constructor directly,

  it must explicitly be added to the session.

* When creating an object using a factory function (like

  ``create_repo``), the returned object has already (by convention)

  been added to the session, and should not be added again.

* When getting an object from the session (via ``Session().query`` or

  any of the utility functions that look up objects in the database),

  it's already part of the session, and should not be added again.

  SQLAlchemy monitors attribute modifications automatically for all

  objects it knows about and syncs them to the database.

SQLAlchemy also flushes changes to the database automatically; manually

calling ``Session().flush`` is usually only necessary when the Python

code needs the database to assign an "auto-increment" primary key ID to

a freshly created model object (before flushing, the ID attribute will

be ``None``).

"Roadmap"

---------

We do not have a road map but are waiting for your contributions. Refer to the

wiki_ for some ideas of places we might want to go -- contributions in these

areas are very welcome.

Thank you for your contribution!

--------------------------------

.. _Weblate: http://weblate.org/

.. _issue tracking: https://bitbucket.org/conservancy/kallithea/issues?status=new&status=open

.. _pull requests: https://bitbucket.org/conservancy/kallithea/pull-requests

.. _bitbucket: http://bitbucket.org/

.. _mailing list: http://lists.sfconservancy.org/mailman/listinfo/kallithea-general

.. _kallithea-general: http://lists.sfconservancy.org/mailman/listinfo/kallithea-general

.. _Hosted Weblate: https://hosted.weblate.org/projects/kallithea/kallithea/

.. _wiki: https://bitbucket.org/conservancy/kallithea/wiki/Home

.. _index:

#######################

Kallithea Documentation

#######################

**Readme**

.. toctree::

   :maxdepth: 1

   readme

**Installation**

.. toctree::

   :maxdepth: 1

   overview

   installation

   installation_win

   installation_win_old

   installation_iis

   setup

   installation_puppet

   upgrade

**Usage**

.. toctree::

   :maxdepth: 1

   usage/general

   usage/vcs_support

   usage/locking

   usage/statistics

**Administrator's guide**

.. toctree::

   :maxdepth: 1

   usage/email

   usage/performance

   usage/backup

   usage/debugging

   usage/troubleshooting

   usage/customization

**Development**

.. toctree::

   :maxdepth: 1

   contributing

   dev/translation

   dev/dbmigrations

**API**

.. toctree::

   :maxdepth: 1

   api/api

Other topics

************

* :ref:`genindex`

* :ref:`search`

.. _virtualenv: http://pypi.python.org/pypi/virtualenv

.. _python: http://www.python.org/

.. _django: http://www.djangoproject.com/

.. _bitbucket: http://bitbucket.org/

.. _subversion: http://subversion.tigris.org/

.. _git: http://git-scm.com/

.. _celery: http://celeryproject.org/

.. _Sphinx: http://sphinx.pocoo.org/

.. _vcs: http://pypi.python.org/pypi/vcs

                req.subprocess_env['X_REMOTE_USER'] = username

                req.subprocess_env['X_REMOTE_EMAIL'] = r[0][1]['mail'][0].lower()

                req.subprocess_env['X_REMOTE_FIRSTNAME'] = "%s" % r[0][1]['givenname'][0]

                req.subprocess_env['X_REMOTE_LASTNAME'] = "%s" % r[0][1]['sn'][0]

            except Exception, e:

                apache.log_error("error getting data from ldap %s" % str(e), apache.APLOG_ERR)

            return apache.OK

.. note::

   If you enable proxy pass-through authentication, make sure your server is

   only accessible through the proxy. Otherwise, any client would be able to

   forge the authentication header and could effectively become authenticated

   using any account of their liking.

Integration with issue trackers

-------------------------------

Kallithea provides a simple integration with issue trackers. It's possible

to define a regular expression that will match an issue ID in commit messages,

and have that replaced with a URL to the issue. To enable this simply

uncomment the following variables in the ini file::

    issue_pat = (?:^#|\s#)(\w+)

    issue_server_link = https://issues.example.com/{repo}/issue/{id}

    issue_prefix = #

``issue_pat`` is the regular expression describing which strings in

commit messages will be treated as issue references. A match group in

parentheses should be used to specify the actual issue id.

The default expression matches issues in the format ``#<number>``, e.g., ``#300``.

Matched issue references are replaced with the link specified in

``issue_server_link``. ``{id}`` is replaced with the issue ID, and

``{repo}`` with the repository name.  Since the # is stripped away,

``issue_prefix`` is prepended to the link text.  ``issue_prefix`` doesn't

necessarily need to be ``#``: if you set issue prefix to ``ISSUE-`` this will

generate a URL in the format:

.. code-block:: html

  <a href="https://issues.example.com/example_repo/issue/300">ISSUE-300</a>

If needed, more than one pattern can be specified by appending a unique suffix to

the variables. For example::

    issue_pat_wiki = (?:wiki-)(.+)

    issue_server_link_wiki = https://wiki.example.com/{id}

    issue_prefix_wiki = WIKI-

With these settings, wiki pages can be referenced as wiki-some-id, and every

such reference will be transformed into:

.. code-block:: html

  <a href="https://wiki.example.com/some-id">WIKI-some-id</a>

Hook management

---------------

Hooks can be managed in similar way to that used in ``.hgrc`` files.

To manage hooks, choose *Admin > Settings > Hooks*.

The built-in hooks cannot be modified, though they can be enabled or disabled in the *VCS* section.

To add another custom hook simply fill in the first textbox with

``<name>.<hook_type>`` and the second with the hook path. Example hooks

can be found in ``kallithea.lib.hooks``.

Changing default encoding

-------------------------

By default, Kallithea uses UTF-8 encoding.

This is configurable as ``default_encoding`` in the .ini file.

This affects many parts in Kallithea including user names, filenames, and

encoding of commit messages. In addition Kallithea can detect if the ``chardet``

library is installed. If ``chardet`` is detected Kallithea will fallback to it

when there are encode/decode errors.

Celery configuration

--------------------

Kallithea can use the distributed task queue system Celery_ to run tasks like

cloning repositories or sending emails.

Kallithea will in most setups work perfectly fine out of the box (without

Celery), executing all tasks in the web server process. Some tasks can however

take some time to run and it can be better to run such tasks asynchronously in

a separate process so the web server can focus on serving web requests.

For installation and configuration of Celery, see the `Celery documentation`_.

Note that Celery requires a message broker service like RabbitMQ_ (recommended)

or Redis_.

The use of Celery is configured in the Kallithea ini configuration file.

To enable it, simply set::

  use_celery = true

and add or change the ``celery.*`` and ``broker.*`` configuration variables.

Remember that the ini files use the format with '.' and not with '_' like

Celery. So for example setting `BROKER_HOST` in Celery means setting

`broker.host` in the configuration file.

To start the Celery process, run::

 paster celeryd <configfile.ini>

.. note::

   Make sure you run this command from the same virtualenv, and with the same

   user that Kallithea runs.

HTTPS support

-------------

Kallithea will by default generate URLs based on the WSGI environment.

Alternatively, you can use some special configuration settings to control

directly which scheme/protocol Kallithea will use when generating URLs:

- With ``https_fixup = true``, the scheme will be taken from the

  ``X-Url-Scheme``, ``X-Forwarded-Scheme`` or ``X-Forwarded-Proto`` HTTP header

  (default ``http``).

- With ``force_https = true`` the default will be ``https``.

- With ``use_htsts = true``, Kallithea will set ``Strict-Transport-Security`` when using https.

Nginx virtual host example

--------------------------

Sample config for Nginx using proxy:

.. code-block:: nginx

    upstream kallithea {

        server 127.0.0.1:5000;

        # add more instances for load balancing

        #server 127.0.0.1:5001;

        #server 127.0.0.1:5002;

    }

    ## gist alias

    server {

       listen          443;

       server_name     gist.example.com;

       access_log      /var/log/nginx/gist.access.log;

       error_log       /var/log/nginx/gist.error.log;

       ssl on;

       ssl_certificate     gist.your.kallithea.server.crt;

       ssl_certificate_key gist.your.kallithea.server.key;

       ssl_session_timeout 5m;

       ssl_protocols SSLv3 TLSv1;

       ssl_ciphers DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:EDH-RSA-DES-CBC3-SHA:AES256-SHA:DES-CBC3-SHA:AES128-SHA:RC4-SHA:RC4-MD5;

       ssl_prefer_server_ciphers on;

       rewrite ^/(.+)$ https://kallithea.example.com/_admin/gists/$1;

       rewrite (.*)    https://kallithea.example.com/_admin/gists;

    }

    server {

       listen          443;

       server_name     kallithea.example.com

       access_log      /var/log/nginx/kallithea.access.log;

       error_log       /var/log/nginx/kallithea.error.log;

       ssl on;

       ssl_certificate     your.kallithea.server.crt;

       ssl_certificate_key your.kallithea.server.key;

       ssl_session_timeout 5m;

       ssl_protocols SSLv3 TLSv1;

       ssl_ciphers DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:EDH-RSA-DES-CBC3-SHA:AES256-SHA:DES-CBC3-SHA:AES128-SHA:RC4-SHA:RC4-MD5;

       ssl_prefer_server_ciphers on;

       ## uncomment root directive if you want to serve static files by nginx

       ## requires static_files = false in .ini file

       #root /srv/kallithea/kallithea/kallithea/public;

       include         /etc/nginx/proxy.conf;

       location / {

            try_files $uri @kallithea;

       }

       location @kallithea {

            proxy_pass      http://127.0.0.1:5000;

       }

    }

Here's the proxy.conf. It's tuned so it will not timeout on long

pushes or large pushes::

    proxy_redirect              off;

    proxy_set_header            Host $host;

    ## needed for container auth

    #proxy_set_header            REMOTE_USER $remote_user;

    #proxy_set_header            X-Forwarded-User $remote_user;

    proxy_set_header            X-Url-Scheme $scheme;

    proxy_set_header            X-Host $http_host;

    proxy_set_header            X-Real-IP $remote_addr;

    proxy_set_header            X-Forwarded-For $proxy_add_x_forwarded_for;

    proxy_set_header            Proxy-host $proxy_host;

    proxy_buffering             off;

    proxy_connect_timeout       7200;

    proxy_send_timeout          7200;

    proxy_read_timeout          7200;

    proxy_buffers               8 32k;

    client_max_body_size        1024m;

    client_body_buffer_size     128k;

    large_client_header_buffers 8 64k;

Apache virtual host reverse proxy example

-----------------------------------------

Here is a sample configuration file for Apache using proxy:

.. code-block:: apache

    <VirtualHost *:80>

            ServerName kallithea.example.com

            <Proxy *>

              # For Apache 2.4 and later:

              Require all granted

              # For Apache 2.2 and earlier, instead use:

              # Order allow,deny

              # Allow from all

            </Proxy>

            #important !

            #Directive to properly generate url (clone url) for pylons

            ProxyPreserveHost On

            #kallithea instance

            ProxyPass / http://127.0.0.1:5000/

            ProxyPassReverse / http://127.0.0.1:5000/

            #to enable https use line below

            #SetEnvIf X-Url-Scheme https HTTPS=1

    </VirtualHost>

Additional tutorial

http://pylonsbook.com/en/1.1/deployment.html#using-apache-to-proxy-requests-to-pylons

Apache as subdirectory

----------------------

Apache subdirectory part:

.. code-block:: apache

    <Location /<someprefix> >

      ProxyPass http://127.0.0.1:5000/<someprefix>

      ProxyPassReverse http://127.0.0.1:5000/<someprefix>

      SetEnvIf X-Url-Scheme https HTTPS=1

    </Location>

Besides the regular apache setup you will need to add the following line

into ``[app:main]`` section of your .ini file::

    filter-with = proxy-prefix

Add the following at the end of the .ini file::

    [filter:proxy-prefix]

    use = egg:PasteDeploy#prefix

    prefix = /<someprefix>

then change ``<someprefix>`` into your chosen prefix

Apache with mod_wsgi

--------------------

Alternatively, Kallithea can be set up with Apache under mod_wsgi. For

that, you'll need to:

- Install mod_wsgi. If using a Debian-based distro, you can install

  the package libapache2-mod-wsgi::

    aptitude install libapache2-mod-wsgi

- Enable mod_wsgi::

    a2enmod wsgi

- Add global Apache configuration to tell mod_wsgi that Python only will be

  used in the WSGI processes and shouldn't be initialized in the Apache

  processes::

    WSGIRestrictEmbedded On

- Create a wsgi dispatch script, like the one below. Make sure you

  check that the paths correctly point to where you installed Kallithea

  and its Python Virtual Environment.

- Enable the ``WSGIScriptAlias`` directive for the WSGI dispatch script,

  as in the following example. Once again, check the paths are

  correctly specified.

Here is a sample excerpt from an Apache Virtual Host configuration file:

.. code-block:: apache

    WSGIDaemonProcess kallithea processes=5 threads=1 maximum-requests=100 \

        python-home=/srv/kallithea/venv

    WSGIProcessGroup kallithea

    WSGIScriptAlias / /srv/kallithea/dispatch.wsgi

    WSGIPassAuthorization On

Or if using a dispatcher WSGI script with proper virtualenv activation:

.. code-block:: apache

    WSGIDaemonProcess kallithea processes=5 threads=1 maximum-requests=100

    WSGIProcessGroup kallithea

    WSGIScriptAlias / /srv/kallithea/dispatch.wsgi

    WSGIPassAuthorization On

Apache will by default run as a special Apache user, on Linux systems

usually ``www-data`` or ``apache``. If you need to have the repositories

directory owned by a different user, use the user and group options to

WSGIDaemonProcess to set the name of the user and group.

Example WSGI dispatch script:

.. code-block:: python

    import os

    os.environ["HGENCODING"] = "UTF-8"

    os.environ['PYTHON_EGG_CACHE'] = '/srv/kallithea/.egg-cache'

    # sometimes it's needed to set the current dir

    os.chdir('/srv/kallithea/')

    import site

    site.addsitedir("/srv/kallithea/venv/lib/python2.7/site-packages")

    ini = '/srv/kallithea/my.ini'

    from paste.script.util.logging_config import fileConfig

    fileConfig(ini)

    from paste.deploy import loadapp

    application = loadapp('config:' + ini)

Or using proper virtualenv activation:

.. code-block:: python

    activate_this = '/srv/kallithea/venv/bin/activate_this.py'

    execfile(activate_this, dict(__file__=activate_this))

    import os

    os.environ['HOME'] = '/srv/kallithea'

    ini = '/srv/kallithea/kallithea.ini'

    from paste.script.util.logging_config import fileConfig

    fileConfig(ini)

    from paste.deploy import loadapp

    application = loadapp('config:' + ini)

Other configuration files

-------------------------

A number of `example init.d scripts`__ can be found in

the ``init.d`` directory of the Kallithea source.

.. __: https://kallithea-scm.org/repos/kallithea/files/tip/init.d/ .

.. _virtualenv: http://pypi.python.org/pypi/virtualenv

.. _python: http://www.python.org/

.. _Celery: http://celeryproject.org/

.. _Celery documentation: http://docs.celeryproject.org/en/latest/getting-started/index.html

.. _RabbitMQ: http://www.rabbitmq.com/

.. _Redis: http://redis.io/

.. _python-ldap: http://www.python-ldap.org/

.. _mercurial-server: http://www.lshift.net/mercurial-server.html

.. _troubleshooting:

===============

Troubleshooting

===============

:Q: **Missing static files?**

:A: Make sure either to set the ``static_files = true`` in the .ini file or

   double check the root path for your http setup. It should point to

   for example:

   ``/home/my-virtual-python/lib/python2.7/site-packages/kallithea/public``

|

:Q: **Can't install celery/rabbitmq?**

:A: Don't worry. Kallithea works without them, too. No extra setup is required.

    Try out the great Celery docs for further help.

|

:Q: **Long lasting push timeouts?**

:A: Make sure you set a longer timeout in your proxy/fcgi settings. Timeouts

    are caused by the http server and not Kallithea.

|

:Q: **Large pushes timeouts?**

:A: Make sure you set a proper ``max_body_size`` for the http server. Very often

    Apache, Nginx, or other http servers kill the connection due to to large

    body.

|

:Q: **Apache doesn't pass basicAuth on pull/push?**

:A: Make sure you added ``WSGIPassAuthorization true``.

|

:Q: **Git fails on push/pull?**

:A: Make sure you're using a WSGI http server that can handle chunked encoding

    such as ``waitress`` or ``gunicorn``.

|

:Q: **How can I use hooks in Kallithea?**

:A: It's easy if they are Python hooks: just use advanced link in

    hooks section in Admin panel, that works only for Mercurial. If

    you want to use Git hooks, just install th proper one in the repository,

    e.g., create a file `/gitrepo/hooks/pre-receive`. You can also use

    Kallithea-extensions to connect to callback hooks, for both Git

    and Mercurial.

|

:Q: **Kallithea is slow for me, how can I make it faster?**

:A: See the :ref:`performance` section.

|

:Q: **UnicodeDecodeError on Apache mod_wsgi**

:A: Please read: https://docs.djangoproject.com/en/dev/howto/deployment/wsgi/modwsgi/#if-you-get-a-unicodeencodeerror.

|

:Q: **Requests hanging on Windows**

:A: Please try out with disabled Antivirus software, there are some known problems with Eset Antivirus. Make sure

    you have installed the latest Windows patches (especially KB2789397).

.. _virtualenv: http://pypi.python.org/pypi/virtualenv

.. _python: http://www.python.org/

.. _celery: http://celeryproject.org/

.. _rabbitmq: http://www.rabbitmq.com/

.. _python-ldap: http://www.python-ldap.org/