Contributing to Kallithea

=========================

Kallithea is developed and maintained by its users. Please join us and scratch

your own itch.

Infrastructure

--------------

The main repository is hosted at Our Own Kallithea (aka OOK) on

https://kallithea-scm.org/repos/kallithea/ (which is our self-hosted instance

of Kallithea).

For now, we use Bitbucket_ for `Pull Requests`_ and `Issue Tracker`_ services. The

issue tracker is for tracking bugs, not for "support", discussion or ideas -

please use the `mailing list`_ to reach the community.

We use Weblate_ to translate the user interface messages into languages other

than English. Join our project on `Hosted Weblate`_ to help us.

To register, you can use your Bitbucket or GitHub account.

Getting started

---------------

To get started with development::

        hg clone https://kallithea-scm.org/repos/kallithea

        cd kallithea

        virtualenv ../kallithea-venv

        source ../kallithea-venv/bin/activate

        python setup.py develop

        paster make-config Kallithea my.ini

        paster setup-db my.ini --user=user --email=user@example.com --password=password --repos=/tmp

        paster serve my.ini --reload &

        firefox http://127.0.0.1:5000/

You can also start out by forking https://bitbucket.org/conservancy/kallithea

on Bitbucket_ and create a local clone of your own fork.

Running tests

-------------

After finishing your changes make sure all tests pass cleanly. You can run

the testsuite running ``nosetest`` from the project root, or if you use tox

run tox for python2.6-2.7 with multiple database test. When using `nosetests`

to change your testing enviroment.

    paster serve test.ini --pid-file=test.pid --daemon

    KALLITHEA_WHOOSH_TEST_DISABLE=1 KALLITHEA_NO_TMP_PATH=1 nosetests -x kallithea/tests/other/test_vcs_operations.py

    kill -9 $(cat test.pid)

Coding/contribution guidelines

------------------------------

Kallithea is GPLv3 and we assume all contributions are made by the

committer/contributor and under GPLv3 unless explicitly stated. We do care a

lot about preservation of copyright and license information for existing code

that is brought into the project.

We don't have a formal coding/formatting standard. We are currently using a mix

of Mercurial (http://mercurial.selenic.com/wiki/CodingStyle), pep8, and

consistency with existing code. Run whitespacecleanup.sh to avoid stupid

whitespace noise in your patches.

We support both Python 2.6.x and 2.7.x and nothing else. For now we don't care

about Python 3 compatibility.

We try to support the most common modern web browsers. IE8 is still supported

to the extent it is feasible but we may stop supporting it very soon.

We primarily support Linux and OS X on the server side but Windows should also work.

Html templates should use 2 spaces for indentation ... but be pragmatic. We

should use templates cleverly and avoid duplication. We should use reasonable

semantic markup with classes and ids that can be used for styling and testing.

We should only use inline styles in places where it really is semantic (such as

display:none).

JavaScript must use ';' between/after statements. Indentation 4 spaces. Inline

multiline functions should be indented two levels - one for the () and one for

{}. jQuery value arrays should have a leading $.

Commit messages should have a leading short line summarizing the changes. For

bug fixes, put "(Issue #123)" at the end of this line.

Contributions will be accepted in most formats - such as pull requests on

bitbucket, something hosted on your own Kallithea instance, or patches sent by

mail to the kallithea-general mailing list.

Make sure to test your changes both manually and with the automatic tests

before posting.

We care about quality and review and keeping a clean repository history. We

In order to gain maximum performance

there are some third-party you must install. When Kallithea is used

together with celery you have to install some kind of message broker,

recommended one is rabbitmq_ to make the async tasks work.

Of course Kallithea works in sync mode also and then you do not have to install

any third party applications. However, using Celery_ will give you a large

speed improvement when using many big repositories. If you plan to use

Kallithea for say 7 to 10 repositories, Kallithea will perform perfectly well

without celery running.

If you make the decision to run Kallithea with celery make sure you run

celeryd using paster and message broker together with the application.

.. note::

   Installing message broker and using celery is optional, Kallithea will

   work perfectly fine without them.

**Message Broker**

- preferred is `RabbitMq <http://www.rabbitmq.com/>`_

- A possible alternative is `Redis <http://code.google.com/p/redis/>`_

For installation instructions you can visit:

http://ask.github.com/celery/getting-started/index.html.

This is a very nice tutorial on how to start using celery_ with rabbitmq_

Next

----

You can now proceed to :ref:`setup`.

Upgrading Kallithea from Python Package Index (PyPI)

-----------------------------------------------------

.. note::

   Firstly, it is recommended that you **always** perform a database and

   configuration backup before doing an upgrade.

   (These directions will use '{version}' to note that this is the version of

   Kallithea that these files were used with.  If backing up your Kallithea

   instance from version 0.1 to 0.2, the ``my.ini`` file could be

   backed up to ``my.ini.0-1``.)

then make a copy of the database file::

 service kallithea stop

 cp kallithea.db kallithea.db.{version}

Back up your configuration file::

 cp my.ini my.ini.{version}

Ensure that you are using the Python Virtual Environment that you'd originally

installed Kallithea in::

 pip freeze

will list all packages installed in the current environment.  If Kallithea

isn't listed, change virtual environments to your venv location::

 source /srv/kallithea/venv/bin/activate

Once you have verified the environment you can upgrade Kallithea with::

 pip install --upgrade kallithea

Then run the following command from the installation directory::

 paster make-config Kallithea my.ini

This will display any changes made by the new version of Kallithea to your

current configuration. It will try to perform an automerge. It's recommended

that you re-check the content after the automerge.

.. note::

   Please always make sure your .ini files are up to date. Often errors are

   caused by missing params added in new versions.

It is also recommended that you rebuild the whoosh index after upgrading since

the new whoosh version could introduce some incompatible index changes. Please

read the changelog to see if there were any changes to whoosh.

The final step is to upgrade the database. To do this simply run::

 paster upgrade-db my.ini

.. _setup:

=====

Setup

=====

Setting up Kallithea

--------------------

First, you will need to create a Kallithea configuration file. Run the

following command to do this::

    paster make-config Kallithea my.ini

- This will create the file `my.ini` in the current directory. This

  configuration file contains the various settings for Kallithea, e.g proxy

  port, email settings, usage of static files, cache, celery settings and

  logging.

Next, you need to create the databases used by Kallithea. It is recommended to

configuration file to use this other database. Kallithea currently supports

the following command::

    paster setup-db my.ini

This will prompt you for a "root" path. This "root" path is the location where

Kallithea will store all of its repositories on the current machine. After

entering this "root" path ``setup-db`` will also prompt you for a username

and password for the initial admin account which ``setup-db`` sets

up for you.

setup process can be fully automated, example for lazy::

    paster setup-db my.ini --user=nn --password=secret --email=nn@your.kallithea.server --repos=/srv/repos

- The ``setup-db`` command will create all of the needed tables and an

  admin account. When choosing a root path you can either use a new empty

  location, or a location which already contains existing repositories. If you

  (Note: make sure you specify the correct path to the root).

  for the application. It's very important since the Kallithea web interface

  will work without write access, but when trying to do a push it will

  eventually fail with permission denied errors unless it has write access.

You are now ready to use Kallithea, to run it simply execute::

    paster serve my.ini

- This command runs the Kallithea server. The web app should be available at the

  127.0.0.1:5000. This ip and port is configurable via the my.ini

  file created in previous step

- Use the admin account you created above when running ``setup-db``

  to login to the web app.

- The default permissions on each repository is read, and the owner is admin.

  Remember to update these if needed.

- In the admin panel you can toggle LDAP, anonymous, permissions settings. As

  well as edit more advanced options on users and repositories

Optionally users can create `rcextensions` package that extends Kallithea

functionality. To do this simply execute::

    paster make-rcext my.ini

This will create `rcextensions` package in the same place that your `ini` file

lives. With `rcextensions` it's possible to add additional mapping for whoosh,

stats and add additional code into the push/pull/create/delete repo hooks.

Please see the `__init__.py` file inside `rcextensions` package

for more details.

Using Kallithea with SSH

------------------------

Kallithea currently only hosts repositories using http and https. (The addition

of ssh hosting is a planned future feature.) However you can easily use ssh in

parallel with Kallithea. (Repository access via ssh is a standard "out of

repositories that Kallithea is hosting. See PublishingRepositories_)

Kallithea repository structures are kept in directories with the same name

as the project. When using repository groups, each group is a subdirectory.

This allows you to easily use ssh for accessing repositories.

In order to use ssh you need to make sure that your web-server and the users

login accounts have the correct permissions set on the appropriate directories.

(Note that these permissions are independent of any permissions you have set up

using the Kallithea web interface.)

If your main directory (the same as set in Kallithea settings) is for example

set to **/srv/repos** and the repository you are using is named `kallithea`, then

to clone via ssh you should run::

    hg clone ssh://user@server.com//srv/repos/kallithea

Using other external tools such as mercurial-server_ or using ssh key based

authentication is fully supported.

Note: In an advanced setup, in order for your ssh access to use the same

permissions as set up via the Kallithea web interface, you can create an

authentication hook to connect to the Kallithea db and runs check functions for

permissions against that.

Setting up Whoosh full text search

----------------------------------

The whoosh index can be build by using the paster

command ``make-index``. To use ``make-index`` you must specify the configuration

file that stores the location of the index. You may specify the location of the

repositories (`--repo-location`).  If not specified, this value is retrieved

from the Kallithea database.

It is also possible to specify a comma separated list of

repositories (`--index-only`) to build index only on chooses repositories

skipping any other found in repos location

You may optionally pass the option `-f` to enable a full index rebuild. Without

the `-f` option, indexing will run always in "incremental" mode.

For an incremental index build use::

    paster make-index my.ini

For a full index rebuild use::

    paster make-index my.ini -f

    issue_pat = (?:^#|\s#)(\w+)

    issue_server_link = https://myissueserver.com/{repo}/issue/{id}

    issue_prefix = #

`issue_pat` is the regular expression describing which strings in

commit messages will be treated as issue references. A match group in

parentheses should be used to specify the actual issue id.

The default expression matches issues in the format '#<number>', e.g. '#300'.

Matched issues are replaced with the link specified as `issue_server_link`

{id} is replaced with issue id, and {repo} with repository name.

Since the # is stripped away, `issue_prefix` is prepended to the link text.

`issue_prefix` doesn't necessarily need to be #: if you set issue

prefix to ISSUE- this will generate a URL in format::

  <a href="https://myissueserver.com/example_repo/issue/300">ISSUE-300</a>

If needed, more than one pattern can be specified by appending a unique suffix to

the variables. For example::

    issue_pat_wiki = (?:wiki-)(.+)

    issue_server_link_wiki = https://mywiki.com/{id}

    issue_prefix_wiki = WIKI-

With these settings, wiki pages can be referenced as wiki-some-id, and every

such reference will be transformed into::

  <a href="https://mywiki.com/some-id">WIKI-some-id</a>

Hook management

---------------

Hooks can be managed in similar way to this used in .hgrc files.

To access hooks setting click `advanced setup` on Hooks section of Mercurial

Settings in Admin.

There are 4 built in hooks that cannot be changed (only enable/disable by

checkboxes on previos section).

To add another custom hook simply fill in first section with

<name>.<hook_type> and the second one with hook path. Example hooks

can be found at *kallithea.lib.hooks*.

Changing default encoding

-------------------------

It is configurable as `default_encoding` in the .ini file.

This affects many parts in Kallithea including user names, filenames, and

encoding of commit messages. In addition Kallithea can detect if `chardet`

library is installed. If `chardet` is detected Kallithea will fallback to it

when there are encode/decode errors.

Celery configuration

--------------------

Celery is configured in the Kallithea ini configuration files.

Simply set use_celery=true in the ini file then add / change the configuration

variables inside the ini file.

Remember that the ini files use the format with '.' not with '_' like celery.

So for example setting `BROKER_HOST` in celery means setting `broker.host` in

the config file.

In order to start using celery run::

 paster celeryd <configfile.ini>

.. note::

   Make sure you run this command from the same virtualenv, and with the same

   user that Kallithea runs.

HTTPS support

-------------

Kallithea will by default generate URLs based on the WSGI environment.

Alternatively, you can use some special configuration settings to control

directly which scheme/protocol Kallithea will use when generating URLs:

- With `https_fixup = true`, the scheme will be taken from the HTTP_X_URL_SCHEME,

  HTTP_X_FORWARDED_SCHEME or HTTP_X_FORWARDED_PROTO HTTP header (default 'http').

- With `force_https = true` the default will be 'https'.

- With `use_htsts = true`, it will set Strict-Transport-Security when using https.

Nginx virtual host example

--------------------------

Sample config for nginx using proxy::

    upstream kallithea {

        server 127.0.0.1:5000;

        # add more instances for load balancing

   root, for examply by adding: `user=www-data group=www-data` to the configuration.

.. note::

   If running Kallithea in multiprocess mode,

   make sure you set `instance_id = \*` in the configuration so each process

   gets it's own cache invalidationkey.

Example wsgi dispatch script::

    import os

    os.environ["HGENCODING"] = "UTF-8"

    os.environ['PYTHON_EGG_CACHE'] = '/srv/kallithea/.egg-cache'

    # sometimes it's needed to set the curent dir

    os.chdir('/srv/kallithea/')

    import site

    site.addsitedir("/srv/kallithea/pyenv/lib/python2.7/site-packages")

    from paste.deploy import loadapp

    from paste.script.util.logging_config import fileConfig

    fileConfig('/srv/kallithea/my.ini')

    application = loadapp('config:/srv/kallithea/my.ini')

Or using proper virtualenv activation::

    activate_this = '/srv/kallithea/venv/bin/activate_this.py'

    execfile(activate_this,dict(__file__=activate_this))

    import os

    os.environ['HOME'] = '/srv/kallithea'

    ini = '/srv/kallithea/kallithea.ini'

    from paste.script.util.logging_config import fileConfig

    fileConfig(ini)

    from paste.deploy import loadapp

    application = loadapp('config:' + ini)

Other configuration files

-------------------------

Some example init.d scripts can be found in init.d directory: https://kallithea-scm.org/repos/kallithea/files/tip/init.d/

.. _virtualenv: http://pypi.python.org/pypi/virtualenv

.. _python: http://www.python.org/

.. _celery: http://celeryproject.org/

.. _rabbitmq: http://www.rabbitmq.com/

.. _python-ldap: http://www.python-ldap.org/

.. _mercurial-server: http://www.lshift.net/mercurial-server.html

.. _PublishingRepositories: http://mercurial.selenic.com/wiki/PublishingRepositories