.. _contributing:

=========================

Contributing to Kallithea

=========================

Kallithea is developed and maintained by its users. Please join us and scratch

your own itch.

Infrastructure

--------------

The main repository is hosted on Our Own Kallithea (aka OOK) at

https://kallithea-scm.org/repos/kallithea/, our self-hosted instance

of Kallithea.

For now, we use Bitbucket_ for `pull requests`_ and `issue tracking`_. The

issue tracker is for tracking bugs, not for support, discussion, or ideas --

please use the `mailing list`_ or :ref:`IRC <readme>` to reach the community.

We use Weblate_ to translate the user interface messages into languages other

than English. Join our project on `Hosted Weblate`_ to help us.

To register, you can use your Bitbucket or GitHub account. See :ref:`translations`

for more details.

Getting started

---------------

To get started with Kallithea development::

        hg clone https://kallithea-scm.org/repos/kallithea

        cd kallithea

        virtualenv ../kallithea-venv

        source ../kallithea-venv/bin/activate

        pip install --upgrade pip setuptools

        pip install --upgrade -e .

        pip install --upgrade -r dev_requirements.txt

        npm install     # install dependencies - both tools and data

        npm run less    # for generating css from less

        kallithea-cli config-create my.ini

        gearbox serve -c my.ini --reload &

        firefox http://127.0.0.1:5000/

If you plan to use Bitbucket_ for sending contributions, you can also fork

Kallithea on Bitbucket_ first (https://bitbucket.org/conservancy/kallithea) and

then replace the clone step above by a clone of your fork. In this case, please

see :ref:`contributing-guidelines` below for configuring your fork correctly.

Contribution flow

-----------------

Starting from an existing Kallithea clone, make sure it is up to date with the

latest upstream changes::

        hg pull

        hg update

Review the :ref:`contributing-guidelines` and :ref:`coding-guidelines`.

If you are new to Mercurial, refer to Mercurial `Quick Start`_ and `Beginners

Guide`_ on the Mercurial wiki.

Now, make some changes and test them (see :ref:`contributing-tests`). Don't

forget to add new tests to cover new functionality or bug fixes.

For documentation changes, run ``make html`` from the ``docs`` directory to

generate the HTML result, then review them in your browser.

Before submitting any changes, run the cleanup script::

        ./scripts/run-all-cleanup

When you are completely ready, you can send your changes to the community for

review and inclusion. Most commonly used methods are sending patches to the

mailing list (via ``hg email``) or by creating a pull request on Bitbucket_.

.. _contributing-tests:

Running tests

-------------

After finishing your changes make sure all tests pass cleanly. Run the testsuite

by invoking ``py.test`` from the project root::

    py.test

=======================

Database schema changes

=======================

Kallithea uses Alembic for :ref:`database migrations <upgrade_db>`

(upgrades and downgrades).

If you are developing a Kallithea feature that requires database schema

changes, you should make a matching Alembic database migration script:

1. :ref:`Create a Kallithea configuration and database <setup>` for testing

   the migration script, or use existing ``development.ini`` setup.

   Ensure that this database is up to date with the latest database

   schema *before* the changes you're currently developing. (Do not

   create the database while your new schema changes are applied.)

2. Create a separate throwaway configuration for iterating on the actual

   database changes::

    kallithea-cli config-create temp.ini

   Edit the file to change database settings. SQLite is typically fine,

   but make sure to change the path to e.g. ``temp.db``, to avoid

   clobbering any existing database file.

3. Make your code changes (including database schema changes in ``db.py``).

4. After every database schema change, recreate the throwaway database

   to test the changes::

    rm temp.db

    kallithea-cli repo-scan -c temp.ini

5. Once satisfied with the schema changes, auto-generate a draft Alembic

   script using the development database that has *not* been upgraded.

   (The generated script will upgrade the database to match the code.)

   ::

    alembic -c development.ini revision -m "area: add cool feature" --autogenerate

6. Edit the script to clean it up and fix any problems.

   Note that for changes that simply add columns, it may be appropriate

   to not remove them in the downgrade script (and instead do nothing),

   to avoid the loss of data. Unknown columns will simply be ignored by

   Kallithea versions predating your changes.

7. Run ``alembic -c development.ini upgrade head`` to apply changes to

   the (non-throwaway) database, and test the upgrade script. Also test

   downgrades.

   The included ``development.ini`` has full SQL logging enabled. If

   you're using another configuration file, you may want to enable it

   by setting ``level = DEBUG`` in section ``[handler_console_sql]``.

The Alembic migration script should be committed in the same revision as

the database schema (``db.py``) changes.

See the `Alembic documentation`__ for more information, in particular

the tutorial and the section about auto-generating migration scripts.

.. __: http://alembic.zzzcomputing.com/en/latest/

Troubleshooting

---------------

* If ``alembic --autogenerate`` responds "Target database is not up to

  date", you need to either first use Alembic to upgrade the database

  to the most recent version (before your changes), or recreate the

  database from scratch (without your schema changes applied).

In a command prompt type (adapting paths if necessary)::

  cd C:\Kallithea\Env\Scripts

  activate

  pip install --upgrade pip setuptools

The prompt will change into "(Env) C:\\Kallithea\\Env\\Scripts" or similar

(depending of your folder structure). Then type::

  pip install kallithea

.. note:: This will take some time. Please wait patiently until it is fully

          complete. Some warnings will appear. Don't worry, they are

          normal.

Step 8 -- Install Git (optional)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Mercurial being a python package, was installed automatically when doing ``pip install kallithea``.

You need to install Git manually if you want Kallithea to be able to host Git repositories.

See http://git-scm.com/book/en/v2/Getting-Started-Installing-Git#Installing-on-Windows for instructions.

The location of the Git binaries (like ``c:\path\to\git\bin``) must be

added to the ``PATH`` environment variable so ``git.exe`` and other tools like

``gzip.exe`` are available.

Step 9 -- Configuring Kallithea

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Steps taken from `<setup.html>`_

You have to use the same command prompt as in Step 7, so if you closed

it, reopen it following the same commands (including the "activate"

one). When ready, type::

  cd C:\Kallithea\Bin

  kallithea-cli config-create my.ini

Then you must edit my.ini to fit your needs (IP address, IP

port, mail settings, database, etc.). `NotePad++`__ or a similar text

editor is recommended to properly handle the newline character

differences between Unix and Windows.

__ http://notepad-plus-plus.org/

For the sake of simplicity, run it with the default settings. After your edits (if any) in the previous command prompt, type::

.. warning:: This time a *new* database will be installed. You must

             follow a different process to later :ref:`upgrade <upgrade>`

             to a newer Kallithea version.

The script will ask you for confirmation about creating a new database, answer yes (y)

The script will ask you for the repository path, answer C:\\Kallithea\\Repos (or similar).

The script will ask you for the admin username and password, answer "admin" + "123456" (or whatever you want)

The script will ask you for admin mail, answer "admin@xxxx.com" (or whatever you want).

If you make a mistake and the script doesn't end, don't worry: start it again.

If you decided not to install Git, you will get errors about it that you can ignore.

Step 10 -- Running Kallithea

^^^^^^^^^^^^^^^^^^^^^^^^^^^^

In the previous command prompt, being in the C:\\Kallithea\\Bin folder, type::

  gearbox serve -c my.ini

Open your web server, and go to http://127.0.0.1:5000

It works!! :-)

Remark:

If it does not work the first time, Ctrl-C the CMD process and start it again. Don't forget the "http://" in Internet Explorer.

What this guide does not cover:

- Installing Celery

- Running Kallithea as a Windows Service. You can investigate here:

  - http://pypi.python.org/pypi/wsgisvc

  - http://ryrobes.com/python/running-python-scripts-as-a-windows-service/

  - http://wiki.pylonshq.com/display/pylonscookbook/How+to+run+Pylons+as+a+Windows+service

- Using Apache. You can investigate here:

  - https://groups.google.com/group/rhodecode/msg/c433074e813ffdc4

   64-bit: For 64-bit you need to modify the shortcut that is used to start the

   Visual Studio 2008 Command Prompt. Use right-mouse click to open properties.

Change commandline from::

%comspec% /k ""C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\vcvarsall.bat"" x86

to::

%comspec% /k ""C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\vcvarsall.bat"" amd64

In that CMD (loaded with VS2008 PATHs) type::

  cd C:\Kallithea\Env\Scripts (or similar)

  activate

  pip install --upgrade pip setuptools

The prompt will change into "(Env) C:\\Kallithea\\Env\\Scripts" or similar

(depending of your folder structure). Then type::

 pip install kallithea

(long step, please wait until fully complete)

Some warnings will appear, don't worry as they are normal.

Step 8 -- Configuring Kallithea

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

steps taken from http://packages.python.org/Kallithea/setup.html

You have to use the same Visual Studio 2008 command prompt as Step7, so

if you closed it reopen it following the same commands (including the

"activate" one). When ready, just type::

  cd C:\Kallithea\Bin

  kallithea-cli config-create my.ini

Then, you must edit my.ini to fit your needs (network address and

port, mail settings, database, whatever). I recommend using NotePad++

(free) or similar text editor, as it handles well the EndOfLine

character differences between Unix and Windows

(http://notepad-plus-plus.org/)

For the sake of simplicity lets run it with the default settings. After

your edits (if any), in the previous Command Prompt, type::

.. warning:: This time a *new* database will be installed. You must

             follow a different process to later :ref:`upgrade <upgrade>`

             to a newer Kallithea version.

The script will ask you for confirmation about creating a NEW database,

answer yes (y)

The script will ask you for repository path, answer C:\\Kallithea\\Repos

(or similar)

The script will ask you for admin username and password, answer "admin"

+ "123456" (or whatever you want)

The script will ask you for admin mail, answer "admin@xxxx.com" (or

whatever you want)

If you make some mistake and the script does not end, don't worry, start

it again.

Step 9 -- Running Kallithea

^^^^^^^^^^^^^^^^^^^^^^^^^^^

In the previous command prompt, being in the C:\\Kallithea\\Bin folder,

just type::

 gearbox serve -c my.ini

Open yout web server, and go to http://127.0.0.1:5000

It works!! :-)

Remark:

If it does not work first time, just Ctrl-C the CMD process and start it

again. Don't forget the "http://" in Internet Explorer

What this Guide does not cover:

- Installing Celery

- Running Kallithea as Windows Service. You can investigate here:

  - http://pypi.python.org/pypi/wsgisvc

  - http://ryrobes.com/python/running-python-scripts-as-a-windows-service/

  - http://wiki.pylonshq.com/display/pylonscookbook/How+to+run+Pylons+as+a+Windows+service

- Using Apache. You can investigate here:

  - https://groups.google.com/group/rhodecode/msg/c433074e813ffdc4

.. _setup:

=====

Setup

=====

Preparing front-end

-------------------

Temporarily, in the current Kallithea version, some extra steps are required to

build front-end files:

Find the right ``kallithea/public/less`` path with::

    python -c "import os, kallithea; print os.path.join(os.path.dirname(os.path.abspath(kallithea.__file__)), 'public', 'less')"

Then run::

    npm install

    npm run less

Setting up Kallithea

--------------------

First, you will need to create a Kallithea configuration file. Run the

following command to do so::

    kallithea-cli config-create my.ini

This will create the file ``my.ini`` in the current directory. This

configuration file contains the various settings for Kallithea, e.g.

proxy port, email settings, usage of static files, cache, Celery

settings, and logging. Extra settings can be specified like::

    kallithea-cli config-create my.ini host=8.8.8.8 "[handler_console]" formatter=color_formatter

Next, you need to create the databases used by Kallithea. It is recommended to

use PostgreSQL or SQLite (default). If you choose a database other than the

default, ensure you properly adjust the database URL in your ``my.ini``

configuration file to use this other database. Kallithea currently supports

PostgreSQL, SQLite and MySQL databases. Create the database by running

the following command::

This will prompt you for a "root" path. This "root" path is the location where

Kallithea will store all of its repositories on the current machine. After

up for you.

Example::

admin account. When choosing a root path you can either use a new

empty location, or a location which already contains existing

repositories. If you choose a location which contains existing

repositories Kallithea will add all of the repositories at the chosen

location to its database.  (Note: make sure you specify the correct

path to the root).

.. note:: the given path for Mercurial_ repositories **must** be write

          accessible for the application. It's very important since

          the Kallithea web interface will work without write access,

          but when trying to do a push it will fail with permission

          denied errors unless it has write access.

You are now ready to use Kallithea. To run it simply execute::

    gearbox serve -c my.ini

- This command runs the Kallithea server. The web app should be available at

  http://127.0.0.1:5000. The IP address and port is configurable via the

  configuration file created in the previous step.

- The default permissions on each repository is read, and the owner is admin.

  Remember to update these if needed.

- In the admin panel you can toggle LDAP, anonymous, and permissions

  settings, as well as edit more advanced options on users and

  repositories.

Internationalization (i18n support)

-----------------------------------

The Kallithea web interface is automatically displayed in the user's preferred

language, as indicated by the browser. Thus, different users may see the

application in different languages. If the requested language is not available

(because the translation file for that language does not yet exist or is

incomplete), the language specified in setting ``i18n.lang`` in the Kallithea

configuration file is used as fallback. If no fallback language is explicitly

specified, English is used.

If you want to disable automatic language detection and instead configure a

fixed language regardless of user preference, set ``i18n.enabled = false`` and

set ``i18n.lang`` to the desired language (or leave empty for English).

Using Kallithea with SSH

------------------------

Kallithea currently only hosts repositories using http and https. (The addition

of ssh hosting is a planned future feature.) However you can easily use ssh in

parallel with Kallithea. (Repository access via ssh is a standard "out of

the box" feature of Mercurial_ and you can use this to access any of the

repositories that Kallithea is hosting. See PublishingRepositories_)

Kallithea repository structures are kept in directories with the same name

as the project. When using repository groups, each group is a subdirectory.

This allows you to easily use ssh for accessing repositories.

In order to use ssh you need to make sure that your web server and the users'

login accounts have the correct permissions set on the appropriate directories.

.. note:: These permissions are independent of any permissions you

          have set up using the Kallithea web interface.

If your main directory (the same as set in Kallithea settings) is for

example set to ``/srv/repos`` and the repository you are using is

named ``kallithea``, then to clone via ssh you should run::

    hg clone ssh://user@kallithea.example.com/srv/repos/kallithea

.. _vcs_notes:

===================================

Version control systems usage notes

===================================

.. _importing:

Importing existing repositories

-------------------------------

There are two main methods to import repositories in Kallithea: via the web

interface or via the filesystem. If you have a large number of repositories to

import, importing them via the filesystem is more convenient.

Importing via web interface

^^^^^^^^^^^^^^^^^^^^^^^^^^^

For a small number of repositories, it may be easier to create the target

repositories through the Kallithea web interface, via *Admin > Repositories* or

via the *Add Repository* button on the entry page of the web interface.

Repositories can be nested in repository groups by first creating the group (via

*Admin > Repository Groups* or via the *Add Repository Group* button on the

entry page of the web interface) and then selecting the appropriate group when

adding the repository.

After creation of the (empty) repository, push the existing commits to the

*Clone URL* displayed on the repository summary page. For Git repositories,

first add the *Clone URL* as remote, then push the commits to that remote.  The

specific commands to execute are shown under the *Existing repository?* section

of the new repository's summary page.

A benefit of this method particular for Git repositories, is that the

Kallithea-specific Git hooks are installed automatically.  For Mercurial, no

hooks are required anyway.

Importing via the filesystem

^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The alternative method of importing repositories consists of creating the

repositories in the desired hierarchy on the filesystem and letting Kallithea

scan that location.

All repositories are stored in a central location on the filesystem. This

at *Admin > Settings > VCS > Location of repositories*. Repository groups

(defined in *Admin > Repository Groups*) are represented by a directory in that

repository location. Repositories of the repository group are nested under that

directory.

To import a set of repositories and organize them in a certain repository group

structure, first place clones in the desired hierarchy at the configured

repository location.

These clones should be created without working directory. For Mercurial, this is

done with ``hg clone -U``, for Git with ``git clone --bare``.

When the repositories are added correctly on the filesystem:

* go to *Admin > Settings > Remap and Rescan* in the Kallithea web interface

* select the *Install Git hooks* checkbox when importing Git repositories

* click *Rescan Repositories*

This step will scan the filesystem and create the appropriate repository groups

and repositories in Kallithea.

*Note*: Once repository groups have been created this way, manage their access

permissions through the Kallithea web interface.

Mercurial-specific notes

------------------------

Working with subrepositories

^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This section explains how to use Mercurial subrepositories_ in Kallithea.

Example usage::

    ## init a simple repo

    hg init mainrepo

    cd mainrepo

    echo "file" > file

    hg add file

    hg ci --message "initial file"

    # clone subrepo we want to add from Kallithea

    hg clone http://kallithea.local/subrepo

    ## specify URL to existing repo in Kallithea as subrepository path

    echo "subrepo = http://kallithea.local/subrepo" > .hgsub

    hg add .hgsub

# -*- coding: utf-8 -*-

# This program is free software: you can redistribute it and/or modify

# it under the terms of the GNU General Public License as published by

# the Free Software Foundation, either version 3 of the License, or

# (at your option) any later version.

#

# This program is distributed in the hope that it will be useful,

# but WITHOUT ANY WARRANTY; without even the implied warranty of

# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the

# GNU General Public License for more details.

#

# You should have received a copy of the GNU General Public License

# along with this program.  If not, see <http://www.gnu.org/licenses/>.

# Alembic migration environment (configuration).

import logging

from logging.config import fileConfig

from alembic import context

from sqlalchemy import engine_from_config, pool

from kallithea.model import db

# The alembic.config.Config object, which wraps the current .ini file.

config = context.config

# Default to use the main Kallithea database string in [app:main].

# For advanced uses, this can be overridden by specifying an explicit

# [alembic] sqlalchemy.url.

database_url = (

    config.get_main_option('sqlalchemy.url') or

    config.get_section_option('app:main', 'sqlalchemy.url')

)

# Configure default logging for Alembic. (This can be overriden by the

# config file, but usually isn't.)

logging.getLogger('alembic').setLevel(logging.INFO)

# Setup Python loggers based on the config file provided to the alembic

# command. If we're being invoked via the Alembic API (presumably for

# and loggers are assumed to already have been configured.

if config.config_file_name:

    fileConfig(config.config_file_name, disable_existing_loggers=False)

def include_in_autogeneration(object, name, type, reflected, compare_to):

    """Filter changes subject to autogeneration of migrations. """

    # Don't include changes to sqlite_sequence.

    if type == 'table' and name == 'sqlite_sequence':

        return False

    return True

def run_migrations_offline():

    """Run migrations in 'offline' (--sql) mode.

    This produces an SQL script instead of directly applying the changes.

    Some migrations may not run in offline mode.

    """

    context.configure(

        url=database_url,

        literal_binds=True,

    )

    with context.begin_transaction():

        context.run_migrations()

def run_migrations_online():

    """Run migrations in 'online' mode.

    Connects to the database and directly applies the necessary

    migrations.

    """

    cfg = config.get_section(config.config_ini_section)

    cfg['sqlalchemy.url'] = database_url

    connectable = engine_from_config(

        cfg,

        prefix='sqlalchemy.',

        poolclass=pool.NullPool)

    with connectable.connect() as connection:

        context.configure(

            connection=connection,

            # Support autogeneration of migration scripts based on "diff" between

# -*- coding: utf-8 -*-

# This program is free software: you can redistribute it and/or modify

# it under the terms of the GNU General Public License as published by

# the Free Software Foundation, either version 3 of the License, or

# (at your option) any later version.

#

# This program is distributed in the hope that it will be useful,

# but WITHOUT ANY WARRANTY; without even the implied warranty of

# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the

# GNU General Public License for more details.

#

# You should have received a copy of the GNU General Public License

# along with this program.  If not, see <http://www.gnu.org/licenses/>.

# 'cli' is the main entry point for 'kallithea-cli', specified in setup.py as entry_points console_scripts

from kallithea.bin.kallithea_cli_base import cli

# import commands (they will add themselves to the 'cli' object)

import kallithea.bin.kallithea_cli_config

import kallithea.bin.kallithea_cli_iis

import kallithea.bin.kallithea_cli_ishell

import kallithea.bin.kallithea_cli_repo

# -*- coding: utf-8 -*-

# This program is free software: you can redistribute it and/or modify

# it under the terms of the GNU General Public License as published by

# the Free Software Foundation, either version 3 of the License, or

# (at your option) any later version.

#

# This program is distributed in the hope that it will be useful,

# but WITHOUT ANY WARRANTY; without even the implied warranty of

# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the

# GNU General Public License for more details.

#

# You should have received a copy of the GNU General Public License

# along with this program.  If not, see <http://www.gnu.org/licenses/>.

import kallithea

from kallithea.lib.db_manage import DbManage

from kallithea.model.meta import Session

    """