.. _contributing:

=========================

Contributing to Kallithea

=========================

Kallithea is developed and maintained by its users. Please join us and scratch

your own itch.

Infrastructure

--------------

The main repository is hosted on Our Own Kallithea (aka OOK) at

https://kallithea-scm.org/repos/kallithea/, our self-hosted instance

of Kallithea.

For now, we use Bitbucket_ for `pull requests`_ and `issue tracking`_. The

issue tracker is for tracking bugs, not for support, discussion, or ideas --

please use the `mailing list`_ or :ref:`IRC <readme>` to reach the community.

We use Weblate_ to translate the user interface messages into languages other

than English. Join our project on `Hosted Weblate`_ to help us.

To register, you can use your Bitbucket or GitHub account. See :ref:`translations`

for more details.

Getting started

---------------

To get started with Kallithea development::

        hg clone https://kallithea-scm.org/repos/kallithea

        cd kallithea

        virtualenv ../kallithea-venv

        source ../kallithea-venv/bin/activate

        pip install --upgrade pip setuptools

        pip install --upgrade -e .

        pip install --upgrade -r dev_requirements.txt

        npm install     # install dependencies - both tools and data

        npm run less    # for generating css from less

        gearbox setup-db -c my.ini --user=user --email=user@example.com --password=password --repos=/tmp

        gearbox serve -c my.ini --reload &

        firefox http://127.0.0.1:5000/

If you plan to use Bitbucket_ for sending contributions, you can also fork

Kallithea on Bitbucket_ first (https://bitbucket.org/conservancy/kallithea) and

then replace the clone step above by a clone of your fork. In this case, please

see :ref:`contributing-guidelines` below for configuring your fork correctly.

Contribution flow

-----------------

Starting from an existing Kallithea clone, make sure it is up to date with the

latest upstream changes::

        hg pull

        hg update

Review the :ref:`contributing-guidelines` and :ref:`coding-guidelines`.

If you are new to Mercurial, refer to Mercurial `Quick Start`_ and `Beginners

Guide`_ on the Mercurial wiki.

Now, make some changes and test them (see :ref:`contributing-tests`). Don't

forget to add new tests to cover new functionality or bug fixes.

For documentation changes, run ``make html`` from the ``docs`` directory to

generate the HTML result, then review them in your browser.

Before submitting any changes, run the cleanup script::

        ./scripts/run-all-cleanup

When you are completely ready, you can send your changes to the community for

review and inclusion. Most commonly used methods are sending patches to the

mailing list (via ``hg email``) or by creating a pull request on Bitbucket_.

.. _contributing-tests:

Running tests

-------------

After finishing your changes make sure all tests pass cleanly. Run the testsuite

by invoking ``py.test`` from the project root::

    py.test

=======================

Database schema changes

=======================

Kallithea uses Alembic for :ref:`database migrations <upgrade_db>`

(upgrades and downgrades).

If you are developing a Kallithea feature that requires database schema

changes, you should make a matching Alembic database migration script:

1. :ref:`Create a Kallithea configuration and database <setup>` for testing

   the migration script, or use existing ``development.ini`` setup.

   Ensure that this database is up to date with the latest database

   schema *before* the changes you're currently developing. (Do not

   create the database while your new schema changes are applied.)

2. Create a separate throwaway configuration for iterating on the actual

   database changes::

   Edit the file to change database settings. SQLite is typically fine,

   but make sure to change the path to e.g. ``temp.db``, to avoid

   clobbering any existing database file.

3. Make your code changes (including database schema changes in ``db.py``).

4. After every database schema change, recreate the throwaway database

   to test the changes::

    rm temp.db

    gearbox setup-db -c temp.ini --repos=/var/repos --user=doe --email doe@example.com --password=123456 --no-public-access --force-yes

    gearbox repo-scan -c temp.ini

5. Once satisfied with the schema changes, auto-generate a draft Alembic

   script using the development database that has *not* been upgraded.

   (The generated script will upgrade the database to match the code.)

   ::

    alembic -c development.ini revision -m "area: add cool feature" --autogenerate

6. Edit the script to clean it up and fix any problems.

   Note that for changes that simply add columns, it may be appropriate

   to not remove them in the downgrade script (and instead do nothing),

   to avoid the loss of data. Unknown columns will simply be ignored by

   Kallithea versions predating your changes.

7. Run ``alembic -c development.ini upgrade head`` to apply changes to

   the (non-throwaway) database, and test the upgrade script. Also test

   downgrades.

   The included ``development.ini`` has full SQL logging enabled. If

   you're using another configuration file, you may want to enable it

   by setting ``level = DEBUG`` in section ``[handler_console_sql]``.

The Alembic migration script should be committed in the same revision as

the database schema (``db.py``) changes.

See the `Alembic documentation`__ for more information, in particular

the tutorial and the section about auto-generating migration scripts.

.. __: http://alembic.zzzcomputing.com/en/latest/

Troubleshooting

---------------

Step 7 -- Install Kallithea

^^^^^^^^^^^^^^^^^^^^^^^^^^^

In order to install Kallithea, you need to be able to run "pip install kallithea". It will use pip to install the Kallithea Python package and its dependencies.

Some Python packages use managed code and need to be compiled.

This can be done on Linux without any special steps. On Windows, you will need to install Microsoft Visual C++ compiler for Python 2.7.

Download and install "Microsoft Visual C++ Compiler for Python 2.7" from http://aka.ms/vcpython27

.. note::

  You can also install the dependencies using already compiled Windows binaries packages. A good source of compiled Python packages is http://www.lfd.uci.edu/~gohlke/pythonlibs/. However, not all of the necessary packages for Kallithea are on this site and some are hard to find, so we will stick with using the compiler.

In a command prompt type (adapting paths if necessary)::

  cd C:\Kallithea\Env\Scripts

  activate

  pip install --upgrade pip setuptools

The prompt will change into "(Env) C:\\Kallithea\\Env\\Scripts" or similar

(depending of your folder structure). Then type::

  pip install kallithea

.. note:: This will take some time. Please wait patiently until it is fully

          complete. Some warnings will appear. Don't worry, they are

          normal.

Step 8 -- Install Git (optional)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Mercurial being a python package, was installed automatically when doing ``pip install kallithea``.

You need to install Git manually if you want Kallithea to be able to host Git repositories.

See http://git-scm.com/book/en/v2/Getting-Started-Installing-Git#Installing-on-Windows for instructions.

The location of the Git binaries (like ``c:\path\to\git\bin``) must be

added to the ``PATH`` environment variable so ``git.exe`` and other tools like

``gzip.exe`` are available.

Step 9 -- Configuring Kallithea

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Steps taken from `<setup.html>`_

You have to use the same command prompt as in Step 7, so if you closed

it, reopen it following the same commands (including the "activate"

one). When ready, type::

  cd C:\Kallithea\Bin

Then you must edit my.ini to fit your needs (IP address, IP

port, mail settings, database, etc.). `NotePad++`__ or a similar text

editor is recommended to properly handle the newline character

differences between Unix and Windows.

__ http://notepad-plus-plus.org/

For the sake of simplicity, run it with the default settings. After your edits (if any) in the previous command prompt, type::

  gearbox setup-db -c my.ini

.. warning:: This time a *new* database will be installed. You must

             follow a different process to later :ref:`upgrade <upgrade>`

             to a newer Kallithea version.

The script will ask you for confirmation about creating a new database, answer yes (y)

The script will ask you for the repository path, answer C:\\Kallithea\\Repos (or similar).

The script will ask you for the admin username and password, answer "admin" + "123456" (or whatever you want)

The script will ask you for admin mail, answer "admin@xxxx.com" (or whatever you want).

If you make a mistake and the script doesn't end, don't worry: start it again.

If you decided not to install Git, you will get errors about it that you can ignore.

Step 10 -- Running Kallithea

^^^^^^^^^^^^^^^^^^^^^^^^^^^^

In the previous command prompt, being in the C:\\Kallithea\\Bin folder, type::

  gearbox serve -c my.ini

Open your web server, and go to http://127.0.0.1:5000

It works!! :-)

Remark:

If it does not work the first time, Ctrl-C the CMD process and start it again. Don't forget the "http://" in Internet Explorer.

What this guide does not cover:

- Installing Celery

- Running Kallithea as a Windows Service. You can investigate here:

  - http://pypi.python.org/pypi/wsgisvc

Step 7 -- Install Kallithea

^^^^^^^^^^^^^^^^^^^^^^^^^^^

Finally, install Kallithea

Close previously opened command prompt/s, and open a Visual Studio 2008

Command Prompt (**IMPORTANT!!**). To do so, go to Start Menu, and then open

"Microsoft Visual C++ 2008 Express Edition" -> "Visual Studio Tools" ->

"Visual Studio 2008 Command Prompt"

.. note::

   64-bit: For 64-bit you need to modify the shortcut that is used to start the

   Visual Studio 2008 Command Prompt. Use right-mouse click to open properties.

Change commandline from::

%comspec% /k ""C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\vcvarsall.bat"" x86

to::

%comspec% /k ""C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\vcvarsall.bat"" amd64

In that CMD (loaded with VS2008 PATHs) type::

  cd C:\Kallithea\Env\Scripts (or similar)

  activate

  pip install --upgrade pip setuptools

The prompt will change into "(Env) C:\\Kallithea\\Env\\Scripts" or similar

(depending of your folder structure). Then type::

 pip install kallithea

(long step, please wait until fully complete)

Some warnings will appear, don't worry as they are normal.

Step 8 -- Configuring Kallithea

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

steps taken from http://packages.python.org/Kallithea/setup.html

You have to use the same Visual Studio 2008 command prompt as Step7, so

if you closed it reopen it following the same commands (including the

"activate" one). When ready, just type::

  cd C:\Kallithea\Bin

Then, you must edit my.ini to fit your needs (network address and

port, mail settings, database, whatever). I recommend using NotePad++

(free) or similar text editor, as it handles well the EndOfLine

character differences between Unix and Windows

(http://notepad-plus-plus.org/)

For the sake of simplicity lets run it with the default settings. After

your edits (if any), in the previous Command Prompt, type::

 gearbox setup-db -c my.ini

.. warning:: This time a *new* database will be installed. You must

             follow a different process to later :ref:`upgrade <upgrade>`

             to a newer Kallithea version.

The script will ask you for confirmation about creating a NEW database,

answer yes (y)

The script will ask you for repository path, answer C:\\Kallithea\\Repos

(or similar)

The script will ask you for admin username and password, answer "admin"

+ "123456" (or whatever you want)

The script will ask you for admin mail, answer "admin@xxxx.com" (or

whatever you want)

If you make some mistake and the script does not end, don't worry, start

it again.

Step 9 -- Running Kallithea

^^^^^^^^^^^^^^^^^^^^^^^^^^^

In the previous command prompt, being in the C:\\Kallithea\\Bin folder,

just type::

 gearbox serve -c my.ini

Open yout web server, and go to http://127.0.0.1:5000

It works!! :-)

Remark:

If it does not work first time, just Ctrl-C the CMD process and start it

again. Don't forget the "http://" in Internet Explorer

What this Guide does not cover:

- Installing Celery

- Running Kallithea as Windows Service. You can investigate here:

.. _setup:

=====

Setup

=====

Preparing front-end

-------------------

Temporarily, in the current Kallithea version, some extra steps are required to

build front-end files:

Find the right ``kallithea/public/less`` path with::

    python -c "import os, kallithea; print os.path.join(os.path.dirname(os.path.abspath(kallithea.__file__)), 'public', 'less')"

Then run::

    npm install

    npm run less

Setting up Kallithea

--------------------

First, you will need to create a Kallithea configuration file. Run the

following command to do so::

This will create the file ``my.ini`` in the current directory. This

configuration file contains the various settings for Kallithea, e.g.

proxy port, email settings, usage of static files, cache, Celery

settings, and logging. Extra settings can be specified like::

Next, you need to create the databases used by Kallithea. It is recommended to

use PostgreSQL or SQLite (default). If you choose a database other than the

default, ensure you properly adjust the database URL in your ``my.ini``

configuration file to use this other database. Kallithea currently supports

PostgreSQL, SQLite and MySQL databases. Create the database by running

the following command::

    gearbox setup-db -c my.ini

This will prompt you for a "root" path. This "root" path is the location where

Kallithea will store all of its repositories on the current machine. After

entering this "root" path ``setup-db`` will also prompt you for a username

and password for the initial admin account which ``setup-db`` sets

up for you.

The ``setup-db`` values can also be given on the command line.

Example::

    gearbox setup-db -c my.ini --user=nn --password=secret --email=nn@example.com --repos=/srv/repos

The ``setup-db`` command will create all needed tables and an

admin account. When choosing a root path you can either use a new

empty location, or a location which already contains existing

repositories. If you choose a location which contains existing

repositories Kallithea will add all of the repositories at the chosen

location to its database.  (Note: make sure you specify the correct

path to the root).

.. note:: the given path for Mercurial_ repositories **must** be write

          accessible for the application. It's very important since

          the Kallithea web interface will work without write access,

          but when trying to do a push it will fail with permission

          denied errors unless it has write access.

You are now ready to use Kallithea. To run it simply execute::

    gearbox serve -c my.ini

- This command runs the Kallithea server. The web app should be available at

  http://127.0.0.1:5000. The IP address and port is configurable via the

  configuration file created in the previous step.

- Log in to Kallithea using the admin account created when running ``setup-db``.

- The default permissions on each repository is read, and the owner is admin.

  Remember to update these if needed.

- In the admin panel you can toggle LDAP, anonymous, and permissions

  settings, as well as edit more advanced options on users and

  repositories.

3. Activate the Kallithea virtual environment (if any)

------------------------------------------------------

Verify that you are using the Python environment that you originally

installed Kallithea in by running::

    pip freeze

This will list all packages installed in the current environment. If

Kallithea isn't listed, activate the correct virtual environment.

See the appropriate installation page for details.

4. Install new version of Kallithea

-----------------------------------

Please refer to the instructions for the installation method you

originally used to install Kallithea.

If you originally installed using pip, it is as simple as::

    pip install --upgrade kallithea

If you originally installed from version control, it is as simple as::

    cd my-kallithea-clone

    hg pull -u

    pip install --upgrade -e .

Temporarily, in the current version, an extra step is required to build

front-end files:

Find the right ``kallithea/public/less`` path with::

    python -c "import os, kallithea; print os.path.join(os.path.dirname(os.path.abspath(kallithea.__file__)), 'public', 'less')"

Then run::

    npm install

    npm run less

5. Upgrade your configuration

-----------------------------

Run the following command to create a new configuration (``.ini``) file::

Then compare it with your old config file and see what changed.

.. note::

    Please always make sure your ``.ini`` files are up to date. Errors

    can often be caused by missing parameters added in new versions.

.. _upgrade_db:

6. Upgrade your database

------------------------

.. note::

    If you are *downgrading* Kallithea, you should perform the database

    migration step *before* installing the older version. (That is,

    always perform migrations using the most recent of the two versions

    you're migrating between.)

First, run the following command to see your current database version::

    alembic -c my.ini current

Typical output will be something like "9358dc3d6828 (head)", which is

the current Alembic database "revision ID". Write down the entire output

for troubleshooting purposes.

The output will be empty if you're upgrading from Kallithea 0.3.x or

older. That's expected. If you get an error that the config file was not

found or has no ``[alembic]`` section, see the next section.

Next, if you are performing an *upgrade*: Run the following command to

upgrade your database to the current Kallithea version::

    alembic -c my.ini upgrade head

If you are performing a *downgrade*: Run the following command to

downgrade your database to the given version::

    alembic -c my.ini downgrade 0.4

Alembic will show the necessary migrations (if any) as it executes them.

If no "ERROR" is displayed, the command was successful.

Should an error occur, the database may be "stranded" half-way

through the migration, and you should restore it from backup.

Enabling old Kallithea config files for Alembic use

# -*- coding: utf-8 -*-

# This program is free software: you can redistribute it and/or modify

# it under the terms of the GNU General Public License as published by

# the Free Software Foundation, either version 3 of the License, or

# (at your option) any later version.

#

# This program is distributed in the hope that it will be useful,

# but WITHOUT ANY WARRANTY; without even the implied warranty of

# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the

# GNU General Public License for more details.

#

# You should have received a copy of the GNU General Public License

# along with this program.  If not, see <http://www.gnu.org/licenses/>.

# 'cli' is the main entry point for 'kallithea-cli', specified in setup.py as entry_points console_scripts

from kallithea.bin.kallithea_cli_base import cli

# -*- coding: utf-8 -*-

# This program is free software: you can redistribute it and/or modify

# it under the terms of the GNU General Public License as published by

# the Free Software Foundation, either version 3 of the License, or

# (at your option) any later version.

#

# This program is distributed in the hope that it will be useful,

# but WITHOUT ANY WARRANTY; without even the implied warranty of

# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the

# GNU General Public License for more details.

#

# You should have received a copy of the GNU General Public License

# along with this program.  If not, see <http://www.gnu.org/licenses/>.

import os

import uuid

from collections import defaultdict

import mako.exceptions

from kallithea.lib import inifile

    The primary high level configuration keys and their default values are

    Additional key=value arguments will be patched/inserted in the [app:main]

    section ... until another section name specifies where any following values

    should go.

    """

    mako_variable_values = {}

    ini_settings = defaultdict(dict)

    section_name = None

        parts = parameter.split('=', 1)

        if len(parts) == 1 and parameter.startswith('[') and parameter.endswith(']'):

            section_name = parameter

        elif len(parts) == 2:

            key, value = parts

            if section_name is None and key in inifile.default_variables:

                mako_variable_values[key] = value

            else:

                if section_name is None:

                    section_name = '[app:main]'

                ini_settings[section_name][key] = value

        else:

            raise ValueError("Invalid name=value parameter %r" % parameter)

    # use default that cannot be replaced

    mako_variable_values.update({

        'uuid': lambda: uuid.uuid4().hex,

    })

    try:

    except Exception:

from setuptools.command import sdist

sdist_org = sdist.sdist

class sdist_new(sdist_org):

    def initialize_options(self):

        sdist_org.initialize_options(self)

        self.owner = self.group = 'root'

sdist.sdist = sdist_new

packages = setuptools.find_packages(exclude=['ez_setup'])

setuptools.setup(

    name='Kallithea',

    version=__version__,

    description=description,

    long_description=long_description,

    keywords=keywords,

    license=__license__,

    author=__author__,

    author_email='kallithea@sfconservancy.org',

    dependency_links=dependency_links,

    url=__url__,

    install_requires=requirements,

    classifiers=classifiers,

    data_files=data_files,

    packages=packages,

    include_package_data=True,

    message_extractors={'kallithea': [

            ('**.py', 'python', None),

            ('templates/**.mako', 'mako', {'input_encoding': 'utf-8'}),

            ('templates/**.html', 'mako', {'input_encoding': 'utf-8'}),

            ('public/**', 'ignore', None)]},

    zip_safe=False,

    entry_points="""

    [console_scripts]

    kallithea-api =    kallithea.bin.kallithea_api:main

    kallithea-gist =   kallithea.bin.kallithea_gist:main

    kallithea-config = kallithea.bin.kallithea_config:main

    kallithea-cli =    kallithea.bin.kallithea_cli:cli

    [paste.app_factory]

    main = kallithea.config.middleware:make_app

    [gearbox.commands]

    cache-keys=kallithea.lib.paster_commands.cache_keys:Command

    celeryd=kallithea.lib.paster_commands.celeryd:Command

    cleanup-repos=kallithea.lib.paster_commands.cleanup:Command

    install-iis=kallithea.lib.paster_commands.install_iis:Command

    ishell=kallithea.lib.paster_commands.ishell:Command

    make-index=kallithea.lib.paster_commands.make_index:Command

    make-rcext=kallithea.lib.paster_commands.make_rcextensions:Command

    repo-scan=kallithea.lib.paster_commands.repo_scan:Command

    setup-db=kallithea.lib.paster_commands.setup_db:Command

    update-repoinfo=kallithea.lib.paster_commands.update_repoinfo:Command

    upgrade-db=kallithea.lib.dbmigrate:UpgradeDb

    """,

)