.. _contributing:

=========================

Contributing to Kallithea

=========================

Kallithea is developed and maintained by its users. Please join us and scratch

your own itch.

Infrastructure

--------------

The main repository is hosted on Our Own Kallithea (aka OOK) at

https://kallithea-scm.org/repos/kallithea/, our self-hosted instance

of Kallithea.

For now, we use Bitbucket_ for `pull requests`_ and `issue tracking`_. The

issue tracker is for tracking bugs, not for support, discussion, or ideas --

please use the `mailing list`_ or :ref:`IRC <readme>` to reach the community.

We use Weblate_ to translate the user interface messages into languages other

than English. Join our project on `Hosted Weblate`_ to help us.

To register, you can use your Bitbucket or GitHub account. See :ref:`translations`

for more details.

Getting started

---------------

To get started with Kallithea development::

        hg clone https://kallithea-scm.org/repos/kallithea

        cd kallithea

        virtualenv ../kallithea-venv

        source ../kallithea-venv/bin/activate

        pip install --upgrade pip setuptools

        pip install --upgrade -e .

        pip install --upgrade -r dev_requirements.txt

        npm install     # install dependencies - both tools and data

        npm run less    # for generating css from less

        gearbox make-config my.ini

        gearbox setup-db -c my.ini --user=user --email=user@example.com --password=password --repos=/tmp

        gearbox serve -c my.ini --reload &

        firefox http://127.0.0.1:5000/

If you plan to use Bitbucket_ for sending contributions, you can also fork

Kallithea on Bitbucket_ first (https://bitbucket.org/conservancy/kallithea) and

then replace the clone step above by a clone of your fork. In this case, please

see :ref:`contributing-guidelines` below for configuring your fork correctly.

Contribution flow

-----------------

Starting from an existing Kallithea clone, make sure it is up to date with the

latest upstream changes::

        hg pull

        hg update

Review the :ref:`contributing-guidelines` and :ref:`coding-guidelines`.

If you are new to Mercurial, refer to Mercurial `Quick Start`_ and `Beginners

Guide`_ on the Mercurial wiki.

Now, make some changes and test them (see :ref:`contributing-tests`). Don't

forget to add new tests to cover new functionality or bug fixes.

For documentation changes, run ``make html`` from the ``docs`` directory to

generate the HTML result, then review them in your browser.

Before submitting any changes, run the cleanup script::

        ./scripts/run-all-cleanup

When you are completely ready, you can send your changes to the community for

review and inclusion. Most commonly used methods are sending patches to the

mailing list (via ``hg email``) or by creating a pull request on Bitbucket_.

.. _contributing-tests:

Running tests

-------------

After finishing your changes make sure all tests pass cleanly. Run the testsuite

by invoking ``py.test`` from the project root::

    py.test

Note that testing on Python 2.6 also requires ``unittest2``.

Note that on unix systems, the temporary directory (``/tmp`` or where

``$TMPDIR`` points) must allow executable files; Git hooks must be executable,

and the test suite creates repositories in the temporary directory. Linux

systems with /tmp mounted noexec will thus fail.

You can also use ``tox`` to run the tests with all supported Python versions

(currently Python 2.6--2.7).

It is possible to avoid recreating the full test database on each invocation of

the tests, thus eliminating the initial delay. To achieve this, run the tests as::

    KALLITHEA_WHOOSH_TEST_DISABLE=1 KALLITHEA_NO_TMP_PATH=1 py.test

    kill -9 $(cat test.pid)

In these commands, the following variables are used::

    KALLITHEA_WHOOSH_TEST_DISABLE=1 - skip whoosh index building and tests

    KALLITHEA_NO_TMP_PATH=1 - disable new temp path for tests, used mostly for testing_vcs_operations

You can run individual tests by specifying their path as argument to py.test.

py.test also has many more options, see `py.test -h`. Some useful options

are::

    -k EXPRESSION         only run tests which match the given substring

                          expression. An expression is a python evaluable

                          expression where all names are substring-matched

                          against test names and their parent classes. Example:

    -x, --exitfirst       exit instantly on first error or failed test.

    --lf                  rerun only the tests that failed at the last run (or

                          all if none failed)

    --ff                  run all tests but run the last failures first. This

                          may re-order tests and thus lead to repeated fixture

                          setup/teardown

    --pdb                 start the interactive Python debugger on errors.

    -s, --capture=no      don't capture stdout (any stdout output will be

                          printed immediately)

Performance tests

^^^^^^^^^^^^^^^^^

A number of performance tests are present in the test suite, but they are

not run in a standard test run. These tests are useful to

evaluate the impact of certain code changes with respect to performance.

To run these tests::

    env TEST_PERFORMANCE=1 py.test kallithea/tests/performance

To analyze performance, you could install pytest-profiling_, which enables the

--profile and --profile-svg options to py.test.

.. _pytest-profiling: https://github.com/manahl/pytest-plugins/tree/master/pytest-profiling

.. _contributing-guidelines:

Contribution guidelines

-----------------------

Kallithea is GPLv3 and we assume all contributions are made by the

committer/contributor and under GPLv3 unless explicitly stated. We do care a

lot about preservation of copyright and license information for existing code

that is brought into the project.

Contributions will be accepted in most formats -- such as pull requests on

Bitbucket, something hosted on your own Kallithea instance, or patches sent by

email to the `kallithea-general`_ mailing list.

When contributing via Bitbucket, please make your fork of

https://bitbucket.org/conservancy/kallithea/ `non-publishing`_ -- it is one of

the settings on "Repository details" page. This ensures your commits are in

"draft" phase and makes it easier for you to address feedback and for project

maintainers to integrate your changes.

.. _non-publishing: https://www.mercurial-scm.org/wiki/Phases#Publishing_Repository

Make sure to test your changes both manually and with the automatic tests

before posting.

We care about quality and review and keeping a clean repository history. We

might give feedback that requests polishing contributions until they are

"perfect". We might also rebase and collapse and make minor adjustments to your

changes when we apply them.

We try to make sure we have consensus on the direction the project is taking.

Everything non-sensitive should be discussed in public -- preferably on the

mailing list.  We aim at having all non-trivial changes reviewed by at least

one other core developer before pushing. Obvious non-controversial changes will

be handled more casually.

There is a main development branch ("default") which is generally stable so that

it can be (and is) used in production. There is also a "stable" branch that is

almost exclusively reserved for bug fixes or trivial changes. Experimental

changes should live elsewhere (for example in a pull request) until they are

ready.

.. _coding-guidelines:

Coding guidelines

-----------------

We don't have a formal coding/formatting standard. We are currently using a mix

of Mercurial's (https://www.mercurial-scm.org/wiki/CodingStyle), pep8, and

consistency with existing code. Run ``scripts/run-all-cleanup`` before

committing to ensure some basic code formatting consistency.

We support both Python 2.6.x and 2.7.x and nothing else. For now we don't care

about Python 3 compatibility.

We try to support the most common modern web browsers. IE9 is still supported

to the extent it is feasible, IE8 is not.

We primarily support Linux and OS X on the server side but Windows should also work.

HTML templates should use 2 spaces for indentation ... but be pragmatic. We

should use templates cleverly and avoid duplication. We should use reasonable

semantic markup with element classes and IDs that can be used for styling and testing.

We should only use inline styles in places where it really is semantic (such as

``display: none``).

JavaScript must use ``;`` between/after statements. Indentation 4 spaces. Inline

multiline functions should be indented two levels -- one for the ``()`` and one for

``{}``.

Variables holding jQuery objects should be named with a leading ``$``.

Commit messages should have a leading short line summarizing the changes. For

bug fixes, put ``(Issue #123)`` at the end of this line.

Use American English grammar and spelling overall. Use `English title case`_ for

page titles, button labels, headers, and 'labels' for fields in forms.

.. _English title case: https://en.wikipedia.org/wiki/Capitalization#Title_case

Template helpers (that is, everything in ``kallithea.lib.helpers``)

should only be referenced from templates. If you need to call a

helper from the Python code, consider moving the function somewhere

else (e.g. to the model).

Notes on the SQLAlchemy session

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Each HTTP request runs inside an independent SQLAlchemy session (as well

as in an independent database transaction). ``Session`` is the session manager

and factory. ``Session()`` will create a new session on-demand or return the

current session for the active thread. Many database operations are methods on

such session instances - only ``Session.remove()`` should be called directly on

the manager.

Database model objects

(almost) always belong to a particular SQLAlchemy session, which means

that SQLAlchemy will ensure that they're kept in sync with the database

(but also means that they cannot be shared across requests).

Objects can be added to the session using ``Session().add``, but this is

rarely needed:

* When creating a database object by calling the constructor directly,

  it must explicitly be added to the session.

* When creating an object using a factory function (like

  ``create_repo``), the returned object has already (by convention)

  been added to the session, and should not be added again.

* When getting an object from the session (via ``Session().query`` or

  any of the utility functions that look up objects in the database),

  it's already part of the session, and should not be added again.

  SQLAlchemy monitors attribute modifications automatically for all

  objects it knows about and syncs them to the database.

SQLAlchemy also flushes changes to the database automatically; manually

calling ``Session().flush`` is usually only necessary when the Python

code needs the database to assign an "auto-increment" primary key ID to

a freshly created model object (before flushing, the ID attribute will

be ``None``).

TurboGears2 DebugBar

^^^^^^^^^^^^^^^^^^^^

It is possible to enable the TurboGears2-provided DebugBar_, a toolbar overlayed

over the Kallithea web interface, allowing you to see:

* timing information of the current request, including profiling information

* request data, including GET data, POST data, cookies, headers and environment

  variables

* a list of executed database queries, including timing and result values

DebugBar is only activated when ``debug = true`` is set in the configuration

file. This is important, because the DebugBar toolbar will be visible for all

users, and allow them to see information they should not be allowed to see. Like

is anyway the case for ``debug = true``, do not use this in production!

To enable DebugBar, install ``tgext.debugbar`` and ``kajiki`` (typically via

``pip``) and restart Kallithea (in debug mode).

"Roadmap"

---------

We do not have a road map but are waiting for your contributions. Refer to the

wiki_ for some ideas of places we might want to go -- contributions in these

areas are very welcome.

Thank you for your contribution!

--------------------------------

.. _Weblate: http://weblate.org/

.. _issue tracking: https://bitbucket.org/conservancy/kallithea/issues?status=new&status=open

.. _pull requests: https://bitbucket.org/conservancy/kallithea/pull-requests

.. _bitbucket: http://bitbucket.org/

.. _mailing list: http://lists.sfconservancy.org/mailman/listinfo/kallithea-general

.. _kallithea-general: http://lists.sfconservancy.org/mailman/listinfo/kallithea-general

.. _Hosted Weblate: https://hosted.weblate.org/projects/kallithea/kallithea/

.. _wiki: https://bitbucket.org/conservancy/kallithea/wiki/Home

.. _DebugBar: https://github.com/TurboGears/tgext.debugbar

.. _Quick Start: https://www.mercurial-scm.org/wiki/QuickStart

.. _Beginners Guide: https://www.mercurial-scm.org/wiki/BeginnersGuides

============

Translations

============

Translations are available on Hosted Weblate at the following URL:

    https://hosted.weblate.org/projects/kallithea/kallithea/

Registered users may contribute to the existing languages, or request a new

language translations.

Translating using Weblate

-------------------------

Weblate_ offers a simple and easy to use interface featuring glossary, machine

translation, suggestions based on similar translations in other projects,

automatic checks etc. Weblate imports the source code tree directly from

the version control system, and commits edits back from time to time.

When registering at Weblate, make sure you name and email address you prefer to

be used when your changes are committed. We can and probably will amend changesets

coming from Weblate, but having things right from the beginning makes things easier.

Weblate performs sanity checks all the time and tries to prevent you from ignoring

them. Most common mistakes are inconsistent punctuation, whitespaces, missing or extra

format parameters, untranslated strings copied into the translation. Please perform

necessary corrections when they're needed, or override the false positives.

Merging translations from Weblate

---------------------------------

Weblate rebases its changes every time it pulls from our repository. Pulls are triggered

by a web hook from Our Own Kallithea every time it receives new commits. Usually merging

the new translations is a straightforward process consisting of a pull from Weblate-hosted

repository which is available under Data Exports tab in Weblate interface.

Weblate tries to minimise the number of commits, but that's not always work, especially

when two translators work with different languages at more or less the same time.

It makes sense sometimes to re-order or fold commits by the same author when they touch

just the same language translation. That, however, may confuse Weblate sometimes, in

which case it should be manually convinced it has to discard the commits it created by

using its administrative interface.

Manual creation of a new language translation

---------------------------------------------

In the prepared development environment, run the following to ensure

all translation strings are extracted and up-to-date::

    python2 setup.py extract_messages

Create new language by executing following command::

    python2 setup.py init_catalog -l <new_language_code>

This creates a new translation under directory `kallithea/i18n/<new_language_code>`

based on the translation template file, `kallithea/i18n/kallithea.pot`.

Edit the new PO file located in `LC_MESSAGES` directory with poedit or your

favorite PO files editor. After you finished with the translations, check the

translation file for errors by executing::

    msgfmt -f -c kallithea/i18n/<new_language_code>/LC_MESSAGES/<updated_file.po>

Finally, compile the translations::

    python2 setup.py compile_catalog -l <new_language_code>

Updating translations

---------------------

Extract the latest versions of strings for translation by running::

    python2 setup.py extract_messages

Update the PO file by doing::

    python2 setup.py update_catalog -l <new_language_code>

Edit the new updated translation file. Repeat all steps after `init_catalog` step from

new translation instructions

Testing translations

--------------------

    py.test

.. _Weblate: http://weblate.org/

import os

import re

import sys

import logging

import pkg_resources

import time

import formencode

from paste.deploy import loadwsgi

from routes.util import URLGenerator

import pytest

from pytest_localserver.http import WSGIServer

from kallithea.controllers.root import RootController

from kallithea.lib import inifile

from kallithea.lib.utils import repo2db_mapper

from kallithea.model.user import UserModel

from kallithea.model.meta import Session

from kallithea.model.db import Setting, User, UserIpMap

from kallithea.model.scm import ScmModel

from kallithea.tests.base import invalidate_all_caches, TEST_USER_REGULAR_LOGIN, TESTS_TMP_PATH, \

    TEST_USER_ADMIN_LOGIN, TEST_USER_ADMIN_PASS

import kallithea.tests.base # FIXME: needed for setting testapp instance!!!

from tg.util.webtest import test_context

def pytest_configure():

    os.environ['TZ'] = 'UTC'

    if not kallithea.is_windows:

        time.tzset() # only available on Unix

    path = os.getcwd()

    sys.path.insert(0, path)

    pkg_resources.working_set.add_entry(path)

    # Disable INFO logging of test database creation, restore with NOTSET

    logging.disable(logging.INFO)

    ini_settings = {

        '[server:main]': {

            'port': '4999',

        },

        '[app:main]': {

            'app_instance_uuid': 'test',

            'show_revision_number': 'true',

            'beaker.cache.sql_cache_short.expire': '1',

            'beaker.session.secret': '{74e0cd75-b339-478b-b129-07dd221def1f}',

        },

        '[handler_console]': {

            'formatter': 'color_formatter',

        },

        # The 'handler_console_sql' block is very similar to the one in

        # development.ini, but without the explicit 'level=DEBUG' setting:

        # it causes duplicate sqlalchemy debug logs, one through

        # handler_console_sql and another through another path.

        '[handler_console_sql]': {

            'formatter': 'color_formatter_sql',

        },

    }

    if os.environ.get('TEST_DB'):

        ini_settings['[app:main]']['sqlalchemy.url'] = os.environ.get('TEST_DB')

    test_ini_file = os.path.join(TESTS_TMP_PATH, 'test.ini')

    inifile.create(test_ini_file, None, ini_settings)

    context = loadwsgi.loadcontext(loadwsgi.APP, 'config:%s' % test_ini_file)

    from kallithea.tests.fixture import create_test_env, create_test_index

    # set KALLITHEA_NO_TMP_PATH=1 to disable re-creating the database and test repos

    if not int(os.environ.get('KALLITHEA_NO_TMP_PATH', 0)):

        create_test_env(TESTS_TMP_PATH, context.config())

    # set KALLITHEA_WHOOSH_TEST_DISABLE=1 to disable whoosh index during tests

    if not int(os.environ.get('KALLITHEA_WHOOSH_TEST_DISABLE', 0)):

        create_test_index(TESTS_TMP_PATH, context.config(), True)

    kallithea.tests.base.testapp = context.create()

    # do initial repo scan

    repo2db_mapper(ScmModel().repo_scan(TESTS_TMP_PATH))

    logging.disable(logging.NOTSET)

    kallithea.tests.base.url = URLGenerator(RootController().mapper, {'HTTP_HOST': 'example.com'})

    # set fixed language for form messages, regardless of environment settings

    formencode.api.set_stdtranslation(languages=[])

@pytest.fixture

def create_test_user():

    """Provide users that automatically disappear after test is over."""

    test_user_ids = []

    def _create_test_user(user_form):

        user = UserModel().create(user_form)

        test_user_ids.append(user.user_id)

        return user

    yield _create_test_user

    for user_id in test_user_ids:

        UserModel().delete(user_id)

    Session().commit()

def _set_settings(*kvtseq):

    session = Session()

    for kvt in kvtseq:

        assert len(kvt) in (2, 3)

        k = kvt[0]

        v = kvt[1]

        t = kvt[2] if len(kvt) == 3 else 'unicode'

        Setting.create_or_update(k, v, t)

    session.commit()

@pytest.fixture

def set_test_settings():

    """Restore settings after test is over."""

    # Save settings.

    settings_snapshot = [

        (s.app_settings_name, s.app_settings_value, s.app_settings_type)

        for s in Setting.query().all()]

    yield _set_settings

    # Restore settings.

    session = Session()

    keys = frozenset(k for (k, v, t) in settings_snapshot)

    for s in Setting.query().all():

        if s.app_settings_name not in keys:

            session.delete(s)

    for k, v, t in settings_snapshot:

        if t == 'list' and hasattr(v, '__iter__'):

            v = ','.join(v) # Quirk: must format list value manually.

        Setting.create_or_update(k, v, t)

    session.commit()

@pytest.fixture

def auto_clear_ip_permissions():

    """Fixture that provides nothing but clearing IP permissions upon test

    exit. This clearing is needed to avoid other test failing to make fake http

    accesses."""

    yield

    # cleanup

    user_model = UserModel()

    user_ids = []

    user_ids.append(User.get_default_user().user_id)

    user_ids.append(User.get_by_username(TEST_USER_REGULAR_LOGIN).user_id)

    for user_id in user_ids:

        for ip in UserIpMap.query().filter(UserIpMap.user_id == user_id):

            user_model.delete_extra_ip(user_id, ip.ip_id)

    # IP permissions are cached, need to invalidate this cache explicitly

    invalidate_all_caches()

@pytest.fixture

def test_context_fixture(app_fixture):

    """

    Encompass the entire test using this fixture in a test_context,

    making sure that certain functionality still works even if no call to

    self.app.get/post has been made.

    The typical error message indicating you need a test_context is:

        TypeError: No object (name: context) has been registered for this thread

    The standard way to fix this is simply using the test_context context

    manager directly inside your test:

        with test_context(self.app):

            <actions>

    but if test setup code (xUnit-style or pytest fixtures) also needs to be

    executed inside the test context, that method is not possible.

    Even if there is no such setup code, the fixture may reduce code complexity

    if the entire test needs to run inside a test context.

    To apply this fixture (like any other fixture) to all test methods of a

    class, use the following class decorator:

        @pytest.mark.usefixtures("test_context_fixture")

        class TestFoo(TestController):

            ...

    """

    with test_context(app_fixture):

        yield

class MyWSGIServer(WSGIServer):

    def repo_url(self, repo_name, username=TEST_USER_ADMIN_LOGIN, password=TEST_USER_ADMIN_PASS):

        """Return URL to repo on this web server."""

        host, port = self.server_address

        proto = 'http' if self._server.ssl_context is None else 'https'

        auth = ''

        if username is not None:

            auth = username

            if password is not None:

                auth += ':' + password

        if auth:

            auth += '@'

        return '%s://%s%s:%s/%s' % (proto, auth, host, port, repo_name)

@pytest.yield_fixture(scope="session")

def webserver():

    """Start web server while tests are running.

    Useful for debugging and necessary for vcs operation tests."""

    server = MyWSGIServer(application=kallithea.tests.base.testapp)

    server.start()

    yield server

    server.stop()

#!/usr/bin/env python2

"""

"""

import re

from kallithea.lib import inifile

# files to be generated from the mako template

ini_files = [

    ('development.ini',

        {

            '[server:main]': {

                'host': '0.0.0.0',

            },

            '[app:main]': {

                'debug': 'true',

                'app_instance_uuid': 'development-not-secret',

                'beaker.session.secret': 'development-not-secret',

            },

            '[handler_console]': {

                'formatter': 'color_formatter',

            },

            '[handler_console_sql]': {

                'formatter': 'color_formatter_sql',

            },

            '[logger_routes]': {

                'level': 'DEBUG',

            },

            '[logger_beaker]': {

                'level': 'DEBUG',

            },

            '[logger_templates]': {

                'level': 'INFO',

            },

            '[logger_kallithea]': {

                'level': 'DEBUG',

            },

            '[logger_tg]': {

                'level': 'DEBUG',

            },

            '[logger_gearbox]': {

                'level': 'DEBUG',

            },

            '[logger_whoosh_indexer]': {

                'level': 'DEBUG',

            },

        },

    ),

]

def main():

    # make sure all mako lines starting with '#' (the '##' comments) are marked up as <text>

    makofile = inifile.template_file

    print 'reading:', makofile

    mako_org = open(makofile).read()

    mako_no_text_markup = re.sub(r'</?%text>', '', mako_org)

    mako_marked_up = re.sub(r'\n(##.*)', r'\n<%text>\1</%text>', mako_no_text_markup, flags=re.MULTILINE)

    if mako_marked_up != mako_org:

        print 'writing:', makofile

        open(makofile, 'w').write(mako_marked_up)

    # create ini files

    for fn, settings in ini_files:

        print 'updating:', fn

        inifile.create(fn, None, settings)

if __name__ == '__main__':

    main()