Changeset - 2c3d30095d5e
[Not reviewed]
default
1 39 0
Mads Kiilerich - 9 years ago 2016-10-25 21:32:24
madski@unity3d.com
gearbox: replace paster with something TurboGears2-ish that still works with the Pylons stack

This is a step towards moving away from the Pylons stack to TurboGears2, but
still independent of it.


Some notes from the porting - it could perhaps be the missing(?) documentation
for migrating from paster to gearbox:

Note: 'gearbox' without parameters will crash - specify '-h' to get started
testing.

Replace paster
summary = 'yada yada'
with the first line of the docstring of the Command class ... or override
get_description.

Note: All newlines in the docstring will be collapsed and mangle the long help
text.

Grouping of commands is not possible. Standard commands (for development) can't
be customized under the same name or hidden. (Like for paster, the conceptual
model also assumes that the sub-command naming is namespaced so commands from
other packages won't conflict.)

The usage help is fully automated from the declared options.

For all deprecated Commands, replace paster
hidden = True
with gearbox
deprecated = True

Note: config_file, takes_config_file, min_args and max_args are not available /
relevant.

The gearbox parser is customized by overriding get_parser - there is nothing
like paster update_parser.

Gearbox is using argparse instead of optparse ... but argparse add_argument is
mostly backwards compatible with optparse add_option.

Instead of overriding command or run as in paster, override take_action in
gearbox. The parsed arguments are passed to take_action, not available on the
command instance.

Paster BadCommand is not available and must be handled manually, terminating
with sys.exit(1).

There is no standard make-config command in gearbox.

Paster appinstall has been replaced by the somewhat different setup_app module
in gearbox. There is still no clean way to pass parameters to SetupAppCommand
and it relies on websetup and other apparently unnecessary complexity. Instead,
implement setup-db from scratch.


Minor change by Thomas De Schampheleire: add gearbox logging configuration.
Because we use logging.config.fileConfig(.inifile) during gearbox command
execution, the logging settings need to be correct and contain a block for
gearbox logging itself. Otherwise, errors in command processing are not even
visible and the command exits silently.
40 files changed with 356 insertions and 439 deletions:
0 comments (0 inline, 0 general)
development.ini
Show inline comments
 
@@ -67,8 +67,8 @@ pdebug = false
 
#smtp_use_tls = false
 

	
 
[server:main]
 
## PASTE ##
 
#use = egg:Paste#http
 
## Gearbox default web server ##
 
#use = egg:gearbox#wsgiref
 
## nr of worker threads to spawn
 
#threadpool_workers = 1
 
## max request before thread respawn
 
@@ -76,6 +76,9 @@ pdebug = false
 
## option to use threads of process
 
#use_threadpool = true
 

	
 
## Gearbox gevent web server ##
 
#use = egg:gearbox#gevent
 

	
 
## WAITRESS ##
 
use = egg:waitress#main
 
## number of worker threads
 
@@ -509,7 +512,7 @@ script_location = kallithea:alembic
 
################################
 

	
 
[loggers]
 
keys = root, routes, kallithea, sqlalchemy, beaker, templates, whoosh_indexer
 
keys = root, routes, kallithea, sqlalchemy, gearbox, beaker, templates, whoosh_indexer
 

	
 
[handlers]
 
keys = console, console_sql
 
@@ -550,6 +553,12 @@ handlers =
 
qualname = kallithea
 
propagate = 1
 

	
 
[logger_gearbox]
 
level = DEBUG
 
handlers =
 
qualname = gearbox
 
propagate = 1
 

	
 
[logger_sqlalchemy]
 
level = INFO
 
handlers = console_sql
docs/contributing.rst
Show inline comments
 
@@ -36,9 +36,9 @@ To get started with development::
 
        source ../kallithea-venv/bin/activate
 
        pip install --upgrade pip setuptools
 
        pip install -e .
 
        paster make-config Kallithea my.ini
 
        paster setup-db my.ini --user=user --email=user@example.com --password=password --repos=/tmp
 
        paster serve my.ini --reload &
 
        TODO make-config Kallithea my.ini
 
        gearbox setup-db -c my.ini --user=user --email=user@example.com --password=password --repos=/tmp
 
        gearbox serve -c my.ini --reload &
 
        firefox http://127.0.0.1:5000/
 

	
 
You can also start out by forking https://bitbucket.org/conservancy/kallithea
 
@@ -66,7 +66,7 @@ SQLite database specified there.
 
It is possible to avoid recreating the full test database on each invocation of
 
the tests, thus eliminating the initial delay. To achieve this, run the tests as::
 

	
 
    paster serve kallithea/tests/test.ini --pid-file=test.pid --daemon
 
    gearbox serve -c kallithea/tests/test.ini --pid-file=test.pid --daemon
 
    KALLITHEA_WHOOSH_TEST_DISABLE=1 KALLITHEA_NO_TMP_PATH=1 py.test
 
    kill -9 $(cat test.pid)
 

	
docs/dev/dbmigrations.rst
Show inline comments
 
@@ -18,7 +18,7 @@ changes, you should make a matching Alem
 
2. Create a separate throwaway configuration for iterating on the actual
 
   database changes::
 

	
 
    paster make-config Kallithea temp.ini
 
    TODO make-config Kallithea temp.ini
 

	
 
   Edit the file to change database settings. SQLite is typically fine,
 
   but make sure to change the path to e.g. ``temp.db``, to avoid
 
@@ -30,8 +30,8 @@ changes, you should make a matching Alem
 
   to test the changes::
 

	
 
    rm temp.db
 
    paster setup-db temp.ini --repos=/var/repos --user=doe --email doe@example.com --password=123456 --no-public-access --force-yes
 
    paster repo-scan temp.ini
 
    gearbox setup-db -c temp.ini --repos=/var/repos --user=doe --email doe@example.com --password=123456 --no-public-access --force-yes
 
    gearbox repo-scan -c temp.ini
 

	
 
5. Once satisfied with the schema changes, auto-generate a draft Alembic
 
   script using the development database that has *not* been upgraded.
docs/installation_iis.rst
Show inline comments
 
@@ -59,7 +59,7 @@ ISAPI handler
 

	
 
The ISAPI handler can be generated using::
 

	
 
    paster install-iis my.ini --virtualdir=/
 
    gearbox install-iis -c my.ini --virtualdir=/
 

	
 
This will generate a ``dispatch.py`` file in the current directory that contains
 
the necessary components to finalize an installation into IIS. Once this file
 
@@ -74,7 +74,7 @@ This accomplishes two things: generating
 

	
 
The ISAPI handler is registered to all file extensions, so it will automatically
 
be the one handling all requests to the specified virtual directory. When the website starts
 
the ISAPI handler, it will start a thread pool managed wrapper around the paster
 
the ISAPI handler, it will start a thread pool managed wrapper around the
 
middleware WSGI handler that Kallithea runs within and each HTTP request to the
 
site will be processed through this logic henceforth.
 

	
 
@@ -111,7 +111,7 @@ Troubleshooting
 
---------------
 

	
 
Typically, any issues in this setup will either be entirely in IIS or entirely
 
in Kallithea (or Kallithea's WSGI/paster middleware). Consequently, two
 
in Kallithea (or Kallithea's WSGI middleware). Consequently, two
 
different options for finding issues exist: IIS' failed request tracking which
 
is great at finding issues until they exist inside Kallithea, at which point the
 
ISAPI-WSGI wrapper above uses ``win32traceutil``, which is part of ``pywin32``.
docs/installation_win.rst
Show inline comments
 
@@ -166,7 +166,7 @@ it, reopen it following the same command
 
one). When ready, type::
 

	
 
  cd C:\Kallithea\Bin
 
  paster make-config Kallithea production.ini
 
  TODO make-config Kallithea production.ini
 

	
 
Then you must edit production.ini to fit your needs (IP address, IP
 
port, mail settings, database, etc.). `NotePad++`__ or a similar text
 
@@ -177,7 +177,7 @@ __ http://notepad-plus-plus.org/
 

	
 
For the sake of simplicity, run it with the default settings. After your edits (if any) in the previous command prompt, type::
 

	
 
  paster setup-db production.ini
 
  gearbox setup-db -c production.ini
 

	
 
.. warning:: This time a *new* database will be installed. You must
 
             follow a different process to later :ref:`upgrade <upgrade>`
 
@@ -200,7 +200,7 @@ Step 10 -- Running Kallithea
 

	
 
In the previous command prompt, being in the C:\\Kallithea\\Bin folder, type::
 

	
 
  paster serve production.ini
 
  gearbox serve -c production.ini
 

	
 
Open your web server, and go to http://127.0.0.1:5000
 

	
docs/installation_win_old.rst
Show inline comments
 
@@ -204,7 +204,7 @@ if you closed it reopen it following the
 
"activate" one). When ready, just type::
 

	
 
  cd C:\Kallithea\Bin
 
  paster make-config Kallithea production.ini
 
  TODO make-config Kallithea production.ini
 

	
 
Then, you must edit production.ini to fit your needs (network address and
 
port, mail settings, database, whatever). I recommend using NotePad++
 
@@ -215,7 +215,7 @@ character differences between Unix and W
 
For the sake of simplicity lets run it with the default settings. After
 
your edits (if any), in the previous Command Prompt, type::
 

	
 
 paster setup-db production.ini
 
 gearbox setup-db -c production.ini
 

	
 
.. warning:: This time a *new* database will be installed. You must
 
             follow a different process to later :ref:`upgrade <upgrade>`
 
@@ -239,7 +239,7 @@ Step 9 -- Running Kallithea
 
In the previous command prompt, being in the C:\\Kallithea\\Bin folder,
 
just type::
 

	
 
 paster serve production.ini
 
 gearbox serve -c production.ini
 

	
 
Open yout web server, and go to http://127.0.0.1:5000
 

	
docs/overview.rst
Show inline comments
 
@@ -84,17 +84,17 @@ to use web server authentication.
 

	
 
There are several web server options:
 

	
 
- Kallithea uses the Paste_ tool as command line interface. Paste provides
 
  ``paster serve`` as a convenient way to launch a Python WSGI / web server
 
- Kallithea uses the Gearbox_ tool as command line interface. Gearbox provides
 
  ``gearbox serve`` as a convenient way to launch a Python WSGI / web server
 
  from the command line. That is perfect for development and evaluation.
 
  Actual use in production might have different requirements and need extra
 
  work to make it manageable as a scalable system service.
 

	
 
  Paste comes with its own built-in web server but Kallithea defaults to use
 
  Gearbox comes with its own built-in web server but Kallithea defaults to use
 
  Waitress_. Gunicorn_ is also an option. These web servers have different
 
  limited feature sets.
 

	
 
  The web server used by ``paster`` is configured in the ``.ini`` file passed
 
  The web server used by ``gearbox`` is configured in the ``.ini`` file passed
 
  to it. The entry point for the WSGI application is configured
 
  in ``setup.py`` as ``kallithea.config.middleware:make_app``.
 

	
 
@@ -113,7 +113,7 @@ There are several web server options:
 
  encryption or special authentication or for other security reasons, to
 
  provide caching of static files, or to provide load balancing or fail-over.
 
  Nginx_, Varnish_ and HAProxy_ are often used for this purpose, often in front
 
  of a ``paster`` server that somehow is wrapped as a service.
 
  of a ``gearbox serve`` that somehow is wrapped as a service.
 

	
 
The best option depends on what you are familiar with and the requirements for
 
performance and stability. Also, keep in mind that Kallithea mainly is serving
 
@@ -126,7 +126,7 @@ continuous hammering from the internet.
 
.. _Gunicorn: http://gunicorn.org/
 
.. _Waitress: http://waitress.readthedocs.org/en/latest/
 
.. _virtualenv: http://pypi.python.org/pypi/virtualenv
 
.. _Paste: http://pythonpaste.org/
 
.. _Gearbox: http://turbogears.readthedocs.io/en/latest/turbogears/gearbox.html
 
.. _PyPI: https://pypi.python.org/pypi
 
.. _Apache httpd: http://httpd.apache.org/
 
.. _mod_wsgi: https://code.google.com/p/modwsgi/
docs/setup.rst
Show inline comments
 
@@ -11,7 +11,7 @@ Setting up Kallithea
 
First, you will need to create a Kallithea configuration file. Run the
 
following command to do so::
 

	
 
    paster make-config Kallithea my.ini
 
    TODO make-config Kallithea my.ini
 

	
 
This will create the file ``my.ini`` in the current directory. This
 
configuration file contains the various settings for Kallithea, e.g.
 
@@ -25,7 +25,7 @@ configuration file to use this other dat
 
PostgreSQL, SQLite and MySQL databases. Create the database by running
 
the following command::
 

	
 
    paster setup-db my.ini
 
    gearbox setup-db -c my.ini
 

	
 
This will prompt you for a "root" path. This "root" path is the location where
 
Kallithea will store all of its repositories on the current machine. After
 
@@ -36,7 +36,7 @@ up for you.
 
The ``setup-db`` values can also be given on the command line.
 
Example::
 

	
 
    paster setup-db my.ini --user=nn --password=secret --email=nn@example.com --repos=/srv/repos
 
    gearbox setup-db -c my.ini --user=nn --password=secret --email=nn@example.com --repos=/srv/repos
 

	
 
The ``setup-db`` command will create all needed tables and an
 
admin account. When choosing a root path you can either use a new
 
@@ -54,7 +54,7 @@ path to the root).
 

	
 
You are now ready to use Kallithea. To run it simply execute::
 

	
 
    paster serve my.ini
 
    gearbox serve -c my.ini
 

	
 
- This command runs the Kallithea server. The web app should be available at
 
  http://127.0.0.1:5000. The IP address and port is configurable via the
 
@@ -111,23 +111,23 @@ Kallithea provides full text search of r
 

	
 
For an incremental index build, run::
 

	
 
    paster make-index my.ini
 
    gearbox make-index -c my.ini
 

	
 
For a full index rebuild, run::
 

	
 
    paster make-index my.ini -f
 
    gearbox make-index -c my.ini -f
 

	
 
The ``--repo-location`` option allows the location of the repositories to be overridden;
 
usually, the location is retrieved from the Kallithea database.
 

	
 
The ``--index-only`` option can be used to limit the indexed repositories to a comma-separated list::
 

	
 
    paster make-index my.ini --index-only=vcs,kallithea
 
    gearbox make-index -c my.ini --index-only=vcs,kallithea
 

	
 
To keep your index up-to-date it is necessary to do periodic index builds;
 
for this, it is recommended to use a crontab entry. Example::
 

	
 
    0  3  *  *  *  /path/to/virtualenv/bin/paster make-index /path/to/kallithea/my.ini
 
    0  3  *  *  *  /path/to/virtualenv/bin/gearbox make-index -c /path/to/kallithea/my.ini
 

	
 
When using incremental mode (the default), Whoosh will check the last
 
modification date of each file and add it to be reindexed if a newer file is
 
@@ -592,7 +592,10 @@ Celery. So for example setting `BROKER_H
 

	
 
To start the Celery process, run::
 

	
 
 paster celeryd <configfile.ini>
 
 gearbox celeryd -c <configfile.ini>
 

	
 
Extra options to the Celery worker can be passed after ``--`` - see ``-- -h``
 
for more info.
 

	
 
.. note::
 
   Make sure you run this command from the same virtualenv, and with the same
docs/upgrade.rst
Show inline comments
 
@@ -93,7 +93,7 @@ If you originally installed from version
 

	
 
Run the following command to upgrade your configuration (``.ini``) file::
 

	
 
    paster make-config Kallithea my.ini
 
    TODO make-config Kallithea my.ini
 

	
 
This will display any changes made by the new version of Kallithea to your
 
current configuration, and attempt an automatic merge. It is recommended
docs/usage/customization.rst
Show inline comments
 
@@ -32,7 +32,7 @@ overwrite an entire function, change a g
 

	
 
To generate a skeleton extensions package, run::
 

	
 
    paster make-rcext my.ini
 
    gearbox make-rcext -c my.ini
 

	
 
This will create an ``rcextensions`` package next to the specified ``ini`` file.
 
See the ``__init__.py`` file inside the generated ``rcextensions`` package
docs/usage/general.rst
Show inline comments
 
@@ -15,7 +15,7 @@ or access the repository.
 

	
 
There is a special command for cleaning up such archived repositories::
 

	
 
    paster cleanup-repos --older-than=30d my.ini
 
    gearbox cleanup-repos --older-than=30d -c my.ini
 

	
 
This command scans for archived repositories that are older than
 
30 days, displays them, and asks if you want to delete them (unless given
docs/usage/vcs_support.rst
Show inline comments
 
@@ -28,10 +28,10 @@ Web server with chunked encoding
 
Large Git pushes require an HTTP server with support for
 
chunked encoding for POST. The Python web servers waitress_ and
 
gunicorn_ (Linux only) can be used. By default, Kallithea uses
 
waitress_ for `paster serve` instead of the built-in `paste` WSGI
 
waitress_ for `gearbox serve` instead of the built-in `paste` WSGI
 
server.
 

	
 
The paster server is controlled in the .ini file::
 
The web server used by gearbox is controlled in the .ini file::
 

	
 
    use = egg:waitress#main
 

	
init.d/celeryd-upstart.conf
Show inline comments
 
@@ -21,7 +21,7 @@ env USER=hg
 
# env GROUP=hg
 

	
 
script
 
    COMMAND="/var/hg/.virtualenvs/kallithea/bin/paster celeryd $APPINI --pidfile=$PIDFILE"
 
    COMMAND="/var/hg/.virtualenvs/kallithea/bin/gearbox celeryd -c $APPINI -- --pidfile=$PIDFILE"
 
    if [ -z "$GROUP" ]; then
 
        exec sudo -u $USER $COMMAND
 
    else
init.d/kallithea-daemon-arch
Show inline comments
 
@@ -12,7 +12,7 @@ APP_PATH="$APP_HOMEDIR/$DAEMON"
 
CONF_NAME="production.ini"
 
LOG_FILE="/var/log/$DAEMON.log"
 
PID_FILE="/run/daemons/$DAEMON"
 
APPL=/usr/bin/paster
 
APPL=/usr/bin/gearbox
 
RUN_AS="*****"
 

	
 
ARGS="serve --daemon \
 
@@ -20,7 +20,7 @@ ARGS="serve --daemon \
 
--group=$RUN_AS \
 
--pid-file=$PID_FILE \
 
--log-file=$LOG_FILE \
 
$APP_PATH/$CONF_NAME"
 
-c $APP_PATH/$CONF_NAME"
 

	
 
[ -r /etc/conf.d/$DAEMON ] && . /etc/conf.d/$DAEMON
 

	
init.d/kallithea-daemon-debian
Show inline comments
 
@@ -26,13 +26,13 @@ PYTHON_PATH="/$APP_HOMEDIR/$APP_NAME-ven
 

	
 
RUN_AS="root"
 

	
 
DAEMON="$PYTHON_PATH/bin/paster"
 
DAEMON="$PYTHON_PATH/bin/gearbox"
 

	
 
DAEMON_OPTS="serve --daemon \
 
 --user=$RUN_AS \
 
 --group=$RUN_AS \
 
 --pid-file=$PID_PATH \
 
 --log-file=$LOG_PATH  $APP_PATH/$CONF_NAME"
 
 --log-file=$LOG_PATH -c $APP_PATH/$CONF_NAME"
 

	
 

	
 
start() {
init.d/kallithea-daemon-gentoo
Show inline comments
 
@@ -16,13 +16,13 @@ PYTHON_PATH="/home/$APP_HOMEDIR/v-env"
 

	
 
RUN_AS="username"
 

	
 
DAEMON="$PYTHON_PATH/bin/paster"
 
DAEMON="$PYTHON_PATH/bin/gearbox"
 

	
 
DAEMON_OPTS="serve --daemon \
 
--user=$RUN_AS \
 
--group=$RUN_AS \
 
--pid-file=$PID_PATH \
 
--log-file=$LOG_PATH  $APP_PATH/$CONF_NAME"
 
--log-file=$LOG_PATH -c $APP_PATH/$CONF_NAME"
 

	
 
#extra options
 
opts="${opts} restartdelay"
init.d/kallithea-daemon-redhat
Show inline comments
 
@@ -20,7 +20,7 @@ APP_PATH="/var/www/$APP_NAME"
 
CONF_NAME="production.ini"
 

	
 
# write to wherever the PID should be stored, just ensure
 
# that the user you run paster as has the appropriate permissions
 
# that the user you run gearbox as has the appropriate permissions
 
# same goes for the log file
 
PID_PATH="/var/run/kallithea/pid"
 
LOG_PATH="/var/log/kallithea/kallithea.log"
 
@@ -31,13 +31,13 @@ PYTHON_PATH="/opt/python_virtualenvironm
 

	
 
RUN_AS="kallithea"
 

	
 
DAEMON="$PYTHON_PATH/bin/paster"
 
DAEMON="$PYTHON_PATH/bin/gearbox"
 

	
 
DAEMON_OPTS="serve --daemon \
 
    --user=$RUN_AS \
 
    --group=$RUN_AS \
 
    --pid-file=$PID_PATH \
 
    --log-file=$LOG_PATH $APP_PATH/$CONF_NAME"
 
    --log-file=$LOG_PATH -c $APP_PATH/$CONF_NAME"
 

	
 
DESC="kallithea-server"
 
LOCK_FILE="/var/lock/subsys/$APP_NAME"
init.d/kallithea-upstart.conf
Show inline comments
 
@@ -19,7 +19,7 @@ env HOME=/var/hg
 
env USER=hg
 
env GROUP=hg
 

	
 
exec /var/hg/.virtualenvs/kallithea/bin/paster serve --user=$USER --group=$GROUP --pid-file=$PIDFILE --log-file=$LOGFILE $APPINI
 
exec /var/hg/.virtualenvs/kallithea/bin/gearbox serve --user=$USER --group=$GROUP --pid-file=$PIDFILE --log-file=$LOGFILE -c $APPINI
 

	
 
post-stop script
 
    rm -f $PIDFILE
init.d/supervisord.conf
Show inline comments
 
@@ -45,7 +45,7 @@ serverurl=http://127.0.0.1:9001 ; use an
 
numprocs = 1
 
numprocs_start = 5000 # possible should match ports
 
directory=/srv/kallithea
 
command = /srv/kallithea/venv/bin/paster serve my.ini
 
command = /srv/kallithea/venv/bin/gearbox serve -c my.ini
 
process_name = %(program_name)s_%(process_num)04d
 
redirect_stderr=true
 
stdout_logfile=/%(here)s/kallithea.log
kallithea/alembic/env.py
Show inline comments
 
@@ -40,7 +40,7 @@ logging.getLogger('alembic').setLevel(lo
 

	
 
# Setup Python loggers based on the config file provided to the alembic
 
# command. If we're being invoked via the Alembic API (presumably for
 
# stamping during "paster setup-db"), config_file_name is not available,
 
# stamping during "gearbox setup-db"), config_file_name is not available,
 
# and loggers are assumed to already have been configured.
 
if config.config_file_name:
 
    fileConfig(config.config_file_name, disable_existing_loggers=False)
kallithea/bin/template.ini.mako
Show inline comments
 
@@ -61,9 +61,9 @@ pdebug = false
 
#smtp_use_tls = false
 

	
 
[server:main]
 
%if http_server == 'paste':
 
<%text>## PASTE ##</%text>
 
use = egg:Paste#http
 
%if http_server == 'gearbox':
 
<%text>## Gearbox default web server ##</%text>
 
use = egg:gearbox#wsgiref
 
<%text>## nr of worker threads to spawn</%text>
 
threadpool_workers = 1
 
<%text>## max request before thread respawn</%text>
 
@@ -71,6 +71,10 @@ threadpool_max_requests = 100
 
<%text>## option to use threads of process</%text>
 
use_threadpool = true
 

	
 
%elif http_server == 'gevent':
 
<%text>## Gearbox gevent web server ##</%text>
 
use = egg:gearbox#gevent
 

	
 
%elif http_server == 'waitress':
 
<%text>## WAITRESS ##</%text>
 
use = egg:waitress#main
 
@@ -512,7 +516,7 @@ script_location = kallithea:alembic
 
<%text>################################</%text>
 

	
 
[loggers]
 
keys = root, routes, kallithea, sqlalchemy, beaker, templates, whoosh_indexer
 
keys = root, routes, kallithea, sqlalchemy, gearbox, beaker, templates, whoosh_indexer
 

	
 
[handlers]
 
keys = console, console_sql
 
@@ -553,6 +557,12 @@ handlers =
 
qualname = kallithea
 
propagate = 1
 

	
 
[logger_gearbox]
 
level = DEBUG
 
handlers =
 
qualname = gearbox
 
propagate = 1
 

	
 
[logger_sqlalchemy]
 
level = INFO
 
handlers = console_sql
kallithea/config/deployment.ini_tmpl
Show inline comments
 
@@ -62,8 +62,8 @@ pdebug = false
 
#smtp_use_tls = false
 

	
 
[server:main]
 
## PASTE ##
 
#use = egg:Paste#http
 
## Gearbox default web server ##
 
#use = egg:gearbox#wsgiref
 
## nr of worker threads to spawn
 
#threadpool_workers = 1
 
## max request before thread respawn
 
@@ -71,6 +71,9 @@ pdebug = false
 
## option to use threads of process
 
#use_threadpool = true
 

	
 
## Gearbox gevent web server ##
 
#use = egg:gearbox#gevent
 

	
 
## WAITRESS ##
 
use = egg:waitress#main
 
## number of worker threads
 
@@ -501,7 +504,7 @@ script_location = kallithea:alembic
 
################################
 

	
 
[loggers]
 
keys = root, routes, kallithea, sqlalchemy, beaker, templates, whoosh_indexer
 
keys = root, routes, kallithea, sqlalchemy, gearbox, beaker, templates, whoosh_indexer
 

	
 
[handlers]
 
keys = console, console_sql
 
@@ -542,6 +545,12 @@ handlers =
 
qualname = kallithea
 
propagate = 1
 

	
 
[logger_gearbox]
 
level = DEBUG
 
handlers =
 
qualname = gearbox
 
propagate = 1
 

	
 
[logger_sqlalchemy]
 
level = INFO
 
handlers = console_sql
kallithea/lib/dbmigrate/__init__.py
Show inline comments
 
from paste.script.command import Command
 
from gearbox.command import Command
 

	
 
class UpgradeDb(Command):
 
    hidden = True
 
    summary = '(removed)'
 
    '''(removed)'''
 

	
 
    deprecated = True
 

	
 
    def run(self, args):
 
        raise SystemExit(
kallithea/lib/paster_commands/cache_keys.py
Show inline comments
 
@@ -15,7 +15,7 @@
 
kallithea.lib.paster_commands.cache_keys
 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 

	
 
cleanup-keys paster command for Kallithea
 
cleanup-keys gearbox command for Kallithea
 

	
 

	
 
This file was forked by the Kallithea project in July 2014.
 
@@ -37,25 +37,14 @@ from kallithea.model.db import CacheInva
 

	
 

	
 
class Command(BasePasterCommand):
 

	
 
    max_args = 1
 
    min_args = 1
 
    "Kallithea: Utilities for managing caching of database content"
 

	
 
    usage = "CONFIG_FILE"
 
    group_name = "Kallithea"
 
    takes_config_file = -1
 
    parser = BasePasterCommand.standard_parser(verbose=True)
 
    summary = "Cache keys utils"
 

	
 
    def command(self):
 
        #get SqlAlchemy session
 
        self._init_session()
 

	
 
    def take_action(self, args):
 
        _caches = CacheInvalidation.query().order_by(CacheInvalidation.cache_key).all()
 
        if self.options.show:
 
        if args.show:
 
            for c_obj in _caches:
 
                print 'key:%s active:%s' % (safe_str(c_obj.cache_key), c_obj.cache_active)
 
        elif self.options.cleanup:
 
        elif args.cleanup:
 
            for c_obj in _caches:
 
                Session().delete(c_obj)
 
                print 'Removing key: %s' % (safe_str(c_obj.cache_key))
 
@@ -63,17 +52,21 @@ class Command(BasePasterCommand):
 
        else:
 
            print 'Nothing done, exiting...'
 

	
 
    def update_parser(self):
 
        self.parser.add_option(
 
    def get_parser(self, prog_name):
 
        parser = super(Command, self).get_parser(prog_name)
 

	
 
        parser.add_argument(
 
            '--show',
 
            action='store_true',
 
            dest='show',
 
            help=("show existing cache keys with together with status")
 
            help="show existing cache keys with together with status",
 
        )
 

	
 
        self.parser.add_option(
 
        parser.add_argument(
 
            '--cleanup',
 
            action="store_true",
 
            dest="cleanup",
 
            help="cleanup existing cache keys"
 
            help="cleanup existing cache keys",
 
        )
 

	
 
        return parser
kallithea/lib/paster_commands/celeryd.py
Show inline comments
 
# -*- coding: utf-8 -*-
 

	
 
import argparse
 

	
 
import kallithea
 
from kallithea.lib.paster_commands.common import BasePasterCommand
 
from kallithea.lib.utils import load_rcextensions
 
@@ -9,26 +11,14 @@ __all__ = ['Command']
 

	
 

	
 
class Command(BasePasterCommand):
 
    """Start the celery worker
 

	
 
    Starts the celery worker that uses a paste.deploy configuration
 
    file.
 
    """
 
    """Kallithea: Celery worker for asynchronous tasks"""
 

	
 
    usage = 'CONFIG_FILE [celeryd options...]'
 
    summary = __doc__.splitlines()[0]
 
    description = "".join(__doc__.splitlines()[2:])
 
    group_name = "Kallithea"
 

	
 
    parser = BasePasterCommand.standard_parser(quiet=True)
 
    # Starts the celery worker using configuration from a paste.deploy
 
    # configuration file.
 

	
 
    def update_parser(self):
 
        from kallithea.lib import celerypylons
 
        cmd = celerypylons.worker.worker(celerypylons.app.app_or_default())
 
        for x in cmd.get_options():
 
            self.parser.add_option(x)
 
    requires_db_session = False # will start session on demand
 

	
 
    def command(self):
 
    def take_action(self, args):
 
        from kallithea.lib import celerypylons
 
        from tg import config
 
        try:
 
@@ -43,4 +33,18 @@ class Command(BasePasterCommand):
 

	
 
        load_rcextensions(config['here'])
 
        cmd = celerypylons.worker.worker(celerypylons.app.app_or_default())
 
        return cmd.run(**vars(self.options))
 

	
 
        celery_args = args.celery_args
 
        if '--' in celery_args:
 
            celery_args.remove('--')
 

	
 
        return cmd.run_from_argv('kallithea celery worker', celery_args)
 

	
 
    def get_parser(self, prog_name):
 
        parser = super(Command, self).get_parser(prog_name)
 

	
 
        parser.add_argument('celery_args', nargs=argparse.REMAINDER,
 
            help="Pass extra options to Celery after a '--' separator",
 
            )
 

	
 
        return parser
kallithea/lib/paster_commands/cleanup.py
Show inline comments
 
@@ -15,7 +15,7 @@
 
kallithea.lib.paster_commands.cleanup
 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 

	
 
cleanup-repos paster command for Kallithea
 
cleanup-repos gearbox command for Kallithea
 

	
 

	
 
This file was forked by the Kallithea project in July 2014.
 
@@ -40,15 +40,7 @@ from kallithea.model.db import Ui
 

	
 

	
 
class Command(BasePasterCommand):
 

	
 
    max_args = 1
 
    min_args = 1
 

	
 
    usage = "CONFIG_FILE"
 
    group_name = "Kallithea"
 
    takes_config_file = -1
 
    parser = BasePasterCommand.standard_parser(verbose=True)
 
    summary = "Cleanup deleted repos"
 
    """Kallithea: Cleanup of backup files of deleted repositories"""
 

	
 
    def _parse_older_than(self, val):
 
        regex = re.compile(r'((?P<days>\d+?)d)?((?P<hours>\d+?)h)?((?P<minutes>\d+?)m)?((?P<seconds>\d+?)s)?')
 
@@ -72,10 +64,7 @@ class Command(BasePasterCommand):
 
        date_part = name[4:19]  # 4:19 since we don't parse milliseconds
 
        return datetime.datetime.strptime(date_part, '%Y%m%d_%H%M%S')
 

	
 
    def command(self):
 
        #get SqlAlchemy session
 
        self._init_session()
 

	
 
    def take_action(self, args):
 
        repos_location = Ui.get_repos_location()
 
        to_remove = []
 
        for dn_, dirs, f in os.walk(safe_str(repos_location)):
 
@@ -97,7 +86,7 @@ class Command(BasePasterCommand):
 

	
 
        #filter older than (if present)!
 
        now = datetime.datetime.now()
 
        older_than = self.options.older_than
 
        older_than = args.older_than
 
        if older_than:
 
            to_remove_filtered = []
 
            older_than_date = self._parse_older_than(older_than)
 
@@ -111,7 +100,7 @@ class Command(BasePasterCommand):
 
                % (len(to_remove), older_than, older_than_date)
 
        else:
 
            print 'Removing all %s deleted repos' % len(to_remove)
 
        if self.options.dont_ask or not to_remove:
 
        if args.dont_ask or not to_remove:
 
            # don't ask just remove !
 
            remove = True
 
        else:
 
@@ -127,8 +116,10 @@ class Command(BasePasterCommand):
 
        else:
 
            print 'Nothing done, exiting...'
 

	
 
    def update_parser(self):
 
        self.parser.add_option(
 
    def get_parser(self, prog_name):
 
        parser = super(Command, self).get_parser(prog_name)
 

	
 
        parser.add_argument(
 
            '--older-than',
 
            action='store',
 
            dest='older_than',
 
@@ -141,9 +132,11 @@ class Command(BasePasterCommand):
 
                 "removed more than 30 days ago.")
 
            )
 

	
 
        self.parser.add_option(
 
        parser.add_argument(
 
            '--dont-ask',
 
            action="store_true",
 
            dest="dont_ask",
 
            help="remove repositories without asking for confirmation."
 
        )
 

	
 
        return parser
kallithea/lib/paster_commands/common.py
Show inline comments
 
@@ -15,7 +15,7 @@
 
kallithea.lib.paster_commands.common
 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 

	
 
Common code for Paster commands.
 
Common code for gearbox commands.
 

	
 
This file was forked by the Kallithea project in July 2014.
 
Original author and date, and relevant copyright and licensing information is below:
 
@@ -26,12 +26,11 @@ Original author and date, and relevant c
 
"""
 

	
 
import os
 
import logging
 
import sys
 
import logging.config
 

	
 
import paste
 
from paste.script.command import Command, BadCommand
 

	
 
from kallithea.lib.utils import setup_cache_regions
 
import paste.deploy
 
import gearbox.command
 

	
 

	
 
def ask_ok(prompt, retries=4, complaint='Yes or no please!'):
 
@@ -47,14 +46,14 @@ def ask_ok(prompt, retries=4, complaint=
 
        print complaint
 

	
 

	
 
class BasePasterCommand(Command):
 
class BasePasterCommand(gearbox.command.Command):
 
    """
 
    Abstract Base Class for paster commands.
 
    Abstract Base Class for gearbox commands.
 
    """
 
    min_args = 1
 
    min_args_error = "Please provide a paster config file as an argument."
 
    takes_config_file = 1
 
    requires_config_file = True
 

	
 
    # override to control how much get_parser and run should do:
 
    takes_config_file = True
 
    requires_db_session = True
 

	
 
    def run(self, args):
 
        """
 
@@ -62,46 +61,48 @@ class BasePasterCommand(Command):
 

	
 
        Checks for a config file argument and loads it.
 
        """
 
        if len(args) < self.min_args:
 
            raise BadCommand(
 
                self.min_args_error % {'min_args': self.min_args,
 
                                       'actual_args': len(args)})
 
        if self.takes_config_file:
 
             self._bootstrap_config(args.config_file)
 
             if self.requires_db_session:
 
                  self._init_session()
 

	
 
        # Decrement because we're going to lob off the first argument.
 
        # @@ This is hacky
 
        self.min_args -= 1
 
        self.bootstrap_config(args[0])
 
        self.update_parser()
 
        return super(BasePasterCommand, self).run(args[1:])
 
        return super(BasePasterCommand, self).run(args)
 

	
 
    def get_parser(self, prog_name):
 
        parser = super(BasePasterCommand, self).get_parser(prog_name)
 

	
 
    def update_parser(self):
 
        if self.takes_config_file:
 
            parser.add_argument("-c", "--config",
 
                help='Kallithea .ini file with configuration of database etc',
 
                dest='config_file', required=True)
 

	
 
        return parser
 

	
 
    def _bootstrap_config(self, config_file):
 
        """
 
        Abstract method.  Allows for the class's parser to be updated
 
        before the superclass's `run` method is called.  Necessary to
 
        allow options/arguments to be passed through to the underlying
 
        celery command.
 
        """
 
        raise NotImplementedError("Abstract Method.")
 

	
 
    def bootstrap_config(self, conf):
 
        """
 
        Loads the app configuration.
 
        Read the config file and initialize logging and the application.
 
        """
 
        from tg import config as pylonsconfig
 

	
 
        self.path_to_ini_file = os.path.realpath(conf)
 
        conf = paste.deploy.appconfig('config:' + self.path_to_ini_file)
 
        path_to_ini_file = os.path.realpath(config_file)
 
        conf = paste.deploy.appconfig('config:' + path_to_ini_file)
 
        logging.config.fileConfig(path_to_ini_file)
 
        pylonsconfig.init_app(conf.global_conf, conf.local_conf)
 

	
 
    def _init_session(self):
 
        """
 
        Inits SqlAlchemy Session
 
        Initialize SqlAlchemy Session from global config.
 
        """
 
        logging.config.fileConfig(self.path_to_ini_file)
 

	
 
        from tg import config
 
        from kallithea.model.base import init_model
 
        from kallithea.lib.utils2 import engine_from_config
 
        from kallithea.lib.utils import setup_cache_regions
 
        setup_cache_regions(config)
 
        engine = engine_from_config(config, 'sqlalchemy.')
 
        init_model(engine)
 

	
 
    def error(self, msg, exitcode=1):
 
        """Write error message and exit"""
 
        sys.stderr.write('%s\n' % msg)
 
        raise SystemExit(exitcode)
kallithea/lib/paster_commands/install_iis.py
Show inline comments
 
@@ -22,37 +22,21 @@ IIS installation tools for Kallithea
 
import os
 
import sys
 
from paste.script.appinstall import AbstractInstallCommand
 
from paste.script.command import BadCommand
 

	
 
from kallithea.lib.paster_commands.common import BasePasterCommand
 

	
 

	
 
class Command(AbstractInstallCommand):
 
    default_verbosity = 1
 
    max_args = 1
 
    min_args = 1
 
    summary = 'Setup IIS given a config file'
 
    usage = 'CONFIG_FILE'
 
class Command(BasePasterCommand):
 
    '''Kallithea: Install into IIS using isapi-wsgi'''
 

	
 
    description = '''
 
    Script for installing into IIS using isapi-wsgi.
 
    '''
 
    parser = AbstractInstallCommand.standard_parser(
 
        simulate=True, quiet=True, interactive=True)
 
    parser.add_option('--virtualdir',
 
                      action='store',
 
                      dest='virtualdir',
 
                      default='/',
 
                      help='The virtual folder to install into on IIS')
 
    requires_db_session = False
 

	
 
    def command(self):
 
        config_spec = self.args[0]
 
        if not config_spec.startswith('config:'):
 
            config_spec = 'config:' + config_spec
 
        config_file = config_spec[len('config:'):].split('#', 1)[0]
 
        config_file = os.path.join(os.getcwd(), config_file)
 
    def take_action(self, args):
 
        config_file = os.path.abspath(args.config_file)
 
        try:
 
            import isapi_wsgi
 
        except ImportError:
 
            raise BadCommand('missing requirement: isapi-wsgi not installed')
 
            self.error('missing requirement: isapi-wsgi not installed')
 

	
 
        file = '''\
 
# Created by Kallithea install_iis
 
@@ -92,7 +76,7 @@ if __name__=='__main__':
 

	
 
        outdata = file % {
 
                'inifile': config_file.replace('\\', '\\\\'),
 
                'virtualdir': self.options.virtualdir
 
                'virtualdir': args.virtualdir,
 
                }
 

	
 
        dispatchfile = os.path.join(os.getcwd(), 'dispatch.py')
 
@@ -102,3 +86,14 @@ if __name__=='__main__':
 
        print ('Run \'python "%s" install\' with administrative privileges '
 
            'to generate the _dispatch.dll file and install it into the '
 
            'default web site') % (dispatchfile,)
 

	
 
    def get_parser(self, prog_name):
 
        parser = super(Command, self).get_parser(prog_name)
 

	
 
        parser.add_argument('--virtualdir',
 
                      action='store',
 
                      dest='virtualdir',
 
                      default='/',
 
                      help='The virtual folder to install into on IIS')
 

	
 
        return parser
kallithea/lib/paster_commands/ishell.py
Show inline comments
 
@@ -15,7 +15,7 @@
 
kallithea.lib.paster_commands.ishell
 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 

	
 
interactive shell paster command for Kallithea
 
interactive shell gearbox command for Kallithea
 

	
 
This file was forked by the Kallithea project in July 2014.
 
Original author and date, and relevant copyright and licensing information is below:
 
@@ -39,20 +39,9 @@ from kallithea.lib.paster_commands.commo
 

	
 

	
 
class Command(BasePasterCommand):
 

	
 
    max_args = 1
 
    min_args = 1
 
    "Kallithea: Interactive Python shell"
 

	
 
    usage = "CONFIG_FILE"
 
    group_name = "Kallithea"
 
    takes_config_file = -1
 
    parser = BasePasterCommand.standard_parser(verbose=True)
 
    summary = "Interactive shell"
 

	
 
    def command(self):
 
        #get SqlAlchemy session
 
        self._init_session()
 

	
 
    def take_action(self, args):
 
        try:
 
            from IPython import embed
 
            from IPython.config.loader import Config
 
@@ -62,6 +51,3 @@ class Command(BasePasterCommand):
 
        except ImportError:
 
            print 'Kallithea ishell requires the IPython Python package'
 
            sys.exit(-1)
 

	
 
    def update_parser(self):
 
        pass
kallithea/lib/paster_commands/make_index.py
Show inline comments
 
@@ -15,7 +15,7 @@
 
kallithea.lib.paster_commands.make_index
 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 

	
 
make-index paster command for Kallithea
 
make-index gearbox command for Kallithea
 

	
 
This file was forked by the Kallithea project in July 2014.
 
Original author and date, and relevant copyright and licensing information is below:
 
@@ -37,30 +37,20 @@ from kallithea.lib.utils import load_rce
 

	
 

	
 
class Command(BasePasterCommand):
 

	
 
    max_args = 1
 
    min_args = 1
 
    "Kallithea: Create or update full text search index"
 

	
 
    usage = "CONFIG_FILE"
 
    group_name = "Kallithea"
 
    takes_config_file = -1
 
    parser = BasePasterCommand.standard_parser(verbose=True)
 
    summary = "Creates or updates full text search index"
 

	
 
    def command(self):
 
        #get SqlAlchemy session
 
        self._init_session()
 
        from tg import config
 
    def take_action(self, args):
 
        from pylons import config
 
        index_location = config['index_dir']
 
        load_rcextensions(config['here'])
 

	
 
        repo_location = self.options.repo_location \
 
            if self.options.repo_location else RepoModel().repos_path
 
        repo_list = map(strip, self.options.repo_list.split(',')) \
 
            if self.options.repo_list else None
 
        repo_location = args.repo_location \
 
            if args.repo_location else RepoModel().repos_path
 
        repo_list = map(strip, args.repo_list.split(',')) \
 
            if args.repo_list else None
 

	
 
        repo_update_list = map(strip, self.options.repo_update_list.split(',')) \
 
            if self.options.repo_update_list else None
 
        repo_update_list = map(strip, args.repo_update_list.split(',')) \
 
            if args.repo_update_list else None
 

	
 
        #======================================================================
 
        # WHOOSH DAEMON
 
@@ -74,33 +64,37 @@ class Command(BasePasterCommand):
 
                                 repo_location=repo_location,
 
                                 repo_list=repo_list,
 
                                 repo_update_list=repo_update_list) \
 
                .run(full_index=self.options.full_index)
 
                .run(full_index=args.full_index)
 
            l.release()
 
        except LockHeld:
 
            sys.exit(1)
 

	
 
    def update_parser(self):
 
        self.parser.add_option('--repo-location',
 
    def get_parser(self, prog_name):
 
        parser = super(Command, self).get_parser(prog_name)
 

	
 
        parser.add_argument('--repo-location',
 
                          action='store',
 
                          dest='repo_location',
 
                          help="Specifies repositories location to index OPTIONAL",
 
                          )
 
        self.parser.add_option('--index-only',
 
        parser.add_argument('--index-only',
 
                          action='store',
 
                          dest='repo_list',
 
                          help="Specifies a comma separated list of repositories "
 
                                "to build index on. If not given all repositories "
 
                                "are scanned for indexing. OPTIONAL",
 
                          )
 
        self.parser.add_option('--update-only',
 
        parser.add_argument('--update-only',
 
                          action='store',
 
                          dest='repo_update_list',
 
                          help="Specifies a comma separated list of repositories "
 
                                "to re-build index on. OPTIONAL",
 
                          )
 
        self.parser.add_option('-f',
 
        parser.add_argument('-f',
 
                          action='store_true',
 
                          dest='full_index',
 
                          help="Specifies that index should be made full i.e"
 
                                " destroy old and build from scratch",
 
                          default=False)
 

	
 
        return parser
kallithea/lib/paster_commands/make_rcextensions.py
Show inline comments
 
@@ -15,7 +15,7 @@
 
kallithea.lib.paster_commands.make_rcextensions
 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 

	
 
make-rcext paster command for Kallithea
 
make-rcext gearbox command for Kallithea
 

	
 
This file was forked by the Kallithea project in July 2014.
 
Original author and date, and relevant copyright and licensing information is below:
 
@@ -34,23 +34,17 @@ from kallithea.lib.paster_commands.commo
 

	
 

	
 
class Command(BasePasterCommand):
 

	
 
    max_args = 1
 
    min_args = 1
 
    """Kallithea: Write template file for extending Kallithea in Python
 

	
 
    group_name = "Kallithea"
 
    takes_config_file = -1
 
    parser = BasePasterCommand.standard_parser(verbose=True)
 
    summary = "Write template file for extending Kallithea in Python."
 
    usage = "CONFIG_FILE"
 
    description = '''\
 
        A rcextensions directory with a __init__.py file will be created next to
 
        the ini file. Local customizations in that file will survive upgrades.
 
        The file contains instructions on how it can be customized.
 
        '''
 
    A rcextensions directory with a __init__.py file will be created next to
 
    the ini file. Local customizations in that file will survive upgrades.
 
    The file contains instructions on how it can be customized.
 
    """
 

	
 
    def command(self):
 
        from tg import config
 
    takes_config_file = False
 

	
 
    def take_action(self, args):
 
        from pylons import config
 

	
 
        here = config['here']
 
        content = pkg_resources.resource_string(
 
@@ -70,6 +64,3 @@ class Command(BasePasterCommand):
 
        with open(ext_file, 'wb') as f:
 
            f.write(content)
 
            print 'Wrote new extensions file to %s' % ext_file
 

	
 
    def update_parser(self):
 
        pass
kallithea/lib/paster_commands/repo_scan.py
Show inline comments
 
@@ -15,7 +15,7 @@
 
kallithea.lib.paster_commands.repo_scan
 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 

	
 
repo-scan paster command for Kallithea
 
repo-scan gearbox command for Kallithea
 

	
 
This file was forked by the Kallithea project in July 2014.
 
Original author and date, and relevant copyright and licensing information is below:
 
@@ -35,20 +35,14 @@ from kallithea.lib.utils import repo2db_
 

	
 

	
 
class Command(BasePasterCommand):
 

	
 
    max_args = 1
 
    min_args = 1
 
    """Kallithea: Scan file system for repositories
 

	
 
    usage = "CONFIG_FILE"
 
    group_name = "Kallithea"
 
    takes_config_file = -1
 
    parser = BasePasterCommand.standard_parser(verbose=True)
 
    summary = "Rescan default location for new repositories"
 
    Search under the repository root configured in the database,
 
    all new repositories, and report missing ones with an option of removing them.
 
    """
 

	
 
    def command(self):
 
        #get SqlAlchemy session
 
        self._init_session()
 
        rm_obsolete = self.options.delete_obsolete
 
    def take_action(self, args):
 
        rm_obsolete = args.delete_obsolete
 
        print 'Now scanning root location for new repos ...'
 
        added, removed = repo2db_mapper(ScmModel().repo_scan(),
 
                                        remove_obsolete=rm_obsolete)
 
@@ -61,10 +55,14 @@ class Command(BasePasterCommand):
 
        else:
 
            print 'Missing: %s' % removed
 

	
 
    def update_parser(self):
 
        self.parser.add_option(
 
    def get_parser(self, prog_name):
 
        parser = super(Command, self).get_parser(prog_name)
 

	
 
        parser.add_argument(
 
            '--delete-obsolete',
 
            action='store_true',
 
            help="Use this flag do delete repositories that are "
 
                 "present in Kallithea database but not on the filesystem",
 
        )
 

	
 
        return parser
kallithea/lib/paster_commands/setup_db.py
Show inline comments
 
@@ -15,112 +15,93 @@
 
kallithea.lib.paster_commands.setup_db
 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 

	
 
Databaset setup paster command for Kallithea
 
Databaset setup gearbox command for Kallithea
 
"""
 

	
 

	
 
import os
 
import sys
 
from paste.script.appinstall import AbstractInstallCommand
 
from paste.script.command import BadCommand
 
from paste.deploy import appconfig
 
import paste.deploy
 

	
 
from kallithea.lib.db_manage import DbManage
 
from kallithea.lib.paster_commands.common import BasePasterCommand
 
from kallithea.model.meta import Session
 

	
 

	
 
class Command(AbstractInstallCommand):
 
# This is almost like SetupAppCommand ... but we have to pass options and it is
 
# thus simpler to drop websetup and reimplement everything
 
class Command(BasePasterCommand):
 
    """Kallithea: Configure the database specified in the .ini file
 

	
 
    default_verbosity = 1
 
    max_args = 1
 
    min_args = 1
 
    summary = "Setup an application, given a config file"
 
    usage = "CONFIG_FILE"
 
    group_name = "Kallithea"
 

	
 
    description = """\
 
        Setup Kallithea according to its configuration file.  This is
 
        the second part of a two-phase web application installation
 
        process (the first phase is prepare-app).  The setup process
 
        consist of things like setting up databases, creating super user
 
    Setup Kallithea according to its configuration file.  This is
 
    the second part of a two-phase web application installation
 
    process (the first phase is prepare-app). The setup process
 
    consist of things like setting up databases and creating the admin user
 
    """
 

	
 
    parser = AbstractInstallCommand.standard_parser(
 
        simulate=True, quiet=True, interactive=True)
 
    parser.add_option('--user',
 
                      action='store',
 
                      dest='username',
 
                      default=None,
 
                      help='Admin Username')
 
    parser.add_option('--email',
 
                      action='store',
 
                      dest='email',
 
                      default=None,
 
                      help='Admin Email')
 
    parser.add_option('--password',
 
                      action='store',
 
                      dest='password',
 
                      default=None,
 
                      help='Admin password min 6 chars')
 
    parser.add_option('--repos',
 
                      action='store',
 
                      dest='repos_location',
 
                      default=None,
 
                      help='Absolute path to repositories location')
 
    parser.add_option('--name',
 
                      action='store',
 
                      dest='section_name',
 
                      default=None,
 
                      help='The name of the section to set up (default: app:main)')
 
    parser.add_option('--force-yes',
 
                       action='store_true',
 
                       dest='force_ask',
 
                       default=None,
 
                       help='Force yes to every question')
 
    parser.add_option('--force-no',
 
                       action='store_false',
 
                       dest='force_ask',
 
                       default=None,
 
                       help='Force no to every question')
 
    parser.add_option('--public-access',
 
                       action='store_true',
 
                       dest='public_access',
 
                       default=None,
 
                       help='Enable public access on this installation (default)')
 
    parser.add_option('--no-public-access',
 
                       action='store_false',
 
                       dest='public_access',
 
                       default=None,
 
                       help='Disable public access on this installation ')
 
    def get_description(self):
 
        return self.__doc__.splitlines()[0]
 

	
 
    requires_db_session = False # only available after this command has been run
 

	
 
    def get_parser(self, prog_name):
 
        parser = super(Command, self).get_parser(prog_name)
 

	
 
    def command(self):
 
        config_spec = self.args[0]
 
        section = self.options.section_name
 
        if section is None:
 
            if '#' in config_spec:
 
                config_spec, section = config_spec.split('#', 1)
 
            else:
 
                section = 'main'
 
        if not ':' in section:
 
            plain_section = section
 
            section = 'app:' + section
 
        else:
 
            plain_section = section.split(':', 1)[0]
 
        if not config_spec.startswith('config:'):
 
            config_spec = 'config:' + config_spec
 
        if plain_section != 'main':
 
            config_spec += '#' + plain_section
 
        config_file = config_spec[len('config:'):].split('#', 1)[0]
 
        config_file = os.path.join(os.getcwd(), config_file)
 
        self.logging_file_config(config_file)
 
        conf = appconfig(config_spec, relative_to=os.getcwd())
 
        ep_name = conf.context.entry_point_name
 
        ep_group = conf.context.protocol
 
        dist = conf.context.distribution
 
        if dist is None:
 
            raise BadCommand(
 
                "The section %r is not the application (probably a filter). "
 
                "You should add #section_name, where section_name is the "
 
                "section that configures your application" % plain_section)
 
        installer = self.get_installer(dist, ep_group, ep_name)
 
        installer.setup_config(
 
            self, config_file, section, self.sysconfig_install_vars(installer))
 
        self.call_sysconfig_functions(
 
            'post_setup_hook', installer, config_file)
 
        parser.add_argument('--user',
 
                          action='store',
 
                          dest='username',
 
                          default=None,
 
                          help='Admin Username')
 
        parser.add_argument('--email',
 
                          action='store',
 
                          dest='email',
 
                          default=None,
 
                          help='Admin Email')
 
        parser.add_argument('--password',
 
                          action='store',
 
                          dest='password',
 
                          default=None,
 
                          help='Admin password min 6 chars')
 
        parser.add_argument('--repos',
 
                          action='store',
 
                          dest='repos_location',
 
                          default=None,
 
                          help='Absolute path to repositories location')
 
        parser.add_argument('--force-yes',
 
                           action='store_true',
 
                           dest='force_ask',
 
                           default=None,
 
                           help='Force yes to every question')
 
        parser.add_argument('--force-no',
 
                           action='store_false',
 
                           dest='force_ask',
 
                           default=None,
 
                           help='Force no to every question')
 
        parser.add_argument('--public-access',
 
                           action='store_true',
 
                           dest='public_access',
 
                           default=None,
 
                           help='Enable public access on this installation (default)')
 
        parser.add_argument('--no-public-access',
 
                           action='store_false',
 
                           dest='public_access',
 
                           default=None,
 
                           help='Disable public access on this installation ')
 

	
 
        return parser
 

	
 
    def take_action(self, opts):
 
        path_to_ini_file = os.path.realpath(opts.config_file)
 
        conf = paste.deploy.appconfig('config:' + path_to_ini_file)
 

	
 
        dbconf = conf['sqlalchemy.url']
 
        dbmanage = DbManage(log_sql=True, dbconf=dbconf, root=conf['here'],
 
                            tests=False, cli_args=vars(opts))
 
        dbmanage.create_tables(override=True)
 
        opts = dbmanage.config_prompt(None)
 
        dbmanage.create_settings(opts)
 
        dbmanage.create_default_user()
 
        dbmanage.admin_prompt()
 
        dbmanage.create_permissions()
 
        dbmanage.populate_default_permissions()
 
        Session().commit()
kallithea/lib/paster_commands/update_repoinfo.py
Show inline comments
 
@@ -15,7 +15,7 @@
 
kallithea.lib.paster_commands.update_repoinfo
 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 

	
 
update-repoinfo paster command for Kallithea
 
update-repoinfo gearbox command for Kallithea
 

	
 
This file was forked by the Kallithea project in July 2014.
 
Original author and date, and relevant copyright and licensing information is below:
 
@@ -38,47 +38,39 @@ from kallithea.model.meta import Session
 

	
 

	
 
class Command(BasePasterCommand):
 

	
 
    max_args = 1
 
    min_args = 1
 
    "Kallithea: Update database cache of repository data"
 

	
 
    usage = "CONFIG_FILE"
 
    group_name = "Kallithea"
 
    takes_config_file = -1
 
    parser = BasePasterCommand.standard_parser(verbose=True)
 
    summary = "Updates repositories caches for last changeset"
 

	
 
    def command(self):
 
        #get SqlAlchemy session
 
        self._init_session()
 

	
 

	
 
        if self.options.repo_update_list is None:
 
    def take_action(self, args):
 
        if args.repo_update_list is None:
 
            repo_list = Repository.query().all()
 
        else:
 
            repo_names = [safe_unicode(n.strip())
 
                          for n in self.options.repo_update_list.split(',')]
 
                          for n in args.repo_update_list.split(',')]
 
            repo_list = list(Repository.query()
 
                .filter(Repository.repo_name.in_(repo_names)))
 
        for repo in repo_list:
 
            repo.update_changeset_cache()
 
        Session().commit()
 

	
 
        if self.options.invalidate_cache:
 
        if args.invalidate_cache:
 
            for r in repo_list:
 
                r.set_invalidate()
 
            print 'Updated repo info and invalidated cache for %s repositories' % (len(repo_list))
 
        else:
 
            print 'Updated repo info for %s repositories' % (len(repo_list))
 

	
 
    def update_parser(self):
 
        self.parser.add_option('--update-only',
 
    def get_parser(self, prog_name):
 
        parser = super(Command, self).get_parser(prog_name)
 

	
 
        parser.add_argument('--update-only',
 
                           action='store',
 
                           dest='repo_update_list',
 
                           help="Specifies a comma separated list of repositories "
 
                                "to update last commit info for. OPTIONAL")
 
        self.parser.add_option('--invalidate-cache',
 
        parser.add_argument('--invalidate-cache',
 
                           action='store_true',
 
                           dest='invalidate_cache',
 
                           help="Trigger cache invalidation event for repos. "
 
                                "OPTIONAL")
 

	
 
        return parser
kallithea/tests/other/manual_test_vcs_operations.py
Show inline comments
 
@@ -18,7 +18,7 @@ kallithea.tests.other.manual_test_vcs_op
 
Test suite for making push/pull operations.
 

	
 
Run it in two terminals::
 
 paster serve kallithea/tests/test.ini
 
 gearbox serve -c kallithea/tests/test.ini
 
 KALLITHEA_WHOOSH_TEST_DISABLE=1 KALLITHEA_NO_TMP_PATH=1 py.test kallithea/tests/other/manual_test_vcs_operations.py
 

	
 
You must have git > 1.8.1 for tests to work fine
kallithea/tests/scripts/create_rc.sh
Show inline comments
 
#!/bin/sh
 
psql -U postgres -h localhost -c 'drop database if exists kallithea;'
 
psql -U postgres -h localhost -c 'create database kallithea;'
 
paster setup-db server.ini --force-yes --user=username --password=qweqwe --email=username@example.com --repos=/home/username/repos --no-public-access
 
gearbox setup-db -c server.ini --force-yes --user=username --password=qweqwe --email=username@example.com --repos=/home/username/repos --no-public-access
 
API_KEY=`psql -R " " -A -U postgres -h localhost -c "select api_key from users where admin=TRUE" -d kallithea | awk '{print $2}'`
 
echo "run those after running server"
 
paster serve server.ini --pid-file=server.pid --daemon
 
gearbox serve -c server.ini --pid-file=server.pid --daemon
 
sleep 3
 
kallithea-api --apikey=$API_KEY --apihost=http://127.0.0.1:5001 create_user username:demo1 password:qweqwe email:demo1@example.com
 
kallithea-api --apikey=$API_KEY --apihost=http://127.0.0.1:5001 create_user username:demo2 password:qweqwe email:demo2@example.com
kallithea/tests/test.ini
Show inline comments
 
@@ -65,8 +65,8 @@ pdebug = false
 
#smtp_use_tls = false
 

	
 
[server:main]
 
## PASTE ##
 
#use = egg:Paste#http
 
## Gearbox default web server ##
 
#use = egg:gearbox#wsgiref
 
## nr of worker threads to spawn
 
#threadpool_workers = 1
 
## max request before thread respawn
 
@@ -74,6 +74,9 @@ pdebug = false
 
## option to use threads of process
 
#use_threadpool = true
 

	
 
## Gearbox gevent web server ##
 
#use = egg:gearbox#gevent
 

	
 
## WAITRESS ##
 
use = egg:waitress#main
 
## number of worker threads
 
@@ -514,7 +517,7 @@ script_location = kallithea:alembic
 
################################
 

	
 
[loggers]
 
keys = root, routes, kallithea, sqlalchemy, beaker, templates, whoosh_indexer
 
keys = root, routes, kallithea, sqlalchemy, gearbox, beaker, templates, whoosh_indexer
 

	
 
[handlers]
 
keys = console, console_sql
 
@@ -555,6 +558,12 @@ handlers =
 
qualname = kallithea
 
propagate = 1
 

	
 
[logger_gearbox]
 
level = DEBUG
 
handlers =
 
qualname = gearbox
 
propagate = 1
 

	
 
[logger_sqlalchemy]
 
level = INFO
 
handlers = console_sql
kallithea/websetup.py
Show inline comments
 
deleted file
scripts/dbmigrate-test
Show inline comments
 
@@ -104,7 +104,7 @@ install_kallithea "$temp/from" "$from_re
 
    cd "$temp/from"
 
    . "$temp/from-env/bin/activate"
 
    announce "Initializing database..."
 
    quiet_if_ok paster setup-db "$config_file" --repos="$temp/repos" --user=doe --email=doe@example.com --password=123456 --no-public-access --force-yes
 
    quiet_if_ok gearbox setup-db -c "$config_file" --repos="$temp/repos" --user=doe --email=doe@example.com --password=123456 --no-public-access --force-yes
 
    alembic -c "$config_file" current -v
 
)
 

	
setup.py
Show inline comments
 
@@ -36,6 +36,7 @@ is_windows = __platform__ in ['Windows']
 

	
 
requirements = [
 
    "alembic>=0.8.0,<0.9",
 
    "GearBox<1",
 
    "waitress>=0.8.8,<1.0",
 
    "webob>=1.7,<2",
 
    "Pylons>=1.0.0,<=1.0.2",
 
@@ -132,7 +133,6 @@ setuptools.setup(
 
    url=__url__,
 
    install_requires=requirements,
 
    classifiers=classifiers,
 
    setup_requires=['PasteScript>=1.6.3'],
 
    data_files=data_files,
 
    packages=packages,
 
    include_package_data=True,
 
@@ -142,7 +142,6 @@ setuptools.setup(
 
            ('templates/**.html', 'mako', {'input_encoding': 'utf-8'}),
 
            ('public/**', 'ignore', None)]},
 
    zip_safe=False,
 
    paster_plugins=['PasteScript', 'Pylons'],
 
    entry_points="""
 
    [console_scripts]
 
    kallithea-api =    kallithea.bin.kallithea_api:main
 
@@ -155,7 +154,7 @@ setuptools.setup(
 
    [paste.app_install]
 
    main = pylons.util:PylonsInstaller
 

	
 
    [paste.global_paster_command]
 
    [gearbox.commands]
 
    setup-db=kallithea.lib.paster_commands.setup_db:Command
 
    cleanup-repos=kallithea.lib.paster_commands.cleanup:Command
 
    update-repoinfo=kallithea.lib.paster_commands.update_repoinfo:Command
0 comments (0 inline, 0 general)