Changeset - 0080ffd8aea0
[Not reviewed]
default
1 3 1
Thomas De Schampheleire - 7 years ago 2018-11-18 20:02:17
thomas.de_schampheleire@nokia.com
cli: convert 'gearbox repo-scan' into 'kallithea-cli repo-scan'
4 files changed with 26 insertions and 42 deletions:
0 comments (0 inline, 0 general)
docs/dev/dbmigrations.rst
Show inline comments
 
=======================
 
Database schema changes
 
=======================
 

	
 
Kallithea uses Alembic for :ref:`database migrations <upgrade_db>`
 
(upgrades and downgrades).
 

	
 
If you are developing a Kallithea feature that requires database schema
 
changes, you should make a matching Alembic database migration script:
 

	
 
1. :ref:`Create a Kallithea configuration and database <setup>` for testing
 
   the migration script, or use existing ``development.ini`` setup.
 

	
 
   Ensure that this database is up to date with the latest database
 
   schema *before* the changes you're currently developing. (Do not
 
   create the database while your new schema changes are applied.)
 

	
 
2. Create a separate throwaway configuration for iterating on the actual
 
   database changes::
 

	
 
    kallithea-cli config-create temp.ini
 

	
 
   Edit the file to change database settings. SQLite is typically fine,
 
   but make sure to change the path to e.g. ``temp.db``, to avoid
 
   clobbering any existing database file.
 

	
 
3. Make your code changes (including database schema changes in ``db.py``).
 

	
 
4. After every database schema change, recreate the throwaway database
 
   to test the changes::
 

	
 
    rm temp.db
 
    gearbox setup-db -c temp.ini --repos=/var/repos --user=doe --email doe@example.com --password=123456 --no-public-access --force-yes
 
    gearbox repo-scan -c temp.ini
 
    kallithea-cli repo-scan -c temp.ini
 

	
 
5. Once satisfied with the schema changes, auto-generate a draft Alembic
 
   script using the development database that has *not* been upgraded.
 
   (The generated script will upgrade the database to match the code.)
 

	
 
   ::
 

	
 
    alembic -c development.ini revision -m "area: add cool feature" --autogenerate
 

	
 
6. Edit the script to clean it up and fix any problems.
 

	
 
   Note that for changes that simply add columns, it may be appropriate
 
   to not remove them in the downgrade script (and instead do nothing),
 
   to avoid the loss of data. Unknown columns will simply be ignored by
 
   Kallithea versions predating your changes.
 

	
 
7. Run ``alembic -c development.ini upgrade head`` to apply changes to
 
   the (non-throwaway) database, and test the upgrade script. Also test
 
   downgrades.
 

	
 
   The included ``development.ini`` has full SQL logging enabled. If
 
   you're using another configuration file, you may want to enable it
 
   by setting ``level = DEBUG`` in section ``[handler_console_sql]``.
 

	
 
The Alembic migration script should be committed in the same revision as
 
the database schema (``db.py``) changes.
 

	
 
See the `Alembic documentation`__ for more information, in particular
 
the tutorial and the section about auto-generating migration scripts.
 

	
 
.. __: http://alembic.zzzcomputing.com/en/latest/
 

	
 

	
 
Troubleshooting
 
---------------
 

	
 
* If ``alembic --autogenerate`` responds "Target database is not up to
 
  date", you need to either first use Alembic to upgrade the database
 
  to the most recent version (before your changes), or recreate the
 
  database from scratch (without your schema changes applied).
kallithea/bin/kallithea_cli.py
Show inline comments
 
# -*- coding: utf-8 -*-
 
# This program is free software: you can redistribute it and/or modify
 
# it under the terms of the GNU General Public License as published by
 
# the Free Software Foundation, either version 3 of the License, or
 
# (at your option) any later version.
 
#
 
# This program is distributed in the hope that it will be useful,
 
# but WITHOUT ANY WARRANTY; without even the implied warranty of
 
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 
# GNU General Public License for more details.
 
#
 
# You should have received a copy of the GNU General Public License
 
# along with this program.  If not, see <http://www.gnu.org/licenses/>.
 

	
 
# 'cli' is the main entry point for 'kallithea-cli', specified in setup.py as entry_points console_scripts
 
from kallithea.bin.kallithea_cli_base import cli
 

	
 
# import commands (they will add themselves to the 'cli' object)
 
import kallithea.bin.kallithea_cli_config
 
import kallithea.bin.kallithea_cli_ishell
 
import kallithea.bin.kallithea_cli_repo
kallithea/bin/kallithea_cli_repo.py
Show inline comments
 
file renamed from kallithea/lib/paster_commands/repo_scan.py to kallithea/bin/kallithea_cli_repo.py
 
# -*- coding: utf-8 -*-
 
# This program is free software: you can redistribute it and/or modify
 
# it under the terms of the GNU General Public License as published by
 
# the Free Software Foundation, either version 3 of the License, or
 
# (at your option) any later version.
 
#
 
# This program is distributed in the hope that it will be useful,
 
# but WITHOUT ANY WARRANTY; without even the implied warranty of
 
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 
# GNU General Public License for more details.
 
#
 
# You should have received a copy of the GNU General Public License
 
# along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
"""
 
kallithea.lib.paster_commands.repo_scan
 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 

	
 
repo-scan gearbox command for Kallithea
 

	
 
This file was forked by the Kallithea project in July 2014.
 
This file was forked by the Kallithea project in July 2014 and later moved.
 
Original author and date, and relevant copyright and licensing information is below:
 
:created_on: Feb 9, 2013
 
:author: marcink
 
:copyright: (c) 2013 RhodeCode GmbH, and others.
 
:license: GPLv3, see LICENSE.md for more details.
 
"""
 

	
 

	
 
import click
 
import kallithea.bin.kallithea_cli_base as cli_base
 

	
 
from kallithea.lib.utils import repo2db_mapper
 
from kallithea.model.scm import ScmModel
 
from kallithea.lib.paster_commands.common import BasePasterCommand
 
from kallithea.lib.utils import repo2db_mapper
 

	
 

	
 
class Command(BasePasterCommand):
 
    """Kallithea: Scan file system for repositories
 

	
 
    Search under the repository root configured in the database,
 
    all new repositories, and report missing ones with an option of removing them.
 
    """
 
@cli_base.register_command(config_file_initialize_app=True)
 
@click.option('--remove-missing', is_flag=True,
 
        help='Remove missing repositories from the Kallithea database.')
 
def repo_scan(remove_missing):
 
    """Scan filesystem for repositories.
 

	
 
    def take_action(self, args):
 
        rm_obsolete = args.delete_obsolete
 
        print 'Now scanning root location for new repos ...'
 
        added, removed = repo2db_mapper(ScmModel().repo_scan(),
 
                                        remove_obsolete=rm_obsolete)
 
        print 'Scan completed.'
 
        if added:
 
            print 'Added: %s' % ', '.join(added)
 
        if removed:
 
            print '%s: %s' % ('Removed' if rm_obsolete else 'Missing',
 
                              ', '.join(removed))
 

	
 
    def get_parser(self, prog_name):
 
        parser = super(Command, self).get_parser(prog_name)
 

	
 
        parser.add_argument(
 
            '--delete-obsolete',
 
            action='store_true',
 
            help="Use this flag do delete repositories that are "
 
                 "present in Kallithea database but not on the filesystem",
 
        )
 

	
 
        return parser
 
    Search the configured repository root for new repositories and add them
 
    into Kallithea.
 
    Additionally, report repositories that were previously known to Kallithea
 
    but are no longer present on the filesystem. If option --remove-missing is
 
    given, remove the missing repositories from the Kallithea database.
 
    """
 
    click.echo('Now scanning root location for new repos ...')
 
    added, removed = repo2db_mapper(ScmModel().repo_scan(),
 
                                    remove_obsolete=remove_missing)
 
    click.echo('Scan completed.')
 
    if added:
 
        click.echo('Added: %s' % ', '.join(added))
 
    if removed:
 
        click.echo('%s: %s' % ('Removed' if remove_missing else 'Missing',
 
                          ', '.join(removed)))
setup.py
Show inline comments
 
@@ -120,54 +120,53 @@ from setuptools.command import sdist
 
sdist_org = sdist.sdist
 
class sdist_new(sdist_org):
 
    def initialize_options(self):
 
        sdist_org.initialize_options(self)
 
        self.owner = self.group = 'root'
 
sdist.sdist = sdist_new
 

	
 
packages = setuptools.find_packages(exclude=['ez_setup'])
 

	
 
setuptools.setup(
 
    name='Kallithea',
 
    version=__version__,
 
    description=description,
 
    long_description=long_description,
 
    keywords=keywords,
 
    license=__license__,
 
    author=__author__,
 
    author_email='kallithea@sfconservancy.org',
 
    dependency_links=dependency_links,
 
    url=__url__,
 
    install_requires=requirements,
 
    classifiers=classifiers,
 
    data_files=data_files,
 
    packages=packages,
 
    include_package_data=True,
 
    message_extractors={'kallithea': [
 
            ('**.py', 'python', None),
 
            ('templates/**.mako', 'mako', {'input_encoding': 'utf-8'}),
 
            ('templates/**.html', 'mako', {'input_encoding': 'utf-8'}),
 
            ('public/**', 'ignore', None)]},
 
    zip_safe=False,
 
    entry_points="""
 
    [console_scripts]
 
    kallithea-api =    kallithea.bin.kallithea_api:main
 
    kallithea-gist =   kallithea.bin.kallithea_gist:main
 
    kallithea-config = kallithea.bin.kallithea_config:main
 
    kallithea-cli =    kallithea.bin.kallithea_cli:cli
 

	
 
    [paste.app_factory]
 
    main = kallithea.config.middleware:make_app
 

	
 
    [gearbox.commands]
 
    cache-keys=kallithea.lib.paster_commands.cache_keys:Command
 
    celeryd=kallithea.lib.paster_commands.celeryd:Command
 
    cleanup-repos=kallithea.lib.paster_commands.cleanup:Command
 
    install-iis=kallithea.lib.paster_commands.install_iis:Command
 
    make-index=kallithea.lib.paster_commands.make_index:Command
 
    make-rcext=kallithea.lib.paster_commands.make_rcextensions:Command
 
    repo-scan=kallithea.lib.paster_commands.repo_scan:Command
 
    setup-db=kallithea.lib.paster_commands.setup_db:Command
 
    update-repoinfo=kallithea.lib.paster_commands.update_repoinfo:Command
 
    upgrade-db=kallithea.lib.dbmigrate:UpgradeDb
 
    """,
 
)
0 comments (0 inline, 0 general)