Changeset - 977e7aca38b1
[Not reviewed]
default
1 3 0
Thomas De Schampheleire - 7 years ago 2018-11-18 20:02:17
thomas.de_schampheleire@nokia.com
cli: convert 'gearbox cleanup-repos' into 'kallithea-cli repo-purge-deleted'

Some changes to user-facing text are done.
4 files changed with 98 insertions and 148 deletions:
0 comments (0 inline, 0 general)
docs/usage/general.rst
Show inline comments
 
.. _general:
 

	
 
=======================
 
General Kallithea usage
 
=======================
 

	
 

	
 
Repository deletion
 
-------------------
 

	
 
Currently when an admin or owner deletes a repository, Kallithea does
 
When an admin or owner deletes a repository, Kallithea does
 
not physically delete said repository from the filesystem, but instead
 
renames it in a special way so that it is not possible to push, clone
 
or access the repository.
 

	
 
There is a special command for cleaning up such archived repositories::
 

	
 
    gearbox cleanup-repos --older-than=30d -c my.ini
 
    kallithea-cli repo-purge-deleted -c my.ini --older-than=30d
 

	
 
This command scans for archived repositories that are older than
 
30 days, displays them, and asks if you want to delete them (unless given
 
the ``--dont-ask`` flag). If you host a large amount of repositories with
 
the ``--no-ask`` flag). If you host a large amount of repositories with
 
forks that are constantly being deleted, it is recommended that you run this
 
command via crontab.
 

	
 
It is worth noting that even if someone is given administrative access to
 
Kallithea and deletes a repository, you can easily restore such an action by
 
renaming the repository directory, removing the ``rm__<date>`` prefix.
 

	
 

	
 
File view: follow current branch
 
--------------------------------
 

	
 
In the file view, left and right arrows allow to jump to the previous and next
 
revision. Depending on the way revisions were created in the repository, this
 
could jump to a different branch.  When the checkbox ``Follow current branch``
 
is checked, these arrows will only jump to revisions on the same branch as the
 
currently visible revision.  So for example, if someone is viewing files in the
 
``beta`` branch and marks the `Follow current branch` checkbox, the < and >
 
arrows will only show revisions on the ``beta`` branch.
 

	
 

	
 
Changelog features
 
------------------
 

	
 
The core feature of a repository's ``changelog`` page is to show the revisions
 
in a repository. However, there are several other features available from the
 
changelog.
 

	
 
Branch filter
 
  By default, the changelog shows revisions from all branches in the
 
  repository. Use the branch filter to restrict to a given branch.
 

	
 
Viewing a changeset
 
  A particular changeset can be opened by clicking on either the changeset
 
  hash or the commit message, or by ticking the checkbox and clicking the
 
  ``Show selected changeset`` button at the top.
 

	
 
Viewing all changes between two changesets
 
  To get a list of all changesets between two selected changesets, along with
 
  the changes in each one of them, tick the checkboxes of the first and
 
  last changeset in the desired range and click the ``Show selected changesets``
 
  button at the top. You can only show the range between the first and last
 
  checkbox (no cherry-picking).
 

	
 
  From that page, you can proceed to viewing the overall delta between the
 
  selected changesets, by clicking the ``Compare revisions`` button.
 

	
 
Creating a pull request
 
  You can create a new pull request for the changes of a particular changeset
 
  (and its ancestors) by selecting it and clicking the ``Open new pull request
 
  for selected changesets`` button.
 

	
 

	
 
Permanent repository URLs
 
-------------------------
 

	
 
Due to the complicated nature of repository grouping, URLs of repositories
 
can often change. For example, a repository originally accessible from::
 

	
 
  http://kallithea.example.com/repo_name
 

	
 
would get a new URL after moving it to test_group::
 

	
 
  http://kallithea.example.com/test_group/repo_name
 

	
 
Such moving of a repository to a group can be an issue for build systems and
 
other scripts where the repository paths are hardcoded. To mitigate this,
 
Kallithea provides permanent URLs using the repository ID prefixed with an
 
underscore. In all Kallithea URLs, for example those for the changelog and the
 
file view, a repository name can be replaced by this ``_ID`` string. Since IDs
 
are always the same, moving the repository to a different group will not affect
 
such URLs.
 

	
 
In the example, the repository could also be accessible as::
 

	
 
  http://kallithea.example.com/_<ID>
 

	
 
The ID of a given repository can be shown from the repository ``Summary`` page,
 
by selecting the ``Show by ID`` button next to ``Clone URL``.
 

	
 

	
 
Email notifications
 
-------------------
 

	
 
With email settings properly configured in the Kallithea
 
configuration file, Kallithea will send emails on user registration and when
 
errors occur.
 

	
 
Emails are also sent for comments on changesets. In this case, an email is sent
 
to the committer of the changeset (if known to Kallithea), to all reviewers of
 
the pull request (if applicable) and to all people mentioned in the comment
 
using @mention notation.
 

	
 

	
 
Trending source files
 
---------------------
 

	
kallithea/bin/kallithea_cli_repo.py
Show inline comments
 
# -*- coding: utf-8 -*-
 
# This program is free software: you can redistribute it and/or modify
 
# it under the terms of the GNU General Public License as published by
 
# the Free Software Foundation, either version 3 of the License, or
 
# (at your option) any later version.
 
#
 
# This program is distributed in the hope that it will be useful,
 
# but WITHOUT ANY WARRANTY; without even the implied warranty of
 
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 
# GNU General Public License for more details.
 
#
 
# You should have received a copy of the GNU General Public License
 
# along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
"""
 
This file was forked by the Kallithea project in July 2014 and later moved.
 
Original author and date, and relevant copyright and licensing information is below:
 
:created_on: Feb 9, 2013
 
:author: marcink
 
:copyright: (c) 2013 RhodeCode GmbH, and others.
 
:license: GPLv3, see LICENSE.md for more details.
 
"""
 
import click
 
import kallithea.bin.kallithea_cli_base as cli_base
 

	
 
from kallithea.lib.utils import repo2db_mapper
 
from kallithea.lib.utils2 import safe_unicode
 
from kallithea.model.db import Repository
 
import datetime
 
import os
 
import re
 
import shutil
 

	
 
from kallithea.lib.paster_commands.common import ask_ok
 
from kallithea.lib.utils import repo2db_mapper, REMOVED_REPO_PAT
 
from kallithea.lib.utils2 import safe_unicode, safe_str
 
from kallithea.model.db import Repository, Ui
 
from kallithea.model.meta import Session
 
from kallithea.model.scm import ScmModel
 

	
 
@cli_base.register_command(config_file_initialize_app=True)
 
@click.option('--remove-missing', is_flag=True,
 
        help='Remove missing repositories from the Kallithea database.')
 
def repo_scan(remove_missing):
 
    """Scan filesystem for repositories.
 

	
 
    Search the configured repository root for new repositories and add them
 
    into Kallithea.
 
    Additionally, report repositories that were previously known to Kallithea
 
    but are no longer present on the filesystem. If option --remove-missing is
 
    given, remove the missing repositories from the Kallithea database.
 
    """
 
    click.echo('Now scanning root location for new repos ...')
 
    added, removed = repo2db_mapper(ScmModel().repo_scan(),
 
                                    remove_obsolete=remove_missing)
 
    click.echo('Scan completed.')
 
    if added:
 
        click.echo('Added: %s' % ', '.join(added))
 
    if removed:
 
        click.echo('%s: %s' % ('Removed' if remove_missing else 'Missing',
 
                          ', '.join(removed)))
 

	
 
@cli_base.register_command(config_file_initialize_app=True)
 
@click.argument('repositories', nargs=-1)
 
def repo_update_metadata(repositories):
 
    """
 
    Update repository metadata in database from repository content.
 

	
 
    In normal operation, Kallithea will keep caches up-to-date
 
    automatically. However, if repositories are externally modified, e.g. by
 
    a direct push via the filesystem rather than via a Kallithea URL,
 
    Kallithea is not aware of it. In this case, you should manually run this
 
    command to update the repository cache.
 

	
 
    If no repositories are specified, the caches of all repositories are
 
    updated.
 
    """
 
    if not repositories:
 
        repo_list = Repository.query().all()
 
    else:
 
        repo_names = [safe_unicode(n.strip()) for n in repositories]
 
        repo_list = list(Repository.query()
 
                        .filter(Repository.repo_name.in_(repo_names)))
 

	
 
    for repo in repo_list:
 
        # update latest revision metadata in database
 
        repo.update_changeset_cache()
 
        # invalidate in-memory VCS object cache... will be repopulated on
 
        # first access
 
        repo.set_invalidate()
 

	
 
    Session().commit()
 

	
 
    click.echo('Updated database with information about latest change in the following %s repositories:' % (len(repo_list)))
 
    click.echo('\n'.join(repo.repo_name for repo in repo_list))
 

	
 
@cli_base.register_command(config_file_initialize_app=True)
 
@click.option('--ask/--no-ask', default=True, help='Ask for confirmation or not. Default is --ask.')
 
@click.option('--older-than',
 
        help="""Only purge repositories that have been removed at least the given time ago.
 
        For example, '--older-than=30d' purges repositories deleted 30 days ago or longer.
 
        Possible suffixes: d (days), h (hours), m (minutes), s (seconds).""")
 
def repo_purge_deleted(ask, older_than):
 
    """Purge backups of deleted repositories.
 

	
 
    When a repository is deleted via the Kallithea web interface, the actual
 
    data is still present on the filesystem but set aside using a special name.
 
    This command allows to delete these files permanently.
 
    """
 
    def _parse_older_than(val):
 
        regex = re.compile(r'((?P<days>\d+?)d)?((?P<hours>\d+?)h)?((?P<minutes>\d+?)m)?((?P<seconds>\d+?)s)?')
 
        parts = regex.match(val)
 
        if not parts:
 
            return
 
        parts = parts.groupdict()
 
        time_params = {}
 
        for (name, param) in parts.iteritems():
 
            if param:
 
                time_params[name] = int(param)
 
        return datetime.timedelta(**time_params)
 

	
 
    def _extract_date(name):
 
        """
 
        Extract the date part from rm__<date> pattern of removed repos,
 
        and convert it to datetime object
 

	
 
        :param name:
 
        """
 
        date_part = name[4:19]  # 4:19 since we don't parse milliseconds
 
        return datetime.datetime.strptime(date_part, '%Y%m%d_%H%M%S')
 

	
 
    repos_location = Ui.get_repos_location()
 
    to_remove = []
 
    for dn_, dirs, f in os.walk(safe_str(repos_location)):
 
        alldirs = list(dirs)
 
        del dirs[:]
 
        if ('.hg' in alldirs or
 
            '.git' in alldirs or
 
            '.svn' in alldirs or
 
            'objects' in alldirs and ('refs' in alldirs or 'packed-refs' in f)):
 
            continue
 
        for loc in alldirs:
 
            if REMOVED_REPO_PAT.match(loc):
 
                to_remove.append([os.path.join(dn_, loc),
 
                                  _extract_date(loc)])
 
            else:
 
                dirs.append(loc)
 
        if dirs:
 
            click.echo('Scanning: %s' % dn_)
 

	
 
    # filter older than (if present)!
 
    now = datetime.datetime.now()
 
    if older_than:
 
        to_remove_filtered = []
 
        older_than_date = _parse_older_than(older_than)
 
        for name, date_ in to_remove:
 
            repo_age = now - date_
 
            if repo_age > older_than_date:
 
                to_remove_filtered.append([name, date_])
 

	
 
        to_remove = to_remove_filtered
 
        click.echo('Purging %s deleted repos older than %s (%s)'
 
            % (len(to_remove), older_than, older_than_date))
 
    else:
 
        click.echo('Purging all %s deleted repos' % len(to_remove))
 

	
 
    if not ask or not to_remove:
 
        # don't ask just remove !
 
        remove = True
 
    else:
 
        remove = ask_ok('The following repositories will be removed completely:\n%s\n'
 
                'Do you want to proceed? [y/n] '
 
                % '\n'.join(['%s deleted on %s' % (safe_str(x[0]), safe_str(x[1]))
 
                                     for x in to_remove]))
 

	
 
    if remove:
 
        for path, date_ in to_remove:
 
            click.echo('Purging repository %s' % path)
 
            shutil.rmtree(path)
 
    else:
 
        click.echo('Nothing done, exiting...')
kallithea/lib/paster_commands/cleanup.py
Show inline comments
 
deleted file
setup.py
Show inline comments
 
@@ -67,104 +67,103 @@ requirements = [
 
if sys.version_info < (2, 7):
 
    requirements.append("importlib == 1.0.1")
 
    requirements.append("argparse")
 

	
 
if not is_windows:
 
    requirements.append("bcrypt >= 3.1.0, < 3.2")
 

	
 
dependency_links = [
 
]
 

	
 
classifiers = [
 
    'Development Status :: 4 - Beta',
 
    'Environment :: Web Environment',
 
    'Framework :: Pylons',
 
    'Intended Audience :: Developers',
 
    'License :: OSI Approved :: GNU General Public License (GPL)',
 
    'Operating System :: OS Independent',
 
    'Programming Language :: Python',
 
    'Programming Language :: Python :: 2.6',
 
    'Programming Language :: Python :: 2.7',
 
    'Topic :: Software Development :: Version Control',
 
]
 

	
 

	
 
# additional files from project that goes somewhere in the filesystem
 
# relative to sys.prefix
 
data_files = []
 

	
 
description = ('Kallithea is a fast and powerful management tool '
 
               'for Mercurial and Git with a built in push/pull server, '
 
               'full text search and code-review.')
 

	
 
keywords = ' '.join([
 
    'kallithea', 'mercurial', 'git', 'code review',
 
    'repo groups', 'ldap', 'repository management', 'hgweb replacement',
 
    'hgwebdir', 'gitweb replacement', 'serving hgweb',
 
])
 

	
 
# long description
 
README_FILE = 'README.rst'
 
try:
 
    long_description = open(README_FILE).read()
 
except IOError as err:
 
    sys.stderr.write(
 
        "[WARNING] Cannot find file specified as long_description (%s)\n"
 
        % README_FILE
 
    )
 
    long_description = description
 

	
 
import setuptools
 

	
 
# monkey patch setuptools to use distutils owner/group functionality
 
from setuptools.command import sdist
 
sdist_org = sdist.sdist
 
class sdist_new(sdist_org):
 
    def initialize_options(self):
 
        sdist_org.initialize_options(self)
 
        self.owner = self.group = 'root'
 
sdist.sdist = sdist_new
 

	
 
packages = setuptools.find_packages(exclude=['ez_setup'])
 

	
 
setuptools.setup(
 
    name='Kallithea',
 
    version=__version__,
 
    description=description,
 
    long_description=long_description,
 
    keywords=keywords,
 
    license=__license__,
 
    author=__author__,
 
    author_email='kallithea@sfconservancy.org',
 
    dependency_links=dependency_links,
 
    url=__url__,
 
    install_requires=requirements,
 
    classifiers=classifiers,
 
    data_files=data_files,
 
    packages=packages,
 
    include_package_data=True,
 
    message_extractors={'kallithea': [
 
            ('**.py', 'python', None),
 
            ('templates/**.mako', 'mako', {'input_encoding': 'utf-8'}),
 
            ('templates/**.html', 'mako', {'input_encoding': 'utf-8'}),
 
            ('public/**', 'ignore', None)]},
 
    zip_safe=False,
 
    entry_points="""
 
    [console_scripts]
 
    kallithea-api =    kallithea.bin.kallithea_api:main
 
    kallithea-gist =   kallithea.bin.kallithea_gist:main
 
    kallithea-config = kallithea.bin.kallithea_config:main
 
    kallithea-cli =    kallithea.bin.kallithea_cli:cli
 

	
 
    [paste.app_factory]
 
    main = kallithea.config.middleware:make_app
 

	
 
    [gearbox.commands]
 
    celeryd=kallithea.lib.paster_commands.celeryd:Command
 
    cleanup-repos=kallithea.lib.paster_commands.cleanup:Command
 
    install-iis=kallithea.lib.paster_commands.install_iis:Command
 
    make-index=kallithea.lib.paster_commands.make_index:Command
 
    make-rcext=kallithea.lib.paster_commands.make_rcextensions:Command
 
    setup-db=kallithea.lib.paster_commands.setup_db:Command
 
    upgrade-db=kallithea.lib.dbmigrate:UpgradeDb
 
    """,
 
)
0 comments (0 inline, 0 general)