celery: move Whoosh indexing task to whoosh library where it belongs

Avoid bundling everything from many different layers in one big task library.

This is more feasible now when we don't need kallithea.CELERY_APP set at import
time.

indexers: drop sys.path mangling in daemon

The original comment in b153a51b1d3b was "to get the pylons_app import" ...
which seems like a bad workaround for not having installed the app properly
before running.

vcs: drop repo contact ... and the odd uses of it

A half-baked and essentially unused hgweb feature.

The reference in get_commits_stats doesn't seem reachable in any relevant use
cases.

config: move config.conf to kallithea.lib

kallithea.config is mainly the WSGI entry point with TG stuff, and thus a high
level controller thing - not a place to store configuration.

Note: The content of conf.py is kind of similar to what we have in
kallithea/__init__.py . These two should perhaps be merged somehow.

py3: remove safe_unicode in places where it no longer is needed because all strings (except bytes) already *are* unicode strings

(The remaining safe_unicode calls are still needed and can't just be removed,
generally because we in these cases still have to convert from bytes to unicode
strings.)

py3: drop .keys when we don't need them

In python 3 they will be iterators and mostly useless ... but they are already
mostly redundant in py2.

vcs: always return bytes from node.content

We will rather have the unicode conversions explicit.

Note: Py3 bytes doesn't have .startswith - replace that with a regexp.

vcs: drop Repository.name_unicode

It wouldn't make much sense in py3 ... and is barely used.

py3: cleanup map usage and avoid py3 ambiguity

Based on 2to3 -f map ... but replace map with something more explicit (unless
born and raised in a lisp world) (but sometimes slightly more verbose).

search: tweak logging from index-create

Try to make it slightly more readable.

codingstyle: replace "not ... in ..." with "... not in ..."

Reported by flake8.

codingstyle: trivial whitespace fixes

Reported by flake8.

model: remove BaseModel class

The only remaining purpose of this class was to provide the "sa" field,
allowing a custom SQLAlchemy session to be used for model operations.
However, this never actually worked, nor was it used anywhere.

There's always a global SQLAlchemy session associated with the current
thread; using another session for a single function call does not make
any sense (as sessions cannot be mixed), unless the code works carefully
to ensure the two sessions (and all objects loaded from them) are kept
completely separate. Suffice to say that Kallithea does no such thing,
thus there's no need to pretend to support multiple concurrent sessions.

indexers: use correct full repository name, which contains group name, at indexing

Before this revision, searching under the specific repository could
cause unexpected result, because repository names used for indexing didn't
contain the group name.

This issue was introduced by 8b7c0ef62427, which uses
repo.name_unicode as repository name instead of
safe_unicode(repo_name) to reduce unicode conversion cost while
repetition at indexing.

To use correct repository name at indexing, this revision replaces
repo.name_unicode by safe_unicode(repo_name). Reducing cost of repeated
unicode conversion cost while will (and should) be addressed in the
future.

This revision also adds a comment to BaseRepository.name property, to
avoid similar misunderstandings in the future.

search: make "repository:" condition work case-insensitively as expected

Before this revision, "repository:" condition at searching for "Commit
messages" never shows revisions in a repository, of which name uses
upper case letter.

Using ID for "repository" of CHGSETS_SCHEMA preserves case of
repository name at indexing. On the other hand, search condition
itself is forcibly lowered before parsing.

- files in repository "FOO" is indexed as "FOO" in "repository" field
- "repository:FOO" condition is treated as "repository:foo:

Then, indexing search itself is executed case-sensitively. Therefore,
"repository:FOO" condition never show revisions in repository "FOO".

But just making "repository" of CHGSETS_SCHEMA case-insensitive isn't
reasonable enough, because it breaks assumptions below, if there is
case-insensitive name collision between repositories, even though
Kallithea itself can manage such repositories at same time.

- combination of "raw_id" (= revision hash ID) and "repository" is
unique between all known revisions under Kallithea

CHGSETS_SCHEMA assumes this.

This unique-ness is required by Whoosh library to determine
whether index table should be updated or not for that repository.

- searching in a repository shows only revisions in that repository

Before this revision, this filtering is achieve by "repository:"
condition with case-preserved repository name from requested URL.

To make "repository:" search condition work case-insensitively as
expected (without any violation of assumptions above), this revision
does:

- make "repository" of CHGSETS_SCHEMA case-insensitive by
"analyzer=ICASEIDANALYZER"

- introduce "repository_rawname" into SCHEMA and CHGSETS_SCHEMA, to
ensure assumptions described above, by preserving case of
repository name

"repository_rawname" of SCHEMA uses not ID but TEXT, because the
former disable "positions" feature, which is required for
highlight-ing file content (see previous revision for detail).

This revision requires full re-building index tables, because indexing
schemas are changed.

cleanup: get rid of dn as shortcut for os.path.dirname

We keep 'dirname' as shortcut despite having removed the 'join' shortcut:
* the name is less ambiguous than 'join'.
* dirname is often applied multiple times - spelling it out would be too
verbose.

cleanup: get rid of jn as shortcut for os.path.join

It is not used in that many places so it is better to be explicit and use os.path.join .

Discussed on https://bitbucket.org/domruf/kallithea/commits/1da05f42bca3f7042c444fda13a1ae1f6b9c300f#comment-2948124 .

Modified by Mads Kiilerich.

indexer: skip documents that can't be retrieved - probably because encoding issues (Issue #175)

This doesn't fix the encoding issue but it makes it less fatal.

indexers: introduce list of filenames for indexing

Before this patch, we cannot search files that without file extension (like
Makefile) and dotfiles (like .hgtags).

This patch makes it possible to search for these files by introducing a list
of filenames for indexing. The list is currently empty.

remove vestiges of Python 2.5 support

We only support Python 2.6 and 2.7; hence we do not need to import
with-statement support from __future__.

indexer: rework get_node to handle alternative path element separators

Problem seen on Windows by Dominik Ruf.

Correct licensing information in individual files.

The top-level license file is now LICENSE.md.

Also, in various places where there should have been joint copyright holders
listed, a single copyright holder was listed. It does not appear easy to add
a link to a large list of copyright holders in these places, so it simply
refers to the fact that various authors hold copyright.

In future, if an easy method is discovered to link to a list from those
places, we should do so.

Finally, text is added to LICENSE.md to point to where the full list of
copyright holders is, and that Kallithea as a whole is GPLv3'd.

Second step in two-part process to rename directories.
This is the actual directory rename.