lib: move some template filter functions from utils2 to webutils

While quite Kallithea specific, we prefer to have these functions in webutils
where they soon can be exposed when templates don't need the whole helpers
module.

model: remove unused 'subject' parameter of NotificationModel.create()

The subject of emails is determined with EmailNotificationModel._subj_map,
based on the notification type. The 'subject' parameter passed to
NotificationModel.create is completely unused.

Remove this parameter and update its callers, removing code that is now no
longer used.

hooks: import whole hooks module

Minimize impact if there should be cycles.

model: changes toward import whole modules

*If* there should be circular dependencies, importing 'from' another module
could fail because the module at that time only was partially imported. That
had to be worked around by importing at runtime instead of globally.

Instead, try to always import whole modules. (But we should still try to avoid
cycles.)

lib: move link_to_ref from helpers to utils2

Less reason to import the messy helpers.

lib: move webhelpers2 and friends to webutils

Gives less of the unfortunate use of helpers - especially in low level libs.

lib: move canonical_url & co to webutils

This gives less of the unfortunate use of helpers - especially in model.

vcs: move changeset diff from controller to vcs

Remove unfortunate model dependency on controller ... and put the VCS details
where they belong, with less VCS specific knowledge in the controllers.

lib: move extract_mentioned_users out of utils2

Avoid model.db reference to keep utils2.py independent of Kallithea classes.

hooks: add extensible create-pullrequest hook

Add a hook that will be called when a new pull request is created, and which
can be implemented in the 'extensions' package.

model: handle redundant reviewers in add_reviewers

Don't attempt to add reviewers that are already a reviewer for the specified
PR (redundant reviewers).

Return the list of added and redundant reviewers, for the controller to
handle.

Under normal circumstances, the pullrequest controller already processes the
list of reviewers and only calls add_reviewers for new reviewers. But, there
could be ways were this checking fails, for example due to a race condition
between two simultaneous requests for the same pullrequest, or due to a bug
in the web server framework that causes the same request to be handled
again.

git: fix pull request deletion - don't crash on deleting refs to PR heads

The refs name was passed as unicode string, and that would cause failure like:
File ".../site-packages/dulwich/repo.py", line 720, in __delitem__
if name.startswith(b"refs/") or name == b"HEAD":
TypeError: startswith first arg must be str or a tuple of str, not bytes

Fixed by correctly passing the ref name as bytes, as we do when creating the PR
refs.

Tests added by Mads Kiilerich.

py3: remove safe_unicode in places where it no longer is needed because all strings (except bytes) already *are* unicode strings

(The remaining safe_unicode calls are still needed and can't just be removed,
generally because we in these cases still have to convert from bytes to unicode
strings.)

pull-request: refactor iteration version bump

Avoid relying on exceptions thrown when re.match fails ... and thus please
pytype and make the code better.

py3: add b'' annotations in some places where they will be needed later

Mostly entirely trivial adding of b prefix that is a ignored for py2 ... and
also a bit of related trivial reformatting/refactorings.

model: move notification types from Notification to NotificationModel

This commit is part of the removal of the UI notification feature from
Kallithea, which is not deemed useful in its current form. Only email
notifications are preserved.

As there is no database storage of notifications anymore, the Notification
class will be removed. However, the notification type definitions are still
used for email notifications, and need to live somewhere. As creating
notifications is always passing via NotificationModel, it makes sense to
move the types there.

codingstyle: trivial whitespace fixes

Reported by flake8.

sqlalchemy: fix warnings from running the test suite

Mainly warnings about strings being passed where unicode was expected.

git: add references for Git pull request heads

When a pull request is created, stick a reference to it in the refs/pull
namespace, similarly to what GitHub does, and remove it when the pull
request is deleted. That reference guarantees the commits stay around
and don't get garbage-collected when a PR is rebased or revised.
Also, that makes it possible to access head of historic pull requests
using Git by fetching a ref from the source repository.

Unlike GitHub though, we don't put the ref into the destination repository
and don't copy commits there to prevent undesired repository growth.
Kallithea uses global pull request identifiers, so there should not be
any confusion as to what pull request the ref corresponds to.
We may later provide a shim to redirect users to the source repository
if that deems needed.

pullrequests: fix broken "new PR iteration" handling of ancestor changes

An earlier refactor (5d60c9a391cd) flubbed this code and accidentally
assumed the most recent common ancestor would not change when iterating,
which is of course only the case when there are no merges from 'other' to
'org' among the new revisions.

This was not only not caught during manual testing (nor review), but
neither did the test suite test this. That has now also been rectified.

tg: minimize future diff by some mocking and replacing some pylons imports with tg

No actual tg dependency yet, just a temporary hack faking tg as an alias for
pylons.

Based on work by Alessandro Molina.

pullrequests: introduce "action objects" for PR creation

Inspired by the command design pattern, this attempts the following:

* move policy and business logic from controllers into the model,
* move validation, authorization and execution logic closer together in
the code,
* establish a reusable pattern for modelling higher-level concepts
(like "create a new PR iteration"),
* make error handling more well-defined, and independent of the
controller layer, and
* provide clear separation between, one one hand, the validation and
authorization of a request, and on the other, the actual execution.

pullrequests: pass around reviewer User objects, not IDs

This moves reviewer user validation into the controller and eliminates
the UserInvalidException.

model: remove BaseModel class

The only remaining purpose of this class was to provide the "sa" field,
allowing a custom SQLAlchemy session to be used for model operations.
However, this never actually worked, nor was it used anywhere.

There's always a global SQLAlchemy session associated with the current
thread; using another session for a single function call does not make
any sense (as sessions cannot be mixed), unless the code works carefully
to ensure the two sessions (and all objects loaded from them) are kept
completely separate. Suffice to say that Kallithea does no such thing,
thus there's no need to pretend to support multiple concurrent sessions.

cleanup: refer less to User.DEFAULT_USER

Down the road we might want to identify the default user in another
way than by username.

pullrequests: refactor PullRequestModel().create

The database lookups of the user and repositories are moved outside,
into the callers, which have already looked up the relevant objects,
thus saving two database lookups when creating a PR.

db: it should be "PullRequestReviewer" (singular)

sed -i 's/\bPullRequestReviewers\b/PullRequestReviewer/g' $(hg files)

Just as it's "User", "PullRequest", etc. It's not a collection.

model: move code from __init__.py to base.py

Having too much code, in particular too much imports, inside a package's
__init__.py is a recipe for circular imports, and considered bad practice in
Python [1]

Move out everything from kallithea/model/__init__.py to a new file
kallithea/model/base.py and adapt the existing imports.

[1] http://docs.python-guide.org/en/latest/writing/structure/#packages

model: change ChangesetComment 'user' to 'author'

Rename the 'user_id' field to 'author_id' and replace other references
to the comment 'user' throughout the model. The database column name
'user_id' remain unchanged for now; a later Alembic script can fix the
name of these and other columns to match their Python name.

db: remove superfluous Session.add calls

Don't re-add objects to the SQLAlchemy Session just because they were
modified. Session.add is only for freshly constructed objects that
SQLAlchemy doesn't know about yet.

The rules are quite simple:

When creating a database object by calling the constructor directly, it
must explicitly be added to the session.

When creating an object using a factory function (like "create_repo"),
the returned object has already (by convention) been added to the
session, and should not be added again.

When getting an object from the session (via Session.query or any of the
utility functions that look up objects in the database), it's already
added, and should not be added again. SQLAlchemy notices attribute
modifications automatically for all objects it knows about.

db: add some PullRequest.query() shortcuts

This makes database query code more explicit and increases readability.

E.g. the function name get_pullrequest_cnt_for_user was bad, because the
concept of "pullrequest for user" is incredibly vague, and could refer
to any kind of association between PRs and users. (Quiz time! Does it
mean that the user is the PR owner, that the user is reviewing, or that
the user has commented on the PR and thus is receiving notifications?)

A descriptive name could be "get_open_pull_request_count_for_reviewer",
because the function is indeed only concerned with reviewers and only
with open pull requests. But at this point, we might as well say
PullRequest.query(reviewer_id=user, include_closed=False).count()
which is only slightly longer, and doesn't require us to write dozens
of little wrapper functions (including, any moment now, a separate
function for listing the PRs instead of counting them).

Note that we're not actually going down an abstraction level by doing
this. We're still operating on the concepts of "pull request", "open"
and "reviewer", and are not leaking database implementation details.

The query() shortcuts are designed so they default to not altering
the query. Any processing requires explicit opt-in by the caller.

model: refactor and simplify _get_instance

_get_instance is a BaseModel method, but never uses "self". Instead
it takes a class argument, which indicates that it's better suited as
a classmethod on said classes.

Also rename to something more descriptive, remove leading underscore
since it's not a private API, and refactor for readability.

model: drop BaseModel.cls

It no longer serves a purpose.

model: inline superfluous "helper" methods

__get_changeset_status wasn't even used and has simply been removed.

Notification.__get_notification reimplemented BaseModel._get_instance,
warts and all, and has been replaced by calls to that method.

The rest were just needless indirection, and have been inlined.

notifications: tweak PR mail subject lines

The subject line is used for mail threading in gmail and can thus not be
changed without impacting users ... but now we do it.

* The tag '[Review]' is more spot-on than '[Added]'.
* The subject should be short so it fits on one line, so abbreviate "pull
request" to PR.
* Add the PR owner - convenient for filtering comments on own PRs from comments
on other PRs.

email: add relevant title to subject of emails

For comment and pullrequest mails, add changeset/pullrequest title (shortened
to first 50 characters of first line)

notifications: make more template strings available for mails

To be used soon ...

pull requests: refactor update_reviewers

This avoids a redundant database lookup for every removed reviewer,
and shows usernames (not just IDs) in associated log messages.

pull requests: prevent adding DEFAULT user as reviewer

Add a helper method to resolve reviewers, with an added check to prevent
the adding of the DEFAULT user as reviewer.

The __add_reviewers method, although internal, had a troubling interface
where the method was responsible for resolving reviewers, but the caller
was responsible for resolving mention_recipients. The method is changed
to always take two sets of database User objects, leaving the resolution
to the caller in both cases.

utils: improve extract_mentioned_users usefulness

Previously, extract_mentioned_users performed a regex search, converted
to a set to remove duplicates, converted that set back into a list, and
sorted the list on username, ignoring case. Every single caller of the
function then promptly took the carefully pruned and sorted result and
put it back into a set, rendering the entire exercise pointless.

In addition, every caller also resolved the usernames to database User
objects.

In this changeset, extract_mentioned_users is changed to return a set
of database User objects directly. A new extract_mentioned_usernames
function, which does only username extraction, is kept for the sole
purpose of providing a more testable interface (no database needed).

Bonus feature: The new extract_mentioned_users will prune non-existent
users, as well as the DEFAULT user. This means it is no longer possible
to @mention (and send notifications to) the DEFAULT user.

pullrequest: linkify changesets in html notification email to reviewers

Make it possible to open a given changeset immediately, instead of having to
go to the pullrequest first.

pull requests: blame the right user when an admin adds reviewers to other peoples PRs

Don't send notifications from the PR owner.

pull requests: call it 'owner' instead of 'author'

Prepare for making it possible to transfer ownership of PRs.

pullrequsts: really create a comment when creating a PR and setting status to 'under review'

14d75d4b03cd changed the pullrequest creation status change comment text from
redundant blurb to just an empty string. Empty comments are however only
created if changing the status ... and in this case we didn't tell the comment
creator that it actually was a status change.

As a result of this, we ended up breaking the implicit invariant that all
status updates have a comment. That showed up as errors dereferencing None when
displaying changesets.

pullrequests: detect invalid reviewers and raise HTTPBadRequest

Normally, the creation of a pullrequest with invalid reviewers is not
possible because the list of reviewers is populated from a form element
that only shows valid reviewers.
However, if creating a pullrequest through an API call, invalid reviewers
can be specified but would not be detected. The reviewer would be encoded
in the database as 'NULL'/None, and opening such a pull request would cause
a server error.

Instead, detect invalid reviewers at pullrequest creation/update time and
raise HTTPBadRequest.

pullrequests: use correct References header when adding reviewers to (new?) PRs

Use the same message id when inviting to a PR as we do when commenting on a PR.

pullrequests: add PullRequest.nice_id method

Currently, a pull request id is referenced as #5, and the '#' symbol is
fixed and repeated in several places. This commit adds a class method
make_nice_id, that returns a string reference to a pull request,
currently in the form '#5'.

This function could be overridden by an organization if they need references
in another form, for example PR-5.

The method nice_id uses this for the pullrequest's own id.

notifications: fix threading - use line number

TODO: Use a more stable link, something that includes the filename.

urls: introduce canonical_url config setting

All URLs that are shown or persisted or emailed will use this instead of the
current url.

This is convenient when the server has multiple names - for instance when
transitioning from one protocol or domain or hostname to another.

notifications: insert 'References' mail headers to help MUA threading

gmail is however fubar.

helpers: user email can be unset

Check for not being empty, not for ''.
Also, don't call person() with email, as it can't be caught properly.

Correct licensing information in individual files.

The top-level license file is now LICENSE.md.

Also, in various places where there should have been joint copyright holders
listed, a single copyright holder was listed. It does not appear easy to add
a link to a large list of copyright holders in these places, so it simply
refers to the fact that various authors hold copyright.

In future, if an easy method is discovered to link to a list from those
places, we should do so.

Finally, text is added to LICENSE.md to point to where the full list of
copyright holders is, and that Kallithea as a whole is GPLv3'd.

Second step in two-part process to rename directories.
This is the actual directory rename.