celery: move send_email task to a better home in notification model

Avoid bundling everything from many different layers in one big task library.

This is more feasible now when we don't need kallithea.CELERY_APP set at import
time.

model: remove unused 'subject' parameter of NotificationModel.create()

The subject of emails is determined with EmailNotificationModel._subj_map,
based on the notification type. The 'subject' parameter passed to
NotificationModel.create is completely unused.

Remove this parameter and update its callers, removing code that is now no
longer used.

lib: consistently import helpers the same way

Make it easier to grep for any remaining potential layering-violating use of
helpers.

comments: add changeset author in mail notification subject

In analogy to the handling of comments on pull requests, as changed in
commit d4061c6cc0e2.

celery: fix send_email to work with JSON encoding (Issue #363)

Long time ago, c935bcaf7086 introduced an optional User object parameter to the
send_email task and used the computed full_name_or_username property. Due to the
magic of pickle, that also worked when using Celery to run the task async.

Now, Celery 4 changed the default encoding from Pickle to JSON, which we
anticipated in e539db6cc0da. That broke send_email in some cases, for example
when a user comments on another user's changeset.

Fixed by passing the "From" name as string instead of passing the whole User
object.

Thanks to vyom for reporting.

cleanup: get rid of most "import *"

Apply script generated with the following hack:
(
hg loc '*.py'|xargs pyflakes-2 | sed -rn "s/([^:]*):.*'(.*)' may be undefined, or defined from star imports.*/sed -ri 's,\\\\<\2\\\\>([^=]|$),XXXX.\2\\\\1,g' \1/gp" | sort -u
hg loc '*.py'|xargs pyflakes-2 | sed -rn "s/([^:]*):.* undefined name '(.*)'$/sed -ri 's,\\\\<\2\\\\>([^=]|$),XXXX.\2\\\\1,g' \1/gp" | sort -u
hg loc '*.py'|xargs pyflakes-2 | sed -rn "s/([^:]*):.*'(from .*)\.([^.]*) import \*' used.*/sed -ri 's,\\\\<XXXX\\\\.,\3.,g' \1/gp" | sort -u
hg loc '*.py'|xargs pyflakes-2 | sed -rn "s/([^:]*):.*'(from .*)\.([^.]*) import \*' used.*/sed -ri 's,\2\\\\.\3 .*,\2 import \3,g' \1/gp" | sort -u
) | grep -v kallithea/bin/kallithea_cli_ishell.py > fix2.sh

model: move notification types from Notification to NotificationModel

This commit is part of the removal of the UI notification feature from
Kallithea, which is not deemed useful in its current form. Only email
notifications are preserved.

As there is no database storage of notifications anymore, the Notification
class will be removed. However, the notification type definitions are still
used for email notifications, and need to live somewhere. As creating
notifications is always passing via NotificationModel, it makes sense to
move the types there.

model: remove UI notification feature

This commit is part of the removal of the UI notification feature from
Kallithea, which is not deemed useful in its current form. Only email
notifications are preserved.

tests: remove tests of UI notifications

This commit is part of the removal of the UI notification feature from
Kallithea, which is not deemed useful in its current form. Only email
notifications are preserved.

tests: notifications: increase indentation of multi-line for loop condition

Code in question is:

for foo in [
item1,
item2,
item3]:
action1
action2

With the above indentation, a quick glance at the code does not show where
the actions start.

Using a double indentation for line continuation avoids this problem:

for foo in [
item1,
item2,
item3]:
action1
action2

There are no actual code changes in this commit.

email templates: add HTML entities to test strings

An issue is observed in Outlook rendering of HTML emails, where the HTML
entity &apos; is not recognized and seen as a literal string.

Add some HTML entities to the test strings to be able to see the issue
(after mailing the resulting HTML to an Outlook client)

tests: notification: clarify that cs_target_repo is a URL

In analogy with the fake URLs used for pullrequest notification testing, use
an URL for cs_target_repo. When testing the resulting HTML in a real email
client, it will become obvious that the mail client transforms URLs to
something clickable, regardless of any present <a> tag.

codingstyle: replace "not ... in ..." with "... not in ..."

Reported by flake8.

codingstyle: trivial whitespace fixes

Reported by flake8.

backend: replace Pylons with TurboGears2

Replace the no-longer-supported Pylons application framework by TurboGears2
which is largely compatible/similar to Pylons.
Some interesting history is described at:
https://en.wikipedia.org/wiki/TurboGears

Changes by Dominik Ruf:
- fix sql config in test.ini

Changes by Thomas De Schampheleire:
- set-up of test suite
- tests: 'fix' repo archival test failure
Between Pylons and TurboGears2, there seems to be a small difference in the
headers sent for repository archive files, related to character encoding.
It is assumed that this difference is not important, and that the test
should just align with reality.
- remove need to import helpers/app_globals in lib
TurboGears2 by default expects helpers and app_globals to be available
in lib. For this reason kallithea/lib/__init__.py was originally changed
to include those files. However, this triggered several types of
circular import problems. If module A imported something from lib (e.g.
lib.annotate), and lib.helpers imported (possibly indirectly) module A,
then there was a circular import. Fix this by overruling the relevant
method of tg AppConfig, which is also hinted in the TurboGears2 code.
Hereby, the include of something from lib does not automatically import
helpers, greatly reducing the chances of circular import problems.
- make sure HTTP error '400' uses the custom error pages
TurboGears2 does not by default handle HTTP status code
'400 (Bad Request)' via the custom error page handling, causing a
standard non-styled error page.
- disable transaction manager
Kallithea currently handles its own transactions and does not need the
TurboGears2 transaction manager. However, TurboGears2 tries to enable it
by default and fails, throwing an error during application initialization.
The error itself seemed to be harmless for normal application functioning,
but was nevertheless confusing.
- add backlash as required dependency: backlash is meant as the WebError
replacement in TurboGears2 (originally WebError is part of Pylons). When
debug==true, it provides an interactive debugger in the browser. When
debug==false, backlash is necessary to show backtraces on the console.
- misc fixes

tests: use test_context for tests needing internationalization

Instead of relying on the top-level handling of the translator, use the
newly introduced test_context.

cleanup: use obj.foo_id instead of obj.foo.foo_id

Don't use constructs like obj.user.user_id when obj.user_id works
equally well (and potentially saves a database load).

tests: add as little code as possible in __init__.py

kallithea/tests/__init__.py contained quite a lot of code, including the test
base class TestController. This in itself may be considered bad practice.

Specifically, this poses a problem when using pytest 3.0+, in which asserts
in some files are not automatically rewritten to give improved assert
output. That problem can be fixed by explicitly registering such files for
assertion rewriting, but that register call should be executed _before_ said
files are imported. I.e. if the register call is in
kallithea/tests/__init__.py, assert calls in __init__.py itself can not be
rewritten.

Since the TestController base class does effectively contain asserts, and we
do not want to execute the register call from somewhere outside the
kallithea/tests directory, we need to move the TestController class to
another file (kallithea/tests/base.py) so we can have a register call in
__init__.py before loading base.py.

While not strictly necessary to fix the mentioned pytest problem, we take
the opportunity to fully clean __init__.py and move everything to
the new kallithea/tests/base.py. While doing so, unnecessary imports are
removed, and imports are ordered alphabetically. Explicit imports of symbols
from modules that were already imported as a whole, are removed in favor of
fully qualifying the references (e.g. tempfile._RandomNameSequence).

celeryd: annotate tasks so they can be run directly without run_task

This also makes the system less forgiving about celery configuration problems
and thus easier to debug. I like that.

mails: restructure notification mail content

* Mention "The Thing" in the header, link to online version
* Show The Thing
* A dense summary of the essentials of the context below it

The html indentation is odd in order to make the next diff smaller.

The text version is based on:

for a in kallithea/templates/email_templates/*.html; do sed -e 's,<\([^%/>]\|/[^%>]\)*>,,g' -e 's,\.html",\.txt",g' -e 's,^ *,,g' -e 's/}/|n,unicode}/g' $a > ${a%%.html}.txt ; done

notifications: tweak PR mail subject lines

The subject line is used for mail threading in gmail and can thus not be
changed without impacting users ... but now we do it.

* The tag '[Review]' is more spot-on than '[Added]'.
* The subject should be short so it fits on one line, so abbreviate "pull
request" to PR.
* Add the PR owner - convenient for filtering comments on own PRs from comments
on other PRs.

email: add relevant title to subject of emails

For comment and pullrequest mails, add changeset/pullrequest title (shortened
to first 50 characters of first line)

notifications: make more template strings available for mails

To be used soon ...

tests: fix unicode issues in test_notifications mail template dump test

Other strings should perhaps be unicode too, but that is fixed silently for
pure ascii strings. Let's hope they are all pure.

tests: introduce tests and reference dump for notification mails

The mails are dumped to a tracked html file:
* changes shows up as diffs and are easy to spot and review
* all mails can easily can be investigated in a browser and checked for content
and consistency

The tests are mocking canonical_url because it has deep dependencies to
pylons.url which requires (thread local?) environment setup that the tests
doesn't have.

tests: even smaller setup method refactor for readability

2ffd56e missed u3.

pytest migration: rename TestControllerPytest back to TestController

The name TestControllerPytest was introduced to allow a temporary situation
where nose/unittest and pytest-based tests could coexist.
This situation is now over, so the base test class can be renamed again.

pytest migration: backout declassification of remove_all_notifications

In order to accomodate both nose/unittest and pytest at the same time,
method remove_all_notifications has been extracted from BaseTestCase in
commit 37d713674f63.
Now that nose/unittest is no longer used, we can backout this change again.

pytest migration: models: switch to standard assert statements

Use unittest2pytest to replace unittest-style assert statements (e.g.
assertEqual) with standard Python assert statements to benefit from pytest's
improved reporting on assert failures.

The conversion by unittest2pytest was correct except for:
- 'import pytest' is not automatically added when needed
- line wrapping in string formatting caused a syntax error in the
transformed code. Reported upstream at
https://github.com/pytest-dev/unittest2pytest/issues/3 .
- in assertRaises with a lambda, the lambda needs to be removed

pytest migration: model: convert all tests to TestControllerPytest

The model tests were based on BaseTestCase which does not exist currently
for pytest-style tests. Nevertheless, there seems to be no advantage of
directly subclassing BaseTestCase over subclassing TestControllerPytest.
Thus, keep things simple and use TestControllerPytest.

tests: move remove_all_notifications outside of BaseTestCase

In preparation of allowing real pytest-style test cases (instead of
unittest-style ones), some reorganization is needed in the base test
classes, for one because we want a transition period where pytest and
unittest style test cases can live alongside each other, and secondly
because the pytest style test classes cannot have an __init__ method.

The BaseTestCase class will not be reused for the pytest test cases, but
the remove_all_notifications method will. To avoid having to duplicate it,
and since it does not use any resources from the class (self), move the
method out of the BaseTestCase class to top-level, and export it in
kallithea.tests.

notifications: mark notifications to self "pre-read"

When a user e.g. comments on its own pull request, that user receives a
notification about its own comment. This is slightly dubious behavior,
but at least brings a level of continuity to the notification history.

However, at the very least, the notification should not show as unread.

notification tests: delete notifications before (not after) tests

Don't clean notifications (and changeset comments) *after* the test,
but *before* the test. Other unit tests don't care if they leave
notifications in the database, and neither should these. Rather,
they should ensure their *own* preconditions before testing.

Admittedly, currently only one test leaves a notification in the
database, but more could come along at any time (and why worry?):
TestPullrequestsController.test_create_with_existing_reviewer

tests: improve robustness of notifications tests

The notifications tests make the assumption that there are no notifications
at the start of the test, which is explicitly asserted.

However, any other test, like the one introduced in commit 9a23b444a7fe
(pullrequests: detect invalid reviewers and raise HTTPBadRequest), could add
new notifications to the database, and thus fail these assertions.

Just like the notifications tests already cleaned all notifications at the
end of the test (tearDown), make sure to clean them at the start (setUp)
too.

Second step in two-part process to rename directories.
This is the actual directory rename.