controllers: move controllers base class from lib/base to controllers

TG quickstart put it in lib/base.py , but it fits better on the controllers
layer as a base there.

The contributing docs were a bit ahead of time ... but with a typo.

lib: move canonical_url & co to webutils

This gives less of the unfortunate use of helpers - especially in model.

journal: don't include email in author name - avoid double data

In journal RSS feed, report author as:
<author>name@example.com (User Name)</author>
instead of double email due to:
<author>name@example.com (User Name &lt;name@example.com&gt;)</author>

In the journal ATOM feed, report author as:
<author>
<name>user Name</name>
<email>test_admin@example.com</email>
</author>
instead of using double email due to:
<name>name@example.com (User Name &lt;name@example.com&gt;)</name>

page: consistently invoke Page with kwargs instead of explicitly providing a custom url generator

Drop the option of providing an url maker as 'url', and thus also hardcode the
assumption that the page parameter is called page.

auth: drop api_access_controllers_whitelist and give API key auth same access as other kinds of auth

All authentication methods are created equal. There is no point in
discriminating api key authentication and limit it to few APIs.

auth: refactor to introduce @LoginRequired(allow_default_user=True) and deprecate @NotAnonymous()

It was error prone that @LoginRequired defaulted to allow anonymous users (if
'default' user is enabled). See also 245b4e3abf39.

Refactor code to make it more explicit and safe by default: Deprecate
@NotAnonymous by making it the default of @LoginRequired. That will make it
safe by default.

To preserve same functionality, set allow_default_user=True in all the cases
where @LoginRequired was *not* followed by @NotAnonymous or other permission
checks - that was done with some script hacks:
sed -i 's/@LoginRequired(\(..*\))/@LoginRequired(\1, allow_default_user=True)/g' `hg mani`
sed -i 's/@LoginRequired()/@LoginRequired(allow_default_user=True)/g' `hg mani`
perl -0pi -e 's/\@LoginRequired\(allow_default_user=True\)\n\s*\@NotAnonymous\(\)/\@LoginRequired()/g' `hg mani`
perl -0pi -e 's/\@LoginRequired\(allow_default_user=True\)(\n\s*\@Has(Repo)?Permission)/\@LoginRequired()\1/g' `hg mani`

It has been reviewed that all uses of allow_default_user=True are in places
where the there indeed wasn't any checking for default user before. These may
or may not be correct, but now they are explicit and can be spotted and fixed.

The few remaining uses of @NotAnonymous should probably be removed somehow.

codingstyle: trivial whitespace fixes

Reported by flake8.

controllers: rename __before__ to _before in preparation of TurboGears2

__before__ in Pylons is called _before in TurboGears2. We can prepare this
rename already in Pylons-based Kallithea, so that the real TG2 migration
commit just changes the BaseController.

Since TurboGears2 _before can pass extra arguments, we add *args and
**kwargs parameters as well.

tg: minimize future diff by some mocking and replacing some pylons imports with tg

No actual tg dependency yet, just a temporary hack faking tg as an alias for
pylons.

Based on work by Alessandro Molina.

templates: properly escape inline JavaScript values

TLDR: Kallithea has issues with escaping values for use in inline JS.
Despite judicious poking of the code, no actual security vulnerabilities
have been found, just lots of corner-case bugs. This patch fixes those,
and hardens the code against actual security issues.

The long version:

To embed a Python value (typically a 'unicode' plain-text value) in a
larger file, it must be escaped in a context specific manner. Example:

>>> s = u'<script>alert("It\'s a trap!");</script>'

1) Escaped for insertion into HTML element context

>>> print cgi.escape(s)
&lt;script&gt;alert("It's a trap!");&lt;/script&gt;

2) Escaped for insertion into HTML element or attribute context

>>> print h.escape(s)
&lt;script&gt;alert(&#34;It&#39;s a trap!&#34;);&lt;/script&gt;

This is the default Mako escaping, as usually used by Kallithea.

3) Encoded as JSON

>>> print json.dumps(s)
"<script>alert(\"It's a trap!\");</script>"

4) Escaped for insertion into a JavaScript file

>>> print '(' + json.dumps(s) + ')'
("<script>alert(\"It's a trap!\");</script>")

The parentheses are not actually required for strings, but may be needed
to avoid syntax errors if the value is a number or dict (object).

5) Escaped for insertion into a HTML inline <script> element

>>> print h.js(s)
("\x3cscript\x3ealert(\"It's a trap!\");\x3c/script\x3e")

Here, we need to combine JS and HTML escaping, further complicated by
the fact that "<script>" tag contents can either be parsed in XHTML mode
(in which case '<', '>' and '&' must additionally be XML escaped) or
HTML mode (in which case '</script>' must be escaped, but not using HTML
escaping, which is not available in HTML "<script>" tags). Therefore,
the XML special characters (which can only occur in string literals) are
escaped using JavaScript string literal escape sequences.

(This, incidentally, is why modern web security best practices ban all
use of inline JavaScript...)

Unsurprisingly, Kallithea does not do (5) correctly. In most cases,
Kallithea might slap a pair of single quotes around the HTML escaped
Python value. A typical benign example:

$('#child_link').html('${_('No revisions')}');

This works in English, but if a localized version of the string contains
an apostrophe, the result will be broken JavaScript. In the more severe
cases, where the text is user controllable, it leaves the door open to
injections. In this example, the script inserts the string as HTML, so
Mako's implicit HTML escaping makes sense; but in many other cases, HTML
escaping is actually an error, because the value is not used by the
script in an HTML context.

The good news is that the HTML escaping thwarts attempts at XSS, since
it's impossible to inject syntactically valid JavaScript of any useful
complexity. It does allow JavaScript errors and gibberish to appear on
the page, though.

In these cases, the escaping has been fixed to use either the new 'h.js'
helper, which does JavaScript escaping (but not HTML escaping), OR the
new 'h.jshtml' helper (which does both), in those cases where it was
unclear if the value might be used (by the script) in an HTML context.
Some of these can probably be "relaxed" from h.jshtml to h.js later, but
for now, using h.jshtml fixes escaping and doesn't introduce new errors.

In a few places, Kallithea JSON encodes values in the controller, then
inserts the JSON (without any further escaping) into <script> tags. This
is also wrong, and carries actual risk of XSS vulnerabilities. However,
in all cases, security vulnerabilities were narrowly avoided due to other
filtering in Kallithea. (E.g. many special characters are banned from
appearing in usernames.) In these cases, the escaping has been fixed
and moved to the template, making it immediately visible that proper
escaping has been performed.

Mini-FAQ (frequently anticipated questions):

Q: Why do everything in one big, hard to review patch?
Q: Why add escaping in specific case FOO, it doesn't seem needed?

Because the goal here is to have "escape everywhere" as the default
policy, rather than identifying individual bugs and fixing them one
by one by adding escaping where needed. As such, this patch surely
introduces a lot of needless escaping. This is no different from
how Mako/Pylons HTML escape everything by default, even when not
needed: it's errs on the side of needless work, to prevent erring
on the side of skipping required (and security critical) work.

As for reviewability, the most important thing to notice is not where
escaping has been introduced, but any places where it might have been
missed (or where h.jshtml is needed, but h.js is used).

Q: The added escaping is kinda verbose/ugly.

That is not a question, but yes, I agree. Hopefully it'll encourage us
to move away from inline JavaScript altogether. That's a significantly
larger job, though; with luck this patch will keep us safe and secure
until such a time as we can implement the real fix.

Q: Why not use Mako filter syntax ("${val|h.js}")?

Because of long-standing Mako bug #140, preventing use of 'h' in
filters.

Q: Why not work around bug #140, or even use straight "${val|js}"?

Because Mako still applies the default h.escape filter before the
explicitly specified filters.

Q: Where do we go from here?

Longer term, we should stop doing variable expansions in script blocks,
and instead pass data to JS via e.g. data attributes, or asynchronously
using AJAX calls. Once we've done that, we can remove inline JavaScript
altogether in favor of separate script files, and set a strict Content
Security Policy explicitly blocking inline scripting, and thus also the
most common kind of cross-site scripting attack.

model: remove BaseModel class

The only remaining purpose of this class was to provide the "sa" field,
allowing a custom SQLAlchemy session to be used for model operations.
However, this never actually worked, nor was it used anywhere.

There's always a global SQLAlchemy session associated with the current
thread; using another session for a single function call does not make
any sense (as sessions cannot be mixed), unless the code works carefully
to ensure the two sessions (and all objects loaded from them) are kept
completely separate. Suffice to say that Kallithea does no such thing,
thus there's no need to pretend to support multiple concurrent sessions.

controllers: avoid setting request state in controller instances - set it in the thread global request variable

In TurboGears, controllers are singletons and we should avoid using instance
variables for any volatile data. Instead, use the "global thread local" request
context.

With everything in request, some use of c is dropped.

Note: kallithea/controllers/api/__init__.py still use instance variables that
will cause problems with TurboGears.

controllers: avoid setting constants as controller instance variables in __before__

Setting constants in __before__ is a weird pattern and we can do fine without doing it.
That makes it more clear that there is no state in the controller instances.

Real constants can just be set at the module level.

Some values depend on configuration and can thus probably not be set as
constants at module import time but could perhaps be set in __init__. But
reading configuration directly when needed will probably be just as good.

db: rename UserFollowing.follows_repo_id to follows_repository_id

Since the relationship is 'follows_repository', rename the column to be
'follows_repository_id', not 'follows_repo_id'. This also makes the
Python column name match the actual database column name.

(The inconsistency dates back to early RhodeCode days.)

helpers: move Page/RepoPage to a separate file page.py

The Page and RepoPage classes are not used from templates and thus do not
belong in helpers.
They could have been put in utils.py, but because they are completely
standalone we can easily split them to a separate file.

cleanup: use obj.foo_id instead of obj.foo.foo_id

Don't use constructs like obj.user.user_id when obj.user_id works
equally well (and potentially saves a database load).

db: PullRequest/Repository/RepoGroup/UserGroup: change 'user' to 'owner'

Rename the 'user' and 'user_id' fields on the four classes to something
more informative. The database column names remain unchanged for now;
a later Alembic script can fix the name of these and other columns to
match their Python name.

This might break rcextensions, though, and external scripts that use
the HTML form interface.

Turbogears2 migration: replace pylons.url by kallithea.config.routing.url

In preparation for the migration to Turbogears2, introduce a
kallithea.config.routing.url to replace pylons.url. The implementation is
basically the same: wrap around routes.url().

This change involves:
- a number of import statement changes
- fixing some tests in test_libs.py; to avoid duplication, the different
implementations of fake_url were grouped in one place.

This change was first proposed by Alessandro Molina in his initial
port. Following changes were made afterwards:
- move UrlGenerator from kallithea.lib.utils to kallithea.config.routing
- add documentation to UrlGenerator
- kallithea/lib/auth.py used url_for instead of url, for no apparent reason
so this was changed.
- fix libs tests
- rebase onto Pylons-based Kallithea first

Turbogears2 migration: remove some references to Pylons in comments

In order to minimize the diff of the actual Turbogears2 migration, this
commit already removes certain unnecessary references to Pylons from the
Kallithea source base.
Places where the reference to Pylons is important are still kept for now, as
well as references in kallithea/config where many changes are made for
Turbogears2 anyway.

controllers: simplify request.GET.get for safe_int

Don't specify the default value twice - the one for safe_int is enough.

spelling: fix title casing on various translated strings

Primarily captions on other not-just-text and data.

controllers: don't pass rendered templates in context variables

Some controllers used the followifng pattern:
- render a data template into a context variable
- for partial (ajax) requests, return the contents of this variable
- for full-page requests, render the full page, which expands the value of
the context variable

Instead, avoid context variables let the controller simply render the full or partial page, and let
the full page template include the partial page.

Remove this context variable for templating and use render exclusively.
From templates, use %include instead of context variables.

This in line with the suggestions in the Pylons documentation:
http://pylons-webframework.readthedocs.org/en/latest/helpers.html#partial-updates-with-ajax

cleanup: avoid some 'except Exception' catching - catch specific exceptions or log it and show what happened

This has a risk of introducing regressions ... but we want to get rid of all
exception muting and make the whole system less fragile and easier to debug.

urls: introduce canonical_url config setting

All URLs that are shown or persisted or emailed will use this instead of the
current url.

This is convenient when the server has multiple names - for instance when
transitioning from one protocol or domain or hostname to another.

Correct licensing information in individual files.

The top-level license file is now LICENSE.md.

Also, in various places where there should have been joint copyright holders
listed, a single copyright holder was listed. It does not appear easy to add
a link to a large list of copyright holders in these places, so it simply
refers to the fact that various authors hold copyright.

In future, if an easy method is discovered to link to a list from those
places, we should do so.

Finally, text is added to LICENSE.md to point to where the full list of
copyright holders is, and that Kallithea as a whole is GPLv3'd.

Second step in two-part process to rename directories.
This is the actual directory rename.