controllers: move controllers base class from lib/base to controllers

TG quickstart put it in lib/base.py , but it fits better on the controllers
layer as a base there.

The contributing docs were a bit ahead of time ... but with a typo.

lib: move webhelpers2 and friends to webutils

Gives less of the unfortunate use of helpers - especially in low level libs.

config: move config.conf to kallithea.lib

kallithea.config is mainly the WSGI entry point with TG stuff, and thus a high
level controller thing - not a place to store configuration.

Note: The content of conf.py is kind of similar to what we have in
kallithea/__init__.py . These two should perhaps be merged somehow.

summary: use calendar.monthrange instead of internal calendar.mday

Fix pytype warning:
No attribute 'mdays' on module 'calendar' [module-attr]

This also takes care of handling leap-years correctly. Not that it matters, but
very nice ...

summary: compute lang_stats consistently

Visiting a /statistics with py3 would fail with:
... in statistics
sorted(lang_stats, reverse=True, key=lambda k: k[1])[:10]
TypeError: '<' not supported between instances of 'dict' and 'dict'

The "summary" computation didn't have that problem. And it put '?' as
description for unknown extensions. And it had stable output as it also sorted
on the file extension as secondary key. Just use that.

summary: drop no_data in all possible ways

The controllers some odd "is" checks did and removed has been.

The javascript variables were never used.

summary.html didn't use no_data at all.

The only real use was in statistics.html ... where it seems like a better fit
to just use c.stats_percentage .

lib: clean up ext_json and how it is used - avoid monkey patching

Note that py3 json.dumps will return ASCII (with all unicode escaped) encoded
as str. But we generally want JSON as bytes (which json.loads also can read),
so also wrap the result with ascii_bytes in many places.

lib: handle both HTML, unsafe strings, and exceptions passed to helpers.flash()

Before, h.flash would trust any input to contain html ... and callers would
convert exceptions to string, often with a simple str() or unicode() ... which
really didn't deserve to be trusted.

Instead, only trust messages that have a __html__ and escape anything else ...
but also apply str/unicode on the parameter so the caller doesn't have to but
*can* pass an exception directly.

vcs: always return bytes from node.content

We will rather have the unicode conversions explicit.

Note: Py3 bytes doesn't have .startswith - replace that with a regexp.

py3: only use safe_str for string conversion - not for arbitrary __str__ invocation

Usually, we actually just want unicode string, and it is better to just unicode().

page: replace RepoPage with Page given the reverse collection

That seems to be all RepoPage did ...

We must still take care to make sure the collection works correctly, even when
filtered so indices might be higher than repo length. vcs module takes care of
that by internally creating a list of hashes (which it can reverse), while the
Changeset instances are only created on demand. We can save some resources by
not retrieving the whole list of Changesets just to reverse it so we can use a
few entries.

summary: handle repo like changelog does

Show "There are no changesets yet" instead of relying on pager to silently eat
EmptyRepositoryError. Use .get_changesets() instead of using
c.db_repo_scm_instance directly.

caching: invalidate Repository cache of README and RSS based on latest revision hash in its .changeset_cache

Avoid using the the more heavy and complex CacheInvalidation.

Note that raw_id only is passed to the getter function as cache key, in order
to make sure new data is retrieved whenever the repo changes.

summary: simplify sorting of language stats - avoid comparing dicts

Make the ordering well defined as primarily decreasing count, secondarily
extension - thus some slight changes in test output.

ssh: show ssh URL on summary page

Original change has been heavily modified by Mads Kiilerich.

clone_url: simplify the logic - move summary handling of different URLs with/without id to db

Drop the half-baked concept of a separate "clone_uri_by_id" - just always use a
single template and change back and forth between {repo} and _{repoid} as
necessary.

Make it clear that clone_uri_tmpl always is passed as argument.

clone_url: clarify variable naming

Minor nits:
user -> username (temporarily more complex)
uri_tmpl -> clone_uri_tmpl
qualified_home_url -> prefix_url
decoded_path -> prefix

summary: fix odd code for allowing URI clone template to either be specified with {repo} or _{repoid} (Issue #336)

Clarify that if the configured URI uses {repo}, then clone_uri_by_id must replace that with _{repoid}.

Also, if the URI uses _{repoid}, it is _def_clone_uri that must be computed by replacing it with {node}.

auth: restore anonymous repository access

Dominik Ruf found that aa25ef34ebab introduced a regression in anonymous access
to repositories ... if that is enabled.

The refactoring was too strict when it missed that not all repo permission
checks require a logged in user. Read access can be granted to the default user
... but not write or admin.

Instead of the commands used in aa25ef34ebab, the following commands are used
to consistently also allow the default user in all decorators where we only need
repo read access:

# Introduce explicit allow_default_user=True - that was the default before aa25ef34ebab
sed -i 's/@LoginRequired()/@LoginRequired(allow_default_user=True)/g' `hg mani`
sed -i 's/@LoginRequired(\(..*\))/@LoginRequired(\1, allow_default_user=True)/g' `hg mani`
# The primary case: Replace @NotAnonymous with removal of allow_default_user=True
perl -0pi -e 's/\@LoginRequired\((?:(.*), )?allow_default_user=True\)\n\s*\@NotAnonymous\(\)/\@LoginRequired(\1)/g' `hg mani`
# If there is a global permission check, no anonymous is ever allowed
perl -0pi -e 's/\@LoginRequired\(allow_default_user=True\)(\n\s*\@HasPermission)/\@LoginRequired()\1/g' `hg mani`
# Repo access for write or admin also assume no default user
perl -0pi -e 's/\@LoginRequired\(allow_default_user=True\)(\n\s*\@HasRepoPermissionLevelDecorator\('"'(write|admin)'"'\))/\@LoginRequired()\1/g' `hg mani`

auth: refactor to introduce @LoginRequired(allow_default_user=True) and deprecate @NotAnonymous()

It was error prone that @LoginRequired defaulted to allow anonymous users (if
'default' user is enabled). See also 245b4e3abf39.

Refactor code to make it more explicit and safe by default: Deprecate
@NotAnonymous by making it the default of @LoginRequired. That will make it
safe by default.

To preserve same functionality, set allow_default_user=True in all the cases
where @LoginRequired was *not* followed by @NotAnonymous or other permission
checks - that was done with some script hacks:
sed -i 's/@LoginRequired(\(..*\))/@LoginRequired(\1, allow_default_user=True)/g' `hg mani`
sed -i 's/@LoginRequired()/@LoginRequired(allow_default_user=True)/g' `hg mani`
perl -0pi -e 's/\@LoginRequired\(allow_default_user=True\)\n\s*\@NotAnonymous\(\)/\@LoginRequired()/g' `hg mani`
perl -0pi -e 's/\@LoginRequired\(allow_default_user=True\)(\n\s*\@Has(Repo)?Permission)/\@LoginRequired()\1/g' `hg mani`

It has been reviewed that all uses of allow_default_user=True are in places
where the there indeed wasn't any checking for default user before. These may
or may not be correct, but now they are explicit and can be spotted and fixed.

The few remaining uses of @NotAnonymous should probably be removed somehow.

summary: inline _load_changelog_summary

It makes more sense to have this summary specific code in the summary
controller.

controllers: remove empty __before__ methods

__before__ methods that only call the super __before__ method are redundant
and can be removed. The super's method will be called directly.

tg: minimize future diff by some mocking and replacing some pylons imports with tg

No actual tg dependency yet, just a temporary hack faking tg as an alias for
pylons.

Based on work by Alessandro Molina.

templates: properly escape inline JavaScript values

TLDR: Kallithea has issues with escaping values for use in inline JS.
Despite judicious poking of the code, no actual security vulnerabilities
have been found, just lots of corner-case bugs. This patch fixes those,
and hardens the code against actual security issues.

The long version:

To embed a Python value (typically a 'unicode' plain-text value) in a
larger file, it must be escaped in a context specific manner. Example:

>>> s = u'<script>alert("It\'s a trap!");</script>'

1) Escaped for insertion into HTML element context

>>> print cgi.escape(s)
&lt;script&gt;alert("It's a trap!");&lt;/script&gt;

2) Escaped for insertion into HTML element or attribute context

>>> print h.escape(s)
&lt;script&gt;alert(&#34;It&#39;s a trap!&#34;);&lt;/script&gt;

This is the default Mako escaping, as usually used by Kallithea.

3) Encoded as JSON

>>> print json.dumps(s)
"<script>alert(\"It's a trap!\");</script>"

4) Escaped for insertion into a JavaScript file

>>> print '(' + json.dumps(s) + ')'
("<script>alert(\"It's a trap!\");</script>")

The parentheses are not actually required for strings, but may be needed
to avoid syntax errors if the value is a number or dict (object).

5) Escaped for insertion into a HTML inline <script> element

>>> print h.js(s)
("\x3cscript\x3ealert(\"It's a trap!\");\x3c/script\x3e")

Here, we need to combine JS and HTML escaping, further complicated by
the fact that "<script>" tag contents can either be parsed in XHTML mode
(in which case '<', '>' and '&' must additionally be XML escaped) or
HTML mode (in which case '</script>' must be escaped, but not using HTML
escaping, which is not available in HTML "<script>" tags). Therefore,
the XML special characters (which can only occur in string literals) are
escaped using JavaScript string literal escape sequences.

(This, incidentally, is why modern web security best practices ban all
use of inline JavaScript...)

Unsurprisingly, Kallithea does not do (5) correctly. In most cases,
Kallithea might slap a pair of single quotes around the HTML escaped
Python value. A typical benign example:

$('#child_link').html('${_('No revisions')}');

This works in English, but if a localized version of the string contains
an apostrophe, the result will be broken JavaScript. In the more severe
cases, where the text is user controllable, it leaves the door open to
injections. In this example, the script inserts the string as HTML, so
Mako's implicit HTML escaping makes sense; but in many other cases, HTML
escaping is actually an error, because the value is not used by the
script in an HTML context.

The good news is that the HTML escaping thwarts attempts at XSS, since
it's impossible to inject syntactically valid JavaScript of any useful
complexity. It does allow JavaScript errors and gibberish to appear on
the page, though.

In these cases, the escaping has been fixed to use either the new 'h.js'
helper, which does JavaScript escaping (but not HTML escaping), OR the
new 'h.jshtml' helper (which does both), in those cases where it was
unclear if the value might be used (by the script) in an HTML context.
Some of these can probably be "relaxed" from h.jshtml to h.js later, but
for now, using h.jshtml fixes escaping and doesn't introduce new errors.

In a few places, Kallithea JSON encodes values in the controller, then
inserts the JSON (without any further escaping) into <script> tags. This
is also wrong, and carries actual risk of XSS vulnerabilities. However,
in all cases, security vulnerabilities were narrowly avoided due to other
filtering in Kallithea. (E.g. many special characters are banned from
appearing in usernames.) In these cases, the escaping has been fixed
and moved to the template, making it immediately visible that proper
escaping has been performed.

Mini-FAQ (frequently anticipated questions):

Q: Why do everything in one big, hard to review patch?
Q: Why add escaping in specific case FOO, it doesn't seem needed?

Because the goal here is to have "escape everywhere" as the default
policy, rather than identifying individual bugs and fixing them one
by one by adding escaping where needed. As such, this patch surely
introduces a lot of needless escaping. This is no different from
how Mako/Pylons HTML escape everything by default, even when not
needed: it's errs on the side of needless work, to prevent erring
on the side of skipping required (and security critical) work.

As for reviewability, the most important thing to notice is not where
escaping has been introduced, but any places where it might have been
missed (or where h.jshtml is needed, but h.js is used).

Q: The added escaping is kinda verbose/ugly.

That is not a question, but yes, I agree. Hopefully it'll encourage us
to move away from inline JavaScript altogether. That's a significantly
larger job, though; with luck this patch will keep us safe and secure
until such a time as we can implement the real fix.

Q: Why not use Mako filter syntax ("${val|h.js}")?

Because of long-standing Mako bug #140, preventing use of 'h' in
filters.

Q: Why not work around bug #140, or even use straight "${val|js}"?

Because Mako still applies the default h.escape filter before the
explicitly specified filters.

Q: Where do we go from here?

Longer term, we should stop doing variable expansions in script blocks,
and instead pass data to JS via e.g. data attributes, or asynchronously
using AJAX calls. Once we've done that, we can remove inline JavaScript
altogether in favor of separate script files, and set a strict Content
Security Policy explicitly blocking inline scripting, and thus also the
most common kind of cross-site scripting attack.

model: remove BaseModel class

The only remaining purpose of this class was to provide the "sa" field,
allowing a custom SQLAlchemy session to be used for model operations.
However, this never actually worked, nor was it used anywhere.

There's always a global SQLAlchemy session associated with the current
thread; using another session for a single function call does not make
any sense (as sessions cannot be mixed), unless the code works carefully
to ensure the two sessions (and all objects loaded from them) are kept
completely separate. Suffice to say that Kallithea does no such thing,
thus there's no need to pretend to support multiple concurrent sessions.

auth: simplify repository permission checks

In practice, Kallithea has the 'repository.admin' permission imply the
'repository.write' permission, which again implies 'repository.read'.

This codifies/enforces this practice by replacing HasRepoPermissionAny
"perm function" with the new HasRepositoryLevel function, reducing the
risk of errors and saving quite a lot of typing.

controllers: avoid setting request state in controller instances - set it in the thread global request variable

In TurboGears, controllers are singletons and we should avoid using instance
variables for any volatile data. Instead, use the "global thread local" request
context.

With everything in request, some use of c is dropped.

Note: kallithea/controllers/api/__init__.py still use instance variables that
will cause problems with TurboGears.

lib: move jsonify from utils to base

Suggested by Mads Kiilerich.

The jsonify method is the only thing in utils that directly uses pylons.
Move it to base where it fits better and we can use existing global imports.

compat: drop unnecessary wrappers for old Python versions

Made unnecessary by other cleanup or upgrade to Python 2.6+.

celeryd: annotate tasks so they can be run directly without run_task

This also makes the system less forgiving about celery configuration problems
and thus easier to debug. I like that.

db: name the scm_instance_cached cache entries - reduce the risk of collisions

- and same with other cache regions

auth: introduce AuthUser.is_default_user attribute

This makes makes a number of checks more readable.

The username of the default user is presently hardcoded to "default" (in
db.User.DEFAULT_USER); this is currently what defines the default user,
and this commit doesn't change that. (Even if the check that defines
is_default_user is a comparison between user IDs and not usernames,
the anonymous_user object used in the comparison is loaded by looking
up the user named "default".)

stats: fix display when no data ready yet

* jQ uses .append(), not .appendChild()
* when no data ready, display a nice message
* rephrase the message, the data isn't ready, not hasn't loaded yet

cleanup: avoid some 'except Exception' catching - catch specific exceptions or log it and show what happened

This has a risk of introducing regressions ... but we want to get rid of all
exception muting and make the whole system less fragile and easier to debug.

Correct licensing information in individual files.

The top-level license file is now LICENSE.md.

Also, in various places where there should have been joint copyright holders
listed, a single copyright holder was listed. It does not appear easy to add
a link to a large list of copyright holders in these places, so it simply
refers to the fact that various authors hold copyright.

In future, if an easy method is discovered to link to a list from those
places, we should do so.

Finally, text is added to LICENSE.md to point to where the full list of
copyright holders is, and that Kallithea as a whole is GPLv3'd.

Second step in two-part process to rename directories.
This is the actual directory rename.