controllers: move controllers base class from lib/base to controllers

TG quickstart put it in lib/base.py , but it fits better on the controllers
layer as a base there.

The contributing docs were a bit ahead of time ... but with a typo.

lib: move webhelpers2 and friends to webutils

Gives less of the unfortunate use of helpers - especially in low level libs.

lib: consistently use webutils.url instead of h.url

Templates are still the valid use case for accessing through h.

Simplify mocking of url() function.

diffs: remove unused argument enable_comments and class no-comment

enable_comments was only used to set/not-set the 'no-comment' CSS class.
While this class was emitted, no CSS rule nor any JavaScript logic was
actually using it. Last real usage of that class was removed with commit
e87baa8f1c5bd2488aefc23b95c0db3a04bc8431.

Cleanup the code by not emitting 'no-comment' and remove the
'enable_comments' flag.

diff: drop per file ignore-whitespace and context - it didn't work and had conceptual issue (Issue #344)

Diffs are currently generated at the low level as one big diff between two vcs
resisions, provided global values for diff context size and flag for ignoring
whitespace. All files use the same flags. There is no way to actually compute
the full diff using these use per file flags, and no simple and efficient way
to add it.

The best option is thus to drop the failed attempt at making it per file, and
just rely on the simple global flags in the URL.

The links for changing whitespace and context is sometimes shown for the whole
"page", and sometimes next to the diff for one file. For now, keep showing the
link in these places, but make sure it navigates back to the FID of the section
where the link was clicked.

The implementation is completely rewritten and moved to a more appropriate
location in helpers.

With a more clean implementation, we also consistently use the simple getters
to extract values from the URL.

diff: fix ignorews/context link to use the right target as anchor

The value in url_fid might not be a valid anchor.

For changesets, url_fid would be like 'C--9c390eb52cd6' even though the actual
target included the changeset hash and were like 'C-1536d03b4869-9c390eb52cd6'.

For pullrequests and compare, it wouldn't link to anything at all, even though
there was a target like 'C--56535da5df40'.

Instead, pass id_fid as anchor value as a separate argument. That one is a
valid anchor.

py3: remove safe_unicode in places where it no longer is needed because all strings (except bytes) already *are* unicode strings

(The remaining safe_unicode calls are still needed and can't just be removed,
generally because we in these cases still have to convert from bytes to unicode
strings.)

py3: use comprehensions and generators instead of filters - it is more explicit, and sometimes shorter

From 2to3 -f filter.

changeset: store hexified source hash in context so it is readily available in template

Avoid hex as "encoding" - it is not supported in py3.

Mercurial store source and transplant_source differently ... but that will now
be handled in the controller instead of in the template.

pull-request: fix missing imports for delete and close

Problem introduced in 21edd7f16681 - apparently a code path without automated
test coverage.

controllers: remove pr_comment flag in delete_cs_pr_comment

A separate comment is not really needed as we can check co.pull_request.

Suggested by Mads Killerich.

controllers: inline changeset.create_comment

As changeset.create_comment is now only used from
changeset.create_cs_pr_comment (used from pullrequests controller as well),
there is no need to have it as a separate method.

controllers: forward pullrequests.comment to changeset

Similar to the way delete_comment is handled.

controllers: align pullrequests.comment with changeset.comment

This commit purely serves to highlight the differences.
The subsequent commit will remove the duplication.

controllers: changeset: always allow status changes

Don't disallow status changes on changesets that are part of a pull request
and on which the last status change happened via the pull request.

This odd restriction was already previously highlighted by Mads Kiilerich as
'RLY?' in commit 7834f845505aec3086f525600c81209a40b495ef, so it seems fair
to remove it.

When dont_allow_on_closed_pull_request no longer is set,
StatusChangeOnClosedPullRequestError will no longer be raised.

controllers: changeset: catch status changes on inline comments

This is a preparation on sharing the same comment method between
pullrequests and changeset controllers.

controllers: forward pullrequests.delete_comment to changeset

Remove duplication between pullrequests and changeset.
We move the code outside ChangesetController to make it callable from
PullrequestsController.

Note:
- instead of keeping the method pullrequests.delete_comment itself and
letting it forward to changeset.delete_comment, an alternative solution
would have been to change the routing table directly. However, the chosen
solution makes it more explicit which operations are supported on each
controller.

controllers: align pullrequests.delete_comment with changeset.delete_comment

This commit purely serves to highlight the differences.
The subsequent commit will remove the duplication.

auth: restore anonymous repository access

Dominik Ruf found that aa25ef34ebab introduced a regression in anonymous access
to repositories ... if that is enabled.

The refactoring was too strict when it missed that not all repo permission
checks require a logged in user. Read access can be granted to the default user
... but not write or admin.

Instead of the commands used in aa25ef34ebab, the following commands are used
to consistently also allow the default user in all decorators where we only need
repo read access:

# Introduce explicit allow_default_user=True - that was the default before aa25ef34ebab
sed -i 's/@LoginRequired()/@LoginRequired(allow_default_user=True)/g' `hg mani`
sed -i 's/@LoginRequired(\(..*\))/@LoginRequired(\1, allow_default_user=True)/g' `hg mani`
# The primary case: Replace @NotAnonymous with removal of allow_default_user=True
perl -0pi -e 's/\@LoginRequired\((?:(.*), )?allow_default_user=True\)\n\s*\@NotAnonymous\(\)/\@LoginRequired(\1)/g' `hg mani`
# If there is a global permission check, no anonymous is ever allowed
perl -0pi -e 's/\@LoginRequired\(allow_default_user=True\)(\n\s*\@HasPermission)/\@LoginRequired()\1/g' `hg mani`
# Repo access for write or admin also assume no default user
perl -0pi -e 's/\@LoginRequired\(allow_default_user=True\)(\n\s*\@HasRepoPermissionLevelDecorator\('"'(write|admin)'"'\))/\@LoginRequired()\1/g' `hg mani`

auth: refactor to introduce @LoginRequired(allow_default_user=True) and deprecate @NotAnonymous()

It was error prone that @LoginRequired defaulted to allow anonymous users (if
'default' user is enabled). See also 245b4e3abf39.

Refactor code to make it more explicit and safe by default: Deprecate
@NotAnonymous by making it the default of @LoginRequired. That will make it
safe by default.

To preserve same functionality, set allow_default_user=True in all the cases
where @LoginRequired was *not* followed by @NotAnonymous or other permission
checks - that was done with some script hacks:
sed -i 's/@LoginRequired(\(..*\))/@LoginRequired(\1, allow_default_user=True)/g' `hg mani`
sed -i 's/@LoginRequired()/@LoginRequired(allow_default_user=True)/g' `hg mani`
perl -0pi -e 's/\@LoginRequired\(allow_default_user=True\)\n\s*\@NotAnonymous\(\)/\@LoginRequired()/g' `hg mani`
perl -0pi -e 's/\@LoginRequired\(allow_default_user=True\)(\n\s*\@Has(Repo)?Permission)/\@LoginRequired()\1/g' `hg mani`

It has been reviewed that all uses of allow_default_user=True are in places
where the there indeed wasn't any checking for default user before. These may
or may not be correct, but now they are explicit and can be spotted and fixed.

The few remaining uses of @NotAnonymous should probably be removed somehow.

users: remove code that is unused after most autocomplete has been switched to ajax

@mention support still require _USERS_AC_DATA as a global variable.

diffs: wrap vcs repo get_diff

Refactor to get a single place for diff error handling outside the vcs lib.

diffs: move as_html and _safe_id from method to a pure function - avoid calling the method as function

as_html was sometimes used in a way where we actually don't want to use the
whole DiffProcessor - we just created a dummy instance and passed custom input
as parameter to the instance method.

Instead, make it the function we apparently want.

Make it clear that as_html not just returns a "diff" but that it is a html diff.

diffs: drop the DiffLimitExceeded container - just make it a flag available as property

Keep it simple.

diffs: inline prepare() into __init__ and make the result available as .parsed

Make it more clear what the DiffProcessor is: Something that works on a raw
diff as input, mainly compute when initialized, and returns an object where the
result is available in different ways.

diffs: cleanup of variable naming around cut_off_limit

A brief summary of this area:

The base controller sets self.cut_off_limit from config and is used for diffs,
unless controllers are given a fulldiff query parameter. In a few cases, these
are passed to templates as c.cut_off_limit or c.fulldiff . Also, if the diff
function returns a LimitedDiffContainer, c.limited_diff is set so the UI can
report the data set is partial.

codingstyle: trivial whitespace fixes

Reported by flake8.

controllers: rename __before__ to _before in preparation of TurboGears2

__before__ in Pylons is called _before in TurboGears2. We can prepare this
rename already in Pylons-based Kallithea, so that the real TG2 migration
commit just changes the BaseController.

Since TurboGears2 _before can pass extra arguments, we add *args and
**kwargs parameters as well.

tg: minimize future diff by some mocking and replacing some pylons imports with tg

No actual tg dependency yet, just a temporary hack faking tg as an alias for
pylons.

Based on work by Alessandro Molina.

templates: properly escape inline JavaScript values

TLDR: Kallithea has issues with escaping values for use in inline JS.
Despite judicious poking of the code, no actual security vulnerabilities
have been found, just lots of corner-case bugs. This patch fixes those,
and hardens the code against actual security issues.

The long version:

To embed a Python value (typically a 'unicode' plain-text value) in a
larger file, it must be escaped in a context specific manner. Example:

>>> s = u'<script>alert("It\'s a trap!");</script>'

1) Escaped for insertion into HTML element context

>>> print cgi.escape(s)
&lt;script&gt;alert("It's a trap!");&lt;/script&gt;

2) Escaped for insertion into HTML element or attribute context

>>> print h.escape(s)
&lt;script&gt;alert(&#34;It&#39;s a trap!&#34;);&lt;/script&gt;

This is the default Mako escaping, as usually used by Kallithea.

3) Encoded as JSON

>>> print json.dumps(s)
"<script>alert(\"It's a trap!\");</script>"

4) Escaped for insertion into a JavaScript file

>>> print '(' + json.dumps(s) + ')'
("<script>alert(\"It's a trap!\");</script>")

The parentheses are not actually required for strings, but may be needed
to avoid syntax errors if the value is a number or dict (object).

5) Escaped for insertion into a HTML inline <script> element

>>> print h.js(s)
("\x3cscript\x3ealert(\"It's a trap!\");\x3c/script\x3e")

Here, we need to combine JS and HTML escaping, further complicated by
the fact that "<script>" tag contents can either be parsed in XHTML mode
(in which case '<', '>' and '&' must additionally be XML escaped) or
HTML mode (in which case '</script>' must be escaped, but not using HTML
escaping, which is not available in HTML "<script>" tags). Therefore,
the XML special characters (which can only occur in string literals) are
escaped using JavaScript string literal escape sequences.

(This, incidentally, is why modern web security best practices ban all
use of inline JavaScript...)

Unsurprisingly, Kallithea does not do (5) correctly. In most cases,
Kallithea might slap a pair of single quotes around the HTML escaped
Python value. A typical benign example:

$('#child_link').html('${_('No revisions')}');

This works in English, but if a localized version of the string contains
an apostrophe, the result will be broken JavaScript. In the more severe
cases, where the text is user controllable, it leaves the door open to
injections. In this example, the script inserts the string as HTML, so
Mako's implicit HTML escaping makes sense; but in many other cases, HTML
escaping is actually an error, because the value is not used by the
script in an HTML context.

The good news is that the HTML escaping thwarts attempts at XSS, since
it's impossible to inject syntactically valid JavaScript of any useful
complexity. It does allow JavaScript errors and gibberish to appear on
the page, though.

In these cases, the escaping has been fixed to use either the new 'h.js'
helper, which does JavaScript escaping (but not HTML escaping), OR the
new 'h.jshtml' helper (which does both), in those cases where it was
unclear if the value might be used (by the script) in an HTML context.
Some of these can probably be "relaxed" from h.jshtml to h.js later, but
for now, using h.jshtml fixes escaping and doesn't introduce new errors.

In a few places, Kallithea JSON encodes values in the controller, then
inserts the JSON (without any further escaping) into <script> tags. This
is also wrong, and carries actual risk of XSS vulnerabilities. However,
in all cases, security vulnerabilities were narrowly avoided due to other
filtering in Kallithea. (E.g. many special characters are banned from
appearing in usernames.) In these cases, the escaping has been fixed
and moved to the template, making it immediately visible that proper
escaping has been performed.

Mini-FAQ (frequently anticipated questions):

Q: Why do everything in one big, hard to review patch?
Q: Why add escaping in specific case FOO, it doesn't seem needed?

Because the goal here is to have "escape everywhere" as the default
policy, rather than identifying individual bugs and fixing them one
by one by adding escaping where needed. As such, this patch surely
introduces a lot of needless escaping. This is no different from
how Mako/Pylons HTML escape everything by default, even when not
needed: it's errs on the side of needless work, to prevent erring
on the side of skipping required (and security critical) work.

As for reviewability, the most important thing to notice is not where
escaping has been introduced, but any places where it might have been
missed (or where h.jshtml is needed, but h.js is used).

Q: The added escaping is kinda verbose/ugly.

That is not a question, but yes, I agree. Hopefully it'll encourage us
to move away from inline JavaScript altogether. That's a significantly
larger job, though; with luck this patch will keep us safe and secure
until such a time as we can implement the real fix.

Q: Why not use Mako filter syntax ("${val|h.js}")?

Because of long-standing Mako bug #140, preventing use of 'h' in
filters.

Q: Why not work around bug #140, or even use straight "${val|js}"?

Because Mako still applies the default h.escape filter before the
explicitly specified filters.

Q: Where do we go from here?

Longer term, we should stop doing variable expansions in script blocks,
and instead pass data to JS via e.g. data attributes, or asynchronously
using AJAX calls. Once we've done that, we can remove inline JavaScript
altogether in favor of separate script files, and set a strict Content
Security Policy explicitly blocking inline scripting, and thus also the
most common kind of cross-site scripting attack.

cleanup: remove SQLAlchemy session argument to action_logger

There's always a global SQLAlchemy session associated with the current
thread; using another session for a single function call does not make
any sense (as sessions cannot be mixed), unless the code works carefully
to ensure the two sessions (and all objects loaded from them) are kept
completely separate. Suffice to say that Kallithea does no such thing,
thus there's no need to pretend to support multiple concurrent sessions.

auth: simplify repository permission checks

In practice, Kallithea has the 'repository.admin' permission imply the
'repository.write' permission, which again implies 'repository.read'.

This codifies/enforces this practice by replacing HasRepoPermissionAny
"perm function" with the new HasRepositoryLevel function, reducing the
risk of errors and saving quite a lot of typing.

controllers: avoid setting request state in controller instances - set it in the thread global request variable

In TurboGears, controllers are singletons and we should avoid using instance
variables for any volatile data. Instead, use the "global thread local" request
context.

With everything in request, some use of c is dropped.

Note: kallithea/controllers/api/__init__.py still use instance variables that
will cause problems with TurboGears.

lib: move jsonify from utils to base

Suggested by Mads Kiilerich.

The jsonify method is the only thing in utils that directly uses pylons.
Move it to base where it fits better and we can use existing global imports.

template: use Bootstrap tooltips and popover instead of handmade tooltips

Based on work from Andrew Shadura <andrew@shadura.me>.

Further modified by Mads Kiilerich.

show_changeset_tooltip is merged into tooltip_activate.

db: rename ChangesetStatus.changeset_comment_id to comment_id

This ensures consistency with the existing "comment" relationship, as
well as the foreign key (ChangesetComment.comment_id).

cleanup: use obj.foo_id instead of obj.foo.foo_id

Don't use constructs like obj.user.user_id when obj.user_id works
equally well (and potentially saves a database load).

model: change ChangesetComment 'user' to 'author'

Rename the 'user_id' field to 'author_id' and replace other references
to the comment 'user' throughout the model. The database column name
'user_id' remain unchanged for now; a later Alembic script can fix the
name of these and other columns to match their Python name.

Turbogears2 migration: remove some references to Pylons in comments

In order to minimize the diff of the actual Turbogears2 migration, this
commit already removes certain unnecessary references to Pylons from the
Kallithea source base.
Places where the reference to Pylons is important are still kept for now, as
well as references in kallithea/config where many changes are made for
Turbogears2 anyway.

diff: correct handling of links to old filename in renames

There were links to the file at the parent revision ... but if the file had
been renamed, it used the wrong name.

diff: rework data structure used by diff_block.diff_block (used for changeset diffs)

Make it a pure reusable function that also can replace diff_block_simple.

Improve code clarity by using Python tuples for tuples.

fid and url_fid are just shuffled around - not cleaned up for now.

diff: minor cleanups

More consistency and preparing for later changes.

auth: properly invoke PermFunctions (CVE-2016-3114)

This fixes a vulnerability that allowed logged-in users to edit or
delete open pull requests associated with any repository to which
they had read access, plus a related vulnerability allowing logged-in
users to delete any comment from any repository, provided they could
determine the comment ID and had read access to just one repository.

comments: extract common comment creation functionality for changesets and pullrequests

So far not big but it will grow later ...

comments: change comment formatting to plain text instead of rst

There might be value in enabling rich markup (especially if it is markdown
instead of rst) ... or it might be a waste of time. We might revisit that
later.

But either way: Changing to plain text makes it more feasible to do markup of
hashes and issues - for now that is more important than rtf.

cleanup: fix 'try ... except ...' formatting missed in d69aa464f373

Examples spotted by Jiří Suchan <yed@vanyli.net>.

cleanup: replace redirect with WebOb exceptions

All redirect does is to log "Generating 302 redirect" with logging the actual
location and raise a WebOb HTTPFound exception, and the logging is redundant,
as WebOb exceptions and their status codes are already logged.

Instead, just raise the exception directly, which is both explicit and simpler
(and finally, gets rid of "return redirect" which never really returns).

cleanup: consistently use 'except ... as ...:'

Use the Python 2.6+ syntax instead of the old confusing 'except ..., ...'
syntax.

cleanup: check for None object identity in cases where that is what the 'contract' says

Avoid applying bool() on complex objects - it might return something unexpected
such as the key (which might be 0 and thus be false). Checking for None is more
safe and faster.

changeset: make code more stable against unexpected comments

Avoid passing None to the template in comments. This makes the code more
correct and fixes a crash seen while hacking, probably never seen in the wild.

changeset: reduce log level of stack trace on innocent exceptions

When the user performs an unallowed action and a flash is displayed, there
is no need to log the stack trace at 'error' level. Reduce the stack trace
log to debug instead.

changeset/pullrequest controller: refactor some old variable naming

c.co is a bit cryptic, use the full name c.comment
comm sounds like 'communication', use the full name, comment

tested by adding a single comment on a changeset/pullreques
and running:
* the kallithea/tests/functional/test_changeset_comments.py test

spelling: fix title casing on various translated strings

Primarily captions on other not-just-text and data.

comments: avoid storing 'No comments' text when changing status

When a general comment (with or without status change) is added to a
changeset or pull request, and no text was added, Kallithea automatically
used 'No comments' as text. The stub text is added to the database as if it has
been entered by the user and it can thus not easily be identified as an
automatic comment.

This commit makes following changes:
- allow adding an empty comment to the database when there is a status
change. An empty comment without status change is ignored.
- do not add a stub text to the database, but generate it on demand
- the stub text is shown in italic font to differentiate it from user-entered
text

Currently there is a large amount of duplication between
controllers/changeset.py and controllers/pullrequests.py, which is to be
cleaned up in a later commit.

cleanup: avoid some 'except Exception' catching - catch specific exceptions or log it and show what happened

This has a risk of introducing regressions ... but we want to get rid of all
exception muting and make the whole system less fragile and easier to debug.

pull requests: rework handling of comments and texts when voting and closing

Close did not leave any records - now it does.

Status changes without comments repeated the status change in text - now it
doesn't repeat itself.

compare: introduce .cs_repo as the repo for .cs_changes - sometimes it is org, sometimes other

Influences pull request creation, display and compare.

Correct licensing information in individual files.

The top-level license file is now LICENSE.md.

Also, in various places where there should have been joint copyright holders
listed, a single copyright holder was listed. It does not appear easy to add
a link to a large list of copyright holders in these places, so it simply
refers to the fact that various authors hold copyright.

In future, if an easy method is discovered to link to a list from those
places, we should do so.

Finally, text is added to LICENSE.md to point to where the full list of
copyright holders is, and that Kallithea as a whole is GPLv3'd.