controllers: move controllers base class from lib/base to controllers

TG quickstart put it in lib/base.py , but it fits better on the controllers
layer as a base there.

The contributing docs were a bit ahead of time ... but with a typo.

imports: try to use global imports unless it is a layering violation

To minimize the impact, do imports that violate the layering at runtime instead
of at import time.

lib: move webhelpers2 and friends to webutils

Gives less of the unfortunate use of helpers - especially in low level libs.

lib: fix bad references to utils3

A problem introduced in when rebasing 5e46f73f0d1c after renaming the temporary
utils3 name to webutils.

routing: separate url handling from routing - move it to webutils

This is a helper method relying on the thread local tg.request. We didn't have
a good place to put it. Now we do.

This (re)moves unfortunate dependencies to the routing module (which almost is
a controller).

auth: refactor permissions

Avoid using complex vague typing in dict-of-dicts.

user: make get_by_username_or_email default to treat username case insensitive

The get_by_username_or_email is a flexible function, intended to find users in
multiple ways, suitable for login prompts. The function was sometimes used
with case sensitive user lookup, sometimes without. Instead, be consistent and
just default to be insensitive.

auth: show a clear "Authentication failed" message if login fails after passing form validation

log_in_user will only set a session cookie after verifying that the user is
valid (for example based on IP). The code is thus safe, but no hint were given
to the user if login failed for that reason.

login: assert that the validated user actually is found

Due to another bug, it was possible that authentication succeeded but the user
object couldn't be obtained. This was for example noticed when the LDAP auth
module did not correctly parse the email attribute, and a login via email
was attempted. In this case, the user was retrieved from email address and LDAP
found the user, but the email attribute in the Kallithea database was then
changed incorrectly and a subsequent retrieval based on the same original email
address would not find the user.

Such problem would lead to an assert in Kallithea:

File ".../kallithea/controllers/login.py", line 104, in index
auth_user = log_in_user(user, c.form_result['remember'], is_external_auth=False, ip_addr=request.ip_addr)
File ".../kallithea/lib/base.py", line 122, in log_in_user
assert not user.is_default_user, user
AttributeError: 'NoneType' object has no attribute 'is_default_user'

This assert cought the problem but is not a spot-on indicator of the real
problem. Instead, we can catch this problem sooner by adding an assert already
in the login controller.

login: fix incorrect CSRF rejection of "Reset Your Password" form (Issue #350)

htmlfill would remove the CSRF token from the form when substituting the query
parameters, causing password reset to break.

By default, htmlfill will clear all input fields that doesn't have a new
"default" value provided. It could be fixed by setting force_defaults to False
- see http://www.formencode.org/en/1.2-branch/modules/htmlfill.html . It could
also be fixed by providing the CSRF token in the defaults to be substituted in
the form.

Instead, refactor password_reset_confirmation to have more explicitly safe
handling of query parameters. Replace htmlfill with the usual template
variables.

The URLs are generated in kallithea/model/user.py send_reset_password_email()
and should only contain email, timestamp (integer as digit string) and a hex
token from get_reset_password_token() .

auth: use other and better checks than is_authenticated

These are the two only uses of is_authenticated, and we are fine without it.

auth: don't ignore login POSTs if already logged in

This is probably only rarely a big deal, but ignoring an explicit command seems
wrong.

This makes it possible to easily re-authenticate - we will need this ...

auth: drop confusing and layering-violating User.AuthUser property

Keep it simple and just explicitly create an AuthUser if "needed". That can
perhaps be simplified further later on.

recaptcha: Update to Google recaptcha API v2 (Issue #313)

Recaptcha stopped working. It was using google recaptcha v1, and
https://developers.google.com/recaptcha/docs/faq says: "Any calls to the v1 API
will not work after March 31, 2018."

Note:

1) Not using 'async defer' - it's not used anywhere else, plus I have no idea
of what exactly it does.

2) The recaptcha div has an ID so the label for="recaptcha_field" can reference
it (although I'm not certain a div can technically have a label, but it's
better than removing the label, or the for="").

controllers: remove empty __before__ methods

__before__ methods that only call the super __before__ method are redundant
and can be removed. The super's method will be called directly.

tg: minimize future diff by some mocking and replacing some pylons imports with tg

No actual tg dependency yet, just a temporary hack faking tg as an alias for
pylons.

Based on work by Alessandro Molina.

controllers: avoid setting request state in controller instances - set it in the thread global request variable

In TurboGears, controllers are singletons and we should avoid using instance
variables for any volatile data. Instead, use the "global thread local" request
context.

With everything in request, some use of c is dropped.

Note: kallithea/controllers/api/__init__.py still use instance variables that
will cause problems with TurboGears.

forms: wrap LoginForm inside function like other forms

All forms except LoginForm are wrapped inside a function. The original
purpose of this wrapping seems to be the ability to pass parameters to tweak
the form.

But, this also has another desired effect: translation of strings
wrapped with _ is no longer attempted when reading the class definition, but
only when the function is instantiated. In the former case, it is not
guaranteed that a translator is actually available because we are not
running in standard application context.

Align LoginForm with the others.

Turbogears2 migration: replace pylons.url by kallithea.config.routing.url

In preparation for the migration to Turbogears2, introduce a
kallithea.config.routing.url to replace pylons.url. The implementation is
basically the same: wrap around routes.url().

This change involves:
- a number of import statement changes
- fixing some tests in test_libs.py; to avoid duplication, the different
implementations of fake_url were grouped in one place.

This change was first proposed by Alessandro Molina in his initial
port. Following changes were made afterwards:
- move UrlGenerator from kallithea.lib.utils to kallithea.config.routing
- add documentation to UrlGenerator
- kallithea/lib/auth.py used url_for instead of url, for no apparent reason
so this was changed.
- fix libs tests
- rebase onto Pylons-based Kallithea first

auth: allow web login with email addresses

Let users log in using their email addresses instead of their user names.
This only applies to the web login, not git+http or hg+http protocols.

auth: note that we never emit authuser "cookies" for the default user

The only place where we set "authuser" in the session is in log_in_user,
which is called only by the internal auth system and by auth plugins.
The internal auth system cannot log a user in as the default user,
because the default user doesn't have a password (and cannot have a
password assigned). Auth plugins cannot log a user in as the default
user, because the user doesn't have the right extern_type. As such, it's
a bug if log_in_user is ever called with the default user (which this
commit documents with an assert).

This realization makes the is_authenticated field of the authuser cookie
redundant, as it's always True. It also emphasizes that is_default_user
and is_authenticated are mutually exclusive.

auth: introduce AuthUser.is_default_user attribute

This makes makes a number of checks more readable.

The username of the default user is presently hardcoded to "default" (in
db.User.DEFAULT_USER); this is currently what defines the default user,
and this commit doesn't change that. (Even if the check that defines
is_default_user is a comparison between user IDs and not usernames,
the anonymous_user object used in the comparison is loaded by looking
up the user named "default".)

cleanup: replace redirect with WebOb exceptions

All redirect does is to log "Generating 302 redirect" with logging the actual
location and raise a WebOb HTTPFound exception, and the logging is redundant,
as WebOb exceptions and their status codes are already logged.

Instead, just raise the exception directly, which is both explicit and simpler
(and finally, gets rid of "return redirect" which never really returns).

login: enhance came_from validation

Drop urlparse and just validate that came_from is a RFC 3986 compliant path.

This blocks an HTTP header injection vulnerability discovered by
Gjoko Krstic <gjoko@zeroscience.mk> of Zero Science Lab (CVE-2015-5285)

login: use server-relative URLs in came_from correctly

Using h.url to combine came_from with query parameters caused the
SCRIPT_NAME to incorrectly be prepended to came_from, even though
it was already present. This was not a problem if the Kallithea
instance was served directly from the server root ('/') as is common,
but broke setups where Kallithea was served from a prefix.

login: include query parameters in came_from

The login controller uses the came_from query argument to determine
the page to continue to after login.

Previously, came_from specified only the URL path (obtained using
h.url.current), and any URL query parameters were passed along as
separate (additional) URL query parameters; to obtain the final redirect
target, h.url was used to combine came_from with the request.GET.

As of this changeset, came_from specifies both the URL path and query
string (obtained using request.path_qs), which means that came_from can
be used directly as the redirect target (as always, WebOb handles the
task of expanding the server relative path to a fully qualified URL).
The mangling of request.GET can also be removed.

The login code appended arbitrary, user-supplied query parameters to
URLs by calling the Routes URLGenerator (h.url) with user-supplied
keyword arguments. This construct is unfortunate, since url only
appends _unknown_ keyword arguments as query parameters, and the
parameter names could overlap with known keyword arguments, possibly
affecting the generated URL in various ways. This changeset removes
this usage from the login code, but other instances remain.

(In practice, the damage is apparently limited to causing an Internal
Server Error when going to e.g. "/_admin/login?host=foo", since WebOb
returns Unicode strings and URLGenerator only allows byte strings for
these keyword arguments.)

login: simplify came_from validation

Even though only server-relative came_from URLs were ever generated,
the login controller allowed fully qualified URLs (URLs including
scheme and server). To avoid an open HTTP redirect (CWE-601), the code
included logic to prevent redirects to other servers. By requiring
server-relative URLs, this logic can simply be removed.

Note: SCRIPT_NAME is still not validated and it is thus possible to redirect
from one app to another on the same netloc.

login: inline _redirect_to_origin

Refactor to simplify code and make next changes simpler.

Let the controller handle came_from early and more explicit, thus simplifying
the redirect flow.

auth: secure password reset implementation

This is a better implementation of password reset function, which
doesn't involve sending a new password to the user's email address
in clear text, and at the same time is stateless.

The old implementation generated a new password and sent it
in clear text to whatever email assigned to the user currently,
so that any user, possibly unauthenticated, could request a reset
for any username or email. Apart from potential insecurity, this
made it possible for anyone to disrupt users' workflow by repeatedly
resetting their passwords.

The idea behind this implementation is to generate
an authentication token which is dependent on the user state
at the time before the password change takes place, so the token
is one-time and can't be reused, and also to bind the token to
the browser session.

The token is calculated as SHA1 hash of the following:

* user's identifier (number, not a name)
* timestamp
* hashed user's password
* session identifier
* per-application secret

We use numeric user's identifier, as it's fixed and doesn't change,
so renaming users doesn't affect the mechanism. Timestamp is added
to make it possible to limit the token's validness (currently hard
coded to 24h), and we don't want users to be able to fake that field
easily. Hashed user's password is needed to prevent using the token
again once the password has been changed. Session identifier is
an additional security measure to ensure someone else stealing the
token can't use it. Finally, per-application secret is just another
way to make it harder for an attacker to guess all values in an
attempt to generate a valid token.

When the token is generated, an anonymous user is directed to a
confirmation page where the timestamp and the usernames are already
preloaded, so the user needs to specify the token. User can either
click the link in the email if it's really them reading it, or to type
the token manually.

Using the right token in the same session as it was requested directs
the user to a password change form, where the user is supposed to
specify a new password (twice, of course). Upon completing the form
(which is POSTed) the password change happens and a notification
mail is sent.

The test is updated to test the basic functionality with a bad and
a good token, but it doesn't (yet) cover all code paths.

The original work from Andrew has been thorougly reviewed and heavily
modified by Søren Løvborg.

cleanup: consistently use 'except ... as ...:'

Use the Python 2.6+ syntax instead of the old confusing 'except ..., ...'
syntax.

BaseController: hide "Log out" link for external login sessions

If user is authorized by external means (API key or container auth),
Kallithea is not actually able to log the user out and should not show
the "Log out" link.

BaseController: enable container authentication on all pages

Previously, user had to visit the login page to log in using container
authentication; this now happens on every page load, unless user has an
existing login session.

The container authentication result is cached in session on success.

log_in_user: extract user session setup from LoginController

The next changeset will need to set up a user login session outside
LoginController. 'log_in_user' extracted and added as a top-level
function in kallithea.lib.base since the code doesn't need access to
the current controller.

Code refactored to take a User object instead of a username, to allow
callers flexibility in how the user should be looked up.

AuthUser: simplify check_ip_allowed and drop is_ip_allowed

check_ip_allowed is always called with user_id and inherit_from_default
arguments taken from the same User/AuthUser object, so just take that
object instead. This simplifies the is_ip_allowed method to the point
where it can be removed.

AuthUser: Drop ip_addr field

None of the AuthUser consumers actually need to get the IP address from
the AuthUser object, so it's just redundant.

Also, AuthUser represents a user session, and should not be used as a
generic user + IP address data structure.

LoginController: Let sessionmiddleware set session cookies

sessionmiddleware already reads cookie_out and adds Set-Cookie header,
no need for LoginController to do it.

login: preserve GET arguments throughout login redirection (issue #104)

When redirecting a user to the login page and while handling this login and
redirecting to the original page, the GET arguments passed to the original
URL are lost through the login redirection process.

For example, when creating a pull request for a specific revision from the
repository changelog, there are rev_start and rev_end arguments passed in
the URL. Through the login redirection, they are lost.

Fix the issue by passing along the GET arguments to the login page, in the
login form action, and when redirecting back to the original page.
Tests are added to cover these cases, including tests with unicode GET
arguments (in URL encoding).

spelling: fix title casing on various translated strings

Primarily captions on other not-just-text and data.

tests: provide _authentication_token when POSTing

So far not used, just preparing for the the time when the actual checking is
introduced ...

This change is very verbose. self.app.post should perhaps just at this value
automagically ...

controllers: consistently use formfill.render with force_defaults=False

The inconsistency could cause confusion for developers. It seems to me like
force_defaults=False should be the default ... and apparently it was that in
older versions of formfill.

It could perhaps make sense for us to have a wrapper that added the defualt
values once, instead of repeating it all over ;-)

Correct licensing information in individual files.

The top-level license file is now LICENSE.md.

Also, in various places where there should have been joint copyright holders
listed, a single copyright holder was listed. It does not appear easy to add
a link to a large list of copyright holders in these places, so it simply
refers to the fact that various authors hold copyright.

In future, if an easy method is discovered to link to a list from those
places, we should do so.

Finally, text is added to LICENSE.md to point to where the full list of
copyright holders is, and that Kallithea as a whole is GPLv3'd.

Second step in two-part process to rename directories.
This is the actual directory rename.