files: fix raw download of repo files with names with unicode points above 256 in name

Raw download had apparently only been tested with non-ascii characters that
were latin1. That was apparently a (too) simple case that worked without
crashing.

Files with unicode code points above 256 in their name would fail to download,
when Waitress failed like this, trying to get a real byte string by encoding
WSGI headers to latin1:
UnicodeEncodeError: 'latin-1' codec can't encode characters in position 84-85: ordinal not in range(256)

HTTP headers are of course byte strings on the network, but Python3 WSGI does
unfortunately neither expose it as bytes nor as unicode strings to be encoded
as utf-8. Instead, it uses unicode strings with byte values encoded as code
points 0-255. That is achieved by decoding the utf-8 encoded bytes as latin1.

For raw downloads, the recommended download filename is provided in the
Content-Disposition header. The problem is that it was provided as a real
unicode string.

Fixed by applying the "proper" latin1-decoding of a utf8-encoding.

controllers: move controllers base class from lib/base to controllers

TG quickstart put it in lib/base.py , but it fits better on the controllers
layer as a base there.

The contributing docs were a bit ahead of time ... but with a typo.

lib: consistently import helpers the same way

Make it easier to grep for any remaining potential layering-violating use of
helpers.

lib: move webhelpers2 and friends to webutils

Gives less of the unfortunate use of helpers - especially in low level libs.

lib: consistently use webutils.url instead of h.url

Templates are still the valid use case for accessing through h.

Simplify mocking of url() function.

diffs: remove unused argument enable_comments and class no-comment

enable_comments was only used to set/not-set the 'no-comment' CSS class.
While this class was emitted, no CSS rule nor any JavaScript logic was
actually using it. Last real usage of that class was removed with commit
e87baa8f1c5bd2488aefc23b95c0db3a04bc8431.

Cleanup the code by not emitting 'no-comment' and remove the
'enable_comments' flag.

routing: separate url handling from routing - move it to webutils

This is a helper method relying on the thread local tg.request. We didn't have
a good place to put it. Now we do.

This (re)moves unfortunate dependencies to the routing module (which almost is
a controller).

db: move URL_SEP constant from db to top level kallithea module

URL_SEP is used in several places - having it in the db module is too high
level.

__init__ might not be the best place for this, but it does no harm to the
dependencies graph to place constants there.

diff: drop per file ignore-whitespace and context - it didn't work and had conceptual issue (Issue #344)

Diffs are currently generated at the low level as one big diff between two vcs
resisions, provided global values for diff context size and flag for ignoring
whitespace. All files use the same flags. There is no way to actually compute
the full diff using these use per file flags, and no simple and efficient way
to add it.

The best option is thus to drop the failed attempt at making it per file, and
just rely on the simple global flags in the URL.

The links for changing whitespace and context is sometimes shown for the whole
"page", and sometimes next to the diff for one file. For now, keep showing the
link in these places, but make sure it navigates back to the FID of the section
where the link was clicked.

The implementation is completely rewritten and moved to a more appropriate
location in helpers.

With a more clean implementation, we also consistently use the simple getters
to extract values from the URL.

lib: drop own asbool implementation and consistently use tg.support.converters as utils2.asbool

str2bool never reported error on odd input such as '' or '-1', but the tg
asbool behaviour of raising ValueError("String is not true/false: %r" % obj) in
that case seems fine.

scripts: drop isort --wrap-length 160 - it is broken with py3 and not really necessary

Under Python 3, isort 4.3.21 failed with
https://github.com/timothycrosley/isort/issues/741 on
kallithea/controllers/api/api.py :

Traceback (most recent call last):
File "data/env/bin/isort", line 10, in <module>
sys.exit(main())
File ".../env/lib64/python3.7/site-packages/isort/main.py", line 379, in main
for sort_attempt in attempt_iterator:
File ".../env/lib64/python3.7/site-packages/isort/main.py", line 377, in <genexpr>
attempt_iterator = (sort_imports(file_name, **arguments) for file_name in file_names)
File ".../env/lib64/python3.7/site-packages/isort/main.py", line 88, in sort_imports
result = SortImports(file_name, **arguments)
File ".../env/lib64/python3.7/site-packages/isort/isort.py", line 207, in __init__
self._add_formatted_imports()
File ".../env/lib64/python3.7/site-packages/isort/isort.py", line 606, in _add_formatted_imports
self._add_from_imports(from_modules, section, section_output, sort_ignore_case)
File ".../env/lib64/python3.7/site-packages/isort/isort.py", line 526, in _add_from_imports
import_statement = self._multi_line_reformat(import_start, from_import_section, comments)
File ".../env/lib64/python3.7/site-packages/isort/isort.py", line 552, in _multi_line_reformat
dynamic_indent, indent, line_length, comments)
File ".../env/lib64/python3.7/site-packages/isort/isort.py", line 705, in _output_grid
if len(next_statement.split(self.line_separator)[-1]) + 1 > line_length:
TypeError: '>' not supported between instances of 'int' and 'str'

lib: handle both HTML, unsafe strings, and exceptions passed to helpers.flash()

Before, h.flash would trust any input to contain html ... and callers would
convert exceptions to string, often with a simple str() or unicode() ... which
really didn't deserve to be trusted.

Instead, only trust messages that have a __html__ and escape anything else ...
but also apply str/unicode on the parameter so the caller doesn't have to but
*can* pass an exception directly.

vcs: always return bytes from node.content

We will rather have the unicode conversions explicit.

Note: Py3 bytes doesn't have .startswith - replace that with a regexp.

py3: only use safe_str for string conversion - not for arbitrary __str__ invocation

Usually, we actually just want unicode string, and it is better to just unicode().

cleanup: minor refactorings and simplification of dict usage

Makes it more py3 compatible.

scm: don't try to get IP address from web request in model

Remove a layering violation and make functions more reusable when they no
longer depend on global state.

At this level, the IP address (and information about the current user) is only
used for hooks logging push / pull operations. Arguably, IP address logging
only belongs in an HTTP access log, not in the log of push/pull operations. But
as long as we have IP addresses in the logs, we have to provide it. The (good?)
alternative would be to drop IP address from the push / pull logs ...

locking: drop the pull-to-lock / push-to-unlock functionality

The feature is not worth the maintenance cost. The locking is too coarse and
unflexible with insufficient UI and UX. The implementation is also quite
invasive in tricky areas of the code, and thus high maintenance.
Dropping this will enable other cleanup ... or at least make it easier.

files: remove need for webhelpers.html.literal

The following code is unnecessarily complex:
h.literal(_('There are no files yet. %s') % add_new)

First of all, the '%s' part in the translatable string is a whole new
sentence, independent of the first. There is no reason it needs to be part
of the same translatable string.

Secondly, the only reason for h.literal is to preserve the link in
'add_new' (which contains the result of 'h.link_to'). But, h.link_to
actually already is a 'literal' object. The problem is that the special
'literal' property is lost due to the coercion into a plain string via the
'%' operator.

The following code would be a possible solution for the second issue:
h.HTML(_('There are no files yet. %s')) % add_new
i.e. make sure that the format string is not a plain string but itself a
literal object (after its contents being escaped), before applying the '%'
operator.

To handle the first issue, this would become:
h.HTML(_('There are no files yet.')) + ' ' + h.HTML(add_new)
but, here h.HTML is unnecessary on the first string because there is nothing
special about it, and equally unnecessary on the 'add_new' variable because
h.link_to already returns a literal object.

So, the final code becomes:
_('There are no files yet.') + ' ' + add_new

git: fix links to nodes that are submodule links

Without this changeset, a link pointing to ./<submodulename> would generate a
'500 Internal Server Error'.

Instead, redirect requests to submodule paths to the target URL.

vcs: introduce 'branches' attribute on changesets, making it possible for Git to show multiple branches for a changeset

Mercurial changesets will always have have exactly one branch (which might be
"default"). The VCS data model was the same.

Git allows for a changeset to have 0 or more branches ... and possibly one of
them as active. The right data model is thus to have an enumerable of branches.

We thus add a 'branches' attribute and use it where applicable.

The existing 'branch' attribute used some heuristics to decide which branch use
as "the" branch ... and in some places code (and tests) rely on that. We thus
keep that old method, knowing that some of its uses probably should move to
'branches'.

The code for retrieving Git branches is based on work by Dominik Ruf.

auth: restore anonymous repository access

Dominik Ruf found that aa25ef34ebab introduced a regression in anonymous access
to repositories ... if that is enabled.

The refactoring was too strict when it missed that not all repo permission
checks require a logged in user. Read access can be granted to the default user
... but not write or admin.

Instead of the commands used in aa25ef34ebab, the following commands are used
to consistently also allow the default user in all decorators where we only need
repo read access:

# Introduce explicit allow_default_user=True - that was the default before aa25ef34ebab
sed -i 's/@LoginRequired()/@LoginRequired(allow_default_user=True)/g' `hg mani`
sed -i 's/@LoginRequired(\(..*\))/@LoginRequired(\1, allow_default_user=True)/g' `hg mani`
# The primary case: Replace @NotAnonymous with removal of allow_default_user=True
perl -0pi -e 's/\@LoginRequired\((?:(.*), )?allow_default_user=True\)\n\s*\@NotAnonymous\(\)/\@LoginRequired(\1)/g' `hg mani`
# If there is a global permission check, no anonymous is ever allowed
perl -0pi -e 's/\@LoginRequired\(allow_default_user=True\)(\n\s*\@HasPermission)/\@LoginRequired()\1/g' `hg mani`
# Repo access for write or admin also assume no default user
perl -0pi -e 's/\@LoginRequired\(allow_default_user=True\)(\n\s*\@HasRepoPermissionLevelDecorator\('"'(write|admin)'"'\))/\@LoginRequired()\1/g' `hg mani`

diffs: cleanup of variable naming around cut_off_limit

A brief summary of this area:

The base controller sets self.cut_off_limit from config and is used for diffs,
unless controllers are given a fulldiff query parameter. In a few cases, these
are passed to templates as c.cut_off_limit or c.fulldiff . Also, if the diff
function returns a LimitedDiffContainer, c.limited_diff is set so the UI can
report the data set is partial.

codingstyle: trivial whitespace fixes

Reported by flake8.

controllers: rename __before__ to _before in preparation of TurboGears2

__before__ in Pylons is called _before in TurboGears2. We can prepare this
rename already in Pylons-based Kallithea, so that the real TG2 migration
commit just changes the BaseController.

Since TurboGears2 _before can pass extra arguments, we add *args and
**kwargs parameters as well.

tg: minimize future diff by some mocking and replacing some pylons imports with tg

No actual tg dependency yet, just a temporary hack faking tg as an alias for
pylons.

Based on work by Alessandro Molina.

auth: simplify repository permission checks

In practice, Kallithea has the 'repository.admin' permission imply the
'repository.write' permission, which again implies 'repository.read'.

This codifies/enforces this practice by replacing HasRepoPermissionAny
"perm function" with the new HasRepositoryLevel function, reducing the
risk of errors and saving quite a lot of typing.

controllers: avoid setting request state in controller instances - set it in the thread global request variable

In TurboGears, controllers are singletons and we should avoid using instance
variables for any volatile data. Instead, use the "global thread local" request
context.

With everything in request, some use of c is dropped.

Note: kallithea/controllers/api/__init__.py still use instance variables that
will cause problems with TurboGears.

lib: move jsonify from utils to base

Suggested by Mads Kiilerich.

The jsonify method is the only thing in utils that directly uses pylons.
Move it to base where it fits better and we can use existing global imports.

controllers: remove redundant Repository database lookups

BaseRepoController already provides the repository object in c.db_repo
(may be None) and BaseController already provides the repository name
in c.repo_name (as given in the URL).

(Arguably, that's a bad design, and we should revisit that decision in
the future. For now, the code just performs slightly better.)

Turbogears2 migration: replace pylons.url by kallithea.config.routing.url

In preparation for the migration to Turbogears2, introduce a
kallithea.config.routing.url to replace pylons.url. The implementation is
basically the same: wrap around routes.url().

This change involves:
- a number of import statement changes
- fixing some tests in test_libs.py; to avoid duplication, the different
implementations of fake_url were grouped in one place.

This change was first proposed by Alessandro Molina in his initial
port. Following changes were made afterwards:
- move UrlGenerator from kallithea.lib.utils to kallithea.config.routing
- add documentation to UrlGenerator
- kallithea/lib/auth.py used url_for instead of url, for no apparent reason
so this was changed.
- fix libs tests
- rebase onto Pylons-based Kallithea first

diff: correct handling of links to old filename in renames

There were links to the file at the parent revision ... but if the file had
been renamed, it used the wrong name.

diff: rework data structure used by diff_block.diff_block (used for changeset diffs)

Make it a pure reusable function that also can replace diff_block_simple.

Improve code clarity by using Python tuples for tuples.

fid and url_fid are just shuffled around - not cleaned up for now.

diff: minor cleanups

More consistency and preparing for later changes.

diff: show correct operation for file diffs instead of '???'

Fix issue seen on the url $repo/diff/$filename?diff2=hash2&diff1=hash2 .

Drop returning unused size from diffs.wrapped_diff and return the operation
instead.

files: always show the requested version - not last changeset that touched the file

The last changeset to touch the file is rarely relevant - the whole annotate of
individual lines of code will give that kind of information in a much more
relevant way. The information is however available in the history drophown.

This also gets rid of some weird revision compare.

diff: make sure context parameter is an integer

Prevent Abort in mdiff on malformed URLs.

windows: node paths should always use posixpath - avoid \ in repo internal paths on windows

Using \ on Windows would result in a wrong node path which could not be checked
out with Git on Windows.

cleanup: fix 'try ... except ...' formatting missed in d69aa464f373

Examples spotted by Jiří Suchan <yed@vanyli.net>.

files: support annotation on files larger than cut_off_limit

When requesting the annotation for a file larger than the cut_off_limit
configured in the ini file, the only current option is to click the useless
'show as raw' (which is not an annotation).

Replace it with a link 'Show full annotation anyway' instead.

cleanup: replace redirect with WebOb exceptions

All redirect does is to log "Generating 302 redirect" with logging the actual
location and raise a WebOb HTTPFound exception, and the logging is redundant,
as WebOb exceptions and their status codes are already logged.

Instead, just raise the exception directly, which is both explicit and simpler
(and finally, gets rid of "return redirect" which never really returns).

remove vestiges of Python 2.5 support

We only support Python 2.6 and 2.7; hence we do not need to import
with-statement support from __future__.

cleanup: consistently use 'except ... as ...:'

Use the Python 2.6+ syntax instead of the old confusing 'except ..., ...'
syntax.

archive: refactor archive handling

Use file descriptors in a more sane way.

archive: fix archive failing first time on Windows - close file descriptor early

File descriptors is not used correctly - this is just a minimal stop gap.

spelling: fix various typos

This commit fixes various typos or basic English grammar mistakes found by
reviewing the kallithea.pot file.

Full correction of sentences that are not very well formulated, like missing
articles, is out of scope for this commit. Likewise for inconsistent
capitalization of strings like 'Repository group'/'Repository Group'.

files: use current revision as default for 'show at' (Issue #31)

Before, clicking 'show at' before selecting a changeset would fail when looking
for 'no revision'.

Correct licensing information in individual files.

The top-level license file is now LICENSE.md.

Also, in various places where there should have been joint copyright holders
listed, a single copyright holder was listed. It does not appear easy to add
a link to a large list of copyright holders in these places, so it simply
refers to the fact that various authors hold copyright.

In future, if an easy method is discovered to link to a list from those
places, we should do so.

Finally, text is added to LICENSE.md to point to where the full list of
copyright holders is, and that Kallithea as a whole is GPLv3'd.

Second step in two-part process to rename directories.
This is the actual directory rename.