lib: move urlification to webutils

Less use of helpers in model.

lib: consistently import helpers the same way

Make it easier to grep for any remaining potential layering-violating use of
helpers.

cleanup: get rid of most "import *"

Apply script generated with the following hack:
(
hg loc '*.py'|xargs pyflakes-2 | sed -rn "s/([^:]*):.*'(.*)' may be undefined, or defined from star imports.*/sed -ri 's,\\\\<\2\\\\>([^=]|$),XXXX.\2\\\\1,g' \1/gp" | sort -u
hg loc '*.py'|xargs pyflakes-2 | sed -rn "s/([^:]*):.* undefined name '(.*)'$/sed -ri 's,\\\\<\2\\\\>([^=]|$),XXXX.\2\\\\1,g' \1/gp" | sort -u
hg loc '*.py'|xargs pyflakes-2 | sed -rn "s/([^:]*):.*'(from .*)\.([^.]*) import \*' used.*/sed -ri 's,\\\\<XXXX\\\\.,\3.,g' \1/gp" | sort -u
hg loc '*.py'|xargs pyflakes-2 | sed -rn "s/([^:]*):.*'(from .*)\.([^.]*) import \*' used.*/sed -ri 's,\2\\\\.\3 .*,\2 import \3,g' \1/gp" | sort -u
) | grep -v kallithea/bin/kallithea_cli_ishell.py > fix2.sh

py3: fix error in template from introducing safe_unicode in 9203621cae03

Add missing test coverage of actual edit form.

lib: handle both HTML, unsafe strings, and exceptions passed to helpers.flash()

Before, h.flash would trust any input to contain html ... and callers would
convert exceptions to string, often with a simple str() or unicode() ... which
really didn't deserve to be trusted.

Instead, only trust messages that have a __html__ and escape anything else ...
but also apply str/unicode on the parameter so the caller doesn't have to but
*can* pass an exception directly.

tests: fix test_files.py json comparison to load json to struct so it is independent of serialization order

Py3 would fail the test because different ordering.

routing: fix files_annotate_home annotate value to be compatible with Routes >= 2

The routing entry for files_annotate_home had annotate=True. That primarily
served to let the controller files.index differentiate files_annotate_home from
files_home and files_home_nopath . Anything true can work.

test_files.py is creating files_annotate_home URLs in an odd way: Instead of
explicitly specifying it is a files_annotate_home URL, it passes expected
controller parameters and expects routing to find a URL that would provide
these parameters. It thus also has to specify the otherwise "invisible"
annotate value. For Routes < 2, True works just fine. For Routes >= 2, it seems
to expect values that actually can be encoded in URLs.

Thus, instead of True, use '1'.

style: use monospace for changeset messages

Similar to commits 9005c37dbe21 and 6e65c2153baa that make PR descriptions
display in a monospace font, apply monospace on changeset messages.

Note that in the files template, the reference to the non-existing 'commit'
class can be replaced instead of appending to it.

diff: fix crash when displaying diff on a single file

File diff on an image file would fail on
%if op in 'DM':
because op was None:
TypeError: 'in <string>' requires string as left operand, not NoneType

But really, if op is None, we also don't want to show invalid "Show images"
links. Thus, guard the whole image display section with having an actual op.

_parse_gitdiff will never return op None, but wrapped_diff is more lazy and
might do that. It could be considered a bug in wrapped_diff, and this change is
just a bad workaround.

codingstyle: trivial whitespace fixes

Reported by flake8.

setup: bump WebOb to >=1.7

TurboGears will soon require WebOb>=1.7 and that cause an innocent test change.

Older WebOb versions also required older WebTest versions but the WebTest
dependency from Pylons will now automatically pick compatible versions.

helpers: refactor and optimize urlify_issues

Avoid parsing the configuration and compiling regexps every time a string is
processed.

Instead, the configuration is parsed once and turned into a function that apply
the compiled regexps in a chain.

A next iteration can perhaps run them in parallel and integrate with the
general urlify machinery.

db: remove superfluous Session.add calls

Don't re-add objects to the SQLAlchemy Session just because they were
modified. Session.add is only for freshly constructed objects that
SQLAlchemy doesn't know about yet.

The rules are quite simple:

When creating a database object by calling the constructor directly, it
must explicitly be added to the session.

When creating an object using a factory function (like "create_repo"),
the returned object has already (by convention) been added to the
session, and should not be added again.

When getting an object from the session (via Session.query or any of the
utility functions that look up objects in the database), it's already
added, and should not be added again. SQLAlchemy notices attribute
modifications automatically for all objects it knows about.

tests: add as little code as possible in __init__.py

kallithea/tests/__init__.py contained quite a lot of code, including the test
base class TestController. This in itself may be considered bad practice.

Specifically, this poses a problem when using pytest 3.0+, in which asserts
in some files are not automatically rewritten to give improved assert
output. That problem can be fixed by explicitly registering such files for
assertion rewriting, but that register call should be executed _before_ said
files are imported. I.e. if the register call is in
kallithea/tests/__init__.py, assert calls in __init__.py itself can not be
rewritten.

Since the TestController base class does effectively contain asserts, and we
do not want to execute the register call from somewhere outside the
kallithea/tests directory, we need to move the TestController class to
another file (kallithea/tests/base.py) so we can have a register call in
__init__.py before loading base.py.

While not strictly necessary to fix the mentioned pytest problem, we take
the opportunity to fully clean __init__.py and move everything to
the new kallithea/tests/base.py. While doing so, unnecessary imports are
removed, and imports are ordered alphabetically. Explicit imports of symbols
from modules that were already imported as a whole, are removed in favor of
fully qualifying the references (e.g. tempfile._RandomNameSequence).

helpers: in urlify_text, use <br> for newlines in pre-formatted text so it can be cut'n'pasted correctly

Before, when copying the preformatted nice looking ASCII-artsy text in a PR or
changeset description from a browser to a text editor, it would be pasted as
text without newline. Simply put, copy/paste ignores that it is inside a <pre>
/ white-space:pre-wrap section.

Instead, translate newlines to <br> which always translates to a newline when
pasted.

pytest migration: rename TestControllerPytest back to TestController

The name TestControllerPytest was introduced to allow a temporary situation
where nose/unittest and pytest-based tests could coexist.
This situation is now over, so the base test class can be renamed again.

pytest migration: functional: switch to standard assert statements

Use unittest2pytest to replace unittest-style assert statements (e.g.
assertEqual) with standard Python assert statements to benefit from pytest's
improved reporting on assert failures.

The conversion by unittest2pytest was correct, except for line wrapping
problems.

tests: fix test_nodes.py and test_files.py which fail on windows due to mimetype differences

The mimetype of files may differ on different platforms (e.g. the mimetype of
.py files on windows is not text/x-python but text/plain). Therefore, compare
the mimetype to the value the platform uses.

windows: node paths should always use posixpath - avoid \ in repo internal paths on windows

Using \ on Windows would result in a wrong node path which could not be checked
out with Git on Windows.

pytest migration: files: convert to TestControllerPytest and parametrize

Migrate the 'files' tests to pytest and its parametrize decorator.
The syntax of this decorator is slightly different: a first argument is a
comma-separated string with the method argument names for which the values
are provided as second argument.

tests: more consistently use unicode where unicode is expected

Nothing but extra u annotation to turn str constants into unicode.

This has been verified by hacking sqlalchemy to fail if wrong string types are passed.

cleanup: use example.com for tests and examples

example.com is explicitly reserved for this purpose. Using that means
we won't accidentally hammer a real server or real email address if an
example value escapes into the wild, e.g. in an automated test.

The domain "kallithea.example.com" has been used throughout to refer to the
example Kallithea server.

tests: provide _authentication_token when POSTing

So far not used, just preparing for the the time when the actual checking is
introduced ...

This change is very verbose. self.app.post should perhaps just at this value
automagically ...

files_browser: use font for file, folder, and submodule icons

This patch introduces a def that returns the <i> element corresponding to the
type (i.e. file, folder, or submodule) and a span for the text. Tests have been
updated. Old png icons have been removed.

Second step in two-part process to rename directories.
This is the actual directory rename.