py3: remove safe_unicode in places where it no longer is needed because all strings (except bytes) already *are* unicode strings
(The remaining safe_unicode calls are still needed and can't just be removed, generally because we in these cases still have to convert from bytes to unicode strings.)
py3: fix kallithea-cli ini parsing after ConfigParser got strict=True
ConfigParser in py3 defaults to strict=True and would thus reject our ssh logging hack of renaming config sections ... which cause duplicate section names.
Fortunately, fileConfig now also allows passing a ConfigParser, and we can avoid using io.StringIO .
py3: update ssh for base64.b64decode raising binascii.Error instead of TypeError
A command like: python -c 'import base64; base64.b64decode("QQ")' would fail in Python2 with: TypeError: Incorrect padding but in python3: binascii.Error: Incorrect padding
There is no point in creating dicts and then logging them as json. Also, json can't handle py3 bytes and it would fail on py3. (ext_json could perhaps handle bytes, but it seems better to keep it simple and explicit.)
If the default repr isn't good enough, it would be better to use pprint. But repr is good enough.
py3: make get_current_authuser handle missing tg context consistently and explicitly
tg context handling ends up using tg.support.registry.StackedObjectProxy._current_obj for attribute access ... which if no context has been pushed will end up in: raise TypeError( 'No object (name: %s) has been registered for this ' 'thread' % self.____name__)
utils2.get_current_authuser used code like: if hasattr(tg.tmpl_context, 'authuser'):
Python 2 hasattr will call __getattr__ and return False if it throws any exception. (It would thus catch the TypeError and silently fall through to use the default user None.) This hasattr behavior is confusing and hard to use correctly. Here, it was used incorrectly. It has been common practice to work around by using something like: getattr(x, y, None) is not None
Python 3 hasattr fixed this flaw and only catches AttributeError. The TypeError would thus (rightfully) be propagated. That is a change that must be handled when introducing py3 support.
The get_current_authuser code could more clearly and simple and py3-compatible be written as: return getattr(tmpl_context, 'authuser', None) - but then we also have to handle the TypeError explicitly ... which we are happy to do.
celery: introduce make_app instead of creating app at import time
It is dirty to instantiate things at import time (unless it really is basic singletons).
In 0.5.1 (and earlier), such dirtyness made partial test execution fail when other things had global side effects and things didn't use the usual import order:
$ py.test kallithea/lib/ collecting ... ――― kallithea/lib/celerypylons/__init__.py ――― kallithea/lib/celerypylons/__init__.py:58: in <module> app.config_from_object(celery_config(tg.config)) kallithea/lib/celerypylons/__init__.py:28: in celery_config assert config['celery.imports'] == 'kallithea.lib.celerylib.tasks', 'Kallithea Celery configuration has not been loaded' data/env/lib/python2.7/site-packages/tg/configuration/tgconfig.py:31: in __getitem__ return self.config_proxy.current_conf()[key] E KeyError: 'celery.imports'
Avoid that by running a "factory" function when the celery app actually is needed.
config: fix pyflakes warning about unused tg import
app_cfg had an unused 'import tg'. tg.hooks was used, but through a separate import. Clean that up by consistently using tg (which always makes tg.hooks available) and dropping the separate hooks import.
(localrepo might already always be available in the mercurial namespace due to side effects from other imports, but it is still better to do it explicit ... and also to please pytype.)
Since a while, the test suite shows following warning:
kallithea/tests/__init__.py:29 /home/tdescham/repo/contrib/kallithea/kallithea-review/kallithea/tests/__init__.py:29: PytestAssertRewriteWarning: Module already imported so cannot be rewritten: kallithea.tests pytest.register_assert_rewrite('kallithea.tests')
The problem can be fixed by moving the register_assert_rewrite call from kallithea/tests/__init__.py to the root-level conftest.py, outside of the 'kallithea' module.
tests: remove race condition in test_forgot_password
One in so many times, test_forgot_password failed with:
kallithea/tests/functional/test_login.py:427: in test_forgot_password assert '\n%s\n' % token in body E assert ('\n%s\n' % 'd71ad3ed3c6ca637ad00b7098828d33c56579201') in "Password Reset Request\n\nHello passwd reset,\n\nWe have received a request to reset the password for your account.\n\nTo s...7e89326ca372ade1d424dafb106d824cddb\n\nIf it weren't you who requested the password reset, just disregard this message.\n"
i.e. the expected token is not the one in the email.
The token is calculated based on a timestamp (among others). And the token is calculated twice: once in the real code and once in the test, each time on a slightly different timestamp. Even though there is flooring of the timestamp to a second resolution, there will always be a race condition where the two timestamps floor to a different second, e.g. 4.99 vs 5.01.
The problem can be reproduced reliably by adding a sleep of e.g. 2 seconds before generating the password reset mail (after the test has already calculated the expected token).
Solve this problem by mocking the time.time() used to generate the timestamp, so that the timestamp used for the real token is the same as the one used for the expected token in the test.
scripts: handle "Python 2.7 reached the end of its life" message
The script failed with:
Error: pip detected following problems: DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Thus, clarify the use of i18n.lang (refining f2f7a8c1281e and 8931078f70db) and set 'en' as default value on app startup.
TurboGears requires an (empty) translation for the source language which is default for i18n.lang . The empty .mo for en is created as the 4 magic .mo bytes followed by lengths of 0: printf '\x95\x04\x12\xde\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0' > kallithea/i18n/en/LC_MESSAGES/kallithea.mo
login: fix incorrect CSRF rejection of "Reset Your Password" form (Issue #350)
htmlfill would remove the CSRF token from the form when substituting the query parameters, causing password reset to break.
By default, htmlfill will clear all input fields that doesn't have a new "default" value provided. It could be fixed by setting force_defaults to False - see http://www.formencode.org/en/1.2-branch/modules/htmlfill.html . It could also be fixed by providing the CSRF token in the defaults to be substituted in the form.
Instead, refactor password_reset_confirmation to have more explicitly safe handling of query parameters. Replace htmlfill with the usual template variables.
The URLs are generated in kallithea/model/user.py send_reset_password_email() and should only contain email, timestamp (integer as digit string) and a hex token from get_reset_password_token() .
setup: bump sqlalchemy minimum version to 1.2.9 to get rid of py3 warning
Avoid verbose warning:
build/minimum-dependency-versions-venv/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py:637 .../site-packages/sqlalchemy/util/langhelpers.py:637: DeprecationWarning: `formatargspec` is deprecated since Python 3.5. Use `signature` and the `Signature` object directly d_args = inspect.formatargspec(spec[0][1:])
setup: bump alembic minimum version to 1.0.10 to get rid of py3 warning
Avoid verbose warning:
.../site-packages/alembic/util/langhelpers.py:92: DeprecationWarning: `formatargspec` is deprecated since Python 3.5. Use `signature` and the `Signature` object directly formatvalue=lambda x: '=' + x)
setup: bump decorator minimum version to 4.2.1 to get rid of py3 warning
Avoid verbose warning:
.../site-packages/decorator.py:95: DeprecationWarning: `formatargspec` is deprecated since Python 3.5. Use `signature` and the `Signature` object directly formatvalue=lambda val: "", *argspec)[1:-1]
setup: bump dulwich minimum version to 0.19.0 to get good py3 support
Fix failure in test_compare_forks_on_branch_extra_commits_git ... but make a major bump to make sure we get good py3 support in this cruicial and complex low level library.
setup: bump webtest minimum version to 2.0.6 to get py3 stdlib support
/usr/lib64/python3.7/http/cookiejar.py:723: in is_third_party if not domain_match(req_host, reach(request.origin_req_host)): E AttributeError: '_RequestCookieAdapter' object has no attribute 'origin_req_host'
File ".../lib/python3.7/site-packages/whoosh/reading.py", line 241, in expand_prefix if fn != fieldname or not text.startswith(prefix): TypeError: startswith first arg must be bytes or a tuple of bytes, not str
setup: bump mako minimum version to 0.9.1 to get py3 support
Avoid test_api_get_pullrequest crash:
data/env3/lib/python3.7/site-packages/mako/template.py:653: in _compile_text code = compile(source, cid, 'exec') E File "email_templates_button_html", line 15 E def render_body(context,url,title='',padding_top=,padding_bottom=,**pageargs): E ^ E SyntaxError: invalid syntax
setup: bump python-dateutil minimum version to 2.1.0 to get py3 support
Avoid py2 syntax:
E File "/home/mk/kallithea-py3/build/minimum-dependency-versions-venv/lib/python3.7/site-packages/dateutil/tz.py", line 78 E `self._name`, E ^ E SyntaxError: invalid syntax
setup: bump formencode minimum version to 1.3.1 to get py3 support
1.3.0 failed with py3:
build/minimum-dependency-versions-venv/lib/python3.7/site-packages/formencode/api.py:153: if unicode is not str: # Python 2 E NameError: name 'unicode' is not defined
E File ".../build/minimum-dependency-versions-venv/lib/python3.7/site-packages/tg/request_local.py", line 37, in languages_best_match E items = [i for i, q in sorted(al._parsed, key=lambda iq: -iq[1])] E TypeError: 'NoneType' object is not iterable
b075693b3214 introduced use of acceptable_offers which with WebOb < 1.8 would fail with:
File ".../kallithea/lib/middleware/pygrack.py", line 189, in __call__ elif req.accept.acceptable_offers(self.valid_accepts): AttributeError: 'MIMEAccept' object has no attribute 'acceptable_offers'
cache: drop setup_cache_regions - tg will already have done that and coerced the types correctly
The configuration and type fixing will be invoked from make_base_app, and we will thus not have to do it:
File "kallithea/config/middleware.py", line 31, in make_app_without_logging return make_base_app(global_conf, full_stack=full_stack, **app_conf) File ".../python3.7/site-packages/tg/configuration/app_config.py", line 176, in make_base_app wrap_app) File ".../python3.7/site-packages/tg/configurator/application.py", line 112, in _make_app app = TGApp(conf) File ".../python3.7/site-packages/tg/wsgiapp.py", line 49, in __init__ app_wrapper = wrapper(self.wrapped_dispatch, self.config) File ".../python3.7/site-packages/tg/appwrappers/caching.py", line 36, in __init__ self.options = parse_cache_config_options(config) File ".../python3.7/site-packages/beaker/util.py", line 430, in parse_cache_config_options
This will fix a py3 problem where setup_cache_regions was run *after* beaker had coerced types, thus introducing string types in the config where beaker expected the integers it had put there.
Visiting a /statistics with py3 would fail with: ... in statistics sorted(lang_stats, reverse=True, key=lambda k: k[1])[:10] TypeError: '<' not supported between instances of 'dict' and 'dict'
The "summary" computation didn't have that problem. And it put '?' as description for unknown extensions. And it had stable output as it also sorted on the file extension as secondary key. Just use that.
lib: clean up ext_json and how it is used - avoid monkey patching
Note that py3 json.dumps will return ASCII (with all unicode escaped) encoded as str. But we generally want JSON as bytes (which json.loads also can read), so also wrap the result with ascii_bytes in many places.
vcs: fix get_file_annotate - consistently bind sha so it has the right value when executing
The Git implementation did *not* save the sha value in the lambda expression for the "changeset lazy loader". Thus, if the generator had moved on and assigned a different value to sha when the expression was executed, it would use the "wrong" sha.
Fixed by doing as the Hg implementation: bind the sha value as value of a default parameter when defining the lambda expression.
The Hg implementation did however also save the line - it is not used, and there is no need for that.
vcs: tweak how revisions and repo names are shown in error messages
Decode bytes to str, and show repo name instead of repr or full server file system path. In some places, it will only report the "basename" of the repository, without any "group names" that also would be nice to have. The easy alternative would be to show the full file system path ... but it would be unfortunate to leak absolute server side paths to end users.