Files
@ 7aff9a999536
Branch filter:
Location: kallithea/docs/usage/performance.rst
7aff9a999536
4.7 KiB
text/prs.fallenstein.rst
templates, controllers: replace webhelpers.html.literal() with webhelpers.html.HTML() where possible
Usage of webhelpers.literal (h.literal) can be a problem when variables are
not correctly escaped. Luckily, this function can be avoided in several
cases.
Several users of the construct:
h.literal(_('..A..') % (..B..))
can be simplified if (..B..) just contains a call to h.link_to. In this
case, there is actually no need to use h.literal, because the object
returned by link_to is already a literal. It is sufficient to use
webhelpers.html.HTML() like so:
h.HTML(_('..A..')) % (..B..)
which is better because it will escape the '..A..' part instead of passing
it literally.
The need to wrap the '..A..' part in HTML() is to make sure the (escaped)
end result is not a plain string but a 'literal' to avoid double escaping
later.
See also the documentation:
https://docs.pylonsproject.org/projects/webhelpers/en/latest/modules/html/builder.html
"
When literal is used in a mixed expression containing both literals and
ordinary strings, it tries hard to escape the strings and return a
literal. However, this depends on which value has “control” of the
expression. literal seems to be able to take control with all
combinations of the + operator, but with % and join it must be on the
left side of the expression. So these all work:
"A" + literal("B")
literal(", ").join(["A", literal("B")])
literal("%s %s") % (16, literal("kg"))
But these return an ordinary string which is prone to double-escaping later:
"\n".join([literal('<span class="foo">Foo!</span>'), literal('Bar!')])
"%s %s" % (literal("16"), literal("<em>kg</em>"))
"
This same escaping with 'HTML()' was already done by default in mako
templates for constructs like ${_("something")} that do not contain format
specifiers. When the translated string _does_ contain format specifiers, we
want to use the same escaping, but we have to do it explicit and earlier so
the escaping happens already when strings are inserted into the template
string.
Usage of webhelpers.literal (h.literal) can be a problem when variables are
not correctly escaped. Luckily, this function can be avoided in several
cases.
Several users of the construct:
h.literal(_('..A..') % (..B..))
can be simplified if (..B..) just contains a call to h.link_to. In this
case, there is actually no need to use h.literal, because the object
returned by link_to is already a literal. It is sufficient to use
webhelpers.html.HTML() like so:
h.HTML(_('..A..')) % (..B..)
which is better because it will escape the '..A..' part instead of passing
it literally.
The need to wrap the '..A..' part in HTML() is to make sure the (escaped)
end result is not a plain string but a 'literal' to avoid double escaping
later.
See also the documentation:
https://docs.pylonsproject.org/projects/webhelpers/en/latest/modules/html/builder.html
"
When literal is used in a mixed expression containing both literals and
ordinary strings, it tries hard to escape the strings and return a
literal. However, this depends on which value has “control” of the
expression. literal seems to be able to take control with all
combinations of the + operator, but with % and join it must be on the
left side of the expression. So these all work:
"A" + literal("B")
literal(", ").join(["A", literal("B")])
literal("%s %s") % (16, literal("kg"))
But these return an ordinary string which is prone to double-escaping later:
"\n".join([literal('<span class="foo">Foo!</span>'), literal('Bar!')])
"%s %s" % (literal("16"), literal("<em>kg</em>"))
"
This same escaping with 'HTML()' was already done by default in mako
templates for constructs like ${_("something")} that do not contain format
specifiers. When the translated string _does_ contain format specifiers, we
want to use the same escaping, but we have to do it explicit and earlier so
the escaping happens already when strings are inserted into the template
string.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 | .. _performance:
================================
Optimizing Kallithea performance
================================
When serving a large amount of big repositories, Kallithea can start performing
slower than expected. Because of the demanding nature of handling large amounts
of data from version control systems, here are some tips on how to get the best
performance.
Fast storage
------------
Kallithea is often I/O bound, and hence a fast disk (SSD/SAN) and plenty of RAM
is usually more important than a fast CPU.
Caching
-------
Tweak beaker cache settings in the ini file. The actual effect of that is
questionable.
Database
--------
SQLite is a good option when having a small load on the system. But due to
locking issues with SQLite, it is not recommended to use it for larger
deployments.
Switching to MySQL or PostgreSQL will result in an immediate performance
increase. A tool like SQLAlchemyGrate_ can be used for migrating to another
database platform.
Horizontal scaling
------------------
Scaling horizontally means running several Kallithea instances and let them
share the load. That can give huge performance benefits when dealing with large
amounts of traffic (many users, CI servers, etc.). Kallithea can be scaled
horizontally on one (recommended) or multiple machines.
It is generally possible to run WSGI applications multithreaded, so that
several HTTP requests are served from the same Python process at once. That can
in principle give better utilization of internal caches and less process
overhead.
One danger of running multithreaded is that program execution becomes much more
complex; programs must be written to consider all combinations of events and
problems might depend on timing and be impossible to reproduce.
Kallithea can't promise to be thread-safe, just like the embedded Mercurial
backend doesn't make any strong promises when used as Kallithea uses it.
Instead, we recommend scaling by using multiple server processes.
Web servers with multiple worker processes (such as ``mod_wsgi`` with the
``WSGIDaemonProcess`` ``processes`` parameter) will work out of the box.
In order to scale horizontally on multiple machines, you need to do the
following:
- Each instance's ``data`` storage needs to be configured to be stored on a
shared disk storage, preferably together with repositories. This ``data``
dir contains template caches, sessions, whoosh index and is used for
task locking (so it is safe across multiple instances). Set the
``cache_dir``, ``index_dir``, ``beaker.cache.data_dir``, ``beaker.cache.lock_dir``
variables in each .ini file to a shared location across Kallithea instances
- If using several Celery instances,
the message broker should be common to all of them (e.g., one
shared RabbitMQ server)
- Load balance using round robin or IP hash, recommended is writing LB rules
that will separate regular user traffic from automated processes like CI
servers or build bots.
Serve static files directly from the web server
-----------------------------------------------
With the default ``static_files`` ini setting, the Kallithea WSGI application
will take care of serving the static files from ``kallithea/public/`` at the
root of the application URL.
The actual serving of the static files is very fast and unlikely to be a
problem in a Kallithea setup - the responses generated by Kallithea from
database and repository content will take significantly more time and
resources.
To serve static files from the web server, use something like this Apache config
snippet::
Alias /images/ /srv/kallithea/kallithea/kallithea/public/images/
Alias /css/ /srv/kallithea/kallithea/kallithea/public/css/
Alias /js/ /srv/kallithea/kallithea/kallithea/public/js/
Alias /codemirror/ /srv/kallithea/kallithea/kallithea/public/codemirror/
Alias /fontello/ /srv/kallithea/kallithea/kallithea/public/fontello/
Then disable serving of static files in the ``.ini`` ``app:main`` section::
static_files = false
If using Kallithea installed as a package, you should be able to find the files
under ``site-packages/kallithea``, either in your Python installation or in your
virtualenv. When upgrading, make sure to update the web server configuration
too if necessary.
It might also be possible to improve performance by configuring the web server
to compress responses (served from static files or generated by Kallithea) when
serving them. That might also imply buffering of responses - that is more
likely to be a problem; large responses (clones or pulls) will have to be fully
processed and spooled to disk or memory before the client will see any
response. See the documentation for your web server.
.. _SQLAlchemyGrate: https://github.com/shazow/sqlalchemygrate
|