Files
@ 73e3599971da
Branch filter:
Location: kallithea/docs/usage/statistics.rst - annotation
73e3599971da
1.2 KiB
text/prs.fallenstein.rst
journal: make "repository:" filtering condition work as expected (Issue #261)
Before this revision, journal filtering conditions like as below never
match against any entry, even if there are corresponded repositories.
- repository:foo/bar
- repository:foo-bar
Whoosh library, which is used to parse filtering condition, does:
- treat almost all non-alphanumeric characters as delimiter at
parsing condition
- join each conditions at filtering by "AND", by default
For example, filtering condition "repository:foo/bar" is translated as
"repository:foo AND repository:bar". This combined condition never
matches against any entry, because it is impossible that "repository"
field in DBMS table "user_logs" has both "foo" and "bar" values at
same time.
Using TEXT for "repository" of JOURNAL_SCHEMA causes this issue,
because TEXT assumes tokenization at parsing.
In addition to it, using TEXT also causes unintentional ignorance of
"stop words" in filtering conditions. For example, "this", "a", "you",
and so on are ignored at parsing, because these are too generic words
(from point of view of generic "text search").
To make "repository:" filtering condition work as expected, this
revision uses ID instead of TEST for "repository" of
JOURNAL_COLUMN. ID avoids both tokenization and removing "stop words".
This replacement should be safe with already existing DBMS instance,
because:
- JOURNAL_SCHEMA is used only to parse filtering condition
- DBMS table "user_logs" itself is defined by UserLog class
(in kallithea/model/db.py)
BTW, using ID also avoids normalization by lowercase-ing. But this
doesn't violate current case-insensitive search policy, because
LOWER-ing in actual SQL query is achieved by get_filterion() or so in
kallithea/controllers/admin/admin.py.
Before this revision, journal filtering conditions like as below never
match against any entry, even if there are corresponded repositories.
- repository:foo/bar
- repository:foo-bar
Whoosh library, which is used to parse filtering condition, does:
- treat almost all non-alphanumeric characters as delimiter at
parsing condition
- join each conditions at filtering by "AND", by default
For example, filtering condition "repository:foo/bar" is translated as
"repository:foo AND repository:bar". This combined condition never
matches against any entry, because it is impossible that "repository"
field in DBMS table "user_logs" has both "foo" and "bar" values at
same time.
Using TEXT for "repository" of JOURNAL_SCHEMA causes this issue,
because TEXT assumes tokenization at parsing.
In addition to it, using TEXT also causes unintentional ignorance of
"stop words" in filtering conditions. For example, "this", "a", "you",
and so on are ignored at parsing, because these are too generic words
(from point of view of generic "text search").
To make "repository:" filtering condition work as expected, this
revision uses ID instead of TEST for "repository" of
JOURNAL_COLUMN. ID avoids both tokenization and removing "stop words".
This replacement should be safe with already existing DBMS instance,
because:
- JOURNAL_SCHEMA is used only to parse filtering condition
- DBMS table "user_logs" itself is defined by UserLog class
(in kallithea/model/db.py)
BTW, using ID also avoids normalization by lowercase-ing. But this
doesn't violate current case-insensitive search policy, because
LOWER-ing in actual SQL query is achieved by get_filterion() or so in
kallithea/controllers/admin/admin.py.
bbd499c7b55e bbd499c7b55e ac7e43325817 ac7e43325817 ac7e43325817 bbd499c7b55e 5ae8e644aa88 ac7e43325817 ac7e43325817 ac7e43325817 bbd499c7b55e ac7e43325817 ac7e43325817 ac7e43325817 ac7e43325817 ac7e43325817 bbd499c7b55e ac7e43325817 ac7e43325817 ac7e43325817 bbd499c7b55e ac7e43325817 ac7e43325817 ac7e43325817 bbd499c7b55e ac7e43325817 bbd499c7b55e ac7e43325817 ac7e43325817 ac7e43325817 ac7e43325817 | .. _statistics:
=====================
Repository statistics
=====================
Kallithea has a *repository statistics* feature, disabled by default. When
enabled, the amount of commits per committer is visualized in a timeline. This
feature can be enabled using the ``Enable statistics`` checkbox on the
repository ``Settings`` page.
The statistics system makes heavy demands on the server resources, so
in order to keep a balance between usability and performance, statistics are
cached inside the database and gathered incrementally.
When Celery is disabled:
On each first visit to the summary page a set of 250 commits are parsed and
added to the statistics cache. This incremental gathering also happens on each
visit to the statistics page, until all commits are fetched.
Statistics are kept cached until additional commits are added to the
repository. In such a case Kallithea will only fetch the new commits when
updating its statistics cache.
When Celery is enabled:
On the first visit to the summary page, Kallithea will create tasks that will
execute on Celery workers. These tasks will gather all of the statistics until
all commits are parsed. Each task parses 250 commits, then launches a new
task.
|