kallithea Changeset - c0b2410d63a5

Changeset - c0b2410d63a5

Parent rev.

Child rev.

[Not reviewed]

default

0 2 0

FUJIWARA Katsunori - 9 years ago 2017-01-22 18:17:38
foozy@lares.dti.ne.jp

search: prevent username related conditions from removing "stop words"

Before this revision, username related conditions below cause
unintentional ignorance of "stop words".

- owner: (for all)
- author: (for "Commit messages")

Therefore, username related conditions with "this", "a", "you", and so
on are completely ignored, even if they are valid username components.

To prevent username related conditions from removing "stop words",
this revision explicitly specifies "analyzer" for username related
fields of SCHEMA and CHGSETS_SCHEMA.

Difference between EMAILADDRANALYZER and default analyzer of TEXT is
whether "stop words" are preserved or not. Tokenization is still
applied on usernames.

For future changing, this revision doesn't make EMAILADDRANALYZER
share analyzer definition with PATHANALYZER, even though their
definitions are identical with each other at this revision.

This revision requires full re-building index tables, because indexing
schemas are changed.

Original patch has been modified by Mads Kiilerich - tests of 'owner' will be
addressed separately.

2 files changed with 13 insertions and 5 deletions:

kallithea/lib/indexers/__init__.py

kallithea/tests/functional/test_search_indexing.py

0 comments (0 inline, 0 general)

kallithea/lib/indexers/__init__.py

➞

Show inline comments

@@ @@ -35,24 +35,32 @@ sys.path.append(dirname(dirname(dirname( @@
 from whoosh.analysis import RegexTokenizer, LowercaseFilter, IDTokenizer
 from whoosh.fields import TEXT, ID, STORED, NUMERIC, BOOLEAN, Schema, FieldType, DATETIME
 from whoosh.formats import Characters
 from whoosh.highlight import highlight as whoosh_highlight, HtmlFormatter, ContextFragmenter
 from kallithea.lib.utils2 import LazyProperty
 log = logging.getLogger(__name__)
 # CUSTOM ANALYZER wordsplit + lowercase filter
 ANALYZER = RegexTokenizer(expression=r"\w+") | LowercaseFilter()
 # CUSTOM ANALYZER wordsplit + lowercase filter, for emailaddr-like text
+#
 # This is useful to:
 # - avoid removing "stop words" from text
 # - search case-insensitively
+#
 EMAILADDRANALYZER =  RegexTokenizer() | LowercaseFilter()
 # CUSTOM ANALYZER raw-string + lowercase filter
+#
 # This is useful to:
 # - avoid tokenization
 # - avoid removing "stop words" from text
 # - search case-insensitively
+#
 ICASEIDANALYZER = IDTokenizer() | LowercaseFilter()
 # CUSTOM ANALYZER raw-string
+#
 # This is useful to:
@@ @@ -63,49 +71,49 @@ IDANALYZER = IDTokenizer() @@
 # CUSTOM ANALYZER wordsplit + lowercase filter, for pathname-like text
+#
 # This is useful to:
 # - avoid removing "stop words" from text
 # - search case-insensitively
+#
 PATHANALYZER = RegexTokenizer() | LowercaseFilter()
 #INDEX SCHEMA DEFINITION
 SCHEMA = Schema(
     fileid=ID(unique=True),
     owner=TEXT(),
+    owner=TEXT(analyzer=EMAILADDRANALYZER),
     # this field preserves case of repository name for exact matching
     repository_rawname=TEXT(analyzer=IDANALYZER),
     repository=TEXT(stored=True, analyzer=ICASEIDANALYZER),
     path=TEXT(stored=True, analyzer=PATHANALYZER),
     content=FieldType(format=Characters(), analyzer=ANALYZER,
                       scorable=True, stored=True),
     modtime=STORED(),
     extension=TEXT(stored=True, analyzer=PATHANALYZER)
+)
 IDX_NAME = 'HG_INDEX'
 FORMATTER = HtmlFormatter('span', between='\n<span class="break">...</span>\n')
 FRAGMENTER = ContextFragmenter(200)
 CHGSETS_SCHEMA = Schema(
     raw_id=ID(unique=True, stored=True),
     date=NUMERIC(stored=True),
     last=BOOLEAN(),
     owner=TEXT(),
+    owner=TEXT(analyzer=EMAILADDRANALYZER),
     # this field preserves case of repository name for exact matching
     # and unique-ness in index table
     repository_rawname=ID(unique=True),
     repository=ID(stored=True, analyzer=ICASEIDANALYZER),
     author=TEXT(stored=True),
+    author=TEXT(stored=True, analyzer=EMAILADDRANALYZER),
     message=FieldType(format=Characters(), analyzer=ANALYZER,
                       scorable=True, stored=True),
     parents=TEXT(),
     added=TEXT(analyzer=PATHANALYZER),
     removed=TEXT(analyzer=PATHANALYZER),
     changed=TEXT(analyzer=PATHANALYZER),
+)
 CHGSET_IDX_NAME = 'CHGSET_INDEX'
 # used only to generate queries in journal
 JOURNAL_SCHEMA = Schema(

kallithea/tests/functional/test_search_indexing.py

➞

Show inline comments

@@ @@ -179,20 +179,20 @@ class TestSearchControllerIndexing(TestC @@
     @parametrize('searchtype,query,hit', [
         # matching against both 2 files
         ('content', 'owner:"this is it"', 0),
         ('content', 'owner:this-is-it', 0),
         ('path', 'owner:"this is it"', 0),
         ('path', 'owner:this-is-it', 0),
         # matching against both 2 revisions
         ('commit', 'owner:"this is it"', 0),
         ('commit', 'owner:"this-is-it"', 0),
         # matching against only 1 revision
         ('commit', 'author:"this is it"', 0),
         ('commit', 'author:"this-is-it"', 0),
         ('commit', 'author:"this is it"', 1),
         ('commit', 'author:"this-is-it"', 1),
     ])
     def test_mailaddr_stopword(self, searchtype, query, hit):
         response = self.app.get(url(controller='search', action='index'),
                                 {'q': query, 'type': searchtype})
         response.mustcontain('>%d results' % hit)

0 comments (0 inline, 0 general)