Revision - 61f6e5c - In the Snowball dictionary, don't try to stem excessively-long words. - origin: https://github.com/postgres/postgres

visit type:

https://github.com/postgres/postgres

07 April 2024, 13:56:23 UTC

Revision 61f6e5c2360382be99846be904af33e3406c938a authored by Tom Lane on 31 August 2022, 14:42:05 UTC, committed by Tom Lane on 31 August 2022, 14:42:05 UTC

In the Snowball dictionary, don't try to stem excessively-long words.

If the input word exceeds 1000 bytes, don't pass it to the stemmer;
just return it as-is after case folding.  Such an input is surely
not a word in any human language, so whatever the stemmer might
do to it would be pretty dubious in the first place.  Adding this
restriction protects us against a known recursion-to-stack-overflow
problem in the Turkish stemmer, and it seems like good insurance
against any other safety or performance issues that may exist in
the Snowball stemmers.  (I note, for example, that they contain no
CHECK_FOR_INTERRUPTS calls, so we really don't want them running
for a long time.)  The threshold of 1000 bytes is arbitrary.

An alternative definition could have been to treat such words as
stopwords, but that seems like a bigger break from the old behavior.

Per report from Egor Chindyaskin and Alexander Lakhin.
Thanks to Olly Betts for the recommendation to fix it this way.

Discussion: https://postgr.es/m/1661334672.728714027@f473.i.mail.ru

1 parent fe9d009

Files
Changes

Permalinks

Tip revision: 61f6e5c2360382be99846be904af33e3406c938a authored by Tom Lane on 31 August 2022, 14:42:05 UTC
In the Snowball dictionary, don't try to stem excessively-long words.

Tip revision: 61f6e5c

File	Mode	Size
config
contrib
doc
src
.cirrus.yml	-rw-r--r--	17.7 KB
.dir-locals.el	-rw-r--r--	730 bytes
.editorconfig	-rw-r--r--	183 bytes
.git-blame-ignore-revs	-rw-r--r--	9.5 KB
.gitattributes	-rw-r--r--	1.4 KB
.gitignore	-rw-r--r--	504 bytes
COPYRIGHT	-rw-r--r--	1.2 KB
GNUmakefile.in	-rw-r--r--	4.1 KB
HISTORY	-rw-r--r--	277 bytes
Makefile	-rw-r--r--	1.8 KB
README	-rw-r--r--	1.2 KB
README.git	-rw-r--r--	721 bytes
aclocal.m4	-rw-r--r--	397 bytes
configure	-rwxr-xr-x	581.6 KB
configure.ac	-rw-r--r--	86.5 KB

Showing with 0 additions and 0 deletions (0 / 0 diffs computed)

Computing file changes ...

https://github.com/postgres/postgres

In the Snowball dictionary, don't try to stem excessively-long words.

README