Language Packs

Language Packs are designed for software developers to extend the performance of their dtSearch Engine powered applications. Language packs include:

Stemming rules and noise-word files

dtSearch products are supplied with stemming rules and a noise-word file for English(US). Stemming is the only search expansion option which is 'on' by default in the dtSearch end-user products; the reason for this is that stemming is almost always useful when making a search, and adds little to the time required to make a search. Unlike some other search engines, dtSearch applies stemming at search time, there is no need to build indexes specifically to apply stemming and no need to build separate indexes for each language in use.

The problem

With the stemming option selected dtSearch will find plurals and many other word variations; for example a search on print will also find prints, printers, printing, printed, printable. However, if you are searching documents written in other languages, the English stemming rules will cause you to miss many word variations which do not occur in English (e.g. verb and noun changes with gender), and you may find that words which are unrelated are found in error.

Furthermore, the English noise word list, which is designed to remove unwanted English words from your index to keep the index size small, is not suitable for other languages; your indexes may contain many words which will not be useful in searches and which will add to the size of your indexes.

The solution

Use language specific files in place of the default US English files. These are supplied in the form of Language Extension Packs which contain files for many languages. All files are in Unicode format. List of languages supported.

The LEP500 Language Extension Pack license includes the "Russian Plus" stemming rules; these combine Cyrillic and Latin rules to enable improved search recall in document collections containing Russian and another language. The language pairs supported are:

Russian plus Czech
Russian plus Estonian
Russian plus Finnish
Russian plus German
Russian plus Greek
Russian plus Hungarian
Russian plus Latvian
Russian plus Lithuanian
Russian plus Polish
Russian plus Slovak
Russian plus Swedish

LEP500 Language Extension Pack can also be licensed for large volume use or wider distribution in your own application please ask for developer licensing options.