GroupDocs.Search for .NET 26.10 Release Notes

Release Highlights

The table below sums up the key gains of this release compared with the previous version. Most gains grow with the size of the index and the number of its segments; the details are given in the Public API and Backward Incompatible Changes section.

Major Features

GroupDocs.Search for .NET 26.10 is a major release. The index format has been redesigned, and most operations on an index now work faster and consume less memory. The changes are grouped by area below.

Index storage and reliability

• The index size on disk now grows in proportion to the indexed data: the larger the index and the more segments it has, the greater the saving.
• The amount of RAM required by a loaded index has been reduced by half.
• Opening an index is 2 to 4 times faster.
• Index data are protected with checksums, and all changes to an index are applied atomically.
• Indexes created by versions 22.7–26.8 are converted to the new index format without re-indexing.

Search

• In indexes with many segments, search for a word is up to 2 times faster, and search for rare words is 10 to 40 times faster.
• In such indexes, fuzzy search is 2.5 to 5 times faster, and wildcard search is 2.5 to 3 times faster.
• In search by chunks, wildcard, regular expression, numeric range and fuzzy search are 7 to 150 times faster, and only the index segments that contain the query terms are searched; date range search is 2 to 27 times faster.
• Search uses as many threads as there are processor cores by default, and the number of search threads can be changed at any time.
• Reverse image search is up to 1.8 times faster; image search data take 2.9 times less memory, are loaded 6.6 times faster and are saved 100 times faster.
• Retrieving the stored text of a document is up to 15 times faster.
• Searches and index-modifying operations no longer block each other.

Indexing and index maintenance

• The indexing operation consumes about half as much peak RAM.
• Deleting documents is hundreds of times faster, requires no additional disk space and uses 4 times less memory; updating an index is accelerated accordingly.
• Index optimization is 4 times faster and uses 4 times less memory.

Text processing

• Letter case is handled in the same way on .NET Framework and .NET.
• Two new index settings are enabled by default: unpaired surrogate characters are replaced with the Unicode replacement character (U+FFFD) instead of being removed, and terms that consist only of blended characters are not indexed.
• Term dictionaries of an index that contains documents can no longer be changed, and indexes that process text differently can no longer be merged; previously both cases led to incomplete search results.

Upgrade Notes

Take the following changes into account when upgrading from a previous version.

• The index format has changed. Loading an index created by a previous version fails until the index version is updated with the IndexUpdater class. Indexes created by versions 22.7–26.8 are converted without re-indexing: the source documents are not required, the converted index is saved to a different folder, and the source index is not modified. For details, see Update index version.
• Term dictionaries are fixed once an index contains documents. The alphabet, tokenization dictionary, character replacements and stop words, as well as the UseStopWords and UseCharacterReplacements settings, have to be configured before the first document is indexed; a later attempt to change them throws InvalidOperationException.
• Incompatible indexes are not merged. Index.Merge throws InvalidOperationException before making any changes if the merged indexes split text into terms differently. A merge is also not performed when a merged index is busy or has failed; the ErrorOccurred event reports it.
• Search is multithreaded by default. With the default value of IndexSettings.SearchThreads, search uses as many threads as there are processor cores instead of one thread, and a value assigned after the index is created is no longer ignored. To limit the number of threads, set the property explicitly.
• Two new index settings are enabled by default. IndexSettings.ReplaceUnpairedSurrogates and IndexSettings.SkipBlendedOnlyTerms affect which terms are indexed and are fixed when an index is created. To keep the previous behavior, set them to false before creating an index.
• Deleted documents are physically removed during optimization. Deleted documents are excluded from search immediately and are physically removed when the index is optimized (Index.Optimize).
• The number of chunks has changed. A search by chunks no longer returns a chunk for every index segment, and the number of chunks can differ from search to search. Iterate over the chunks while SearchResult.NextChunkSearchToken is not null instead of relying on the number of segments.

Full List of Issues Covering all Changes in this Release

Public API and Backward Incompatible Changes

Implement the ability to change the number of search threads at any time

The IndexSettings.SearchThreads property can now be changed at any time after an index is created. Searches started after the change use the new number of threads, and the new value is saved in the index. Previously, a value assigned after the index was created was ignored.

Use cases

The following example demonstrates how to change the number of search threads of an existing index.

string indexFolder = @"c:\MyIndex\"; // Specify path to the index folder

Index index = new Index(indexFolder); // Opening an existing index

index.IndexSettings.SearchThreads = NumberOfThreads.Four; // Changing the number of threads used for the search

SearchResult result = index.Search("Einstein"); // Searching using the new number of threads

Implement an option to replace unpaired surrogate characters

The new IndexSettings.ReplaceUnpairedSurrogates property (enabled by default) replaces unpaired UTF-16 surrogate characters in document text with the Unicode replacement character (U+FFFD) instead of removing them, so the characters around them are no longer joined into a single term. The setting is fixed when an index is created.

Public API changes

Property Boolean ReplaceUnpairedSurrogates has been added to GroupDocs.Search.IndexSettings class.

Use cases

The following example demonstrates how to disable the replacement of unpaired surrogate characters when creating an index.

string indexFolder = @"c:\MyIndex\"; // Specify path to the index folder
string documentsFolder = @"c:\MyDocuments\"; // Specify the path to a folder containing documents to search

IndexSettings settings = new IndexSettings(); // Creating an index settings instance
settings.ReplaceUnpairedSurrogates = false; // Disabling the replacement of unpaired surrogate characters

Index index = new Index(indexFolder, settings); // Creating an index with the specified settings

index.Add(documentsFolder); // Synchronous indexing documents from the specified folder

Implement an option to skip terms consisting only of blended characters

The new IndexSettings.SkipBlendedOnlyTerms property (enabled by default) excludes from indexing the terms that consist only of blended characters (characters defined in the alphabet as both letters and separators), for example a standalone hyphen when the hyphen is a blended character. This reduces the number of meaningless terms in the index. The setting is fixed when an index is created.

Public API changes

Property Boolean SkipBlendedOnlyTerms has been added to GroupDocs.Search.IndexSettings class.

Use cases

The following example demonstrates how to keep the terms consisting only of blended characters in an index.

string indexFolder = @"c:\MyIndex\"; // Specify path to the index folder
string documentsFolder = @"c:\MyDocuments\"; // Specify the path to a folder containing documents to search

IndexSettings settings = new IndexSettings(); // Creating an index settings instance
settings.SkipBlendedOnlyTerms = false; // Disabling the skipping of terms consisting only of blended characters

Index index = new Index(indexFolder, settings); // Creating an index with the specified settings

index.Add(documentsFolder); // Synchronous indexing documents from the specified folder

Implement index version update to the new index format without reindexing the documents

IndexUpdater now converts indexes created by versions 22.7–26.8 to the new index format without re-indexing: the source documents do not have to be available. Indexed documents, their attributes and stored texts, image search data, index settings and dictionaries are preserved, and the source index is never modified.

Use cases

The following example demonstrates how to update an index of a previous version without reindexing the documents.

string sourceIndexFolder = @"c:\MyOldIndex\"; // Specify path to the index of a previous version
string targetIndexFolder = @"c:\MyNewIndex\"; // Specify path to the folder for the updated index

IndexUpdater updater = new IndexUpdater(); // Creating an index updater

if (updater.CanUpdateVersion(sourceIndexFolder)) // Checking whether the index version can be updated
{
    VersionUpdateResult result = updater.UpdateVersion(sourceIndexFolder, targetIndexFolder); // Updating the index version; the source index is not changed
}

Implement optimization of disk space consumption of an index

The index format has been redesigned. In previous versions, the size of an index grew faster than the amount of indexed data: the more documents were added, the larger the overhead. Now the index size grows in proportion to the indexed data, so the larger the index and the more segments it has, the greater the saving.

Implement optimization of RAM consumption of a loaded index

The amount of RAM required by a loaded index has been reduced by half.

Implement optimization of index loading speed

Opening an index is 2 to 4 times faster; the more segments the index has, the greater the gain.

Implement optimization of search speed in large indexes

Search no longer accesses every index segment for each search term: only the segments that contain the term are read. In indexes with many segments, search for a word is up to 2 times faster, and search for rare words is 10 to 40 times faster. The number of simultaneously open files no longer depends on the number of segments.

Implement optimization of fuzzy search

Fuzzy search is 2.5 to 5 times faster in indexes with many segments and uses all processor cores by default.

Implement optimization of wildcard search

Wildcard search is 2.5 to 3 times faster in indexes with many segments. Multithreaded wildcard search has been implemented.

Implement optimization of deleting and updating documents

Deleting documents no longer rewrites the index: deleted documents are excluded from search immediately and are physically removed when the index is optimized. As a result, deleting documents is hundreds of times faster, requires no additional disk space and uses 4 times less memory. Updating an index with changed or deleted documents is accelerated accordingly.

Implement optimization of the index optimization operation

Index optimization is 4 times faster and uses 4 times less memory. The memory consumption and the number of simultaneously open files no longer depend on the index size; a very large index is optimized into several segments of limited size.

Allow searching while the index is being modified

Searches and index-modifying operations (indexing, update, deletion, merging, optimization) no longer block each other. Each search works with a consistent snapshot of the index taken when the search starts.

Improve index reliability

Index data are protected with checksums, so damaged data are detected and reported. All changes to an index are applied atomically: if an operation is interrupted by an error, a power failure or termination of the process, the index remains in its previous consistent state, and an indexing operation interrupted by a failure is completed or rolled back the next time the index is opened. The CheckSegments method repairs more types of damage without losing documents.

Implement optimization of memory usage during indexing

Indexing no longer reserves memory in advance. Now memory grows with the amount of indexed data, and the collected data are kept in a compact form. The indexing operation now consumes about half as much peak RAM.

Implement optimization of retrieving document text

Retrieving the stored text of a document (Index.GetDocumentText) is up to 15 times faster.

Implement optimization of image search

Image search data take 2.9 times less memory, are loaded 6.6 times faster and are saved 100 times faster; reverse image search is up to 1.8 times faster.

Implement handling letter case identically on all platforms

Conversion of letters to lower case no longer depends on the .NET runtime: indexing and search handle letter case in the same way on .NET Framework and .NET, so search results do not depend on the platform.

Implement optimization of search by chunks

Search by chunks has been optimized. The following kinds of search have been improved (the time of returning all chunks of a search, measured on an index with 256 segments):

• Wildcard search: the words that match the pattern are found once and reused by all chunks of the search and by repeated searches with the same query; 7 to 26 times faster.
• Regular expression search: the words that match the expression are found once and reused in the same way; 75 to 110 times faster.
• Numeric range search: the numbers of the range are found once and reused in the same way; 9 to 85 times faster.
• Fuzzy search: the similar words are found once and reused in the same way; 30 to 150 times faster.
• Date range search: the numbers of the day, month and year are read once and reused for all dates of the range; 2 to 9 times faster, and 2 to 27 times faster without search by chunks.
• Search for words that occur in few segments: only the index segments that contain the query terms are searched, so fewer chunks are returned; for example, a word that occurs in 10 of 256 segments gives 10 chunks instead of 256 and is found 4.5 times faster.

To keep the found words between chunks, an index stored on disk can create temporary files in the index folder (normally up to 256 MB in total); they are deleted automatically.

Fix the default number of search threads

With the default value of IndexSettings.SearchThreads, search now uses as many threads as there are processor cores, as documented; previously only one thread was used on all platforms. Regular expression, numeric range, fuzzy and wildcard searches, spelling correction and image search are now parallelized by default: for example, a regular expression search is 1.7 times faster on a dual-core processor, and the gain grows with the number of cores.

Prevent inconsistent indexing after changing term dictionaries

The alphabet, tokenization dictionary, character replacements and stop words of an index, as well as the UseStopWords and UseCharacterReplacements settings, can now be changed only while the index contains no documents; an attempt to change them later throws InvalidOperationException. Previously such a change was accepted, and documents indexed before and after it were processed by different rules, which led to missed search results and incorrect highlighting.

Prevent merging indexes that process text differently

Index.Merge now throws InvalidOperationException before making any changes if the merged indexes split text into terms differently (alphabet, tokenization dictionary, stop words, character replacements or related settings). Previously such indexes were merged into an index that returned incomplete search results. A merge is also not performed when a merged index is busy or has failed; the ErrorOccurred event reports it.