meilisearch

Meilisearch v1.50 revamps the Dynamic Search Rules, adds support for federated document fetch in sharded configurations, among other improvements

Breaking changes

This release introduces breaking changes for users using some experimental features

dynamicSearchRules experimental feature

Request type changes

1. priority has been replaced with precedence, which better reflects the behavior (lower precedence means the rule is applied first)
2. conditions has been modified from an array to an object with two fields: "query" of type QueryCondition and "time" of time TimeCondition
3. New type QueryCondition that contains the fields isEmpty (as previously) and words instead of contains (same type)
4. It is now possible to pass isEmpty: false with words in a QueryCondition. Passing isEmpty:true with words still results in a synchronous error.
5. New type TimeCondition with fields start and end (unchanged semantics from previous type).
6. When specifying the selector of an Action, it is now mandatory to specify an id. Previously, it was optional, but the action would never trigger.
7. When listing rules with POST /dynamic-search-rules, filter.attributePatterns has been replaced with filter.query, an optional string that searches in rule description and conditions.query.words.
8. When calling DELETE /dynamic-search-rules/{:ruleUid} or PATCH /dynamic-search-rules/{:ruleUid} in a sharded configuration, endpoint will not return a HTTP 400 error if called on a follower remote rather than on the leader.

Response changes

1. PATCH /dynamic-search-rules/{:ruleUid} and DELETE /dynamic-search-rules/{:ruleUid} now register an asynchronous task..
2. The response is modified to return the registered task instead of the modified dynamic search rule.
3. HTTP 404 is no longer returned if the {:ruleUid} portion of the URL refers to a rule that doesn't exist. This is because rules are processed asynchronously, and is consistent with the behavior of DELETE /indexes/{:indexUid}/documents/{:docId} for {:docId}

network experimental feature

The default behavior for users using the network experimental feature with sharding configured (leader not null) will change on the following routes:

• GET indexes/:uid/documents
• GET indexes/:uid/documents/:document_id
• POST indexes/:uid/documents/fetch

Meilisearch will now fetch the documents from all the shards and not only on the local machine when processing the request.
To keep the same behavior as before, users will have to set useNetwork to false when making their request.

🌈 Improvements

Scaling up the Dynamic Search Rules

• Dynamic search rules scale up to 75K rules without any impact on the search
• The API of Dynamic Search Rules has been simplified
  • It is harder to send conditions that will result in the rules never activating
• This also unlocks future improvements such as filter activation conditions for search rules

Additions

• Add a new DELETE /dynamic-search-rule route that deletes all the DSRs
• Add the concept of "DSR fuel" that determines how much energy is spent resolving DSR during a search. The fuel is initialized with some default variables that can be overridden using environment variables:
  • MEILI_EXPERIMENTAL_DSR_FUEL_MAX_COUNTED_WORDS: max number of words considered inside of a search query for the purpose of finding conditions.query.words constraints. Defaults to 10, max value is 255
  • MEILI_EXPERIMENTAL_DSR_FUEL_MAX_ACTIVE_RULES: max number of active rules whose actions are evaluated. Defaults to 1000, max value is 4294967295
  • MEILI_EXPERIMENTAL_DSR_FUEL_MAX_PIN_ACTIONS: max number of pin actions that are applied. Defaults to 100, max value is 4294967295
  • MEILI_EXPERIMENTAL_DSR_FUEL_WORD_FUEL: max number of constraint combinations that are evaluated for the purpose of finding conditions.query.words constraints. Defaults to 4096, max value is 4294967295

By @dureuill in https://github.com/meilisearch/meilisearch/pull/6484 and https://github.com/meilisearch/meilisearch/pull/6506

Behavior changes

• The conditions.query.words behaves differently from query.contains: previously, a rule would match if its conditions query.contains would be substrings of q in the search query in the sense of str::contains. Now, a rule matches if all the words in conditions.query.words appear in q (after normalization). For q = hero super, query.contains = super hero would not match, whereas conditions.query.words = super hero does now match. This behavior is more in line with regular search, and allows improving performance.
• Dynamic search rules are now replicated from the leader to its follower, when in a sharded configuration

Federated document fetch routes

GET indexes/:uid/documents, GET indexes/:uid/documents/:document_id and POST indexes/:uid/documents/fetch will now fetch the documents from all the shards in the configured network.

Moreover, a new useNetwork parameter is available to activate or deactivate the usage of the network.

By @ManyTheFish in https://github.com/meilisearch/meilisearch/pull/6495

Support partial wildcards when requesting facets

The facets parameter in search and federated search now supports more wildcards. Previously, only the single wildcard "*" was supported, requesting all filterable fields.

Now, patterns containing * are supported with the same matching rules as in filterableAttributes.attributePatterns and localizedAttributes.attributePatterns, such as dogs.*, which will add to the facet distribution all filterable fields that match the pattern (such as dogs.intel, dogs.kefir, etc.).

By @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6497

🦋 Fixes

Fix migration from v1.48 and earlier

Migration via --experimental-dumpless-upgrade would fail in some cases in v1.49, when trying to migrate synonyms that contained no words (empty synonyms, or containing only separator tokens such as &).

Such synonyms are now ignored during migration, avoiding the issue.

By @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6501

Fix filter memory consumption in some cases

In some conditions, the memory consumption of filters would increase quadratically with the length of the filter. This is now resolved for these cases.

By @ManyTheFish in https://github.com/meilisearch/meilisearch/pull/6509

No longer reject some correctly-escaped filters

Fix a bug where some filters containing escaped characters (such as \) would cause search requests to fail with invalid_search_filter

By @dureuill in https://github.com/meilisearch/meilisearch/pull/6499

More fault-tolerant S3 snapshots

Potentially fix an issue when sending a request to AWS S3 to create a new multipart upload, ensuring we resend the request if it fails.

By @Kerollmops in https://github.com/meilisearch/meilisearch/pull/6494

🔩 Miscellaneous changes

• Add missing route descriptions for documentation by @curquiza in https://github.com/meilisearch/meilisearch/pull/6500
• Make the prototype docs clearer by @curquiza in https://github.com/meilisearch/meilisearch/pull/6503
• Improve maintenability by simplifying partitioning step by @ManyTheFish in https://github.com/meilisearch/meilisearch/pull/6511
• Fix frequently failing tests on Windows by @dureuill in https://github.com/meilisearch/meilisearch/pull/6520

Full Changelog: https://github.com/meilisearch/meilisearch/compare/v1.49.0...v1.50.0