Fixes 🦋

Better search handling under heavy loads

All of the next PR should make meilisearch behave better under heavy loads:

• Only spawn one search queue in actix-web by @irevoire in #4893
• Make sure the index scheduler never stops running by @irevoire in #4896
• Explicitly drop the search permits by @irevoire in #4898
• Stop trying to process searches after one minute by @irevoire in #4899

Speed improvement 🐎

We made the autobatching of the document deletion with the document deletion by filter possible which should uncklog the task queue of the people using these two operations heavily.
Meilisearch still cannot autobatch the document deletion by filter and the document addition, though.

• Autobatch document deletion by filter by @irevoire in #4901
• Do not fail the whole batch when a single document deletion by filter fails by @irevoire in #4905

Full Changelog: v1.10.0...v1.10.1

Fixes 🪲

• Return an empty list of embeddings for embedders that have no document for an embedder. by @dureuill in #4889

This fixes an issue where dumps created for indexes with:

1. A user-provided embedder
2. At least one documents that opt-out of vectors for that user-provided embedder

would fail to import correctly.

Upgrade path to v1.10.0 🚀

If you are a Cloud user affected by the above issue, please contact customer support so we perform the upgrade for you.

If you are an OSS user affected by the above, perform the following operations:

1. Upgrade from v1.9.0 to v1.9.1 without using a dump
2. Upgrade to v1.10.0 using a dump created from v1.9.1

Full Changelog

v1.10.0 release changelogs

Meilisearch v1.10 introduces federated search. This innovative feature allows you to receive a single list of results for multi-search requests. v1.10 also includes a setting to manually define which language or languages are present in your documents, and two new new experimental features: the CONTAINS filter operator and the ability to update a subset of your dataset with a function.

🧰 All official Meilisearch integrations (including SDKs, clients, and other tools) are compatible with this Meilisearch release. Integration deployment happens between 4 to 48 hours after a new version becomes available.

Some SDKs might not include all new features. Consult the project repository for detailed information. Is a feature you need missing from your chosen SDK? Create an issue letting us know you need it, or, for open-source karma points, open a PR implementing it (we'll love you for that ❤️).

New features and updates 🔥

Federated search

Use the new federation setting of the /multi-search route to return a single search result object:

curl \
  -X POST 'https://localhost:7700/multi-search' \
  -H 'Content-Type: application/json' \
  --data-binary '{
    "federation": {
      "offset": 5,
      "limit": 10
    }
    "queries": [
      {
        "q": "Batman",
        "indexUid": "movies"
      },
      {
        "q": "Batman",
        "indexUid": "comics"
      }
    ]
  }'

Response:

{
  "hits": [
    {
      "id": 42,
      "title": "Batman returns",
      "overview": "..",
      "_federation": {
        "indexUid": "movies",
        "queriesPosition": 0
      }
    },
    {
      "comicsId": "batman-killing-joke",
      "description": "..",
      "title": "Batman: the killing joke",
      "_federation": {
        "indexUid": "comics",
        "queriesPosition": 1
      }
    },
    …
 ],
  processingTimeMs: 0,
  limit: 20,
  offset: 0,
  estimatedTotalHits: 2,
  semanticHitCount: 0,
}

When performing a federated search, Meilisearch merges the results coming from different sources in descending ranking score order.

If federation is empty ({}), Meilisearch sets offset and limit to 0 and 20 respectively.

If federation is null or missing, multi-search returns one list of search result objects for each index.

Federated results relevancy

When performing federated searches, use federationOptions in the request's queries array to configure the relevancy and the weight of each index:

curl \
 -X POST 'https://localhost:7700/multi-search' \
 -H 'Content-Type: application/json' \
 --data-binary '{
  "federation": {},
  "queries": [
    {
      "q": "apple red",
      "indexUid": "fruits",
      "filter": "BOOSTED = true",
      "_showRankingScore": true,
      "federationOptions": {
        "weight": 3.0
      }
    },
    {
      "q": "apple red",
      "indexUid": "fruits",
      "_showRankingScore": true,
    }
  ]
}'

federationOptions must be an object. It supports a single field, weight, which must be a positive floating-point number:

• if weight < 1.0, results from this index are less likely to appear in the results
• if weight > 1.0, results from this index are more likely to appear in the results
• if not specified, weight defaults to 1.0

📖 Consult the usage page for more information about the merge algorithm.

Done by @dureuill in #4769.

Experimental: CONTAINS filter operator

Enable the containsFilter experimental feature to use the CONTAINS filter operator:

curl \
  -X PATCH 'https://localhost:7700/experimental-features/' \
  -H 'Content-Type: application/json' \
  --data-binary '{
    "containsFilter": true
  }'

CONTAINS filters results containing partial matches to the specified string, similar to a SQL LIKE:

curl \
  -X POST https://localhost:7700/indexes/movies/search \
  -H 'Content-Type: application/json' \
  --data-binary '{
    "q": "super hero",
    "filter": "synopsis CONTAINS spider"
  }'

🗣️ This is an experimental feature, and we need your help to improve it! Share your thoughts and feedback on this GitHub discussion.

Done by @irevoire in #4804.

Language settings

Use the new localizedAttributes index setting and the locales search parameter to explicitly set the languages used in document fields and the search query itself. This is particularly useful for <=v1.9 users who have to occasionally resort to alternative Meilisearch images due to language auto-detect issues in Swedish and Japanese datasets.

Done by @ManyTheFish in #4819.

Set language during indexing with localizedAttributes

Use the newly introduced localizedAttributes setting to explicitly declare which languages correspond to which document fields:

curl \
  -X PATCH 'https://localhost:7700/indexes/movies/settings' \
  -H 'Content-Type: application/json' \
  --data-binary '{
    "localizedAttributes": [
      {"locales": ["jpn"], "attributePatterns": ["*_ja"]},
      {"locales": ["eng"], "attributePatterns": ["*_en"]},
      {"locales": ["cmn"], "attributePatterns": ["*_zh"]},
      {"locales": ["fra", "ita"], "attributePatterns": ["latin.*"]},
      {"locales": [], "attributePatterns": ["*"]}
    ]
  }'

locales is a list of ISO-639-3 language codes to assign to a pattern. The currently supported languages are: epo, eng, rus, cmn, spa, por, ita, ben, fra, deu, ukr, kat, ara, hin, jpn, heb, yid, pol, amh, jav, kor, nob, dan, swe, fin, tur, nld, hun, ces, ell, bul, bel, mar, kan, ron, slv, hrv, srp, mkd, lit, lav, est, tam, vie, urd, tha, guj, uzb, pan, aze, ind, tel, pes, mal, ori, mya, nep, sin, khm, tuk, aka, zul, sna, afr, lat, slk, cat, tgl, hye.

attributePattern is a pattern that can start or end with a * to match one or several attributes.

If an attribute matches several rules, only the first rule in the list will be applied. If the locales list is empty, then Meilisearch is allowed to auto-detect any language in the matching attributes.

These rules are applied to the searchableAttributes, the filterableAttributes, and the sortableAttributes.

Set language at search time with locales

The /search route accepts a new parameter, locales. Use it to define the language used in the current query:

curl \
  -X POST https://localhost:7700/indexes/movies/search \
  -H 'Content-Type: application/json' \
  --data-binary '{
    "q": "進撃の巨人",
    "locales": ["jpn"]
  }'

The locales parameter overrides eventual locales in the index settings.

Experimental: Edit documents with a Rhai function

Use a Rhai function to edit documents in your database directly from Meilisearch:

First, activate the experimental feature:

curl \
  -X PATCH 'https://localhost:7700/experimental-features/' \
  -H 'Content-Type: application/json' \
  --data-binary '{
    "editDocumentsByFunction": true
  }'

Then query the /documents/edit route with the editing function:

curl https://localhost:7700/indexes/movies/documents/edit \
  -H 'content-type: application/json' \
  -d '{
   "function": "doc.title = `✨ ${doc.title.to_upper()} ✨`",
   "filter": "id > 3000"
  }'

/documents/edit accepts three parameters in its payload: function, filter, and context.

function must be a string with a Rhai function. filter must be a filter expression.. context must be an object with data you want to make available for the editing function.

📖 More information here.

🗣️ This is an experimental feature and we need your help to improve it! Share your thoughts and feedback on this GitHub discussion.

Done by @Kerollmops in #4626.

Experimental AI-powered search: quality of life improvements

For the purpose of future stabilization of the feature, we are applying changes and quality-of-life improvements.

Done by @dureuill in #4801, #4815, #4818, #4822.

⚠️ Breaking changes: Changing the parameters of the REST API

The old parameters of the REST API are too numerous and confusing.

Removed parameters: query , inputField, inputType, pathToEmbeddings and embeddingObject.
Replaced by:

• request : A JSON value that represents the request made by Meilisearch to the remote embedder. The text to embed must be replaced by the placeholder value “{{text}}”.
• response: A JSON value that represents a fragment of the response made by the remote embedder to Meilisearch. The embedding must be replaced by the placeholder value "{{embedding}}".

Before:

// v1.10 version ✅
{
  "source": "rest",
  "url": "https://localhost:10006",
  "request": {
    "model": "minillm",
    "prompt": "{{text}}"
  },
  "response": {
    "embedding": "{{embedding}}"
  }
}

// v1.9 version ❌
{
  "source": "rest",
  "url": "https://localhost:10006",
  "query": {
    "model": "minillm",
  },
  "inputField": ["prompt"],
  "inputType": "text",
  "embeddingObject": ["embedding"]
}

What's Changed

• also intersect the universe for searchOnAttributes by @dureuill in #4858
• Don't remove facet value when multiple original values map to the same normalized value by @dureuill in #4864
• Fix perf regression facet strings by @dureuill in #4845
• Infer locales from index settings by @ManyTheFish in #4881

⚠️ Since this is a release candidate (RC), we do NOT recommend using it in a production environment. Is something not working as expected? We welcome bug reports and feedback about new features.

What's Changed

Bugs fixes 🐞

Use a fixed date format regardless of features

If you were compiling Meilisearch yourself, we found out that using one version compiled from the root of the repository and then one from the meilisearch directory could cause database corruption.

Fixed by @dureuill in #4850

Fix edit by function deletion of documents

When deleting documents from the new edit by function route, if you had more than one thread, some documents would randomly not be deleted.

Fixed by @dureuill in #4853

Maintainance 🐢

• Add OpenAI tests by @dureuill in #4846

Full Changelog: v1.10.0-rc.1...v1.10.0-rc.2

⚠️ Since this is a release candidate (RC), we do NOT recommend using it in a production environment. Is something not working as expected? We welcome bug reports and feedback about new features.

Bug fixes 🪲

• Add missing PUT and GET /indexes/{indexUid}/settings/localized-attributes routes by @dureuill in #4836

Other improvements

• Errors emitted at the main level of the Meilisearch binary are now logged with level ERROR by @dureuill in #4835

Full Changelog

With Meilisearch v1.10 we keep innovating by introducing the really demanded federated search! You can now apply multi-search requests and get one single list of results 🎉. This version also includes a setting to define your index languages (even if multiple languages are in your documents!), and new experimental features like the CONTAINS operator and the ability to update a subset of your dataset by using a simple function.

New features and updates 🔥

Federated search

By using the POST /multi-search endpoint, you can now return a single search result object, whose list of hits is built by merging the hits coming from all the queries in descending ranking score order.

curl \
 -X POST 'https://localhost:7700/multi-search' \
 -H 'Content-Type: application/json' \
 --data-binary '{
  "federation": {
    "offset": 5,
    "limit": 10,
 }
  "queries": [
 {
      "q": "Batman",
      "indexUid": "movies"
 },
 {
      "q": "Batman",
      "indexUid": "comics"
 }
 ]
}'

Response:

{
  "hits": [
    {
      "id": 42,
      "title": "Batman returns",
      "overview": "..",
      "_federation": {
        "indexUid": "movies",
        "queriesPosition": 0
      }
    },
    {
      "comicsId": "batman-killing-joke",
      "description": "..",
      "title": "Batman: the killing joke",
      "_federation": {
        "indexUid": "comics",
        "queriesPosition": 1
      }
    },
    ...
 ],
  processingTimeMs: 0,
  limit: 20,
  offset: 0,
  estimatedTotalHits: 2,
  semanticHitCount: 0,
}

If federation is empty ({}) default values of offset and limit are used, so respectively 0 and 20.
If federation is null or missing, a classic multi-search will be applied, so a list of search result objects for each index will be returned.

To customize the relevancy and the weight applied to each index in the search result, use the federationOptions parameter in your request:

curl \
 -X POST 'https://localhost:7700/multi-search' \
 -H 'Content-Type: application/json' \
 --data-binary '{
  "federation": {},
  "queries": [
    {
      "q": "apple red",
      "indexUid": "fruits",
      "filter": "BOOSTED = true",
      "_showRankingScore": true,
      "federationOptions": {
        "weight": 3.0
      }
    },
    {
      "q": "apple red",
      "indexUid": "fruits",
      "_showRankingScore": true,
    }
  ]
}'

weight must be positive (>=0)

• if < 1.0, the hits from this query are less likely to appear in the results.
• if > 1.0, the hits from this query are more likely to appear in the results.
• if missing, the default value is applied (1.0)

📖 More information about the merge algorithm on the here.

Done by @dureuill in #4769.

Experimental: CONTAINS filter operator

Enabling the experimental feature will make a new CONTAINS operator available while filtering on strings.
This is similar to the SQL LIKE operator used with %.

Activate the experimental feature:

curl \
 -X PATCH 'https://localhost:7700/experimental-features/' \
  -H 'Content-Type: application/json' \
 --data-binary '{
    "containsFilter": true
 }'

Use the newly introduced CONTAINS operator:

curl \
 -X POST https://localhost:7700/indexes/movies/search \
  -H 'Content-Type: application/json' \
 --data-binary '{
      "q": "super hero",
      "filter": "synopsis NOT CONTAINS spider"
 }'

🗣️ This is an experimental feature, and we need your help to improve it! Share your thoughts and feedback on this GitHub discussion.

Done by @irevoire in #4804.

Languages settings

You can now set up the language of your index in your settings and during the search. This will prevent users from using alternative Meilisearch images we were separately created until now (like for Swedish and Japanese)

Done by @ManyTheFish in #4819.

Index settings

Use the newly introduced localizedAttributes setting (here is an example of handling multi-language documents):

curl \
  -X PATCH 'https://localhost:7700/indexes/movies/settings' \
  -H 'Content-Type: application/json' \
  --data-binary '{
    "localizedAttributes": [
      {"locales": ["jpn"], "attributePatterns": ["*_ja"]},
      {"locales": ["eng"], "attributePatterns": ["*_en"]},
      {"locales": ["cmn"], "attributePatterns": ["*_zh"]},
      {"locales": ["fra", "ita"], "attributePatterns": ["latin.*"]},
      {"locales": [], "attributePatterns": ["*"]}
    ]
}'

locales is a list of language codes to assign to a pattern, the supported codes are: epo, eng, rus, cmn, spa, por, ita, ben, fra, deu, ukr, kat, ara, hin, jpn, heb, yid, pol, amh, jav, kor, nob, dan, swe, fin, tur, nld, hun, ces, ell, bul, bel, mar, kan, ron, slv, hrv, srp, mkd, lit, lav, est, tam, vie, urd, tha, guj, uzb, pan, aze, ind, tel, pes, mal, ori, mya, nep, sin, khm, tuk, aka, zul, sna, afr, lat, slk, cat, tgl, hye.

attributePattern is a pattern that can start or end with a * to match one or several attributes.

"locales": [], "attributePatterns": ["*"] means the is the default rule.

Notes:

• if an attribute matches several rules, only the first rule in the list will be applied
• if the locales list is empty, then Meilisearch is allowed to auto-detect any language in the matching attributes
• These rules are applied to the searchableAttributes, the filterableAttributes, and the sortableAttributes.

At search time

The search route accepts a new parameter, locales allowing the end-user to define the language used in the current query:

curl \
 -X POST https://localhost:7700/indexes/movies/search \
  -H 'Content-Type: application/json' \
 --data-binary '{
  "q": "進撃の巨人",
  "locales": ["jpn"]
}'

The locales parameter overrides eventual locales in the index settings.

Experimental: edit documents by using a function

You can edit documents by executing a Rhai function on all the documents of your database or a subset of them that you can select by a Meilisearch filter.

Activate the experimental feature:

curl \
 -X PATCH 'https://localhost:7700/experimental-features/' \
  -H 'Content-Type: application/json' \
 --data-binary '{
    "editDocumentsByFunction": true
 }'

By indexing this movies dataset, you can run the following Rhai function on all of them. This function will uppercase the titles of the movies with an id > 3000 and add sparkles around it. Rhai templating syntax is applied here:

curl https://localhost:7700/indexes/movies/documents/edit \
 -H 'content-type: application/json' \
  -d '{
 "filter": "id > 3000",
 "function": "doc.title = `✨ ${doc.title.to_upper()} ✨`"
 }'

📖 More information here.

🗣️ This is an experimental feature and we need your help to improve it! Share your thoughts and feedback on this GitHub discussion.

Done by @Kerollmops in #4626.

Experimental AI-powered search: quality of life improvements

For the purpose of future stabilization of the feature, we are applying changes and quality-of-life improvements.

Done by @dureuill in #4801, #4815, #4818, #4822.

⚠️ Breaking changes - changing the parameters of the REST API

The old parameters of the REST API are too numerous and confusing.

Removed parameters: query , inputField, inputType, pathToEmbeddings and embeddingObject.
Replaced by

• request : A JSON value that represents the request made by Meilisearch to the remote embedder. The text to embed must be replaced by the placeholder value “{{text}}”.
• response: A JSON value that represents a fragment of the response made by the remote embedder to Meilisearch. The embedding must be replaced by the placeholder value "{{embedding}}".

Before:

// v1.9 (old) version
{
  "source": "rest",
  "url": "https://localhost:10006",
  "query": {
    "model": "minillm",
 },
  "inputField": ["prompt"],
  "inputType": "text",
  "embeddingObject": ["embedding"]
}

// v1.10 (new) version
{
  "source": "rest",
  "url": "https://localhost:10006",
  "request": {
    "model": "minillm",
    "prompt": "{{text}}"
 },
  "response": {
    "embedding": "{{embedding}}"
 }
}

Upgrade procedure (Cloud):

1. Remove any embedder with source "rest"
2. Follow the usual steps described [here in the documentation](https://www.meilisearch.com/docs/lear...

Improvements

• Generate vectors in dumps by @dureuill in #4796
  • this version provides an upgrade path to v1.9 without regenerating embeddings for autogenerating embedders.
  • to upgrade from v1.8.x to v1.9.0 without regenerating embeddings, please follow the following procedure:
    1. Upgrade from v1.8.x to v1.8.4 by simply restarting Meilisearch v1.8.4 on you v1.8.x DB. No dump is necessary at this step
    2. Create a dump.
    3. Import the created dump in a Meilisearch v1.9

Full Changelog

Meilisearch v1.9 includes performance improvements for hybrid search and the addition/updating of settings. This version benefits from multiple requested features, such as the new frequency matching strategy and the ability to retrieve similar documents.

🧰 All official Meilisearch integrations (including SDKs, clients, and other tools) are compatible with this Meilisearch release. Integration deployment happens between 4 to 48 hours after a new version becomes available.

Some SDKs might not include all new features. Consult the project repository for detailed information. Is a feature you need missing from your chosen SDK? Create an issue letting us know you need it, or, for open-source karma points, open a PR implementing it (we'll love you for that ❤️).

New features and updates 🔥

Hybrid search updates

This release introduces multiple hybrid search updates.

Done by @dureuill and @irevoire in #4633 and #4649

⚠️ Breaking change: Empty _vectors.embedder arrays

Empty _vectors.embedder arrays are now interpreted as having no vector embedding.

Before v1.9, Meilisearch interpreted these as a single embedding of dimension 0. This change follows user feedback that the previous behavior was unexpected and unhelpful.

⚠️ Breaking change: _vectors field no longer present in search results

When the experimental vectorStore feature is enabled, Meilisearch no longer includes _vectors in returned search results by default. This will considerably improve performance.

Use the new retrieveVectors search parameter to display the _vectors field:

curl \
  -X POST 'https://localhost:7700/indexes/INDEX_NAME/search' \
  -H 'Content-Type: application/json' \
  --data-binary '{
    "q": "SEARCH QUERY",
    "retrieveVectors": true
  }'

⚠️ Breaking change: Meilisearch no longer preserves the exact representation of embeddings appearing in _vectors

In order to save storage and run faster, Meilisearch is no longer storing your vector "as-is". Meilisearch now returns the float in a canonicalized representation rather than the user-provided representation.

For example, 3 may be represented as 3.0

Document _vectors accepts object values

The document _vectors field now accepts objects in addition to embedding arrays:

{
  "id": 42,
  "_vectors": {
    "default": [0.1, 0.2 ],
    "text": {
      "embeddings": [[0.1, 0.2, 0.3], [0.4, 0.5, 0.6]],
      "regenerate": false
    },
    "translation": {
      "embeddings": [0.1, 0.2, 0.3, 0.4],
      "regenerate": true
    }
  }
}

The _vectors object may contain two fields: embeddings and regenerate.

If present, embeddings will replace this document's embeddings.

regenerate must be either true or false. If regenerate: true, Meilisearch will overwrite the document embeddings each time the document is updated in the future. If regenerate: false, Meilisearch will keep the last provided or generated embeddings even if the document is updated in the future.

This change allows importing embeddings to autoembedders as a one-shot process, by setting them as regenerate: true. This change also ensures embeddings are not regenerated when importing a dump created with Meilisearch v1.9.

Meilisearch v1.9.0 also improves performance when indexing and using hybrid search, avoiding useless operations and optimizing the important ones.

New feature: Ranking score threshold

Use rankingScoreThreshold to exclude search results with low ranking scores:

curl \
 -X POST 'https://localhost:7700/indexes/movies/search' \
 -H 'Content-Type: application/json' \
 --data-binary '{
    "q": "Badman dark returns 1",
    "showRankingScore": true,
    "limit": 5,
    "rankingScoreThreshold": 0.2
 }'

Meilisearch does not return any documents below the configured threshold. Excluded results do not count towards estimatedTotalHits, totalHits, and facet distribution.

⚠️ For performance reasons, if the number of documents above rankingScoreThreshold is higher than limit, Meilisearch does not evaluate the ranking score of the remaining documents. Results ranking below the threshold are not immediately removed from the set of candidates. In this case, Meilisearch may overestimate the count of estimatedTotalHits, totalHits and facet distribution.

Done by @dureuill in #4666

New feature: Get similar documents endpoint

This release introduces a new AI-powered search feature allowing you to send a document to Meilisearch and receive a list of similar documents in return.

Use the /indexes/{indexUid}/similar endpoint to query Meilisearch for related documents:

curl \
  -X POST /indexes/:indexUid/similar
  -H 'Content-Type: application/json' \
  --data-binary '{
    "id": "23",
    "offset": 0,
    "limit": 2,
    "filter": "release_date > 1521763199",
    "embedder": "default",
    "attributesToRetrieve": [],
    "showRankingScore": false,
    "showRankingScoreDetails": false
  }'

• id: string indicating the document needing similar results, required
• offset: number of results to skip when paginating, optional, defaults to 0
• limit: number of results to display, optional, defaults to 20
• filter: string with a filter expression Meilisearch should apply to the results, optional, defaults to null
• embedder: string indicating the embedder Meilisearch should use to retrieve similar documents, optional, defaults to "default"
• attributesToRetrieve: array of strings indicating which fields Meilisearch will include in the response, optional, defaults to ["*"]
• showRankingScore: boolean indicating if results should include ranking score information, optional, defaults to false
• showRankingScoreDetails: boolean indicating if results should include detailed ranking score information, optional, defaults to false
• rankingScoreThreshold: Excludes search results with a ranking score lower than the defined number, optional, defaults to null.

/indexes/{indexUid}/similar supports GET and POST routes. Use URL query parameters to configure your GET request, or include your parameters in the request body if using the POST route. Both offer identical functionality.

Done by @dureuill in #4647

New feature: frequency matching strategy

This release adds a new matching strategy, frequency. Use it to prioritize results containing the least frequent query terms:

curl \
 -X POST 'https://localhost:7700/indexes/{index_uid}/search' \
 -H 'Content-Type: application/json' \
 --data-binary '{
    "q": "cheval blanc",
    "matchingStrategy": "frequency"
 }'

Done by @ManyTheFish in #4667

Set distinctAttribute at search time

This release introduces a new search parameter: distinct which you can use to specify the distinct attribute at search time:

curl \
 -X POST 'https://localhost:7700/indexes/{index_uid}/search' \
 -H 'Content-Type: application/json' \
 --data-binary '{
    "q": "kefir le double poney",
    "distinct": "book.isbn"
 }'

If a distinct attribute is already defined in the settings it'll be ignored in favor of the one defined at search time.

Done by @Kerollmops in #4693

Improve indexing speed when updating/adding settings

Meilisearch now limits operations when importing settings by avoiding unnecessary writing operations in its internal database and reducing disk usage.

Additionally, when changing embedding settings, Meilisearch will now only regenerate the embeddings for the embedders whose settings have been modified, instead of for all embedders. When only the documentTemplate is modified, embeddings will only be regenerated for documents where this modification leads to a different text to embed.

Done by @irevoire, @Kerollmops, @ManyTheFish and @dureuill in #4646, #4680, #4631 and #4649

Other improvements

• Speed up filter ANDs operations during the search (#4682) @Kerollmops
• Speed up facet distribution during the search (#4713) @Kerollmops
• Improve language support (#4684) @ManyTheFish @Soham1803 @mosuka @tkhshtsh0917
  • Add new normalizer to normalize œ to oe and æ to ae
  • Fix chinese-normalization-pinyin feature flag compilation
• Prometheus experimental feature: Use HTTP path pattern instead of full path in metrics (#4619) @gh2k
• ⚠️ Remove exportPuffinReport experimental feature. Use logs routes and logs modes instead (#4655) @Kerollmops

Fixes 🐞

• All fields now have the same impact on relevancy when searchableAttributes: ["*"]. Consult the GitHub issue for a detailed breakdown of these changes (#4631) @irevoire
• Fix searchableAttributes behavior when handling nested fields. Consult the GitHub issue for more information (#4631) @irevoire
• Fix security issue in dependency: bump Rustls to non-vulnerable versions (#4622) @Kerollmops
• Reset other embedding settings when changing the source of an embedder. This prevents misleading error messages when configuring the embedders (#4649) @dureuill
• Fix panic in hybrid search when removing all embedders from the DB (#4715) @irevoire
• Hybrid search now respects the offset and limit parameters when returning keyword results earl...

⚠️ Since this is a release candidate (RC), we do NOT recommend using it in a production environment. Is something not working as expected? We welcome bug reports and feedback about new features.

Bug fixes 🪲

• Fix hybrid search limit offset by @dureuill in #4746
• Make embeddings optional and improve error message for regenerate by @irevoire in #4740

❤️ Thanks to @inventor123 for first reporting #4745

Full Changelog