Tasks and asynchronous operations

Meilisearch uses a task queue to handle asynchronous operations. This in-depth guide explains tasks, their uses, and how to manage them using Meilisearch’s API.

Many operations in Meilisearch are processed asynchronously. These API requests are not handled immediately. Instead, Meilisearch places them in a queue and processes them in the order they were received.

Which operations are asynchronous?

Every operation that might take a long time to be processed is handled asynchronously. Processing operations asynchronously allows Meilisearch to handle resource-intensive tasks without impacting search performance.

Currently, these are Meilisearch’s asynchronous operations:

• Creating an index
• Updating an index
• Swapping indexes
• Deleting an index
• Updating index settings
• Adding documents to an index
• Updating documents in an index
• Deleting documents from an index
• Canceling a task
• Deleting a task
• Creating a dump
• Creating snapshots

Understanding tasks

When an API request triggers an asynchronous process, Meilisearch creates a task and places it in a task queue.

Task objects

Tasks are objects containing information that allow you to track their progress and troubleshoot problems when things go wrong.

A task object includes data not present in the original request, such as when the request was enqueued, the type of request, and an error code when the task fails:

{
    "uid": 1,
    "indexUid": "movies",
    "status": "enqueued",
    "type": "documentAdditionOrUpdate",
    "canceledBy": null,
    "details": {
        "receivedDocuments": 67493,
        "indexedDocuments": null
    },
    "error": null,
    "duration": null,
    "enqueuedAt": "2021-08-10T14:29:17.000000Z",
    "startedAt": null,
    "finishedAt": null
}

For a comprehensive description of each task object field, consult the task API reference.

Summarized task objects

Every POST and PUT endpoint that triggers an asynchronous operation returns a summarized task object rather than the full task. The summary contains only the fields required to track the new task:

Field	Type	Description
taskUid	integer	Unique identifier of the newly created task.
indexUid	string	Index the task targets. null for global tasks.
status	string	Always enqueued at this point.
type	string	Task type, for example indexCreation or documentAdditionOrUpdate.
enqueuedAt	string (RFC 3339)	Date and time the task entered the queue.

{
  "taskUid": 0,
  "indexUid": "movies",
  "status": "enqueued",
  "type": "indexCreation",
  "enqueuedAt": "2021-08-11T09:25:53.000000Z"
}

Use the summarized task’s taskUid to track the progress of a task and retrieve the full task object.

Task status

Tasks always contain a field indicating the task’s current status. This field has one of the following possible values:

• enqueued: the task has been received and will be processed soon
• processing: the task is being processed
• succeeded: the task has been successfully processed
• failed: a failure occurred when processing the task. No changes were made to the database
• canceled: the task was canceled

succeeded, failed, and canceled tasks are finished tasks. Meilisearch keeps them in the task database but has finished processing these tasks. It is possible to configure a webhook to notify external services when a task is finished.

enqueued and processing tasks are unfinished tasks. Meilisearch is either processing them or will do so in the future.

Global tasks

Some task types are not associated with a particular index but apply to the entire instance. These tasks are called global tasks. Global tasks always display null for the indexUid field.

Meilisearch considers the following task types as global:

• dumpCreation
• taskCancelation
• taskDeletion
• indexSwap
• snapshotCreation

In a protected instance, your API key must have access to all indexes ("indexes": [*]) to view global tasks.

Task queue

After creating a task, Meilisearch places it in a queue. Enqueued tasks are processed one at a time, following the order in which they were requested.

When the task queue reaches its limit (about 10GiB), it will throw a no_space_left_on_device error. Users will need to delete tasks using the delete tasks endpoint to continue write operations.

Task queue priority

Meilisearch considers certain tasks high-priority and always places them at the front of the queue.

The following types of tasks are always processed as soon as possible in this order:

1. taskCancelation
2. upgradeDatabase
3. taskDeletion
4. indexCompaction
5. export
6. snapshotCreation
7. dumpCreation

All other tasks are processed in the order they were enqueued.

Task workflow

When you make a request for an asynchronous operation, Meilisearch processes all tasks following the same steps:

1. Meilisearch creates a task, puts it in the task queue, and returns a summarized task object. Task status set to enqueued
2. When your task reaches the front of the queue, Meilisearch begins working on it. Task status set to processing
3. Meilisearch finishes the task. Status set to succeeded if task was successfully processed, or failed if there was an error

Terminating a Meilisearch instance in the middle of an asynchronous operation is completely safe and will never adversely affect the database. Tasks are not canceled when the instance shuts down: any task that was processing is reset to enqueued on restart, and task handling proceeds as normal once the instance is relaunched.

Task batches

Meilisearch processes tasks in batches, grouping tasks for the best possible performance. In most cases, batching should be transparent and have no impact on the overall task workflow. Use the /batches route to obtain more information on batches and how they are processing your tasks.

A batch’s uid is incremented globally across all indexes in an instance, not per-index. If you see a large batchUid on a task, that number reflects batches processed across every index combined.

How auto-batching works

Meilisearch automatically groups consecutive compatible tasks into a single batch. For tasks to be batched together, they must meet all of the following conditions:

• They target the same index
• They are the same task type (for example, multiple documentAdditionOrUpdate tasks)
• They use the same content type (for example, all JSON or all NDJSON)

When Meilisearch encounters a task that cannot be grouped with the current batch (because it targets a different index, is a different task type, or is a deleteByFilter operation), it closes the current batch and starts a new one. Task ordering is always preserved: tasks within a batch are applied in the order they were enqueued.

Settings update tasks are also batched together when they target the same index. However, a settings update and a document addition cannot be part of the same batch, even if they target the same index.

Canceling tasks

Use the cancel tasks endpoint to cancel any number of tasks based on their uid, status, type, indexUid, or the date at which they were enqueued (enqueuedAt) or processed (startedAt). Canceling a task changes its status to canceled.

Only enqueued and processing tasks can be canceled. Calling cancel on finished tasks (succeeded, failed, or canceled) has no effect on their state: the operation still succeeds, but the response reports canceledTasks: 0.

Task cancellation is itself an asynchronous operation that creates a taskCancelation task. Because cancellation is an atomic transaction, either all matched tasks are successfully canceled, or none are.

POST /tasks/cancel requires at least one filter. To prevent users from accidentally canceling every enqueued and processing task, Meilisearch rejects unfiltered cancel requests with the missing_task_filters error.

You can also cancel taskCancelation tasks themselves as long as they are in the enqueued or processing state. This is possible because taskCancelation tasks are processed in reverse order: the last one you enqueue is processed first.

The canceledBy field

Every task object includes a canceledBy field:

• If the task was canceled, canceledBy contains the uid of the taskCancelation task that canceled it.
• If the task was not canceled, canceledBy is always null.

You can filter tasks by canceledBy in the list tasks endpoint to retrieve every task canceled by a given taskCancelation.

Deleting tasks

Finished tasks remain visible in the task list. To delete them manually, use the delete tasks route. Like cancellation, task deletion is an atomic transaction: either all matched tasks are successfully deleted, or none are.

Only finished tasks can be deleted. If you call the delete endpoint on enqueued or processing tasks, the request succeeds but deletedTasks is 0. Cancel those tasks first and then delete them once they reach a finished state.

Meilisearch stores up to 1M tasks in the task database. If enqueuing a new task would exceed this limit, Meilisearch automatically tries to delete the oldest 100K finished tasks. If there are no finished tasks in the database, Meilisearch does not delete anything and enqueues the new task as usual.

Task details by type

Every task object contains a details field whose shape depends on the task’s type. The table below lists the most common task types and the fields their details contains.

Task type	details fields
documentAdditionOrUpdate	receivedDocuments: number of documents in the payload. indexedDocuments: number successfully indexed.
documentDeletion	providedIds: number of document ids received. originalFilter: the filter used, if any. deletedDocuments: number of documents actually removed.
indexCreation / indexUpdate	primaryKey: the primary key set on the index, or null.
indexDeletion	deletedDocuments: number of documents removed along with the index.
indexSwap	swaps: array of swapped index pairs.
settingsUpdate	A copy of the settings payload applied to the index.
dumpCreation	dumpUid: identifier of the generated dump.
taskCancelation / taskDeletion	matchedTasks: number of tasks matching the filter. canceledTasks or deletedTasks: number of tasks actually affected. originalFilter: the query string used.
snapshotCreation	details is always null for snapshotCreation tasks.

Count fields such as indexedDocuments, deletedDocuments, canceledTasks, and deletedTasks are null while the task is enqueued or processing. They receive their final value only once the task reaches a finished state.

Examples

Suppose you add a new document to your instance using the add documents endpoint and receive a taskUid in response.

When you query the get task endpoint using this value, you see that it has been enqueued:

{
    "uid": 1,
    "indexUid": "movies",
    "status": "enqueued",
    "type": "documentAdditionOrUpdate",
    "canceledBy": null,
    "details": {
        "receivedDocuments": 67493,
        "indexedDocuments": null
    },
    "error": null,
    "duration": null,
    "enqueuedAt": "2021-08-10T14:29:17.000000Z",
    "startedAt": null,
    "finishedAt": null
}

Later, you check the task’s progress one more time. It was successfully processed and its status changed to succeeded:

{
    "uid": 1,
    "indexUid": "movies",
    "status": "succeeded",
    "type": "documentAdditionOrUpdate",
    "canceledBy": null,
    "details": {
            "receivedDocuments": 67493,
            "indexedDocuments": 67493
    },
    "error": null,
    "duration": "PT1S",
    "enqueuedAt": "2021-08-10T14:29:17.000000Z",
    "startedAt": "2021-08-10T14:29:18.000000Z",
    "finishedAt": "2021-08-10T14:29:19.000000Z"
}

Had the task failed, the response would have included a detailed error object:

{
    "uid": 1,
    "indexUid": "movies",
    "status": "failed",
    "type": "documentAdditionOrUpdate",
    "canceledBy": null,
    "details": {
            "receivedDocuments": 67493,
            "indexedDocuments": 0
    },
    "error": {
        "message": "Document does not have a `:primaryKey` attribute: `:documentRepresentation`.",
        "code": "internal",
        "type": "missing_document_id",
        "link": "https://docs.meilisearch.com/errors#missing-document-id"
    },
    "duration": "PT1S",
    "enqueuedAt": "2021-08-10T14:29:17.000000Z",
    "startedAt": "2021-08-10T14:29:18.000000Z",
    "finishedAt": "2021-08-10T14:29:19.000000Z"
}

If the task had been canceled while it was enqueued or processing, it would have the canceled status and a non-null value for the canceledBy field.

After a task has been deleted, trying to access it returns a task_not_found error.