API/behavior changes

Describe routes and behavior changes

First proposition

The task object

The task object gets updated with a new field: batchUid.

It is set to null for all enqueued tasks and is set to a single u32 number for all other tasks (succeeded, processing, canceled, or failed).

The batch object

Introduction of a new batch object containing the following fields:

• uid - The batch unique identifier. Starting at 0 and increasing by one for every new batch.
• details - This is akin to the task details but with the information of everything that happened during this batch. For example, if multiple document additions were processed together with a document deletion, we should see the total of documents updated and removed
• progress - Shows the progress and indexing step of the current batch. Once a batch is finished, it must display the progress: null:
  • steps: An array of all the steps currently being processed
    • currentStep: A string representing the name of the current step - It will change over time and break. It is NOT stable, and users should not rely on it. It’s only for the eyes, not for the code
    • finished: How many tasks are finished
    • total: The total number of tasks that must be finished before moving to the next step
  • percentage: A percentage of progression computed as the sum of the percentage of all steps running currently
• stats -
  • totalNbTasks: number of tasks in this batch
  • status :
    • succeeded - Number of tasks succeded
    • failed - same
    • canceled - same
    • processing - same
    • enqueued - same
  • types:
    • types associated with the number of tasks matching
  • indexUids:
    • List all the indexes with their number of tasks matching"canceledBy": null,
• started_at - The date at which the batch started processing, follow the rfc3339 format
• finished_at - The date at which the batch finished processing, follows the rfc3339 format
• duration - The duration the batch took to process its tasks

Example object:

{
  "uid": 160,
  "progress": {
    "steps": [
      {
        "currentStep": "processing tasks",
        "finished": 0,
        "total": 2
      },
      {
        "currentStep": "indexing",
        "finished": 2,
        "total": 3
      },
      {
        "currentStep": "extracting words",
        "finished": 3,
        "total": 13
      },
      {
        "currentStep": "document",
        "finished": 12300,
        "total": 19546
      }
    ],
    "percentage": 37.986263
  },
  "details": {
    "receivedDocuments": 19547,
    "indexedDocuments": null
  },
  "stats": {
    "totalNbTasks": 1,
    "status": {
      "processing": 1
    },
    "types": {
      "documentAdditionOrUpdate": 1
    },
    "indexUids": {
      "mieli": 1
    }
  },
  "duration": null,
  "startedAt": "2024-12-12T09:44:34.124726733Z",
  "finishedAt": null
}

Remarks:

• The batch doesn’t have an enqueued_at because a batch is never enqueued; it’s only living in RAM until it’s finished and written to disk.
• We cannot know in advance which batch a task will end up in. Thus, enqueued tasks should never appear in a batch.