Async jobs🔗

Async jobs define recurrent execution of Slurpers and the type of collection where the data are persisted.

Usage🔗

The recurrence is set using the crontab syntax. Recurrence can also be omitted, and user can run the job manually via the detail page of the async job.

Recurrence examples

Run at 04:00 (daily)

0 4 * * *

Run at 00:00 on Monday (weekly)

0 0 * * 1

At 22:00 on every day-of-week from Monday through Friday

0 22 * * 1-5

At minute 15 past every 3rd hour

15 */3 * * *

Async job with recurrence can be paused.

Async job execution are all logged, and user can access the history of execution.

Collections🔗

By default, an async job doesn't have any collection. You have to create and activate a collection of the right type.

There are six possibilities:

Slurpy collections

Normal collection🔗

Stored data is replaced every time the async job runs. You can only configure the name of the collection

Example

Iteration

Output data from slurper

Stored data

1

[
 {
  "id": 1,
  "name": "Leanne Graham",
  "username": "Bret"
 }
]

[
 {
  "id": 1,
  "name": "Leanne Graham",
  "username": "Bret"
 }
]

2

[
 {
  "id": 2,
  "name": "Ervin Howell",
  "username": "Antonette"
 }
]

[
 {
  "id": 2,
  "name": "Ervin Howell",
  "username": "Antonette"
 }
]

3

[
 {
  "id": 3,
  "name": "Clementine Bauch",
  "username": "Samantha"
 }
]

[
 {
  "id": 3,
  "name": "Clementine Bauch",
  "username": "Samantha"
 }
]

Upserted collection🔗

Stored data is updated every time the async job runs. New data is inserted, existing data is updated if needed. Absent data is not removed. You have to configure a key (column name) which will be used to define if the data is already present or not. Most of the times, this will correspond to a primary key: id. This has the advantage of keeping old data if it is removed in the datasource.

Example

Upsert key is id.

Iteration

Output data from slurper

Stored data

1

[
 {
  "id": 1,
  "name": "Leanne Graham",
  "username": "Bret"
 }
]

[
 {
  "id": 1,
  "name": "Leanne Graham",
  "username": "Bret"
 }
]

2

[
 {
  "id": 2,
  "name": "Ervin Howell",
  "username": "Antonette"
 }
]

[
 {
  "id": 1,
  "name": "Leanne Graham",
  "username": "Bret"
 },
 {
  "id": 2,
  "name": "Ervin Howell",
  "username": "Antonette"
 }
]

3

[
 {
  "id": 1,
  "name": "Leanne Graham",
  "username": "Samantha"
 }
]

[
 {
  "id": 1,
  "name": "Leanne Graham",
  "username": "Samantha"
 },
 {
  "id": 2,
  "name": "Ervin Howell",
  "username": "Antonette"
 }
]

Accumulative collection🔗

Stored data is updated every time the async job runs. New data is always inserted.

Example

Iteration

Output data from slurper

Stored data

1

[
 {
  "id": 1,
  "name": "Leanne Graham",
  "username": "Bret"
 }
]

[
 {
  "id": 1,
  "name": "Leanne Graham",
  "username": "Bret"
 }
]

2

[
 {
  "id": 2,
  "name": "Ervin Howell",
  "username": "Antonette"
 }
]

[
 {
  "id": 1,
  "name": "Leanne Graham",
  "username": "Bret"
 },
 {
  "id": 2,
  "name": "Ervin Howell",
  "username": "Antonette"
 }
]

3

[
 {
  "id": 1,
  "name": "Leanne Graham",
  "username": "Samantha"
 }
]

[
 {
  "id": 1,
  "name": "Leanne Graham",
  "username": "Bret"
 },
 {
  "id": 2,
  "name": "Ervin Howell",
  "username": "Antonette"
 },
 {
  "id": 1,
  "name": "Leanne Graham",
  "username": "Samantha"
 }
]

Versioned collection🔗

A new version of the data is created everytime the async job runs. Old versions are kept separately. You have to configure the number of versions you want to keep.

Example

Iteration

Output data from slurper

Stored data

1

[
 {
  "id": 1,
  "name": "Leanne Graham",
  "username": "Bret"
 }
]

V1:

[
 {
  "id": 1,
  "name": "Leanne Graham",
  "username": "Bret"
 }
]

2

[
 {
  "id": 2,
  "name": "Ervin Howell",
  "username": "Antonette"
 }
]

V1:

[
 {
  "id": 1,
  "name": "Leanne Graham",
  "username": "Bret"
 }
]

V2:

[
 {
  "id": 2,
  "name": "Ervin Howell",
  "username": "Antonette"
 }
]

3

[
 {
  "id": 3,
  "name": "Clementine Bauch",
  "username": "Samantha"
 }
]

V1:

[
 {
  "id": 1,
  "name": "Leanne Graham",
  "username": "Bret"
 }
]

V2:

[
 {
  "id": 2,
  "name": "Ervin Howell",
  "username": "Antonette"
 }
]

V3:

[
 {
  "id": 3,
  "name": "Clementine Bauch",
  "username": "Samantha"
 }
]

Upserted and versioned collection🔗

A new version of the data is created and then upserted every time the async job runs. This is a mix between versioning and upserting.

every time the async job runs, the current version of the data is duplicated and then the new version is upserted with the result of the async job.

Example

Upsert key is id.

Iteration

Output data from slurper

Stored data

1

[
 {
  "id": 1,
  "name": "Leanne Graham",
  "username": "Bret"
 }
]

V1:

[
 {
  "id": 1,
  "name": "Leanne Graham",
  "username": "Bret"
 }
]

2

[
 {
  "id": 2,
  "name": "Ervin Howell",
  "username": "Antonette"
 }
]

V1:

[
 {
  "id": 1,
  "name": "Leanne Graham",
  "username": "Bret"
 }
]

V2:

[
 {
  "id": 1,
  "name": "Leanne Graham",
  "username": "Bret"
 },
 {
  "id": 2,
  "name": "Ervin Howell",
  "username": "Antonette"
 }
]

3

[
 {
  "id": 1,
  "name": "Leanne Graham",
  "username": "Samantha"
 }
]

V1:

[
 {
  "id": 1,
  "name": "Leanne Graham",
  "username": "Bret"
 }
]

V2:

[
 {
  "id": 1,
  "name": "Leanne Graham",
  "username": "Bret"
 },
 {
  "id": 2,
  "name": "Ervin Howell",
  "username": "Antonette"
 }
]

V3:

[
 {
  "id": 1,
  "name": "Leanne Graham",
  "username": "Samantha"
 },
 {
  "id": 2,
  "name": "Ervin Howell",
  "username": "Antonette"
 }
]

Accumulative and versioned collection🔗

A new version of the data is created and then inserted everytime the async job runs. This is a mix between versioning and inserting.

Everytime the async job runs, the current version of the data is duplicated and then the new version is inserted with the result of the async job.

Example

Iteration

Output data from slurper

Stored data

1

[
 {
  "id": 1,
  "name": "Leanne Graham",
  "username": "Bret"
 }
]

V1:

[
 {
  "id": 1,
  "name": "Leanne Graham",
  "username": "Bret"
 }
]

2

[
 {
  "id": 2,
  "name": "Ervin Howell",
  "username": "Antonette"
 }
]

V1:

[
 {
  "id": 1,
  "name": "Leanne Graham",
  "username": "Bret"
 }
]

V2:

[
 {
  "id": 1,
  "name": "Leanne Graham",
  "username": "Bret"
 },
 {
  "id": 2,
  "name": "Ervin Howell",
  "username": "Antonette"
 }
]

3

[
 {
  "id": 1,
  "name": "Leanne Graham",
  "username": "Samantha"
 }
]

V1:

[
 {
  "id": 1,
  "name": "Leanne Graham",
  "username": "Bret"
 }
]

V2:

[
 {
  "id": 1,
  "name": "Leanne Graham",
  "username": "Bret"
 },
 {
  "id": 2,
  "name": "Ervin Howell",
  "username": "Antonette"
 }
]

V3:

[
 {
  "id": 1,
  "name": "Leanne Graham",
  "username": "Bret"
 },
 {
  "id": 2,
  "name": "Ervin Howell",
  "username": "Antonette"
 },
 {
  "id": 1,
  "name": "Leanne Graham",
  "username": "Samantha"
 }
]

Staging collection behavior🔗

For some collections, a staging collection is created (and initialized in some cases) at the beggining of the job, and used as a destination collection for this slurp. If the job complete successfully, the staging collection replace the actual collection. If not, the staging collection is dropped and the actual collection is kept. It works like an atomic transaction where we commit or rollback depending of the job result.

When an endpoint cache is accessed, the data are served from the actual collection, even if a staging collection is being filled, there is no downtime for the endpoint.

This applies to Normal and all Versionned collection types.