Slurpers

Slurpers define how to transform data from one or more datasources. The type attribute of the Slurper define the programming language in which the user want to write its transformation code.

For now, only the Python programming language is supported.

Python

Usage

The user write code in the selected programming language, and when a Slurper get executed, its code will be called after fetching the data, and the returned result will be stored in a NoSQL database.

The user write a function called slurp(ctx, data_map).

The first argument is ctx, a dict containing the context.

The second argument is data_map, a dict containing data for each selected datasources. The keys are the names of the datasources, and the values are list of dict, each one being an item of the datasource (row of database, row of csv, item of webservice REST response, etc.)

Using Python's typing module, the type hints would be:

from typing import Any, Dict, List

def slurp(ctx: Dict[str, Any], data_map: Dict[str, List[Mapping[str, Any]]]):
    ...

Examples

Filtering a datasource named Users by username

def slurp(ctx, data_map):
    users = data_map.get("Users")
    return [user for user in users if user["username"] != "admin"]

Enhance items with a new field slurp_at

from datetime import datetime

def slurp(ctx, data_map):
    users = data_map.get("Users")
    slurp_at = datetime.now().isoformat()
    return [{**user, "slurp_at": now} for user in users]