deepr.prepros.Serial

class deepr.prepros.Serial(*preprocessors, fuse=True, num_parallel_calls=None)

Chain preprocessors to define complex preprocessing pipelines.

It will apply each preprocessing step one after the other on each element. For performance reasons, it fuses Map and Filter operations into single tf.data calls.

For an example, see the following snippet:

import deepr

def gen():
    yield {"a": [0], "b": [0, 1]}
    yield {"a": [0, 1], "b": [0]}
    yield {"a": [0, 1], "b": [0, 1]}

prepro_fn = deepr.prepros.Serial(
    deepr.prepros.Map(deepr.layers.Sum(inputs=("a", "b"), outputs="c")),
    deepr.prepros.Filter(deepr.layers.IsMinSize(inputs="a", outputs="a_size", size=2)),
    deepr.prepros.Filter(deepr.layers.IsMinSize(inputs="b", outputs="b_size", size=2)),
)

dataset = tf.data.Dataset.from_generator(gen, {"a": tf.int32, "b": tf.int32}, {"a": (None,), "b": (None,)})
reader = deepr.readers.from_dataset(prepro_fn(dataset))
expected = [{"a": [0, 1], "b": [0, 1], "c": [0, 2]}]

fuse

If True (default), will fuse Map and Filter.

Type:

bool, Optional

preprocessors

Positional arguments of Prepro instance or Tuple / List / Generator of prepro instances

Type:

Union[Prepro, Tuple[Prepro], List[Prepro], Generator[Prepro, None, None]]

__init__(*preprocessors, fuse=True, num_parallel_calls=None)

Methods

__init__(*preprocessors[, fuse, ...])
apply(dataset[, mode])	Pre-process a dataset