deepr.utils package

Submodules

deepr.utils.broadcasting module

Tensorflow Broadcasting utilities

deepr.utils.broadcasting.make_same_shape(tensors, broadcast=True)

Make list of tensors the same shape

Parameters:

• tensors (List[tf.Tensor]) – List of tensors

• broadcast (bool, optional) – If True, not only add missing dims, also broadcast

Returns:

Return type:

List[tf.Tensor]

deepr.utils.checkpoint module

Checkpoint utilities

deepr.utils.checkpoint.save_variables_in_ckpt(path, variables, num_shards_embeddings=1)

Save variables in checkpoint

deepr.utils.datastruct module

Generic Iter Utilities

deepr.utils.datastruct.dict_to_item(data, keys)

Convert dictionary into object or tuple of objects.

deepr.utils.datastruct.item_to_dict(items, keys)

Convert tuple or object to dictionary.

Return type:

Dict

deepr.utils.datastruct.to_flat_tuple(items)

Convert nested list, tuples and generators to a flat tuple.

Flatten any nested structure of items. Will unpack lists, tuple and generators. Any other type will not be unpacked, meaning that you can safely use this function on other iterable types like strings or tf.Tensor. For example:

to_flat_tuple(1)  # (1,)
to_flat_tuple("hello")  # ("hello",)
to_flat_tuple(tf.ones([2, 2]))  # (tf.ones([2, 2]),)
to_flat_tuple((x for x in range(2)))  # (0, 1)
to_flat_tuple((1, 2))  # (1, 2)
to_flat_tuple(((0, 1), 2))  # (0, 1, 2)

Parameters:

items (Item, Tuple, List or Generator (nested)) – Items to transform to a flat tuple

Returns:

Return type:

Tuple

deepr.utils.exceptions module

Utilities for dealing with exceptions.

deepr.utils.exceptions.handle_exceptions(fn)

Handle Exceptions Decorator.

deepr.utils.field module

Field.

class deepr.utils.field.Field(name, shape, dtype, default=None, sequence=None)

Bases: object

Convenient way to define fields for features.

default

Default value of the field for padding

Type:

Any

dtype

Tensorflow type of the field (automatically inferred if string)

Type:

tf.DType

name

Name of the field

Type:

str

sequence

If True, the field represents a sequence.

Used for tf.Example message serialization : if sequence is True, the field with be stored in the feature_list entry of a tf.train.SequenceExample.

Automatically set if not given : True if shape’s first dimension is None.

Type:

bool

shape

Shape of the field

Type:

Tuple

as_placeholder(batch=False)

Return type:

placeholder

property batch_shape

property feature_specs

Return feature specs for parsing Example messages.

is_featurizable()

Return type:

bool

is_sparse()

Return type:

bool

startswith(prefix)

to_feature(value)

Convert value to tf.train.Feature or tf.train.FeatureList.

For shapes with more than 2 dimensions, uses np.ravel to flatten tensors in a list of values. Note that because tf.Example uses row-major to parse list of values, we make sure to use the same order with NumPy.

For that reason, if any of the dimensions is not set (i.e. is None), a ValueError is raised.

Parameters:

value (np.array) – Tensor values

Return type:

Union[Feature, FeatureList]

Returns:

• tf.train.FeatureList – If sequence is True

• tf.train.Feature – If sequence is False

Raises:

ValueError – If sequence, len(shape) > 2 and one of the non-first dimensions is not set (i.e. is None). If not sequence, len(shape) > 2 and any of the dimensions is not set (i.e. is None).

deepr.utils.field.TensorType(dtype)

Return TensorType from Python, TensorFlow or NumPy type

deepr.utils.graph module

Tensorflow Graph utilities.

deepr.utils.graph.get_by_name(graph, name)

Return op in Graph with name or None if not found.

Parameters:

graph (tf.Graph) – A Tensorflow Graph

Returns:

Return type:

tf.Operation or None

deepr.utils.graph.get_feedable_tensors(graph, names)

Retrieve feed tensors from graph.

Parameters:

• graph (tf.Graph) – A Tensorflow Graph

• names (List[str]) – List of operations or tensor names.

Returns:

Mapping of names to tf.Tensor

Return type:

Dict[str, tf.Tensor]

deepr.utils.graph.get_fetchable_tensors(graph, names)

Retrieve fetch tensors from graph.

Parameters:

• graph (tf.Graph) – A Tensorflow Graph

• names (List[str]) – List of operations or tensor names

Returns:

Mapping of names to tf.Tensor

Return type:

Dict[str, tf.Tensor]

deepr.utils.graph.import_graph_def(path_pb, name='')

Import Graph Definition from protobuff into the current Graph.

Parameters:

path_pb (str) – Path to .pb file

deepr.utils.graphite module

Graphite Utilities

deepr.utils.graphite.get_sender(host=None, port=None, prefix=None, postfix=None, timeout=5, interval=None, queue_size=None, log_sends=False, protocol='tcp', batch_size=1000)

Get Graphite Sender.

deepr.utils.graphite.log_metric(metric, value, postfix=None)

Log metric to graphite.

deepr.utils.graphite.log_metrics(metrics, postfix=None)

Log metrics to graphite

deepr.utils.iter module

Utilities for logging.

deepr.utils.iter.chunks(iterable, chunk_size)

Split Iterable into Iterable chunks.

>>> from deepr.utils import chunks
>>> for chunk in chunks(range(5), chunk_size=2):
...     print("-")
...     for idx in chunk:
...         print(idx)
-
0
1
-
2
3
-
4

deepr.utils.iter.progress(iterable, secs=60)

Log progress on Iterable.iterable

>> from deepr.utils import progress >>> for idx in progress(range(2), secs=10): … print(idx) 0 1

deepr.utils.mlflow module

MLFlow utilities.

deepr.utils.mlflow.clear_run()

Clear run (remove from MLFlow stack and unset ENV variable).

deepr.utils.mlflow.download_artifacts(run_id, path, dst_path=None, tracking_uri=None)

Download artifacts from MLFlow over HTTP if possible.

Parameters:

• run_id (str) – MLFlow Run Id

• path (str) – Path of the artifact

• dst_path (str, optional) – Path on the local system

deepr.utils.mlflow.log_dict(data, filename)

Log dictionary to MLFlow as an artifact under filename.

deepr.utils.mlflow.set_or_create_experiment(name, artifact_location=None)

Set Experiment with specific artifact_location.

deepr.utils.tables module

Tables Utilities

class deepr.utils.tables.TableContext

Bases: object

Context Manager to reuse Tensorflow tables.

Tensorflow does not have a tf.get_variable equivalent for tables. The TableContext is here to provide this functionality.

Example

>>> import deepr
>>> with deepr.utils.TableContext() as tables:
...     table = deepr.utils.table_from_mapping(name="my_table", mapping={1: 2})
...     tables.get("my_table") is table
True

>>> with deepr.utils.TableContext():
...     table = deepr.utils.table_from_mapping(name="my_table", mapping={1: 2})
...     reused = deepr.utils.table_from_mapping(name="my_table", reuse=True)
...     table is reused
True

classmethod active()

close()

get(name)

classmethod is_active()

set(name, table)

deepr.utils.tables.index_to_string_table_from_file(name, path=None, vocab_size=None, default_value='UNK', reuse=False)

Create reverse table from file

deepr.utils.tables.table_from_file(name, path=None, key_dtype=None, reuse=False, default_value=-1)

Create table from file

deepr.utils.tables.table_from_mapping(name, mapping=None, default_value=None, key_dtype=None, value_dtype=None, reuse=False)

Create table from mapping

deepr.utils.tf2xla_pb2 module

deepr.utils.uuid module

Uuid

deepr.utils.uuid.msb_lsb_to_str(msb, lsb)

Convert two 64 bit integers MSB and LSB to a 128 bit UUID.

deepr.utils.uuid.str_to_msb_lsb(el)

Convert a 128 bit UUID to two 64 bit integers MSB and LSB.

Module contents