gfw.common.beam.transforms.SampleAndLogElements#

class SampleAndLogElements(sample_size=None, window_size=60, pretty_print=False, message='Element: {e}')[source]#

A Beam PTransform that logs elements of a PCollection.

Parameters:

sample_size (int | None) – The number of elements to log. If not provided, logs all elements.
window_size (int) – The window duration in seconds used when sampling unbounded sources; only applicable when sample_size is set.
pretty_print (bool) – If True, formats each element as pretty-printed JSON when possible.
message (str) – A custom string format for the log message. Must contain the placeholder {e}.

Methods

`annotations`
`default_label`
`default_type_hints`
`display_data`	Returns the display data associated to a pipeline component.
`expand`	Log elements of a PCollection, optionally sampling a `sample_size` elements.
`from_runner_api`
`get_resource_hints`
`get_type_hints`	Gets and/or initializes type hints for this object.
`get_windowing`	Returns the window function to be associated with transform's output.
`infer_output_type`
`register_urn`
`runner_api_requires_keyed_input`
`to_runner_api`
`to_runner_api_parameter`
`to_runner_api_pickled`
`type_check_inputs`
`type_check_inputs_or_outputs`
`type_check_outputs`
`with_input_types`	Annotates the input type of a `PTransform` with a type-hint.
`with_output_types`	Annotates the output type of a `PTransform` with a type-hint.
`with_resource_hints`	Adds resource hints to the `PTransform`.

Attributes

expand(pcoll)[source]#

Log elements of a PCollection, optionally sampling a sample_size elements.