gfw.common.beam.transforms.WritePartitionedParquet#

class WritePartitionedParquet(path, schema, window_size=60, allowed_lateness=0, accumulation_mode=1, num_shards=6, codec='snappy', file_suffix='.parquet', partition=None, sink_factory=None)[source]#

Writes elements to hive-partitioned Parquet files.

Output files are organized using Hive-style partitioning. Time partitioning is always present and derived from the window start. Extra dimensions are user-defined and prepended before the time component:

{path}/{prefix}{dim1}={v1}/{prefix}{dim2}={v2}/.../{prefix}date=YYYY-MM-DD/{prefix}hour=HH/

For example, with HivePartitionConfig(fields={"source": fn}):

{path}/event_source=kpler/event_date=2024-01-15/event_hour=08/

## Tuning window_size and num_shards

The primary tuning knob is the combination of window_size and num_shards, which together determine output file size:

rows_per_file = (rows_per_second x window_size) / num_shards

File size directly controls:

  • Memory consumption per worker: larger files = more memory during flush.

  • Query performance: files that are too small increase metadata overhead in BigQuery external table scans; files that are too large reduce parallelism.

  • GCS write latency: files must be written within the window duration, so very large files with short windows can cause the pipeline to stall.

As a rule of thumb, target file sizes of 10-50 MB.

## Row groups

Each output file contains a single Parquet row group, which is optimal for BigQuery external table scans. This is a natural consequence of writing all rows for a shard in a single flush at window close.

Parameters:

  • path (str) – Output path where files will be written.

  • schema (pa.Schema) – PyArrow schema used by the Parquet writer.

  • window_size (int) – Size of the fixed window in seconds. Controls how frequently files are written. Shorter windows reduce latency and file size but produce more files. Defaults to 60 seconds.

  • allowed_lateness (int) – Allowed lateness in seconds. Late records arriving within this duration after the window closes will still be included. Defaults to 0.

  • accumulation_mode (AccumulationMode) – Beam accumulation mode. Defaults to DISCARDING, which is correct for most streaming use cases.

  • num_shards (int) – Number of output files per partition per window. Must be tuned together with window_size to hit the target file size. Defaults to 6.

  • codec (str) – Compression codec for Parquet. Defaults to "snappy".

  • file_suffix (str) – File suffix. Defaults to ".parquet".

  • partition (HivePartitionConfig | None) – Hive partition layout. Defaults to hourly time-only partitioning with "event_" prefix. See HivePartitionConfig.

  • sink_factory (Callable[..., ParquetSink] | None) – A callable (schema, codec) -> FileSink used to create the sink for each output file. Defaults to ParquetSink. Inject FakeParquetSink to bypass GCS writes in tests while still exercising windowing, partitioning, and file-naming logic.

Methods

annotations

default_label

default_type_hints

display_data

Returns the display data associated to a pipeline component.

expand

Applies windowing, partition keying, and Parquet file writing.

from_runner_api

get_resource_hints

get_type_hints

Gets and/or initializes type hints for this object.

get_windowing

Returns the window function to be associated with transform's output.

infer_output_type

register_urn

runner_api_requires_keyed_input

to_runner_api

to_runner_api_parameter

to_runner_api_pickled

type_check_inputs

type_check_inputs_or_outputs

type_check_outputs

with_input_types

Annotates the input type of a PTransform with a type-hint.

with_output_types

Annotates the output type of a PTransform with a type-hint.

with_resource_hints

Adds resource hints to the PTransform.

Attributes

label

pipeline

side_inputs

expand(pcoll)[source]#

Applies windowing, partition keying, and Parquet file writing.

Return type:

PCollection