gfw.common.beam.transforms.HivePartitionConfig#

class HivePartitionConfig(fields=<factory>, prefix='event_', time_granularity='hour')[source]#

Hive-style partition layout for WritePartitionedParquet.

Parameters:

fields (dict[str, Callable[[str], str]]) – Extra partition dimensions applied before the time component, as an ordered {field_name: fn} dict. field_name is read from the element and fn maps its value to a partition string. Insertion order determines the directory hierarchy. Defaults to no extra dimensions (time-only partitioning).
prefix (str) – Prefix applied to every partition key name, including the time components. Avoids collisions with element field names when BigQuery reads the external table. Defaults to "event_".
time_granularity (Literal['hour', 'day']) – Granularity of the time partition component. "hour" produces {prefix}date=YYYY-MM-DD/{prefix}hour=HH/; "day" produces {prefix}date=YYYY-MM-DD/ only. Should be chosen based on query patterns, independently of window_size. Defaults to "hour".

Methods

Attributes