gfw.common.beam.transforms.ParquetSink

class ParquetSink(schema, codec='snappy')

A FileSink that buffers rows and flushes as one row group.

Parameters:

• schema (pa.Schema) – PyArrow schema for the Parquet file.

• codec (str) – Parquet compression codec (e.g. "snappy", "zstd").

Methods

create_metadata
flush	Write all buffered rows as a single row group and close the writer.
open	Open the Parquet writer and initialise the row buffer.
write	Buffer one keyed row for writing.

open(fh)

Open the Parquet writer and initialise the row buffer.

write(element)

Buffer one keyed row for writing.

flush()

Write all buffered rows as a single row group and close the writer.