gfw.common.beam.testing#

Testing utilities for Apache Beam pipelines.

Functions#

equal_to(expected[, equals_fn])

Drop-in replacement for apache_beam.testing.util.equal_to() with rich diff output.
equal_to(expected, equals_fn=<function _default_equals_fn>)[source]#

Drop-in replacement for apache_beam.testing.util.equal_to() with rich diff output.

This matcher performs unordered comparison of top-level elements in actual and expected PCollection outputs, just like Apache Beam’s equal_to(). However, it adds a rich diff visualization to help debug mismatches by rendering side-by-side differences.

Use in tests with assert_that(pcoll, equal_to(expected)).

Note

  • Only top-level permutations are considered equal: [1, 2] and [2, 1] are equal, but [[1, 2]] and [[2, 1]] are not.

  • If elements are not directly comparable, a fallback comparison using a custom equality function or deep diff is used. This helps handle:

    • Collections with types that don’t have a deterministic sort order (e.g., pyarrow.Tables as of 0.14.1).

    • Collections containing elements of different types.

Parameters:

  • expected (List[Any]) – Iterable of expected PCollection elements.

  • equals_fn (Callable[[Any, Any], bool]) – Optional function (expected_item, actual_item) -> bool to customize equality.

Returns:

A matcher function for use with apache_beam.testing.util.assert_that.

Return type:

Callable[[List[Any]], None]