Skip to content

Add FAIL API to pynumaflow-lite SDK (trfm/map) #378

Description

@vaibhavtiwari33

Summary

Add the FAIL API to the pynumaflow-lite SDK so that user-defined functions can explicitly mark a message as failed. A failed message is retried according to the configured retryStrategy, matching the behavior recently introduced in Numaflow's transformer/map retry mechanism.

Background

Numaflow added a retry mechanism for transformer and map UDFs in numaproj/numaflow#3570. As part of that work, the ability to fail a message from within a UDF was exposed across SDKs:

The full pynumaflow SDK already provides this API. pynumaflow-lite does not yet expose it, so users on the lite SDK cannot fail/retry a message.

Goal

Provide the FAIL API in pynumaflow-lite for the UDF types that support it in the full SDK, so a handler can return a "failed" message and have it retried per retryStrategy.

Target UDF surfaces (mirroring pynumaflow#370):

  • Map
  • Batch Map
  • Map Stream
  • Source Transformer

Acceptance Criteria

  • A UDF handler in pynumaflow-lite can mark a message as failed for the map, batch map, map stream, and source transformer types.
  • The failure signal is propagated correctly to the Numaflow runtime over the existing gRPC contract (consistent with the wire behavior used by pynumaflow and numaflow-rs).
  • Unit tests cover the new API (message-level and, where applicable, server/handler-level).
  • Docs / examples updated as needed.

References

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions