fix: cancel sibling tasks when concurrent retrieval fails

[hxaxd]

Related Issues

• fixes asyncio.gather leaks orphaned tasks in MultiRetriever/MultiQueryTextRetriever/MultiQueryEmbeddingRetriever when one concurrent call fails #11965

Proposed Changes:

MultiRetriever.run_async, MultiQueryTextRetriever.run_async, and MultiQueryEmbeddingRetriever.run_async used asyncio.gather() to run retrieval calls concurrently. When one call failed, the exception was propagated immediately, but the remaining sibling tasks continued running.

These components share an intentional implementation lineage:

• MultiRetriever was introduced in feat: Add new components TextEmbeddingRetriever and MultiRetriever #10872 with parallel sync and async retrieval paths.
• MultiQueryTextRetriever.run_async and MultiQueryEmbeddingRetriever.run_async were introduced in feat: add run_async to TextEmbeddingRetriever, MultiQueryEmbeddingRetriever, and MultiQueryTextRetriever #11367, which explicitly followed the pattern established by MultiRetriever.run_async.
• Their async-first execution and thread fallback behavior now use the same _execute_component_async utility as Pipeline.

Since Pipeline already cancels and drains its remaining in-flight tasks when one component fails, these components should apply the same failure semantics to the nested concurrent tasks they manage internally.

This change aligns them with the existing Pipeline behavior:

• Keep explicit references to the concurrent tasks.
• Cancel sibling tasks when one task fails.
• Await the cancelled tasks with return_exceptions=True before re-raising the original exception.
• Add a regression test for each affected component that verifies an in-flight sibling receives cancellation.

The existing exception behavior and successful result handling remain unchanged.

How did you test it?

Affected unit tests

Ran the unit test suites for all three affected retriever components. The new regression tests verify that an in-flight sibling task is cancelled when another concurrent retrieval task fails.

50 passed, 10 deselected

The 10 deselected tests are marked as integration tests and are excluded by the standard unit test configuration. They were not failures or skips caused by this change.

Additional checks

Ran the project type checks, Ruff linting and formatting, and all applicable pre-commit hooks.

Mypy:      Success — no issues found in 370 source files
Ruff:      Passed
Pre-commit: All applicable hooks passed

Notes for the reviewer

The implementation intentionally follows the existing Pipeline cancel-and-drain pattern, preserving the current first-exception behavior.

This PR was developed with the assistance of an AI assistant. I have reviewed and understand the changes, and I take responsibility for this contribution.

Checklist

• I have read the contributors guidelines and the code of conduct.
• I have updated the related issue with new insights and changes.
• I have added unit tests and updated the docstrings.
• I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test: and added ! in case the PR includes breaking changes.
• I have documented my code.
• I have added a release note file, following the contributors guidelines.
• I have run pre-commit hooks and fixed any issue.

[vercel]

@hxaxd is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

[CLAassistant]

All committers have signed the CLA.

[Copilot]

Pull request overview

• Aligns async fan-out retrieval components with Pipeline failure semantics by cancelling + draining sibling tasks when one concurrent retrieval fails (fixes #11965), preventing orphaned in-flight tasks.

Changes:

• Replace direct asyncio.gather([...]) usage with explicit asyncio.create_task(...) plus cancel-and-drain behavior in the three affected run_async implementations.
• Add regression tests for MultiRetriever, MultiQueryTextRetriever, and MultiQueryEmbeddingRetriever to verify sibling cancellation on failure.
• Add a release note describing the behavior change.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
haystack/components/retrievers/multi_retriever.py	Track spawned retrieval tasks and cancel/drain siblings on failure before re-raising.
haystack/components/retrievers/multi_query_text_retriever.py	Apply the same cancel/drain pattern to per-query concurrent tasks.
haystack/components/retrievers/multi_query_embedding_retriever.py	Apply the same cancel/drain pattern to per-query concurrent tasks.
test/components/retrievers/test_multi_retriever.py	Add regression test asserting sibling retriever task receives cancellation.
test/components/retrievers/test_multi_query_text_retriever_async.py	Add regression test asserting sibling query task receives cancellation.
test/components/retrievers/test_multi_query_embedding_retriever_async.py	Add regression test asserting sibling query task receives cancellation.
releasenotes/notes/cancel-retriever-sibling-tasks-2d76cfc76a9b3ced.yaml	Document the fix in release notes.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[github-actions]

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
haystack/components/retrievers
multi_query_embedding_retriever.py
multi_query_text_retriever.py
multi_retriever.py
haystack/utils
async_utils.py
Project Total

This report was generated by python-coverage-comment-action

[bogdankostic]

This is duplicate in all components, maybe we can export this to a utility function in haystack/utils/async_utils.py to avoid duplicate code?

[hxaxd]

Thanks for pointing this out! I extracted the shared cancel-and-drain logic into _gather_tasks_with_cancel in haystack/utils/async_utils.py and updated all three components to use it. The three existing cancellation regression tests still pass.