Skip to content

Release and reuse the clients held by ClientManager - #2853

Merged
bert-e merged 3 commits into
development/9.6from
bugfix/BB-877
Sep 17, 2026
Merged

bert-e merged 3 commits into
development/9.6from
bugfix/BB-877

Conversation

@francoisferrand

Copy link
Copy Markdown
Contributor

ClientManager starts a 30 min interval to sweep inactive credentials, keeps the handle, and never uses it again. Nothing clears it and the timer is not unref'd, so a process cannot exit on its own. The lifecycle processors look like they handle this, but they clear the property on themselves rather than on the ClientManager, so those calls have always been no-ops.

unref() alone would not be enough: a live interval is a GC root, and its closure captures the manager, so the keepAlive agents, the client caches and the CredentialsManager stay reachable for the life of the process. The class gets a close() instead, called by the services that own a manager once their consumers and producers are done with the clients.

That matters more since BB-812, which made managers per endpoint and role rather than one per process. Looking at the other holders, ReplicateObject turned out to be the worse case: it built a brand-new manager for every entry it replicated, and another one on each retry that moved to the next host, dropping the previous one every time. Beyond the leak, each entry started from an empty credentials cache, so a role was assumed again for every single object. Those clients are now held on the queue processor and keyed by destination host and role, the way the copy location tasks already share theirs. The two caches stay separate because they assume their roles on different STS endpoints — the remote site's for the sources, the destination's for the targets.

Idle managers are not evicted from either cache while the process runs; they drain their own credentials and clients, so what remains is the shell. Bounded by endpoints and roles rather than by object count, and followed up in BB-886.

Issue: BB-877

@bert-e

bert-e commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

Hello francoisferrand,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options
name description privileged authored
/after_pull_request Wait for the given pull request id to be merged before continuing with the current one.
/bypass_author_approval Bypass the pull request author's approval
/bypass_build_status Bypass the build and test status
/bypass_commit_size Bypass the check on the size of the changeset TBA
/bypass_incompatible_branch Bypass the check on the source branch prefix
/bypass_jira_check Bypass the Jira issue check
/bypass_peer_approval Bypass the pull request peers' approval
/bypass_leader_approval Bypass the pull request leaders' approval
/bypass_source_branch_lineage Bypass the cross-branch contamination check
/approve Instruct Bert-E that the author has approved the pull request. ✍️
/create_pull_requests Allow the creation of integration pull requests.
/create_integration_branches Allow the creation of integration branches.
/no_octopus Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
/unanimity Change review acceptance criteria from one reviewer at least to all reviewers
/wait Instruct Bert-E not to run until further notice.
Available commands
name description privileged
/help Print Bert-E's manual in the pull request.
/status Print Bert-E's current status in the pull request.
/clear Remove all comments from Bert-E from the history TBA
/retry Re-start a fresh build TBA
/build Re-start a fresh build TBA
/force_reset Delete integration branches & pull requests, and restart merge process from the beginning.
/reset Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

@codecov

codecov Bot commented Sep 15, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 76.60%. Comparing base (9bf34a7) to head (3a2229b).
⚠️ Report is 3 commits behind head on development/9.6.

Additional details and impacted files

Impacted file tree graph

Files with missing lines Coverage Δ
extensions/gc/GarbageCollector.js 48.61% <100.00%> (+2.23%) ⬆️
...ecycle/bucketProcessor/LifecycleBucketProcessor.js 85.36% <100.00%> (+4.75%) ⬆️
...tensions/lifecycle/conductor/LifecycleConductor.js 86.91% <100.00%> (+0.02%) ⬆️
...ecycle/objectProcessor/LifecycleObjectProcessor.js 83.63% <100.00%> (+0.30%) ⬆️
...sions/replication/queueProcessor/QueueProcessor.js 77.98% <100.00%> (+0.47%) ⬆️
extensions/replication/tasks/ReplicateObject.js 92.77% <100.00%> (+0.08%) ⬆️
lib/clients/ClientManager.js 81.66% <100.00%> (+5.74%) ⬆️

... and 2 files with indirect coverage changes

Components Coverage Δ
Bucket Notification 80.25% <ø> (ø)
Core Library 83.81% <100.00%> (-0.01%) ⬇️
Ingestion 75.65% <ø> (ø)
Lifecycle 81.65% <100.00%> (+0.27%) ⬆️
Oplog Populator 85.83% <ø> (ø)
Replication 63.92% <100.00%> (+0.20%) ⬆️
Bucket Scanner 85.76% <ø> (ø)
@@                 Coverage Diff                 @@
##           development/9.6    #2853      +/-   ##
===================================================
+ Coverage            76.50%   76.60%   +0.09%     
===================================================
  Files                  206      206              
  Lines                14427    14450      +23     
===================================================
+ Hits                 11038    11069      +31     
+ Misses                3379     3371       -8     
  Partials                10       10              
Flag Coverage Δ
api:retry 9.18% <0.00%> (-0.02%) ⬇️
api:routes 8.95% <0.00%> (-0.02%) ⬇️
bucket-scanner 85.76% <ø> (ø)
ft_test:queuepopulator 11.18% <0.00%> (+0.26%) ⬆️
ingestion 12.72% <0.00%> (-0.03%) ⬇️
lib 8.94% <0.00%> (-0.04%) ⬇️
lifecycle 20.47% <21.21%> (+<0.01%) ⬆️
notification 0.98% <0.00%> (-0.01%) ⬇️
oplogPopulator 0.13% <0.00%> (-0.01%) ⬇️
replication 18.76% <12.12%> (-0.01%) ⬇️
unit 57.88% <100.00%> (+0.22%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Comment thread tests/unit/replication/QueueProcessor.spec.js Outdated
@bert-e

bert-e commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

Waiting for approval

The following approvals are needed before I can proceed with the merge:

  • the author

  • 2 peers

@francoisferrand
francoisferrand requested review from a team, benzekrimaha and delthas September 16, 2026 06:05
The interval sweeping inactive credentials was started and never
stopped: it kept the event loop alive, so a process could not exit on
its own, and it kept the whole ClientManager reachable along with its
keepAlive agents and client caches.

The lifecycle processors meant to clear it, but did so on themselves
rather than on the ClientManager, which never had anything to clear it
with. Give the class a close() and call it from the services owning a
manager, once their consumers and producers are done with the clients.

Issue: BB-877
A queue processor built a new ClientManager for every entry it
replicated, and another one each time a retry moved to the next host.
Each of them was left behind with its credentials cache, its keepAlive
agents and the sweep timer holding it, so a role was assumed again for
every single object.

Hold them on the queue processor, keyed by destination host and role, the
way the copy location tasks already share theirs, and release them on
shutdown. They assume their role on the destination STS, so they cannot
share the cache the copy location tasks read their sources with.

Issue: BB-877
The spec ended up with two adjacent describe('stop') blocks, one left by
the redis subscriber teardown and one by the client manager teardown, so
mocha reported the shutdown tests under two identically titled sections.

Keep a single suite holding all four of them, which is what the output is
expected to show for a class with one stop().

Issue: BB-877
@francoisferrand

Copy link
Copy Markdown
Contributor Author

/approve

@bert-e

bert-e commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

I have successfully merged the changeset of this pull request
into targetted development branches:

  • ✔️ development/9.6

The following branches have NOT changed:

  • development/7.10
  • development/7.4
  • development/7.70
  • development/8.6
  • development/9.0
  • development/9.1
  • development/9.2
  • development/9.3
  • development/9.4
  • development/9.5

This pull request did not target the following hotfix branch(es) so they
were left untouched:

  • hotfix/7.10.8
  • hotfix/9.0.7
  • hotfix/7.4.4
  • hotfix/7.4.5
  • hotfix/7.4.6
  • hotfix/7.8.0
  • hotfix/7.9.0
  • hotfix/7.4.2
  • hotfix/7.10.4
  • hotfix/7.4.7
  • hotfix/7.6.0
  • hotfix/7.4.8
  • hotfix/7.4.10
  • hotfix/7.4.9
  • hotfix/7.4.3
  • hotfix/7.2.0
  • hotfix/7.10.0
  • hotfix/7.10.12
  • hotfix/7.70.12
  • hotfix/7.7.0
  • hotfix/7.10.1
  • hotfix/9.0.4
  • hotfix/8.2.12
  • hotfix/7.70.15
  • hotfix/7.4.0
  • hotfix/7.4.1
  • hotfix/7.10.2
  • hotfix/7.10.3
  • hotfix/7.70.1
  • hotfix/7.10.17

Please check the status of the associated issue BB-877.

Goodbye francoisferrand.

The following options are set: approve

@bert-e
bert-e merged commit 3a2229b into development/9.6 Sep 17, 2026
24 checks passed
@bert-e
bert-e deleted the bugfix/BB-877 branch September 17, 2026 13:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants