Skip to content

feat(pegs): L2 oracle health monitor — stables/oracles.py (#301)#306

Open
spalen0 wants to merge 3 commits into
feat/peg-monitoring-foundationfrom
feat/peg-monitoring-l2-oracles
Open

feat(pegs): L2 oracle health monitor — stables/oracles.py (#301)#306
spalen0 wants to merge 3 commits into
feat/peg-monitoring-foundationfrom
feat/peg-monitoring-l2-oracles

Conversation

@spalen0

@spalen0 spalen0 commented Jun 29, 2026

Copy link
Copy Markdown
Collaborator

Closes #301. Layer 2 of peg monitoring.

⚠️ Stacked on #305 (foundation). Base is feat/peg-monitoring-foundation, not main — review/merge #305 first, then this retargets to main automatically. The diff below is L2-only.

What this is

Where protocols/stables/main.py watches market price (DeFiLlama, fast profile), this watches the on-chain oracles our lending markets actually liquidate on — the real liquidation-risk signal. Driven entirely by the shared PeggedAsset registry from the foundation.

Checks

Chainlink-backed assets (USDC, USDT, USDS, USDe, cbBTC, LBTC):

  • Stalenessnow − updatedAt > heartbeat + buffer.
  • Round sanity / monotonicity — positive answer, completed round, answeredInRound ≥ roundId, and a non-decreasing roundId vs the cached run.
  • Deviation from peg — oracle price vs the asset's PegTarget.
  • Oracle ↔ market divergence — Chainlink vs DeFiLlama.

Each check is a pure Alert | None function, so the forced-stale / forced-divergence / off-peg / round-malfunction cases are unit-tested without a chain (acceptance criterion).

Rate / fundamental oracles: generic monotonicity + delta-vs-cached handler (reuses the apyusd approach). A monotonic/capped decrease is CRITICAL (per #196).

Severity

Condition Severity
Non-positive answer / incomplete round / roundId backwards CRITICAL
Lagging answeredInRound HIGH
Staleness, peg deviation, market divergence HIGH
Fundamental-oracle monotonic decrease CRITICAL
Fundamental-oracle delta beyond threshold HIGH

Tenderly coverage / gap analysis (per #196 step 6)

TENDERLY_COVERED documents the LBTC Redstone fundamental oracle (0xb415…0bc81, capped @ 1) which already has a Tenderly alert — so it is not double-polled here. The registry currently exposes no uncovered fundamental oracle; the docstring records exactly what data a new one needs (address, read fn + precision, monotonic/capped flag) to be wired as a rate_oracle.

Foundation change

Added ChainlinkFeed.quote: PegTarget (default USD) so BTC-denominated feeds (LBTC/BTC) convert to a USD basis for the divergence comparison; LBTC's feed marked quote=BTC. Backward-compatible.

Tunables (env-overridable)

PEG_ORACLE_STALENESS_BUFFER (600s), PEG_ORACLE_DIVERGENCE_THRESHOLD (1%), PEG_ORACLE_RATE_DELTA_THRESHOLD (5%).

Automation

Added stables-oracles to the hourly profile in automation/jobs.yaml (renders; hourly now 24 tasks; test_automation_* green).

Validation

  • ruff format + ruff check clean; mypy clean on new modules (remaining errors pre-exist in utils/abi.py/utils/web3_wrapper.py).
  • 52 passed peg tests (incl. forced-stale/-divergence) + 13 passed automation tests.
  • Live mainnet smoke run reads all 6 feeds and computes peg/market deviations correctly — e.g. LBTC oracle $60,405 = 1.0043 BTC × $60,146, all within tolerance → 0 alerts, 1 Tenderly-covered oracle skipped.

Decisions for review

  • Divergence threshold defaults to 1% globally; could be made per-asset if BTC assets need a looser band.
  • LBTC Redstone left document-only (Tenderly-covered) rather than actively polled, to avoid duplicate alerts. Say the word if you'd prefer active polling here too.

🤖 Generated with Claude Code

spalen0 and others added 2 commits June 29, 2026 18:58
Layer 2 of peg monitoring: an hourly script that watches the on-chain oracles
our lending markets actually liquidate on, driven by the shared PeggedAsset
registry. Stacks on the foundation branch (#299 / #305).

Per Chainlink-backed asset (USDC, USDT, USDS, USDe, cbBTC, LBTC):
- staleness (now - updatedAt > heartbeat + buffer),
- round sanity / monotonicity (positive answer, completed round,
  answeredInRound >= roundId, non-decreasing roundId vs cached run),
- deviation from peg (oracle price vs PegTarget),
- oracle <-> market divergence (Chainlink vs DeFiLlama) — the liquidation signal.

Per-check logic is pure (Alert | None) so forced-stale / forced-divergence /
off-peg / round-malfunction cases are unit-tested without a chain.

Rate / fundamental oracles: generic monotonicity + delta-vs-cached handler
(apyusd approach); a monotonic/capped decrease is CRITICAL (per #196). LBTC
Redstone is already covered by a Tenderly alert, so it is documented in
TENDERLY_COVERED and not double-polled; gap-analysis notes included.

Foundation: add ChainlinkFeed.quote (PegTarget) so BTC-denominated feeds
(LBTC/BTC) convert to a USD basis for divergence; LBTC feed marked quote=BTC.

Wired into the hourly profile in automation/jobs.yaml (renders; 24 tasks).

Validation: ruff + mypy clean (new files); 52 peg tests + 13 automation tests
pass; live mainnet smoke run reads all 6 feeds and computes peg/market
deviations correctly (LBTC $60,405 = 1.0043 BTC x $60,146).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…301)

Address review on stables/oracles.py:

1. Emergency dispatch (utils/dispatch.dispatch_emergency_withdrawal) keys off
   Alert.protocol and skips anything outside DISPATCHABLE_PROTOCOLS. Alerts were
   hardcoding protocol="pegs" and putting the owner only in channel, so an USDe
   oracle failure routed to Telegram but never triggered ethena's emergency
   withdrawal. Rename PeggedAsset.channel -> protocol (the logical owner) and add
   an optional channel override (empty falls back to protocol routing). Alerts
   now emit protocol=asset.protocol, channel=asset.channel, so dispatch fires for
   ethena/cap/infinifi while Telegram routing is unchanged.

2. The round cache was overwritten with the current roundId unconditionally, so a
   backwards roundId became the new baseline and the regression was caught only
   once. Add pure next_cached_round(): a health-gated high-water mark that never
   lowers the cached round or stores a malfunctioning one.

Tests: add dispatch-routing coverage (alert.protocol in DISPATCHABLE_PROTOCOLS)
and next_cached_round cases. Full suite: 637 passed, 6 skipped. ruff + mypy clean
(new files). Live no-send smoke reads all six feeds, 0 alerts under current
conditions.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@spalen0 spalen0 marked this pull request as ready for review June 30, 2026 15:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant