Skip to content

Drop the internal daily cap on YouTube analytics jobs - #402

Merged
paulocastellano merged 1 commit into
mainfrom
fix/youtube-analytics-daily-cap
Oct 10, 2026
Merged

paulocastellano merged 1 commit into
mainfrom
fix/youtube-analytics-daily-cap

Conversation

@paulocastellano

Copy link
Copy Markdown
Contributor

Problem

Nightwatch #18 (MaxAttemptsExceededException), 528 times in 24 hours, for CollectPublicationMetrics. The job fails at pickup in under a millisecond, which Worker::markJobAsFailedIfAlreadyExceedsMaxAttempts only does once retryUntil (the end of the day after the observation date) has passed. The jobs were not slow: they were never let through in time.

AnalyticsRateLimits gave YouTube a project-wide Limit::perDay(3000) (youtube:project), shared by metrics and discovery jobs. Once it runs out, RateLimited releases every YouTube job until the 24-hour window resets. The window is fixed and anchored at its first hit, so every blocked job comes back at the same instant, and the ones past retryUntil fail together:

  • All 23 failures on the prod clone are YouTube metrics jobs observed on 07/10 and failed at the same second (09/10 13:19:46). Nightwatch's last occurrence is 10/10 13:21: the same minute, a day later.
  • The large channels whose discovery was held behind the same key first appeared at 13:19:30 on 08/10 and 13:19:39 on 09/10.
  • 07/10 (the feat: independent social posts, workspace analytics and media editing #374 launch backfill of channels with 450–506 videos each) had 2,798 YouTube reads. 730 videos of four channels were never measured; none had failed, they never ran.

The cap was far below what it was protecting. It assumed each job spent up to 2 Data API units and shared the 10,000-unit quota with uploads. In reality:

  • A metrics job reads videos.list in batches of 50 ids, one unit for 50 videos (videos.list), plus one YouTube Analytics API report.
  • videos.insert has its own bucket of 100 calls a day (quota costs, getting started).
  • The project console shows 1,637 of 10,000 Data API units used, and 1,575 of 100,000 Analytics API queries. No quota went above 90% in 7 days.
  • The cap also counted every job that passed the middleware, including discovery, duplicates and jobs that end without reading.

Fix

Remove the YouTube shared daily cap. The per-channel limit (30 jobs a minute) stays. A quotaExceeded, rateLimitExceeded or 429 from Google is already classified as rate_limited by AnalyticsCollectionException::fromResponse and retried; if that happens, the project asks Google for more quota.

Tests

  • PublicationMetricsJobsTest: YouTube is limited only per channel (fails with the old cap).
  • Full suite in parallel (PostgreSQL): 9577 passed. One unrelated test, RemoteMediaImporterTest "a transfer that runs…", failed once in the parallel run; it compares temp file names, and it passed 4 of 4 times on its own, both with and without this change.

AnalyticsRateLimits capped YouTube analytics at 3,000 jobs a day on one
project-wide key, assuming each job spent up to 2 Data API units and shared the
10,000-unit quota with uploads. Neither holds: a metrics job reads videos.list in
batches of 50 ids, and videos.insert now has its own 100-call bucket. The
console shows 16% of the Data API and 1.6% of the Analytics API used.

The cap counted every job that passed the middleware, including discovery and
duplicates, so the launch backfill of large channels exhausted it. Every YouTube
metrics and discovery job was then released until the 24-hour window reset, and
the ones past retryUntil failed together with MaxAttemptsExceededException each
day at the same minute. A quota or 429 from Google is still retried as
rate_limited; more quota is requested from Google when that happens.
@paulocastellano
paulocastellano merged commit 7b9eeec into main Oct 10, 2026
8 checks passed
@paulocastellano
paulocastellano deleted the fix/youtube-analytics-daily-cap branch October 10, 2026 16:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant