DevByte Engine — V2

(pre dockerization version, to read the documentation of possible changes after containerization See DOCKER_NOTES.md )

DevByte Engine – An autonomous AI‑driven digital newsroom that discovers, evaluates, scripts, renders, and publishes developer‑focused short‑form videos on YouTube Shorts, Instagram Reels, and Facebook Pages. Keywords: AI video automation, YouTube Shorts, Instagram Reels, multi‑platform publishing, Gemini LLM, edge‑TTS, Remotion rendering.

Features

Autonomous content discovery from 4 sources
AI‑driven editorial ranking with Gemini
Concurrent video rendering using Remotion
Multi‑platform publishing (YouTube Shorts, Instagram Reels, Facebook Pages)
Fully‑automated AWS S3 temporary asset bridge
Windows‑compatible automation via Task Scheduler

DevByte Engine V2 transforms the original single-source video pipeline into an industrial-grade content automation and multi-platform publishing system. It pulls from four independent news sources, filters noise with rule-based signal detection, evaluates candidates through a cached two-pass AI editorial layer, writes scripts using category-specific prompts, concurrently synthesizes scripts and neural audio, renders videos utilizing 100% of host CPU cores, and broadcasts across YouTube Shorts, Instagram Reels, and Facebook Pages simultaneously via an automated AWS S3 temporary bridge.

Editorial Mission

DevByte covers products, releases, tools, developer infrastructure, AI breakthroughs, and major engineering announcements — not essays, tutorials, opinion pieces, or long-form discussions.

Architecture

flowchart TD
    subgraph discovery["Content Discovery (4 Sources)"]
        direction TB
        S1["Hacker News<br/>(JSON API)"]
        S2["Official Blogs<br/>(RSS Feeds)"]
        S3["GitHub Releases<br/>(REST API)"]
        S4["Product Hunt<br/>(RSS Feed)"]
    end

    subgraph ingestion["Ingestion Pipeline"]
        direction TB
        N["Normalizer"] --> SF["Signal Filter"]
        SF --> QF["Quality Filter"]
        QF --> DD["Deduplicator"]
    end

    subgraph intelligence["AI Newsroom"]
        direction TB
        EV["Gemini Editor<br/>(batch scoring)"] --> SC["Algorithmic<br/>Scoring"]
        SC --> ED["Editorial Engine<br/>(diversity + rotation)"]
    end

    subgraph production["Concurrent Video Production"]
        direction TB
        GS["Gemini Scriptwriter<br/>(Concurrent)"] --> TTS["Edge TTS<br/>(Concurrent)"]
        TTS --> REM["Remotion Renderer<br/>(100% CPU Concurrency)"]
    end

    subgraph publishing["Multi-Platform Publishing Engine"]
        direction TB
        OUT["Rendered Video"] --> YT["YouTube Shorts<br/>(Resumable Chunks)"]
        OUT --> S3["AWS S3 Temporary Asset Bridge"]
        S3 --> IG["Instagram Reels<br/>(Graph API)"]
        S3 --> FB["Facebook Page<br/>(Graph API)"]
        IG --> DEL["S3 Auto-Cleanup"]
        FB --> DEL
    end

    discovery --> ingestion
    ingestion --> intelligence
    intelligence --> production
    production --> publishing

What Changed from V1

Aspect	V1	V2
Sources	GitHub Trending only	HN + Blogs + GitHub Releases + Product Hunt
Selection	Random trending repo	AI-scored editorial queue with category rotation
Filtering	None	Signal filter → Quality filter → Deduplicator
Output	1 video (sequential)	Up to 5 videos (isolated workers with concurrent pre-production)
Publishing	Manual upload	Multi-platform: YouTube Shorts + Instagram Reels + Facebook Pages
S3 Bridge	None	Automated temporary S3 presigned asset distribution & auto-cleanup
Concurrency	Single-threaded	Concurrent pre-production, 100% CPU core rendering, overlapped upload I/O
History	None	PostgreSQL `publications` table + append-only ledger
Prompts	Single generic template	8 category-specific editorial prompts

The production pipeline follows Gemini → Validator → TTS → Remotion, with Remotion templates utilizing dynamic, sound-synchronized scene timing, attention-grabbing vector bullet badges, and full multi-core thread saturation.

Key Capabilities

Content Discovery

4 independent collectors querying official APIs, RSS feeds, and release endpoints
Configurable sources — add new blogs, repos, or filter keywords by editing JSON files in sources/
Signal filter with HN keyword whitelists and global noise blacklists (blocks tutorials, guides, essays, roundups)

AI Newsroom

Two-pass Gemini evaluation — uncached candidates are mission-checked in chunks of 10, then publishable items are relatively ranked
Parallel mission checks — up to 5 chunks are evaluated concurrently with ThreadPoolExecutor
Evidence-driven decisions — Gemini returns publish or reject decisions and editorial reasons using factual evidence rather than an arbitrary composite score
Editorial rotation — balances daily output across 8 content categories using publishing history
Source diversity limit — max 2 stories per source in a single daily batch
Breaking news override — high-scoring announcements (85+) bypass category rotation

Production & Multi-Platform Publishing

High-throughput batch processing — up to 5 isolated worker directories with concurrent pre-production (script + TTS) and pipelined rendering
20+ animated React components — spring physics, frosted glass, mesh gradients, typewriter subtitles
Dynamic Remotion timing — hook, body bullets, and CTA boundaries are calculated from sentence timing metadata, with word-weighted proportional fallback when metadata is missing
Sound-synchronized feature bullets — each body bullet enters when its sentence starts, and its accent border, glass surface, badge, and pulse state follow the active narration window
Vector bullet badges — deterministic SVG badges use four rotating tech themes (zap, neural core, rocket, and sparkle) instead of platform-dependent emoji glyphs
Automated YouTube upload — OAuth 2.0 integration, private staging, dynamic metadata injection, 2 MB resumable chunks, and progress logging
Automated Instagram Reels upload — Meta Graph API integration with direct REELS media containers, container status polling, and instant publishing
Automated Facebook Page upload — Meta Graph API Page Video integration with asynchronous processing verification
AWS S3 Temporary Asset Bridge — uploads the rendered video once to an Amazon S3 temporary bucket, generates a 2-hour presigned URL, broadcasts concurrently to Instagram and Facebook, and automatically deletes the temporary asset upon publication
Maximum Hardware Utilization — concurrent pre-production, 100% host CPU core saturation for Remotion rendering, and overlapped I/O multi-platform uploads
Clear, Uncluttered Logging — buffered subprocess execution with clean ANSI milestone lines, suppressing noisy terminal clutter while capturing full traces in logs/pipeline.log
Performance visibility — per-video timings for script generation, validation, TTS, rendering, and all upload targets with a visual ASCII bar chart

Tech Stack

Layer	Technology
Runtime	Python 3.11, Node.js 18
LLM	Google Gemini 2.5 Flash / 3.5 Flash (automatic fallback)
Voice	Edge TTS (Microsoft Azure Neural Voices)
Video	Remotion 4.x, React 19, Tailwind CSS v4, TypeScript
Publishing	YouTube Data API v3 (resumable chunks), Meta Graph API v19.0 (Instagram Reels, Facebook Page Videos)
Asset Relay	AWS S3 (boto3 presigned URLs with automated lifecycle cleanup)
Media	FFmpeg 6.x
Storage	PostgreSQL `publications` table (append‑only) + legacy `history.json` for fallback during migration

Repository Structure

devbyte-engine/
├── config.json                          # Voice, timeouts, retry limits
├── package.json                         # Node deps — npm run batch
├── .env                                 # GEMINI_API_KEY, META tokens, AWS keys (gitignored)
├── client_secrets.json                  # YouTube OAuth credentials (gitignored)
├── token.json                           # Saved YouTube access token (gitignored)
│
├── sources/                             # Editable newsroom configuration
│   ├── official_feeds.json              #   RSS feed registry (8 blogs)
│   ├── github_repos.json                #   Tracked GitHub repositories (8 repos)
│   ├── filters.json                     #   HN whitelist + global noise blacklist
│   └── github.py                        #   Legacy V1 GitHub Trending scraper
│
├── collectors/                          # Source-specific data fetchers
│   ├── hackernews.py                    #   HN Top Stories JSON API
│   ├── blogs.py                         #   RSS aggregator (OpenAI, Anthropic, etc.)
│   ├── github_releases.py              #   GitHub /releases/latest API
│   └── product_hunt.py                  #   Product Hunt RSS feed
│
├── ingestion/                           # Data cleaning pipeline
│   ├── normalizer.py                    #   Maps all sources → unified schema
│   ├── signal_filter.py                 #   Keyword whitelist/blacklist gatekeeper
│   ├── quality_filter.py                #   Drops malformed or thin candidates
│   └── deduplicator.py                  #   Story-level dedup (ID, URL, title)
│
├── evaluation/                          # Scoring and ranking
│   ├── evaluator.py                     #   Batch Gemini editorial scoring
│   └── scoring.py                       #   Freshness, popularity, trust formulas
│
├── editorial/                           # Content decisions
│   ├── editorial_engine.py              #   Queue builder (rotation + diversity)
│   ├── editorial_policy.json            #   Category weights and thresholds
│   └── prompts/                         #   Category-specific Gemini prompts
│       ├── update.txt                   #     Breaking news / releases
│       ├── free_alternative.txt         #     "Stop paying for X"
│       ├── hidden_gem.txt               #     "Nobody is talking about this"
│       ├── productivity.txt             #     "Saves 2 hours a day"
│       ├── comparison.txt               #     "ChatGPT vs Claude"
│       ├── weekly_roundup.txt           #     "5 AI tools this week"
│       ├── best_for.txt                 #     "Best AI for students"
│       └── prompt_trick.txt             #     "One prompt that saves hours"
│
├── channels/
│   └── ai_tools.json                    # Channel identity, tone, template map
│
├── services/                            # Core production services
│   ├── gemini.py                        #   Script + metadata generator
│   ├── tts.py                           #   Text-to-speech compiler
│   ├── render.js                        #   Remotion multi-core video renderer
│   ├── upload.py                        #   YouTube Data API v3 uploader
│   ├── upload_instagram.py              #   Instagram Reels Graph API uploader
│   ├── upload_facebook.py               #   Facebook Page Graph API uploader
│   └── s3_temp_upload.py                #   AWS S3 temporary bridge & lifecycle cleanup
│
├── orchestrator/                        # Pipeline controllers
│   ├── run_pipeline.js                  #   Sequential single-video runner
│   └── batch_generate_and_upload.js     #   Up to 5-worker sequential batch executor
│
├── utils/                               # Shared helpers
│   ├── logger.py                        #   Timestamped file + console logger
│   ├── validator.py                     #   Post-LLM script sanitizer
│   ├── config.py                        #   config.json loader
│   └── file_utils.py                    #   Path and JSON read/write helpers
│
├── data/                                # Runtime artifacts (gitignored)
│   ├── history.json                     #   Permanent publish ledger (append-only)
│   ├── raw_candidates.json              #   Post-ingestion candidates
│   ├── evaluated_candidates.json        #   Post-scoring candidates
│   ├── content_queue.json               #   Editorial-selected queue
│   ├── selected_tool.json               #   Current video target
│   ├── script.json / audio.mp3          #   Per-video production artifacts
│   └── worker_0/ ... worker_4/          #   Isolated batch worker directories
│
└── render/                              # Remotion video engine
    ├── remotion.config.ts               #   1080×1920 output, Tailwind, Webpack
    └── src/
        ├── Root.tsx                      #   Dynamic duration compositions and timing metadata schema
        ├── design/
        │   └── theme.ts                 #   Colors, typography, spacing tokens
        ├── backgrounds/
        │   ├── AnimatedMeshGradient.tsx  #   Floating sinusoidal mesh panels
        │   ├── DarkNoise.tsx            #   Glass-texture dark overlay
        │   ├── RadialGlow.tsx           #   Radial light source effect
        │   └── Grid.tsx                 #   Digital alignment grid
        ├── effects/
        │   ├── GlassMorphism.tsx        #   Frosted glass surface
        │   ├── Glow.tsx                 #   Neon glow border
        │   └── LightSweep.tsx           #   Horizontal light sweep
        ├── motion/
        │   ├── Fade.tsx / Slide.tsx      #   Entry animations
        │   ├── Scale.tsx / Pop.tsx       #   Attention animations
        │   ├── GlowPulse.tsx            #   Pulsing highlight
        │   ├── CountUp.tsx              #   Animated number counter
        │   └── Typewriter.tsx           #   Character-by-character reveal
        ├── components/
        │   ├── BrowserWindow.tsx        #   Chrome-style browser frame
        │   ├── TerminalWindow.tsx       #   macOS terminal frame
        │   ├── ToolCard.tsx             #   Product info card
        │   ├── FeatureChip.tsx          #   Pill-shaped feature tag
        │   ├── SubtitleBar.tsx          #   TTS-synced subtitle overlay
        │   └── ...
        ├── scenes/
        │   └── HeroScene.tsx            #   Full-screen title reveal
        ├── utils/
        │   └── timing.ts                 #   Sentence-aware scene and bullet timing engine
        └── templates/
            ├── FreeAlternative.tsx       #   Dynamic vertical video composition and vector badges
            └── Stubs.ts                 #   Template routing for other categories

Prerequisites

Dependency	Verify with
Python 3.11+	`python --version`
Node.js 18+	`node --version`
FFmpeg 6+	`ffmpeg -version`

Installation

# 1. Clone and initialize
git clone https://github.com/vishnu108shanker/Youtube-video-automation.git
cd Youtube-video-automation
mkdir -p data logs

# 2. Python dependencies
pip install python-dotenv google-genai requests beautifulsoup4 edge-tts feedparser python-slugify google-auth google-auth-oauthlib google-api-python-client boto3

# 3. Node orchestrator dependencies
npm install

# 4. Remotion renderer dependencies
cd render && npm install && cd ..

API Credentials & Environment Configuration

Configure .env in the project root with the following keys:

# ── Core AI & Database ──
GEMINI_API_KEY=your_google_gemini_api_key_here
DATABASE_URL=postgresql://user:password@localhost:5432/devbyte

# ── Meta Platforms (Instagram Reels & Facebook Page) ──
META_LONG_LIVED_TOKEN=your_meta_long_lived_user_access_token
INSTAGRAM_ACCOUNT_ID=your_instagram_business_account_id
FB_PAGE_ID=your_facebook_page_id
FB_LONG_LIVED_TOKEN=your_facebook_page_access_token

# ── AWS S3 Temporary Asset Bridge ──
S3_TEMP_BUCKET=your_s3_temporary_bucket_name
AWS_ACCESS_KEY_ID=your_aws_access_key_id
AWS_SECRET_ACCESS_KEY=your_aws_secret_access_key
AWS_DEFAULT_REGION=us-east-1

# ── Optional Concurrency Overrides ──
REMOTION_CONCURRENCY=4  # Defaults to 100% of host CPU cores

YouTube Data API Configuration:

Enable the YouTube Data API v3 in Google Cloud Console
Create an OAuth 2.0 Client ID (Desktop application)
Download the credentials JSON, rename to client_secrets.json, and place in the project root
On the first upload, a browser window opens for authentication — token.json is saved for future automated runs

Initialize the history ledger:

echo [] > data/history.json

Usage

Batch Production (Recommended)

The primary workflow — ingests from all sources, filters, evaluates, generates up to 5 videos sequentially, and uploads each to YouTube:

npm run batch

This executes five phases:

Phase 1 — Collect → Normalize → Signal Filter → Quality Filter → Deduplicate → Staleness Gate → Evaluate → Editorial Queue
Phase 2 — Create isolated worker directories and copy the selected candidates
Phases 3–4 — For each candidate, run Script → Validate → TTS → Render → Upload sequentially; the next video starts only after the previous one finishes
Phase 5 — Append successful publications to data/history.json
Performance reporting — Print and append a timing report after each successful video

Single Video Pipeline

For debugging or manual runs, the sequential pipeline generates one video:

node orchestrator/run_pipeline.js --fresh

Without --fresh, the orchestrator skips completed steps and resumes from the last failure point.

Running Modules Independently

Every Python module accepts --input and --output flags:

# Collect from one source
python collectors/hackernews.py --input config.json --output data/temp_hn.json

# Run the full ingestion chain
python ingestion/normalizer.py --input data/temp_hn.json --output data/raw_candidates.json
python ingestion/signal_filter.py --input data/raw_candidates.json --output data/raw_candidates.json
python ingestion/quality_filter.py --input data/raw_candidates.json --output data/raw_candidates.json
python ingestion/deduplicator.py --input data/raw_candidates.json --output data/raw_candidates.json
python ingestion/staleness_gate.py --input data/raw_candidates.json --output data/raw_candidates.json --max-days 14

# Evaluate and select
python evaluation/evaluator.py --input data/raw_candidates.json --output data/evaluated_candidates.json
python editorial/editorial_engine.py `
  --input data/evaluated_candidates.json `
  --output data/content_queue.json `
  --channel channels/ai_tools.json `
  --policy editorial/editorial_policy.json `
  --history data/history.json

  
# Generate a video from the top candidate
python services/gemini.py --input data/selected_tool.json --output data/script.json
python utils/validator.py --input data/script.json --output data/validated_script.json
python services/tts.py --input data/validated_script.json --output data/audio.mp3
node services/render.js --input data/validated_script.json --audio data/audio.mp3 --output data/video.mp4
# Multi-Platform Publishing (independent execution)
python services/upload.py --video data/worker_0/video.mp4 --script data/worker_0/script.json
python services/upload_instagram.py --video data/worker_0/video.mp4 --caption "DevByte AI Tool Spotlight"
python services/upload_facebook.py --video data/worker_0/video.mp4 --caption "DevByte AI Tool Spotlight"
on docker 
docker compose run --rm --entrypoint python3 devbyte services/upload_facebook.py --video data/worker_0/video.mp4 --caption "DevByte AI Tool Spotlight"

# Or manage the S3 temporary asset manually:
python services/s3_temp_upload.py --file data/worker_0/video.mp4
# Output: {"presigned_url": "...", "bucket": "...", "key": "..."}
python services/s3_temp_upload.py --delete --key meta_uploads/...mp4

services/render.js is JavaScript and must be run with Node, including in Docker:

docker compose run --rm devbyte node services/render.js \
    --input data/validated_script.json \
    --audio data/audio.mp3 \
    --output data/video.mp4

Dynamic Timing and Bullet Badges

render/src/templates/FreeAlternative.tsx delegates timeline calculation to render/src/utils/timing.ts. The renderer creates three contiguous scenes:

Hook — starts at frame 0 and lasts for the hook sentence duration.
Body bullets — starts immediately after the hook and allocates one duration per body sentence. Each bullet appears at the cumulative start frame of its sentence.
CTA — starts immediately after the final body sentence and runs to the end of the audio-driven composition.

The timing engine accepts sentence metadata from sentence_timings, sentence_durations, metadata.sentence_timings, metadata.sentence_durations, or timings. It also accepts structured timings values such as:

{
    "timings": {
        "hook": 2.5,
        "body": [3.1, 3.8, 2.9],
        "cta": 1.7
    }
}

Durations may be expressed in seconds, frames, or { "start": ..., "end": ... } objects. When metadata is absent or incomplete, durations are estimated from the actual word counts and scaled to the total video duration. Minimum scene guards and proportional scaling keep the scene boundaries contiguous without gaps.

Body bullets use deterministic vector badges rather than system emoji: zap (cyan/blue), neural core (pink/violet), rocket (amber/red), and sparkle (emerald/cyan). While a sentence is narrated, its card receives an accent border and glow, and its badge pulses with the active narration state.

You can have one SSH session and watch the log file live:

cmd :     tail -f logs/pipeline.log

That means: "Show me the last part of this file, and whenever new lines are appended, keep displaying them."

The extremely basic but powerful Unix philosophy: tail -f logs/pipeline.log

Then you'll probably find yourself wanting things like: tail -f logs/pipeline.log | grep ERROR or: *grep "worker_3" logs/pipeline.log *or: grep "RENDER" logs/pipeline.log

And suddenly your EC2 machine becomes much easier to operate.

Customizing the Newsroom

All editorial behavior is controlled through JSON configuration files — no code changes required.

What	File	Example
Add an RSS feed	`sources/official_feeds.json`	Append `{"url": "https://blog.example.com/rss", "source": "example_blog"}`
Track a GitHub repo	`sources/github_repos.json`	Append `"owner/repo"` to the array
Block noise keywords	`sources/filters.json`	Add words to `noise_blacklist` array (e.g., `"vercel"`, `"cloudflare"`)
Tune editorial constraints	`editorial/editorial_policy.json`	Adjust `queue_size` (5), `max_per_source` (3), or `max_per_company` (2)
Tune staleness cutoff	`editorial/editorial_policy.json`	Adjust `staleness_max_days` (default: 14 days)
Change cache TTL	`editorial/editorial_policy.json`	Adjust `cache_ttl_hours` (default: 30 hours)
Change TTS voice	`config.json`	Set `tts_voice` to any Edge TTS voice name
Edit video colors	`render/src/design/theme.ts`	Modify palette, typography, or spacing tokens

Automated Scheduling (Windows)

A run_automation.bat is included. Schedule it with Windows Task Scheduler:

schtasks /create /tn "DevByteAutomation" /tr "\"C:\path\to\run_automation.bat\"" /sc daily /st 23:30 /f

Pipeline Data Flow

                        ┌─────────────┐
                        │  4 Sources  │ (HN, Blogs, GitHub, Product Hunt)
                        └──────┬──────┘
                               ↓
                ┌───────────────────────────────┐
                │       Normalizer              │  Schema standardization
                │       Signal Filter           │  Noise & blacklist removal
                │       Quality Filter          │  Structure & field validation
                │       Deduplicator            │  ID, URL & fuzzy title dedup
                │       Staleness Gate          │  Drops items > 14 days old
                └───────────────┬───────────────┘
                               ↓
                ┌───────────────────────────────┐
                │       Evidence Builder        │  Objective facts (no arbitrary points)
                │       Pass 1: Mission Check   │  PUBLISH / REJECT in chunks of 10, up to 5 workers
                │       Pass 2: Relative Rank   │  Strict rank 1..N (scales to 50+ items)
                │       Editorial Engine        │  Top 5, source & company diversity
                └───────────────┬───────────────┘
                               ↓
                ┌───────────────────────────────┐
                │     Up to 5 Sequential Workers│
                │     Script → Validate → TTS   │
                │     → Render → Upload         │
                └───────────────────────────────┘

Editorial Evaluation Engine (V2 Architecture)

The legacy 4-factor arithmetic composite score (freshness + popularity + trust + quality = base * editor_multiplier) has been replaced with an evidence-driven editorial engine:

Code determines what is allowed. Gemini determines what is worth publishing.

1. Deterministic Hard Gates

Signal Filter & Blacklist: Discards noise (tutorial, opinion, essay, vercel, cloudflare).
Quality Filter: Enforces schema and required field presence.
Deduplicator: Prevents duplicate stories across sources.
Staleness Gate: Enforces a 14-day recency cutoff before sending items to the LLM.
History Gate: Skips previously published stories unless marked as major_update.

2. Evidence Builder (Facts, Not Points)

Instead of converting metrics into arbitrary points, the evidence builder compiles factual context for the editor:

Recency: days_ago, is_today, is_this_week.
Community Engagement: Raw hn_points, github_stars, ph_upvotes (null when not tracked).
Source Authority: source_label (e.g. "NVIDIA Official Blog/Newsroom").
Cross-Source Coverage: Number of distinct sources covering the story.

3. Two-Pass Gemini Editorial Judgment

Pass 1 — Mission Check: Evaluates items in chunks of 10. Up to five chunks are sent concurrently through ThreadPoolExecutor; Gemini classifies each item as publish or reject with a concrete 1–2 sentence editorial reason based on DevByte's core editorial mission.
Pass 2 — Relative Ranking: Eligible publishable stories are ranked from 1 (the single most important/viral video of the day) to $N$. Large batches (50+ items) are ranked using segmented tournament ranking.

4. Probabilistic Robustness & Structural Verification

temperature=0.0 is used as a stability heuristic, but the system is engineered to handle probabilistic variance.
Correctness Invariants: Every input candidate ID must be returned, decisions must be valid, publishable ranks must be strictly unique integers from $1..N$ with no gaps, and rejects must have rank: null.
Self-Healing: Malformed outputs trigger automated retries (up to 2 attempts) across fallback models (gemini-2.5-flash, gemini-3.5-flash, gemini-3.1-flash-lite).
ID and rank validation: Responses containing unknown candidate IDs, missing candidates, invalid decisions, duplicate ranks, or malformed ranks are rejected and retried. If every ranking attempt fails, the original candidate order is preserved.

5. 30-Hour Caching & Re-Scoring

Evaluations are cached in data/evaluation_cache.json for 30 hours.
Automatic re-scoring is triggered if an item goes viral (hn_points surges by > 50%) or gains cross-source validation before the cache expires.

6. Operational Constraints & Queue Output

Company Diversity: Max 2 stories per company per queue (prevents single-company domination).
Source Diversity: Max 3 stories per source.
Queue Size: Emits up to 5 ranked stories for sequential batch production.

7. High-Throughput Concurrent Production & Resource Saturation

Concurrent Pre-Production: Script generation (Gemini), validation, and neural voice synthesis (Edge TTS) run in parallel across all batch candidates at the start of Phase 3. All scripts and audio files are ready in ~10–15 seconds total.
100% Host CPU Saturation: Remotion rendering dynamically detects host hardware and invokes --concurrency=${os.cpus().length} so all CPU cores on the EC2 instance are pinned at 100% throughout the render.
Pipelined CPU & I/O Overlapping: As soon as Worker $i$ finishes rendering, it hands off its output to the background multi-platform publishing engine, and Worker $i+1$ immediately begins rendering on the CPU. The CPU is never waiting on outbound network uploads, and outbound network bandwidth is never waiting on the CPU.
Phase timing & Visual Report: The orchestrator measures Gemini Script, Validator, TTS Voice, Remotion Render, YT Upload, S3 Temp Upload, IG Upload, and FB Upload with millisecond timers, generating a proportional ASCII bar chart in the terminal and logs/pipeline.log.

Example:

⏱️   PERFORMANCE REPORT: "NVIDIA to Acquire Hugging Face"
────────────────────────────────────────────────────────────
  Gemini Script   :    6.2s  █
  Validator       :    0.2s
  TTS Voice       :    4.1s  █
  Remotion Render :   58.4s  ████████████████████
  YT Upload       :   18.2s  ██████
  S3 Temp Upload  :    4.3s  █
  IG Upload       :   26.5s  █████████
  FB Upload       :   24.1s  ████████
────────────────────────────────────────────────────────────
  Total Pipeline  :  142.0s

8. Multi-Platform Publishing & Zero-Clutter Logging

Multi-Platform Distribution: Simultaneously publishes to YouTube Shorts (resumable 2 MB chunks), Instagram Reels (Meta Graph API container lifecycle), and Facebook Pages (Page Video processing status polling).
Temporary S3 Bridge: Uploads the finished MP4 once to an Amazon S3 temporary bucket, generates a 2-hour presigned URL, broadcasts concurrently to Instagram and Facebook, and automatically deletes the temporary asset when both platforms finish.
Zero-Clutter Terminal Logs: Subprocess stdout and stderr are buffered and filtered. High-level milestone lines with tagged ANSI worker prefixes keep the terminal clean and readable, while full diagnostics and error stack traces are written to logs/pipeline.log.

Content Categories

Category	Hook Style	Example
`update`	Breaking news	"OpenAI just released..."
`free_alternative`	Cost savings	"Stop paying for X"
`hidden_gem`	Curiosity	"Nobody is talking about this"
`productivity`	Time savings	"Saves 2 hours a day"
`comparison`	Decision help	"ChatGPT vs Claude"
`weekly_roundup`	Breadth	"5 AI tools this week"
`best_for`	Search intent	"Best AI for students"
`prompt_trick`	Engagement	"One prompt that saves hours"

Each category has a dedicated prompt template in editorial/prompts/ that shapes the scriptwriting style, hook formula, and body structure.

Version History

Version	Date	Highlights
2.5.0	Sep 22, 2026	Multi-platform publishing engine (YouTube Shorts, Instagram Reels, Facebook Pages), AWS S3 temporary asset bridge with automated lifecycle cleanup, high-concurrency pipelining (concurrent pre-production, 100% multi-core Remotion CPU saturation, parallel platform uploads), and clean buffered terminal logging
2.4.0	Sep 8, 2026	Parallel Pass 1 evaluation, sequential resource-safe video production, resumable 2 MB upload chunks, upload progress logging, and per-video performance reports
2.3.0	Sep 7, 2026	Editorial evaluation engine rewrite — scrapped 4-factor scoring for evidence-driven LLM classification & relative ranking, 50+ batch scaling, 30h caching, company diversity cap
2.2.1	Jul 4, 2026	Deduplicator rewrite (story-level identity), source diversity cap
2.2.0	Jul 4, 2026	Newsroom overhaul — 4 sources, signal filter, batch Gemini scoring
2.1.0	Jul 4, 2026	Queue truncation fix, hype bonus, prompt persona refinement
2.0.0	Jun 23, 2026	Full architectural rewrite — editorial engine, parallel batch, YouTube upload
1.0.0	Jun 2026	Initial pipeline — GitHub Trending → Gemini → TTS → Remotion

Name		Name	Last commit message	Last commit date
.vscode		.vscode
DOCUMENTATIONS HISTORY		DOCUMENTATIONS HISTORY
channels		channels
collectors		collectors
database_layer		database_layer
editorial		editorial
evaluation		evaluation
ingestion		ingestion
migrations		migrations
orchestrator		orchestrator
render		render
scratch		scratch
scripts		scripts
services		services
sources		sources
tests		tests
utils		utils
.dockerignore		.dockerignore
.gitignore		.gitignore
CONTRIBUTING.md		CONTRIBUTING.md
Dockerfile		Dockerfile
LICENSE		LICENSE
README.md		README.md
command.md		command.md
config.json		config.json
docker-compose.yml		docker-compose.yml
package-lock.json		package-lock.json
package.json		package.json
requirements.txt		requirements.txt
test_postgres.py		test_postgres.py

Latest commit

History

Folders and files

Repository files navigation