Skip to content

feat(docloaders): add Xquik tweet loader - #1138

Open
kriptoburak wants to merge 1 commit into
ScrapeGraphAI:pre/betafrom
kriptoburak:codex/xquik-tweet-loader
Open

feat(docloaders): add Xquik tweet loader#1138
kriptoburak wants to merge 1 commit into
ScrapeGraphAI:pre/betafrom
kriptoburak:codex/xquik-tweet-loader

Conversation

@kriptoburak

Copy link
Copy Markdown

Problem

Public X status pages can return login or client-rendered shells to the generic Chromium loader. SmartScraperGraph then lacks stable post data for extraction.

Changes

  • Add XquikLoader for public x.com and twitter.com status URLs or numeric tweet IDs.
  • Retrieve the published tweet response through the optional x_twitter_scraper SDK.
  • Pass structured JSON and source metadata into the existing graph pipeline.
  • Skip HTML-to-Markdown conversion for the structured Xquik response.
  • Document installation, configuration, and a complete SmartScraperGraph example.
  • Cover URL validation, request order, client ownership, graph routing, metadata, and FetchNode behavior.

The SDK stays optional. Users install it with scrapegraphai when they enable the xquik configuration.

Tests

  • 107 deterministic unit tests passed, including the repository CI suite and 12 Xquik tests.
  • Black, isort, and Ruff passed for every changed Python file.
  • Mypy passed for the new loader and its unit tests.

Security

The examples contain placeholder environment variables only. The integration uses the published Xquik SDK contract and public tweet data.

@dosubot dosubot Bot added size:L This PR changes 100-499 lines, ignoring generated files. enhancement New feature or request labels Aug 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request size:L This PR changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant