From f636938bc28a164e32050276de9f7c4d641d8586 Mon Sep 17 00:00:00 2001 From: deswanth12 Date: Thu, 8 Oct 2026 13:00:04 +0530 Subject: [PATCH 1/2] docs(cli): clarify --output json behavior and examples --- docs/codebase/cli.md | 2 +- docs/md_v2/core/cli.md | 14 ++++++++------ 2 files changed, 9 insertions(+), 7 deletions(-) diff --git a/docs/codebase/cli.md b/docs/codebase/cli.md index 6f5a348b1..d280892d9 100644 --- a/docs/codebase/cli.md +++ b/docs/codebase/cli.md @@ -31,7 +31,7 @@ | **List saved profiles** | `crwl profiles` → choose **1** | Shows name, browser type, size, last-modified. | | **Delete a profile** | `crwl profiles` → choose **3** → pick the profile index → confirm | Removes the folder. | | **Crawl with a profile (default alias)** | `crwl https://site.com/dashboard -p my-profile` | Keeps login cookies, sets `use_managed_browser=true` under the hood. | -| **Crawl + verbose JSON output** | `crwl https://site.com -p my-profile -o json -v` | Any other `crawl` flags work the same. | +| **Crawl + full JSON output** | `crwl https://site.com -p my-profile -o all -v` | Any other `crawl` flags work the same. | | **Crawl with extra browser tweaks** | `crwl https://site.com -p my-profile -b "headless=true,viewport_width=1680"` | CLI overrides go on top of the profile. | | **Same but via explicit sub-command** | `crwl crawl https://site.com -p my-profile` | Identical to default alias. | | **Use profile from inside Profile Manager** | `crwl profiles` → choose **4** → pick profile → enter URL → follow prompts | Handy when demo-ing to non-CLI folks. | diff --git a/docs/md_v2/core/cli.md b/docs/md_v2/core/cli.md index ded35f2f6..bae7b1945 100644 --- a/docs/md_v2/core/cli.md +++ b/docs/md_v2/core/cli.md @@ -31,8 +31,8 @@ crwl https://example.com # Get markdown output crwl https://example.com -o markdown -# Verbose JSON output with cache bypass -crwl https://example.com -o json -v --bypass-cache +# Full crawl result as JSON with cache bypass (default output format) +crwl https://example.com -o all -v --bypass-cache # See usage examples crwl --example @@ -230,19 +230,21 @@ crwl https://example.com -f filter_bm25.yml -o markdown-fit ## Output Formats -- `all` - Full crawl result including metadata -- `json` - Extracted structured data (when using extraction) +- `all` - Full crawl result as formatted JSON, including HTML, markdown, and metadata (default) +- `json` - Extracted structured data only (requires an extraction strategy) - `markdown` / `md` - Raw markdown output - `markdown-fit` / `md-fit` - Filtered markdown for better readability +> **Important:** `--output json` (`-o json`) outputs structured data from `result.extracted_content`. It requires an extraction strategy configured via `--json-extract` / `-j` or `--extraction-config` / `-e` (optionally with `--schema` / `-s`). If no extraction strategy is configured, `extracted_content` is not populated and running with `-o json` will fail (as `extracted_content` is `None`). To export the full crawl result in JSON format without an extraction strategy, use `--output all` (the default). + ## Complete Examples -1. Basic Extraction: +1. Full Crawl Result as JSON: ```bash crwl https://example.com \ -B browser.yml \ -C crawler.yml \ - -o json + -o all ``` 2. Structured Data Extraction: From d8417087bc2fb2ed24dd542329b082e8dbb448a0 Mon Sep 17 00:00:00 2001 From: deswanth12 Date: Thu, 15 Oct 2026 17:31:17 +0530 Subject: [PATCH 2/2] docs(cli): address review feedback on examples, note wording, and missed copies --- docs/codebase/cli.md | 2 +- docs/md_v2/assets/llm.txt/diagrams/cli.txt | 2 +- docs/md_v2/assets/llm.txt/txt/cli.txt | 4 ++-- docs/md_v2/core/cli.md | 11 +++++++---- 4 files changed, 11 insertions(+), 8 deletions(-) diff --git a/docs/codebase/cli.md b/docs/codebase/cli.md index d280892d9..0028dede3 100644 --- a/docs/codebase/cli.md +++ b/docs/codebase/cli.md @@ -31,7 +31,7 @@ | **List saved profiles** | `crwl profiles` → choose **1** | Shows name, browser type, size, last-modified. | | **Delete a profile** | `crwl profiles` → choose **3** → pick the profile index → confirm | Removes the folder. | | **Crawl with a profile (default alias)** | `crwl https://site.com/dashboard -p my-profile` | Keeps login cookies, sets `use_managed_browser=true` under the hood. | -| **Crawl + full JSON output** | `crwl https://site.com -p my-profile -o all -v` | Any other `crawl` flags work the same. | +| **Crawl + full JSON output to file** | `crwl https://site.com -p my-profile -o all -O result.json -v` | Any other `crawl` flags work the same. | | **Crawl with extra browser tweaks** | `crwl https://site.com -p my-profile -b "headless=true,viewport_width=1680"` | CLI overrides go on top of the profile. | | **Same but via explicit sub-command** | `crwl crawl https://site.com -p my-profile` | Identical to default alias. | | **Use profile from inside Profile Manager** | `crwl profiles` → choose **4** → pick profile → enter URL → follow prompts | Handy when demo-ing to non-CLI folks. | diff --git a/docs/md_v2/assets/llm.txt/diagrams/cli.txt b/docs/md_v2/assets/llm.txt/diagrams/cli.txt index d18013307..c84f5ba6e 100644 --- a/docs/md_v2/assets/llm.txt/diagrams/cli.txt +++ b/docs/md_v2/assets/llm.txt/diagrams/cli.txt @@ -360,7 +360,7 @@ timeline Error Handling : Debug authentication issues section Production Phase - Automated Crawls : crwl URL -p profile -o json + Automated Crawls : crwl URL -p profile -o all -O result.json Batch Processing : Multiple URLs with same profile Monitoring : Check profile validity Maintenance : Update profiles as needed diff --git a/docs/md_v2/assets/llm.txt/txt/cli.txt b/docs/md_v2/assets/llm.txt/txt/cli.txt index b9107bb30..f0c8d0ddd 100644 --- a/docs/md_v2/assets/llm.txt/txt/cli.txt +++ b/docs/md_v2/assets/llm.txt/txt/cli.txt @@ -11,8 +11,8 @@ crwl https://example.com # Get markdown output crwl https://example.com -o markdown -# JSON output with cache bypass -crwl https://example.com -o json --bypass-cache +# Full crawl result as JSON with cache bypass, written to a file +crwl https://example.com -o all -O result.json --bypass-cache # Verbose mode with specific browser settings crwl https://example.com -b "headless=false,viewport_width=1280" -v diff --git a/docs/md_v2/core/cli.md b/docs/md_v2/core/cli.md index bae7b1945..0aeaa487f 100644 --- a/docs/md_v2/core/cli.md +++ b/docs/md_v2/core/cli.md @@ -31,8 +31,8 @@ crwl https://example.com # Get markdown output crwl https://example.com -o markdown -# Full crawl result as JSON with cache bypass (default output format) -crwl https://example.com -o all -v --bypass-cache +# Full crawl result as JSON with cache bypass, written to a file +crwl https://example.com -o all -O result.json -v --bypass-cache # See usage examples crwl --example @@ -235,7 +235,9 @@ crwl https://example.com -f filter_bm25.yml -o markdown-fit - `markdown` / `md` - Raw markdown output - `markdown-fit` / `md-fit` - Filtered markdown for better readability -> **Important:** `--output json` (`-o json`) outputs structured data from `result.extracted_content`. It requires an extraction strategy configured via `--json-extract` / `-j` or `--extraction-config` / `-e` (optionally with `--schema` / `-s`). If no extraction strategy is configured, `extracted_content` is not populated and running with `-o json` will fail (as `extracted_content` is `None`). To export the full crawl result in JSON format without an extraction strategy, use `--output all` (the default). +Use `-O ` to write the output to a file instead of the terminal. + +> **Note:** `-o json` prints only the extracted data, so it needs an extraction strategy (`-j` or `-e`, optionally with `-s`). Without one, the CLI exits with an error. Use `-o all` (the default) for the full crawl result as JSON. ## Complete Examples @@ -244,7 +246,8 @@ crwl https://example.com -f filter_bm25.yml -o markdown-fit crwl https://example.com \ -B browser.yml \ -C crawler.yml \ - -o all + -o all \ + -O result.json ``` 2. Structured Data Extraction: