diff --git a/docs/codebase/cli.md b/docs/codebase/cli.md index 6f5a348b1..0028dede3 100644 --- a/docs/codebase/cli.md +++ b/docs/codebase/cli.md @@ -31,7 +31,7 @@ | **List saved profiles** | `crwl profiles` → choose **1** | Shows name, browser type, size, last-modified. | | **Delete a profile** | `crwl profiles` → choose **3** → pick the profile index → confirm | Removes the folder. | | **Crawl with a profile (default alias)** | `crwl https://site.com/dashboard -p my-profile` | Keeps login cookies, sets `use_managed_browser=true` under the hood. | -| **Crawl + verbose JSON output** | `crwl https://site.com -p my-profile -o json -v` | Any other `crawl` flags work the same. | +| **Crawl + full JSON output to file** | `crwl https://site.com -p my-profile -o all -O result.json -v` | Any other `crawl` flags work the same. | | **Crawl with extra browser tweaks** | `crwl https://site.com -p my-profile -b "headless=true,viewport_width=1680"` | CLI overrides go on top of the profile. | | **Same but via explicit sub-command** | `crwl crawl https://site.com -p my-profile` | Identical to default alias. | | **Use profile from inside Profile Manager** | `crwl profiles` → choose **4** → pick profile → enter URL → follow prompts | Handy when demo-ing to non-CLI folks. | diff --git a/docs/md_v2/assets/llm.txt/diagrams/cli.txt b/docs/md_v2/assets/llm.txt/diagrams/cli.txt index d18013307..c84f5ba6e 100644 --- a/docs/md_v2/assets/llm.txt/diagrams/cli.txt +++ b/docs/md_v2/assets/llm.txt/diagrams/cli.txt @@ -360,7 +360,7 @@ timeline Error Handling : Debug authentication issues section Production Phase - Automated Crawls : crwl URL -p profile -o json + Automated Crawls : crwl URL -p profile -o all -O result.json Batch Processing : Multiple URLs with same profile Monitoring : Check profile validity Maintenance : Update profiles as needed diff --git a/docs/md_v2/assets/llm.txt/txt/cli.txt b/docs/md_v2/assets/llm.txt/txt/cli.txt index b9107bb30..f0c8d0ddd 100644 --- a/docs/md_v2/assets/llm.txt/txt/cli.txt +++ b/docs/md_v2/assets/llm.txt/txt/cli.txt @@ -11,8 +11,8 @@ crwl https://example.com # Get markdown output crwl https://example.com -o markdown -# JSON output with cache bypass -crwl https://example.com -o json --bypass-cache +# Full crawl result as JSON with cache bypass, written to a file +crwl https://example.com -o all -O result.json --bypass-cache # Verbose mode with specific browser settings crwl https://example.com -b "headless=false,viewport_width=1280" -v diff --git a/docs/md_v2/core/cli.md b/docs/md_v2/core/cli.md index ded35f2f6..0aeaa487f 100644 --- a/docs/md_v2/core/cli.md +++ b/docs/md_v2/core/cli.md @@ -31,8 +31,8 @@ crwl https://example.com # Get markdown output crwl https://example.com -o markdown -# Verbose JSON output with cache bypass -crwl https://example.com -o json -v --bypass-cache +# Full crawl result as JSON with cache bypass, written to a file +crwl https://example.com -o all -O result.json -v --bypass-cache # See usage examples crwl --example @@ -230,19 +230,24 @@ crwl https://example.com -f filter_bm25.yml -o markdown-fit ## Output Formats -- `all` - Full crawl result including metadata -- `json` - Extracted structured data (when using extraction) +- `all` - Full crawl result as formatted JSON, including HTML, markdown, and metadata (default) +- `json` - Extracted structured data only (requires an extraction strategy) - `markdown` / `md` - Raw markdown output - `markdown-fit` / `md-fit` - Filtered markdown for better readability +Use `-O ` to write the output to a file instead of the terminal. + +> **Note:** `-o json` prints only the extracted data, so it needs an extraction strategy (`-j` or `-e`, optionally with `-s`). Without one, the CLI exits with an error. Use `-o all` (the default) for the full crawl result as JSON. + ## Complete Examples -1. Basic Extraction: +1. Full Crawl Result as JSON: ```bash crwl https://example.com \ -B browser.yml \ -C crawler.yml \ - -o json + -o all \ + -O result.json ``` 2. Structured Data Extraction: