Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/codebase/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@
| **List saved profiles** | `crwl profiles` → choose **1** | Shows name, browser type, size, last-modified. |
| **Delete a profile** | `crwl profiles` → choose **3** → pick the profile index → confirm | Removes the folder. |
| **Crawl with a profile (default alias)** | `crwl https://site.com/dashboard -p my-profile` | Keeps login cookies, sets `use_managed_browser=true` under the hood. |
| **Crawl + verbose JSON output** | `crwl https://site.com -p my-profile -o json -v` | Any other `crawl` flags work the same. |
| **Crawl + full JSON output to file** | `crwl https://site.com -p my-profile -o all -O result.json -v` | Any other `crawl` flags work the same. |
| **Crawl with extra browser tweaks** | `crwl https://site.com -p my-profile -b "headless=true,viewport_width=1680"` | CLI overrides go on top of the profile. |
| **Same but via explicit sub-command** | `crwl crawl https://site.com -p my-profile` | Identical to default alias. |
| **Use profile from inside Profile Manager** | `crwl profiles` → choose **4** → pick profile → enter URL → follow prompts | Handy when demo-ing to non-CLI folks. |
Expand Down
2 changes: 1 addition & 1 deletion docs/md_v2/assets/llm.txt/diagrams/cli.txt
Original file line number Diff line number Diff line change
Expand Up @@ -360,7 +360,7 @@ timeline
Error Handling : Debug authentication issues

section Production Phase
Automated Crawls : crwl URL -p profile -o json
Automated Crawls : crwl URL -p profile -o all -O result.json
Batch Processing : Multiple URLs with same profile
Monitoring : Check profile validity
Maintenance : Update profiles as needed
Expand Down
4 changes: 2 additions & 2 deletions docs/md_v2/assets/llm.txt/txt/cli.txt
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,8 @@ crwl https://example.com
# Get markdown output
crwl https://example.com -o markdown

# JSON output with cache bypass
crwl https://example.com -o json --bypass-cache
# Full crawl result as JSON with cache bypass, written to a file
crwl https://example.com -o all -O result.json --bypass-cache

# Verbose mode with specific browser settings
crwl https://example.com -b "headless=false,viewport_width=1280" -v
Expand Down
17 changes: 11 additions & 6 deletions docs/md_v2/core/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,8 +31,8 @@ crwl https://example.com
# Get markdown output
crwl https://example.com -o markdown

# Verbose JSON output with cache bypass
crwl https://example.com -o json -v --bypass-cache
# Full crawl result as JSON with cache bypass, written to a file
crwl https://example.com -o all -O result.json -v --bypass-cache

# See usage examples
crwl --example
Expand Down Expand Up @@ -230,19 +230,24 @@ crwl https://example.com -f filter_bm25.yml -o markdown-fit

## Output Formats

- `all` - Full crawl result including metadata
- `json` - Extracted structured data (when using extraction)
- `all` - Full crawl result as formatted JSON, including HTML, markdown, and metadata (default)
- `json` - Extracted structured data only (requires an extraction strategy)
- `markdown` / `md` - Raw markdown output
- `markdown-fit` / `md-fit` - Filtered markdown for better readability

Use `-O <file>` to write the output to a file instead of the terminal.

> **Note:** `-o json` prints only the extracted data, so it needs an extraction strategy (`-j` or `-e`, optionally with `-s`). Without one, the CLI exits with an error. Use `-o all` (the default) for the full crawl result as JSON.

## Complete Examples

1. Basic Extraction:
1. Full Crawl Result as JSON:
```bash
crwl https://example.com \
-B browser.yml \
-C crawler.yml \
-o json
-o all \
-O result.json
```

2. Structured Data Extraction:
Expand Down