From 324a74e75d83cc3fff1276e90f3ce6fd6f096bec Mon Sep 17 00:00:00 2001 From: David Yaffe Date: Thu, 20 Aug 2026 14:58:54 -0400 Subject: [PATCH 1/2] Fix dead install.quickwit.io references and update the AWS S3 tutorial install.quickwit.io no longer resolves, which breaks the very first command in the quickstart, the installation guide, and both HDFS tutorials. The install.sh script itself is still present and maintained at the repo root, so point the docs at its raw GitHub URL rather than replacing it. That keeps the script's platform detection and latest-version resolution, and adds no version literals to maintain. This also fixes distribution/docker/ubuntu/Dockerfile, which cannot build today. Because a shell pipeline reports only the status of its last command, "curl | sh" exits 0 and installs nothing, so the build instead fails further down at the COPY of /quickwit-v*/quickwit with a misleading error. That Dockerfile is not referenced by any workflow, which is why this went unnoticed. Published images are unaffected because they compile from source. The remaining changes fix the AWS S3 tutorial, verified end to end against a live three-node cluster on EC2: - Attach the cluster security group to the first node as well. Without this the pre-existing node drops inbound gossip and the cluster never forms. - Use the private IP of searcher-1 for peer_seeds. Quickwit advertises its private address, so a public seed cannot match a rule scoped to the security group. - Narrow the security group to TCP 7280-7281 and UDP 7280 with the group itself as the source, rather than the 7200-7300 range. - Declare enabled_services in all three node configs, so searcher-1 runs the metastore, control plane, indexer, searcher and janitor while the other two run only the searcher. This replaces the --service searcher flag on the command line. - Correct the metastore path to include the prefix set in S3_PATH. - Verify cluster formation through /api/v1/cluster, naming ready_nodes, and note that the joining cluster log line is emitted before any peer is contacted. - Offer the search step as CLI or cURL through tabs, matching the hdfs-logs tutorial. - Document recovery from a partial ingest using index clear, since re-running the ingest appends and inflates the expected 345 hits. - Use an IAM instance profile instead of static access keys on EC2. - Note that quickwit run holds the foreground and needs a second terminal. - Expand cleanup to cover instances, root volumes, the security group, and the bucket. The config format version 0.8 is intentionally left unchanged, as it is independent of the 0.9.0 binary version. Both control_plane and control-plane parse correctly in enabled_services, so the underscore here matches the node configuration reference. Closes #6660 Closes #4004 --- CONTRIBUTING.md | 2 +- distribution/docker/ubuntu/Dockerfile | 2 +- docs/get-started/installation.md | 2 +- docs/get-started/quickstart.md | 2 +- ...ial-hdfs-logs-distributed-search-aws-s3.md | 109 +++++++++++++++--- .../tutorials/tutorial-hdfs-logs.md | 2 +- 6 files changed, 100 insertions(+), 19 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index ba4e68c831c..827ab0ad3ee 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -106,7 +106,7 @@ As the react UI is embedded in the rust binary, we need to build the react app b ## Testing release (alpha, beta, rc) -The following Quickwit installation command `curl -L https://install.quickwit.io | sh` always installs the latest stable version of quickwit. To make it easier in installing and testing new (alpha, beta, rc) releases, you can manually pull and execute the script as `./install.sh --allow-any-latest-version`. This will force the script to install any latest available release package. +The following Quickwit installation command `curl -L https://raw.githubusercontent.com/quickwit-oss/quickwit/main/install.sh | sh` always installs the latest stable version of quickwit. To make it easier in installing and testing new (alpha, beta, rc) releases, you can manually pull and execute the script as `./install.sh --allow-any-latest-version`. This will force the script to install any latest available release package. ## Tracking licenses diff --git a/distribution/docker/ubuntu/Dockerfile b/distribution/docker/ubuntu/Dockerfile index a3ddc75df00..a8e24c6ce28 100644 --- a/distribution/docker/ubuntu/Dockerfile +++ b/distribution/docker/ubuntu/Dockerfile @@ -1,7 +1,7 @@ FROM ubuntu:noble@sha256:66460d557b25769b102175144d538d88219c077c678a49af4afca6fbfc1b5252 AS builder RUN apt-get update && apt-get install -y curl -RUN curl -L https://install.quickwit.io | sh +RUN curl -L https://raw.githubusercontent.com/quickwit-oss/quickwit/main/install.sh | sh FROM ubuntu:noble@sha256:66460d557b25769b102175144d538d88219c077c678a49af4afca6fbfc1b5252 AS quickwit diff --git a/docs/get-started/installation.md b/docs/get-started/installation.md index 52f1097e035..277a06b1ff9 100644 --- a/docs/get-started/installation.md +++ b/docs/get-started/installation.md @@ -139,7 +139,7 @@ To easily install Quickwit on your machine, just run the command below from your The script detects the architecture and then downloads the correct binary archive for the machine. ```bash -curl -L https://install.quickwit.io | sh +curl -L https://raw.githubusercontent.com/quickwit-oss/quickwit/main/install.sh | sh ``` All this script does is download the correct binary archive for your machine and extracts it in the current working directory. This means you can download any desired archive from [github](https://github.com/quickwit-oss/quickwit/releases) that matches your OS architecture and manually extract it anywhere. diff --git a/docs/get-started/quickstart.md b/docs/get-started/quickstart.md index 311fbaf6c03..64e23ae8266 100644 --- a/docs/get-started/quickstart.md +++ b/docs/get-started/quickstart.md @@ -14,7 +14,7 @@ The Quickwit installer automatically picks the correct binary archive for your e This method works only for [some OS/architectures](installation.md#download), and you will also need to install some [external dependencies](installation.md#note-on-external-dependencies). ```bash -curl -L https://install.quickwit.io | sh +curl -L https://raw.githubusercontent.com/quickwit-oss/quickwit/main/install.sh | sh ``` ```bash diff --git a/docs/get-started/tutorials/tutorial-hdfs-logs-distributed-search-aws-s3.md b/docs/get-started/tutorials/tutorial-hdfs-logs-distributed-search-aws-s3.md index 7e444e08b7e..e3cf41d2f18 100644 --- a/docs/get-started/tutorials/tutorial-hdfs-logs-distributed-search-aws-s3.md +++ b/docs/get-started/tutorials/tutorial-hdfs-logs-distributed-search-aws-s3.md @@ -6,6 +6,9 @@ icon_url: /img/tutorials/aws-logo.png sidebar_position: 6 --- +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + In this guide, we will index about 20 million log entries (7 GB decompressed) on AWS S3 using an EC2 instance and launch a three-node distributed search cluster. Example of a log entry: @@ -34,7 +37,7 @@ First of all, let's create an EC2 instance, install a Quickwit binary, and [conf ## Install ```bash -curl -L https://install.quickwit.io | sh +curl -L https://raw.githubusercontent.com/quickwit-oss/quickwit/main/install.sh | sh cd quickwit-v*/ ``` @@ -47,8 +50,10 @@ export S3_PATH=s3://{path/to/bucket}/indexes ``` :::note -You'll want to include the necessary authorization for the given bucket, this can be done by setting the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` -environment variables, or via the AWS credentials file. Usually located at `~/.aws/credentials`. +Quickwit needs access to the bucket and the objects it contains. On EC2, attach an IAM role through an instance +profile. Quickwit picks up those credentials automatically. The role needs `s3:ListBucket` on the bucket, plus +`s3:GetObject`, `s3:PutObject`, `s3:DeleteObject`, `s3:ListMultipartUploadParts`, and `s3:AbortMultipartUpload` on +its objects. For more information, check out [our AWS setup guide](../../guides/aws-setup). ::: @@ -60,6 +65,12 @@ Now we can create a Quickwit config file. echo "version: 0.8 node_id: searcher-1 listen_address: 0.0.0.0 +enabled_services: + - metastore + - control_plane + - searcher + - indexer + - janitor metastore_uri: ${S3_PATH} default_index_root_uri: ${S3_PATH} " > config.yaml @@ -75,12 +86,23 @@ default_index_root_uri: ${S3_PATH} > default_index_root_uri: ${S3_PATH} >``` +`enabled_services` controls which services run on each node. Here `searcher-1` runs the five services listed above, +while nodes 2 and 3 run only the searcher. + We are now ready to start Quickwit. ```bash ./quickwit run --config config.yaml ``` +:::note + +`quickwit run` stays in the foreground. Keep it running and open a second connection to the same instance for the +index and ingest commands that follow. To keep access to the node logs after disconnecting, run it under `tmux` or +redirect its output with `nohup`. + +::: + ## Create your index ```bash @@ -136,7 +158,7 @@ We can now create the index with the `create` subcommand. :::note The `create` command sends the index configuration to the running Quickwit node. The node stores the file-backed -metastore at `s3://path-to-your-bucket/hdfs-logs/metastore.json`. +metastore at `s3://path-to-your-bucket/indexes/hdfs-logs/metastore.json`, under the prefix set in `S3_PATH`. To run this command from your local machine, add `--endpoint http://:7280`. @@ -164,11 +186,41 @@ To run this command from your local machine, add `--endpoint http:// + + + ```bash ./quickwit index search --index hdfs-logs --query "severity_text:ERROR" ``` + + + + +```bash +curl "http://127.0.0.1:7280/api/v1/hdfs-logs/search?query=severity_text:ERROR" +``` + + + + + which returns the json ```json @@ -205,13 +257,20 @@ Also, it needs `{rest.listen_port} + 1` for gRPC communication between instances In AWS, you can create a security group to group these inbound rules. Check out the [network section](../../guides/aws-setup) of our AWS setup guide. -To make things easier, let's create a security group that opens the TCP/UDP port range [7200-7300]. -Next, create two additional EC2 instances using the previously created security group. Take note of each instance's public IP address. +Create one security group for the Quickwit cluster. Set the security group itself as the source and allow TCP ports +7280 and 7281, plus UDP port 7280. + +Attach this group to all three instances, including the `searcher-1` instance you launched earlier. Nodes 2 and 3 +connect to node 1, so node 1 must accept inbound traffic on these ports. If `searcher-1` does not have this group +attached, the other nodes cannot reach it and the cluster does not form. + +Next, create two additional EC2 instances using this security group, and note the private IP address of `searcher-1`. -SSH into the second and third EC2 instances, install Quickwit, and [configure the environment](../../guides/aws-setup) to let Quickwit access the index S3 bucket. +Connect to the second and third EC2 instances, install Quickwit, and +[configure AWS access](../../guides/aws-setup) for the index bucket. ```bash -curl -L https://install.quickwit.io | sh +curl -L https://raw.githubusercontent.com/quickwit-oss/quickwit/main/install.sh | sh cd quickwit-v*/ ``` @@ -219,9 +278,17 @@ And configure the environment so instances can form a cluster: ```bash export S3_PATH=s3://{path/to/bucket}/indexes -export IP_NODE_1={first-ec2-instance-public-ip} +export IP_NODE_1={first-ec2-instance-private-ip} ``` +:::note + +Because all three nodes are in the same VPC, set `peer_seeds` to the private IP address of `searcher-1`. In this setup +Quickwit advertises each node's private address, and traffic sent to a public IP does not match the self-referencing +security group rule above. + +::: + ```bash # configuration for our second node echo "version: 0.8 @@ -229,12 +296,14 @@ node_id: searcher-2 metastore_uri: ${S3_PATH} default_index_root_uri: ${S3_PATH} listen_address: 0.0.0.0 +enabled_services: + - searcher peer_seeds: - ${IP_NODE_1} # searcher-1 " > config.yaml # Start a Quickwit searcher. -./quickwit run --service searcher --config config.yaml +./quickwit run --config config.yaml ``` ```bash @@ -242,6 +311,8 @@ peer_seeds: echo "version: 0.8 node_id: searcher-3 listen_address: 0.0.0.0 +enabled_services: + - searcher peer_seeds: - ${IP_NODE_1} # searcher-1 metastore_uri: ${S3_PATH} @@ -249,11 +320,12 @@ default_index_root_uri: ${S3_PATH} " > config.yaml # Start a Quickwit searcher. -./quickwit run --service searcher --config config.yaml +./quickwit run --config config.yaml ``` -You will see in the terminal the confirmation that the instance has joined the existing cluster. Example of such a log: +Each searcher logs that it is joining the cluster. This line is emitted before any peer has been contacted, so on its +own it does not prove that the cluster formed. Example of such a log: ``` 2023-03-19T16:44:56.918Z INFO quickwit_cluster::cluster: Joining cluster. cluster_id=quickwit-default-cluster node_id=searcher-2 enabled_services={Searcher} gossip_listen_addr=0.0.0.0:7280 gossip_advertise_addr=172.31.30.168:7280 grpc_advertise_addr=172.31.30.168:7281 peer_seed_addrs=172.31.91.203:7280 @@ -265,7 +337,13 @@ Now we can query one of our instance directly by issuing http requests to one of curl -v "http://127.0.0.1:7280/api/v1/hdfs-logs/search?query=severity_text:ERROR" ``` -Check out the logs of all instances and you will see that all nodes are working. +To confirm that the cluster actually formed, list its members: + +```bash +curl -s http://127.0.0.1:7280/api/v1/cluster +``` + +The `ready_nodes` array should list `searcher-1`, `searcher-2`, and `searcher-3`. ## Load balancing incoming requests @@ -282,7 +360,10 @@ Let's do some cleanup by deleting the index: ./quickwit index delete --index hdfs-logs ``` -Also remember to remove the security group to protect your EC2 instances. You can just remove the instances if you don't need them. +Then terminate all three EC2 instances and delete the cluster security group. Stopping the instances continues to +incur EBS charges, and when you terminate them, delete any EBS volumes that are not set to delete on termination. The +`index delete` command removes only the `hdfs-logs` index, so if the bucket is dedicated to this tutorial, empty and +delete it to stop its storage charges. Congratulations! You finished this tutorial! diff --git a/docs/get-started/tutorials/tutorial-hdfs-logs.md b/docs/get-started/tutorials/tutorial-hdfs-logs.md index aec3499cf24..b6029342f01 100644 --- a/docs/get-started/tutorials/tutorial-hdfs-logs.md +++ b/docs/get-started/tutorials/tutorial-hdfs-logs.md @@ -33,7 +33,7 @@ Here is an example of a log entry: Let's download and install Quickwit. ```bash -curl -L https://install.quickwit.io | sh +curl -L https://raw.githubusercontent.com/quickwit-oss/quickwit/main/install.sh | sh cd quickwit-v*/ ``` From b5a7f80b11e5646671a89bbcbebef2b4398dcb3e Mon Sep 17 00:00:00 2001 From: David Yaffe Date: Fri, 21 Aug 2026 14:56:57 -0400 Subject: [PATCH 2/2] Clarify bidirectional cluster traffic in AWS S3 tutorial --- .../tutorial-hdfs-logs-distributed-search-aws-s3.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/get-started/tutorials/tutorial-hdfs-logs-distributed-search-aws-s3.md b/docs/get-started/tutorials/tutorial-hdfs-logs-distributed-search-aws-s3.md index e3cf41d2f18..6734e8ddd48 100644 --- a/docs/get-started/tutorials/tutorial-hdfs-logs-distributed-search-aws-s3.md +++ b/docs/get-started/tutorials/tutorial-hdfs-logs-distributed-search-aws-s3.md @@ -260,9 +260,11 @@ In AWS, you can create a security group to group these inbound rules. Check out Create one security group for the Quickwit cluster. Set the security group itself as the source and allow TCP ports 7280 and 7281, plus UDP port 7280. -Attach this group to all three instances, including the `searcher-1` instance you launched earlier. Nodes 2 and 3 -connect to node 1, so node 1 must accept inbound traffic on these ports. If `searcher-1` does not have this group -attached, the other nodes cannot reach it and the cluster does not form. +Attach this group to all three instances, including the `searcher-1` instance you launched earlier. All three nodes +exchange cluster gossip over UDP port 7280 in both directions, and the searcher nodes communicate with each other +over gRPC on TCP port 7281, also in both directions. Each instance must therefore accept inbound traffic from the +other instances in the security group. Cluster communication can fail if an instance does not have this group +attached. Next, create two additional EC2 instances using this security group, and note the private IP address of `searcher-1`.