diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index ba4e68c831c..827ab0ad3ee 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -106,7 +106,7 @@ As the react UI is embedded in the rust binary, we need to build the react app b ## Testing release (alpha, beta, rc) -The following Quickwit installation command `curl -L https://install.quickwit.io | sh` always installs the latest stable version of quickwit. To make it easier in installing and testing new (alpha, beta, rc) releases, you can manually pull and execute the script as `./install.sh --allow-any-latest-version`. This will force the script to install any latest available release package. +The following Quickwit installation command `curl -L https://raw.githubusercontent.com/quickwit-oss/quickwit/main/install.sh | sh` always installs the latest stable version of quickwit. To make it easier in installing and testing new (alpha, beta, rc) releases, you can manually pull and execute the script as `./install.sh --allow-any-latest-version`. This will force the script to install any latest available release package. ## Tracking licenses diff --git a/distribution/docker/ubuntu/Dockerfile b/distribution/docker/ubuntu/Dockerfile index a3ddc75df00..a8e24c6ce28 100644 --- a/distribution/docker/ubuntu/Dockerfile +++ b/distribution/docker/ubuntu/Dockerfile @@ -1,7 +1,7 @@ FROM ubuntu:noble@sha256:66460d557b25769b102175144d538d88219c077c678a49af4afca6fbfc1b5252 AS builder RUN apt-get update && apt-get install -y curl -RUN curl -L https://install.quickwit.io | sh +RUN curl -L https://raw.githubusercontent.com/quickwit-oss/quickwit/main/install.sh | sh FROM ubuntu:noble@sha256:66460d557b25769b102175144d538d88219c077c678a49af4afca6fbfc1b5252 AS quickwit diff --git a/docs/get-started/installation.md b/docs/get-started/installation.md index 52f1097e035..277a06b1ff9 100644 --- a/docs/get-started/installation.md +++ b/docs/get-started/installation.md @@ -139,7 +139,7 @@ To easily install Quickwit on your machine, just run the command below from your The script detects the architecture and then downloads the correct binary archive for the machine. ```bash -curl -L https://install.quickwit.io | sh +curl -L https://raw.githubusercontent.com/quickwit-oss/quickwit/main/install.sh | sh ``` All this script does is download the correct binary archive for your machine and extracts it in the current working directory. This means you can download any desired archive from [github](https://github.com/quickwit-oss/quickwit/releases) that matches your OS architecture and manually extract it anywhere. diff --git a/docs/get-started/quickstart.md b/docs/get-started/quickstart.md index 311fbaf6c03..64e23ae8266 100644 --- a/docs/get-started/quickstart.md +++ b/docs/get-started/quickstart.md @@ -14,7 +14,7 @@ The Quickwit installer automatically picks the correct binary archive for your e This method works only for [some OS/architectures](installation.md#download), and you will also need to install some [external dependencies](installation.md#note-on-external-dependencies). ```bash -curl -L https://install.quickwit.io | sh +curl -L https://raw.githubusercontent.com/quickwit-oss/quickwit/main/install.sh | sh ``` ```bash diff --git a/docs/get-started/tutorials/tutorial-hdfs-logs-distributed-search-aws-s3.md b/docs/get-started/tutorials/tutorial-hdfs-logs-distributed-search-aws-s3.md index 7e444e08b7e..6734e8ddd48 100644 --- a/docs/get-started/tutorials/tutorial-hdfs-logs-distributed-search-aws-s3.md +++ b/docs/get-started/tutorials/tutorial-hdfs-logs-distributed-search-aws-s3.md @@ -6,6 +6,9 @@ icon_url: /img/tutorials/aws-logo.png sidebar_position: 6 --- +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + In this guide, we will index about 20 million log entries (7 GB decompressed) on AWS S3 using an EC2 instance and launch a three-node distributed search cluster. Example of a log entry: @@ -34,7 +37,7 @@ First of all, let's create an EC2 instance, install a Quickwit binary, and [conf ## Install ```bash -curl -L https://install.quickwit.io | sh +curl -L https://raw.githubusercontent.com/quickwit-oss/quickwit/main/install.sh | sh cd quickwit-v*/ ``` @@ -47,8 +50,10 @@ export S3_PATH=s3://{path/to/bucket}/indexes ``` :::note -You'll want to include the necessary authorization for the given bucket, this can be done by setting the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` -environment variables, or via the AWS credentials file. Usually located at `~/.aws/credentials`. +Quickwit needs access to the bucket and the objects it contains. On EC2, attach an IAM role through an instance +profile. Quickwit picks up those credentials automatically. The role needs `s3:ListBucket` on the bucket, plus +`s3:GetObject`, `s3:PutObject`, `s3:DeleteObject`, `s3:ListMultipartUploadParts`, and `s3:AbortMultipartUpload` on +its objects. For more information, check out [our AWS setup guide](../../guides/aws-setup). ::: @@ -60,6 +65,12 @@ Now we can create a Quickwit config file. echo "version: 0.8 node_id: searcher-1 listen_address: 0.0.0.0 +enabled_services: + - metastore + - control_plane + - searcher + - indexer + - janitor metastore_uri: ${S3_PATH} default_index_root_uri: ${S3_PATH} " > config.yaml @@ -75,12 +86,23 @@ default_index_root_uri: ${S3_PATH} > default_index_root_uri: ${S3_PATH} >``` +`enabled_services` controls which services run on each node. Here `searcher-1` runs the five services listed above, +while nodes 2 and 3 run only the searcher. + We are now ready to start Quickwit. ```bash ./quickwit run --config config.yaml ``` +:::note + +`quickwit run` stays in the foreground. Keep it running and open a second connection to the same instance for the +index and ingest commands that follow. To keep access to the node logs after disconnecting, run it under `tmux` or +redirect its output with `nohup`. + +::: + ## Create your index ```bash @@ -136,7 +158,7 @@ We can now create the index with the `create` subcommand. :::note The `create` command sends the index configuration to the running Quickwit node. The node stores the file-backed -metastore at `s3://path-to-your-bucket/hdfs-logs/metastore.json`. +metastore at `s3://path-to-your-bucket/indexes/hdfs-logs/metastore.json`, under the prefix set in `S3_PATH`. To run this command from your local machine, add `--endpoint http://:7280`. @@ -164,11 +186,41 @@ To run this command from your local machine, add `--endpoint http:// + + + ```bash ./quickwit index search --index hdfs-logs --query "severity_text:ERROR" ``` + + + + +```bash +curl "http://127.0.0.1:7280/api/v1/hdfs-logs/search?query=severity_text:ERROR" +``` + + + + + which returns the json ```json @@ -205,13 +257,22 @@ Also, it needs `{rest.listen_port} + 1` for gRPC communication between instances In AWS, you can create a security group to group these inbound rules. Check out the [network section](../../guides/aws-setup) of our AWS setup guide. -To make things easier, let's create a security group that opens the TCP/UDP port range [7200-7300]. -Next, create two additional EC2 instances using the previously created security group. Take note of each instance's public IP address. +Create one security group for the Quickwit cluster. Set the security group itself as the source and allow TCP ports +7280 and 7281, plus UDP port 7280. + +Attach this group to all three instances, including the `searcher-1` instance you launched earlier. All three nodes +exchange cluster gossip over UDP port 7280 in both directions, and the searcher nodes communicate with each other +over gRPC on TCP port 7281, also in both directions. Each instance must therefore accept inbound traffic from the +other instances in the security group. Cluster communication can fail if an instance does not have this group +attached. + +Next, create two additional EC2 instances using this security group, and note the private IP address of `searcher-1`. -SSH into the second and third EC2 instances, install Quickwit, and [configure the environment](../../guides/aws-setup) to let Quickwit access the index S3 bucket. +Connect to the second and third EC2 instances, install Quickwit, and +[configure AWS access](../../guides/aws-setup) for the index bucket. ```bash -curl -L https://install.quickwit.io | sh +curl -L https://raw.githubusercontent.com/quickwit-oss/quickwit/main/install.sh | sh cd quickwit-v*/ ``` @@ -219,9 +280,17 @@ And configure the environment so instances can form a cluster: ```bash export S3_PATH=s3://{path/to/bucket}/indexes -export IP_NODE_1={first-ec2-instance-public-ip} +export IP_NODE_1={first-ec2-instance-private-ip} ``` +:::note + +Because all three nodes are in the same VPC, set `peer_seeds` to the private IP address of `searcher-1`. In this setup +Quickwit advertises each node's private address, and traffic sent to a public IP does not match the self-referencing +security group rule above. + +::: + ```bash # configuration for our second node echo "version: 0.8 @@ -229,12 +298,14 @@ node_id: searcher-2 metastore_uri: ${S3_PATH} default_index_root_uri: ${S3_PATH} listen_address: 0.0.0.0 +enabled_services: + - searcher peer_seeds: - ${IP_NODE_1} # searcher-1 " > config.yaml # Start a Quickwit searcher. -./quickwit run --service searcher --config config.yaml +./quickwit run --config config.yaml ``` ```bash @@ -242,6 +313,8 @@ peer_seeds: echo "version: 0.8 node_id: searcher-3 listen_address: 0.0.0.0 +enabled_services: + - searcher peer_seeds: - ${IP_NODE_1} # searcher-1 metastore_uri: ${S3_PATH} @@ -249,11 +322,12 @@ default_index_root_uri: ${S3_PATH} " > config.yaml # Start a Quickwit searcher. -./quickwit run --service searcher --config config.yaml +./quickwit run --config config.yaml ``` -You will see in the terminal the confirmation that the instance has joined the existing cluster. Example of such a log: +Each searcher logs that it is joining the cluster. This line is emitted before any peer has been contacted, so on its +own it does not prove that the cluster formed. Example of such a log: ``` 2023-03-19T16:44:56.918Z INFO quickwit_cluster::cluster: Joining cluster. cluster_id=quickwit-default-cluster node_id=searcher-2 enabled_services={Searcher} gossip_listen_addr=0.0.0.0:7280 gossip_advertise_addr=172.31.30.168:7280 grpc_advertise_addr=172.31.30.168:7281 peer_seed_addrs=172.31.91.203:7280 @@ -265,7 +339,13 @@ Now we can query one of our instance directly by issuing http requests to one of curl -v "http://127.0.0.1:7280/api/v1/hdfs-logs/search?query=severity_text:ERROR" ``` -Check out the logs of all instances and you will see that all nodes are working. +To confirm that the cluster actually formed, list its members: + +```bash +curl -s http://127.0.0.1:7280/api/v1/cluster +``` + +The `ready_nodes` array should list `searcher-1`, `searcher-2`, and `searcher-3`. ## Load balancing incoming requests @@ -282,7 +362,10 @@ Let's do some cleanup by deleting the index: ./quickwit index delete --index hdfs-logs ``` -Also remember to remove the security group to protect your EC2 instances. You can just remove the instances if you don't need them. +Then terminate all three EC2 instances and delete the cluster security group. Stopping the instances continues to +incur EBS charges, and when you terminate them, delete any EBS volumes that are not set to delete on termination. The +`index delete` command removes only the `hdfs-logs` index, so if the bucket is dedicated to this tutorial, empty and +delete it to stop its storage charges. Congratulations! You finished this tutorial! diff --git a/docs/get-started/tutorials/tutorial-hdfs-logs.md b/docs/get-started/tutorials/tutorial-hdfs-logs.md index aec3499cf24..b6029342f01 100644 --- a/docs/get-started/tutorials/tutorial-hdfs-logs.md +++ b/docs/get-started/tutorials/tutorial-hdfs-logs.md @@ -33,7 +33,7 @@ Here is an example of a log entry: Let's download and install Quickwit. ```bash -curl -L https://install.quickwit.io | sh +curl -L https://raw.githubusercontent.com/quickwit-oss/quickwit/main/install.sh | sh cd quickwit-v*/ ```