[docs] Add HTTP Gateway lakehouse quickstart - #4231
Conversation
pranavshuklaa
changed the title
|
Could someone please review this PR when you get a chance? It adds the HTTP Gateway lakehouse quickstart for #4221. Thanks!! |
pbanakar
left a comment
pbanakar
left a comment
There was a problem hiding this comment.
Hi @pranavshuklaa, thanks for the PR! left some comments
| @@ -0,0 +1,490 @@ | |||
| --- | |||
| title: Real-Time Lakehouse via the HTTP Gateway | |||
| sidebar_position: 3 | |||
There was a problem hiding this comment.
check if this conflicts with an existing quickstart. Our page_user_profile.md is at 4 and lakehouse.md and security.md both using sidebar_position 2 could you correct it? currently no file at 3?
There was a problem hiding this comment.
Okay so in my latest commit I am fixing the underlying collision rather than just slotting around it. Renumbered so every quickstart page has a unique, sequential position: flink.md=1, lakehouse.md=2, security.md=3 (was 2), this page gateway-lakehouse.md=4 (was 3), page_user_profile.mdx=5 (was 4). Pushed as part of this PR.
| (however, Compose v1 might work with a few adaptions). | ||
| ::: | ||
|
|
||
| ### Build the Gateway image |
There was a problem hiding this comment.
every other quickstart uses a pre-built Docker Hub image. This one requires the user to have the Fluss source repo checked out locally and run docker/fluss-gateway/build.sh. That's not a quickstart experience that's a dev setup. The guide should either wait for a published Gateway image or clearly state this is a developer preview requiring a source build upfront, and this feature is for 1.0v that is not released yet
There was a problem hiding this comment.
Yeah agreed. Added a statement that this is a developer preview requiring a source build. I think this solves it:
This guide covers a feature shipping in the upcoming fluss 1.0 release.
A pre-built Docker Hub image will be available at GA - until then,
building it locally takes one extra step: run the script below from the
root of your ...
| ### Quitting SQL Client | ||
|
|
||
| ```sql title="Flink SQL" | ||
| quit; |
There was a problem hiding this comment.
all other Fluss quickstarts use exit;
There was a problem hiding this comment.
Fixed this. switched to exit; to match the convention in the other quickstarts.
beryllw
left a comment
beryllw
left a comment
There was a problem hiding this comment.
Could we also link to this quickstart from the Gateway overview so readers can easily find the end-to-end example?
| REST API instead of Flink SQL. You'll create a datalake-enabled table with | ||
| `curl`, ingest JSON records with `curl`, then use Flink SQL only to run the | ||
| Lakehouse Tiering Service and query the unified real-time + historical data | ||
| (Union Read). |
There was a problem hiding this comment.
Could we introduce a concrete application scenario and clarify that the application writes through HTTP without separately writing to Paimon?
| ``` | ||
|
|
||
| ```sql title="Flink SQL" | ||
| SELECT order_id, customer, amount_cents, status FROM orders$lake; |
There was a problem hiding this comment.
Could we query orders before orders$lake to highlight that querying does not need to wait for tiering?
| performance, but reflecting only what's been tiered so far. | ||
|
|
||
| ```sql title="Flink SQL" | ||
| SELECT order_id, customer, amount_cents, status FROM orders; |
There was a problem hiding this comment.
Could we include expected SQL results, lake row counts, and a small comparison of the two query forms?
| starts, not once it's actually accepting connections, so the first | ||
| `/ready` call (or the first database-creation call below) can briefly | ||
| return an error or HTTP 503 right after `docker compose up`. Retry after a | ||
| few seconds if that happens. |
There was a problem hiding this comment.
Could we wait for a successful metadata request with a timeout? /ready alone does not verify Fluss connectivity.
| Enter the Flink SQL CLI container: | ||
|
|
||
| ```shell | ||
| docker compose run sql-client |
There was a problem hiding this comment.
Could we use flink run -d and docker compose run --rm sql-client, and explain that the non-interactive SQL client container may exit normally?
| After finishing the tutorial, run `exit` to exit the Flink SQL CLI | ||
| container. | ||
|
|
||
| ## Preview limitations |
There was a problem hiding this comment.
Could we add a short takeaway explaining which application-side components are no longer needed, while clarifying that the shared Tiering Service still runs on Flink?
|
I put together a similar end-to-end demo earlier: HTTP Gateway lakehouse quickstart. Feel free to use it as a reference. |
3 participants
Purpose
Linked issue: close #4221
Add an end-to-end quickstart showing how to build a real-time lakehouse using the Fluss HTTP Gateway, Fluss, Flink, Paimon, and RustFS.
Brief change log
$lakeand$lake$snapshots.Tests
API and Format
No API or storage format changes.
Documentation
Adds a new HTTP Gateway quickstart for a real-time lakehouse and documents the existing Gateway REST API, Fluss datalake integration, Flink Union Read, and Paimon tiering workflow.