Repository navigation
Add examples #1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
elijahpetty
wants to merge
24
commits into
main
Choose a base branch
from
add-example
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Add examples #1
Changes from all commits
Commits
Show all changes
24 commits
Select commit
Hold shift + click to select a range
08e0360
.
elijahpetty 2059ecb
.
elijahpetty 1bfc5d3
.
elijahpetty 6da0501
gitignore file
elijahpetty d5c749d
wip
elijahpetty 99946fb
wip
elijahpetty 339df38
.
elijahpetty f111a48
impl suggestions
elijahpetty bc71cf1
Give each example a single clear story and make READMEs usable end-to…
Copilot dfab129
Clarify Example 3, surface sample data, document toml dependencies, s…
Copilot c8d8dff
Fix link for creating command-line tools
chipkent d6fb791
Reject identical batch input and output directories
Copilot 7173ab4
Clarify packaging guide links and examples
Copilot 1c589d6
impl suggestions from claude & kimi reviews, restructure
elijahpetty 211ca32
Tighten batch file selection and streamline packaging guide
Copilot 9f1800f
impl suggestions
elijahpetty 9064aef
impl suggestions
elijahpetty b487cbd
rm guide to move to core repo
elijahpetty acda4a6
Address remaining README review feedback (#2)
margaretkennedy 333eac6
Handle Deephaven startup failures in CLI examples
Copilot 2730abd
pydeephaven
elijahpetty 5377aef
.
elijahpetty fabf324
claude changes
elijahpetty 4136ce5
impl copilot comment
elijahpetty File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,44 @@ | ||
| # Python | ||
| __pycache__/ | ||
| *.py[cod] | ||
| *$py.class | ||
| *.so | ||
| .Python | ||
| build/ | ||
| develop-eggs/ | ||
| dist/ | ||
| downloads/ | ||
| eggs/ | ||
| .eggs/ | ||
| lib/ | ||
| lib64/ | ||
| parts/ | ||
| sdist/ | ||
| var/ | ||
| wheels/ | ||
| *.egg-info/ | ||
| .installed.cfg | ||
| *.egg | ||
| MANIFEST | ||
|
|
||
| # Virtual environments | ||
| venv/ | ||
| env/ | ||
| ENV/ | ||
| .venv | ||
|
|
||
| # IDEs | ||
| .vscode/ | ||
| .idea/ | ||
| *.swp | ||
| *.swo | ||
| *~ | ||
|
|
||
| # OS | ||
| .DS_Store | ||
| Thumbs.db | ||
|
|
||
| # Test results | ||
| results/ | ||
| output/ | ||
| *.log |
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| Name,Score,Value,Category | ||
| Kara,81,130,A | ||
| Liam,94,175,B | ||
| Mona,77,100,A |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| Name,Score,Value,Category | ||
| Nina,86,115,C | ||
| Omar,90,155,B | ||
| Pria,74,80,A |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| Name,Score,Value,Category | ||
| Quinn,93,165,C | ||
| Rosa,84,125,B | ||
| Sam,79,95,A | ||
| Tara,88,145,C |
|
elijahpetty marked this conversation as resolved.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| Name,Score,Value,Category | ||
| Alice,85,120,A | ||
| Bob,92,150,B | ||
| Charlie,78,95,A | ||
| Diana,88,110,C | ||
| Eve,95,180,B | ||
| Frank,72,85,A | ||
| Grace,91,160,C | ||
| Henry,83,105,B | ||
| Iris,89,140,A | ||
| Jack,76,90,C |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,64 @@ | ||
| # My Deephaven CLI | ||
|
|
||
| An example of packaging a Deephaven script as a command line tool. Installing this package creates one terminal command, `my-dh-query`. No library code is exposed; the package is used only through that command, so no Python needs to be written to use it. | ||
|
|
||
| The command is defined by the `[project.scripts]` entry point in [`pyproject.toml`](pyproject.toml): | ||
|
|
||
| ```toml | ||
| [project.scripts] | ||
| my-dh-query = "my_dh_cli.cli:main" | ||
| ``` | ||
|
|
||
| ## Installation | ||
|
|
||
| From the repository root: | ||
|
|
||
| ```bash | ||
| pip install ./my_dh_cli | ||
| ``` | ||
|
|
||
| Or in editable mode for development: | ||
|
|
||
| ```bash | ||
| pip install -e ./my_dh_cli | ||
| ``` | ||
|
|
||
| ## Usage | ||
|
|
||
| Run the installed command on a CSV file. The command starts its own Deephaven server, so no separate setup is needed: | ||
|
|
||
| ```bash | ||
| my-dh-query data/sample.csv --verbose | ||
| ``` | ||
|
|
||
| It reads the file, adds a `DoubleScore` computed column, and reports the number of rows processed. | ||
|
|
||
| The command binds its server to port 10000 by default. If that port is already in use (for example, by Deephaven running in Docker), pass a free port with `--port`. | ||
|
|
||
| During development, the package also runs without an entry point via [`__main__.py`](src/my_dh_cli/__main__.py): | ||
|
|
||
| ```bash | ||
| python -m my_dh_cli data/sample.csv --verbose | ||
| ``` | ||
|
|
||
| ## Command reference | ||
|
|
||
| ### my-dh-query | ||
|
|
||
| Process a CSV file with Deephaven. The file must contain a numeric `Score` column. | ||
|
|
||
| **Arguments:** | ||
|
|
||
| - `input_file` - Path to the CSV file to process. | ||
|
|
||
| **Options:** | ||
|
|
||
| - `--port` - Port for the embedded Deephaven server. Default: `10000`. | ||
| - `--verbose, -v` - Enable verbose output. | ||
|
|
||
| ## Requirements | ||
|
|
||
| - Python 3.9 or later | ||
| - Java 17 or later | ||
| - deephaven-server 0.35.0 or later (installed automatically as a dependency) | ||
| - Click 8.0.0 or later (installed automatically as a dependency) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| [build-system] | ||
| requires = ["setuptools>=61.0", "wheel"] | ||
| build-backend = "setuptools.build_meta" | ||
|
|
||
| [project] | ||
| name = "my_dh_cli" | ||
| version = "0.1.0" | ||
| description = "Command line tool for data processing" | ||
| readme = "README.md" | ||
| requires-python = ">=3.9" | ||
| dependencies = [ | ||
| # deephaven-server also provides the deephaven module (through its deephaven-core dependency). | ||
| "deephaven-server>=0.35.0", | ||
| # click implements the command line interface. | ||
| "click>=8.0.0", | ||
| ] | ||
|
|
||
| [project.scripts] | ||
| my-dh-query = "my_dh_cli.cli:main" | ||
|
|
||
| [tool.setuptools.packages.find] | ||
| where = ["src"] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| """Command line tool that processes a CSV file with Deephaven.""" | ||
|
|
||
| __version__ = "0.1.0" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| from my_dh_cli.cli import main | ||
|
|
||
| if __name__ == "__main__": | ||
| main() |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,68 @@ | ||
| import click | ||
|
|
||
|
|
||
| def my_dh_query(input_file: str, verbose: bool = False): | ||
| """Read a CSV file and perform a simple query operation on the data.""" | ||
| # Imported here, not at module level: deephaven requires a running server. | ||
| # The entry point starts the server first, then calls this function. | ||
| from deephaven import read_csv | ||
| from pathlib import Path | ||
|
|
||
| input_path = Path(input_file) | ||
|
|
||
| if not input_path.exists(): | ||
| raise click.ClickException(f"Input file does not exist: '{input_path}'") | ||
| if not input_path.is_file(): | ||
| raise click.ClickException(f"Input path is not a file: '{input_path}'") | ||
|
|
||
| if verbose: | ||
| click.echo(f"Processing {input_file}...") | ||
|
|
||
| try: | ||
| source = read_csv(input_file) | ||
| except Exception as e: | ||
| raise click.ClickException(f"Failed to read CSV file '{input_file}': {e}") | ||
|
|
||
| column_names = [col.name for col in source.columns] | ||
| if "Score" not in column_names: | ||
| raise click.ClickException( | ||
| f"File '{input_path.name}' is missing required column 'Score'. " | ||
| f"Available columns: {', '.join(column_names)}" | ||
| ) | ||
|
|
||
| try: | ||
| result = source.update(formulas=["DoubleScore = Score * 2"]) | ||
| except Exception: | ||
| raise click.ClickException( | ||
| f"Failed to compute DoubleScore for '{input_path.name}'. " | ||
| "The 'Score' column must be numeric." | ||
| ) | ||
|
|
||
| if verbose: | ||
| click.echo(f"Processed {result.size} rows") | ||
|
|
||
| return result | ||
|
|
||
|
|
||
| @click.command() | ||
| @click.argument("input_file", type=click.Path(exists=True)) | ||
| @click.option("--port", default=10000, show_default=True, help="Port for the embedded Deephaven server") | ||
| @click.option("--verbose", "-v", is_flag=True, help="Enable verbose output") | ||
| def main(input_file: str, port: int, verbose: bool) -> None: | ||
| """Process data with Deephaven.""" | ||
| try: | ||
| from deephaven_server import Server | ||
|
|
||
| server = Server(port=port, jvm_args=["-Xmx4g"]) | ||
| server.start() | ||
| except Exception as e: | ||
| raise click.ClickException( | ||
| f"Failed to start Deephaven server on port {port}: {e}" | ||
| ) | ||
|
|
||
| my_dh_query(input_file, verbose) | ||
| click.echo("Processing complete!") | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| main() |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,83 @@ | ||
| # My Deephaven Client | ||
|
|
||
| An example of packaging a [`pydeephaven`](https://pypi.org/project/pydeephaven/) client program as a command line tool. Installing this package creates one terminal command, `my-dh-client`, which connects to a Deephaven server that is already running, uploads a CSV file to it, processes the data there, and binds the result under a name so it appears in the server's IDE. | ||
|
|
||
| Compare it with [`my_dh_cli`](../my_dh_cli/), which does similar work with an embedded server. The two differ in three ways: | ||
|
|
||
| - The dependency is `pydeephaven` rather than `deephaven-server`, so installing it does not pull in a JVM and the command starts quickly. | ||
| - The command does not start a server. It needs one to connect to, and many copies of the command can run against the same server at once. | ||
| - `pydeephaven` is imported at the top of `cli.py`. The embedded-server command has to delay its `deephaven` import until after the server starts; the client has no such constraint. | ||
|
|
||
| The command is defined by the `[project.scripts]` entry point in [`pyproject.toml`](pyproject.toml): | ||
|
|
||
| ```toml | ||
| [project.scripts] | ||
| my-dh-client = "my_dh_client.cli:main" | ||
| ``` | ||
|
|
||
| ## Installation | ||
|
|
||
| From the repository root: | ||
|
|
||
| ```bash | ||
| pip install ./my_dh_client | ||
| ``` | ||
|
|
||
| Or in editable mode for development: | ||
|
|
||
| ```bash | ||
| pip install -e ./my_dh_client | ||
| ``` | ||
|
|
||
| ## Usage | ||
|
|
||
| > [!NOTE] | ||
| > A Deephaven server must already be running. By default the command connects to `localhost:10000` with anonymous authentication. See [Start a server for the client examples](../README.md#start-a-server-for-the-client-examples) in the repository README for ways to start one, and for connecting to a server that uses a pre-shared key. | ||
|
|
||
| Run the installed command on a CSV file: | ||
|
|
||
| ```bash | ||
| my-dh-client data/sample.csv --verbose | ||
| ``` | ||
|
|
||
| It reads the file locally with pyarrow, uploads it to the server, adds a `DoubleScore` computed column there, and binds the result as a table named `sample` (the file's stem). Open the server's IDE at `http://localhost:10000` to see the table, or pick a different name with `--name`. | ||
|
|
||
| To connect to a different server, pass `--host` and `--port`. For a server that requires a token, pass `--auth-type` and put the token in the `DH_AUTH_TOKEN` environment variable. Set the variable without typing the token into a command, so it stays out of your shell history; for example, enter it at a hidden prompt with `read -s`: | ||
|
|
||
| ```bash | ||
| read -s DH_AUTH_TOKEN && export DH_AUTH_TOKEN | ||
| my-dh-client data/sample.csv \ | ||
| --host dh.example.com \ | ||
| --auth-type io.deephaven.authentication.psk.PskAuthenticationHandler | ||
| ``` | ||
|
|
||
| During development, the package also runs without an entry point via [`__main__.py`](src/my_dh_client/__main__.py): | ||
|
|
||
| ```bash | ||
| python -m my_dh_client data/sample.csv --verbose | ||
| ``` | ||
|
|
||
| ## Command reference | ||
|
|
||
| ### my-dh-client | ||
|
|
||
| Upload a CSV file to a running Deephaven server and process it there. The file must contain a numeric `Score` column. | ||
|
|
||
| **Arguments:** | ||
|
|
||
| - `input_file` - Path to the CSV file to upload. | ||
|
|
||
| **Options:** | ||
|
|
||
| - `--host` - Deephaven server host. Default: `localhost`. | ||
| - `--port` - Deephaven server port. Default: `10000`. | ||
| - `--auth-type` - Authentication type. Default: `Anonymous`. For other types, set the token in the `DH_AUTH_TOKEN` environment variable. | ||
| - `--name` - Name to bind the result table under on the server. Default: the input file's stem. | ||
| - `--verbose, -v` - Enable verbose output. | ||
|
|
||
| ## Requirements | ||
|
|
||
| - Python 3.9 or later | ||
| - pydeephaven 0.35.0 or later (installed automatically as a dependency) | ||
| - Click 8.0.0 or later (installed automatically as a dependency) | ||
| - A running Deephaven server to connect to. Java is not required on the client machine. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| [build-system] | ||
| requires = ["setuptools>=61.0", "wheel"] | ||
| build-backend = "setuptools.build_meta" | ||
|
|
||
| [project] | ||
| name = "my_dh_client" | ||
| version = "0.1.0" | ||
| description = "Command line client for a running Deephaven server" | ||
| readme = "README.md" | ||
| requires-python = ">=3.9" | ||
| dependencies = [ | ||
| # pydeephaven is the Python client. It connects to a running server and does not start one. | ||
| "pydeephaven>=0.35.0", | ||
| # pyarrow reads the local CSV file. pydeephaven depends on it too, but the command | ||
| # imports it directly, so it is declared here rather than relied on transitively. | ||
| "pyarrow", | ||
| # click implements the command line interface. | ||
| "click>=8.0.0", | ||
| ] | ||
|
|
||
| [project.scripts] | ||
| my-dh-client = "my_dh_client.cli:main" | ||
|
|
||
| [tool.setuptools.packages.find] | ||
| where = ["src"] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| """Command line client that uploads a CSV file to a running Deephaven server.""" | ||
|
|
||
| __version__ = "0.1.0" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| from my_dh_client.cli import main | ||
|
|
||
| if __name__ == "__main__": | ||
| main() |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.