Skip to content

Commit 4d1fbde

Browse files
sp16743sp16743
authored
Merge pull request #85 from NetApp/feature/CSTACKEX-237
feature/CSTACKEX-237: Pilot changes for scalebenmark framework
2 parents 2ccdb23 + 2ee0413 commit 4d1fbde

18 files changed

Lines changed: 2893 additions & 0 deletions

‎private-cicd/benchmark/ontap/.gitignore‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
18+
config.yaml
19+
config_sanity.yaml
20+
.venv/
21+
__pycache__/
22+
*.pyc
23+
results/*.csv
24+
results/*.md
25+
results/*.log

‎private-cicd/benchmark/ontap/AGENTS.md‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
<!--
2+
Licensed to the Apache Software Foundation (ASF) under one
3+
or more contributor license agreements. See the NOTICE file
4+
distributed with this work for additional information
5+
regarding copyright ownership. The ASF licenses this file
6+
to you under the Apache License, Version 2.0 (the
7+
"License"); you may not use this file except in compliance
8+
with the License. You may obtain a copy of the License at
9+
10+
http://www.apache.org/licenses/LICENSE-2.0
11+
12+
Unless required by applicable law or agreed to in writing,
13+
software distributed under the License is distributed on an
14+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
KIND, either express or implied. See the License for the
16+
specific language governing permissions and limitations
17+
under the License.
18+
-->
19+
# ONTAP scale benchmark
20+
21+
Scale matrix: [ONTAP Plugin - CloudStack Operations, Scale & Parallel Test Matrix](https://netapp.atlassian.net/wiki/spaces/OSSG/pages/608854350).
22+
23+
Test ids written to the CSVs and report map to the page sections as follows:
24+
25+
| Test id | Page section | What it measures |
26+
|---|---|---|
27+
| `pool_sequential_create` | 5.1.1 | Sequential create (1 pool at a time, cumulative to N) |
28+
| `pool_sequential_delete` | 5.1.2 | Sequential delete (1 pool at a time, from N remaining) |
29+
| `pool_concurrent_create` | 6.1.1 | Parallel pool creation |
30+
| `pool_concurrent_delete` | 6.1.2 | Parallel pool deletion |
31+
| `vm_sequential_create` | 5.2.1 | Sequential create (1 VM at a time, cumulative to N) |
32+
| `vm_sequential_delete` | 5.2.2 | Sequential delete (1 VM at a time, from N remaining) |
33+
| `vm_concurrent_create` | 6.2.1 | Parallel VM creation |
34+
| `vm_concurrent_delete` | 6.2.2 | Parallel VM deletion |
35+
| Report "Results log" table | 9 | Results log |
36+
37+
Keep section numbers and the page link out of the code. How to run each script is in `README.md` and `USAGE.md`.
38+
- Protocols, checkpoints, and concurrency levels live in `config.yaml`. `run.py` is the entry point. Do not add a CLI flag for a field that already exists in config.
39+
- Use `benchmark_support` for run ids, resource names, `mean_seconds`, and logging. Sequential and concurrency scripts must use the same helpers and the same log format. Do not hand-roll a second average or a second logger.
40+
- Run ids are letters, digits, and underscores. Storage pool names disallow `-`. VM hostnames disallow `_`; `format_vm_name` / `vm_name_token` are the only rewrite, and cleanup must use them so created VMs are the ones deleted.
41+
- Cleanup must stay inside the configured name prefix, zone, cluster (pools), and storage provider. Do not delete a resource just because its name contains a short substring.

‎private-cicd/benchmark/ontap/README.md‎

Lines changed: 233 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,233 @@
1+
<!--
2+
Licensed to the Apache Software Foundation (ASF) under one
3+
or more contributor license agreements. See the NOTICE file
4+
distributed with this work for additional information
5+
regarding copyright ownership. The ASF licenses this file
6+
to you under the Apache License, Version 2.0 (the
7+
"License"); you may not use this file except in compliance
8+
with the License. You may obtain a copy of the License at
9+
10+
http://www.apache.org/licenses/LICENSE-2.0
11+
12+
Unless required by applicable law or agreed to in writing,
13+
software distributed under the License is distributed on an
14+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
KIND, either express or implied. See the License for the
16+
specific language governing permissions and limitations
17+
under the License.
18+
-->
19+
# ONTAP plugin - storage pool benchmark (Step 1)
20+
21+
> Looking for a quick "what does script X do / how do I run it" cheat
22+
> sheet instead of the full design write-up below? See [USAGE.md](USAGE.md).
23+
24+
Downstream-only benchmarking tool (not for Apache upstream) that drives
25+
`createStoragePool` / `deleteStoragePool` over the CloudStack HTTP/REST API
26+
to reproduce the **Storage Pool** rows of the Sequential Scale Matrix (5.1)
27+
and Parallel/Concurrency Matrix (6.1) from the Confluence page
28+
["ONTAP Plugin - CloudStack Operations, Scale & Parallel Test Matrix"](https://netapp.atlassian.net/wiki/spaces/OSSG/pages/608854350).
29+
30+
This is step 1 of the automation follow-up (storage pool only). VM instance
31+
benchmarks are now also covered (see below); volume/snapshot benchmarks will
32+
be added as separate scripts later, reusing `cloudstack_client.py`.
33+
34+
The storage-pool benchmark is split into **two independent scripts** (on
35+
purpose — a sequential run is cheap/safe and worth reviewing before deciding
36+
to launch a concurrency run, which is the one most likely to stress the
37+
mgmt-server/plugin job queue):
38+
39+
- `benchmark_storage_pool_sequential.py` — Section 5.1 (5.1.1 / 5.1.2)
40+
- `benchmark_storage_pool_concurrency.py` — Section 6.1 (6.1.1 / 6.1.2)
41+
42+
Both share the same createStoragePool/deleteStoragePool call shape, CSV
43+
formats, and cleanup logic via `storage_pool_common.py`. Run them with the
44+
**same `--run-id`** to accumulate both into a single
45+
`results/summary_<run_id>.csv` (and therefore a single `render_report.py`
46+
output covering 5.1.x and 6.1.x together).
47+
48+
The VM-instance benchmark is split into **three scripts** sharing
49+
`vm_instance_common.py` (same call shape/CSV formats/cleanup logic):
50+
51+
- `benchmark_vm_instance_sequential.py` — Section 5.2 (5.2.1 / 5.2.2)
52+
- `benchmark_vm_instance_concurrency.py` — Section 6.2 (6.2.1 / 6.2.2)
53+
- `benchmark_vm_instance_combined.py` — runs both of the above back to back
54+
under one shared `--run-id`, for when you already trust the config/
55+
environment and just want the full 5.2.x + 6.2.x matrix in one invocation
56+
57+
See "VM instance benchmark" below for prerequisites and usage.
58+
59+
## What it does
60+
61+
1. **Sequential scale test (5.1.1 / 5.1.2)** — for each protocol (NFS3,
62+
iSCSI), creates storage pools one at a time up to N=30, logging every
63+
single create call, and reports cumulative-total/avg-per-op timing at
64+
checkpoints N = 1, 5, 10, 20, 30. Then deletes them one at a time and
65+
reports the same at "N remaining" = 30, 20, 10, 5, 1.
66+
2. **Concurrency test (6.1.1 / 6.1.2)** — for each concurrency level
67+
C = 2, 5, 10, 20, 30 and each protocol, creates C pools in parallel
68+
(`ThreadPoolExecutor`), records wall-clock time for the whole batch plus
69+
success/failure/avg-per-op, then deletes the same C pools in parallel.
70+
3. Writes **every** individual API call (start/end timestamp, duration,
71+
success/failure, pool id, error) to `results/raw_ops_<run_id>.csv`, and a
72+
checkpoint-level roll-up to `results/summary_<run_id>.csv`.
73+
4. `render_report.py` turns those two CSVs into markdown tables that match
74+
the Confluence page's table layout (including min/avg/p95/max stats for
75+
the "Results Log" section), ready to paste back into the page.
76+
77+
## Setup
78+
79+
```bash
80+
cd private-cicd/benchmark/ontap
81+
python3 -m venv .venv && source .venv/bin/activate
82+
pip install -r requirements.txt
83+
cp config.example.yaml config.yaml
84+
```
85+
86+
Edit `config.yaml`:
87+
88+
- `cloudstack.*` — API URL, admin username/password of your lab's
89+
management server.
90+
- `infrastructure.*` — zone/pod/cluster UUIDs to create pools under.
91+
- `ontap.nfs3` / `ontap.iscsi` — connection details of the ONTAP SVM(s) you
92+
want to benchmark against for each protocol. Delete a block if you only
93+
have one protocol available.
94+
- `benchmark.*` — checkpoints, concurrency levels, pool name prefix, output
95+
directory. Defaults match the Confluence page (N/C = 1,5,10,20,30 /
96+
2,5,10,20,30).
97+
98+
`config.yaml` and everything under `results/` are gitignored — never commit
99+
real lab IPs/credentials or run output.
100+
101+
## Running
102+
103+
`run.py` is the entry point. Protocols, checkpoints, and concurrency levels
104+
come from `config.yaml` (`ontap` / `benchmark` for pools, `vm_bench` for
105+
VMs). Set `enabled_protocols` under `benchmark` or `vm_bench` to select
106+
`[nfs3]`, `[iscsi]`, or `[nfs3, iscsi]`, and edit `sequential_checkpoints` or
107+
`concurrency_levels` to change the matrix.
108+
109+
Run ids are generated as `RUN_<UTC timestamp>_<4 hex chars>` (for example
110+
`RUN_20260927_081500_a1b2`). Letters, digits, and underscores only: a hyphen
111+
is rewritten to `_` because it is embedded in the storage pool name. Reuse
112+
one id across sequential and concurrency when you want a single report.
113+
114+
```bash
115+
# Sanity-check without hitting a real management server:
116+
python3 run.py --test storage-pool-sequential --config config.yaml --dry-run
117+
python3 run.py --test storage-pool-concurrency --config config.yaml --dry-run
118+
119+
# Sequential scale matrix, using the protocols and checkpoints in config.yaml:
120+
python3 run.py --test storage-pool-sequential --config config.yaml
121+
122+
# Concurrency matrix, using config.yaml's concurrency_levels:
123+
python3 run.py --test storage-pool-concurrency --config config.yaml
124+
125+
# Same --run-id across both tests to combine 5.1.x + 6.1.x into one report.
126+
# Run sequential first, review it, then decide on concurrency:
127+
python3 run.py --test storage-pool-sequential --config config.yaml --run-id RUN_20260927_081500_a1b2
128+
python3 run.py --test storage-pool-concurrency --config config.yaml --run-id RUN_20260927_081500_a1b2
129+
```
130+
131+
At the end of a real run each script deletes leftover pools from that run id.
132+
The sweep only deletes pools that match this config's name prefix, name
133+
pattern, zone, cluster, and storage provider, so other pools in a shared zone
134+
are left alone. Pass `--skip-cleanup` to inspect the pools yourself.
135+
136+
If a run crashes and leaves pools behind:
137+
138+
```bash
139+
# Deletes benchmark pools for this config's pool_name_prefix (still provider/zone/cluster scoped):
140+
python3 run.py --test storage-pool-sequential --config config.yaml --cleanup-only
141+
142+
# Or scope it to one specific run (underscores, not hyphens):
143+
python3 run.py --test storage-pool-sequential --config config.yaml --cleanup-only RUN_20260927_081500_a1b2
144+
```
145+
146+
## Rendering the Confluence-ready report
147+
148+
```bash
149+
python3 render_report.py --run-id RUN_20260722_101500_a1b2 \
150+
--cloudstack-build 4.23.0.0-SNAPSHOT --ontap-version 9.15.1
151+
```
152+
153+
This prints markdown tables for 5.1.1, 5.1.2, 6.1.1, 6.1.2, and a set of
154+
pre-filled rows for the page's Section 9 "Results Log" table (with Min/Avg/
155+
P95/Max computed from the raw per-op log), and also saves them to
156+
`results/report_<run_id>.md`.
157+
158+
## VM instance benchmark
159+
160+
Drives `deployVirtualMachine` / `destroyVirtualMachine` to reproduce the
161+
**VM Instance** rows of the same Confluence page's Sequential Scale Matrix
162+
(5.2) and Parallel/Concurrency Matrix (6.2). Every VM gets a root disk (from
163+
the service offering) AND a data disk (from the disk offering), both landing
164+
on the same tagged storage pool.
165+
166+
### Prerequisites (one-time setup, per protocol)
167+
168+
Unlike the storage-pool benchmark's transient pools, the VM-instance
169+
benchmark needs a **persistent, pre-created** `bench_vm_<protocol>` pool plus
170+
matching offerings - it does not create/destroy the pool itself:
171+
172+
1. A storage pool (e.g. `bench_vm_nfs3_pool`) with a distinct storage tag
173+
(e.g. `bench_vm_nfs3`), sized comfortably below your aggregate's capacity.
174+
2. A service offering (root disk) and a disk offering (data disk), both
175+
tagged with that same storage tag, so root + data land on the same pool.
176+
Set the service offering's `rootdisksize` (GB) to comfortably exceed the
177+
template's actual `qemu-img` **virtual** size (not its sparse/download
178+
file size) - too small silently "succeeds" on NFS but hard-fails on iSCSI
179+
with `qemu-img: Cannot grow device files` (see Confluence Issue #2).
180+
3. Fill in `vm_bench.templateid` / `networkid` / per-protocol
181+
`serviceofferingid` / `diskofferingid` in `config.yaml` with the above.
182+
183+
### Running
184+
185+
```bash
186+
# Sanity-check without hitting a real management server:
187+
python3 run.py --test vm-instance-sequential --config config.yaml --dry-run
188+
python3 run.py --test vm-instance-concurrency --config config.yaml --dry-run
189+
190+
# Sequential scale matrix (5.2.1/5.2.2), protocols from config.yaml:
191+
python3 run.py --test vm-instance-sequential --config config.yaml
192+
193+
# Concurrency matrix (6.2.1/6.2.2). Levels come from vm_bench.concurrency_levels:
194+
python3 run.py --test vm-instance-concurrency --config config.yaml
195+
196+
# Both matrices in one invocation:
197+
python3 run.py --test vm-instance-combined --config config.yaml
198+
199+
# Or run the two tests under the SAME --run-id (underscores only):
200+
python3 run.py --test vm-instance-sequential --config config.yaml --run-id RUN_20260927_081500_a1b2
201+
python3 run.py --test vm-instance-concurrency --config config.yaml --run-id RUN_20260927_081500_a1b2
202+
203+
python3 render_report.py --run-id RUN_20260927_081500_a1b2 --raw-prefix raw_ops_vm --summary-prefix summary_vm --report-suffix _vm
204+
```
205+
206+
All three scripts share the same `--skip-cleanup` / `--cleanup-only` safety
207+
net as the storage-pool scripts (see above), via `vm_instance_common.py`.
208+
Failed `deployVirtualMachine` calls are logged but their VM/disk are
209+
intentionally left in place (not auto-destroyed) so you can inspect exactly
210+
what got left behind - use `--cleanup-only` once you're done.
211+
212+
## Notes / caveats
213+
214+
- Auth is session-key based (`login` -> `sessionkey` + `JSESSIONID` cookie),
215+
matching the approach documented on the Confluence API page. API
216+
key/secret signed requests are not implemented (add to
217+
`cloudstack_client.py` if your lab requires it).
218+
- Timing is measured end-to-end from the caller's perspective, including any
219+
async job polling (`queryAsyncJobResult`) — this is the number an operator
220+
or automation would actually observe, not raw server-side processing time.
221+
- The single `CloudStackClient`/`requests.Session` is shared across threads
222+
during concurrency tests. We never mutate session state after login, so
223+
this is safe; if you prefer full isolation, instantiate one
224+
`CloudStackClient` per thread instead.
225+
- `createStoragePool`/`deleteStoragePool`/`updateStoragePool` are
226+
synchronous in this CloudStack version (no `jobid` in the response);
227+
`enableStorageMaintenance`/`cancelStorageMaintenance` are async — the
228+
client polls `queryAsyncJobResult` automatically for any response that
229+
does contain a `jobid`, so the script works either way.
230+
- This script intentionally covers **create/delete only** (per the current
231+
ask). Maintenance mode and enable/disable pool are documented in
232+
`cloudstack_client.py`'s API surface but not yet wired into a benchmark
233+
phase — flag if you want those added to the matrix too.

0 commit comments

Comments
 (0)