Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions .github/workflows/python-package.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,7 @@ jobs:
python-version: "3.14"
cache: "pip"
- name: Install ruff
# Keep this version aligned with the ruff-pre-commit revision.
run: pip install ruff==0.16.1
run: pip install -e .[lint]
- name: Lint with ruff
run: |
ruff check . --output-format=github
Expand Down
2 changes: 1 addition & 1 deletion .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ repos:
exclude: ^tests/data/

- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.16.1
rev: v0.16.5
hooks:
- id: ruff-check
args: [--fix]
Expand Down
5 changes: 3 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,8 +70,9 @@ can predict where a thing is defined.
- Python >= 3.10; the CI test matrix is 3.10, 3.13, 3.14.

## Commands
- Lint: `ruff check .` and `ruff format --check .` (pinned to the version in
`.pre-commit-config.yaml` and the CI lint job — keep them aligned).
- Lint: `ruff check .` and `ruff format --check .`
(`pip install -e '.[lint]'` installs the pinned version;
keep the `.pre-commit-config.yaml` revision aligned with it).
- Tests: `coverage run -m pytest tests/ && coverage report`, or focused like
`pytest tests/waterdata_test.py::test_mock_get_samples`. `coverage report` is
a merge gate: branch coverage with a `fail_under` ratchet in
Expand Down
2 changes: 2 additions & 0 deletions NEWS.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

**09/01/2026:** **Announcement:** We at USGS Water Data for the Nation want your feedback! Tell us how we're doing by taking our quick [survey](https://usgswaterresources.gov1.qualtrics.com/jfe/form/SV_07gX8G1DeOtVrH8), available through September 2026.

**08/30/2026:** **Bug fix:** `waterdata.get_ratings(..., file_path=...)` wrote each rating with no explicit encoding, so on a non-UTF-8 locale an unrepresentable character raised `UnicodeEncodeError` and the rating was silently dropped from the returned dict; ratings are now written as UTF-8, byte for byte with the response. **Bug fix:** that same write blocked the event loop, so on a multi-location request every other in-flight download stalled for its duration; it now runs in a worker thread.

**08/27/2026:** **Bug fix:** `nwis.get_discharge_peaks` and `nwis.get_record(service='peaks')` discarded every peak whose date is only partly known. NWIS zero-fills the unknown part of a historical peak's date -- `YYYY-MM-00` when the day is not known, `YYYY-00-00` when the month is not either (the `Bd` and `Bm` `peak_cd` qualifiers) -- and neither parses as a date, so `preformat_peaks_response` coerced both to `NaT` and then dropped the row along with its discharge value. These are real peaks, and disproportionately a site's largest: site 14105700 lost 20 of its 167 peaks, among them an 1859 flood of 847,000 ft3/s, and 06934500 lost its 1844 peak of 700,000 ft3/s. Such a peak is now kept, with `datetime` left as `NaT`. The date is not completed into one NWIS does not have: a `datetime64` column cannot hold a partial date, so any value there would assert a day the record does not contain. **Behavior change:** peaks queries return more rows than before, and `datetime` may now be `NaT` -- a caller selecting on the datetime index will not see those peaks and should filter on `peak_dt` instead. **Behavior change:** `peak_dt` is no longer removed from the returned frame. It is the only column that holds a censored peak's year, since the peaks response has no `water_yr`, and the only dependable way to tell an unknown day from a known one -- `peak_cd` does not always include the qualifier (22 of 24 censored dates across six sites tested). For peaks with a resolved timestamp alongside explicit `year`/`month`/`day` and a `qualifier` field, use `waterdata.get_peaks()`.

**08/26/2026:** **Bug fix:** `nwis.format_response(df, service='peaks')` and `nwis.preformat_peaks_response` raised `KeyError('peak_dt')` on an empty peaks response instead of returning an empty frame. Both are public, and every other service already treated an empty result as a legitimate empty frame rather than an error (issue #171); the peaks branch was missed because it reformats the datetime column before the empty-frame check. Callers can now check `df.empty` rather than catching an exception. A *non-empty* frame with no `peak_dt` column is malformed rather than empty, and still raises.
Expand Down
6 changes: 6 additions & 0 deletions dataretrieval/codes/__init__.py
Original file line number Diff line number Diff line change
@@ -1,2 +1,8 @@
"""Facade over the ``states`` and ``timezones`` lookup tables.

Re-exports the state code maps, their normalizers (``to_state``,
``apply_state``), and the ``tz`` UTC-offset map.
"""

from .states import *
from .timezones import *
4 changes: 2 additions & 2 deletions dataretrieval/configuration.py
Original file line number Diff line number Diff line change
Expand Up @@ -804,7 +804,7 @@ def _accepts(adapter: str, name: str) -> bool:
return name in _ALL_SETTINGS if accepted is None else name in accepted


def _display_api_key(adapter: str | None = None) -> str:
def _display_api_key(_adapter: str | None = None) -> str:
"""Render the key's presence, never its value."""
return "<set>" if api_key() else "<not set>"

Expand All @@ -814,7 +814,7 @@ def _display_concurrency(adapter: str | None = None) -> str:
return CONCURRENCY_UNBOUNDED if value is None else str(value)


def _display_progress(adapter: str | None = None) -> str:
def _display_progress(_adapter: str | None = None) -> str:
setting = progress()
return "auto" if setting is None else ("on" if setting else "off")

Expand Down
99 changes: 81 additions & 18 deletions dataretrieval/nwis.py
Original file line number Diff line number Diff line change
Expand Up @@ -148,15 +148,15 @@ def _parse_json_or_raise(response: httpx.Response) -> pd.DataFrame:
raise


def _localize_datetime_index(df: pd.DataFrame) -> pd.DataFrame:
def _localized_datetime_index(index: pd.Index) -> pd.Index:
"""Localize a naive datetime index (or multi-index level) to UTC."""
if hasattr(df.index, "levels"):
if hasattr(index, "levels"):
# Multi-index: localize the datetime level (level 1)
if hasattr(df.index.levels[1], "tzinfo") and df.index.levels[1].tzinfo is None:
df = df.tz_localize("UTC", level=1)
elif hasattr(df.index, "tzinfo") and df.index.tzinfo is None:
df = df.tz_localize("UTC")
return df
if hasattr(index.levels[1], "tzinfo") and index.levels[1].tzinfo is None:
return index.set_levels(index.levels[1].tz_localize("UTC"), level=1)
elif hasattr(index, "tzinfo") and index.tzinfo is None:
return index.tz_localize("UTC")
return index


def format_response(
Expand Down Expand Up @@ -198,11 +198,19 @@ def format_response(
return df

if len(df["site_no"].unique()) > 1 and mi:
df.set_index(["site_no", "datetime"], inplace=True)
keys = ["site_no", "datetime"]
else:
df.set_index(["datetime"], inplace=True)
keys = ["datetime"]

# Index our own frame, never the caller's. The shallow copy shares the
# columns; ``set_index`` without ``inplace`` duplicates them, because
# pandas deep-copies the frame whenever copy-on-write is off.
df = df.copy(deep=False)
df.set_index(keys, inplace=True) # noqa: PD002 # our copy, not the caller's

df = _localize_datetime_index(df)
# Retag the index alone; ``DataFrame.tz_localize`` relabels the axis by
# duplicating every column whenever copy-on-write is off.
df.index = _localized_datetime_index(df.index)
return df.sort_index()


Expand Down Expand Up @@ -238,6 +246,8 @@ def preformat_peaks_response(df: pd.DataFrame) -> pd.DataFrame:
# still raise.
return df

# Derive the column on our own frame, never the caller's.
df = df.copy(deep=False)
df["datetime"] = pd.to_datetime(df["peak_dt"], errors="coerce")
return df

Expand Down Expand Up @@ -897,6 +907,44 @@ def what_sites(
return df, NWIS_Metadata(response, **kwargs)


# Each ignored ``get_record`` parameter maps to its declared default and
# replacement getter guidance, not restrictions on the replacement getter.
_DEFUNCT_RECORD_OPTIONS: dict[str, tuple[object, str]] = {
"wide_format": (True, "`waterdata.get_samples()`"),
"datetime_index": (
True,
"`waterdata.get_continuous()` or `waterdata.get_daily()`",
),
"state": (None, "`nwdc.get_wateruse(state=...)`"),
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So this is saying you can't ask for the wide format in the samples function, a datetime index in the water data continuous function, or pass a state in the wateruse function? And these are all passed to _warn_defunct_record_options() to let the user know?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The keys are ignored parameters of nwis.get_record, not restrictions on the replacement getters. Each tuple contains the default value and the replacement guidance to include in a warning.

  • get_record(wide_format=False) warns; waterdata.get_samples returns one row per result and has no wide_format parameter.
  • get_record(datetime_index=False) warns; waterdata.get_continuous and get_daily return time as a column and have no datetime_index parameter.
  • get_record(state="OH") warns and ignores the state, but nwdc.get_wateruse(state="OH") supports that filter.

Yes, _warn_defunct_record_options() checks these three values. It emits a parameter warning only when a value differs from its declared default. Local documentation edits clarify the distinction; they are not pushed yet.



def _warn_defunct_record_options(**given: object) -> None:
"""Warn when an ignored ``get_record`` parameter differs from its default.

``given`` must contain only names from ``_DEFUNCT_RECORD_OPTIONS``.
Values equal to the declared defaults do not produce parameter warnings,
whether passed explicitly or omitted by the caller of ``get_record``.
Replacement names in the table identify getters for retrieving the data;
they do not mean those getters accept the ignored parameters.
"""
for name, value in given.items():
unset, replacement = _DEFUNCT_RECORD_OPTIONS[name]
if value != unset:
warn_deprecated(
f"`nwis.get_record`'s `{name}` argument",
replacement=replacement,
removal=_NWIS_REMOVAL_DATE,
detail=(
"It is ignored, and has been since the service that read "
"it was retired."
),
# _warn_defunct_record_options -> get_record -> @_deprecated
# wrapper -> the caller's own line.
stacklevel=4,
)


@_deprecated
def get_record(
sites: list[str] | str | None = None,
Expand Down Expand Up @@ -929,12 +977,25 @@ def get_record(
If False, return a dataframe with a single-level index (datetime).
Default is True.
wide_format : bool, optional
If True, return data in wide format, with multiple samples per row and
one row per time. Default is True.
(defunct) Previously shaped output from the retired 'qwdata' service.
``get_record`` ignores this parameter; passing ``False`` emits a
``DeprecationWarning``.
To retrieve sample data, use ``waterdata.get_samples``,
which returns one row per result and has no ``wide_format`` parameter.
datetime_index : bool, optional
If True, create a datetime index. Default is True.
(defunct) Previously shaped output from the retired 'qwdata' and
'gwlevels' services.
``get_record`` ignores this parameter; passing ``False`` emits a
``DeprecationWarning``.
To retrieve time-series data, use ``waterdata.get_continuous`` or
``waterdata.get_daily``, which return ``time`` as a column and have
no ``datetime_index`` parameter.
state: string, optional, default is None
State full name, abbreviation, or id.
(defunct) Previously selected sites for the retired 'water_use' service.
``get_record`` ignores this parameter; passing a value other than
``None`` emits a ``DeprecationWarning``.
To retrieve water-use data by state, use ``nwdc.get_wateruse(state=...)``.
Its ``state`` parameter is supported and filters the query.
service: string, default is 'iv'
- 'iv' : instantaneous data
- 'dv' : daily mean data
Expand Down Expand Up @@ -1013,6 +1074,10 @@ def get_record(
),
)

_warn_defunct_record_options(
wide_format=wide_format, datetime_index=datetime_index, state=state

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems weird that you have to do name = name in this function? And it takes as many random things as you give it, right?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is keyword-argument syntax: in wide_format=wide_format, the left side names the argument being passed and the right side reads the local variable. Python does not have a shorthand for this when the names match.

**given collects the supplied keyword arguments into a dictionary. The signature accepts any number of keywords, but the body only supports the three names in _DEFUNCT_RECORD_OPTIONS; an unknown name would raise KeyError at the table lookup. This call passes exactly those three parameters, not the additional **kwargs supplied to get_record.

)

if service == "iv":
df, _ = get_iv(
sites=sites,
Expand Down Expand Up @@ -1099,15 +1164,13 @@ def _parse_parameter_record(
record_df["qualifiers"] = (
record_df["qualifiers"].astype(str).str.strip("[]").str.replace("'", "")
)
record_df.rename(
return record_df.rename(
columns={
"value": col_name,
"dateTime": "datetime",
"qualifiers": col_name + "_cd",
},
inplace=True,
}
)
return record_df


def _parse_site_block(site_block: list[dict[str, Any]]) -> pd.DataFrame:
Expand Down
4 changes: 3 additions & 1 deletion dataretrieval/ogc/dates.py
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,9 @@ def _parse_datetime(value: str) -> datetime | None:
candidate = value[:-1] + "+00:00" if value.endswith("Z") else value
for fmt in _DATETIME_FORMATS:
try:
return datetime.strptime(candidate, fmt)
# DTZ007: naive is the documented outcome for a naive input --
# ``_DATETIME_FORMATS`` carries both the ``%z`` and the bare forms.
return datetime.strptime(candidate, fmt) # noqa: DTZ007
except ValueError:
continue
return None
Expand Down
4 changes: 2 additions & 2 deletions dataretrieval/transport/pagination.py
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,7 @@ def report_page(page: httpx.Response, frame: pd.DataFrame) -> None:

try:
frame, cursor = parse_response(response)
except Exception as exc: # noqa: BLE001
except Exception as exc:
logger.warning("Initial response parse failed.")
raise DataRetrievalError(
paginated_failure_message(0, exc, response.url)
Expand All @@ -140,7 +140,7 @@ def report_page(page: httpx.Response, frame: pd.DataFrame) -> None:
nrows += len(frame)
total_elapsed += _safe_elapsed(response)
report_page(response, frame)
except Exception as exc: # noqa: BLE001
except Exception as exc:
logger.warning(
"Request failed at cursor %r. Data download interrupted.", cursor
)
Expand Down
4 changes: 2 additions & 2 deletions dataretrieval/transport/retry.py
Original file line number Diff line number Diff line change
Expand Up @@ -289,7 +289,7 @@ async def attempt_once() -> _T:
while True:
try:
return await attempt_once()
except Exception as exc: # noqa: BLE001 - re-raised unless retryable
except Exception as exc:
attempt += 1
wait = _retry_delay(exc, attempt, policy)
if wait is None:
Expand All @@ -311,7 +311,7 @@ def retry_sync(fn: Callable[[], _T], policy: RetryPolicy | None = None) -> _T:
while True:
try:
return fn()
except Exception as exc: # noqa: BLE001 - re-raised unless retryable
except Exception as exc:
attempt += 1
wait = _retry_delay(exc, attempt, policy)
if wait is None:
Expand Down
4 changes: 2 additions & 2 deletions dataretrieval/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,9 @@

import dataretrieval._querying as _querying
import dataretrieval.transport.http as _transport_http
from dataretrieval._ambient import Ambient # noqa: F401 - compatibility re-export
from dataretrieval._ambient import Ambient # noqa: F401 # compatibility re-export
from dataretrieval._response_metadata import (
BaseMetadata, # noqa: F401 compatibility re-export; defined there now
BaseMetadata, # noqa: F401 # compatibility re-export; defined there now
)
from dataretrieval.codes import tz

Expand Down
12 changes: 10 additions & 2 deletions dataretrieval/waterdata/ratings.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
from collections.abc import Iterable
from typing import Any, Literal, get_args

import anyio
import httpx
import pandas as pd

Expand Down Expand Up @@ -337,6 +338,12 @@ def _inert_response(
return httpx.Response(status, headers=headers, request=httpx.Request("GET", url))


def _write_rating(path: str, body: str) -> None:
"""Write one rating to disk as UTF-8, preserving line endings."""
with open(path, "w", encoding="utf-8", newline="") as f:
f.write(body)


async def _fetch_rating(
feature: dict[str, Any], file_path: str | None
) -> tuple[pd.DataFrame, httpx.Response]:
Expand All @@ -362,8 +369,9 @@ async def _fetch_rating(
_raise_for_non_200(response)

if file_path is not None:
with open(os.path.join(file_path, fid), "w") as f:
f.write(response.text)
await anyio.to_thread.run_sync(
_write_rating, os.path.join(file_path, fid), response.text
)

df = read_rdb(response.text)
df.attrs["comment"] = extract_rdb_comment(response.text)
Expand Down
38 changes: 26 additions & 12 deletions dataretrieval/waterdata/types.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,21 @@
"""Accepted argument values for the Water Data getters.

Each ``Literal`` type alias lists the values an argument accepts:
``CODE_SERVICES`` for the Samples code services,
``METADATA_COLLECTIONS`` for the reference-table collections,
``SERVICES`` and ``PROFILES`` for the Samples resources and output profiles,
and ``WATERDATA_COLLECTIONS`` for the collections ``get_cql`` queries.
``PROFILE_LOOKUP`` maps each Samples resource to its valid output profiles.

``WATERDATA_SERVICES`` is the previous name for ``WATERDATA_COLLECTIONS``.
Both names refer to the same type alias.
The previous name remains supported for compatibility and is not scheduled
for removal.

Callers can use these aliases in type annotations.
The getters use the same definitions to validate argument values at runtime.
"""

from typing import Literal, get_args

from dataretrieval._validation import require_one_of
Expand Down Expand Up @@ -54,10 +72,9 @@
"results",
]

# OGC API time-series/monitoring collections queryable via ``get_cql``. Keep in sync
# with ``utils._OUTPUT_ID_BY_COLLECTION`` (same keys): that dict maps each service to
# its user-facing ``id`` column and is the runtime definition ``get_cql`` validates
# against.
# OGC API collections queryable via ``get_cql``. Keep in sync with the keys of
# ``utils._OUTPUT_ID_BY_COLLECTION``, which maps each collection to its ``id``
# column and is used by ``get_cql`` to validate the collection argument.
WATERDATA_COLLECTIONS = Literal[
"channel-measurements",
"combined-metadata",
Expand All @@ -72,9 +89,8 @@
"time-series-metadata",
]

#: Permanent alias. OGC API - Features calls these collections -- the value is
#: the ``collectionId`` in ``/collections/{id}/items`` -- but this name is the one
#: the package published first, so it keeps resolving.
#: Previous name for ``WATERDATA_COLLECTIONS``, retained for compatibility.
#: Both names refer to the same object; neither is scheduled for removal.
WATERDATA_SERVICES = WATERDATA_COLLECTIONS

PROFILES = Literal[
Expand Down Expand Up @@ -117,16 +133,14 @@ def _check_profiles(
service: SERVICES,
profile: PROFILES,
) -> None:
"""Check whether a service profile is valid.
"""Check whether an output profile is valid for a Samples resource.

Parameters
----------
service : string
One of the service names from the "services" list.
A Samples resource name from ``SERVICES``.
profile : string
One of the profile names from "results_profiles",
"locations_profiles", "activities_profiles",
"projects_profiles" or "organizations_profiles".
An output profile name from ``PROFILE_LOOKUP[service]``.
"""
require_one_of(service, get_args(SERVICES), name="service")
require_one_of(
Expand Down
2 changes: 1 addition & 1 deletion dataretrieval/wateruse.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@

from dataretrieval import nwdc as _nwdc
from dataretrieval._deprecation import REMOVALS, warn_deprecated
from dataretrieval.nwdc import * # noqa: F403 (re-export the public surface)
from dataretrieval.nwdc import * # noqa: F403 # re-export the public surface

#: When the alias may be deleted. Read from the shared horizon table rather than spelled
#: here, so it is reviewed and extended with every other published removal; matches the
Expand Down
Loading