feat(scripts): Add dependency version scanner tool

scripts/version_scanner/.gitignore

@@ -0,0 +1,2 @@
+.conductor/
+scanner_report.csv

scripts/version_scanner/.scannerignore

@@ -0,0 +1,22 @@
+# Directories and files to ignore by the version scanner
+.git
+__pycache__
+.tox
+.nox
+venv
+.venv
+.conductor
+version_scanner
+docs
+samples
+changelog.md
+.librarian
+goldens
+# Ignore pandoc references in repositories.bzl
+repositories.bzl
+
+# Ignore binary media files
+*.jpg
+*.png
+*.gif
+*.ico

scripts/version_scanner/README.md

@@ -0,0 +1,160 @@
+# Automated Dependency Version Scanner
+
+This tool scans the repository for hardcoded references to specific dependency versions (like Python 3.7) that need to be upgraded or removed.
+
+## Usage
+
+Run the script from the repository root:
+
+```bash
+python3 scripts/version_scanner/version_scanner.py -d <dependency> -v <version> [options]
+```
+
+### Options
+
+*   `-d`, `--dependency`: Name of the dependency (e.g., python, protobuf)
+*   `-v`, `--version`: Specific version to search for (e.g., 3.7, 4.25.8)
+*   `-p`, `--path`: Root directory to scan (defaults to current directory)
+*   `--package`: Specific subdirectory filter (useful for monorepos)
+*   `--package-file`: Path to a file containing a list of package directories to scan
+*   `--config`: Path to the regex configuration file (defaults to scripts/version_scanner/regex_config.yaml)
+*   `-o`, `--output`: Path to the output CSV file (defaults to <dependency>-<version>-<timestamp>.csv)
+*   `--github-repo`: GitHub repository URL base (defaults to https://github.com/googleapis/google-cloud-python)
+*   `--branch`: GitHub branch for links (defaults to main)
+
+## Configuration
+
+The scanner uses a YAML configuration file (`regex_config.yaml`) to define rules and regex patterns.
+
+## Ignoring Directories
+
+You can create a `.scannerignore` file in the directory you are scanning (usually the repo root) to list directories to skip, one per line.
+
+## Known Issues & Future Investigations
+- **Binary Ignores in `.scannerignore`**: Recursive wildcard ignores (e.g., `*.jpg`) currently do not effectively ignore deeply nested binary files. The scanner logic should be investigated to support robust globbing or full-path suffix matching.
+
+---
+
+## Universal Prompt for EOL Runtime & Dependency Migration
+
+### Context & Overview
+
+#### Overview
+This plan outlines the approach to update Python packages to drop support for end-of-life Python runtimes (3.7, 3.8, 3.9) OR for deprecated dependencies, and ensure the packages are configured for modern Python.
+
+#### High-Level Strategy
+- **One Branch Per Package**: To keep PRs manageable and isolated, we suggest a dedicated worktree and branch for each package (e.g., `feat/drop-<dependency>-<version>-<package-name>` i.e. `feat/drop-protobuf-4.25.8-google-cloud-bigquery`).
+- **Small & Reversible Commits**: Group changes into logical commits (Metadata, Nox, Docs, Cleanup, Tests) following Conventional Commits.
+
+---
+
+### Per-Package Workflow
+
+Follow these steps for each package in the target list. Context and warnings are provided inline before the steps where they apply.
+
+#### Step 1: Sync & Branch
+1. Ensure `main` branch is up to date.
+2. Create the feature branch: `git checkout -b feat/drop-<dependency>-<version>-<package-name>`.
+
+#### Step 2: Scan (Baseline)
+1. Run the `version_scanner` for the package to get a list of all occurrences of the dependency and version.
+   > [!TIP]
+   > Use `# version-scanner: ignore` or `ignore-next-line` in code to silence true false-positives and maintain clean reports.
+
+---
+
+#### 💡 Context for Step 3: Standards & Cleanup
+*Before applying changes, review these standards to ensure consistency:*
+
+##### Runtime Version Checks
+- **Standard**: Use `sys.version_info < (X, Y)`.
+- **Rationale**: Python compares tuples lexicographically, making this robust.
+- **Avoid**: `sys.version_info.minor < Y` or string conversions.
+
+##### Pytest Skips
+- **Standard**: `@pytest.mark.skipif(sys.version_info < (X, Y), reason="Requires Python X.Y+")`.
+- **Avoid**: String-based conditions like `@pytest.mark.skipif("sys.version_info < ...")`.
+
+##### Noxfile Version Matches
+- **Standard**: `session.python == "X.Y"` (Nox uses strings).
+- **Avoid**: `float(session.python) < X.Y` (fails for `3.10`).
+
+##### Cleanup Rules
+- **Polyfills**: Remove dead `try/except` blocks guarding polyfills for features now standard in 3.10+.
+- **Obsolete Skips**: Remove pytest skips for features now universally available.
+
+##### Dependency Specific rules
+- Use idiomatic python references to detect dependency versions and to compare against the target version.
+
+---
+
+#### 💡 Context for Step 3: Disposition Rules
+*Every reference to the dependency version found by the scanner must be dispositioned in one of these ways:*
+
+1. **Update**: Update the reference if still necessary (e.g., changing `3.9` to `3.10` in support files).
+2. **Delete**: Delete if no longer relevant (dead code, obsolete comments).
+3. **Pragma Ignore**: Use `# version-scanner: ignore` or `# version-scanner: ignore-next-line` but ONLY for immutable historical facts or true false positives. Do NOT use for things that might change in future upgrades.
+
+#### Step 3: Apply Changes
+1. Update `setup.py` or `pyproject.toml` metadata and `requires-python`.
+2. Update `noxfile.py` to remove old versions from sessions.
+3. Update `README.rst` and `CONTRIBUTING.rst` documentation.
+4. Remove compatibility code and skips based on the standards above.
+5. **Sync Documentation**: If the package has a `docs` folder containing a `README.rst`, copy the updated top-level `README.rst` to overwrite it (unless it is a symlink).
+6. Continue with the update process until all rows from the scan have been properly dispositioned.
+
+---
+
+#### Step 4: Verify (Post-Scan)
+1. Run the `version_scanner` again. The result should be 0 matches (or only valid ignores).
+
+---
+
+#### 💡 Context for Step 5: Constraints & Conflicts
+*Review these lessons learned when dealing with constraints:*
+
+- **Lowest Runtime Constraints**: The file for the lowest accepted runtime (e.g., `constraints-3.10.txt`) must have pins matching the lowest acceptable versions in `setup.py` or `pyproject.toml`.
+- **Philosophy on Warnings**: Do not simply block warnings (like `six` or `pkg_resources`) to make tests pass. **Bump the lower bounds** of dependencies to versions that don't trigger warnings on the current lowest acceptable runtime. This protects customers who use strict warning filters.
+- **SQLAlchemy Transition**: For libraries supporting both 1.4 and 2.0, use `SQLALCHEMY_SILENCE_UBER_WARNING=1` in specific legacy Nox sessions rather than silencing globally.
+
+---
+
+#### Step 5: Local Test
+1. Run unit tests using Nox (e.g., `nox -s unit`).
+   > [!TIP]
+   > Use `nox -s unit-3.10` to save time when debugging specific runtime failures.
+2. Run `blacken` and `lint` sessions.
+
+#### Step 6: Push & PR
+1. Push the branch and create the PR using the template in the Appendix.
+
+---
+
+## Appendix
+
+### PR Template [^1]
+```text
+This PR updates `<dependency>` to establish version x.y.z as the minimum supported version.
+
+### Changes
+* Configuration: Updated `setup.py` and `noxfile.py` to require <dependency> <version> and remove references to older versions.
+* Cleanup: Removed dead code and polyfills no longer needed.
+
+Fixes internal issue: http://b/482126936 🦕
+```
+
+---
+
+## Candidates for `.conductor` or `gemini.md`
+
+*The following guidelines are universal for AI assistants workin' in this repo and should be moved to `.conductor` files or Gemini memories:*
+
+1. **AI & LLM Guidelines for Verification**:
+   - Use Git Worktrees to scan branches without switching.
+   - Run scanner from main branch pointing to worktree.
+   - Bypass env artifacts by worktree only checking out tracked files.
+2. **Automated Bisection**:
+   - Use `version_bisector.py` to find lowest workable versions.
+   - Abort tests early as soon as collection succeeds to save time.
+
+[^1]: Adapted from the standard PR template used in this repository.

scripts/version_scanner/benchmark.py

@@ -0,0 +1,166 @@
+import argparse
+import os
+import random
+import subprocess
+import sys
+import tempfile
+import time
+from typing import List, Dict
+
+def get_package_subset(packages_dir: str, count: int) -> List[str]:
+    """
+    Get a randomized subset of package names from the specified directory.
+
+    Args:
+        packages_dir: Path to the directory containing packages.
+        count: Number of packages to return.
+
+    Returns:
+        A list of package directory names.
+    """
+    all_packages = [d for d in os.listdir(packages_dir) if os.path.isdir(os.path.join(packages_dir, d))]
+
+    if count >= len(all_packages):
+        return all_packages
+
+    return random.sample(all_packages, count)
+
+def run_benchmark(
+    scanner_path: str,
+    root_path: str,
+    package_file: str,
+    dependency: str,
+    version: str
+) -> float:
+    """
+    Run the scanner and return the duration in seconds.
+    """
+    cmd = [
+        "python3", scanner_path,
+        "-d", dependency,
+        "-v", version,
+        "-p", root_path,
+        "--package-file", package_file
+    ]
+
+    start_time = time.perf_counter()
+
+    try:
+        result = subprocess.run(cmd, check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
+    except subprocess.CalledProcessError as e:
+        print(f"Error running benchmark: {e}")
+        return -1.0
+
+    duration = time.perf_counter() - start_time
+    return duration
+
+def run_benchmarks(
+    scanner_path: str,
+    root_path: str,
+    packages_dir: str,
+    counts: List[int],
+    dependency: str,
+    version: str
+) -> Dict[int, float]:
+    """Runs benchmarks for specified counts and returns a dict of results."""
+    results = {}
+
+    for count in counts:
+        subset = get_package_subset(packages_dir, count)
+        print(f"  Testing {len(subset)} packages (e.g., {subset[:3]}...)")
+
+        # Create temp package file
+        with tempfile.NamedTemporaryFile(mode='w', delete=False) as f:
+            for pkg in subset:
+                f.write(f"packages/{pkg}\n")
+            pkg_file = f.name
+
+        try:
+            duration = run_benchmark(scanner_path, root_path, pkg_file, dependency, version)
+            results[count] = duration
+        finally:
+            # Clean up
+            if os.path.exists(pkg_file):
+                os.remove(pkg_file)
+
+    return results
+
+def main():
+    parser = argparse.ArgumentParser(description="Benchmark the version scanner.")
+
+    parser.add_argument(
+        "-s", "--scanner-path",
+        default="version_scanner.py",
+        help="Path to version_scanner.py"
+    )
+
+    parser.add_argument(
+        "-r", "--root-path",
+        required=True,
+        help="Path to the monorepo root directory"
+    )
+
+    parser.add_argument(
+        "-p", "--packages-dir",
+        help="Path to packages directory (defaults to <root-path>/packages)"
+    )
+
+    parser.add_argument(
+        "-d", "--dependency",
+        default="python",
+        help="Dependency to search for"
+    )
+
+    parser.add_argument(
+        "-v", "--version",
+        default="3.7",
+        help="Version to search for"
+    )
+
+    parser.add_argument(
+        "-c", "--counts",
+        default="1,10,50",
+        help="Comma-separated list of package counts to test"
+    )
+
+    args = parser.parse_args()
+
+    packages_dir = args.packages_dir or os.path.join(args.root_path, "packages")
+
+    if not os.path.exists(packages_dir):
+        print(f"Error: Packages directory not found: {packages_dir}", file=sys.stderr)
+        sys.exit(1)
+
+    counts = [int(c) for c in args.counts.split(',')]
+
+    all_packages = [d for d in os.listdir(packages_dir) if os.path.isdir(os.path.join(packages_dir, d))]
+
+    total_packages = len(all_packages)
+
+    print(f"Found {total_packages} packages in {packages_dir}")
+
+    # Filter counts that are greater than total packages
+    counts = [c for c in counts if c <= total_packages]
+    # Add total if not already there
+    if total_packages not in counts:
+        counts.append(total_packages)
+
+    print(f"Running benchmarks for counts: {counts}")
+
+    results = run_benchmarks(
+        scanner_path=args.scanner_path,
+        root_path=args.root_path,
+        packages_dir=packages_dir,
+        counts=counts,
+        dependency=args.dependency,
+        version=args.version
+    )
+
+    print("\nBenchmark Results:")
+    print(f"{'Packages':<10} | {'Time (seconds)':<15}")
+    print("-" * 30)
+    for count, duration in results.items():
+        print(f"{count:<10} | {duration:<15.4f}")
+
+if __name__ == "__main__":
+    main()