Skip to content

test: 100% coverage + CI coverage gate#208

Open
AkashBrowserStack wants to merge 3 commits into
masterfrom
test/coverage-100-and-ci-gate
Open

test: 100% coverage + CI coverage gate#208
AkashBrowserStack wants to merge 3 commits into
masterfrom
test/coverage-100-and-ci-gate

Conversation

@AkashBrowserStack

@AkashBrowserStack AkashBrowserStack commented Jun 12, 2026

Copy link
Copy Markdown

What & why

This SDK had no coverage enforcement in CI and sat at ~95% line locally. This PR raises it to 100% line coverage with meaningful tests and wires a hard CI gate so coverage can't silently regress.

Every line is executed by a real unit test — no # pragma skips, no coverage-gaming. Tests exercise real logic: validation, error paths, and fallbacks.

Tests added

Area Coverage added
PercyOnAutomate (was untested) invalid driver, disabled, name/options validation, alt camelCase region-key normalisation + element-id extraction, swallow-and-return-None error path — new tests/test_percy_automate.py
percy/__init__.py percySnapshot alias + percy_snapshot/percy_screenshot import fallbacks (the latter via a reload-based test) — new tests/test_init.py
cli_wrapper old-CLI minor-version rejection, unsuccessful healthcheck, post_failed_event error swallow, post_poa_screenshots success + error
app_automate remote-uploads-disabled fallback, begin() JSON parse, execute_percy_screenshot error, re-raise after failure notification
app_percy sync / test_case / th_test_case_execution_id type guards
generic_provider full-page falls back to single page
metadata remote_url new-client path + old-client URL fallback
android_metadata malformed viewportRect falls back to driver bars

Instrumentation / gate

  • .coveragercsource = percy, fail_under = 100, show_missing.
  • coverage added to development.txt; make coverage reports and enforces the threshold.
  • test.yml runs make coverage across the 3.9 / 3.11 / 3.12 matrix → red on any drop below 100%.

Verification

Ran 158 tests ... OK
TOTAL  775 stmts  0 miss  100%   (no pragma exclusions)
pylint: 10.00/10

🤖 Generated with Claude Code

Raise unit coverage from 95% line / 99% branch to 100% line + branch
with meaningful tests (not coverage-gaming), and wire a hard gate into
CI so it cannot regress.

Tests added:
- test_percy_automate.py: full coverage of the previously-untested
  PercyOnAutomate class (invalid driver, disabled, name/options
  validation, alt camelCase region-key normalisation + id extraction,
  and the swallow-and-return-None error path).
- test_init.py: defensive percy_snapshot/percy_screenshot import
  fallbacks that surface a helpful "package not installed" error.
- cli_wrapper: old-CLI-minor-version rejection, unsuccessful
  healthcheck, post_failed_event error swallow, and post_poa_screenshots
  success + error paths.
- app_automate: remote-uploads-disabled fallback, begin() JSON parse,
  execute_percy_screenshot error path, and screenshot re-raise after a
  failure notification.
- app_percy: sync / test_case / th_test_case_execution_id type guards.
- generic_provider: full-page falls back to single page.
- metadata: remote_url new-client path and old-client URL fallback.
- android_metadata: malformed viewportRect falls back to the driver.

Instrumentation:
- Add .coveragerc (branch=True, fail_under=100, show_missing).
- Add coverage to development.txt; Makefile `coverage` now reports and
  enforces the threshold instead of only emitting HTML.
- CI (test.yml) runs `make coverage` across the Python matrix, turning
  the gate red on any drop below 100%.
- percy_options: mark two genuinely-unreachable branch arcs with
  `# pragma: no branch` (options is always {} when those guards run).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@AkashBrowserStack AkashBrowserStack requested a review from a team as a code owner June 12, 2026 11:58
AkashBrowserStack and others added 2 commits June 12, 2026 18:16
Drop the branch-coverage gate and the two no-branch pragmas in
percy_options. Every line is executed by real unit tests; the only
branch arcs that pragma silenced were unreachable (control reaches
those guards only when options is {}), so line coverage at 100% is the
honest, pragma-free gate.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The org-managed 'Automatic Dependency Submission' workflow runs
setup-python against .python-version; the pinned patch 3.10.3 is not
available on the GitHub Ubuntu runner, failing the check on every push.
3.10 (latest available patch) matches the sibling SDKs and resolves it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@AkashBrowserStack

Copy link
Copy Markdown
Author

Claude Code PR Review

PR: #208Head: db34cf5Reviewers: stack:code-reviewer

Summary

Adds unit tests across seven modules (app_automate, generic_provider, cli_wrapper, android_metadata, metadata, percy_automate, app_percy/init) covering previously-untested error, fallback, and edge-case paths, and introduces a CI coverage gate: a new .coveragerc with fail_under = 100, a coverage==7.* dev dependency, a Makefile coverage target switched to coverage report, and CI (test.yml) switched from make test to make coverage. No production/source code under percy/ is modified — this is a test-and-tooling-only PR.

Review Table

Priority Category Check Status Notes
High Security No hardcoded secrets or credentials Pass No secrets/tokens introduced. Test fixtures use dummy hub URLs and fake session ids only.
High Security Authentication/authorization checks present N/A No auth surface touched; tests-and-tooling-only PR.
High Security Input validation and sanitization N/A No new input-handling code; tests only assert existing validation (e.g. type checks in app_percy.screenshot).
High Security No IDOR — resource ownership validated N/A No resource-access code in scope.
High Security No SQL injection (parameterized queries) N/A No DB/SQL in this codebase.
High Correctness Logic is correct, handles edge cases Pass Test assertions verified against source. Version-gate (1.26.0 rejected, minor < 27), viewportRect-malformed fallback, remote_url new/old client, POA alt-key normalization, and screenshot failure-notification re-raise all match actual percy/ behavior.
High Correctness Error handling is explicit, no swallowed exceptions Pass Tests deliberately assert the swallow-and-return-None contracts (post_failed_event, percy_automate.screenshot, screenshot_begin) rather than hiding them; lru_cache is cleared in try/finally so failures don't leak cached state.
High Correctness No race conditions or concurrency issues N/A No concurrency introduced; tests are synchronous.
Medium Testing New code has corresponding tests Pass This PR is the tests; they target real, exercisable source paths (verified each assertion against the module under test).
Medium Testing Error paths and edge cases tested Pass Strong coverage of error/fallback branches: malformed viewportRect, disabled remote uploads, finalize-failure re-raise, healthcheck unsuccessful, old CLI version, POA exception swallow, package import-fallback ModuleNotFoundError.
Medium Testing Existing tests still pass (no regressions) Pass Additive; no existing test bodies changed. test_init reloads percy and restores it in finally; cli_wrapper tests clear the is_percy_enabled lru_cache around the call. (Low-sev isolation nit noted in Findings, non-blocking.)
Medium Performance No N+1 queries or unbounded data fetching N/A No data-fetch loops added; test code only.
Medium Performance Long-running tasks use background jobs N/A Not applicable to an SDK test suite.
Medium Quality Follows existing codebase patterns Pass Matches existing unittest + unittest.mock style, # pylint: disable=[...] headers, mock_methods fixtures, and PropertyMock patching used elsewhere in the suite.
Medium Quality Changes are focused (single concern) Pass Single concern: raise coverage to 100% and gate it in CI. The .python-version 3.10.3 -> 3.10 tweak is a minor adjacent change but consistent with the tooling focus.
Low Quality Meaningful names, no dead code Pass Test names are descriptive and intent-revealing; explanatory comments justify non-obvious branches. No dead code.
Low Quality Comments explain why, not what Pass Comments explain why a branch is exercised (e.g. "full page is only supported on App Automate -> warn and produce a single tile").
Low Quality No unnecessary dependencies added Pass Only coverage==7.* added, to a dev-only requirements file (development.txt); appropriate and from PyPI.

Findings

No Critical or High findings. Two non-blocking, low-severity observations:

  • File: tests/test_android_metadata.py:71

  • Severity: Low

  • Reviewer: stack:code-reviewer

  • Issue: The test mutates the module-level shared android_capabilities dict (self.mock_webdriver.capabilities['viewportRect'] = {'left': 5}) and the finally restores it to None rather than to its original loaded value. android_capabilities is loaded once at import and shared by reference across test classes, so restoring to None leaves a slightly different state than the original. In practice each test re-reads capabilities via setUp and no other test asserts on viewportRect, so there is no observed cross-test contamination — but it is a latent isolation fragility.

  • Suggestion: Capture and restore the original value, e.g. orig = self.mock_webdriver.capabilities.get('viewportRect') before mutating and restore orig in finally; or deepcopy the caps in setUp.

  • File: percy/providers/app_automate.py:27 (context surfaced by tests/test_app_automate.py:108 test_screenshot_reraises_after_failure_notification, not a defect in the test)

  • Severity: Low (pre-existing source observation, out of this PR's scope)

  • Reviewer: stack:code-reviewer

  • Issue: In AppAutomate.screenshot, if super().screenshot() raises before percy_screenshot_url is assigned, the except handler references the then-unbound percy_screenshot_url, causing an UnboundLocalError that would mask the original exception. The new test correctly exercises only the post-assignment failure path, so it does not assert this buggy path — flagging for awareness, not as a PR blocker.

  • Suggestion: Initialize percy_screenshot_url = '' before the try in a follow-up source PR. No action required for this test-only PR.


Verdict: PASS

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants