Fix estimate_pass_at_k returning 1.0 when there are no correct samples (#35)

[LeSingh1]

Problem

estimate_pass_at_k returns 1.0 for a problem that has no correct samples when k exceeds the number of samples. That is wrong: with zero correct samples, pass@k is exactly 0 for any k.

The estimator uses a shortcut:

def estimator(n: int, c: int, k: int) -> float:
    if n - c < k:
        return 1.0
    return 1.0 - np.prod(1.0 - k / np.arange(n - c + 1, n + 1))

The n - c < k branch encodes "there are too few incorrect samples to avoid drawing a correct one, so a correct sample is guaranteed." That reasoning only holds when at least one correct sample exists (c > 0). When c == 0 and k > n (so n - c = n < k), the branch is taken and returns 1.0, even though there is nothing correct to draw.

This was reported in #35.

>>> from human_eval.evaluation import estimate_pass_at_k
>>> estimate_pass_at_k([2], [0], 5)   # 0 correct out of 2, k=5
array([1.])                            # expected: array([0.])

evaluate_functional_correctness guards its own calls with (total >= k).all() (so n >= k), which is why the main pipeline never hits this branch. But estimate_pass_at_k is a public, widely-imported helper, and downstream callers that compute pass@k with k > n get an incorrect 1.0 for unsolved problems.

Fix

Return 0.0 up front whenever c == 0:

if c == 0:
    return 0.0
if n - c < k:
    return 1.0
return 1.0 - np.prod(1.0 - k / np.arange(n - c + 1, n + 1))

For c == 0 with k <= n the estimator formula already evaluated to 0.0 (the np.arange(n + 1, n + 1) product is empty → 1.0, giving 1 - 1 = 0), so behavior is unchanged there and for every c > 0 case. Only the previously incorrect c == 0, k > n branch changes.

Verification

• Added human_eval/test_evaluation.py (stdlib unittest, no new dependencies) covering the regression plus the all-correct, guaranteed-pass, known-value (pass@1 == 0.5 for 3/6), and scalar-num_samples cases. All pass:

  $ python -m unittest human_eval.test_evaluation
  ......
  Ran 6 tests in 0.000s
  OK
  

• Before the fix estimate_pass_at_k([2], [0], 5) returned [1.0]; after, it returns [0.0].

• The README sanity check is unchanged: evaluate_functional_correctness on the bundled example samples still yields pass@1: 0.4999999999999999.

Fixes #35