Repository navigation
Fix every finding of both 0.3.0 pre-release audits - #13
Merged
Merged
Conversation
A board that dropped the connection halfway through a 200 had its prefix kept with the ETag, and every later 304 sent the broken body back until the board changed. http.client returns a short read with no error, and a truncated gzip stream unzips to a prefix with no error either. - Keep a body only when all of its Content-Length arrived (and http.client expected nothing more) and, for gzip, the stream reached its end. - A 304 whose kept body is not JSON drops the entry and asks once more without validators; the entry counts as used only after it parsed. - Each cache file carries a format version; files without it (anything a pre-release build wrote) are misses. - A corrupt gzip body raises BadGzipFile, which the retry handles, instead of a zlib.error escaping request(). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The size cap was checked on the first write and then every 500th, so one run could overshoot by 499 times the largest body, and a run of only 304s never pruned at all. A running total of what this process writes stands in for the folder's size between looks; the folder is looked at whole on its first use in a process (a read counts, so 304s alone still age files out), every hour after for a long-running server, and whenever the total passes CACHE_BYTES. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The cache was keyed on the URL alone, so a 304 handed one caller the body another had read: all of it whatever the caller's limit (discover reads a page at 3 MB, careersite at 5 MB), and the JSON representation to a caller asking for HTML where a host sends only Last-Modified. Accept is now part of the key. The limit is not: the same page under two limits is one answer whenever it fits both, and keying on it would keep it twice. A kept body longer than the caller's limit is simply not asked about, so that caller gets what an uncached read gives (the body cut at its limit, not kept) and the whole copy stays for the caller that can take it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
configure(cache_dir=b"/tmp/x") was accepted, and then every GET raised a TypeError from os.path.join before the cache's own error handling. Bytes, a PathLike that gives bytes, and anything else now raise ValueError at configure time and leave the setting as it was. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
_unframed dropped comment and CDATA lines with lines.pop(0) in a loop, and json_values worked it out up to three times. A script opening with 200,000 comment lines took 21 seconds here, 64 when its JSON was invalid, and a 5 MB careers page could hold one fetch for many minutes. Find the first and last real lines in one pass from each end and slice once, and work out the unframed text once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
period_of stripped digits before matching, so Ashby's "2 WEEK", "0.5 MONTH" and "6 MONTH" came out as "week" and "month", and tiers at "1 WEEK" and "2 WEEK" passed the one-kind check and merged into one weekly band. "semi-annual" read as "year". A number in the value other than one now gives "", "semi", "biannual" and "twice" are not periods, and Ashby's one-kind check compares the interval itself wherever no period field can hold it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A salary written as a plain JSON integer too large for a float (json.loads keeps it an int) raised OverflowError from amount() for Lever and Ashby, and from the wording code for Recruitee, Rippling, ClearCompany and careers sites. Ashby's compensationTiers, summaryComponents and components were iterated whatever they held, so a number there raised TypeError. Neither is a BoardError, so fetch() lost every board in the call. amount() gives None on overflow, each wording helper falls back to the text as written, and Ashby reads those three fields only when they are lists. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Lever salaryRange {"min": 65000, "max": 50000} came out as salary_min 65000,
salary_max 50000, so a caller filtering on salary_max >= x got the wrong
answer. pay() now orders the two figures when both are present (swapped,
never changed or invented), and Ashby orders each component before taking
the lowest minimum and highest maximum across tiers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Workday: the reader's address percent-encodes the path ("&" is %26,
non-ASCII letters are UTF-8 escapes) while Job.id used it raw, so the two
disagreed. job_id decodes the segment, and the reader decodes its id
segment too, so a path that arrives already encoded and one that does not
are one id (unchanged for every path that carries no %XX).
- Lever, Ashby, Rippling: an upper-case UUID gave an upper-case id; the
boards give lower case.
- A gh_jid that disagrees with the id in the path (or a second gh_jid that
disagrees with the first) gives "", since either could be wrong.
- A protocol-relative link and a lower-case Rippling locale are read.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
iso_day("0001-01-01T00:00:00") gave "0001-01-01", and the same for 1900 and
the Unix epoch: .NET's DateTime.MinValue is what a ClearCompany blank looks
like. Such a role was dropped by since_days, sorted last, and blocked the
openDate fallback the CHANGELOG says was fixed.
A year before 1990 or a day more than two years ahead now reads as empty
and the next value is tried. Lever's epoch converter, Teamtailor's RFC 822
reader and the careers-site parser pass their results through iso_day, so
the rule holds for every reader.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- "Greenhouse and Lever, among others, answer 304" could not be checked offline: say a board that honours the validators and has not changed answers 304 (README, llms.txt, CHANGELOG). - Kept answers: only whole ones, and the file name is a hash of the request, not the address alone. - The JSON-LD repair is not "the cases extruct's reader handles": read against extruct's jsonld.py, it is a subset. extruct also strips JavaScript comments and trailing commas and mends escapes and stray quotes. The framing tokens are the same as extruct's, now credited beside the pattern, and "written afresh" is dropped. - salary_period can be "" alongside figures (every two weeks). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pruned only to the cap, the folder passed it again on the very next write, so every write after that scanned, stat-ed and sorted the whole folder while holding the lock every read waits on. 200 writes into a full folder of 20,000 files took 21 s; they now take 0.2 s, as on main. A look that finds the folder over its cap now takes it down to PRUNE_TO (0.9) of it, so the next full look comes a tenth of the cap later. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LrnN33G65kWMuv8y7xpUk4
Each insert added up every body already kept, so a run of n answers cost n squared: 20,000 answers of 1 kB took 39 s, and now take 0.03 s. The total is kept beside the store with the store it counts, so a store put in its place (as the tests do for a fresh process) is counted once, the first time, and 0.2.0's and main's own tests still run against it unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LrnN33G65kWMuv8y7xpUk4
The single regular expression for a whole script looked to the end of the page for a
closing tag after every opening one that had none, and for a ">" after every "<script"
that had none. 8,000 unclosed openers (280 kB) took 31 s and 20,000 took 209 s; a 5 MB
page, the most a careers page is read to, could hold one fetch() or one MCP call for
hours. Now each opening tag, its ">" and its closing tag are looked for once, and when
one is missing no later script can be complete either. The same hostile pages at 5 MB
read in under 0.15 s.
The auditor's prototype lowered the page and used those offsets on the original, which
goes wrong after a character whose lower case is longer (U+0130), and missed tags the
expression matched under re.I ("<ſcript"). This version keeps the expression's own
case-insensitive pieces, and a test checks it against the old expression on 5,000 pages
built from the pieces that decide a match. It also matched on every page the suite reads,
on the 29 JSON-LD cases, and on 200,000 such pages run once by hand.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LrnN33G65kWMuv8y7xpUk4
html_to_text's two tag patterns looked to the end of the text for a ">" from every "<" that had none after it. 60 kB of "<a " took 0.8 s and of "<br" 1.6 s; 180 kB took 7 s and 15 s on 0.2.0 and on this branch. No tag can start after the last ">" in the text, and before it every "<" has a ">" to stop at, so the same two patterns now run only up to the last ">", where each looks no further than the next one. The output is the same by construction; a test also checks it against the patterns run over the whole text on 5,000 texts built from the pieces that decide them (300,000 compared once by hand, no difference). 5 MB of "<a " or "<br" now takes under 0.2 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LrnN33G65kWMuv8y7xpUk4
A board answer or a JSON-LD script nested many thousands deep (3,000 on Python 3.9) raised RecursionError, which is not a BoardError, out of fetch(), so one board's answer lost every board in the call. _http.request now gives "the response was not JSON", as for any other body it cannot parse, and a careers page's script gives no posting, whether the JSON reader or the lower-casing of its keys runs out of depth; the page's other scripts are still read. The MCP server answered such a line by stopping; it now answers with a parse error and carries on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LrnN33G65kWMuv8y7xpUk4
An answer with an ETag but no Content-Length, not chunked and not gzipped, ends when the connection closes, and one dropped halfway ends the same way. Such a prefix was kept and request_raw sent it back on every 304 until the board changed (careers-site sitemaps and pages, discover, and the Personio, Teamtailor, JazzHR and Homerun feeds). Now a body that is not gzipped is kept only when it came chunked (http.client raises when a chunked answer is cut) or with all of its Content-Length; a gzip stream is still kept when it reaches its end. The README says exactly that. 0.2.0 kept such an answer in memory too, so a host that answers this way is now read in full on every request. The test fakes that served bodies with no length now declare it, as a real server does; what those tests check is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LrnN33G65kWMuv8y7xpUk4
A board that answered something other than JSON with an ETag (a maintenance page) had that page kept; the next run's 304 sent it back, it was found not to be JSON, dropped and read again, so every run cost two requests instead of one. _one now hands a new answer back to its caller to keep: request_raw keeps it as it comes, and request keeps it only once it has parsed. An answer that stops being JSON also drops the one kept before it, since that no longer describes the board. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LrnN33G65kWMuv8y7xpUk4
"payRangeDetails": 5, true or 1.5 raised TypeError out of rippling.fetch(details=True), which is not a BoardError, so every board in the call was lost, under a release whose headline says one posting's odd pay no longer sinks the fetch. The two pay readers now share one helper that reads the field only when it is a list. A new test replaces every part of every reader's structured pay (Lever, Ashby, Recruitee, Rippling, ClearCompany and a careers site's baseSalary), the whole field included, with each of 25 odd values: every board still reads, and every figure is a positive number or none, lowest first. Rippling was the only reader it caught. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LrnN33G65kWMuv8y7xpUk4
A Workday path that arrived percent-encoded was decoded into Job.id, so an id 0.2.0
users had stored changed ("M%C3%BCnchen" became "München"), a broken escape became
U+FFFD, and %2F put a "/" inside the id. The id is again the path's last segment exactly
as the listing sends it; on 2,978 listing paths built to hold every awkward case, every
id and every address is what 0.2.0 gave.
job_id(url) reads that id back from the address. The address percent-encodes the path
and leaves any "%" as it is, so job_id writes a link the same way (a copied link with a
raw "ü" or "&" is then the reader's own address) and decodes it wherever the reader would
have written exactly that address from the decoded text; where it would not (%2F, %25, a
broken escape) the escapes are the id. One case cannot be told apart, and it is named in
the docstring and a test: a listing that sends "M%C3%BCnchen" has the same address as one
that sends "München", and job_id reads it as the second.
A path holding a lone surrogate, which JSON can carry and which has no UTF-8, raised
UnicodeEncodeError out of fetch() when its address was written (0.2.0 did the same). Its
posting is now left out, and job_id gives "" for such an address.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LrnN33G65kWMuv8y7xpUk4
Each process keeps the folder to its cap from what it saw at its last full look plus what it has written since, so two processes writing to one folder at once (a long-running application and a command-line run) can take it past the cap until one of them next looks: at its start, hourly while in use, and when its own writes pass the cap. With a cap of 100,000 bytes the auditor's two processes left 179,449. The README, the configure docstring and the comment in _http now say "about 500 MB" and say why, and the configure docstring says only answers shown to be whole are kept. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LrnN33G65kWMuv8y7xpUk4
The comments said the framing tokens were extruct's and that its JSON-LD reader makes these repairs and more. Its reader (extruct 0.18.0, jsonld.py, as the audit read it) does json.loads(strict=False), and on failure strips one leading "//" line or one-line "<!-- -->" and hands the rest to jstyleson for comments and trailing commas. The only repair the two share is reading raw newlines in strings; no code or token list here came from it. The docstring now says what openroles itself leaves unmended. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LrnN33G65kWMuv8y7xpUk4
… user Everything new since 0.2.0 (the cache, job_id, the salary fields and the loose JSON-LD read) is under New, with the properties that make each one right folded into its entry: only answers shown to be whole are kept, a JSON reader keeps an answer only once it parses, the cache holds about 500 MB per process and why, ranges come out lowest first, odd pay gives no figure, framing lines are read in linear time. Fixed keeps only what a 0.2.0 user could have met: a body cut short sent back on a 304, dates that were not dates, odd pay and deeply nested JSON losing every board in the call, and pages that stall the reader. The job_id entry names the seven systems and the two cases where job_id(job.url) does not give job.id back, instead of "for every posting". The extruct comparison is gone. Each headline is one sentence with a full stop; rendered by scripts/release_notes.py the page has 10 headlines and none falls under "And N more". The README names the one ambiguous Workday case. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LrnN33G65kWMuv8y7xpUk4
A host whose Last-Modified stays put while its data changes answers 304 to every If-Modified-Since, and each 304 counted as recent use, so its kept copy was the answer for good: for the life of a process, and with a cache folder, across runs. Now an answer kept with a Last-Modified date and no ETag is asked about only for MODIFIED_MAX_AGE (24 hours) after it was last read in full, a 304 not counting, and then read in full once. An ETag names the body itself, so an answer with one is asked about however old it is. Each kept answer now carries when it was last read in full, in memory and in its file; a file that does not say (a build before this one wrote it) reads as read long ago. The clock is _http._now, so the test moves it a day instead of waiting one, against a real local host whose Last-Modified never moves while its body does: before, that board's new body never arrived. The README, the configure docstring and the CHANGELOG's Changed entry say so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LrnN33G65kWMuv8y7xpUk4
One posting with a field in a shape its reader did not expect raised TypeError or
AttributeError out of fetch(), which lost every board in the call: a number where a
list belongs (Greenhouse departments, Lever lists and allLocations, Ashby
secondaryLocations, Recruitee, Rippling, Gem and ClearCompany locations or offices,
Workday additionalLocations), text where an object belongs (Greenhouse location, Lever
categories), anything but text where a description or an address belongs. 0.2.0 did
the same on all nine of those readers.
Every reader now reads a field only in its shape, with the empty value otherwise:
clean() and html_to_text() give "" for anything that is not text (or, for clean, a
number), so no field holds Python's rendering of an odd value ("{'name': 'Leeds'}",
"True"), and models gains as_list, as_dict, as_str and ident. A posting whose id or title
cannot be read is left out, where it was kept as "gh-None" or with no title at all; a
JazzHR posting with no id was "jz-".
tests/test_odd_fields.py replaces every part of one posting, and of each record read for
a posting's details, with each of 18 odd values on all fifteen JSON readers (4,572 reads),
and changes each element of one posting in the four XML feeds (754 reads): every board
still reads, keeps its other postings, and gives no junk, no unreadable id and no empty
title. Before this, 17 of its 20 cases failed. Every reader's to_dict() over this
suite, main's and 0.2.0's is unchanged except where the pay fuzz test itself sends odd
values. The CHANGELOG has one Fixed headline for it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LrnN33G65kWMuv8y7xpUk4
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Two independent audits looked at everything unreleased since 0.2.0. Every finding they confirmed is fixed in its own commit, each with a test that fails before the fix.
First audit (nine findings)
Accept, and the caller's size limit is respected.configure(cache_dir=...)refuses anything that is not text or a path.salary_periodkeeps the count a board gives.job_id(job.url)gives backjob.id.Second audit
html_to_textis linear on hostile input, and its output is identical.request()cannot parse is not kept, so a board answering non-JSON costs one request a run, not two.Last-Modifieddate is asked about conditionally for at most 24 hours from its last full read (MODIFIED_MAX_AGE). That bounds staleness on a host whose date does not move with its data.gh-Noneor with an empty title.BoardErroror gives no posting, and the MCP server returns a parse error instead of stopping.job_id(url)cannot equaljob.idis documented.The version is still 0.2.0, so merging publishes nothing. The 0.3.0 release follows in its own PR.
How it was checked
Tests: 369 pass on Python 3.13 and 3.9, and
ruff check .is clean.Compatibility with 0.2.0:
Content-Length; that is the intended change.to_dict()over main's suite (349 jobs) and 0.2.0's (311 jobs) is identical, apart from the four new keys.Leak scan: the branch diff has 0 findings against the maintainer's five denylists. The tree's findings are the same as main's (the author attribution).
Not tested: any live board. Timings were measured on one machine.
pytestandruff check .passNo runtime dependency added; works on Python 3.9
robots.txt and the request limits are respected
🤖 Generated with Claude Code
https://claude.ai/code/session_01LrnN33G65kWMuv8y7xpUk4