Licensing: record-and-track policy, and the per-dataset inventory for migration #35

Activity

1. mmcky commented on Aug 7, 2026

   

   mmcky

   on Aug 7, 2026

   ContributorAuthor

   More actions

   The six high_dim_data files are the next wave to arrive (P3 in migration.yml), so I went through their upstream provenance. The licence expectations in the assessment table all hold. Two things do not, and both need deciding before the fold PR is written rather than at migration time.

   1. SCF+ has no existing status to inherit

   Policy rule 2 says a file arrives with its recorded status, not a fresh review. For SCF+ there is nothing recorded to carry over. SCF_plus/README.md (4,974 B) is a variable dictionary — a header, four lines on units and imputation implicates, then a 60-row variable/label table. It contains zero URLs, zero dates, no licence, no DOI, and the word "source" does not appear anywhere in it. The provenance table lives only in cross_section/README.md (2,173 B), which covers the four cross-section files and says nothing about SCF+.

   So the SCF row needs a positive decision, and there are two honest ones. Either establish it against the Kuhn–Schularick–Steins (JPE 2020) deposit record — which is what the table already asks for, and which needs someone to match the deposit to these bytes — or, if that match cannot be made, land retrieved: null, license.name: null and integrity.upstream.status: unverifiable rather than inventing a value. Specifically: do not use the high_dim_data commit date as retrieved. That dates our copy, not the retrieval.

   Two related mechanics. The minis are derived, not retrieved: SCF_plus/generating_mini.md reads SCF_plus.dta, keeps 7 of its columns and renames them; both to_csv calls are commented out upstream, so the committed builder writes nothing while the fetch and transform are complete. And SCF_plus.dta appears in the table above but not in P3's dataset list — no lecture reads the .dta directly, only generating_mini.md does. Whichever way that goes, the deposit-record question still has to be answered, because both minis inherit their record from it.

   2. webscrape_forbes.ipynb is a separate row from the Forbes data

   The table treats the two CSVs and their builder as one item. They are two different questions, and they need not resolve the same way. Caching data we already serve is the policy's case. Republishing the scraper from a canonical, publicly-served dataset repo is not.

   The global2000 cell requests https://www.forbes.com/forbesapi/org/global2000/2020/position/true.json?limit=2000 — an undocumented internal endpoint, still live — with a spoofed Chrome 87 user-agent, a matching referer, and hardcoded GDPR consent cookies (notice_behavior: "expressed,eu", notice_gdpr_prefs). The billionaires cell hits a second undocumented endpoint, http://www.forbes.com/ajax/list/data. That is a terms-of-service question about the method, independent of whatever redistribution the CSVs get. The data can land restricted-and-tracked while the notebook either does not come across at all, or comes across with the consent-cookie and user-agent spoof stripped and a note that it no longer runs as-is.

   The six, with evidence

   Dataset	Source as recorded upstream	Expectation
   forbes-global2000.csv, forbes-billionaires.csv	forbes.com/lists/global2000/, forbes.com/billionaires/ — recorded in cross_section/README.md, no grant of any kind	restricted, as the table says. Treat as the countries.csv case
   cities_us.csv, cities_brazil.csv	worldpopulationreview.com/us-cities, /countries/cities/brazil	restricted expected. Like countries.csv, the substitute is abundant — both are plain population-by-city tables
   SCF_plus_mini.csv, SCF_plus_mini_no_weights.csv	nothing recorded; derived from SCF_plus.dta	needs the deposit record or an explicit unverifiable record, per above

   Two files the inventory does not cover

   Both are still to migrate and appear in neither the Open items nor the assessment table. usa-gini-nwealth-tincome-lincome.csv is a second-order derivative — _static/lecture_specific/inequality/data.ipynb:100 writes it from SCF_plus_mini — so it inherits whatever record SCF+ gets and cannot be assessed ahead of it. graph.txt is QuantEcon's own generated output (short_path.md:272 writes it with %%file, 100 nodes), so it belongs in the author-assembled row with longprices.xls and chapter_3.xlsx.

   None of this blocks the wave, consistent with the policy. The SCF+ record is the one item that has to be chosen rather than copied, and it has to be chosen while the manifest is being written.
2. mmcky

   added a commit that references this issue on Aug 10, 2026

   Fold in the six high_dim_data datasets (PR B1) (#62)

   ...

   55d4e05
3. mmcky commented on Aug 10, 2026

   

   mmcky

   on Aug 10, 2026

   ContributorAuthor

   More actions

   P3 migrated — five of the "to assess" rows are now settled, and four came back restricted

   The six high_dim_data datasets landed in #62 and SCF_plus.dta in #63, so the top three rows of the "To assess as they migrate" table have answers. Two of the three expectations recorded there held; the third was right about the licence and wrong about how it would be established.

   Dataset(s)	Expected	Recorded
   forbes-global2000.csv, forbes-billionaires.csv	most likely restricted	redistribution: restricted, license.name: null	as expected
   cities_us.csv, cities_brazil.csv	expect to need a decision	redistribution: restricted, license.name: null	as expected
   SCF_plus_mini.csv, SCF_plus_mini_no_weights.csv, SCF_plus.dta	verify against the deposit record	redistribution: permitted, license.name: null	no deposit record exists to verify against

   All five are inherited exposures — served publicly from lecture-python-intro and lecture-wasm for years — so under the record-and-track policy they migrated with the licence recorded as found. None blocked its wave.

   3. Forbes ×2 — restricted, migrated, about to be in use

   Commercial publisher, editorially assembled ranking, obtained through an undocumented internal API. No redistribution grant of any kind is stated, and attribution alone does not cure that. Both manifests carry redistribution: restricted with a note pointing here.

   Unlike countries.csv, there is no trivial substitute: heavy_tails uses these to illustrate that firm size and wealth are heavy-tailed, and the pedagogical point depends on the data being a recognisable real-world rich list. An openly-licensed replacement would change what the lecture demonstrates, not just its numbers. So permission-seeking is the more likely route here, and it is worth starting earlier than the others.

   4. World Population Review ×2 — restricted, migrated, about to be in use

   Same publisher class as countries.csv and the same absence of a redistribution grant. heavy_tails uses one column of each — pop2023 — to plot a city-size CCDF.

   This is the strongest replacement candidate of the four: the lectures use a single population column, and openly-licensed city-population tables are abundant. Worth pairing with the countries.csv decision, since resolving that publisher class once covers three of the five open rows.

   5. SCF+ — permitted, but the deposit could not be located at all

   The expectation in the table above was to "verify against the deposit record rather than a homepage". That turned out not to be possible. Searched to exhaustion on 2026-08-10:

   Where	Result
   Crossref, DOI 10.1086/708815	resolves to the article; license: null, and no data relation registered
   DataCite	zero results
   Harvard Dataverse	zero results
   openICPSR	403
   JPE supplementary-material path	403

   So license.name, license.url and retrieved are honest nulls with a note in all three records, and integrity.upstream.status is unverifiable rather than a soft unverified. redistribution: permitted reflects that this is replication data for a published article that QuantEcon has served publicly for years — recorded as a reasoned position, like ames_house_prices.csv above, not as a verified licence.

   Worth stating plainly so it is not re-derived: nothing that migrated supplies SCF+ provenance. The SCF_plus/README.md inherited upstream is a variable dictionary — zero URLs, zero dates, no licence, no DOI, not even the word "source". Its content went into SCF_plus_mini.csv.yml's schema.columns[].description rather than landing as a second unstructured record. The audit trail is sources/README.md.

   6. webscrape_forbes.ipynb — a republication question separate from the data licence

   This is not the same question as rows 3 above, and it should not be resolved by answering those. The builder now committed at builders/webscrape_forbes.ipynb (builder_status: committed-frozen):

   • spoofs a browser user-agent string;
   • hardcodes GDPR consent cookies (notice_behavior, notice_gdpr_prefs) to defeat a consent gate;
   • targets an undocumented internal endpoint, forbesapi/org/global2000/…, not a published API;
   • reaches the billionaires list over plain http://.

   The data licence question is may we cache and serve these bytes with attribution. This one is should QuantEcon publish, under its own name, a working recipe for circumventing a commercial publisher's access controls — which stands whatever the answer on the data is, and would stand even if Forbes granted redistribution tomorrow.

   It is committed deliberately: a constructed dataset without its builder is a bug, and committed-frozen records that it will not be re-run. The options if that reads as too much are to redact the headers and cookies with a note (which costs the artifact its value as a record of what actually ran), or to remove it and mark both manifests unrecovered (which reintroduces the gap the status exists to make visible). Recording rather than deciding, since it is a judgement call about what the org publishes rather than a data question.

   ---

   Suggested edit to the table above when someone next touches the body: the three settled rows move into Open items as 3, 4 and 5, leaving maketable1/2/4.dta through mpd2020.xlsx to assess.
4. mmcky

   added 2 commits that reference this issue on Aug 12, 2026

   Record D11: invest in qeld, defer data.quantecon.org indefinitely (#76)

   ...

   990eea6

   Land wave B2' — the hansen_singleton pair and their builders (Track B) (

   ...

   16ddbbf
5. mmcky commented on Aug 17, 2026

   

   mmcky

   on Aug 17, 2026

   ContributorAuthor

   More actions

   Correction: the two BBH files are not simply CC BY 4.0

   The inventory in this issue's body records:

   | bbh_macro_quarterly.csv, bbh_michigan_monthly.csv | Zenodo replication package | already known CC BY 4.0 |

   That is right about the deposit and wrong about one of the two files. Established while writing their manifests for #92.

   bbh_macro_quarterly.csv — the row stands. All fourteen series are US federal statistics (BEA, BLS, CBO, Federal Reserve Board) delivered through FRED, plus PIRIC, a series contributed to FRED by Riccardo DiCecio and tagged "public domain: citation requested". None of the fourteen FRED series pages carries a copyright or no-redistribution block — checked against a positive control, FRED's SP500 page, which does carry S&P Dow Jones Indices' "Reproduction … is prohibited" notice, so the zero is a real zero rather than a broken check. redistribution: permitted, attribution carried by the manifest's citation.

   bbh_michigan_monthly.csv lands redistribution: restricted. Four of its five columns are University of Michigan Surveys of Consumers aggregates. Michigan's usage agreement (https://data.sca.isr.umich.edu/agreement.php, read 2026-08-17) states that the data "may be displayed, reformatted, and printed for your organization's use" and then: "You agree not to reproduce, retransmit, distribute, sell, publish, or broadcast the data and materials from the Surveys of Consumers website without the express written consent of the University of Michigan."

   The CC BY 4.0 grant is the depositors', and Bhandari, Borovička and Ho are not the rights holder for that material. Whether they obtained Michigan's written consent is not stated anywhere in the deposit and could not be established. So the deposit's licence does not settle the question for those four columns — it only tells us what the depositors granted over what was theirs to grant. The fifth column, UNRATE, is a BLS series and clean.

   Decision taken 2026-08-17: migrate as restricted and register here, on the same reasoning as countries.csv (workspace-lectures#20). This is an inherited exposure, not a new one — the bytes have been served publicly from lecture-python-advanced.myst since 2026-06-12 — and per AGENTS.md an already-served file may be cached when marked restricted and logged for review. Pausing the migration would not have reduced the existing exposure.

   Two facts that bear on how much is actually at stake, recorded so a future review does not have to re-derive them. What is published is four aggregate columns for the all-households cell only — not the demographic breakdowns the deposit ships, and not the microdata. And these are the same published aggregates Michigan puts on its own tables pages. Michigan's terms bind someone who obtains the data from its website; QuantEcon obtained these bytes from a CC BY 4.0 Zenodo deposit.

   Suggested edit to the body table, so the inventory stops asserting the opposite of the manifest:

   dataset	source	licence
   bbh_macro_quarterly.csv	Zenodo replication package (doi:10.5281/zenodo.10194324)	CC BY 4.0 — permitted; underlying series are US federal statistics via FRED
   bbh_michigan_monthly.csv	same deposit	restricted — deposit declares CC BY 4.0, but 4 of 5 columns are Michigan Surveys of Consumers aggregates whose terms forbid redistribution without written consent; depositors are not the rights holder. Inherited exposure, served publicly since 2026-06-12

   I have not edited the body — this is the register's own record to keep, and the row above is the correction it needs.
6. mmcky

   added a commit that references this issue on Aug 17, 2026

   Land wave C1 — the three advanced.myst datasets that needed no rename…

   ...

   ca2614c
7. mmcky commented on Sep 29, 2026

   

   mmcky

   on Sep 29, 2026

   ContributorAuthor

   More actions

   2026-09-29: QuantEcon/workspace-lectures#20, a private workspace issue, is folded in here and closed in favour of this one. It tracked a single file, countries.csv, already Open item 1 above. That issue was where each item's fate (confirm, replace or remove) was to be decided, and that role moves here too, so this issue is now the one licence register. It stays open as a standing record, as the P4 work plan (#118) already classes it. It does not gate the datasets migration: licensing does not gate migration (settled 2026-08-06), and the only gate, promoting a branded public open-data host, is unscheduled because D11 (2026-08-12) deferred data.quantecon.org indefinitely (#37).

   Correction to Open item 1: countries.csv has six readers, not two. Measured 2026-09-29, lectures/pandas_panel.md reads this repo's copy in lecture-python-programming, lecture-python.myst and lecture-python.zh-cn, and in the programming translations lecture-python-programming.zh-cn, lecture-python-programming.fr and lecture-python-programming.fa. The manifest's consumers lists the first three.

   To do when the body is refreshed. The body stops at the first ten datasets and never absorbed its own comments. The manifests and CATALOG.md remain the register of record; as of 2026-09-29 the body lacks the following.

   • Rebuild the tables from all 44 manifests on main (Add us_household_net_worth_2022.csv: SCF 2022 household net worth with survey weights #144 would add a 45th, permitted). Every one records license.redistribution with a verified date: 38 permitted, 6 restricted. Drop "To assess as they migrate".
   • List the six restricted files among the Open items, with their readers: countries.csv (above), cities_us.csv, cities_brazil.csv, forbes-global2000.csv, forbes-billionaires.csv and bbh_michigan_monthly.csv. This absorbs the 2026-08-10 comment's items 3 and 4 and the 2026-08-17 correction that bbh_michigan_monthly.csv is restricted, not CC BY 4.0. That comment's item 5 (SCF+, permitted with no deposit record found) moves into the body too, and its item 6, whether to keep publishing builders/webscrape_forbes.ipynb, stays an open owner decision.
   • Add the 14 files whose manifests say they are registered or logged here but have no recorded outcome yet: NEWQDATA.csv, caron.npy, dataBHS.csv, fp.dta, hansen_jagannathan_1991_data.json, hansen_singleton_1982_data.csv, hansen_singleton_1983_data.csv, life-expectancy-vs-gdp-per-capita.csv, longprices.xls, maketable1.dta, maketable2.dta, maketable4.dta, nom_balances.npy and usa-gini-nwealth-tincome-lincome.csv. sources/README.md makes the same claim for two builder inputs, NEWQDATA.MAT and dataBHS.mat.
   • Add the four business_cycle snapshots whose manifests landed on 2026-09-01: gdp_growth_annual.csv (in P4 groundwork: manifest, four-stage builder and provenance/ for business_cycle #109, as business_cycle_data.csv, renamed in Record the four schema and naming decisions: rename, pattern capture groups, null placement, dtype sweep #124), and private_credit_to_gdp.csv, unemployment_rate_annual.csv and us_business_cycle_monthly.csv (in P4: the business_cycle snapshot set — three World Bank tables and a FRED composite #114).
   • Correct the row labels. The published file is dataBHS.csv (the .mat is a builder input in sources/), SCF_plus.dta is an unpublished builder input, and graph.txt was retired in place, never migrated. Of the seven "author-assembled" files only chapter_3.xlsx is QuantEcon's own transcription: longprices.xls, assignat.xlsx, dette.xlsx and fig_3.xlsx are verbatim, and caron.npy and nom_balances.npy have no recorded source, so no licence can be established for them yet.
   • Register two questions for an owner decision (measured 2026-09-29; no legal assessment made). First, the University of Michigan's terms are read two ways: bbh_michigan_monthly.csv is restricted on Michigan's usage agreement, while us_business_cycle_monthly.csv records its UMCSENT column, which FRED classes "Copyrighted: Citation Required", as permitted, reading Michigan's terms as permitting use with citation. That file has no consumers yet, so this is cheapest to settle before Include business_cycle: what it reads, what data-lectures now serves, and the sequence lecture-wasm#70 adopts it. Second, clause (p) of FRED's Terms of Use, on storing, caching or archiving FRED content, has not been weighed for the FRED-built files.
   • Restate the gate in AGENTS.md's current words, before this repo "is ever promoted as a branded public open-data host", which D11 leaves unscheduled. Replace the pointers to Phase 4 cutover: swap interim raw URLs → data.quantecon.org/lectures #15 (not planned, 2026-08-12), to QuantEcon/workspace-lectures#20 and to "Part of Scaffolding: QuantEcon/data → data-lectures (canonical lecture-data repo) #8" (Scaffolding: QuantEcon/data → data-lectures (canonical lecture-data repo) #8 completed 2026-08-31).
8. mmcky commented on Sep 30, 2026

   

   mmcky

   on Sep 30, 2026

   ContributorAuthor

   More actions

   2026-09-30: @mmcky has decided the two questions registered on 2026-09-29, and #152 records them.

   1. UMCSENT in us_business_cycle_monthly.csv is permitted, on Michigan's FAQ. Michigan publishes two texts. The data portal's usage agreement forbids redistribution without written consent. Its FAQ says "Any data, tables, or charts available to the public on our main website can be used without permission" and asks for a citation. The headline index is main-website data: tables.html serves it as tbmics.csv, and every committed value matches that table. So it sits on the permitted side. The four Michigan columns of bbh_michigan_monthly.csv come from portal-only tables and stay restricted. Both manifests now record that line.
   2. FRED stays a known yes, with clause (p) recorded. Clause (p) of FRED's full Terms of Use bars storing or caching FRED content without the Bank's prior written consent; the summarized terms list no such prohibition. AGENTS.md records the posture once for the source, and the seven FRED-derived manifests point to it. Asking the Bank for consent waits for the promotion gate, like every other item on this register.

   The check also turned up two errors, both corrected in #152. FRED labels USREC and M0892AUSM156SNBR "Copyrighted: Citation Required" too; the composite and fred_data.csv said otherwise. And M0892AUSM156SNBR is seasonally adjusted, not NSA. This settles the item "Register two questions for an owner decision" in the comment of 2026-09-29.
9. mmcky

   added a commit that references this issue on Sep 30, 2026

   licence: rest UMCSENT on Michigan's FAQ and record FRED's caching cla…

   ...

   bbb4cc6