feat(dataset): add contributor metadata storage and API - #6952
Conversation
Automated Reviewer SuggestionsBased on the
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #6952 +/- ##
=========================================
Coverage 80.58% 80.58%
- Complexity 3890 3891 +1
=========================================
Files 1162 1162
Lines 46264 46284 +20
Branches 5153 5155 +2
=========================================
+ Hits 37280 37298 +18
- Misses 7319 7320 +1
- Partials 1665 1666 +1
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🔴 | bs=10 sw=10 sl=64 | 491 | 0.3 | 19,730/28,688/28,688 us | 🔴 +11.0% / 🔴 +77.1% |
| 🔴 | bs=100 sw=10 sl=64 | 1,023 | 0.624 | 96,775/114,012/114,012 us | 🟢 -19.2% / 🟢 -6.8% |
| 🔴 | bs=1000 sw=10 sl=64 | 1,162 | 0.709 | 851,445/998,626/998,626 us | 🔴 +11.4% / 🟢 +17.1% |
Baseline details
Latest main 348fa88 from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 491 tuples/sec | 531 tuples/sec | 765.25 tuples/sec | -7.5% | -35.8% |
| bs=10 sw=10 sl=64 | MB/s | 0.3 MB/s | 0.324 MB/s | 0.467 MB/s | -7.4% | -35.8% |
| bs=10 sw=10 sl=64 | p50 | 19,730 us | 17,781 us | 12,703 us | +11.0% | +55.3% |
| bs=10 sw=10 sl=64 | p95 | 28,688 us | 27,447 us | 16,199 us | +4.5% | +77.1% |
| bs=10 sw=10 sl=64 | p99 | 28,688 us | 27,447 us | 19,557 us | +4.5% | +46.7% |
| bs=100 sw=10 sl=64 | throughput | 1,023 tuples/sec | 1,040 tuples/sec | 965.22 tuples/sec | -1.6% | +6.0% |
| bs=100 sw=10 sl=64 | MB/s | 0.624 MB/s | 0.635 MB/s | 0.589 MB/s | -1.7% | +5.9% |
| bs=100 sw=10 sl=64 | p50 | 96,775 us | 91,655 us | 103,830 us | +5.6% | -6.8% |
| bs=100 sw=10 sl=64 | p95 | 114,012 us | 141,053 us | 110,778 us | -19.2% | +2.9% |
| bs=100 sw=10 sl=64 | p99 | 114,012 us | 141,053 us | 118,713 us | -19.2% | -4.0% |
| bs=1000 sw=10 sl=64 | throughput | 1,162 tuples/sec | 1,176 tuples/sec | 992.33 tuples/sec | -1.2% | +17.1% |
| bs=1000 sw=10 sl=64 | MB/s | 0.709 MB/s | 0.718 MB/s | 0.606 MB/s | -1.3% | +17.1% |
| bs=1000 sw=10 sl=64 | p50 | 851,445 us | 849,865 us | 1,016,959 us | +0.2% | -16.3% |
| bs=1000 sw=10 sl=64 | p95 | 998,626 us | 896,328 us | 1,062,501 us | +11.4% | -6.0% |
| bs=1000 sw=10 sl=64 | p99 | 998,626 us | 896,328 us | 1,082,611 us | +11.4% | -7.8% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,407.06,200,128000,491,0.300,19729.69,28688.22,28688.22
1,100,10,64,20,1955.49,2000,1280000,1023,0.624,96775.48,114012.34,114012.34
2,1000,10,64,20,17215.42,20000,12800000,1162,0.709,851445.15,998625.50,998625.50There was a problem hiding this comment.
Pull request overview
This PR extends Texera’s dataset metadata support by introducing first-class dataset contributor records in the database and exposing them through the file-service dataset APIs, enabling proper attribution beyond the dataset owner.
Changes:
- Adds a new
dataset_contributortable via DDL + incremental migration and registers it in the SQL changelog. - Extends
file-servicedataset creation/detail responses to persist and return contributor metadata. - Introduces a new endpoint to replace a dataset’s contributor list, with ScalaTest coverage for persistence, replacement, validation, and cascade deletion.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
sql/updates/29.sql |
Adds incremental migration for the new dataset_contributor table. |
sql/texera_ddl.sql |
Updates baseline schema to include dataset_contributor. |
sql/changelog.xml |
Registers migration 29 in the changelog. |
file-service/src/main/scala/org/apache/texera/service/resource/DatasetResource.scala |
Implements contributor persistence/retrieval + new update endpoint. |
file-service/src/test/scala/org/apache/texera/service/resource/DatasetResourceSpec.scala |
Adds tests covering contributor lifecycle, validation, permissions, and cascade behavior. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
e3c36cb to
52188de
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.
Suppressed comments (3)
file-service/src/main/scala/org/apache/texera/service/resource/DatasetResource.scala:186
getContributorsByDidreturns aList[Contributor]but the query has noORDER BY, so API responses may return contributors in nondeterministic order depending on query plan/vacuum. Since the API models this as a list, it should return a stable order (e.g., insertion order viacid).
ctx
.selectFrom(DATASET_CONTRIBUTOR)
.where(DATASET_CONTRIBUTOR.DID.eq(did))
.fetch()
file-service/src/main/scala/org/apache/texera/service/resource/DatasetResource.scala:588
updateDatasetContributorstreats a missingcontributorsfield (None) asNil, which will silently clear all contributors if a client accidentally sends{ "did": ... }. For a dedicated "replace contributors" endpoint, it’s safer to require the field explicitly (clients can send an empty list to clear).
DatasetResource.insertContributors(
ctx,
modificator.did,
modificator.contributors.getOrElse(Nil)
)
file-service/src/test/scala/org/apache/texera/service/resource/DatasetResourceSpec.scala:576
- This test is titled "reject contributor fields longer than 256 characters" but it only exercises the over-length name path. Since the implementation also validates
emailandaffiliation, add cases for those fields so the length validation is fully covered.
assertThrows[BadRequestException] {
datasetResource.updateDatasetContributors(
DatasetResource.DatasetContributorsModification(
createdDataset.dataset.getDid,
Some(
What changes were proposed in this PR?
This PR adds contributor metadata to datasets so that dataset authors and other contributors can be properly acknowledged. Each dataset can record a list of contributors, where each contributor has a name, a creator flag, and optional email, affiliation, and free-text comments fields.
Changes:
dataset_contributortable with a foreign key todatasetand cascade deletion intexera_ddl.sql, along with the incremental migration scriptsql/updates/30.sqland changelog entry 30.file-serviceso that contributors are:POST /dataset/update/contributorsendpoint (withdidin the request body, matching the existingupdate/descriptionandupdate/nameendpoints), protected by the existing WRITE-access check.Currently, contributors are stored independently for each dataset. Cross-dataset contributor identity is left as a future extension. The UI will be added in a follow-up PR.
Any related issues, documentation, discussions?
Related to #6926
How was this PR tested?
7 ScalaTest cases were added, covering persisting contributors at dataset creation, replacing and clearing the list, validation failures (missing name, over-length fields), the WRITE-access check, and cascade deletion when the dataset is removed.
Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Fable 5)