Skip to content

feat(dataset): add contributor metadata storage and API - #6952

Open
xuang7 wants to merge 2 commits into
apache:mainfrom
xuang7:feat/dataset-contributor-backend
Open

feat(dataset): add contributor metadata storage and API#6952
xuang7 wants to merge 2 commits into
apache:mainfrom
xuang7:feat/dataset-contributor-backend

Conversation

@xuang7

@xuang7 xuang7 commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

This PR adds contributor metadata to datasets so that dataset authors and other contributors can be properly acknowledged. Each dataset can record a list of contributors, where each contributor has a name, a creator flag, and optional email, affiliation, and free-text comments fields.

Changes:

  • Adds a new dataset_contributor table with a foreign key to dataset and cascade deletion in texera_ddl.sql, along with the incremental migration script sql/updates/30.sql and changelog entry 30.
  • Updates file-service so that contributors are:
    • returned with the dataset detail response,
    • accepted during dataset creation, and
    • replaceable through a new POST /dataset/update/contributors endpoint (with did in the request body, matching the existing update/description and update/name endpoints), protected by the existing WRITE-access check.
  • Validates that each contributor has a name, and that name, email, and affiliation do not exceed 256 characters.
  • Models email, affiliation, and comments as optional in the API: omitted fields deserialize to null.

Currently, contributors are stored independently for each dataset. Cross-dataset contributor identity is left as a future extension. The UI will be added in a follow-up PR.

Any related issues, documentation, discussions?

Related to #6926

How was this PR tested?

7 ScalaTest cases were added, covering persisting contributors at dataset creation, replacing and clearing the list, validation failures (missing name, over-length fields), the WRITE-access check, and cascade deletion when the dataset is removed.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Fable 5)

@github-actions github-actions Bot added feature ddl-change Changes to the TexeraDB DDL platform Non-amber Scala service paths labels Jul 27, 2026
@github-actions

github-actions Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • Contributors with relevant context: @mengw15, @kunwp1, @Neilk1021
    You can notify them by mentioning @mengw15, @kunwp1, @Neilk1021 in a comment.

@codecov-commenter

codecov-commenter commented Jul 27, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 90.90909% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.58%. Comparing base (348fa88) to head (990cb4f).

Files with missing lines Patch % Lines
...ache/texera/service/resource/DatasetResource.scala 90.90% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff            @@
##               main    #6952   +/-   ##
=========================================
  Coverage     80.58%   80.58%           
- Complexity     3890     3891    +1     
=========================================
  Files          1162     1162           
  Lines         46264    46284   +20     
  Branches       5153     5155    +2     
=========================================
+ Hits          37280    37298   +18     
- Misses         7319     7320    +1     
- Partials       1665     1666    +1     
Flag Coverage Δ *Carryforward flag
access-control-service 70.00% <ø> (ø)
agent-service 77.42% <ø> (ø) Carriedforward from 0f7a426
amber 75.59% <ø> (ø)
computing-unit-managing-service 20.49% <ø> (ø)
config-service 65.97% <ø> (ø)
file-service 67.74% <90.90%> (+0.93%) ⬆️
frontend 83.22% <ø> (ø) Carriedforward from 0f7a426
notebook-migration-service 78.89% <ø> (ø)
pyamber 97.36% <ø> (ø) Carriedforward from 0f7a426
workflow-compiling-service 26.31% <ø> (ø)

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

⚠️ Benchmark changes need a look

🟢 2 better · 🔴 6 worse · ⚪ 7 noise (<±5%) · 0 without baseline

Compared against main 348fa88 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

config throughput MB/s latency max Δ latest / 7d
🔴 bs=10 sw=10 sl=64 491 0.3 19,730/28,688/28,688 us 🔴 +11.0% / 🔴 +77.1%
🔴 bs=100 sw=10 sl=64 1,023 0.624 96,775/114,012/114,012 us 🟢 -19.2% / 🟢 -6.8%
🔴 bs=1000 sw=10 sl=64 1,162 0.709 851,445/998,626/998,626 us 🔴 +11.4% / 🟢 +17.1%
Baseline details

Latest main 348fa88 from same runner

config metric PR latest main 7d avg Δ latest Δ 7d
bs=10 sw=10 sl=64 throughput 491 tuples/sec 531 tuples/sec 765.25 tuples/sec -7.5% -35.8%
bs=10 sw=10 sl=64 MB/s 0.3 MB/s 0.324 MB/s 0.467 MB/s -7.4% -35.8%
bs=10 sw=10 sl=64 p50 19,730 us 17,781 us 12,703 us +11.0% +55.3%
bs=10 sw=10 sl=64 p95 28,688 us 27,447 us 16,199 us +4.5% +77.1%
bs=10 sw=10 sl=64 p99 28,688 us 27,447 us 19,557 us +4.5% +46.7%
bs=100 sw=10 sl=64 throughput 1,023 tuples/sec 1,040 tuples/sec 965.22 tuples/sec -1.6% +6.0%
bs=100 sw=10 sl=64 MB/s 0.624 MB/s 0.635 MB/s 0.589 MB/s -1.7% +5.9%
bs=100 sw=10 sl=64 p50 96,775 us 91,655 us 103,830 us +5.6% -6.8%
bs=100 sw=10 sl=64 p95 114,012 us 141,053 us 110,778 us -19.2% +2.9%
bs=100 sw=10 sl=64 p99 114,012 us 141,053 us 118,713 us -19.2% -4.0%
bs=1000 sw=10 sl=64 throughput 1,162 tuples/sec 1,176 tuples/sec 992.33 tuples/sec -1.2% +17.1%
bs=1000 sw=10 sl=64 MB/s 0.709 MB/s 0.718 MB/s 0.606 MB/s -1.3% +17.1%
bs=1000 sw=10 sl=64 p50 851,445 us 849,865 us 1,016,959 us +0.2% -16.3%
bs=1000 sw=10 sl=64 p95 998,626 us 896,328 us 1,062,501 us +11.4% -6.0%
bs=1000 sw=10 sl=64 p99 998,626 us 896,328 us 1,082,611 us +11.4% -7.8%
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,407.06,200,128000,491,0.300,19729.69,28688.22,28688.22
1,100,10,64,20,1955.49,2000,1280000,1023,0.624,96775.48,114012.34,114012.34
2,1000,10,64,20,17215.42,20000,12800000,1162,0.709,851445.15,998625.50,998625.50

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR extends Texera’s dataset metadata support by introducing first-class dataset contributor records in the database and exposing them through the file-service dataset APIs, enabling proper attribution beyond the dataset owner.

Changes:

  • Adds a new dataset_contributor table via DDL + incremental migration and registers it in the SQL changelog.
  • Extends file-service dataset creation/detail responses to persist and return contributor metadata.
  • Introduces a new endpoint to replace a dataset’s contributor list, with ScalaTest coverage for persistence, replacement, validation, and cascade deletion.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
sql/updates/29.sql Adds incremental migration for the new dataset_contributor table.
sql/texera_ddl.sql Updates baseline schema to include dataset_contributor.
sql/changelog.xml Registers migration 29 in the changelog.
file-service/src/main/scala/org/apache/texera/service/resource/DatasetResource.scala Implements contributor persistence/retrieval + new update endpoint.
file-service/src/test/scala/org/apache/texera/service/resource/DatasetResourceSpec.scala Adds tests covering contributor lifecycle, validation, permissions, and cascade behavior.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread sql/updates/29.sql Outdated
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@xuang7
xuang7 force-pushed the feat/dataset-contributor-backend branch from e3c36cb to 52188de Compare July 31, 2026 22:54

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

Suppressed comments (3)

file-service/src/main/scala/org/apache/texera/service/resource/DatasetResource.scala:186

  • getContributorsByDid returns a List[Contributor] but the query has no ORDER BY, so API responses may return contributors in nondeterministic order depending on query plan/vacuum. Since the API models this as a list, it should return a stable order (e.g., insertion order via cid).
    ctx
      .selectFrom(DATASET_CONTRIBUTOR)
      .where(DATASET_CONTRIBUTOR.DID.eq(did))
      .fetch()

file-service/src/main/scala/org/apache/texera/service/resource/DatasetResource.scala:588

  • updateDatasetContributors treats a missing contributors field (None) as Nil, which will silently clear all contributors if a client accidentally sends { "did": ... }. For a dedicated "replace contributors" endpoint, it’s safer to require the field explicitly (clients can send an empty list to clear).
      DatasetResource.insertContributors(
        ctx,
        modificator.did,
        modificator.contributors.getOrElse(Nil)
      )

file-service/src/test/scala/org/apache/texera/service/resource/DatasetResourceSpec.scala:576

  • This test is titled "reject contributor fields longer than 256 characters" but it only exercises the over-length name path. Since the implementation also validates email and affiliation, add cases for those fields so the length validation is fully covered.
    assertThrows[BadRequestException] {
      datasetResource.updateDatasetContributors(
        DatasetResource.DatasetContributorsModification(
          createdDataset.dataset.getDid,
          Some(

@mengw15 mengw15 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ddl-change Changes to the TexeraDB DDL feature platform Non-amber Scala service paths

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants