Skip to content

An attribute whose range is xsd:hexBinary is imported as text - #1008

Merged
WaylandYang merged 2 commits into
deeplethe:devfrom
Maya-Kid:fix/binary-xsd-ranges-are-text
Sep 29, 2026
Merged

WaylandYang merged 2 commits into
deeplethe:devfrom
Maya-Kid:fix/binary-xsd-ranges-are-text

Conversation

@Maya-Kid

Copy link
Copy Markdown
Contributor

Follows the #1001 review, which pointed at the 13 UCO attributes still skipped because their range is xsd:hexBinary or xsd:base64Binary.

Problem

  • ADR 0001 decision 20 skips a range whose values the extractor could never read out of prose, and it counts hexBinary among them. The dividing line it states is "can this value appear in a sentence".
  • hexBinary is written as hex digits. In prose it carries a hash, an address or a flag, for example "The file's SHA-256 is 9f86d0…" or "the region starts at 7FFE0000".
  • UCO types 12 attributes this way, among them types:hashValue, observable:androidID, the region start and end addresses, and the ICMP type and code. On the merged UCO 1.5.0 all 12 are skipped at import as unusable_range, although a forensic report states them in sentences.

Change

  • In crates/utopia-ingest/src/ontology_rdf.rs, datatype_of maps xsd:hexBinary to text, alongside anyURI and token: the value is kept as it is written. unusable no longer lists it, and its doc comment says why.
  • base64Binary stays skipped, as do XMLLiteral and the XML identifiers.
    • The same doc comment's own example is a base64 floor plan: a block, not a value that a sentence states, and a fact read out of it would be wrong.
    • UCO's one base64Binary attribute, observable:byteStringValue ("the raw, byte-string representation of the extracted string"), is that kind of payload.
  • In docs/decisions/0001-ontology-import-and-governance.md, decision 20 moves hexBinary out of the skipped list, and a revision entry records the change.

Verification

Run in containers on a Linux host: Rust 1.98.1 and PostgreSQL 16 with pgvector.

  • a_value_we_can_store_is_kept_even_when_we_cannot_type_it now checks that hexBinary maps to text, and that base64Binary, XMLLiteral and QName stay unusable. On dev with only the test changed, it fails at the hexBinary check.
  • cargo fmt --all --check and cargo clippy --locked --workspace --all-targets -- -D warnings are clean.
  • UTOPIA_TEST_REQUIRE_PDFTOTEXT=1 cargo test --locked --workspace passes: 1304 passed, 0 failed.

UCO 1.5.0 was run through the import plan that the preview returns, on an empty base, as measured for #1001:

dev this branch
Datatype properties imported 549 561
Skipped: key collision 16 16
Skipped: unusable range 13 1 (byteStringValue)
Skipped: no domain 1 1 (core:informalType)

Everything else in the plan is unchanged. None of the five bundled packs uses hexBinary or base64Binary.

Signed-off-by: dada-yan <BinjunYann@gmail.com>

@WaylandYang WaylandYang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. Thanks @Maya-Kid for picking this up from the #1001 review.

Reviewed at d14140d. The line decision 20 draws is whether a value can appear in a sentence, and a hash or an address does; base64Binary stays skipped for the reason the comment already gives. The test now pins both sides, and the record carries the revision.

CI did not run on this PR (the CI workflow is not registered on the repository at the moment, which we are looking into), so I ran the backend steps locally on this commit: cargo fmt --check and clippy with -D warnings are clean, the migrations run twice, and cargo test --workspace against Postgres gives 1304 passed, 0 failed. Two chat stop tests timed out on the first run while the machine was compiling and passed on the second; they are not touched by this change.

It will be merged once CI is back and green.

@WaylandYang

Copy link
Copy Markdown
Contributor

Closing and reopening to trigger CI: the workflow did not run when this was opened. Nothing is wrong with the PR.

@WaylandYang WaylandYang reopened this Sep 29, 2026
@WaylandYang
WaylandYang merged commit c7ac51a into deeplethe:dev Sep 29, 2026
6 checks passed
@Maya-Kid
Maya-Kid deleted the fix/binary-xsd-ranges-are-text branch September 30, 2026 01:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants