Skip to content

proposal: mapping on arrival - a business key becomes a relation, and a gate ignores what is not understood #42

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
iliyan-velichkov wants to merge 1 commit into
base: main
Choose a base branch
from
Open

proposal: mapping on arrival - a business key becomes a relation, and a gate ignores what is not understood #42

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
185 changes: 185 additions & 0 deletions proposals/0021-arrival-mapping.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,185 @@
# Mapping on arrival — a business key becomes a relation, and a gate ignores what is not understood

- **Status:** draft
- **Issue:** [IntentFile/intent-specification#30](https://github.com/IntentFile/intent-specification/issues/30)
- **Implementation:** [eclipse-dirigible/dirigible#6769](https://github.com/eclipse-dirigible/dirigible/issues/6769)
([PR #6839](https://github.com/eclipse-dirigible/dirigible/pull/6839))
- **Companion:** [`0017-declared-payload.md`](0017-declared-payload.md) — the same problem in the
other direction, whose value vocabulary this one mirrors; and
[`0018-outbound-departures.md`](0018-outbound-departures.md), whose own issue named this half.

## The problem

An `inbound` entry ingests the arriving JSON **as the entity**. That is only expressible when the
sender's payload already *is* the entity, field for field — and a real arrival contract is not. It is
an envelope, frozen between two products long before either one modelled anything:

```json
{ "messageId": "9f9d1c9e-...", "type": "user.assignment.requested", "version": 1,
"tenantId": "acme", "appId": "library", "email": "new.user@example.com", "role": "User" }
```

Ingesting that needs three things the construct cannot say:

1. **Field mapping** — the envelope's keys are not the entity's names.
2. **A business key resolved to a relation** — `tenantId: "acme"` and `role: "User"` identify the
`Tenant` and the `AssignmentRole` records, and what the entity stores are references to them. This
is the single most common requirement of any arrival, and on its own it is enough to force a
hand-written consumer.
3. **A type and version gate** — accept `user.assignment.requested` at version 1, and *ignore*
anything else with a warning rather than failing it, so a sender rolling out version 2 does not
fill the receiver's error queue with messages it will keep sending.

The consequence is the one worth avoiding: an application can declare its entity, its deduplication,
its process trigger **and** its arrival, and still need custom code purely because a relation arrives
as a name instead of a reference. The construct looks like it covers the case, and the hand-off is
discovered when the first payload lands.

The asymmetry is also hard to defend on its own terms. The format admitted the declared envelope for
messages **leaving** ([`0017`](0017-declared-payload.md)) on the argument that real contracts are not
records; an arriving message is the same contract read in the other direction, and until now it had to
be a record.

## The proposed shape

Two optional keys on an `inbound` entry, valid on **every** arrival, because they describe the payload
rather than the transport it came over.

```yaml
inbound:
- name: userAssignments
source: { queue: "codbex.user-assignment-requests" }
accept: { type: user.assignment.requested, version: 1 } # anything else: warn and ignore
create: TenantUserAssignment
map:
messageId: messageId # entity field <- envelope key
email: email
seats: seatCount
tenant: { lookup: Tenant, by: tenantId, from: tenantId } # business key -> relation
role: { lookup: AssignmentRole, by: name, from: role }
```

`map:` omitted keeps the current behaviour exactly — the payload is the record — so no existing file
changes meaning and the simple case stays one line.

## Expected behaviour

**The gate.** `accept:` names envelope keys and the value each must carry. A message matching all of
them is ingested. A message that does not match is **acknowledged and ignored**, with a warning
recording what arrived and what was expected. It is deliberately *not* failed: failing an arrival asks
the sender to deliver it again, and a message this receiver structurally does not understand will be
identical every time. Over a request/response arrival the answer says the same thing — the message was
received and not acted on — rather than reporting a fault the sender did not commit.

**The projection.** `map:` fills the named property of `create:` from the named envelope key. A key the
map does not mention is not part of the record; an envelope key that is absent leaves its property
unset, which the entity's own required-value rules then judge like any other missing value. The
authored order is irrelevant to the result.

**The lookup.** `{ lookup: <Entity>, by: <field>, from: <envelopeKey> }` reads the envelope key, finds
the `lookup` entity whose `by` field equals it, and stores a reference to that record in the mapped
relation. Exactly one match fills the relation. **No match rejects the whole arrival** — with a
diagnostic naming the value that could not be resolved — because a record stored with an unresolved
reference is a record nobody can trace back to the party that asked for it, and it is indistinguishable
from one that never named a party at all.

**The write path is unchanged.** Whatever is declared, the record is saved through the entity's
ordinary write path, so validations, translations and the create event behave exactly as for any other
write. Mapping changes what the record is built from, never how it is stored.

## Edge rules

- **`by:` must identify at most one record** — a field declared unique, or the entity's key. A lookup
that could match several rows would have to pick one, and picking silently is a worse outcome than
failing; so a non-unique `by:` is refused when the file is read, not when two rows first collide.
- **`by:` must be a field a business key can travel as** — text or an integer. A temporal key would
make the comparison depend on how the sender formatted a date, which is not a key.
- **A lookup fills a to-one relation only.** On a plain field it is meaningless: there is nothing to
resolve to.
- **`lookup:` must name the relation's own target.** Two ways of saying which entity is read could only
drift apart.
- **A mapped key must be a field or a to-one relation of `create:`.** A key naming neither is an
authoring error, reported with the nearest declared name (per
[`0014`](0014-unknown-keys-are-errors.md)).
- **A lookup value's keys are closed** — `lookup`, `by`, `from`, all three required. An unrecognised
one is an error.
- **The entity's key MUST NOT be mapped.** It is assigned when the record is created. An arrival that
carries its own identifier gives it a field of its own, declared unique — which is also what makes a
redelivery of the same message refuse itself.
- **A valueless key is an error, not an empty one.** `accept: { type: }` and `map: { email: }` are
reported. Read as "no value", the first would gate on nothing — accepting every message — and the
second would fill nothing, in both cases losing an authored promise silently.
- **`accept:` values are scalars.** A nested object or list is not a gate.
- **A plain envelope key on a to-one relation** copies the target's key as it arrives, which remains
valid: a sender that already speaks in references needs no lookup.
- **Same-model targets in this revision.** `lookup:` reads an entity declared in the same model.
Resolving a business key against a model that only declares the entity by reference is a larger
question and is left out deliberately rather than half-specified.

## Prior art / workarounds

A hand-written subscriber or endpoint alongside the generated application: parse the envelope, query
each register by its business key, build the entity, save it through the generated write path. It is
mechanical, it is written again in every application that consumes the same contract, and — because the
contract and the resolution live in that code — the arrival is invisible to anyone reading the model.

Some formats offer a general transformation language for this. That is the trade this proposal
deliberately does not make: three value forms and one lookup shape express the contracts people
actually have, while a transformation language moves the arrival out of the model and into a second
program written in a worse language. An arrival that needs more than this is an algorithm, and the
honest answer is a hand-written handler over the same write path.

## Specification text

**Anchor:** Declarative glue > `inbound — arrivals from outside` (immediately after that section's
normative block)

An arrival may declare how its payload is **read**. Without these keys the payload is the record; with
them it is an envelope, which is what an integration contract normally is.

```yaml
inbound:
- name: userAssignments
source: { queue: "user-assignment-requests" }
accept: { type: user.assignment.requested, version: 1 }
create: TenantUserAssignment
map:
messageId: messageId
seats: seatCount
tenant: { lookup: Tenant, by: tenantId, from: tenantId }
```

`accept` gates the arrival on the envelope keys it names. `map` fills a property of `create` from an
envelope key, or — given `{ lookup, by, from }` — from the record of another entity that a **business
key** identifies: the envelope carries a name, the entity stores a reference.

> **Normative.**
> `accept` and `map` are independent, both optional, and both valid on every arrival — they describe
> the payload, not the transport. An arrival declaring neither behaves exactly as one specified before
> them.
>
> A message matching every key of `accept` is ingested. A message that does not match MUST be
> acknowledged and ignored, with a diagnostic; it MUST NOT be failed, since redelivery cannot change
> the outcome. `accept` values are scalars.
>
> Each key of `map` MUST name a field or a to-one relation of `create`; the entity's key MUST NOT be
> mapped, being assigned on creation. A value is an envelope key, or a lookup declaring exactly
> `lookup`, `by` and `from`. A lookup MUST fill a to-one relation, `lookup` MUST name that relation's
> target, and `by` MUST name a field of the target that identifies at most one record — a field
> declared unique, or the target's key — and one that a business key can travel as, namely text or an
> integer. A generator MUST refuse a `by` that does not, rather than resolve a lookup that could match
> several records.
>
> A lookup matching exactly one record fills the relation. A lookup matching none MUST reject the
> arrival, reporting the unresolved value; it MUST NOT store the record with the relation unset. A key
> declared with no value at all — in either block — MUST be reported as an authoring error rather than
> read as an empty one.
>
> A mapped record MUST be saved through the entity's ordinary write path, exactly as an unmapped one
> is: mapping changes what the record is built from, never how it is stored.

## DSL index

| Construct | What it gives you |
| --- | --- |
| [`inbound.accept` / `inbound.map`](#inbound--arrivals-from-outside) | read an arrival as an envelope: gate on its type and version, map its keys onto the record, resolve a business key to a relation |