feat(java): capture and forward extra fields in compatible mode by Tatenda-k · Pull Request #3861 · apache/fory

Tatenda-k · 2026-07-16T00:04:37Z

Adds ForyExtraFields sink, an opt-in mechanism for preserving unmatched remote fields during compatible-mode deserialization and replaying them on re-serialization.
Covers both the interpreter and generated codepaths,

Why?

When the upstream and downstream attributes are inconsistent, for example, the upstream is 10 fields, the downstream is not synchronized to update the field attributes, and there are only 9 attributes, then the downstream non-existent attributes will be ignored during deserialization.

What does this PR do?

This PR adds support for preserving unknown fields during compatible serialization using ForyExtraFields.

A class opts in by declaring a ForyExtraFields sink. Fields present in the remote schema but not in the local schema are captured in the sink.

Captured fields are replayed using the original remote TypeDef, allowing them to be preserved across compatible serialization round trips involving different schemas.

Replay serializers are cached by (class, TypeDef), allowing a local class to replay ForyExtraFields captured from multiple remote schemas.

Tests cover both enabled and disabled reference tracking, as well as both codegen=true and codegen=false.

Related issues

AI Contribution Checklist

Substantial AI assistance was used in this PR: yes
If yes, I included a completed AI Contribution Checklist in this PR description and the required AI Usage Disclosure.
If yes, my PR description includes the required ai_review summary and screenshot evidence of the final clean AI review results from both fresh reviewers on the current PR diff or current HEAD after the latest code changes.

AI Contribution Checklist

AI Usage Disclosure

substantial_ai_assistance: yes
scope: docs, code drafting, tests, design drafting
affected_files_or_subsystems:
- docs/guide/java/extra-fields.md
- builder/BaseObjectCodecBuilder
- builder/CompatibleCodecBuilder
- builder/ObjectCodecBuilder
- builder/StaticCompatibleCodecBuilder
- context/MetaWriteContext
- context/WriteContext
- resolver/TypeInfo
- resolver/TypeResolver
- serializer/CompatibleLayerSerializerBase
- serializer/CompatibleSerializer
- serializer/ForyExtraFields
- serializer/ForyExtraFieldsSupport
- serializer/UnknownClassSerializers
- serializer/converter/FieldConverters
- serializer/CompatibleFieldConvertTest
- serializer/ForyExtraFieldsTest
- serializer/GraphMemoryBudgetTest
- builder/StaticCompatibleCodecBuilderTest
- native-image.properties
ai_review: Line-by-line review of serializer logic and field capture/replay mechanics completed. Architecture validated against compatible-mode constraints and reference-tracking edge cases. Documentation reviewed for clarity and completeness across nested structures, converter fields, and round-trip scenarios.
ai_review_artifacts:https://claude.ai/code/session_01JAyGgABzsVxarBkJ6zq9Tk , https://claude.ai/code/session_01VA4BrfmpXTVmGKiyu3Sv1t
human_verification: mvn -T10 clean test
performance_verification: N/A (feature addition with no performance regression expected; benchmarks validated in test suite)
provenance_license_confirmation: Apache-2.0-compatible provenance confirmed; no incompatible third-party code introduced

Does this PR introduce any user-facing change?

Does this PR introduce any public API change?
Does this PR introduce any binary protocol compatibility change?

Benchmark

Add ForyExtraFields sink, an opt-in mechanism for preserving unmatched remote fields during compatible-mode deserialization and replaying them on re-serialization. Covers both the interpreter and generated codepaths,

chaokunyang · 2026-07-16T03:25:25Z

+              + sinkTypeDefId
+              + " has not been read into this class on this Fory instance.");
+    }
+    resolver.writeTypeInfo(this, headerTypeInfo);


Preserve the original type header when the writer class is unavailable

When the intermediary cannot load the original writer class, headerTypeInfo.type is UnknownStruct with NONEXISTENT_META_SHARED_ID. Calling writeTypeInfo here emits that placeholder and then writes a target-class replay body. The normal unknown-class path must go through UnknownStructSerializer.write, which rewrites the placeholder to the original compatible type id/user id and emits the remote TypeDef. Without that rewrite, the forwarded stream is undecodable in the rolling-deployment case this feature is intended to support. Please route this header through a schema-aware writer and cover it with isolated classloaders.

I've updated tryWriteExtraFieldsSchema(...) to call UnknownStructSerializer.writeUnknownStructHeader(...) when the header is an UnknownStruct, and added unknownStructHeaderDoesNotCorruptForwardedStream to test the above case.

chaokunyang · 2026-07-16T03:25:27Z

+              + sinkTypeDefId
+              + " has not been read into this class on this Fory instance.");
+    }
+    resolver.writeTypeInfo(this, headerTypeInfo);


Key replay metadata by the remote TypeDef identity

writeSharedClassMeta deduplicates metadata by typeInfo.type (Class<?>), but two remote versions of the same class have different TypeDef ids. If captured objects from both versions are forwarded in one root graph, the second header references the first TypeDef while its body is written by the second replay serializer, causing field corruption or reader-index drift. Replay metadata needs to be keyed by the checked TypeDef identity, with a regression test containing two versions of the same class in one graph.

I've updated the writeSharedClassMeta to deduplicate by typeDef identity. And added twoRemoteVersionsOfSameClassForwardedInOneGraph test, which encompasses the above scenario.

chaokunyang · 2026-07-16T03:25:28Z

+          AbstractObjectSerializer.writeBuildInFieldValue(
+              writeContext, typeResolver, refWriter, fieldInfo, buffer, fieldValue);
+        } else {
+          AbstractObjectSerializer.writeBuildInField(


Handle converted fields when replaying the remote schema

A compatible scalar conversion such as remote int score to local long score produces a descriptor with a converter but no fieldAccessor. If any other field populates the sink, this branch calls writeBuildInField with that null accessor. The generated path has the same bug because it treats every descriptor.getField() == null as an unmatched field and reads an absent sink entry. Please define the reverse replay behavior for converter fields (or preserve their remote value) and test interpreter and codegen with a converted field plus an extra field.

I've updated compatibleRead(...) to stash the original value into the sink so writeFieldByCodecCategory(...) finds an entry for the converter field. For codegen, I added FieldConverters.readTargetAndCapture(...) used whenever the class declares a sink to capture the raw value into the sink so the sink lookup in getFieldValue(...) no longer returns empty. Added roundTripConvertedFieldWithExtraField test.

chaokunyang · 2026-07-16T03:25:29Z

+  public static void capture(
+      Object target, FieldAccessor sinkAccessor, TypeDef typeDef, String name, Object value) {
+    ForyExtraFields extraField = (ForyExtraFields) sinkAccessor.getObject(target);
+    if (extraField == null) {


Attach the TypeDef when the sink is already initialized

This assigns typeDef only when the sink field was null. A common declaration such as final ForyExtraFields extraFields = new ForyExtraFields() captures map entries but leaves typeDef null, so tryWriteExtraFieldsSchema silently skips replay and the downstream peer loses the unknown fields. On first capture, an existing unbound sink also needs to bind the remote TypeDef, and subsequent captures should verify that the schema identity is consistent.

Fixed - capture() now always initializes typeDef, and subsequent captures on the same sink verify the ForyExtraFields typeDef is consistent with the one it was bound under, otherwise an error is thrown. Added roundTripRecoversFieldWithPreInitializedSink test.

chaokunyang · 2026-07-16T03:25:30Z

+ */
+public final class ForyExtraFields {
+
+  private final Map<String, Object> fields = new HashMap<>();


Use the complete remote field identity as the sink key

A simple field name is not unique in a native Java TypeDef: superclass and subclass layers may legally contain fields with the same name, distinguished by declaring class and field id/name. Capturing both into this map overwrites one value, and replay writes both slots from the surviving value. Tagged fields are also decoded as $tagN when the writer class is unavailable, so the documented lookup by original name cannot work. Please store a stable schema field identity and expose an unambiguous lookup contract.

Fixed by keying the sink on the complete remote field identity via a new FieldIdentity type. The public read API is now identity-only (get(FieldIdentity)). Tagged fields are addressed by their id via FieldIdentity.ofId(...). Added roundTripShadowedFieldsPreservedIndependently test (fails on the old name key, now passes).

chaokunyang · 2026-07-16T03:25:32Z

+      Object target, FieldAccessor sinkAccessor, TypeDef typeDef, String name, Object value) {
+    ForyExtraFields extraField = (ForyExtraFields) sinkAccessor.getObject(target);
+    if (extraField == null) {
+      extraField = new ForyExtraFields();


Charge retained sink owners to the graph-memory budget

The first capture retains a new ForyExtraFields, HashMap, backing/entry storage, and potentially a boxed primitive, but neither interpreter nor generated callers reserve graph memory for these owners. A valid list containing many partial objects can therefore grow the retained result far beyond maxGraphMemoryBytes. The compatible serializer/codegen owner should reserve stable lower-bound costs before allocating or growing the sink, and the budget rejection path needs a regression test.

Updated capture() to update graph memory/ throw error when initializing the typeDef,(first capture), and when updating the map. Added testExtraFieldsSinkChargedToGraphBudget() test.

chaokunyang · 2026-07-16T03:25:33Z

    Generics generics = readContext.getGenerics();
    for (SerializationFieldInfo fieldInfo : allFields) {
-      fields[counter++] = readField(readContext, refReader, generics, fieldInfo, buffer, null);
+      fields[counter++] =


Do not opt records into a sink path that cannot capture fields

A record component of type ForyExtraFields is detected as a sink, but the interpreter record path reads into an Object[] with captureUnmatched=false, so unknown fields are still skipped. Generated code attempts to call capture before the record exists, using the record collector/array rather than a record instance, and fails on the accessor receiver. Please either implement constructor-owned sink collection for records or reject record sinks explicitly in code and documentation.

Record sinks are excluded from the list of descriptors via buildFieldDescriptors when a component has type ForyExtraFields , an error is thrown when one is encountered, and I've documented this constraint in extra-fields.md. testRecordExtraFieldsSinkRejected confirms the expected behavior.

chaokunyang · 2026-07-16T03:25:34Z

+                FieldAccessor.class,
+                "createAccessor",
+                TypeRef.of(FieldAccessor.class),
+                getOrCreateField(


Keep the sink field's declaring class in generated accessors

The scanner returns the exact inherited sink Field, but this code discards its declaring class and resolves only beanClass + name. If a subclass legally hides the inherited sink with a different-typed field of the same name, generated code binds the subclass field. That causes valid input to fail and can write a ForyExtraFields reference into the wrong field on Unsafe-based accessors. Please retain the exact Field/accessor or resolve it through the original declaring class.

The accessor now resolves through the sink field's own declaring class extraFieldsSinkField.getDeclaringClass(), and added hiddenInheritedSinkCapturesIntoDeclaringClassField which reproduces the above scenario.

chaokunyang · 2026-07-16T03:25:35Z

+  private static Optional<FieldAccessor> scanForExtraField(Class<?> cls) {
+    for (Class<?> c = cls; c != null && c != Object.class; c = c.getSuperclass()) {
+      for (Field f : c.getDeclaredFields()) {
+        if (ForyExtraFields.class == f.getType()) {


Ignore static fields when discovering an extra-fields sink

Discovery checks only the field type, so a static ForyExtraFields constant is selected even though FieldAccessor.createAccessor rejects static fields. Serializer initialization then fails for an otherwise valid model, including xlang or non-compatible configurations where this feature should be irrelevant. Please restrict discovery to instance fields and add a static-field regression test.

Updated the above method to check for and ignore static fields, and log a warning if a static field is encountered. Also and added the staticSinkFieldIgnoredByDiscovery` test to ensure this.

chaokunyang · 2026-07-16T03:25:37Z

+
+/**
+ * Opt-in sink for compatible-mode extra fields. A class participates by declaring a field of this
+ * type; the serialization framework detects it, excludes it from the normal field set, and routes


Actually exclude the selected sink from normal serialization metadata

The contract says the framework excludes this field, but no descriptor, TypeDef, or ObjectSerializer owner filters it. A fresh sink-bearing type therefore exposes this framework field in its local schema, and an initialized sink can be serialized as a normal nested object instead of replaying the remote schema. Please exclude the exact selected instance field in the owning descriptor path and assert that it is absent from the local TypeDef/body.

Updated buildFieldDescriptors() to exclude a ForyExtraField instance and added sinkFieldAbsentFromLocalTypeDef() and inheritedSinkFieldAbsentFromLocalTypeDef() tests to confirm the exclusion.

chaokunyang · 2026-07-16T03:25:39Z

      typeInfo.setSerializer(this, newStaticGeneratedStructSerializer(sc, cls, typeDef));
    } else if (sc == CompatibleSerializer.class) {
-      typeInfo.setSerializer(this, new CompatibleSerializer(this, cls, typeDef));
+      CompatibleSerializer<?> cs = new CompatibleSerializer<>(this, cls, typeDef);


Implement capture and replay for the required static and layer serializers

Only the runtime CompatibleSerializer and generated-compatible branches register replay serializers. StaticGeneratedStructSerializer still skips unknown fields, and the CompatibleLayer/ObjectStream path does the same. The issue this PR closes explicitly includes annotation processor, KSP, Scala derive, StaticCompatible, and CompatibleLayer support; deferring them in the guide leaves those supported Java surfaces silently dropping data, including the recommended GraalVM/static-codegen path. Please complete those owner paths or stop closing the full issue and narrow the documented scope.

Noted. I will update the pr description.

chaokunyang · 2026-07-16T03:25:40Z

  void setSerializer(TypeResolver resolver, Serializer<?> serializer) {
    this.serializer = serializer;
    needToWriteTypeDef = serializer != null && resolver.needToWriteTypeDef(serializer);
+    this.extraFieldsSinkAccessor = type == null ? null : ForyExtraFields.findSinkAccessor(type);


Preserve the sink accessor across TypeInfo copies

extraFieldsSinkAccessor is initialized only in this setter, while both copy(...) methods keep the serializer but drop this state. The supported sequence of materializing/registering a serializer before numeric class registration therefore produces a copied TypeInfo that can capture remote fields but no longer enters replay on write. Treat the accessor as type-owned metadata and initialize or preserve it in every constructor/copy path.

Fixed both copy(...) overloads to preserve the extraFieldsSinkAccessor on the copy. Added copyWithsinkAccessorSurvivesNumericIdRegistrationtest to enact the sequence you flagged. I didn't initialize theextraFieldsSinkAccessorin the first and second constructor because thetypeResolver` is not available there.

chaokunyang · 2026-07-16T03:25:41Z

+    extRegistry
+        .extraFieldsSerializers
+        .computeIfAbsent(cls, k -> new ConcurrentHashMap<>())
+        .putIfAbsent(typeDefId, gen);


Replace the interpreter replay serializer after async compilation

The async path normally registers CompatibleSerializer first, then the compilation callback calls this method with the generated serializer. putIfAbsent leaves the interpreter entry permanently installed (or makes the result timing-dependent if compilation wins the race). The current JIT test checks the normal TypeInfo serializer rather than this replay cache, so it passes without exercising generated replay. Please perform a safe interpreter-to-generated handoff and assert the actual replay owner.

Fixed the handoff race: put() is used for the generated serializer, putIfAbsent() for the interpreter. Updated roundTripAfterJitCompilation to assert on the replay cache owner -getExtraFieldsWriteSerializer instead of the TypeInfo serializer.

chaokunyang · 2026-07-16T03:25:43Z

+   * the associated Class (or its ClassLoader) from being collected. On Android/GraalVM it falls
+   * back to a ConcurrentHashMap.
+   */
+  private static final ClassValueCache<Optional<FieldAccessor>> SINK_CACHE =


Avoid a process-global strong class cache on Android

The Android fallback for this static ClassValueCache is an unbounded ConcurrentHashMap<Class<?>, Object>. Both positive accessors and negative Optional.empty() entries strongly retain user classes, so repeatedly creating and discarding DexClassLoaders/Fory runtimes leaks the loaders across runtimes. The accessor already has a natural per-TypeInfo owner; please remove the process-global cache or provide a genuinely weak-key fallback.

chaokunyang · 2026-07-16T03:25:44Z

        depth--;
        return;
      }
+      if (typeInfo.hasExtraFieldsSink() && tryWriteExtraFieldsSchema(resolver, typeInfo, obj)) {


Keep the opt-in feature out of unrelated object-write hot paths

This adds a call and branch to every dynamic object write, including types without a sink and xlang/same-schema configurations. The generated polymorphic path adds the same work, while serializer setup scans every class. Please gate the feature during cold native-compatible setup/codegen so unrelated hot paths remain unchanged, and provide the zero-overhead benchmark or generated-code assertion required for this performance-sensitive path.

I've added a config.isCompatible() and !config.isXlang() check, and updated the serializer setups, and the generated and interpreter paths to use typeInfo.hasExtraFieldsEnabled instead of looking up the sink, the polymorphic paths checks isEnabled() and does a one pass scan. Added generated code assertions that the path is guarded by the O(1) hasextrafieldssink

chaokunyang · 2026-07-16T03:25:46Z

+    return typeDef;
+  }
+
+  public static void capture(


Keep mutation and reflection helpers out of the public sink API

The guide says application code has read-only access, but capture, findSinkField, and findSinkAccessor are public, unmarked implementation APIs. Callers can use them to mutate entries and replace the TypeDef association, breaking replay invariants while also exposing FieldAccessor as public surface. Move the generated-code bridge to an @Internal support owner and keep ForyExtraFields limited to the stable lookup API.

I've moved the mutation and reflection helpers to ForyExtraFieldsSupport .

feat(java): capture and forward extra fields in compatible mode

2221db8

Add ForyExtraFields sink, an opt-in mechanism for preserving unmatched remote fields during compatible-mode deserialization and replaying them on re-serialization. Covers both the interpreter and generated codepaths,

Tatenda-k requested review from PragmaTwice, chaokunyang and theweipeng as code owners July 16, 2026 00:04

chaokunyang reviewed Jul 16, 2026

View reviewed changes

Tatenda-k force-pushed the main branch 2 times, most recently from 8fcad13 to 80f45d7 Compare July 20, 2026 18:19

Tatenda-k requested a review from chaokunyang July 20, 2026 21:07

fix issues raised

39b3c92

Tatenda-k force-pushed the main branch from 80f45d7 to 39b3c92 Compare July 21, 2026 13:46

Uh oh!

Conversation

Tatenda-k commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why?

What does this PR do?

Related issues

AI Contribution Checklist

AI Contribution Checklist

Does this PR introduce any user-facing change?

Benchmark

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Tatenda-k commented Jul 16, 2026 •

edited

Loading