Optimization: special null values

[greenrobot]

In addition to existing "optional" support for C++, there should also be a variant to represent "null" values in FlatBuffers via special values. Why? The current C++11 solution with unique_ptr is heap based and thus comes with an extra indirection with potential cache misses (not to mention additional allocs and deallocs). C++17's optional is much better but still comes with space overhead (one byte plus padding; resulting in 2x space requirements in some cases).

Special values to indicate a null in FlatBuffers could be configurable in the future, but to start with we can only consider the "obvious" ones: empty string, zero for integer types, and NAN for floating points. This requires a new annotation or an addition to the existing one. E.g. /// objectbox:optional="special-value"

Btw, https://cpp.objectbox.io/entity-annotations does not mention optional values at all yet.

[vaind]

isn't that basically FlatBuffer's default value? In standard flatbuffer generated code, such a value is not stored (i.e. it is stored as NULL)

[greenrobot]

isn't that basically FlatBuffer's default value?

Yes, as in "a specialized variant" of that as the "defaults" are fixed.

PS.: to clarify, the existing behavior should stay as is. Storing nulls from zeros is something one should be very aware of (too troublesome to make it the default behavior).

[vaind]

If I get this right, the desired behavior would be:

1. When reading from the database where the value was stored as null then the scalar field values in c++ would get a configured "default" value, e.g. 0 for integers, NaN for floating point numbers, empty std::string for strings.
2. When writing to the database, the value of a field is compared to the configured "default" value and stored as null if it matches.

If that's the case it would behave like flatbuffers normally do, i.e. don't write default values to the buffer. Since we have changed this behaviour to write all the time, we can switch back for individual fields using the /// objectbox:optional annotation (no need for actual value here for the annotation) and use the default value specified in the schema (or a zero equivalent if none is specified in the .fbs), which is already supported by the flatbuffers schema parser - https://github.com/google/flatbuffers/blob/4b9123baffc7c83d5d39f311d72d8cfcdf80e7e1/tests/monster_test.fbs#L78

[greenrobot]

Hmm, this is not exactly what I had in mind. I had started with some code locally some days back; maybe we can sync on that in a vcall next week?