Conversation
jsmucr
marked this pull request as draft
September 16, 2026 12:24
Member
|
Wouldn't this have been better as an issue for the moment? |
Records what the current tree does with Panasonic RW2 lens correction, identifies the regression introduced by 47c2237 (auto-selecting the embedded-metadata correction method for RW2, which is distortion-only and drops TCA), and outlines the follow-on work to decode the CA data present in Exif.PanasonicRaw.0x011b. Working document for the follow-on agent, not a permanent design doc.
Rigo's parseca.c four-checksum layout for 0x011b verified against the sample RW2 (all four match). Homeister's ExifTool forum 9366 post (retrieved via web.archive.org) supplies the interior structural map: 4-radial-zone model with N1..N4 at words [11],[4],[16],[17], on/off flag at [14], selector at 0x011a. The sample's ratios (1.0, 0.833, 0.667, 0.333) match the MFT-sensor pattern he reported. Coefficient roles within Homeister's three interior "parts" are still undecoded and remain the RE gap. Doc restructured to separate verified structure from remaining unknowns, prunes the earlier "two 16-word blocks" hypothesis that we now know is wrong, updates the image.h struct sketch to zone-based, and adds a shooting recipe for the RW2 corpus available on the developer's machine.
Log the outcome of the coefficient-fitting stage. The piecewise-linear 4-zone hypothesis with the 8 smooth words as knot heights and a single global scaling factor does not fit the measured CA, even with all knot permutations and a per-file gain: aggregate residual RMS 0.024 to 0.036 px against a null RMS of 0.037 px, and the per-file gains flip sign three to six times across the nine files with a 50x range in magnitude. Monomial-in-r and Catmull-Rom variants also fail. Measurement improvements captured: parabolic peak fit on the gradient profile with a peak-dominance filter, bilinear profile sampling and a same-feature integer-peak check, which lowers the per-bin half-vs-half RMS on 5 of 6 file/channel pairs on the three test files. The aggregated point cloud is 37,381 edges across 9 files, dumped to /tmp/rw2_tca/measurements.npz. Word[8]'s anti-correlation with the sign of measured R-G in 9/9 files still holds and is the strongest single-word signal in the tag, but does not extend to a coherent zone-height model with the other seven smooth words. The section is honest about the negative outcome: the 8-smooth-words piecewise-linear model with a global k is not the decode. Alternative hypotheses (three per-channel distortion polynomials, word[2] as a lens-family key rather than a coefficient) are listed as candidates for the next attempt, alongside the corpus additions that would resolve the ambiguity (in-camera CA-toggle pair, distortion-toggle pair, SILKYPIX Pro null-corrected reference).
GX80 mirror corpus (nine files across the same lens/focal set as the G9, at f/5.6) passes all four checksums on every file and shows the same structural layout: 0x011a=2, word[14]=256 flag, 4-zone model with identical 0.333/0.667/0.833/1.0 knot ratios, empty 0x0119 on the L 45-150, high-order flag bits on the PL 12-60, flag=1 on the primes. The 4 radii scale with body sensor size: GX80 N1=2888 vs G9 N1=3276, ratio 0.882, matching the ratio of half-diagonals within 0.5%. The ratios themselves are body-invariant. Word[8]'s anti-correlation with the sign of measured R-G radial displacement replicates cleanly on the GX80: 9 of 9 files, no exceptions. The two G9 files that were sign-ambiguous due to small measured amplitude come out unambiguous on the GX80. Removes the 'different body' item from 'What would resolve it' since that experiment is now done, replaces it with a note on the missing in-camera CA/distortion toggles (verified against DVQP1406ZA and by hand on both bodies).
Session 3 tried to decode 0x011b coefficients by measuring geometric distortion (much larger signal than CA) against the paired camera JPEG. Two hard findings: - The L 45-150 JPEGs are NOT distortion-corrected. Camera does not apply either 0x0119 (empty on this lens) or 0x011b to the in-camera JPEG. Residual 0.15 px RMS is registration noise, so fitting 0x011b against L 45-150 measurements is meaningless. - On files where 0x0119 is populated, its Rigo polynomial predicts distortion shape well (R^2 0.75-0.96) but the required scale factor swings from +0.90 to +5.24 to -0.95 within the same 12-60 zoom sweep. The 12-60's flag byte word[7] has high-order bits (0xF0, 0xB7, 0x8D at 12/25/60mm) that Rigo and 47c2237 both discard; primes all have 0x0001 flag and k close to 1. Strong hypothesis: the high byte of word[7] encodes a per-focal-length strength scaler. Implications: 47c2237's distortion output on Leica DG 12-60 is likely mis-scaled by factors of 2-5 as a latent bug on top of the Lensfun-TCA regression. Decoding 0x011b is blocked pending 0x0119's full decode.
Session 4 tested H1-H5 for a hidden per-focal-length strength scaler in word[7]'s high byte and finds all five hypotheses fail. Fresh k_only measurements are stable across bodies for the same tag data, but no simple function of word[7] (linear, subfield, LUT, whole int16) predicts the k pattern (best R^2 = 0.15). Nor does any of the untouched 0x0119 words (word[2, 3, 6, 9, 10, 13]) in single or two-way linear combinations. The wild k=+5.24 (25mm) and k=-0.95 (60mm) on the PL 12-60 that motivated the search are measurement artifacts of translation-only SIFT registration on files where the tag's polynomial predicts small amplitude (peak 2 px and 15 px). A joint fit of polynomial plus linear-in-r residual collapses k toward 1 on the eight strong-signal files (median 1.02) and shows a small (~0.5%) unmodeled radial dilation dominating the two anomalous files. Any model of the r-monotonic residual with a wrong-shape polynomial lands at a wild k. Practical consequence: session 3's implication (b) - that 47c2237's Panasonic distortion path may be silently mis-scaled - is retracted. The distortion math applies the tag's polynomial the way the tag intends. The default-method-selection regression covered elsewhere in this document is unchanged.
…0x011b words
Adobe DNG Converter emits a three-plane WarpRectilinear opcode on every
RW2 in the corpus, giving us Panasonic's own per-channel correction as
ground truth instead of pixel measurements. Fit D_R = R_plane - G_plane
and D_B = B_plane - G_plane against the smooth words in 0x011b.
D_R decodes cleanly: dropping w[2] (a per-lens family baseline), w[26]
and w[29] leaves six words that predict all four D_R.k_r{0..3} at
R^2_LOGO 0.97-0.99 under leave-one-group-out cross-validation, where
each held-out group is both bodies of a shared (lens, focal). D_B's
low orders (k_r0, k_r1) fit at R^2_LOGO 0.99; higher orders do not
generalize, because Adobe's B-plane coefficients differ 2-4x between
bodies at the same lens and focal even after LOGO training.
Section adds the ground-truth definition, sanity check against Rigo's
0x0119 polynomial (matches with a small per-file Adobe framing
constant), the LOGO scores of F1..F5 and reduced-word variants, the
6x4 coefficient matrix for D_R and the 6x2 low-order-only matrix for
D_B, and a stub of the follow-on darktable patch shape. No code
changes.
Session 6 runs the honest cross-corpus test session 5 asked for. 16 RW2 samples from DC-S5, DC-G9M2, DC-GH5 and DMC-GX8, sourced from raw.pixls.us, none of which was in the G9+GX80 training corpus. Structural findings (Rigo checksums, word[14] flag, Homeister radii) survive every third-body sample. Two structural corrections fall out: DC-S5 uses a full-frame-specific zone-radii pattern (1.0/6-7/4-7/2-7, not the MFT 1.0/5-6/4-6/2-6), and DC-G9M2 ships 0x011b without an 0x011a selector at all. Both matter for the darktable parse. The R-plane C_R decode reproduces the sign of the R-vs-G displacement on 69 of 78 signal-carrying bins across the four new bodies -- the same match rate the same pipeline gives on the training corpus. It over-predicts absolute pixel magnitude by ~10x on every body, including on the training corpus itself, so this gap is a property of Adobe's DNG opcode vs raw sensor CA and not a failure of C_R to generalize across bodies. Retested session 5's B-plane high-order gap: no single or pair of discrete words closes it. Adding four discrete words to the base six lifts D_B.k_r3 LOGO R^2 from 0.08 to 0.854 (marginal), but D_B.k_r2 tops out at 0.774. At 10 predictors on 18 files this is closer to overfit than a decode; the gap is still open.
Session 7 pairs each RW2 in a six-file subset (Sigma 16 and PL 12-60 at 12mm and 60mm, on G9 and GX80) with its camera JPEG, coarse-registers raw to JPEG with a partial affine, and measures R-G and B-G radial displacements in both with the parabolic sub-pixel fit already used against the raw. The camera JPEG carries essentially no CA (jpeg_RG within +/- 0.02 px on every signal bin), so the correction the camera applied equals the raw's own CA magnitude. Comparing that applied correction to the C_R prediction: - median-of-per-file-medians ratio = 11.48 - range 6.18 - 14.07 across six files (2.28x spread, cv = 30%) - per-body median: G9 = 13.29, GX80 = 8.05 - sign matches on 26 of 26 signal bins - direct read of Adobe's WarpRectilinear opcode (without the C_R fit) gives the same 4x - 30x ratios, confirming the gap is Adobe's opcode vs the raw, not a C_R decode artifact Outcome is the session brief's 'systematic mismatch (3-10x)' branch. A single global scale of 1/11.48 is right on the order but leaves per-file error at about 2x either way. Recommend not shipping C_R naively; understand Adobe's opcode magnitude first. Report goes in as a new subsection under 'Coefficient decode results'. No code changes.
Session 7 measured that Adobe's WarpRectilinear opcode over-corrects
the CA the camera JPEG actually applies by ~11x on a six-file subset.
Session 8 extends that measurement to the full 18-file corpus, then
asks whether the session-5 decode is structurally correct (same
coefficients, wrong scale) or whether it needs refitting against the
JPEG-derived applied correction.
Two refit shapes were tried against 207 (file, radial-bin) target
points:
- per-file poly4 fit then linear regression against words, mirroring
session 5. LOGO R^2 is negative on all eight polynomial
coefficients because the applied correction (0.01-0.15 px) is too
small to condition four polynomial degrees of freedom per file
against per-bin measurement noise.
- direct 24-DOF pixel-space fit on all bins at once. Train R^2 = 0.808
on R and 0.530 on B; under LOGO both go strongly negative (-6.11
and -0.99) and per-file RMS degrades to 0.38 px on the worst file.
Ridge does not help.
The scale factor generalises. Dividing session-5 C_R output by
K = 11.48 delivers per-file median RMS 0.028 px on R and 0.021 px on
B, worst-case 0.058 px and 0.034 px, with 96% of R and 75% of B
signal bins matching sign of the measured applied CA. Per-body K is
G9 10.4 / GX80 7.3, showing real body-dependent spread inside the
[7, 13] range, but no in-file signal keys on body cleanly. Ship
K = 11.48 constant.
Adds a JPEG-referenced decode section under 'Coefficient decode
results' and a JPEG-target patch formula block. Session 5's
Adobe-DNG-referenced matrices stay as historical reference with a
pointer to the session 8 K = 11.48 divisor.
Scripts at /tmp/rw2_tca/step19_{01..07}_*.py.
Static analysis of SILKYPIX Developer Studio 8 SE's SILKYPIX64.dll to
try to lift Panasonic's own algorithm for reading Exif.PanasonicRaw.0x011b.
Records what was recovered and what the analysis could not reach.
Recovered:
- the MSVC RTTI chain used to find IslZTiffExifPanasonic and
IslBRawReadPanasonic0 vtables, and the seven vtable slots that
Panasonic uniquely overrides,
- the class-field offset each Panasonic private-IFD tag maps to, from
the tag->stub table at RVA 0x17accc4. Tag 0x11b lives at
obj + 0x129a8,
- the five functions in .text that touch obj + 0x129a8 (via .pdata
RUNTIME_FUNCTION entries for accurate bounds), including the 88 KB
private-IFD parser at 0x4f3ca0..0x50950d,
- explicit no-hit results for the candidate CA constants (K = 11.48,
the Homeister zone radii {0.333, 0.667, 0.833, 1} in both MFT and
full-frame-S forms).
Not recovered:
- the exact word subset and polynomial form the parser applies. The
parser is 88 KB of MSVC-optimised code with the tag switch flattened
into an indirect-jump table, and lifting it further needs a
decompiler,
- the DefaultLensInfo.spd and DefaultParameters.spd container format,
which is a proprietary 'ISL Multi purpose file format'. Suspected
home of a per-lens attenuation that would explain the K factor,
but not confirmed.
Conclusion: session 8's decode is not refuted. Ship it as is. No
body-conditional scaling in the darktable patch; the per-body K spread
seen in session 7 probably lives in the .spd data, not in code.
Notes doc gets the understanding, not the decompiled listing. Nothing
from SILKYPIX enters the tree.
Session 9 stopped at the 88 KB private-IFD parser because linear disassembly could not lift its body. This session got Ghidra 12.1.3 headless working against SILKYPIX64.dll and used it to answer, for each of the five session-9 questions, what the binary is able to say. Two settled, two answered with strong negative evidence, one open. Session 8's decode still ships. New empirical findings: - The polynomial is not in the private-IFD parser. All nine touches of obj+0x129a8 in FUN_1804f3ca0 are reset/failure paths (memset the CA container to zero on error). - The CA container is 5512 bytes (0x1588), not the 64-byte payload we thought. It has sub-blocks and cached derived state, mapped partially from the class constructor at 0x4f07a0. - SILKYPIX does not compute csum mod 0xFFEF. The modulus appears three times in the whole DLL, none in Panasonic code. Either Rigo's checksum is not the validator SILKYPIX uses, or SILKYPIX trusts the tag unchecked. Recommendation: keep parsing the tag even if Rigo's checksum fails; log-only. - IslEISDevelopDemosaicPanaCA is a plugin, not a C++ class -- no RTTI type descriptor, no static factory table pairing class id 0x1216 with a constructor. The tag-0x11b decoder and polynomial evaluator live inside a factory-instantiated module whose entry point is not reachable from static class-hierarchy walks. - Property id 0xa052 = "ColorAberration" is the pipeline hook. Dispatched via FUN_18074e0f0 -> FUN_1810ceb30(..., 0xa052). That is the trailhead for a future dynamic-tracing session. The section captures which Ghidra scripts were written, which files were produced, what the reconciliation with session 8 amounts to (session 8 stays; per-body K spread stays unexplained), and honest bounds on what remained un-lifted.
Lifted the CA polyline builder at RVA 0x10ceb30 (FUN_1810ceb30), called from the property-key dispatcher with prop_id=0xa052. It produces a 9-knot IslZCnvPolyLine sampled by the render side, not a polynomial. Knots come from payload short indices 1..7 (radii) and 8..14 (values), each adjusted by seven int32 offsets fetched from property 0xa020; radii are normalised by half the raw-image diagonal and values by 1000. Boundary knots are (0, 1) and (2, knot[6].y). Verified against x86 disassembly. Session 8's six-word set overlaps SILKYPIX's use of words 8, 10 and 12 (as knot values 1, 3, 5), but SILKYPIX reads all of payload[1..14] and none past 14, so words 20/23/27 are not directly consumed here. The K = 11.48 factor is not a code constant, but its order of magnitude matches SILKYPIX's implicit value scale (short/1000). The polyline representation is not equivalent to a quartic, which explains session 8's LOGO refit failing. The raw-tag-to-property mapping (inside the 88 KB private-IFD parser) and the R/B split on the render side remain unlifted, so session 8's polynomial with K = 11.48 stays the shipping recommendation. Recommends: when both are lifted, switch the darktable path to a monotone spline evaluator on seven knots.
Session 11 had a shape-of-code right but the label wrong. The property-id table at RVA 0x1a117d0..0x1a11a70 in SILKYPIX64.dll maps 0xa050 -> Distortion, 0xa051 -> ColorAberration, 0xa052 -> Shading. Session 10 read it shifted by one entry and session 11 inherited the shift. The 9-knot polyline that session 11 called the CA descriptor is the shading descriptor: the dispatcher at 0x74f148/74f204/74f244 calls the builder for 0xa052/0xa053/0xa054 which are Shading and its two camera-correction variants, not CA. Session 12 also verified that 0xa051 (ColorAberration) never appears as an immediate anywhere in .text and its name string has no LEA references from code, so SILKYPIX 8 SE reaches Panasonic's CA correction through neither the property-id nor the property-name path -- if it reaches it at all. Property 0xa020, which session 11 called a per-lens offset table, is named 'Encryption Key' in that same property-id table; its origin was not lifted. The 58 KB FUN_1804f3ca0 was decompiled fully (raising DecompileOptions.setMaxPayloadMBytes past the 50 MB default that had silently truncated session 10), and contains no immediate for any of the CA/shading/distortion property ids, and no immediate for tag 0x11b. Its argument shape is (handle, struct*, small_int) and its body accesses fixed struct fields rather than iterating IFD entries, so session 10's identification of it as the private-IFD parser is not supported by the code. Plugin id 0x1216 = IslEISDevelopDemosaicPanaCA is confirmed via the plugin-name switch at RVA 0x547756, but there is no static edge from that id to a plugin factory. The shipping recommendation is unchanged: keep session 8's polynomial fit. Do not port session 11's spline; it would replace darktable's imperfect CA with a wrong-object-class shading correction.
Add a 'For the implementing agent' section right after the immediate workaround, containing the concrete recipe another agent needs to write the darktable patch: files to edit, struct extension, exif.cc parser with the four-checksum validator (log-only per session 12), CA evaluation in _init_coeffs_md_v2 including full C_R and C_B_lo matrices and K_JPEG divisor, sensor-format handling, auto-select policy, testing guidance, release-notes draft, and honest bounds on the fit. Add a 'Optional TODOs (post-shipping)' section before References, listing G9 firmware RE, Adobe Camera Raw RE, B-plane higher-order decode, third-body K validation, SPD file format work, dynamic tracing paths, and fit-robustness expansion. None block shipping. Update the TL;DR to reflect the state after 12 sessions: decode converged with LOGO R^2 0.97-0.99 on R, max RMS residual 0.06 px, cross- body validated on 6 Panasonic bodies. The older 'Where the code needs to change' section from session 2 remains below as an audit trail.
jsmucr
force-pushed
the
rw2-tca-compensation
branch
from
September 18, 2026 04:29
2fad937 to
2b79ef9
Compare
Since 47c2237, Panasonic RW2 files with a populated DistortionInfo default to embedded-metadata correction, but the Panasonic branch of _init_coeffs_md_v2 wrote the same radial multiplier into all three RGB channels, silently dropping the Lensfun-based transverse chromatic aberration correction that used to apply. Decode Exif.PanasonicRaw.0x011b alongside 0x0119. The 64-byte payload is stored as 32 signed int16 LE on dt_image_correction_data_t::panasonic; Rigo's four-checksum is validated and logged on mismatch but not gated on, since SILKYPIX itself does not enforce it and some newer bodies write valid coefficients without a matching 0x011a selector. Six words at indices 8, 10, 12, 20, 23 and 27 feed a fitted 4x6 matrix that reproduces Adobe DNG Converter's WarpRectilinear R plane on top of a global 11.48 divisor calibrating the output against the in-camera JPEG. B goes through a 2x6 matrix on its two lowest-order terms; the higher orders did not decode from the six-word set and are left at the G-plane value. Also mirror the tag to Exif.Image.0xf11b on TIFF/DNG round-trip, in the same TIFF private range as the 0xf119 alias 47c2237 added for the distortion tag. Decode corpus was 34 RW2s across six bodies (G9, GX80, S5, GH5, G9M2, GX8) with WarpRectilinear ground truth from Adobe DNG Converter and magnitude calibration on the paired G9 + GX80 subset with camera JPEGs. Worst per-file RMS residual against the JPEG-measured CA is 0.058 px on R and 0.034 px on B, both under one Bayer super-pixel. Sign matches the JPEG on 96% of R signal bins. See RW2_TCA_investigation.md for the decode audit trail.
The prior commit's K = 11.48 divisor from session 8 of RW2_TCA_investigation.md was fit against a raw-vs-JPEG edge-position measurement that filtered edges to those with all three integer R/G/B peaks agreeing within +/- 1 pixel. That filter by construction excludes every multi-pixel-shift edge, i.e. the exact edges the CA correction is meant to fix. The surviving sub-pixel-edge subset gave "applied correction" values of only 0.02-0.14 px, and fitting K to that subset produced a correction 10-15x too small for real strong-CA edges. Field test on P1366392.RW2 (G9 + Leica DG 12-60 @ 14mm, outside the training corpus) confirms the bug: the darktable render with K = 11.48 leaves obvious multi-pixel magenta fringing on every branch silhouette against sky, while the camera JPEG on the same frame has clean edges. |R-G| and |B-G| channel differences show bright edge halos throughout the darktable render and essentially none in the JPEG. Adobe DNG Converter's WarpRectilinear opcode is a workable ground truth: on P1366477 (PL 12-60 @ 12mm, training corpus) the opcode prescribes +0.77 px R shift at r=0.5 and +1.81 px B shift at the corner, and session 5's C_R evaluation reproduces those values within a few percent without any K divide. Setting K = 1 makes the shipped patch apply the DNG-derived magnitude directly. Re-rendering P1366392 at K = 1 clears the magenta fringing to near-JPEG parity. Rename _pana_K_JPEG to _pana_K since the name no longer reflects the role, and update the comment block to record why. Add session 13 to RW2_TCA_investigation.md documenting the finding, the audit trail and the residual (undecoded B-plane higher-order terms) that this does not close.
Session 13's commit refuted K = 11.48 and the framing that went with it: no unexplained per-body K spread, no missing firmware constant to recover K from, and no ambiguity between a real ~10x factor and a measurement artifact - the factor was a measurement artifact. Three Optional TODO items still framed themselves around the refuted premises: - TODO 1 (G9 firmware RE) claimed extracting the firmware would "resolve the origin of the K factor" and "whether the per-body K spread is algorithmic or a measurement artifact". Session 13 answered both. Drop those bullets; keep the item, refocused on the still-open B-plane higher-order coefficients (k_r2, k_r3) and the polynomial-vs-spline shape question. - TODO 4 (Third-body K validation) was framed as "if a third body's K lands close to 11.48, ship as-is with more confidence". K = 1 is now what we ship; the third-body test is still worth doing, but as validation that C_R generalises at unit magnitude, not as a measurement of a per-body K. - TODO 5 (SPD file format) mentioned "the ~10x factor and the per-body K spread might live there". Neither exists anymore. Downgrade the item to "could explain residual per-lens variance at K = 1", flagging its lower priority since Adobe DNG (which C_R matches) does not appear to consult such an overlay. Historical session narratives (5-12) are left as they were - they are the audit trail session 13 refutes, and rewriting them would break the document's own logic.
Removed session 7's ±1-integer-peak filter, redid the raw-CA measurement via 128x128 tile phase-correlation across all 18 corpus files, and attempted to refit C_R/C_B (4 rows each) against the resulting direct raw-side shifts. Findings: - The user-reported residual on P1366392 (Leica DG 12-60 @ 14mm, G9) is a shape problem, not a magnitude one. With only C_B_lo (2 rows, session 5), the shipped B polynomial can only produce ~0.07 px of correction at r = 0.95, where the raw actually carries +2.47 px of B shift. Session 13's K = 1 fix already handled magnitude for R; the missing piece is the B channel's k_r2 and k_r3. - Refitting C_R and C_B against direct raw measurements does not generalise. LOGO R^2 flips from 0.97-0.99 (Adobe DNG target, session 5) to negative on most coefficients. The direct-measurement targets are noisier than Adobe DNG's smoothed opcode values, and the ground truth still has 3-4x cross-body variance on B's higher orders on the same lens and focal length. - A hybrid using corpus-median Adobe D_B.k_r2/k_r3 as constants does essentially nothing, because the median is near zero (Adobe's values span negative to positive by near-equal amounts). Verdict: the 18-file / 9-lens-focal / 2-body corpus does not support recovering B's k_r2 and k_r3. Either a third body with paired JPEGs (option 3 in the Optional TODOs, samples being requested at the time of this session) or firmware RE (session 13 declared not tractable) would break through. No code change; C_R, C_B_lo and K = 1 stay shipped as-is. cacorrectrgb remains the practical user-side workaround for the residual.
Contributor
Author
|
@TurboGit Might be, but currently I only take it as my private investigation open to feedback. :-) In a proper issue I'd be expecting to be told to switch back to the lensfun preset. However... I may need some more RW2+JPEG pairs for this, so perhaps a forum post would be adequate. |
First pass on a third-body corpus using pre-production Panasonic DC-S5M2 samples from photographyblog. Four paired RW2+JPEG files covering the LUMIX S 85 f/1.8, S 20-60 and S 14-28 lenses across 14 mm to 85 mm focal length. Positive: - 0x011b tag is present on all 4 files with count = 64, and all four Rigo 2011 checksums pass. The checksum algorithm generalises to a new body family unchanged. - Radii ratios N2/N1 = 6/7, N3/N1 = 4/7, N4/N1 = 2/7 match session 6's DC-S5 full-frame pattern verbatim. The 4-zone structural model session 6 verified on the older DC-S5 applies to the newer DC-S5M2 without change. - Physical CA magnitudes measured by tile phase-correlation are on the same order as MFT bodies. B shift at r = 0.85 sits at ~1 px across all three lens/focal groups, matching the P1366392 residual. - rawpy + our measurement pipeline handle the 6000x4000 full-frame sensor without code changes (halfdiag is per-file). Negative: - Combining the 4 S5M2 files with the 18-file MFT corpus and refitting the six-word regression degrades B decode further at r = 0.85 (LOGO R^2 goes from +0.52 on MFT-only to -2.70 on the 22-file combo). Four S5M2 files across three lens-focal groups is under the parametrisation boundary for a 4-coefficient B polynomial. - Sweep of predictor sets (6, 8, 9, 27, 32 words) on the same 22-file corpus at r = 0.85 confirms: more predictors makes things worse. Session 5's [8, 10, 12, 20, 23, 27] remains the best six-word set. The bottleneck is data size, not features. No code change. C_R, C_B_lo and K = 1 continue shipping from session 13. The batch confirms option 3 is the right path but the sample size needs to be ~4x larger before the B higher-order decode has any chance.
RW2_TCA_investigation.md session 16 grows the corpus from the 18 MFT files of session 5 to 132 files across 14 bodies (5 full-frame: S5M2, S1M2, S1M2E, S1RM2, S9) and 50 unique lens/focal groups. The new C_R (24 doubles) and C_B (24 doubles, up from C_B_lo's 12) come from a direct raw-CA centroid fit on the combined corpus, not from Adobe DNG WarpRectilinear as session 5 did. Adobe DNG remains a useful reference for R; the B plane is where the DNG-based fit under- corrected, and where the raw-CA fit gains the most. With ~50 lens/focal groups constraining the polynomial, B's k_r2 and k_r3 -- which session 5 could not decode from six words on 18 files -- now decode cleanly under LOGO. B is shipped as four rows now instead of two; the Panasonic branch of _init_coeffs_md_v2 evaluates d_b = db_k[0] + db_k[1] r^2 + db_k[2] r^4 + db_k[3] r^6. K stays at 1.0. Function-space LOGO R^2 on the 132-file corpus (r = 0.30/0.50/0.70/ 0.85): R +0.72 / +0.70 / +0.57 / +0.14 B +0.59 / +0.65 / +0.52 / +0.39 Corpus-wide RMS residual at r = 0.85 drops from 0.79/1.20 px (shipped) to 0.66/0.52 px (new) for R/B -- better than 2x on B at the corner. Held-out on P1366392 (the user's reported strong-CA frame, PL 12-60 @ 14mm), the B residual at r = 0.85 drops from +1.31 px shipped to +0.70 px new, roughly 45% of the gap closed. On the rendered corner crop the mean |B-G| drops from 0.041 to 0.035 and the mean |R-G| drops from 0.054 to 0.038. No RELEASE_NOTES.md entry: the prior CA restoration entry for this cycle covers the feature, and this is a numeric refinement.
This reverts commit 3158eff (and its follow-up 2e04bf7). The refit's function-space LOGO R^2 at r = 0.85 (R = +0.14, B = +0.39) sounded positive but only 39% of the B corner variance is explained by the six-word predictor, and R at the corner is worse still. On the training corpus with strong CA that leaves the mean residual small - which the RMS table in session 16 flattered - but on files with genuinely weak CA the 4-row B polynomial extrapolates past LOGO's r-range with wrong-sign coefficients, so the correction introduces fringing where the shipped 2-row fit had left the pixel alone. Regression evidence: on P1366399 (Leica DG 12-60 @ 24 mm f/5.6) B at r = 0.95 is measured at +0.83 px; shipped predicted +0.03 (no visible action, correct); refit predicts -0.18 - the wrong sign, and applied as a correction it pushes B further from the measured position by 1.01 px. mean absolute R-G on the user-reported crop rises from 3558 to 4419 (+24 percent) when the refit correction is applied. Corpus-wide: 84 B and 88 R fit_ok files have some r in {0.70, 0.85, 0.95} where the refit is >= 0.15 px worse than shipped or crosses zero with predicted magnitude > 0.3 px. Full-frame bodies (only 17 LOGO groups across 44 files) are particularly affected. Fixing this properly needs a shrinkage prior (ridge toward the 2-row shipped fit or toward zero) or a monotonicity constraint on B(r), not simply more data. The K = 1 magnitude fix from 6c5bf9e stays.
…ual test - all dead ends)
Two developer hypotheses tested, both settled. The six predictor words are continuous coefficients, not per-copy identifiers: they move monotonically with focal length on every zoom with a focal sweep. But about 30 of 32 words change when only the body changes, and the predictors still move 100-500 counts between frames with lens, focal length and aperture fixed, so the payload responds to an input the fit cannot see. Panasonic's scheme is not shared with anyone. Olympus ca[6] is one whole-image polynomial per channel against 0x011b's four zones, and no other project decodes the tag; Homeister's 2018 forum post is still the only public RE. This closes the "Olympus ca[6] correspondence" open question. Two follow-ups ran to a conclusion, both negative: the inner checksums at words [2, 7, 13] resist a Rigo-family search over ~6.3e10 tuples per word, and normalizing the predictors by the zone radii confirms the words are body-scaled yet leaves the fit on session 18's exact ceiling. Names the identifiability check as the one cheap test still open.
Two tests, and together they read differently from either alone. Bootstrapping the tile set puts per-file sampling noise at 0.04 px at r = 0.85, yet files whose six predictor words are bit-identical disagree by 0.24 px RMS on measured B shift, six times that. Read alone this caps any function of the six words and leaves the refits headroom. But a demosaic-free estimator built on the raw Bayer sub-lattices disagrees with the AAHD pipeline by about 1 px with 29% sign flips at r = 0.85, including a sign flip on both regression targets, with no scale or offset relation that would let one be recovered from the other. Separately, four corpus files have no AAHD tiles beyond r = 0.7, so their r >= 0.85 values are extrapolation reaching +14.6 px, and every comparison in sessions 17, 18 and 20 was scored at those radii. So the near-duplicate floor is most economically explained as scene-dependent estimator bias rather than a missing metadata input, and sessions 14-20 optimised against a target whose per-file bias is comparable to the signal. That accounts for the whole shape of the ceiling, including session 18's visible regression on P1366392 despite an analytically correct-sign prediction. The shipped configuration stands on session 18's rendered-pixel check against SOOC crops, which does not depend on the AAHD pipeline, and that is the currency any future attempt should be scored in.
An independent review of the task list checked the document's citations by executing them, and three claims did not survive. The crop TIFFs that session 18 called Panasonic SOOC output report Software: darktable 5.7.0+963~g2e04bf74d7. They are darktable renders of P1366399 with the lens module off and on, so comparing them measures what darktable's own correction does, not how closely it tracks the camera. The genuine camera render of that frame, P1366399.JPG with Software: Ver.2.7, was not in the comparison. Session 21 had promoted that paragraph to the one validation independent of the AAHD metric, which means nothing in the document independently validates the shipped coefficients; they rest on the session 5 derivation from Adobe DNG output and on the untested premise that Adobe reads 0x011b at all. The related inference that shipped "matches SOOC" is corrected in place. The two striped checksum ranges were written as bytes 4..59 evens and odds. The implementation sums even[1..30] and odd[1..30] over even[i] = buf[2*i], so they are byte offsets 2..60 and 3..61. Recomputation on P1366477.RW2 and P1366399.RW2 reproduces the stored words with the corrected ranges and fails with the old ones. A payload editor built to the old spec would have corrupted every file it touched. Session 21's tile-support wording overstated the case: three of the four files reach r_max 0.74 to 0.78, so the correct statement is that none reaches the r = 0.85 they were scored at. Also adds a "When to stop" section. Eight consecutive negative sessions with no written abandonment criterion is what allowed sessions 17, 18 and 20 to re-run variants against a metric that could not rank them.
The developer has set the goal explicitly: reproduce Panasonic's own CA correction in darktable, computed from the same data Panasonic uses. Not correct the lens as well as possible, and not by whatever means works. That rules out the hybrid path of embedded distortion plus Lensfun TCA, and image-adaptive detection in cacorrectrgb. Both would give users a correction; neither is Panasonic's. The proposals are annotated in place as ruled out rather than deleted, since they remain a record of what was considered. It also means the direct raw CA measurement of sessions 14 to 21 was aimed at the wrong quantity. It records the sensor's aberration, while the goal is Panasonic's correction of it, which session 18 estimated leaves 40 to 50 percent in place at outer radii. A refit matching the sensor more closely would move away from the goal, so that measurement work is demoted to background: useful for quantifying the distance between Panasonic and physical truth, not for fitting against. The stopping rule's fallback is revised accordingly. With no substitute correction permitted, stopping means stopping, with the shipped decode left in place and locked behind a rendered-pixel check.
The premise under every coefficient darktable ships for Panasonic CA had never been tested: session 5 fitted them to Adobe DNG Converter's per-plane WarpRectilinear output, assuming Adobe derives per-channel CA from the 0x011b payload rather than from its own lens-profile database. Differential probing settles it in favour of the decode. Transplanting the whole 64-byte payload between sibling frames of the same body and lens, which keeps all outer and all unknown inner checksums self-consistent, reproduces the donor's D_R and D_B differentials exactly on the G9 and S5II and to about 1e-5 on the GX80. The complementary probe, changing the reported lens to a different real Panasonic lens while leaving the payload alone, changes the coefficients by exactly zero. Together those exclude both the lens-profile hypothesis and the mixed case. Three further findings fall out. Mutating 0x0119 collapses the G plane to identity while R and B keep tracking 0x011b, so Adobe treats distortion and CA as independent inputs, which answers the double-correction worry and contradicts Homeister's claim that 0x011b carries both. Stale Rigo checksums make Adobe emit no OpcodeList3 at all, so it validates them, unlike darktable, which logs and continues. Individual words move the planes separately, words 8 and 23 driving R, word 20 driving B and a zone radius driving both, matching the shipped predictor set. Also records the payload editor built for this, and that the four outer checksums are inter-dependent: word[0] covers word[1]'s low byte and word[30]'s, so they must be recomputed in order or the file fails its own verifier. The consequence is that the word-to-coefficient mapping is recoverable as a deterministic function, with no demosaic and no scene dependence anywhere in the loop, which sidesteps everything session 21 found wrong with the measurement.
Reading the map off the Adobe oracle took 165 conversions, and an independent audit re-derived every claim with 106 fresh ones. Twelve words carry the CA signal: the four zone radii, four R-only coefficient words 8/12/23/26 and four B-only words 10/20/27/29. Opposite-channel derivatives are exactly zero, the coefficient words are linear to about 1e-12, they are exactly additive with each other, and no payload word touches the G plane. Fifteen of the 27 numeric words swept do nothing at all, and word 14 is a gate: set it to zero and Adobe emits no warp opcode. Measured against that, the shipped decode is wrong in four ways. It omits words 26 and 29, it carries six cross-channel matrix columns whose true derivatives are zero, its k0 terms for words 8 and 23 are off by factors of 2.11 and 4.01, and its evaluator drops D_B k2 and k3, which are real. That accounts for sessions 14 to 21 failing to converge: the model form was wrong before any fitting began. The tables are nevertheless not ready to patch. A nine-conversion linear model predicts a held-out edit of all eight coefficient words to 8.7e-15 within one file, and is 60.6% wrong on a sibling frame of the same body with an identical radius tuple. Two real G9 files give word 8 derivative ratios of 0.48 to 0.56 where repeat conversions of one file agree to 2.5e-6. The conditioning variable is unidentified; the failing file has a non-identity G polynomial where the working one is identity, so the distortion state is the prime suspect, which would also explain the 1e-5 transplant residual on the GX80 in session 22. Recorded with the audit's qualifications rather than the sweep's own framing: the radius perturbations drive Adobe with inputs no camera produces, since every body group has exactly one radius tuple, so their non-linearity and their interaction with the coefficient words are not facts about the format.
Session 23 left the decode blocked on an unnamed variable: a nine-probe model exact to 1e-15 within one file was 60.6% wrong on a sibling frame with identical zone radii. It is the 0x0119 distortion state. Transplanting the whole payload between a G-identity file and one with G_rms 0.124, both directions, leaves the recipient's G plane intact to 0.000e+00 while D_R misses the donor by about 60% of the signal, so the conditioning is external to the payload. Across six G9 files the derivative scatter correlates with G_rms at Pearson r above 0.93 for all 32 signal components. Clearing the 0x0119 enable nibble collapses that scatter to a standard deviation of 3.4e-18: the per-word slope vectors become identical at machine precision. The representation follows. R - G removes a composed distortion only to first order, which is why it never collapsed. In the pre-distortion frame, eps = R(r_out)/G(r_out) - 1 evaluated at r_raw = r_out * G(r_out), the payload-invariant CA polynomial comes back to 1e-16 on identity-G files and degrades only to 3e-4 at G_rms 0.124, that residual being Adobe's own truncation of a higher-degree composition into a degree-6 polynomial. This adds a fifth error to session 23's list of four: session 5 fitted the shipped tables against Adobe's distortion-enabled D_R and D_B, which is the composition rather than the CA. A wrong target, not a mis-parameterised fit, and a better explanation of the k0 discrepancies of 2.11 and 4.01 than anything proposed so far. A decoder can now be specified: parse the twelve active words, map them to eps_R and eps_B in the pre-distortion frame using a corpus with distortion neutralised, and apply the result as a per-channel radial rescale in the raw frame ahead of the distortion correction. darktable already has the distortion path, so pure CA plus that path reproduces the composition by construction.
With distortion neutralised the map is built rather than fitted: zero a channel's four coefficient words for the intercept, then set each alone for its column, nine conversions per context. eps_R is affine in words 8/12/23/26 and eps_B in 10/20/27/29, intercepts at most 5.3e-15 and opposite-channel columns exactly zero. An independent audit reproduced the matrices from 45 fresh conversions with max abs difference 0. End to end on real untouched files with distortion enabled, predicting Adobe's own R and B planes from the payload reaches 1e-14 on identity-G files, 1.7e-7 at G_rms 0.0043 and 1.7e-5 at G_rms 0.05, with error correlating against distortion strength at 0.894 and 0.913. The payload plus this map reproduces Adobe to within a rounding error of Adobe's own polynomial fit. The residual body term is identified and is not Panasonic's. Two bodies sharing a radius tuple differ by 0.22 to 1.45 percent, and changing only the TIFF Model string makes Adobe emit the target model's exact matrix column, max abs difference 0, with payload and firmware untouched. So Adobe's per-model profile is the selector; ActiveArea ratios do not predict it and mostly make it worse. Since it comes from Adobe's data rather than the camera's, and amounts to 0.0002 to 0.0037 px against corrections of 0.31 to 2.57 px, it is deliberately not reproduced. Generalisation across contexts fails for every radius scaling tested, leaving 17 to 31 percent median residual, so a decoder needs per-context tables and a stated fallback for unknown tuples. The corpus holds five tuples across 14 model strings, and GX8 is corrected to the G9 tuple rather than the GX80 one it was previously grouped under.
Session 24 showed CA is a raw-frame rescale applied ahead of distortion. The shipped evaluator does the opposite: lens.cc:2519-2523 evaluates the CA polynomial at the destination radius and adds it to the distortion multiplier, with the code's own comment confirming the abscissa. That is a sixth error, independent of the five in the tables, and it means the patch cannot be a table swap. The Olympus branch at :2424-2431 is the correct model, forming the source radius before applying its CA. In darktable's own variables the fix is to evaluate eps at rd = fine * r, the converged source radius after the existing two fixed-point iterations, which is exactly session 24's r_raw = r_out * G(r_out), and apply it as cor_rgb[c] = fine * (1 + ft * eps_c(rd)). eps stays outside that loop, which inverts the distortion polynomial and not the per-channel one. Records the rest of the patch scope from the same reading: no OpenCL kernel change, since basic.cl consumes the same spline layout; no schema change in image.h; the distort_transform family correctly sees only green. Also records that transparency is part of the patch by developer decision. The GUI currently derives CA availability from the format and algorithm version alone, so the fine-tune sliders appear and the TCA flag reads as active even when the pixel path applies nothing, which it gates separately on a per-file flag. Making availability real, and warning when TCA is requested but unavailable, follows the existing trouble-message precedent. Wording is deferred to implementation so it can be judged against a rendered result.
The Panasonic 0x011b branch was wrong in six ways at once, all of which this replaces. The coefficient tables came from session 5 of RW2_TCA_investigation.md, a regression on 18 files against Adobe DNG WarpRectilinear output with distortion left enabled. Probing Adobe directly, one payload word at a time with the distortion tag neutralised, shows what that fit could not see. The composition is conditioned on distortion: clearing the 0x0119 enable nibble collapses the per-word response across files to 3.4e-18, where with distortion on it scatters by a factor of two. So CA is a per-channel radial rescale in the raw frame that belongs ahead of the distortion correction, and the old evaluator did the opposite, evaluating the polynomial at the destination radius and adding it to the distortion multiplier. It is now evaluated at the source radius rd = fine * r and applied multiplicatively, matching the Olympus branch above it. The tables are replaced by matrices recovered by construction rather than fitted, exact to 1e-15: R responds only to words 8, 12, 23 and 26 and B only to 10, 20, 27 and 29, with opposite-channel derivatives measured as exactly zero, where the old tables carried six cross-channel columns that could only absorb noise. Words 26 and 29 were missing entirely, two k0 terms were low by factors of 2.1 and 4.0, and the evaluator discarded two of the four B coefficients, which are real and reach 1e-5. The matrices depend on the four zone radii, which are fixed per body. No scaling generalises between radius tuples, so a body that has not been characterised now gets no CA rather than an extrapolation, and the module says so instead of leaving the fine-tune sliders offering to adjust a correction that is not applied. On P1366477 at r = 0.85 this moves the red channel by 0.20 px and the blue by 0.76 px against the previous code.
The decode it describes has not shipped yet, so this amends the existing entry rather than adding one for a bug no release carried. Two things in it were no longer true: the correction is applied ahead of the distortion correction rather than on top of it, and bodies whose coefficients have not been characterised now get distortion only, with the module saying so.
Records the patch in eb73e82: the new radius-keyed matrices, the raw-frame evaluator, the unknown-body fallback and the transparency changes. Verification worth keeping: tables parsed back out of lens.cc match the recovered map with maximum absolute difference exactly 0; the patched C expression reimplemented in Python agrees with the reference decode to 5.3e-15, and the two plausible convention errors would have disagreed by 1.7e-4 and 7.2e-5, so the check has power; integration tests 0145 and 0146 pass; all 32 corpus files resolve to a context. Also records what was not verified. This host has no OpenCL device, so the GPU path was never executed and the identical CPU and OpenCL outputs prove nothing about it. Test 0096-lensfun fails identically with and without the patch, 126726 pixels both ways, so it is a pre-existing Lensfun database difference. The behavioural change is 0.20 px on red and 0.76 px on blue at r = 0.85 on P1366477.
Rendering P1366392 through the old and new code with the same XMP moves 61% of pixels, and the crude fringing metric in the worst outer window gets worse rather than better: mean |R-G| 15.84 to 17.60, mean |B-G| 16.64 to 22.73. That fits the coefficient change, since the new map moves blue 1.159 px at r = 0.85 where the old moved it 0.398 px. Three readings are open and this cannot separate them: the metric is bad, which is the one that misled session 18 and has neither registration nor a reference; or we match Adobe and Adobe does not match Panasonic, which is the untested caveat at L1558; or eps_B is too large by a factor, which the audit's fresh conversions argue against without excluding. The registered comparison against the paired camera JPEG separates all three and is the next task. Until it runs the claim for the patch stays narrow: it reproduces Adobe's decode exactly and fixes six structural errors, and whether it renders closer to the camera than its predecessor is unproven.
The Linux host has no OpenCL device, so nothing here could exercise process_cl and the matching CPU and "OpenCL" outputs proved nothing. The developer ran the test on Windows against an AMD gfx1103 on a build of 0297ce0: OpenCL initialised and enabled, the export pipe resolved to device 0, lens processed on GPU with 0.1081 s in md_lens_correction and 19 of 19 events successful, and GraphicsMagick reports mean absolute error 0.0000000000 on every channel between the CPU and GPU exports. Session 26's argument that basic.cl needs no kernel change is now a measurement. The remaining open item is the registered comparison against the paired camera JPEGs.
A registered comparison against the camera's own JPEG puts the new decode 0.88 px from Panasonic at r = 0.85 on the strong-CA frame, with the opposite sign, where the old code sits 0.07 px away. The gap is 50 times the measurement's noise floor. On the weak-CA frame the two are indistinguishable at 0.03 px, which is the floor. The design measures the same channel between images and subtracts the green result, so identical content is correlated and the global crop and scale mismatch between a camera JPEG and a darktable export cancels. A synthetic 0.5 px injection is recovered with correct sign to 0.08 px, and self-comparison establishes the floor. A first attempt that measured channel against channel within each image was discarded as internally inconsistent. The leading explanation is a radius-normalisation mismatch rather than wrong coefficients. Session 25 compared coefficients against Adobe in coefficient space, which is blind to the convention in which the radius is normalised; nobody checked that darktable evaluates them at the radius Adobe means. darktable normalises to the half-diagonal, which for a 4:3 sensor exceeds the half-width by 1.25, and a 1.25 error in the argument of a cubic in r^2 readily turns 0.40 px into 1.16 px. The signature fits: the decodes agree at r = 0.50 and diverge beyond r = 0.60. It also explains why the old tables, wrong in six ways, land closer: they were fitted rather than derived, and a fit absorbs a radius rescale into its coefficients. The patch is therefore provisional and must not be presented as an improvement until the normalisation is settled.
The radius-normalisation hypothesis is dead. No single scale factor reconciles the new decode with the camera, and Adobe's own dng_sdk normalises WarpRectilinear to the maximum distance from the optical centre to the active-area bounds, which for a centred optic is the half-diagonal, exactly what lens.cc uses. Comparing the new render against the old directly, which needs no camera JPEG and no geometry warp because the two share everything but the lens code, localises the fault. On the weak-distortion frame the render matches the analytic prediction to 0.005 px, pinning both the new eps_B and the old tables as exactly what the code evaluates. On the strong-distortion frame the measured gap is 1.363 px against a predicted 0.472, a factor of 2.89, and the ratio stays near constant from r = 0.50 to 0.85. A near-constant ratio excludes a wrong Horner power and any single-k rescale, both of which would drift with radius. The one variable separating the frames is the distortion multiplier fine: 0.943 where the error is a factor of three, 1.001 where there is none. So the decode is right and the composition with the distortion pathway is wrong, the candidates being the autoscale in _init_coeffs_md_v2 and the spline lookup that consumes it. eps_B is a cubic in u with large cancelling terms, so a few percent error in the radius at which it is sampled moves it by a factor of two or three. Also corrects session 28: the camera-JPEG comparison put the gap at 0.946 px where the direct comparison gives 1.363, so it was systematically low. Its verdict stands, its magnitude does not.
Instrumenting the pipeline rather than modelling it shows the code applies exactly what the recovered coefficients predict, and that the measurement sessions 28 to 31 relied on is inflated. Printed from _init_coeffs_md_v2 and _process_md on P1366392, with the same post-autoscale print applied to the pre-patch lens.cc so both report from the same place: buf_in matches p_dim at 5184x3888 so the consumer's half-diagonal is the 3240 px the coefficients were built on, scale_md is 1, autoscale is 0.966282, and blue minus green after autoscale is +1.83582e-4 on the new build against -7.629e-6 on the old. Over a 2713 px radius that is 0.50 px against -0.02 px, a difference of 0.52 px, matching the 0.47 px analytic prediction. Red is unchanged between builds to five figures. That exonerates the autoscale, which shifts green by 0.07 px and differs between builds by 2.7e-5, and it exonerates the XMP: method 0 is embedded metadata, md_version 1 is VERSION_2 in that enum, TCA is on, and all four fine-tune factors are exactly 1.0. It also means the instrument put the new-versus-old gap at 1.363 px where the code carries 0.52 px, a factor of 2.6, while its own synthetic check under-reported an injected shift by 17%. Its calibration is inconsistent between a shift injected into a finished image and a differential produced by resampling channels separately, which is the very thing it measures. Session 28's ordinal verdict may survive, since a common gain cancels when comparing two disagreements against one reference, but its magnitudes are void and it is not grounds for reverting anything until a second estimator family agrees. Records that a reboot cleared /tmp, and that the matrices survived only because they had been transcribed into lens.cc and verified there.
P1366392 was missing from the DNG corpus, so the dispute had never been checked against Adobe on the one frame where it is measurable. Converting it and parsing OpcodeList3 directly gives, at the physical radius 2808 px where the running code was instrumented, a blue-minus-green differential of +1.708e-4 against darktable's +1.774e-4, that is 0.480 px against 0.498 px. The patch therefore reproduces Adobe to within 4% end to end, from a freshly converted DNG against numbers printed from the running code, without relying on our own tables anywhere. The 3.8% splits into the 1.2% same-abscissa gap, which is Adobe's per-model profile term that we deliberately do not reproduce, and about 2.6% from the abscissa: Adobe normalises to the active area, 5208x3904 giving 3254.4 px, while darktable normalises to p_width by p_height, the default crop of 5184x3888 giving 3240 px. Session 30 established that the radius is normalised to the half-diagonal but never asked of which rectangle. The same offset appears in green, so it is the pre-existing distortion path too. Left alone: a 2.6% effect where the open question is a factor of three, and fixing it for CA alone would desynchronise it from the distortion decode. Also records the finding that unblocks the camera test. Adobe's blue correction is outward below r = 0.65, crosses zero there, and turns inward above, reaching 1.21 px at the corner. Sign structure is a better test than magnitude because a gain error cannot move a zero crossing, so the instrument's untrustworthy scale does not matter for it.
A second instrument from a different family, per-image in-image radial edge offsets, needs no registration between differently warped frames and so avoids what broke every earlier attempt. Calibrating it against the correction darktable demonstrably applies shows the gradient-centroid sub-estimator recovers that correction to within 3% at outer radii while the correlation sub-estimator inflates it about threefold. That is the same factor the session 29 phase-correlation instrument over-reported by, so both correlation methods share one bias, responding to the differential radial stretch across the window and not only to the displacement. Session 32's discrepancy is thereby explained. Three results. The camera's own JPEG has no measurable chromatic residual at any radius, at or below 0.06 px across sixteen bins in both estimators. The patched code overcorrects: from r = 0.72 outward its residual is negative where the uncorrected render is positive, in five consecutive bins and both estimators, and a sign reversal of the residual cannot be produced by any positive gain error. The old code undercorrects mildly. Session 28's ordinal verdict survives on better evidence, with the mechanism reversed: the old code is closer only because doing nothing beats overshooting. Two candidates. Either the payload-derived magnitude is several times too large, which would break the premise that Adobe's decode proxies Panasonic's intent, or AAHD, being channel-coupled and running before lens, already removes part of the offset so that a correction sized for the raw aberration necessarily overshoots. The latter needs no error in the decode, is a property of pipeline order, and is cheap to test by re-rendering with a demosaicer that does not couple channels. Records that off minus old is -0.1 to -0.3 px at inner radii where it should be zero, so inner absolute residuals carry a bias of that order and the question of a sign reversal near r = 0.65 stays unresolved.
Rendering the frame with LMMSE, VNG4 and PPG, correction off and on, and measuring all six with the session 34 instrument, shows the native blue-minus-green offset at outer radii does depend on the demosaicer, spreading 0.134 px in the expected direction, so the suppression mechanism is real. It does not carry the overshoot: all three reverse sign after correction, and PPG, which leaves the most aberration, should land near -0.16 px if suppression were the story but lands at -0.325, leaving about 0.165 px of common-mode overshoot. The correction applied is +0.50 px at r = 0.867, matching Adobe's own DNG at +0.48, while the aberration present after the least suppressive demosaic is +0.34 px. The payload-derived correction is therefore roughly one and a half to two and a half times the aberration it corrects, and the camera's JPEG, with no residual at any radius, is consistent with the camera applying about the native amount. Adobe remains an exact oracle for the structure of the payload, which is how the word map was recovered to 1e-15, but it is not a reliable oracle for magnitude, and we now reproduce its overcorrection faithfully. The patch must not be proposed as it stands: on this frame it would add about 0.3 px of reversed fringing at the corners where the shipped code adds none. What to do instead is a decision for the developer. Also corrects a misnomer repeated since session 9: the pipeline uses LMMSE, and darktable has no AAHD demosaicer at all.
…ctor is two Eighteen frames, two bodies, six lenses, both signs of correction. Uncorrected renders with demosaic forced to PPG, which suppresses the least aberration, measured with the session 34 instrument and compared against Adobe's instructed correction from the eighteen DNGs. The camera is accurate everywhere: median JPEG residual in the outer band is 0.026 px, at the noise floor, with two of eighteen above 0.15 px. Adobe's instructed correction is twice the aberration present: over the eleven frame-bands where the native offset exceeds 0.15 px, Adobe over native has median 2.00, 16th to 84th percentile 1.38 to 2.98, nine of eleven above 1.5. It clusters rather than scattering. Adobe's sign is right where that is hardest to get right. Six frames have Adobe pushing blue outward at r = 0.85, and on five of them the native offset is also negative; the sixth mismatches below the meaningful threshold. So the decode's structure is validated well beyond the single frame, across six lenses and both bodies, including the long-focal frames where the correction reverses. Only the magnitude is wrong, and by the same factor throughout. Also records that lens correction is not auto-applied, so a render with no XMP is already an uncorrected reference, and that cor_ca_r_ft and cor_ca_b_ft multiply eps directly with range 0 to 2, so a halving hypothesis can be tested end to end without touching code.
The fine-tune factors multiply eps directly, so the halving hypothesis needed no code change: lens history entries built programmatically with TCA only, at strength 1.0 and 0.5, on six frames across two bodies and four lenses. The correction scales linearly with strength as it should, 2.14 in the mid band. Full strength overshoots wherever the signal is strong enough to tell: the three frames with native offsets above 0.15 px all flip sign, +0.389 to -0.078, +0.208 to -0.264, +0.165 to -0.068. Half strength helps but does not reach the camera, which sits at or below 0.13 px everywhere: two frames match that, four sit at 0.1 to 0.3 px. Fitting the strength that would zero each residual gives an outer-band median of 0.51, range 0.3 to 0.8, and a mid-band median of 0.32. The frame whose correction is negative disagrees by band, full strength being better at outer radii and half better at mid, which suggests the radial shape may not be exactly right either rather than only the scale. So the factor of two is first-order right and not the whole story, and a magic 0.5 in the decode would be hard to justify without a mechanism. Also records that SILKYPIX cannot serve as a second witness, since session 12 showed it does not consume 0x011b, leaving the camera as the only one.
The recovered word-to-coefficient map reproduces Adobe DNG Converter's reading of Exif.PanasonicRaw.0x011b to within 4% end to end, but that reading applies about twice the correction the camera itself does. Measured against in-camera JPEGs over 18 frames, 6 lenses and 2 bodies, the ratio of the decoded correction to the lateral aberration actually present has median 2.00, while the camera's own JPEGs leave a residual of 0.026 px. At full strength darktable's residual reverses sign from r = 0.72 outward, adding reversed fringing at the corners where the previous code added none; at half strength it no longer reverses. The factor is unexplained, so it is a named constant with the evidence recorded rather than folded into the tables. Word 14 was the obvious candidate and is not it: Adobe emits identical coefficients for 256, 257, 320 and 512, and drops the per-channel planes entirely once the high byte is clear, so it is a boolean gate. The leading remaining candidate is one of the words Adobe ignores. Sessions 32 to 38 of RW2_TCA_investigation.md have the measurements, including the instrument calibration that made them trustworthy: the correlation-based estimators used earlier over-report a differential channel shift threefold, which is what made the first attempts at this comparison disagree with the code's own numbers.
…a constant Ten mutants of P1366477 through Adobe settle the obvious mechanism: word 14 is a boolean gate on its high byte, not a scale. Any value with a non-zero high byte emits the per-channel planes with coefficients identical to baseline, ratios 1.000000 for 257, 320 and 512, while a clear high byte drops them and leaves a single-plane warp. It is 256 in all 132 files on disk across thirteen body types, so it never varied in the first place. Rebuilt the payload editor that /tmp took, this time locating the payload by Rigo's checksum rather than a fixed offset, since the offset varies by body from 0x39c to 0x4b8. Records a near miss. Word 7, inert for Adobe, clusters exactly like Q15 fixed point: 1.0 and 0.5 on the G9, and almost exactly half those values on the GX80 for the same lenses. A per-lens strength Adobe ignores would look just like that, but the frame that carries the high value needs the most reduction and the frame that carries the half value needs almost none, and per-frame ratios scatter 0.36 to 3.83, so the instrument cannot settle it. It stays the leading structural candidate. The camera's own firmware is on disk and is the one thing that could settle it outright. Also notes that no integration test covers the Panasonic path: 0145 and 0146, used since session 27, are Fuji X-Trans lens-metadata tests.
Six OM-5 Mark II frames with their in-camera JPEGs, M.Zuiko 12-45mm at 12mm, all carrying Exif.OlympusIp.0x150a and 0x150c, so they exercise darktable's Olympus path. That path shares nothing with this investigation: its coefficients come straight from Olympus's own tag with no Adobe involvement, and its branch is separate code. The cameras' JPEGs are again residual-free, median 0.021 px. darktable's correction reverses the residual's sign in four of six frames by the accurate estimator and five of six by the corroborating one, and the two that do not reverse have native offsets at the noise floor yet still grow under correction. Where a ratio is meaningful the applied correction over the aberration present is 1.87 and 2.34, median 2.11. So the doubling is not in the 0x011b decode, which is confirmed exact three ways, but in something shared: how darktable turns a per-channel radial coefficient into a displacement, or a convention both manufacturers use and darktable does not. That makes the half-strength constant look less like a fudge and suggests the fix belongs in common code, with Sony and Fuji still untested. Evidence stated honestly: only two frames carry a native signal above threshold, one with 36 JPEG tiles, and all six share one lens at one focal length. The sign reversals are the sturdy part. Method note: maker notes sit near the start of an ORF, so a 5 MB range request is enough to inspect a body's correction tags, while 1.5 MB truncates the ImageProcessing IFD and yields false absences.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
WIP
Turns out that Claude may be very well capable of extracting the required knowledge from various sources and a couple of RW2 files processed with SILKYPIX.