Reverse-Engineering Gen7+ Pen Data Format

[qzed]

This is a technical discussion for the gen7+ pen data format. Please don't post any issues/problems in this thread and keep this a discussion about technical aspects/the protocol.

Some information from other threads:

• Paper explaining Active / Projected Capacitive Stylus technology: https://ieeexplore.ieee.org/abstract/document/8350275. This seems to be the technology used in IPTS.
• Excerpts from the aforementioned paper: What kind of help I can provide to improve IPTS for Surface Pro 7? linux-surface#553 (comment), What kind of help I can provide to improve IPTS for Surface Pro 7? linux-surface#553 (comment).
• Report ID names of protocol: IPTS on Surface Gen7 #4 (comment)
• HID report descriptor: IPTS on Surface Gen7 #4 (comment)

More info about the protocol can also be found in #4.

[haymanpf]

I'll just contribute the little detective work I've been able to do in my spare time playing with my SP7 and Renaisser pen.

I've found three HID report types that consistently show up when the pen is in use, pressed to the screen or just hovering, doesn't matter: 0x0D, 0x1A, and 0x1C. (There are also two others that show up with or without the pen, 0x07 and 0x08, which I think are some sort of noise? They have an ipts_data size of just 64 bytes).

The first and third are very similar, they show up almost the same number of times and have similar ipts_data sizes, 1500 and 2000, respectively. The second ID report shows up about half the time but has a lot more data, it has an ipts_data size of 7488. As an example of frequency, one data set I generated had 330 instances of 0D, 329 instances of 1C, and only 145 instances of 1A. They appear to show up in the order 1C, 0D, 1A (when the last one is there, and assuming I understand the buffer entry correctly).

Two quick guesses: 1) the report types don't seem to line up with any of the IDs posted before, so I'm guessing the other report IDs might be in these data structures somewhere? 2) the 1C and 0D structures are similar in size and always show up next to each other, so maybe these are the RX and TX data from those papers?

I'll attach a couple of binary dumps for each of the report types (data1 is 0D, data2 is 1A and data3 is 1C).

data1-4.bin.txt
data1-14.bin.txt
data2-4.bin.txt
data2-14.bin.txt
data3-4.bin.txt
data3-14.bin.txt

[qzed]

1. the report types don't seem to line up with any of the IDs posted before, so I'm guessing the other report IDs might be in these data structures somewhere?

That's correct. The HID report IDs are mostly useless: HID reports have fixed size, but the (digitizer) "heatmap" data containing all information can be of variable length. So what they did is simply specify a couple of HID reports all containing HID digitizer heatmap data, each with a different length. That means that the controller sending us the data just chooses whatever report ID best fits the data, i.e. whatever report ID has the next larger size, and pads the rest with zeros.

The IDs linked above are sub-IDs inside the heatmap data. Here's a parser for that data (see usage e.g. here) and here are some partially annotated data dumps of the heatmap data. Note that the naming is a bit different: I've called the "reports" in the parser and issue p-frame in the data dumps. Also the report IDs are two bytes as far as I can tell, but from reverse engineering the DLL there's only one byte so the first byte (usually 0x04) may be some bit-field indicating some status or may be a parent category or something. For example 0x0425 is what we currently parse as actual touch heatmap data, not to be confused with digitizer heatmap data which is everything inside the HID report, so 0x04 would be maybe the status/parent thing and 0x25 the actual ID. From the parser notebook I've linked above, the first byte is either 0x00, 0x04, or 0x08 and both instances seem to have the same payload so my bet is on some bitflag stuff.

2. the 1C and 0D structures are similar in size and always show up next to each other, so maybe these are the RX and TX data from those papers?

Possible, but as I've said the HID report IDs don't really mean anything. There may still be some correlation. Note that it's also possible that some of it is multitouch data. I've seen the pen produce some interference (albeit small) with regards to touch. So I think it's quite possible that you also get some multitouch data mixed in.

(There are also two others that show up with or without the pen, 0x07 and 0x08, which I think are some sort of noise? They have an ipts_data size of just 64 bytes).

You could look at the HID report descriptor to figure that out. Using the one from #4 (comment) (I think that should fit given the other IDs), reports 0x07 and 0x08 are also heatmap data. 0x07 with 61 bytes (1 byte report-id + 2 bytes scan-time + 61 bytes payload = 64 bytes) and 0x08 with 209 bytes (1 byte report-id + 2 bytes scan-time + 209 bytes payload = 212 bytes).

[haymanpf]

Ah shoot. Thanks for the tools though!

[qzed]

Feel free to poke around at the data with them, let me know if something doesn't work.

[StollD]

Should probably post this here too:

I've spent some time figuring out how the reports for the pen data are formatted. Together with the report names that @qzed already figured out, maybe someone has any ideas about the data in the reports

https://gist.github.com/StollD/ae901f46bf693a0c5355cc048cb21073

[quo]

I had a quick look at the pen data on a SP7+. The most interesting data seems to be in the DFT window packets with length 1548. I did not really look at the other packets yet.

Typically, there are three of these DFT packets in a row, which contain slightly different data. Let's call them P, Q, and R.

The structure of the packets is roughly as follows:

struct pen_dft {
    u32 timestamp; // counting at approx 8MHz
    u8 a; // always 16
    u8 b; // usually P=6, Q=7, R=9, very rarely P=3, Q=4, R=5
    u8 c; // usually 1, can be 0 if there are simultaneous touch events
    u8 d; // usually 1, can be 0 if there are simultaneous touch events
    u8 e; // usually 1, but can be higher (2,3,4) for the first few packets of a pen interaction
    u8 f; // data type? P=10, Q=10, R=11
    i16 g; // always -1

    struct pen_dft_row {
        u32 h; // fixed value, P/Q x and y use the same set of 16 values, R x and y use another set of 16 values
        u32 i; // value related to magnitude of dft components, possibly some kind of amplitude
        i16 real[9], imag[9]; // DFT components
        u32 j; // related to pen position: high for top/left, low for bottom/right
    } x[16], y[16]; // x rows encode x-coordinate data, y rows encode y-coordinate data
};

There are some additional patterns in the row data:

• For P and Q, there appear to be two types of rows, alternating in pairs: row 0/1 are type PQ1, row 2/3 are type PQ2, row 4/5 are type PQ1, etc.
• For R, there also appear to be two types of rows: the first 7 rows are type R1, and the remaining 9 are type R2.
• The PQ1 rows become all zero (including h/i/j) when the pen is moved away from the screen.
• For PQ2, R1 and R2: When the pen is at the top/left of the screen, the first 5 real/imag components are nonzero, the other 4 are zero. When the pen is at the bottom/right, the last 5 components are nonzero, the first 4 are zero.
• For R1: The middle row (index 3) has the largest values.
• All x rows in the same packet with the same type have the same value for j. Same goes for all y rows. The j values only change as the pen moves across the screen.

Edit: It looks like there are actually 64 DFT bins in the X direction and 44 bins in the Y direction, and we only get a 9 bin window from that. The j value is actually 3 signed bytes + a nul byte. The signed bytes are the indices of the first/last/middle bins in the window (since there 9 bins, they are always related by: first + 4 == last - 4 == middle). When the pen is at the edge of the screen, the first index can be negative or the last index can be larger than 63 or 43, and the corresponding components are zero.

[quo]

The smaller DFT window packets have the same format, but contain fewer rows. The "a" value is the number of rows per coordinate.

When the pen is touching the screen, the position can be determined fairly simply from the R1 rows by finding the bin index with the largest absolute value, then interpolating with its neighbors. E.g.:

#define NUM_DFT 9
struct pen_dft_row {
    u32 unknown;
    u32 magnitude;
    s16 real[NUM_DFT], imag[NUM_DFT];
    s8 first, last, mid, zero;
};

#define SCALE 0x10000
static int pen_get_pos(struct pen_dft_row *r) {
    int maxampsq = 0, maxidx = 0;
    for (int i = 0; i < NUM_DFT; i++) {
        int ampsq = r->real[i] * r->real[i] + r->imag[i] * r->imag[i];
        if (ampsq > maxampsq) {
            maxampsq = ampsq;
            maxidx = i;
        }
    }
    // interpolate using Eric Jacobsen's modified quadratic estimator
    int i = maxidx < 1 ? 1 : maxidx > NUM_DFT-2 ? NUM_DFT-2 : maxidx;
    int ra = r->real[i+1] - r->real[i-1], rb = 2*r->real[i] - r->real[i-1] - r->real[i+1];
    int ia = r->imag[i+1] - r->imag[i-1], ib = 2*r->imag[i] - r->imag[i-1] - r->imag[i+1];
    int div = rb*rb + ib*ib;
    int d = div ? SCALE * (s64)(ra*rb + ia*ib) / div : 0;
    return (r->first + i) * SCALE + (d > SCALE/2 ? SCALE/2 : d < -SCALE/2 ? -SCALE/2 : d);
}

The pen pressure can also be determined from R1. When pressure is high, the first row will have the largest DFT components and magnitude (i) value. When pressure is low, later rows will have the largest values. So pressure can be calculated by finding the index of the row with the largest magnitude value, then interpolating with neighboring rows.

Of course this is only using a fraction of the data the device sends, so presumably you would be able to get much better accuracy by using the rest of the data. Especially when the pen is not touching the screen, the R1 data is very noisy.

[qzed]

Awesome! And kinda interesting... so you're saying the DFT bins relate to position instead of some frequency? Or maybe a better interpretation is the DFT bin for some single frequency at some position (could fit the "antenna" name)... somehow this starts to make sense to me.

[quo]

I was mostly just looking at the data without thinking too much about what's going on at the hardware level.

From skimming the papers, it looks like they have horizontal and vertical wires, and are transmitting different frequencies on one set, while receiving on the other set. The pen creates a coupling between the wires, so running a DFT on the received data lets you figure which transmissions are most strongly coupled, ie. which wires the pen is nearest to. The pen can modulate the coupling or transmit its own frequencies to transmit data about pressure, etc.

After thinking about it a bit more, I'm not entirely sure the 9 values we get are actually bins from a single DFT. It could also be that they are all bins calculated at the exact same frequency, but from different RX wires (calculating a single bin is computationally very simple, you'd just multiply every sample with a sin and cos value). The unknown value at the start of each row could represent the bin frequency for the entire row.

Also, in the R1 data, the center bin almost always seems to have the largest value, so in the code I gave the for loop is unnecessary and you can just do int i = NUM_DFT/2 instead.

I should mention that I'm testing with an old SP4 pen, which according to MS doesn't support more advanced stuff like tilt. The PQ packet data just looks like noise to me, it's possible that's supposed to be something like tilt data which my pen just isn't sending.

[qzed]

After thinking about it a bit more, I'm not entirely sure the 9 values we get are actually bins from a single DFT. It could also be that they are all bins calculated at the exact same frequency, but from different RX wires (calculating a single bin is computationally very simple, you'd just multiply every sample with a sin and cos value). The unknown value at the start of each row could represent the bin frequency for the entire row.

Yes, that's what I meant with "bins relate to position [i.e. wire] instead of some frequency". I haven't had a detailed look at the papers yet, but I think that this would make sense from a logical perspective: Assuming the pen sends some fixed single-frequency signal (at least one frequency for position) and the wires receive that, then you could just check for where reception of that particular frequency is best (assuming there's notable damping over that "small" space).

Alternatively I guess the pen could be the receiver and wires could send different frequencies each, meaning that a single bin again relates to a specific wire (this time not position but frequency). Assuming damping over space and that each wire has the same output power, you'd then look for the most prominent frequency which gives you the closest wire. I'm not sure if that's the case though because then the pen would need to send the received data back somehow... having the pen just send data seems a lot easier to implement.

If you have one set of wires transmitting and another one receiving, maybe you again send different frequencies on different sender wires and look for them on the receiver ones. So on one hand you have strength over the specific frequencies giving the position over sending wires and strength over the full spectrum giving the position on the receiving side. Now I have no idea if that's physically possible because that'd somehow assume that without the pen receiving and sending sides are entirely (or mostly) disconnected and only the pen can "connect" them (maybe via frequency shifting?).

Again, I probably need to read the papers...

At some point I'll hopefully have access to the Pro X touch data (with new pen and all), but that will probably still take a while.

What I found quite interesting on the SB2: You can somehow have two pens on the panel and one still works as expected (i.e. there's no jumping between them). The other one just gets ignored completely. Not sure if that can be done entirely with software based filtering. But then again pens also have a serial number so that also needs to be sent somehow (so filtering could work via that if you can actually associate pen data directly with it).

[gurrgur]

I recommend this paper which goes into more detail about MFDM than the overview paper mentioned in #14 (comment).

From my understanding (and assuming MFDM), the pen encodes it's coordinate, pressure and (if applicable) tilt by transmitting a signal containing frequencies fs1, fs2 and fs3 from the pen tip.
fs1 encodes the pen coordinate and is coupled into electrodes Rx and Tx for which charge signals Qs are measured. Therefore we can assume that at least some of the DFT bins we receive represent charge signals Qs of Tx and Rx evaluated at the single frequency bin fs1.
On the other hand fs2 and fs3 are used to encode pressure and tilt. However the pen does not transmit them unaltered but shifts them proportional to the quantity they describe. Accordingly Δfs2 and Δfs3 are proportional to pressure and tilt. It's reasonable to assume that for pressure and tilt neighbouring DFT bins (frequency wise) of fs2 and fs3 are reported.

Finally, the reason for multiple pens not interfering with each other is that the MCU instructs each pen to transmit at specific frequencies. Thus with multiple pens available the MCU can assign them different frequencies.
This seems to be more of a side benefit though, since the main reason for adjusting the pen's transmitting frequencies is to improve the SNR by avoiding parts of the spectrum polluted by externally induced noise.

Note that I'm writing this purely based on my understanding of some of the papers mentioned. I did not yet have a look at actual pen data reports.

[csdvrx]

On the other hand fs2 and fs3 are used to encode pressure and tilt. However the pen does not transmit them unaltered but shifts them proportional to the quantity they describe.

Would it be possible that some of these measurements are only possible dynamically, not statically?

After doing some tests with a Lenovo Active Pen # 5T70J33309
(Wacom), I found the hovering function didn't work unless the top part (the one with the writing tip) was screwed in.

There doesn't seem to be any electrical contact mechanism, unlike in the bottom part (AAAA battery holder with a spring).

Given how we also know the pen is continuously discharging its battery, this makes me believe:

• the pen is always emitting

• the X,Y coordinates can be read using the 2d antenna that exist behind the laptop screen (if it's anything like a thinkpad: they contain a metal foil that seems necessary to the wacom function)

• the Z coordinate is far less precise, and the variance (noise) might be too high to detect a click: no static detections of clicks

• so the tip of the pen must contain a pressure sensitive component, with non linear increases, to be more discriminative for a click detection

• when the top part is present, it applies a small but necessary amount of pressure: this gives a baseline, the measured (min,max) are used by some sort of calibration to give the "hover" function

• a "click" creates a discrete jump far away from the max of this range: this delta is measured

• the pressure sensitive tip can encodes the pressure more accurately than (click, no click) in some kind of stepwise function with N levels, N being a function of both the sensitivity of the receptor and the encoding capacity of the pen hardware/firmware. There could be a non linear scale (a log scale would make sense for pressure), using 12 bits for the 4096 levels, and fewer in earlier version of the pen (less sensitive)

It's just a bunch of speculations, and I'd need to test with different models and generations of Wacom pens, but it would seem to make sense.

Tilt was introduced after pressure sensitivity, so it's likely to be a very small improvement of the exact same method.

To do tilt in the same way, besides the tip of the pen Z1 I'd have added another mapping along the Z axis , for example the middle or the top of the pen: let's call it Z2.

Even if Z measurements are noisy, if both Z1 and Z2 are subject to the same noise, you could estimate the tilt by projecting both Z1 and Z2 on the X,Y plane - maybe after correcting by this noise if it's non homogeneous: the click event would be the perfect opportunity to get Z1star, from which you could compute Z1-Z1p as the noise on this part of the screen, and use it to better estimate the projection of Z2.

Accordingly Δfs2 and Δfs3 are proportional to pressure and tilt. It's reasonable to assume that for pressure and tilt neighbouring DFT bins (frequency wise) of fs2 and fs3 are reported.

Another explanation is that the deltas you are getting are Δfs2=Z1 and Δfs3 is either Z2 or Z1-Z2, while the actual click replaces Z1 by Z1star: so outside of a click event, you get an approximate proportional measurement, during a click event you get the precise value that allow you to get the correct pressure and tilt.

Especially when the pen is not touching the screen, the R1 data is very noisy

If my speculations above are correct, the R1 data is noisy due to being within the (min,max) that are dynamically calibrated to get the "hovering" function.

It should be possible to test my hypothesis by simulating clicks at various degrees (for 90 deg, put the pen very close to the screen and trigger a click of its tip with your fingernail), and plotting.

I've played a bit too much with my 5T70J33309 pen, testing it in various conditions and uh... now I can't unscrew it anymore lol so I won't be able to finish that testing :)

Also, I have some weird i915 issues on my lakefield X1 fold so I can't event do more basic stuff, like a high frequency display of the readings while also doing an external video record of the pen position on the screen: on linux, I just can't get the i915 driver to claim the video hardware, regardless of the forceprobe options (i915.forceprobe=9840 given lspci -nn), the distribution tried (Ubuntu 22 or Fedora 22 both containing the latest linux-firwmare for the huc and guc) or the kernel (5.17)

It's likely to be a stupid bug, hopefully my speculations may help a little with the IPTS protocol.