Tx sample scale merge to main

[Abraxas3d]

The Case of the Missing 24 dB Transmit Power

The Opulent Voice modem for the LibreSDR graduated from the lab to the field in late March 2026. Instead of coaxial cables connecting transmitter to receiver, and receiver to transmitter, we now connected our brave little radios to filters and outdoor antennas.

And, nothing was received. The signal levels appeared to be very low. Even moving the antennas right next to each other resulted in only a few scattered frames demodulated and decoded.

On 24 March 2026, we selected one of the amplifiers at random in order to characterize it in ORI’s Remote Labs. We connected the input of the amplifier to the output of the DSG821A signal generator. The signal generator was set to 431 MHz, which was the frequency we wanted to use. We connected the output of the amplifier through a 6 dB attenuator to the Rigol RSA5065N Spectrum Analyzer. We fitted a JST-HX power cable to the power connector of the amplifier. We provided 12 volts of power from the DP832 lab power supply.

The amplifier made 27 dB of gain from -100 dBm input to about -3 dBm input, made 20 dB at 0 dBm input, and worked pretty well up to 9 dBm input.

The next test was to remove the signal generator and connect a LibreSDR running Opulent Voice. Instead of a carrier wave from the signal generator we’d be sending an 81 kHz wide minimum shift key (MSK) signal from a real modem through the amplifier. We were intending to repeat the measurements we’d made with the signal generator. However, we noticed something very interesting. The signal level from the LibreSDR was expected to be about 0 dBm, which would provide enough drive to the amplifier to create enough gain to help our over-the-air tests succeed. However, when the LibreSDR, running Locutus and Dialogus, was commanded to transmit with PTT and audio frames from Interlocutor, the peak of the main lobe of the MSK signal was at 1 microwatt. If this was the true power output of the LibreSDR, then no wonder the over-the-air tests had failed.

The transmit power hardware attenuation setting was confirmed to be at 0 dB. This is set through an Industrial Input and Output (IIO) library attribute call, was correctly reported, and we saw that changing the attribute caused the signal to increase or decrease by the exact amount of gain. So, it wasn’t a configuration error. As far as the hardware was concerned, it was transmitting at 0 dBm.

The other possibility was that the I and Q signals were not being generated for transmit at full scale. If we weren’t filling up the registers correctly, then maybe we were accidentally dividing our signal down before it got to the antenna. Investigation turned to the Hardware Descriptive Language (HDL) files.

The Opulent Voice VHDL language modem, called Locutus, runs inside the LibreSDR FPGA. Data frames arrive via direct memory access, pass through the Opulent Voice frame encoder, are convolutionally encoded (K=7, rate 1/2), go through a byte-to-bit deserializer, and the resulting bits are sent to the MSK modulator. The modulator produces the I and Q samples that drive the AD9363 digital to analog converter (DAC). Software in the general purpose processor of the LibreSDR configures the IIO context and controls PTT.

The direct memory access transfers protocol data frames into the LibreSDR, and not IQ samples. So, the classic PlutoSDR bug of 12-bit samples being miscounted in a 16-bit word did not apply here. The modulator itself generates all I and Q waveforms. The frequencies are set by Dialogus at startup.

The Integrated Logic Analyzer (ILA) in the bitstream already had probes on two very important signals, tx_i_sync and tx_q_sync. These signals were measured right at the point where the samples enter the AD9361 core. A January 2026 ILA capture told the story clearly. The waveform showed clean MSK signals. No corruption, no skips, and with the exact right relationship to each other. At the time, this was a big milestone and part of the process of troubleshooting the porting of the HDL code from the PlutoSDR to the LibreSDR. But we’d overlooked something critical. The bug was right there in an otherwise perfect image.

The peak values of the I and Q waveforms were only plus and minus 1100 or so, in a 16-bit signed word. At first glance, a value of 1100 in a 16-bit word might not raise any red flags. The alarm bells ring when you know how the axi_ad9361 core actually reads those 16 bits.

There are two different conventions on the same bus. The axi_ad9361 core uses 16-bit data buses internally. However, the AD9361 and AD9363 (the chips used in these software-defined radios) have only 12-bit digital to analog converters. The documented convention, confirmed by a tour through Analog Devices Engineer Zone forum, is as follows.

RX (ADC Output) is 12-bit value in [11:0], sign extended to [15:12]
TX (DAC Output) is 12-bit value expected in [15:4], which is the top 12 bits

In plain English, RX gives you the data right-justified. TX expects it left-justified. These are opposite conventions on the same 16-bit bus, and the apply whether the interface is CMOS (PlutoSDR) or LVDS (LibreSDR).

Our code, from msk_modulator.vhd, in the carrier_mod_proc section looks like this.

tx_samples_I <= std_logic_vector(resize(s1s + s2s, SAMPLE_W));
tx_samples_Q <= std_logic_vector(resize(s1c + s2c, SAMPLE_W));

s1s and s2s are each signed 12-bit values from the numerically controlled oscillator (NCO). The lookup table fills using the command

ROUND(SIN(theta) * 1024.0)

which gives a peak value of plus or minus 1024. VHDL addition of two such values produces a 12-bit result that ranges from -2048 to +2048. So far so good. The resize call then sign-extends that 13-bit result into 16 bits. This is a right-justified 16-bit word, which is the opposite of what the Analog Devices core expects.

The full chain of what happens to the signal amplitude can be calculated.

The lookup table output is [11:0] signed and is a 12-bit sinusoid.
s1s + s2s is [12:0] signed and is a 13-bit sum.
Resize(…, 16) [15:13] sign extension with [12:0] as the data. This is right-justified.
Analog Devices chip reads transmit values as [15:4], sending the top 12 bits to the DAC.
Analog Devices reads [15:4], we drive [12:0], and this is a divide by 16 to the amplitude.

What’s the damage? -24 dB.

Why did this work in the PlutoSDR? Well, it didn’t. It did not produce full power, either. The same modulator code drove the Pluto variant of Opulent Voice. The -24 dB bug was there too. Why did we not notice it? We never graduated to over-the-air tests with the PlutoSDR. All of the tests transmissions were in the lab and were either conducted through coaxial cables or done with Vivaldi lab antennas right next to each other on the bench. With conducted tests, everything worked perfectly.

For ORI’s LibreSDR work, we were now in the field. We wanted to characterize the modem output before adding an amplifier. That scrutiny revealed the long-lived bug in the HDL.

Matthew Wishek NB0X implemented a fix on the tx_sample_scale branch of the published repository, with changes to two submodules, the NCO and the msk_modulator. No changes to the block design TCL or to msk_top.vhd were required.

In the NCO (sin_cos_lut.vhd), a new constant was introduced:

CONSTANT FULL_SCALE : INTEGER := 2**(SINUSOID_W-1) -1. 

And, the lookup table fill function was changed from the hardcoded

1024.0 to * real(FULL_SCALE). 

With SINUSOID_W = 12, this gives FULL_SCALE = 2047, filling the entire signed 12-bit range. The fix is fully generic. It works for any value of SINUSOID_W.

-- Before:
tmp := ROUND(SIN(theta) * 1024.0);
-- After:
CONSTANT FULL_SCALE : INTEGER := 2**(SINUSOID_W-1) - 1;
tmp := ROUND(SIN(theta) * real(FULL_SCALE));

In the modulator (msk_modulator.vhd), a new 3-bit input port tx_shift : IN std_logic_vector(2 DOWNTO 0) was added. The IQ output assignment was changed from a plain resize() to a shift_left() whose amount is driven by tx_shift at runtime.

-- Before:
tx_samples_I <= std_logic_vector(resize(s1s + s2s, SAMPLE_W));
-- After:
tx_samples_I <= std_logic_vector(
    shift_left(resize(s1s + s2s, SAMPLE_W), 
        to_integer(unsigned(tx_shift))));

The full 12-bit scale was achieved. With the sum now peaking at plus or minus 4094, left-shifting by 3 puts the signal in the correct place, which is [15:3]. The Analog Devices core reads [15:4], which is the full DAC scale. Making tx_shift a configurable port rather than a hardcoded constant is an elegant touch. Dialogus sets it through the register map at runtime, with no bitstream rebuild needed.

With the tx_sample_scale fix integrated and a new bitstream loaded, the Opulent Voice modem then achieved its first successful over the air transmission. This was from one building to another, with the full signal chain, from a LibreSDR to another LibreSDR. Voice traffic and text messages were received, with excellent audio quality. The ~30 dB shortfall that had been quietly sitting in the hardware since the original modulator was gone.

Lessons Learned

RX and TX use opposite justify directions in axi_ad9361. This is documented, but really only in a so-called Verified Answer on Analog Devices Engineer Zone forum. It’s not prominently documented in the IP wiki. The wiki describes the 16-bit data base and mentions that the IP “always works in 16 bits”, but does not call out the left/right justification asymmetry in a way that is easy to find. If you are writing custom HDL that drives DACs, then you should read the forum thread at https://ez.analog.com/fpga/f/q-a/112155/axi_ad9361-data-format

ILA probes are worth their cost. The screenshot from the ILA capture back in January 2026 told us the answer, if we had known what the question was. Running ILA and keeping the results pays off because you can go back and look at signals that may not be accessible otherwise. Wire up ILA early and often and be curious about your signals. Go for a tour. Explore your design and the design of any infrastructure that you are working with.

-24 dB is a recognizable signature. In fact, any multiple of -6 dB is significant. Each bit of DAC resolution is 6 dB, so if you’re missing something like 24 dB, then an inadvertent four-bit shift might be the culprit.

Fix things at the right layer. The initial discussions included assumptions such as “the fix should live in the block design TCL file” or maybe in msk_top. Matthew chose to fix it inside the modulator and NCO submodules. This is the better choice. It makes the modules self-consistent, removes the need for platform-specific fancy workarounds or settings, and ensures that any future target automatically benefits. When a submodule’s output format is wrong, fix the submodule rather than papering over it at the integration layer.

The Receiver Fix

The modem was working, and the transmitter was producing 0 dBm signals, but the receiver levels seemed very puny. Weak 1 and weak 0 were observed in conducted loopback. This was wrong. The primary fix for this came down to a single line of VHDL, a >>5 shift that was silently discarding 97% of the signal energy before it ever reached the frame sync detector.

The receive chain processes incoming MSK-modulated RF signal through a Costas loop demodulator, frame sync detector, convolutional FEC decoder (soft Viterbi, K=7), deinterleaver, and derandomizer before delivering decoded frames to the ARM processor via DMA.

The Case of the Confused Receiver

The approach was to instrument the receive chain with Vivado ILA (Integrated Logic Analyzer) probes at key stages and trace the signal level from the RF connector to the frame sync correlator. Using ILA, we confirmed that the ADC was receiving clean signal. The Costas loops were locking, when the window of integration time was turned up to the maximum number (1023). But frame sync would never transition from HUNTING to LOCKED.

What we expected:

The sync word correlator computes a weighted dot product of 24 incoming soft values against the known 24-bit sync word (0x02B8DB). With healthy soft values of ±600, the peak correlation should reach approximately 24 × 600 = 14,400. This is just above the HUNTING_THRESHOLD of 14,000.

What we measured:

The ILA showed the ADC outputting a healthy ±594 counts — right where it should be. But soft values at the input to the frame sync detector were only ±209. The peak correlation was reaching only 5,016 — permanently below the 14,000 threshold. Frame sync couldn't lock because the math made it physically impossible.

Finding the culprit:

Tracing the signal through the fixed-point chain stage by stage:

rx_samples (ADC):         ±594    12-bit
rx_cos (mixer):           ±1.2M   24-bit  (12×12 product)
rx_cos_acc (integrate):   ±27M    32-bit  (~45 samples/symbol)
rx_cos_dump (>>12):       ±6,679  15-bit effective  (normalize product)
data_out (>>5):           ±209    16-bit  ← ROOT CAUSE

The >>5 shift in costas_loop.vhd was reducing the signal by a factor of 32. This shift was introduced in commit f604e97 (October 6, 2024) with the message: "Fixed a data-scaling issue, works in simulation, but needs further review." So, we are doing that review now!

Why simulation didn't catch it:

The HUNTING_THRESHOLD in simulation was calibrated for the attenuated ±209 soft values, so the correlator worked in simulation by accident. Hardware used the same threshold but the real-world signal path exposed the underlying problem.

---

The Fix

One line changed in costas_loop.vhd:

-- Before (32× signal loss):
data_out <= std_logic_vector(resize(shift_right(rx_cos_dump, 5), DATA_W));

-- After (minimum safe shift, prevents 16-bit overflow):
data_out <= std_logic_vector(resize(shift_right(rx_cos_dump, 1), DATA_W));

Why >>1 specifically: with full-scale ADC input (±2047), the integrate-and-dump output (rx_cos_dump) reaches ±23,018. Shifting right by 1 gives ±11,509 — which fits in a 16-bit signed value (max ±32,767). Shifting right by 0 would overflow at full scale.

After the fix, three other values needed recalibration for the new ±3,340 soft value range:

1. quantize_soft() thresholds in frame_sync_detector_soft.vhd — scaled proportionally from ±360 simulation values to ±3,340 hardware values
2. HUNTING_THRESHOLD — updated from 10,000 to 60,000 (75% of expected peak correlation 80,160)
3. LOCKED_THRESHOLD — updated from 5,000 to 36,000 (45% of peak)

---

A Second Receiver Bug

With the soft values fixed, the frame sync detector locked immediately. But frames were still not being decoded — approximately 25 errors per second (one per frame) with zero frames received by Dialogus.

ILA and register readback revealed the OV frame decoder was stuck in state 5 (PREP_FEC_DECODE) permanently. This state packs the deinterleaved soft G1/G2 streams into the Viterbi decoder input arrays, iterating over 1,072 symbols.

The bug: the loop counter byte_idx was declared as:

SIGNAL byte_idx : NATURAL RANGE 0 TO ENCODED_BYTES;  -- max 268

But the PREP_FEC_DECODE loop needed to count to 1,072 (NUM_SYMBOLS). Vivado synthesized a 9-bit counter (sufficient for 268), which overflowed at 512 and wrapped to 0 — restarting the loop forever. The VHDL simulator allows NATURAL RANGE violations at runtime and counted correctly to 1,072, masking the bug entirely.

Fix: one line:

SIGNAL byte_idx : NATURAL RANGE 0 TO ENCODED_BITS;  -- max 2144

After both fixes, the decoder cycled through all states correctly and frames arrived at the ARM processor.

---

Results

ILA confirmation after fixes:

• dbg_rx_data_soft: ±3,255 (within 2.5% of predicted ±3,340) ✓
• dbg_rx_soft_quantized: strong decisions (0/1 and 6/7) replacing weak decisions (2/3/4) ✓
• dbg_rx_sync_corr_peak: 153,688 — well above HUNTING_THRESHOLD 60,000 ✓
• Frame sync: locked and stable ✓
• Frames decoded: confirmed in Dialogus ✓

Demo at DMEMS (IEEE San Diego):
The system was demonstrated over the air between two LibreSDR units at the IEEE DMEMS conference in San Diego on 22-23 April 2026. Text messages were transmitted and received successfully. Some missed messages were observed. This is likely related to symbol lock threshold calibration and AGC settings not yet optimized for real-world RF conditions.

---

What's Next

We believe the system is working but not yet fully calibrated for hardware. Here is a list of where we need to start in the lab with this merged build:

• Symbol lock thresholds — the costas_lock_detect accumulator is now 32-bit (fixed this session) but the symbol_lock_threshold register field is still only 16-bit. Matthew (NB0X) needs to widen the register to 32 bits in the SystemRDL and regenerate the register map.
• PI loop gains — current values work but lock time is slower than optimal. A systematic search around the current working values should reduce acquisition time. We started on this process and have a lab procedure that works.
• AGC target — currently -12 dBFS, should be -6 dBFS for MSK to better use ADC dynamic range. We have a constant envelope signal, so we can take advantage of this (up to a point).
• Quantization thresholds — currently calibrated for RF loopback. We will benefit from adjustment for over-the-air conditions.
• Split-frequency operation — full duplex TX/RX on separate frequencies for real two-way voice operation. This needs additional circuits and some purchases.

---

What did we Learn?

Trace signal levels end-to-end before tuning thresholds. A single misplaced shift (>>5 instead of >>1) reduced signal energy by 32× and propagated invisibly through the entire receive chain. No amount of threshold tuning would have fixed a correlation peak that was mathematically bounded below the threshold. Or, if we forced a low calibration threshold, we'd be working with much less resolution than we thought we were working with.

Synthesized counter widths must match the full range required. Vivado silently synthesizes the narrowest counter that fits the declared range. If the range is too narrow for the actual loop bound, the counter overflows and the state machine stalls. With no warning from synthesis and no failure in simulation. This is Vivado "working as intended" and the NATURAL numbers are needed for the function that allows us to assign our complicated addressing scheme to BRAM, but code reviews need to include this in the future.

Instrument aggressively. ILA probes on rx_samples_I_raw, rx_data_soft, dbg_rx_sync_corr_peak, and decoder_debug_state were essential for finding both bugs. Without visibility into the intermediate signal levels, the root cause would have remained invisible for much longer than it was.

---

Thanks to Paul (KB5MU) for patient hardware operation and register read/write testing throughout the bring-up process, and to Matthew (NB0X) for the underlying demodulator and decoder design.

Branch: tx_sample_scale merge pull request · Platform: LibreSDR Rev.5 (Zynq Z7020 + AD9361) · Toolchain: Vivado 2022.2