Debugging

In Part 1, we used “video over TCP” as a stress test for TCP’s behavior—examining how zero-window events, retransmits, and the masking effect reveal what’s happening inside a struggling connection.

But during those experiments, I discovered TCP throughput degraded rapidly as jitter worsened. While I knew that packet loss would destroy TCP throughput, I hadn’t quite expected the jitter-induced cliff.

At a certain jitter threshold, throughput collapses so severely that measurements become unreliable. Single tests can vary by over 100%. This “chaos zone” makes diagnosis treacherous: the same network conditions can produce wildly different results depending on when you measure.

This post explores TCP’s behavior under jitter and loss, comparing CUBIC and BBR. It’s common knowledge that TCP is inappropriate for delay-sensitive streaming data, and this post will try to demonstrate how and why.

Your security camera feed stutters. Your video call over the corporate VPN freezes. The question isn’t whether something is wrong—that’s obvious. The question is what is wrong, because the fix depends entirely on the diagnosis.

Is the problem the sender, the network, or the receiver? These require fundamentally different interventions. Telling someone to “upgrade their internet connection” when the real issue is their overloaded NVR is worse than useless—it wastes time and money while the actual problem persists.

Sender-side problems—where the source isn’t transmitting at the expected rate—are straightforward to detect: compare actual throughput to expected throughput. The harder question is distinguishing network problems from receiver problems when data is being sent. TCP’s built-in feedback mechanisms give us the answer.

UDP is the natural transport for real-time video—it tolerates loss gracefully and avoids head-of-line blocking. But video often ends up traveling over TCP whether we like it or not. VPN tunnels may encapsulate everything in TCP. Security cameras fall back to RTSP interleaved mode (RTP-over-TCP) when UDP is blocked. Some equipment simply doesn’t offer a choice.

The research question driving this investigation: Can we identify reliable TCP metrics that distinguish network problems from receiver problems?

Through controlled experiments, I found the answer is yes—with important caveats. This post builds a complete diagnostic framework covering all three problem types, with the experiments focused on the harder network-vs-receiver distinction. Part 2 will explore what happens when TCP goes chaotic.

Many consumer and industrial devices—from home security cameras to the HDMI-over-IP extenders I investigated in a previous post—are designed as simple appliances. They often have hard-coded, unroutable IP addresses (e.g., 192.168.1.100) and expect to live on a simple, isolated network. This becomes a major problem when you need to get their multicast video or data streams from that isolated segment to users on a main LAN. My goal was to solve this with a standard Linux server, creating a simple, high-performance multicast router without a dedicated, expensive hardware box.

Can a standard Linux server act as a simple, kernel-native multicast router for devices with hard-coded, unroutable IP addresses? This article chronicles an investigation into combining nftables SNAT with direct control of the Multicast Forwarding Cache (MFC).

On a good day, my day job involves building networking tools in python. Too many python networking tools look like shell scripts, spawning subprocesses for basic tools like ‘ping’ or ‘ip’ - often resulting in a fragile mess due to poor, or inconsistent error handling.

I was quite excited to find icmplib. It provides a much simpler, less fragile way to do things like reachability tests, RTT measurements, path discovery and path MTU discovery in python code. Hopefully it finds its way into Fedora soon!

Armed with icmplib, I went on a journey of discovery to develop my understanding of Path MTU. I specifically wanted to understand the differences between IPv4 and IPv6, and the effect of VETH, VLAN and Bridge virtual devices.