Metric</th>	Value</th></tr></thead>
Commits</td>	97</td></tr>
Development time</td>	6.5 months</td></tr>
Lines added</td>	27,293</td></tr>
Lines removed</td>	1,786</td></tr>
Net change</td>	+25,507 lines</td></tr>
Files modified</td>	277</td></tr>
Test code</td>	3,448 lines</td></tr>
Test topologies</td>	7</td></tr> </tbody></table> The tests cover multi-area flooding, mixed legacy/E-LSA interoperability, E-LSA-only networks, broadcast DR/BDR election, and graceful restart. The code works. The approach handles the complexity.</p> “Not How Things Are Done Here”</h1> The implementation went through three PR iterations:</p> PR #16199</a> — Initial approach</li> PR #16532</a> — Revised approach</li> PR #17343</a> — Final iteration</li> </ul> The feedback: callback-based iteration is wrong for this codebase. Callbacks add indirection. They make code harder to follow, harder to debug. This isn’t how things are done here. The alternative: separate code paths for legacy and E-LSA.</p> I understood the concern. I’ve worked in codebases where callbacks created impenetrable spaghetti. The objection wasn’t unreasonable.</p> But I also saw the duplication path leading somewhere bad—a lot of nearly-identical code to maintain in sync forever. I asked for more specific guidance. What would an acceptable middle ground look like? The answer was conceptual rather than concrete: not callbacks, but no specific alternative I could implement.</p> I tried variations—inlining some callbacks, reducing indirection, restructuring the handler registration. None landed. Eventually the discussion ended without resolution.</p> Was I Wrong, or Just Unwelcome?</h1> Here’s what I’m still sitting with.</p> The code works. The tests pass. The iterator pattern handles both formats correctly across all 9 LSA types, SPF, ABR, ASBR, flooding, and graceful restart. We had confidence in the technical approach because we had working software proving it.</p> What I don’t know is whether my instincts were right for this context.</p> Maybe the duplication approach would have been fine—maybe the maintenance burden I imagined wouldn’t materialize, or would be acceptable to people who know this codebase better than I do. Maybe there’s a third approach I didn’t see because I was anchored on the patterns I knew. Maybe my years of “abstraction over duplication” instincts are a bias, not wisdom, in environments where those patterns aren’t established.</p> Or maybe I was right, and I just couldn’t communicate it effectively. Or maybe there’s no clean answer and the feature is just genuinely hard to add to this architecture. I don’t know.</p> What I do know is that experience from one context doesn’t transfer automatically to another. The instincts that made me effective elsewhere made me confident in an approach that wasn’t accepted here. Whether that’s because I was wrong, or because I failed to build consensus, or because the environment has different values—I genuinely can’t tell.</p> I’ve written a detailed analysis</a> comparing approaches, trying to work through the trade-offs. I still don’t have a clear answer.</p> A Working Branch, An Open Question</h1> The branch</a> works. It passes tests. It’s a complete reference implementation of RFC 8362 for ospf6d, if anyone wants it.</p> If you need E-LSA support in production today, Holo Routing</a> is a Rust-based routing suite with RFC 8362 support and active development. For FRR, the feature remains unimplemented.</p> This is one of those experiences that shaped me. I spent months pushing against a door that wasn’t going to open—not because the work was wrong, but because it wasn’t wanted in that form, in that place, at that time. The conclusion I came to: put your efforts where they’re welcomed, or go your own way.</p> Acknowledgments</h1> Thanks to Jake Lodge, who found and fixed countless bugs in this code. This implementation wouldn’t exist without his work and support.</p> Linux Network Configuration: A Decade Later 2025-12-22T00:00:00+00:00 In 2014 I wrote about the state of Linux network configuration</a>, lamenting the proliferation of netlink libraries and how most projects hadn’t progressed past shell scripting and iproute2. I concluded that “there is a need for a good netlink library for one of the popular scripting languages.”</p> A decade later, that library exists. More importantly, the ecosystem has matured enough that every major language has a credible netlink option - and production systems are using them.</p> To compare them, I’ll use the same example throughout: create a bridge, a network namespace, and a veth pair connecting them.</p> MCR: A High-Performance Userspace Multicast Relay 2025-11-18T00:00:00+00:00 In my previous post</a>, we hit a wall with the kernel’s multicast forwarding logic. The core issue was the RPF (Reverse Path Forwarding) check failure</strong> for sources on unroutable networks. While digging deeper into this, I found it’s a well-documented challenge in multicast literature, often called the “last-hop/first-hop problem”</strong>. It describes the difficulty of bridging the gap between an isolated multicast source and the main network.</p> The investigation concluded that a userspace relay was the right approach. This post introduces MCR (Multicast Relay)</strong> (https://github.com/acooks/mcr</a>), a specialized userspace relay built in Rust, as a modern, high-performance implementation of that architectural pattern. The project was built by myself, with the help of AI models Gemini and Claude. Its goal is to provide a robust solution to the first-hop problem where throughput and dynamic control are critical.</p> Why Userspace? A Necessary Trade-off</h3> Before diving into the architecture, it’s important to address a fundamental question: why move packet forwarding into userspace at all? For raw throughput, kernel-native forwarding is almost always faster. The decision to build a userspace relay was not driven by a desire for more performance, but by necessity, after concluding that kernel-level solutions were insufficient or impractical for this specific problem.</p> The core challenge is the kernel’s Reverse Path Forwarding (RPF) check. While it’s true that this check can be disabled on a per-interface basis (by setting the `rp_filter</code> sysctl to 0), this is often an inadequate solution for two key reasons:</p>` It’s a Blunt Instrument:</strong> Disabling rp_filter</code> turns off a key anti-spoofing security mechanism for all</em> traffic on that interface, not just the multicast stream in question. This is a security trade-off many administrators are unwilling to make.</li> It Doesn’t Solve the Whole Problem:</strong> More importantly, simply disabling the RPF check on the first router doesn’t “clean” the packet. The packet is still forwarded with its original, unroutable source address, which will cause RPF failures on correctly configured downstream routers. It pushes the problem one hop further away without solving it.</li> </ol> With the rp_filter</code> approach being insufficient, we are left with other kernel-level mechanisms, which also have limitations:</p> No Inbound Multicast SNAT:</strong> Our previous investigation proved that it’s architecturally unsupported to use Netfilter’s conntrack</code> to perform SNAT on inbound multicast, which would be the most direct way to “clean” the source address.</li> Static Forwarding Limits:</strong> Using direct control of the Multicast Forwarding Cache (MFC) works for routable sources, but the kernel has a hard-coded limit of 32 Virtual Interfaces (MAXVIFS</code>), making it unsuitable for high-density scenarios.</li> </ul> Given these constraints, a targeted userspace application becomes the most practical way to solve the problem robustly at the network edge. It allows us to “launder” the packet once, making it fully compliant for the rest of its journey through the network. This trade-off—sacrificing the raw speed of the kernel for the correctness and flexibility of a userspace application—means that performance cannot be taken for granted and must be a central goal of the design.</p> How it Works: MCR’s Architecture</h3> MCR’s architecture is designed to solve the RPF problem by bypassing the kernel’s IP stack and giving the operator direct, dynamic control over forwarding. It consists of two main components:</p> The Supervisor & Workers:</strong> The main multicast_relay</code> process acts as a supervisor. It spawns high-performance “worker” processes, pinning each to a specific CPU core. These workers form the data plane, doing the actual packet processing.</li> The Control Plane:</strong> MCR is managed dynamically at runtime. A separate control_client</code> tool communicates with the supervisor over a UNIX socket, allowing you to add, remove, and list forwarding rules without ever stopping the service.</li> </ol> This design separates the data plane, which is optimized for performance, from the control plane, which is optimized for flexible management.</p> Data Plane Design for Performance</h4> At its core, MCR operates on a simple but powerful principle: if the kernel won’t forward the packet, MCR will intercept it before the kernel has a chance to drop it. This is achieved through several key architectural decisions:</p> Unified io_uring</code> Event Loop:</strong> At the heart of each worker is a single, unified event loop built on Linux’s most advanced I/O interface, io_uring</code>. This allows a single thread to manage both ingress and egress asynchronously with minimal syscall overhead, a design which avoids the need for complex and often slower cross-thread communication.</li> AF_PACKET</code> for Raw Sockets:</strong> By operating at Layer 2, MCR sidesteps the kernel’s entire IP stack on the ingress path. This is the key to bypassing the RPF check and gives MCR full control over packet handling.</li> Efficient In-Memory Fan-Out:</strong> When a single input stream must be replicated to multiple outputs, MCR uses a reference-counted pointer (Arc<[u8]></code>) to share a single packet’s memory buffer across multiple send operations. This avoids expensive memory copies in userspace, a critical optimization for high-density replication scenarios.</li> Core-Local Buffer Pools:</strong> Each worker pre-allocates and manages its own memory buffers, a strategy aimed at avoiding the performance penalty of dynamic memory allocation in the fast path.</li> </ul> While even higher performance might be achievable with technologies like eBPF/XDP, the AF_PACKET</code> and io_uring</code> approach was chosen deliberately. It offers excellent performance while maintaining compatibility with a wider range of widely deployed enterprise Linux kernels, avoiding the tighter coupling to specific kernel versions and complex toolchain dependencies that an eBPF-based solution would entail.</p> Example Workflow</h4> A typical workflow looks like this: an operator starts the multicast_relay</code> service and then uses the control_client</code> to provision forwarding rules. For example, to relay a stream arriving on eth0</code> to a new destination via eth1</code>, the command is simple and declarative:</p> ./control_client add-rule \ </span> --input-interface eth0 \ </span> --input-group 239.0.0.1 \ </span> --input-port 5001 \ </span> --output-interface eth1 \ </span> --output-group 239.0.0.2 \ </span> --output-port 6001 </span></code></pre> This command instructs a worker to listen for multicast traffic for 239.0.0.1:5001</code> on eth0</code> and re-transmit any received packets as a new stream to 239.0.0.2:6001</code> via eth1</code>. The source address of the new stream is the relay server’s own, making it fully routable.</p> What About socat</code>?</h3> For many network tasks, the versatile socat</code> utility is an excellent choice. It can be used to solve the RPF problem by creating a userspace relay, and it’s important to understand the architectural and operational differences between that approach and MCR’s.</p> Architectural Difference: Layer 2 vs. Layer 4</h4> The most significant difference is how each tool interacts with the network stack.</p> socat</code> operates at Layer 4.</strong> It uses standard UDP sockets to receive and send packets. When it receives a packet, that packet has already been fully processed by the kernel’s network stack (checksums verified, IP headers parsed, etc.). socat</code> then takes the payload and sends it out via another UDP socket, causing the kernel to build a new packet. This is a simple and robust model.</li> MCR operates at Layer 2.</strong> As detailed in the architecture above, it uses AF_PACKET</code> raw sockets to capture entire Ethernet frames directly from the network driver. This allows it to bypass the kernel’s IP stack and, crucially, the RPF check.</li> </ul> Both tools solve the RPF problem, but in different ways. socat</code> cleverly uses “local delivery”—the kernel sees a local application is the destination and doesn’t apply the forwarding</em> RPF check. MCR sidesteps the check entirely by intercepting the frame at a lower level.</p> Operational and Feature Differences</h4> The architectural differences lead to different operational models:</p> Process Model:</strong> socat</code> is a point-to-point tool. To relay a single multicast stream, you run a single socat</code> process. To relay ten streams, you must run ten separate socat</code> processes, which can be cumbersome to manage at scale. MCR uses a supervisor model where a single daemon can manage hundreds of forwarding rules concurrently across its worker processes.</li> Head-End Replication:</strong> A key feature of MCR is its ability to perform “fan-out” or “head-end replication,” where a single input stream is replicated to multiple different outputs. socat</code>’s “one-to-one” model cannot do this; it can only forward a stream from one point to another.</li> </ul> Comparative Example: 1-to-1 Relay</h4> Let’s say we want to relay a stream from 239.1.1.1:5001</code> on eth0</code> to 239.10.10.10:6001</code> on eth1</code>.</p> With MCR</strong>, you would ensure the supervisor is running and then add a rule dynamically:</p> # Add a rule to a running MCR instance </span>./control_client add-rule \ </span> --input-interface eth0 \ </span> --input-group 239.1.1.1 \ </span> --input-port 5001 \ </span> --output-interface eth1 \ </span> --output-group 239.10.10.10 \ </span> --output-port 6001 </span></code></pre> With socat</code></strong>, you would launch a dedicated process for this specific task:</p> socat -u \ </span> UDP4-RECV:5001,ip-add-membership=239.1.1.1:eth0 \ </span> UDP4-SEND:239.10.10.10:6001,bind=10.1.1.2 </span></code></pre> (Note: socat</code> requires binding to the specific IP address of the egress interface, here assumed to be 10.1.1.2</code> on eth1</code>.)</em></p> In summary, socat</code> is an excellent tool for simple, static, one-to-one relay tasks. MCR is designed for more complex scenarios requiring high-density, dynamic rule management, and features like head-end replication.</p> Measured Performance</h3> To complement the architectural and operational comparison, I conducted a series of benchmarks. The full performance validation reports are available in the project’s documentation, but here’s a summary of the key findings:</p> Baseline Reliability (Layer 3 Routing):</strong> In a standard Layer 3 routing test (e.g., forwarding between veth</code> pairs), both MCR and socat</code> demonstrated 100% packet delivery with 0% loss</strong> at moderate loads (e.g., 50,000 packets per second). For simple L3 scenarios, both tools are equally reliable.</p> </li> Scalability Under High Load:</strong> Under a sustained load of 400,000 packets per second</strong> in the same Layer 3 topology, MCR maintained excellent performance with a negligible 0.2% packet loss</strong>, whereas socat</code> dropped over 15.3%</strong> of packets. This highlights MCR’s architectural advantage with io_uring</code>’s batched I/O, significantly reducing syscall overhead at high rates.</p> </li> Architectural Suitability for L2 Bridging:</strong> A key use case for MCR is relaying traffic between logically separate Layer 2 domains, such as two different Linux bridges. In this scenario, MCR forwarded traffic with 0% loss, whereas socat</code>, as a Layer 4 tool, is not designed to operate at this level and could not forward the traffic</strong>. This doesn’t represent a flaw in socat</code>, but rather illustrates a foundational difference in capability: MCR is explicitly designed for L2 operations that are outside the scope of standard UDP socket-based tools.</p> </li> Extreme Fan-out:</strong> MCR was successfully validated performing a 1-to-50 head-end replication</strong>, where a single input stream was replicated to 50 unique outputs. This demonstrates MCR’s ability to bypass the kernel’s hard-coded MAXVIFS</code> limit of 32 output interfaces for native multicast forwarding.</p> </li> </ul> These results, gathered in a virtualized test environment, confirm MCR’s capabilities for scenarios demanding high throughput and architectural versatility. A comprehensive analysis of performance on physical hardware is planned for a future article.</p> Why Rust?</h3> Rust was chosen for MCR because its language features align well with the project’s goals. Its focus on memory safety and its ownership model help prevent common classes of bugs, like buffer overflows and use-after-free errors, which are particularly critical in networking applications that deal directly with raw memory buffers and low-level system calls. This allows the development focus to remain on the core logic, with the compiler providing strong guarantees against many potential vulnerabilities. Of course, operating at Layer 2 requires powerful capabilities (CAP_NET_RAW</code>), which brings significant security responsibility; future work will explore strategies for dropping these privileges as soon as they are no longer needed.</p> Conclusion: The Right Tool for the Job</h3> The journey from the “disappearing multicast packet” to MCR highlights a core engineering principle: when you hit a fundamental limitation in one layer of the system, an effective solution is often to move up a layer and build a more specialized tool.</p> MCR is that tool. It attempts to solve the specific, real-world problem of unroutable multicast sources by providing a manageable and high-performance userspace relay. It’s an exploration of how combining a deep understanding of the kernel’s boundaries with modern APIs and a language like Rust can create powerful and reliable solutions to complex networking challenges.</p> This approach of a lightweight, userspace-driven protocol that leverages the existing unicast network is not a new idea. It follows the design philosophy of protocols like SLIM (Self-configuring Lightweight Internet Multicast), which argued that the key to multicast adoption was to “nullify the management complexity” by avoiding multicast-specific infrastructure. MCR can be seen as a modern realization of that vision, implementing a similar architectural pattern with the high-performance, asynchronous tools available to us today.</p> References</h3> Hjálmtýsson, G., Brynjúlfsson, B., & Helgason, Ó. R. (2003). Overcoming Last-Hop/First-Hop Problems in IP Multicast</em>. Lecture Notes in Computer Science.</p> </li> Hjálmtýsson, G., Brynjúlfsson, B., & Helgason, Ó. R. (2004). Self-configuring lightweight Internet multicast protocol specification</em>. IEEE International Conference on Systems, Man and Cybernetics.</p> </li> </ol> The Curious Case of the Disappearing Multicast Packet 2025-11-02T00:00:00+00:00 Many consumer and industrial devices—from home security cameras to the HDMI-over-IP extenders I investigated in a previous post</a>—are designed as simple appliances. They often have hard-coded, unroutable IP addresses (e.g., 192.168.1.100</code>) and expect to live on a simple, isolated network. This becomes a major problem when you need to get their multicast video or data streams from that isolated segment to users on a main LAN. My goal was to solve this with a standard Linux server, creating a simple, high-performance multicast router without a dedicated, expensive hardware box.</p> Can a standard Linux server act as a simple, kernel-native multicast router for devices with hard-coded, unroutable IP addresses? This article chronicles an investigation into combining nftables</code> SNAT with direct control of the Multicast Forwarding Cache (MFC).</p> The DSP Lens on Real-World Events, Part 1: A New Dimension for Your Data 2025-08-09T00:00:00+00:00 The Universal Challenge: From Events to Insight</h3> Every field that deals with streams of events over time shares a common challenge. A factory manager tracking items on a conveyor belt, a data scientist analyzing user clicks, and a network engineer monitoring packets are all trying to turn a series of discrete, often chaotic, events into meaningful insight. The tools may differ, but the fundamental problem is the same: how do we perceive the true nature of a system through the lens of the events it generates?</p> My own journey into this problem began with a network analysis tool, JitterTrap</a>. I was seeing things that shouldn’t exist: a prominent, slow-moving wave in my network throughput graph that my knowledge of the underlying system said couldn’t be real. This “ghost” in the machine forced me to look for answers in an unexpected place: the field of Digital Signal Processing (DSP).</p> This three-part series shares the lessons from that journey. Part 1 introduces the core DSP mental model as a powerful, universal framework. Part 2 lays out a theoretical framework for analysis, and Part 3 applies that framework to perform a rigorous, quantitative analysis of a real-world measurement problem.</p> The DSP Lens on Real-World Events, Part 2: A Framework for Analysis 2025-08-09T00:00:00+00:00 Introduction: The Rosetta Stone</h3> In Part 1, we introduced the “ghost in the machine”—an aliased signal created by the very act of measuring a stream of discrete events. We established that any time-bucketed analysis is a form of filtering. But to truly understand our measurements and build a trustworthy instrument, we need a more formal framework.</p> This article is that framework—a “Rosetta Stone” to connect the different concepts at play. Before we can analyze our system quantitatively in Part 3, we must first understand the precise relationship between our sample rate, the frequencies we want to observe, and the characteristics of the filter we are unknowingly using.</p> The DSP Lens on Real-World Events, Part 3: Building an Honest Instrument 2025-08-09T00:00:00+00:00 Introduction: Applying the Framework</h3> In Part 1, we introduced the DSP lens and the problem of aliasing. In Part 2, we built a detailed theoretical framework, our “Rosetta Stone,” connecting the concepts of sample rates, filters, and critical frequencies.</p> Now, in this final part, we apply that framework to a real-world scenario. We will use the specific parameters of the Jittertrap tool to perform a rigorous quantitative analysis of the system’s imperfections. This will allow us to determine, with numerical confidence, exactly what we can and cannot trust from our measurements, and ultimately, how to build a more honest instrument.</p> A fresh coat of paint for Jittertrap 2025-08-03T00:00:00+00:00 My long-running project, Jittertrap, had been neglected for a long time… Mostly, because of a lack of interest.</p> However, I recently had need for it at my day job, and I dusted it off, wrote down a list of things I need to fix for it to be useful to me in that one particular use case, and decided to fix the things that embarrass me and implement the things that would improve my life.</p> The backend code is mostly fine. There are some tricky things about it, and I’ve been thinking about a rewrite in Rust and eBPF, but it mostly works.</p> The frontend, however, has always been a different story. It’s a web UI, and to be blunt, my interest in the nuances of JavaScript, CSS, and the ever-shifting sands of web frameworks is minimal. It’s necessary, but it’s not the work I enjoy.</p> For years, this meant the UI languished. It was functional, but it was built on an old version of Bootstrap and was riddled with the kind of minor, irritating bugs that I had little motivation to track down. The project stalled, not because the core idea was finished, but because it wasn’t fun, and it wasn’t appreciated. The uninteresting work had become a barrier to the interesting work. I decided that this is my opportunity to see what all the AI coding fuss is about.</p> I started using Gemini as a coding assistant, specifically to tackle the frontend. My goal was to offload the tedious parts - the parts I had no desire to master myself.</p> The results have been surprisingly good. Over the past month, I’ve managed to clear a huge backlog of UI debt. Here’s a quick rundown of the progress:</p> Modernized Frameworks:</strong> The entire UI has been migrated from Bootstrap 3 to Bootstrap 4. This was a significant undertaking that I’d been putting off for ages, and one milestone on the journey of removing the jQuery dependency. Because apparently all browsers are good now and jQuery is a liability.</p> </li> Richer Charting Features:</strong> The charts have become more powerful for analysis. The “Top Talkers” chart now aggregates smaller, less relevant flows into a single “Other” category, making the visualization much cleaner. I also added a switch to toggle the Y-axis between logarithmic and linear scales, allowing for a more nuanced view of the data.</p> </li> Improved IPv6 Support:</strong> The backend now properly handles IPv6 traffic. I fixed several bugs in the packet decoding logic for IPv6 addresses and extension headers, which is critical for monitoring modern networks.</p> </li> Security Hardening:</strong> I took the time to harden the application by fixing several security vulnerabilities, including a potential XSS issue in the WebSocket handler and a memory leak in the backend C code.</p> </li> Code Modernization:</strong> The JavaScript codebase has been brought into the modern era. All the old var</code> declarations are gone, replaced by let</code> and const</code>, and I’ve been able to systematically replace old jQuery patterns and callbacks with native JS and arrow functions.</p> </li> Performance Overhaul:</strong> The d3.js charts, which are central to Jittertrap, have been substantially optimized. We moved the “Top Talkers” chart to a <canvas></code> element, which dramatically improved rendering performance. The old setInterval</code> loop was replaced with requestAnimationFrame</code> for smoother, more efficient updates.</p> </li> Responsive Layout:</strong> The UI is now properly responsive. The main container is fluid, and on wide screens, the throughput and top talkers charts now sit side-by-side, which is a much better use of the space.</p> </li> Bug Fixes:</strong> And of course, a whole host of nagging bugs have been squashed, from layout issues with the chart legends to incorrect resizing behavior.</p> </li> </ul> The interesting part of this process is how it has changed my relationship with my own project. I can now focus on the What and not get bogged down in the How of web development. I describe the desired outcome—“make the legend responsive,” “fix the grid resizing”—and the AI generates the code. It’s a powerful new tool, not unlike a compiler or a static analyzer, that handles a specific domain of work for me.</p> It’s a more efficient and, frankly, more enjoyable way to work on a passion project. The AI isn’t driving the project, but it’s clearing the road so I can.</p> Rapido - rapidly creating test VMs for driver development 2023-11-25T00:00:00+00:00 Thanks to rapido</a>, it’s become much simpler to test Linux device drivers for real PCIe devices in VMs.</p> The advantages of this approach are:</p> the host is protected from memory corruption errors caused by buggy kernel drivers</li> the PCI peripheral can be physically installed in a multi-use machine, reducing hardware & lab requirements</li> debugging info is easily available</li> the development cycle is short and simple - rapid even :)</li> </ul> PMTU weirdness 2023-03-11T00:00:00+00:00 On a good day, my day job involves building networking tools in python. Too many python networking tools look like shell scripts, spawning subprocesses for basic tools like ‘ping’ or ‘ip’ - often resulting in a fragile mess due to poor, or inconsistent error handling.</p> I was quite excited to find icmplib</a>. It provides a much simpler, less fragile way to do things like reachability tests, RTT measurements, path discovery and path MTU discovery in python code. Hopefully it finds its way into Fedora soon!</p> Armed with icmplib, I went on a journey of discovery to develop my understanding of Path MTU. I specifically wanted to understand the differences between IPv4 and IPv6, and the effect of VETH</a>, VLAN and Bridge virtual devices.</p> Status update 2017-12-03T00:00:00+00:00 I’ll be speaking at linux.conf.au</a> 2018 in Sydney.</p> Jittertrap</a> testing infrastructure is slowly improving. The idea is to include known test data so that the front-end rendering can be verified and refactored. I expect a sprint over the holidays and some new bugs for lca. :)</p> Clean up of the tn40xx driver</a> is continuing - hopefully towards mainline inclusion. The driver project gets more visitors and clones than Jittertrap, which I find interesting, unexpected and somewhat sobering.</p> Listing Network interface, MAC address and PCI bus/device/function 2016-03-28T00:00:00+00:00 New hardware for BufferUpr</a> has arrived! I’m working on adding Wifi capability into the product and with new hardware comes new challenges.</p> Challenge:</strong> Find a simple way to list all the network interfaces in the system, including the PCIe slot and MAC address for each.</p> Solution:</strong></p> find /sys/devices/pci0000* \ </span>-wholename "net//address" \ </span>-printf "%h/%f " \ </span>-exec cat '{}' \; </span></code></pre> This depends on a proper find</code>, of course, so Busybox won’t do.</p> Example output:</strong></p> /sys/devices/pci0000:00/0000:00:03.0/0000:03:00.0/net/p3p0/address 90:e2:ba:7f:97:00 </span>/sys/devices/pci0000:00/0000:00:03.0/0000:03:00.1/net/p3p1/address 90:e2:ba:7f:97:01 </span>/sys/devices/pci0000:00/0000:00:03.0/0000:03:00.2/net/p3p2/address 90:e2:ba:7f:97:02 </span>/sys/devices/pci0000:00/0000:00:03.0/0000:03:00.3/net/p3p3/address 90:e2:ba:7f:97:03 </span>/sys/devices/pci0000:00/0000:00:19.0/net/em1/address 88:88:88:88:87:88 </span></code></pre> Notice from the path to the address files that some of the interfaces on this system are named according to their PCI BDF (bus/device/function) identifiers. For example, the p3p2</code> interface is bus 3, device 0, function 2.</p> From these BDF numbers, we can guess that there is a 4-port card in PCIe slot 3. However, it may not be easy to tell which physical connector on the motherboard maps to slot 3. It might even involve screwdrivers and dust… shudder</em>.</p> There are some more examples and a better explanation of device naming rules over there</a>.</p> If the disappearance of eth0</code> is news to you and you’re frustrated that nobody asked you whether you wanted interface naming fixed, you might want to read about why this change was made</a>.</p> The Xen wiki</a> has a good page on BDF numbers and Wikipedia has more detail</a>.</p> Installing SciPy on Fedora 23 2016-01-11T00:00:00+00:00 Fedora 23</a> (the current release at the time of writing) ships an outdated version of SciPy</a> that doesn’t include the spectrogram function. Installing the latest Scipy was kind-of a pain, so I thought I’d record some instructions for future-me and share it with you.</p> These instructions install dependencies, set up a python virtualenv container and then install the latest matplotlib</a>, SciPy</a> and Numpy</a> on Fedora 23</a></p> Dependencies:</p> python-virtualenv</p> </li> libpng-devel freetype-devel pygobject2-devel</p> </li> redhat-rpm-config</p> </li> gcc-gfortran gcc-c++</p> </li> atlas-devel lapack-devel</p> </li> </ul> Install these from Fedora repositories:</p> sudo dnf install python-virtualenv libpng-devel freetype-devel pygobject2-devel redhat-rpm-config gcc-gfortran gcc-c++ atlas-devel lapack-devel </span></code></pre> And then the python packages:</p> virtualenv env </span>source env/bin/activate </span> </span>pip install -U pip && </span>pip install numpy && </span>pip install matplotlib && </span>pip install scipy </span></code></pre> Installing numpy, matplotlib and scipy simultaneously fails. Thanks, pip.</p> JitterTrap: Baby steps in DSP 2015-12-01T00:00:00+00:00 This is the story of my first expensive lesson in Digital Signal Processing. It is about JitterTrap</a>, the free software that powers BufferUpr</a>.</p> The premise of BufferUpr is to combine commodity hardware with open source software to create a product that can measure data stream delays of 1-100 milliseconds. This is the range of delay that most multimedia developers and consumers are concerned with.</p> To measure the delay, we count the packets and bytes as they fly past and look for changes in throughput. My naive logic was that if we look at consecutive intervals of 1ms, we’ll be able to measure the throughput over that interval, plot it and show any variation in throughput.</p> This seemed like a very simple task, except that I had no previous experience in Digital Signal Processing and didn’t see why that would be relevant.</p> Lightweight containers in Fedora using systemd 2015-04-02T00:00:00+00:00 It was the year of 2015 and people were still developing new applications in PHP… but for those who could no longer accept the idea of installing a system-wide LAMP stack, there was a new-old fassionable thing: Containers!</p> This is a quick howto for creating a throw-away container for messing around with PHP apps on Fedora. Most of it is taken from the examples at the bottom of ‘man 1 systemd-nspawn’</p> Create the directory for the container filesystem:</p> [user@host]$ mkdir web-dev-container </span></code></pre> Install a minimal Fedora image (including apache and php) into the new container root:</p> [user@host]$ sudo yum -y --releasever=21 --nogpg --installroot=$(pwd)/web-dev-container --disablerepo='' --enablerepo=fedora install systemd passwd yum fedora-release less vim iproute httpd php </span></code></pre> Update the SELinux context:</p> [user@host]$ sudo semanage fcontext -a -t svirt_sandbox_file_t "web-dev-container(/.)?" </span>[user@host]$ sudo restorecon -R web-dev-container </span></code></pre> Spawn the new container:</p> [user@host]$ sudo systemd-nspawn -D web-dev-container </span></code></pre> In the new container, change the root password:</p> [root@container]# passwd </span>[root@container]# logout </span></code></pre> Boot the new container:</p> [user@host]$ sudo systemd-nspawn -bD web-dev-container </span></code></pre> Enable and start the web server and check that it’s running:</p> [root@container]# systemctl enable httpd.service </span>[root@container]# systemctl start httpd.service </span>[root@container]# ss -lt </span></code></pre> Check http://localhost in a web browser.</p> Add an index.html file:</p> [root@container]# echo "Hello World" >> /var/www/html/index.html </span>[root@container]# echo "<?php echo \"Hello PHP World\" ?>" > /var/www/html/index.php </span></code></pre> Refresh http://localhost</p> Check http://localhost/index.php</p> You can now logout of the container (it will keep running).</p> Change the ownership and hack away!</p> [user@host]$ sudo chown -R user.user web-dev-container/var/www/html </span>[user@host]$ vim web-dev-container/var/www/html/index.php </span></code></pre> Login to the container, if needed:</p> [user@host]$ sudo machinectl login web-dev-container </span></code></pre> Exit the container by pressing ^] three times in one second.</p> Using policy routing to loopback over external interfaces 2015-04-01T00:00:00+00:00 Sometimes there is good reason to talk to yourself. You might be doing a sound check, for example.</p> Likewise, it can be useful to route IP packets between two interfaces on the same machine using an external path. One reason to do this is to test other network devices like routers or switches.</p> For many years this has been much trickier to do than it might seem, until Patrick McHardy’s introduction</a> of the ‘accept_local’ setting</a> in December 2009.</p> A little more than a year after that, Kirill Smelkov replied with a functional example script</a> that shows how to achieve full loopback bi-directional IP traffic over physical interfaces. See also this GitHub gist</a> with a similar script.</p> Even so, there are numerous red herring posts on StackOverflow and elsewhere that don’t really help: 1</a> 2</a> 3</a> 4</a> 5</a></p> So, how does it work? Policy Routing, that’s how!</p> Basic preparation:</h2> If you have not already done so, step 0 is to tell NetworkManager to leave the interfaces alone!</em></p> Select the interface names and addresses:</p> IFACE_A=p3p1 </span>IFACE_B=p3p4 </span> </span>IFACE_A_IP="10.10.3.1" </span>IFACE_B_IP="10.10.3.4" </span></code></pre> Assign addresses:</p> ip addr replace $IFACE_A_IP dev $IFACE_A </span>ip addr replace $IFACE_B_IP dev $IFACE_B </span></code></pre> Ensure the links are up:</p> ip link change $IFACE_A up </span>ip link change $IFACE_B up </span></code></pre> Policy Routing</h2> Enable receiving of local source addresses on external ifaces (see Documentation/networking/ip-sysctl.txt in the kernel source tree).</p> sysctl -w net.ipv4.conf.$IFACE_A.accept_local=1 </span>sysctl -w net.ipv4.conf.$IFACE_B.accept_local=1 </span></code></pre> Decrease priority of the ‘local’ routing table (change preference from 0 to 1000):</p> ip rule del pref 0 lookup local </span>ip rule add pref 1000 lookup local </span></code></pre> Prevent routing loop by using preference rules to route incoming packets using ‘local’ table.</p> ip rule add pref 100 iif $IFACE_A lookup local </span>ip rule add pref 101 iif $IFACE_B lookup local </span></code></pre> Use a routing table for each interface, that is, destination policy routing:</p> ip route add default dev $IFACE_A table 200 </span>ip route add default dev $IFACE_B table 201 </span></code></pre> Note: There’s no need to create the tables first, adding a route entry is sufficient.</em></p> Add preference rules to route each destination using a coresponding table:</p> ip rule add pref 200 to ${IFACE_B_IP} lookup 200 # IFACE_A to IFACE_B </span>ip rule add pref 201 to ${IFACE_A_IP} lookup 201 </span></code></pre> Note: Do not be confused by the numbers. ‘pref 200’ refers to the preference/priority of the rule, whereas ‘lookup 200’ refers to table number 200. The kernel only cares about the number, but ‘ip’ will use the optional number-to-name mapping defined in /etc/iproute2/rt_tables.</em></p> Finally, flush the route cache to ensure the tables are reread:</p> ip route flush cache </span></code></pre> Check the routing rules</em>:</p> ip rule show </span>100: from all iif p3p1 lookup local </span>101: from all iif p3p4 lookup local </span>200: from all to 10.10.3.4 lookup 200 </span>201: from all to 10.10.3.1 lookup 201 </span>1000: from all lookup local </span>32766: from all lookup main </span>32767: from all lookup default </span></code></pre> Test that the traffic is routed via the external interface using ‘tcpdump’ and ‘ping’:</p> tcpdump:</p> # tcpdump -nlq -i p3p1 -c 6 </span>tcpdump: verbose output suppressed, use -v or -vv for full protocol decode </span>listening on p3p1, link-type EN10MB (Ethernet), capture size 262144 bytes </span>15:51:00.965010 IP 10.10.3.1 > 10.10.3.4: ICMP echo request, id 27352, seq 1, length 64 </span>15:51:00.965217 IP 10.10.3.4 > 10.10.3.1: ICMP echo reply, id 27352, seq 1, length 64 </span>15:51:01.964694 IP 10.10.3.1 > 10.10.3.4: ICMP echo request, id 27352, seq 2, length 64 </span>15:51:01.964825 IP 10.10.3.4 > 10.10.3.1: ICMP echo reply, id 27352, seq 2, length 64 </span>15:51:02.964785 IP 10.10.3.1 > 10.10.3.4: ICMP echo request, id 27352, seq 3, length 64 </span>15:51:02.964987 IP 10.10.3.4 > 10.10.3.1: ICMP echo reply, id 27352, seq 3, length 64 </span>6 packets captured </span>6 packets received by filter </span>0 packets dropped by kernel </span></code></pre> ping:</p> $ ping -c 3 10.10.3.4 </span>PING 10.10.3.4 (10.10.3.4) 56(84) bytes of data. </span>64 bytes from 10.10.3.4: icmp_seq=1 ttl=64 time=0.247 ms </span>64 bytes from 10.10.3.4: icmp_seq=2 ttl=64 time=0.183 ms </span>64 bytes from 10.10.3.4: icmp_seq=3 ttl=64 time=0.243 ms </span> </span>--- 10.10.3.4 ping statistics --- </span>3 packets transmitted, 3 received, 0% packet loss, time 1999ms </span>rtt min/avg/max/mdev = 0.183/0.224/0.247/0.031 ms </span></code></pre> Thats it! There’s no need to mess around with ARP or reverse path filtering!</p> DSCP vs Linux socket priorities 2014-09-27T00:00:00+00:00 I received some encouraging comments on G+</a> from Jesper Dangaard Brouer about my previous post</a> on DSCP, Linux and VLAN priorities. Those comments and the work linked to (here</a>) points to a few long-standing (but minor) issues with the way DSCP priorities are handled in Linux.</p> Some DSCP values, like Expedited Forwarding, are not currently (3.17 and earlier) handled correctly.</li> Linux Priorities, defined in include/uapi/linux/pkt_sched.h</em>, are not documented particularly well, but forms part of the stable interface with userspace. Working with traffic classification (tc), queuing disciplines (qdisc) or VLANs requires at least a basic understanding of Linux socket priorities.</li> </ol> Looking into DSCP and IEEE 802.1p (VLAN priorities). 2014-09-17T00:00:00+00:00 I recently discovered a flaw in the VLAN implementation I did at work. It seemed that the normal TCP traffic had the correct VLAN priorities applied, but audio streaming UDP traffic did not.</p> This was due to DSCP being applied to the streaming audio and the fact that the VLAN device’s egress-qos-map was incorrect.</p> I had assumed, incorrectly, that VLAN priorities are applied to all traffic as long as we’re not using fancy queuing disciplines (qdiscs). After all, 802.1Q is strictly a layer 2 thing.</p> Linux Network Configuration 2014-08-12T00:00:00+00:00 This concerns the proliferation of netlink libraries and a lack of direction and documentation.</p> Background:</h1> I’ve configured a router with netem</a> (see Bandwidth Throttling with NetEM Network Emulation</a> and the tc-netem man page</a>)</em> to test Tieline</a> devices under various delay and loss network conditions.</p> It’s not really feasible for the tester to use ‘tc’ on the command line for the various tests, so my simple requirement is to build a little web interface. It’s really easy to hack something together quickly (parsing the output of iproute2), but I’ve been developing a bit of an allergy for this kind of hacking things together.</p> I’d like to use a scripting language (Python or Perl, not C) for the webby stuff, so I started looking at what modules these languages provide for netlink, which seems like a better idea than parsing output of “tc” and “ip”. I couldn’t find anything that plays well with netem, but the python bindings for libnl seems to be closest to what I need.</p> This led me to look at the larger landscape for userspace netlink libraries…</p> Existing Netlink Libraries:</h1> There are a few different userspace netlink “libraries”, with somewhat different goals. The libnl documentation</a> makes the distinction between the different subsystems clear in this diagram:</p> </p> libnl</a>:</p> seems to be aimed at general use,</li> includes Python bindings,</li> used by NetworkManager, libvirt, libpcap, etc.</li> decent documentation.</li> </ul> However, libnl is not the only netlink library in use.</p> libmnl</a>:</p> available for general use.</li> used by nftables</li> </ul> libnetlink</a>:</p> internal to iproute2, not for general use.</li> </ul> sd-netlink</a>:</p> internal to systemd-networkd, not for general use.</li> </ul> Network Configuration Managers</h1> Lets look at this from another angle: what do the typical network managers / configuration applications use?</p> NetworkManager</a>:</p> mature</li> maintained</li> uses libnl for direct netlink comms</li> C</li> provides DBus interface</li> </ul> ConnMan</a>:</p> maturity / maintenance concerns</li> C</li> </ul> Wicd</a>:</p> unmaintained</li> Python</li> calls external tools like ifconfig</li> provides DBus interface</li> </ul> netctl</a> (Arch package</a>):</p> maintained</li> Bash</li> calls external tools like iproute2, etc.</li> </ul> netifd (OpenWRT)</a>:</p> active / maintained</li> C</li> </ul> systemd-networkd</a>:</p> active / maintained</li> C</li> uses internal sd-rtnl</li> </ul> vyconfd</a>:</p> maintenance concerns after vyatta sold to brocade.</li> Python</li> external iproute2, etc. tools?</li> </ul> LNST</a>:</p> Python</li> external tools or NetworkManager via DBus</li> </ul> netcf</a>:</p> tries to bridge distro-specific network config files with its own and provides a C library.</li> does not seem to modify the state directly.</li> seems to be used by virsh</li> </ul> ncfg (slides no longer available)</em>:</p> does it, or will it matter?</li> </ul> I also fell down the rabbit warren of looking what the various SDN and router OS projects are doing - and I’m still recovering from the experience - so I’ll write about that when I have a better understanding of it.</em></p> Back to solving the problem:</h1> Remember that the initial problem was how to configure Netem.</p> Option 1: the quick and easy hack. This is what I ended up doing at work, for now, because it’s that kind of place.</em></p> exec tc</li> </ul> Option 2:</em></p> Use a network manager with a dbus interface. Both NetworkManager and Wicd can do this.</li> </ul> Alternative 2:</em></p> Use libnl’s python bindings.</li> </ul> Conclusion</h1> Basically, many projects for network configuration have not progressed past shell scripting and iproute2 - some kind of lowest common denominator - and make a large effort to tie individual tools together into something with a semi-coherent interface. I believe there is a need for a good netlink library for one of the popular scripting languages, so that we can move past the old crufty shell scripts and parsing output of stand-alone tools.</p> On the other hand, many of these basic tools have implemented their own abstractions for using the netlink kernel interface. A Grand Unified Netlink Userspace certainly seems like it would be an improvement over the status quo, but each netlink library probably exists for good reason and a merger seems unlikely.</p> libnl seems like the natural choice for mere mortals. I’ll post some examples as I learn more about it.</p> See also</h1> Video: Can Linux network configuration suck less?</del> (link dead)</em></p> Slides</a> OpenWRT comparison of network managers</a></p> Plain text email attachments. 2014-07-24T00:00:00+00:00 After finally completing task 01 of the Eudyptula Challenge</a>, I’d like to share a few things I’ve learned, without divulging any crucial details about the task or solution.</p> tl;dr</strong> Pay attention.</p> Patience…</h1> Maybe it’s just the timezone, but the turn-around time for a response to a submission meant that I could only send one message per day. That fits with my experience on the netdev and iommu mailing lists.</p> Paying attention to detail.</h1> Having a longer turn-around time really discourages small, quick changes and “is it good enough now?” messages. This is a Good Thing. It makes you think about what you’re doing. It reinforces the concept that it’s good to take as long as you need to do things properly and double check your work, because submitting rubbish will waste a lot of your own time, not just reviewer time.</p> Pay even more attention to detail, or, email clients suck.</h1> It took me four attempts to complete the first task. It’s embarrassing.</p> The task involves sending plain text attachments, as opposed to base64 encoded attachments. To achieve this, I first went through some effort to get this domain up and running and get back some of the control over email that gmail doesn’t provide. Then, I thought I’d use Alpine as a safe bet for an email client, but it turns out Alpine always</em> uses base64 encoding for attachments. Sigh…</p> So, for the first attempt I submitted a patch instead of a simple attachment. I thought I could cheat, just a little, and avoid a problem with base64 encoded attachments. Unfortunately the script wasn’t expecting it and by submitting an unexpected result I wasted my time as well as my anonymous teacher’s time.</p> So, I switched to Thunderbird.</p> The second attempt was rejected, because I didn’t read the requirement properly. I won’t go into more details, but doh!</p> The third attempt… this is where the email hiccup came in. For the second attempt I carefully tested that Thunderbird was creating plain text attachments, as follows:</p> --------------040002060807090301030305 </span>Content-Type: text/plain; charset=UTF-8; </span> name="Makefile" </span>Content-Transfer-Encoding: 7bit </span>Content-Disposition: attachment; </span> filename="Makefile" </span></code></pre> It turns out that the character encoding of the email influences what Thunderbird does with the attachment. Of course I didn’t know that and I didn’t think to check it. After all, the previous message had the correct attachment encoding. So, for the record, don’t use UTF-8 encoding for the message body, or your attachment will be base64 encoded.</p> Conclusion</h1> The Eudyptula Challenge requires careful attention to detail, more than the standard expected of me in my day job. It encourages me to pursue a higher standard of work.</p> Self-hosting email again. 2014-07-18T00:00:00+00:00 It’s been a few years since I set up or maintained a public mail server. It took much longer than expected to get everything working again, but here we are!</p> acooks @ rationali.st</p> Let’s see how long it takes before I need to tweak spam filters again.</p> Hello World 2014-07-14T00:00:00+00:00 What’s this? This little bit of interweb detritus was created for a few reasons:</p> Transparency. Employers want to know what you’ve done before.</li> To learn about “the cloud”. I’ve done plenty</em> of sysadmin work in the past, but that was all on my own bare metal. This helps to keep my sysadmin skills current, since my day job doesn’t involve that at the moment and I might need those skills again.</li> The first task of the Eudyptula Challenge</a> suggests not using Gmail. This blog is another side-effect of getting my own infrastructure up again.</li> To document some projects I’ve been working on.</li> </ol> About me 2014-07-14T00:00:00+00:00 Hi!</p> I’m a Software Engineer and have been a Linux and networking enthusiast since the late nineties.</p> Born and raised in Pretoria, South Africa. I moved to Perth, Western Australia in December 2011, before settling in Brisbane in 2016.</p> In 2015 I founded a startup producing test and measurement tools for real-time IP communications applications. As a business, it failed to gain any customers, but the Jittertrap</a> software is freely available.</p> In 2017 I worked on the tn40xx Ethernet driver</a> (as a hobby) with the hope of upstreaming it. That ended after the chip vendor went out of business and my employement and family changed.</p> Since 2018 I’ve been building software for software defined networks and communication systems</a>.</p> I love working in the low level details, though recently I’ve been spending more time on collaborative work like sketching designs on white boards or Visio and finding less time for independent technical contributions.</p> A couple of trivial patches of mine made it into the kernel</a> and I’m always talking about how I want to be a kernel developer when I grow up.</p> </p> That’s a Red Hat Linux 5</a> (Not RHEL) user’s guide circa 1998.</p> </p> TCP/IP Unleashed from 1996. Seems I started young and haven’t moved on.</p>

rationali.st

When Packets Can't Wait: Comparing Protocols for Delay-Sensitive Data

2026-01-01T00:00:00+00:00

In Diagnosing Video Stuttering Over TCP</a>, we built a diagnostic framework—identifying zero-window events (receiver overwhelmed) versus retransmits (network problems). In The TCP Jitter Cliff</a>, we discovered that throughput collapses unpredictably when jitter exceeds ~20% of RTT, and the chaos zone makes diagnosis treacherous.

The conclusion from both posts is clear: TCP is inappropriate for delay-sensitive streaming. Its guaranteed-delivery model creates unbounded latency during loss recovery. When a packet is lost, TCP will wait—potentially forever—rather than skip ahead. For a live video frame or audio sample, arriving late is the same as not arriving at all.

But “don’t use TCP” isn’t a complete answer. What should you use? The protocol landscape for delay-sensitive data is vast—spanning media streaming, industrial automation, robotics, financial messaging, and IoT. Each protocol answers the fundamental question differently.

Protocol Reference: Transport for Data with Deadlines

2026-01-01T00:00:00+00:00

This is a quick-reference list of protocols for applications where data has a deadline, and late delivery is failure—whether that deadline is 10μs (servo loop), 20ms (audio buffer), or 300ms (video call).
For taxonomy, analysis, and context, see When Packets Can’t Wait</a>. This page is just the inventory—a living reference that grows as protocols become relevant to JitterTrap</a> development.

The Jitter Cliff: When TCP Goes Chaotic

2025-12-28T00:00:00+00:00

In Part 1</a>, we used “video over TCP” as a stress test for TCP’s behavior—examining how zero-window events, retransmits, and the masking effect reveal what’s happening inside a struggling connection.
But during those experiments, I discovered TCP throughput degraded rapidly as jitter worsened. While I knew that packet loss would destroy TCP throughput, I hadn’t quite expected the jitter-induced cliff.
At a certain jitter threshold, throughput collapses so severely that measurements become unreliable. Single tests can vary by over 100%. This “chaos zone” makes diagnosis treacherous: the same network conditions can produce wildly different results depending on when you measure.
This post explores TCP’s behavior under jitter and loss, comparing CUBIC and BBR. It’s common knowledge that TCP is inappropriate for delay-sensitive streaming data, and this post will try to demonstrate how and why.

Diagnosing Video Stuttering Over TCP: A JitterTrap Investigation

2025-12-25T00:00:00+00:00

Your security camera feed stutters. Your video call over the corporate VPN freezes. The question isn’t whether something is wrong—that’s obvious. The question is what is wrong, because the fix depends entirely on the diagnosis.
Is the problem the sender, the network, or the receiver? These require fundamentally different interventions. Telling someone to “upgrade their internet connection” when the real issue is their overloaded NVR is worse than useless—it wastes time and money while the actual problem persists.
Sender-side problems—where the source isn’t transmitting at the expected rate—are straightforward to detect: compare actual throughput to expected throughput. The harder question is distinguishing network problems from receiver problems when data is being sent. TCP’s built-in feedback mechanisms give us the answer.
UDP is the natural transport for real-time video—it tolerates loss gracefully and avoids head-of-line blocking. But video often ends up traveling over TCP whether we like it or not. VPN tunnels may encapsulate everything in TCP. Security cameras fall back to RTSP interleaved mode (RTP-over-TCP) when UDP is blocked. Some equipment simply doesn’t offer a choice.
The research question driving this investigation: Can we identify reliable TCP metrics that distinguish network problems from receiver problems?
Through controlled experiments, I found the answer is yes—with important caveats. This post builds a complete diagnostic framework covering all three problem types, with the experiments focused on the harder network-vs-receiver distinction. Part 2 will explore what happens when TCP goes chaotic.

RFC 8362: Extended LSA Support for FRR ospf6d

2025-12-24T00:00:00+00:00

This is a story about six months of work that didn’t ship, and what I’m still figuring out about why.
The surface story is technical: I implemented RFC 8362 Extended LSA support for FRR’s ospf6d. The code works. The tests pass. It wasn’t accepted upstream.
The deeper story is about what happens when you bring architectural instincts from one context into another, and they don’t translate. When your experience says one thing and the environment says another. When you’re left wondering whether your expertise is an asset or a liability in unfamiliar territory.
Why OSPFv3 Needed Extended LSAs</h1>
OSPFv3’s original extensibility mechanism was defining new LSA types—and this worked. TLV-based LSA types were added for Traffic Engineering (RFC 5329</a>) and other features. But for the core LSA types (Router, Network, Inter-Area-Prefix, etc.), the format remained fixed C structs mapped directly to wire format.
This has limitations:

64KB size limit — Large topologies with many prefixes can exceed this</li>
No sub-structure in core LSAs — Metadata like Segment Routing SIDs can’t be cleanly attached to prefixes or adjacencies</li> </ul> RFC 8362</a> addresses this with Extended LSA versions of the core types, using TLV encoding:
0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Type | Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Value... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ </code></pre> E-LSA is foundational for:

Segment Routing (RFC 8666</a>, RFC 9513</a>) — SID/Label sub-TLVs attach to prefixes and adjacencies</li>
Flex-Algorithm (RFC 9350</a>) — Algorithm-specific metrics and constraints via sub-TLVs</li>
Multi-Topology Routing (draft-ietf-ospf-mt-ospfv3</a>) — Multiple independent topologies over shared infrastructure</li> </ul>
If you’re deploying OSPFv3 with Segment Routing, you need E-LSA support.
Two Wire Formats, One Codebase</h1>
Here’s where it gets interesting—architecturally.
FRR’s ospf6d operates directly on wire-format structs. Algorithms cast the LSA body to the appropriate C struct and access fields directly. This is efficient: no parsing overhead, no intermediate representation, no allocation churn. For a single wire format, it’s a reasonable design.
But RFC 8362 introduces a second wire format for the same semantic data. Now you have two representations of “inter-area prefix”: the legacy fixed struct and the TLV-encoded E-LSA version. Every algorithm that touches LSA contents—SPF calculation, ABR summarization, ASBR external routes, flooding, show commands—needs to handle both.
The obvious approaches:
Duplicate everything. Write parallel code paths for legacy and E-LSA. Every function that processes Inter-Area-Prefix-LSA gets a sibling that processes E-Inter-Area-Prefix-LSA. This keeps the code style consistent but roughly doubles the surface area. Every bug fix, every enhancement, applied twice. Semantic drift over time as the paths evolve independently.
Abstract over the difference. Introduce an iterator or handler pattern that walks LSA contents uniformly, with format-specific parsing isolated behind it. Algorithms see one interface regardless of wire format. Less duplication, but new patterns the codebase doesn’t use elsewhere.
My instincts, shaped by years of systems work in other contexts, said: abstract. Don’t duplicate business logic. Isolate the format-specific stuff.
But it wasn’t just about this feature. The duplication approach would work, but it would leave the codebase harder to maintain than I found it. The next person adding TLV-based functionality faces the same problem, plus more code to navigate. The abstraction, if accepted, makes future work easier. Leave the place better than you found it—that’s the instinct.
There’s also consistency to consider. ospf6d already has TLV-based LSAs—Grace-LSA for graceful restart uses the same TLV header structure. The iterator pattern aligns E-LSA handling with existing TLV-based functionality rather than introducing something foreign.
Iterator Pattern Across 9 LSA Types</h1> We implemented all 9 E-LSA types from RFC 8362, using an iterator pattern. Commit b919740</a> introduces the mechanism—the foreach_lsdesc</code> iterator and handler registration. Commit 589795c</a> shows it applied to existing code, replacing direct struct access with handler callbacks.
The same iterator walks legacy LSA descriptors or E-LSA TLVs depending on LSA type. Format-specific parsing stays in handlers; algorithms see a uniform interface.
The scope of the work:
Metric</th> Value</th></tr></thead>

Commits</td> 97</td></tr>
Development time</td> 6.5 months</td></tr>
Lines added</td> 27,293</td></tr>
Lines removed</td> 1,786</td></tr>
Net change</td> +25,507 lines</td></tr>
Files modified</td> 277</td></tr>
Test code</td> 3,448 lines</td></tr>
Test topologies</td> 7</td></tr> </tbody></table>
The tests cover multi-area flooding, mixed legacy/E-LSA interoperability, E-LSA-only networks, broadcast DR/BDR election, and graceful restart. The code works. The approach handles the complexity.
“Not How Things Are Done Here”</h1>
The implementation went through three PR iterations:

PR #16199</a> — Initial approach</li>
PR #16532</a> — Revised approach</li>
PR #17343</a> — Final iteration</li> </ul>
The feedback: callback-based iteration is wrong for this codebase. Callbacks add indirection. They make code harder to follow, harder to debug. This isn’t how things are done here. The alternative: separate code paths for legacy and E-LSA.
I understood the concern. I’ve worked in codebases where callbacks created impenetrable spaghetti. The objection wasn’t unreasonable.
But I also saw the duplication path leading somewhere bad—a lot of nearly-identical code to maintain in sync forever. I asked for more specific guidance. What would an acceptable middle ground look like? The answer was conceptual rather than concrete: not callbacks, but no specific alternative I could implement.
I tried variations—inlining some callbacks, reducing indirection, restructuring the handler registration. None landed. Eventually the discussion ended without resolution.
Was I Wrong, or Just Unwelcome?</h1>
Here’s what I’m still sitting with.
The code works. The tests pass. The iterator pattern handles both formats correctly across all 9 LSA types, SPF, ABR, ASBR, flooding, and graceful restart. We had confidence in the technical approach because we had working software proving it.
What I don’t know is whether my instincts were right for this context.
Maybe the duplication approach would have been fine—maybe the maintenance burden I imagined wouldn’t materialize, or would be acceptable to people who know this codebase better than I do. Maybe there’s a third approach I didn’t see because I was anchored on the patterns I knew. Maybe my years of “abstraction over duplication” instincts are a bias, not wisdom, in environments where those patterns aren’t established.
Or maybe I was right, and I just couldn’t communicate it effectively. Or maybe there’s no clean answer and the feature is just genuinely hard to add to this architecture. I don’t know.
What I do know is that experience from one context doesn’t transfer automatically to another. The instincts that made me effective elsewhere made me confident in an approach that wasn’t accepted here. Whether that’s because I was wrong, or because I failed to build consensus, or because the environment has different values—I genuinely can’t tell.
I’ve written a detailed analysis</a> comparing approaches, trying to work through the trade-offs. I still don’t have a clear answer.
A Working Branch, An Open Question</h1>
The branch</a> works. It passes tests. It’s a complete reference implementation of RFC 8362 for ospf6d, if anyone wants it.
If you need E-LSA support in production today, Holo Routing</a> is a Rust-based routing suite with RFC 8362 support and active development. For FRR, the feature remains unimplemented.
This is one of those experiences that shaped me. I spent months pushing against a door that wasn’t going to open—not because the work was wrong, but because it wasn’t wanted in that form, in that place, at that time. The conclusion I came to: put your efforts where they’re welcomed, or go your own way.
Acknowledgments</h1>
Thanks to Jake Lodge, who found and fixed countless bugs in this code. This implementation wouldn’t exist without his work and support.

Linux Network Configuration: A Decade Later

2025-12-22T00:00:00+00:00

In 2014 I wrote about the state of Linux network configuration</a>, lamenting the proliferation of netlink libraries and how most projects hadn’t progressed past shell scripting and iproute2. I concluded that “there is a need for a good netlink library for one of the popular scripting languages.”
A decade later, that library exists. More importantly, the ecosystem has matured enough that every major language has a credible netlink option - and production systems are using them.
To compare them, I’ll use the same example throughout: create a bridge, a network namespace, and a veth pair connecting them.

MCR: A High-Performance Userspace Multicast Relay

2025-11-18T00:00:00+00:00

In my previous post</a>, we hit a wall with the kernel’s multicast forwarding logic. The core issue was the RPF (Reverse Path Forwarding) check failure for sources on unroutable networks. While digging deeper into this, I found it’s a well-documented challenge in multicast literature, often called the “last-hop/first-hop problem”. It describes the difficulty of bridging the gap between an isolated multicast source and the main network.
The investigation concluded that a userspace relay was the right approach. This post introduces MCR (Multicast Relay) (https://github.com/acooks/mcr</a>), a specialized userspace relay built in Rust, as a modern, high-performance implementation of that architectural pattern. The project was built by myself, with the help of AI models Gemini and Claude. Its goal is to provide a robust solution to the first-hop problem where throughput and dynamic control are critical.
Why Userspace? A Necessary Trade-off</h3>
Before diving into the architecture, it’s important to address a fundamental question: why move packet forwarding into userspace at all? For raw throughput, kernel-native forwarding is almost always faster. The decision to build a userspace relay was not driven by a desire for more performance, but by necessity, after concluding that kernel-level solutions were insufficient or impractical for this specific problem.
The core challenge is the kernel’s Reverse Path Forwarding (RPF) check. While it’s true that this check can be disabled on a per-interface basis (by setting the rp_filter</code> sysctl to 0), this is often an inadequate solution for two key reasons:
It’s a Blunt Instrument: Disabling rp_filter</code> turns off a key anti-spoofing security mechanism for all traffic on that interface, not just the multicast stream in question. This is a security trade-off many administrators are unwilling to make.</li> It Doesn’t Solve the Whole Problem: More importantly, simply disabling the RPF check on the first router doesn’t “clean” the packet. The packet is still forwarded with its original, unroutable source address, which will cause RPF failures on correctly configured downstream routers. It pushes the problem one hop further away without solving it.</li> </ol> With the rp_filter</code> approach being insufficient, we are left with other kernel-level mechanisms, which also have limitations: No Inbound Multicast SNAT: Our previous investigation proved that it’s architecturally unsupported to use Netfilter’s conntrack</code> to perform SNAT on inbound multicast, which would be the most direct way to “clean” the source address.</li> Static Forwarding Limits: Using direct control of the Multicast Forwarding Cache (MFC) works for routable sources, but the kernel has a hard-coded limit of 32 Virtual Interfaces (MAXVIFS</code>), making it unsuitable for high-density scenarios.</li> </ul> Given these constraints, a targeted userspace application becomes the most practical way to solve the problem robustly at the network edge. It allows us to “launder” the packet once, making it fully compliant for the rest of its journey through the network. This trade-off—sacrificing the raw speed of the kernel for the correctness and flexibility of a userspace application—means that performance cannot be taken for granted and must be a central goal of the design. How it Works: MCR’s Architecture</h3> MCR’s architecture is designed to solve the RPF problem by bypassing the kernel’s IP stack and giving the operator direct, dynamic control over forwarding. It consists of two main components: The Supervisor & Workers: The main multicast_relay</code> process acts as a supervisor. It spawns high-performance “worker” processes, pinning each to a specific CPU core. These workers form the data plane, doing the actual packet processing.</li> The Control Plane: MCR is managed dynamically at runtime. A separate control_client</code> tool communicates with the supervisor over a UNIX socket, allowing you to add, remove, and list forwarding rules without ever stopping the service.</li> </ol> This design separates the data plane, which is optimized for performance, from the control plane, which is optimized for flexible management. Data Plane Design for Performance</h4> At its core, MCR operates on a simple but powerful principle: if the kernel won’t forward the packet, MCR will intercept it before the kernel has a chance to drop it. This is achieved through several key architectural decisions: Unified io_uring</code> Event Loop: At the heart of each worker is a single, unified event loop built on Linux’s most advanced I/O interface, io_uring</code>. This allows a single thread to manage both ingress and egress asynchronously with minimal syscall overhead, a design which avoids the need for complex and often slower cross-thread communication.</li> AF_PACKET</code> for Raw Sockets: By operating at Layer 2, MCR sidesteps the kernel’s entire IP stack on the ingress path. This is the key to bypassing the RPF check and gives MCR full control over packet handling.</li> Efficient In-Memory Fan-Out: When a single input stream must be replicated to multiple outputs, MCR uses a reference-counted pointer (Arc<[u8]></code>) to share a single packet’s memory buffer across multiple send operations. This avoids expensive memory copies in userspace, a critical optimization for high-density replication scenarios.</li> Core-Local Buffer Pools: Each worker pre-allocates and manages its own memory buffers, a strategy aimed at avoiding the performance penalty of dynamic memory allocation in the fast path.</li> </ul> While even higher performance might be achievable with technologies like eBPF/XDP, the AF_PACKET</code> and io_uring</code> approach was chosen deliberately. It offers excellent performance while maintaining compatibility with a wider range of widely deployed enterprise Linux kernels, avoiding the tighter coupling to specific kernel versions and complex toolchain dependencies that an eBPF-based solution would entail. Example Workflow</h4> A typical workflow looks like this: an operator starts the multicast_relay</code> service and then uses the control_client</code> to provision forwarding rules. For example, to relay a stream arriving on eth0</code> to a new destination via eth1</code>, the command is simple and declarative: ./control_client add-rule \ --input-interface eth0 \ --input-group 239.0.0.1 \ --input-port 5001 \ --output-interface eth1 \ --output-group 239.0.0.2 \ --output-port 6001 </code></pre> This command instructs a worker to listen for multicast traffic for 239.0.0.1:5001</code> on eth0</code> and re-transmit any received packets as a new stream to 239.0.0.2:6001</code> via eth1</code>. The source address of the new stream is the relay server’s own, making it fully routable. What About socat</code>?</h3> For many network tasks, the versatile socat</code> utility is an excellent choice. It can be used to solve the RPF problem by creating a userspace relay, and it’s important to understand the architectural and operational differences between that approach and MCR’s. Architectural Difference: Layer 2 vs. Layer 4</h4> The most significant difference is how each tool interacts with the network stack. socat</code> operates at Layer 4. It uses standard UDP sockets to receive and send packets. When it receives a packet, that packet has already been fully processed by the kernel’s network stack (checksums verified, IP headers parsed, etc.). socat</code> then takes the payload and sends it out via another UDP socket, causing the kernel to build a new packet. This is a simple and robust model.</li> MCR operates at Layer 2. As detailed in the architecture above, it uses AF_PACKET</code> raw sockets to capture entire Ethernet frames directly from the network driver. This allows it to bypass the kernel’s IP stack and, crucially, the RPF check.</li> </ul> Both tools solve the RPF problem, but in different ways. socat</code> cleverly uses “local delivery”—the kernel sees a local application is the destination and doesn’t apply the forwarding RPF check. MCR sidesteps the check entirely by intercepting the frame at a lower level. Operational and Feature Differences</h4> The architectural differences lead to different operational models: Process Model: socat</code> is a point-to-point tool. To relay a single multicast stream, you run a single socat</code> process. To relay ten streams, you must run ten separate socat</code> processes, which can be cumbersome to manage at scale. MCR uses a supervisor model where a single daemon can manage hundreds of forwarding rules concurrently across its worker processes.</li> Head-End Replication: A key feature of MCR is its ability to perform “fan-out” or “head-end replication,” where a single input stream is replicated to multiple different outputs. socat</code>’s “one-to-one” model cannot do this; it can only forward a stream from one point to another.</li> </ul> Comparative Example: 1-to-1 Relay</h4> Let’s say we want to relay a stream from 239.1.1.1:5001</code> on eth0</code> to 239.10.10.10:6001</code> on eth1</code>. With MCR, you would ensure the supervisor is running and then add a rule dynamically: # Add a rule to a running MCR instance ./control_client add-rule \ --input-interface eth0 \ --input-group 239.1.1.1 \ --input-port 5001 \ --output-interface eth1 \ --output-group 239.10.10.10 \ --output-port 6001 </code></pre> With socat</code>, you would launch a dedicated process for this specific task: socat -u \ UDP4-RECV:5001,ip-add-membership=239.1.1.1:eth0 \ UDP4-SEND:239.10.10.10:6001,bind=10.1.1.2 </code></pre> (Note: socat</code> requires binding to the specific IP address of the egress interface, here assumed to be 10.1.1.2</code> on eth1</code>.) In summary, socat</code> is an excellent tool for simple, static, one-to-one relay tasks. MCR is designed for more complex scenarios requiring high-density, dynamic rule management, and features like head-end replication. Measured Performance</h3> To complement the architectural and operational comparison, I conducted a series of benchmarks. The full performance validation reports are available in the project’s documentation, but here’s a summary of the key findings: Baseline Reliability (Layer 3 Routing): In a standard Layer 3 routing test (e.g., forwarding between veth</code> pairs), both MCR and socat</code> demonstrated 100% packet delivery with 0% loss at moderate loads (e.g., 50,000 packets per second). For simple L3 scenarios, both tools are equally reliable. </li> Scalability Under High Load: Under a sustained load of 400,000 packets per second in the same Layer 3 topology, MCR maintained excellent performance with a negligible 0.2% packet loss, whereas socat</code> dropped over 15.3% of packets. This highlights MCR’s architectural advantage with io_uring</code>’s batched I/O, significantly reducing syscall overhead at high rates. </li> Architectural Suitability for L2 Bridging: A key use case for MCR is relaying traffic between logically separate Layer 2 domains, such as two different Linux bridges. In this scenario, MCR forwarded traffic with 0% loss, whereas socat</code>, as a Layer 4 tool, is not designed to operate at this level and could not forward the traffic. This doesn’t represent a flaw in socat</code>, but rather illustrates a foundational difference in capability: MCR is explicitly designed for L2 operations that are outside the scope of standard UDP socket-based tools. </li> Extreme Fan-out: MCR was successfully validated performing a 1-to-50 head-end replication, where a single input stream was replicated to 50 unique outputs. This demonstrates MCR’s ability to bypass the kernel’s hard-coded MAXVIFS</code> limit of 32 output interfaces for native multicast forwarding. </li> </ul> These results, gathered in a virtualized test environment, confirm MCR’s capabilities for scenarios demanding high throughput and architectural versatility. A comprehensive analysis of performance on physical hardware is planned for a future article. Why Rust?</h3> Rust was chosen for MCR because its language features align well with the project’s goals. Its focus on memory safety and its ownership model help prevent common classes of bugs, like buffer overflows and use-after-free errors, which are particularly critical in networking applications that deal directly with raw memory buffers and low-level system calls. This allows the development focus to remain on the core logic, with the compiler providing strong guarantees against many potential vulnerabilities. Of course, operating at Layer 2 requires powerful capabilities (CAP_NET_RAW</code>), which brings significant security responsibility; future work will explore strategies for dropping these privileges as soon as they are no longer needed. Conclusion: The Right Tool for the Job</h3> The journey from the “disappearing multicast packet” to MCR highlights a core engineering principle: when you hit a fundamental limitation in one layer of the system, an effective solution is often to move up a layer and build a more specialized tool. MCR is that tool. It attempts to solve the specific, real-world problem of unroutable multicast sources by providing a manageable and high-performance userspace relay. It’s an exploration of how combining a deep understanding of the kernel’s boundaries with modern APIs and a language like Rust can create powerful and reliable solutions to complex networking challenges. This approach of a lightweight, userspace-driven protocol that leverages the existing unicast network is not a new idea. It follows the design philosophy of protocols like SLIM (Self-configuring Lightweight Internet Multicast), which argued that the key to multicast adoption was to “nullify the management complexity” by avoiding multicast-specific infrastructure. MCR can be seen as a modern realization of that vision, implementing a similar architectural pattern with the high-performance, asynchronous tools available to us today. References</h3> Hjálmtýsson, G., Brynjúlfsson, B., & Helgason, Ó. R. (2003). Overcoming Last-Hop/First-Hop Problems in IP Multicast. Lecture Notes in Computer Science. </li> Hjálmtýsson, G., Brynjúlfsson, B., & Helgason, Ó. R. (2004). Self-configuring lightweight Internet multicast protocol specification. IEEE International Conference on Systems, Man and Cybernetics. </li> </ol>
The Curious Case of the Disappearing Multicast Packet 2025-11-02T00:00:00+00:00 Many consumer and industrial devices—from home security cameras to the HDMI-over-IP extenders I investigated in a previous post</a>—are designed as simple appliances. They often have hard-coded, unroutable IP addresses (e.g., 192.168.1.100</code>) and expect to live on a simple, isolated network. This becomes a major problem when you need to get their multicast video or data streams from that isolated segment to users on a main LAN. My goal was to solve this with a standard Linux server, creating a simple, high-performance multicast router without a dedicated, expensive hardware box. Can a standard Linux server act as a simple, kernel-native multicast router for devices with hard-coded, unroutable IP addresses? This article chronicles an investigation into combining nftables</code> SNAT with direct control of the Multicast Forwarding Cache (MFC). The DSP Lens on Real-World Events, Part 1: A New Dimension for Your Data 2025-08-09T00:00:00+00:00 The Universal Challenge: From Events to Insight</h3> Every field that deals with streams of events over time shares a common challenge. A factory manager tracking items on a conveyor belt, a data scientist analyzing user clicks, and a network engineer monitoring packets are all trying to turn a series of discrete, often chaotic, events into meaningful insight. The tools may differ, but the fundamental problem is the same: how do we perceive the true nature of a system through the lens of the events it generates? My own journey into this problem began with a network analysis tool, JitterTrap</a>. I was seeing things that shouldn’t exist: a prominent, slow-moving wave in my network throughput graph that my knowledge of the underlying system said couldn’t be real. This “ghost” in the machine forced me to look for answers in an unexpected place: the field of Digital Signal Processing (DSP). This three-part series shares the lessons from that journey. Part 1 introduces the core DSP mental model as a powerful, universal framework. Part 2 lays out a theoretical framework for analysis, and Part 3 applies that framework to perform a rigorous, quantitative analysis of a real-world measurement problem. The DSP Lens on Real-World Events, Part 2: A Framework for Analysis 2025-08-09T00:00:00+00:00 Introduction: The Rosetta Stone</h3> In Part 1, we introduced the “ghost in the machine”—an aliased signal created by the very act of measuring a stream of discrete events. We established that any time-bucketed analysis is a form of filtering. But to truly understand our measurements and build a trustworthy instrument, we need a more formal framework. This article is that framework—a “Rosetta Stone” to connect the different concepts at play. Before we can analyze our system quantitatively in Part 3, we must first understand the precise relationship between our sample rate, the frequencies we want to observe, and the characteristics of the filter we are unknowingly using. The DSP Lens on Real-World Events, Part 3: Building an Honest Instrument 2025-08-09T00:00:00+00:00 Introduction: Applying the Framework</h3> In Part 1, we introduced the DSP lens and the problem of aliasing. In Part 2, we built a detailed theoretical framework, our “Rosetta Stone,” connecting the concepts of sample rates, filters, and critical frequencies. Now, in this final part, we apply that framework to a real-world scenario. We will use the specific parameters of the Jittertrap tool to perform a rigorous quantitative analysis of the system’s imperfections. This will allow us to determine, with numerical confidence, exactly what we can and cannot trust from our measurements, and ultimately, how to build a more honest instrument. A fresh coat of paint for Jittertrap 2025-08-03T00:00:00+00:00 My long-running project, Jittertrap, had been neglected for a long time… Mostly, because of a lack of interest. However, I recently had need for it at my day job, and I dusted it off, wrote down a list of things I need to fix for it to be useful to me in that one particular use case, and decided to fix the things that embarrass me and implement the things that would improve my life. The backend code is mostly fine. There are some tricky things about it, and I’ve been thinking about a rewrite in Rust and eBPF, but it mostly works. The frontend, however, has always been a different story. It’s a web UI, and to be blunt, my interest in the nuances of JavaScript, CSS, and the ever-shifting sands of web frameworks is minimal. It’s necessary, but it’s not the work I enjoy. For years, this meant the UI languished. It was functional, but it was built on an old version of Bootstrap and was riddled with the kind of minor, irritating bugs that I had little motivation to track down. The project stalled, not because the core idea was finished, but because it wasn’t fun, and it wasn’t appreciated. The uninteresting work had become a barrier to the interesting work. I decided that this is my opportunity to see what all the AI coding fuss is about. I started using Gemini as a coding assistant, specifically to tackle the frontend. My goal was to offload the tedious parts - the parts I had no desire to master myself. The results have been surprisingly good. Over the past month, I’ve managed to clear a huge backlog of UI debt. Here’s a quick rundown of the progress: Modernized Frameworks: The entire UI has been migrated from Bootstrap 3 to Bootstrap 4. This was a significant undertaking that I’d been putting off for ages, and one milestone on the journey of removing the jQuery dependency. Because apparently all browsers are good now and jQuery is a liability. </li> Richer Charting Features: The charts have become more powerful for analysis. The “Top Talkers” chart now aggregates smaller, less relevant flows into a single “Other” category, making the visualization much cleaner. I also added a switch to toggle the Y-axis between logarithmic and linear scales, allowing for a more nuanced view of the data. </li> Improved IPv6 Support: The backend now properly handles IPv6 traffic. I fixed several bugs in the packet decoding logic for IPv6 addresses and extension headers, which is critical for monitoring modern networks. </li> Security Hardening: I took the time to harden the application by fixing several security vulnerabilities, including a potential XSS issue in the WebSocket handler and a memory leak in the backend C code. </li> Code Modernization: The JavaScript codebase has been brought into the modern era. All the old var</code> declarations are gone, replaced by let</code> and const</code>, and I’ve been able to systematically replace old jQuery patterns and callbacks with native JS and arrow functions. </li> Performance Overhaul: The d3.js charts, which are central to Jittertrap, have been substantially optimized. We moved the “Top Talkers” chart to a <canvas></code> element, which dramatically improved rendering performance. The old setInterval</code> loop was replaced with requestAnimationFrame</code> for smoother, more efficient updates. </li> Responsive Layout: The UI is now properly responsive. The main container is fluid, and on wide screens, the throughput and top talkers charts now sit side-by-side, which is a much better use of the space. </li> Bug Fixes: And of course, a whole host of nagging bugs have been squashed, from layout issues with the chart legends to incorrect resizing behavior. </li> </ul> The interesting part of this process is how it has changed my relationship with my own project. I can now focus on the What and not get bogged down in the How of web development. I describe the desired outcome—“make the legend responsive,” “fix the grid resizing”—and the AI generates the code. It’s a powerful new tool, not unlike a compiler or a static analyzer, that handles a specific domain of work for me. It’s a more efficient and, frankly, more enjoyable way to work on a passion project. The AI isn’t driving the project, but it’s clearing the road so I can. Rapido - rapidly creating test VMs for driver development 2023-11-25T00:00:00+00:00 Thanks to rapido</a>, it’s become much simpler to test Linux device drivers for real PCIe devices in VMs. The advantages of this approach are: the host is protected from memory corruption errors caused by buggy kernel drivers</li> the PCI peripheral can be physically installed in a multi-use machine, reducing hardware & lab requirements</li> debugging info is easily available</li> the development cycle is short and simple - rapid even :)</li> </ul> PMTU weirdness 2023-03-11T00:00:00+00:00 On a good day, my day job involves building networking tools in python. Too many python networking tools look like shell scripts, spawning subprocesses for basic tools like ‘ping’ or ‘ip’ - often resulting in a fragile mess due to poor, or inconsistent error handling. I was quite excited to find icmplib</a>. It provides a much simpler, less fragile way to do things like reachability tests, RTT measurements, path discovery and path MTU discovery in python code. Hopefully it finds its way into Fedora soon! Armed with icmplib, I went on a journey of discovery to develop my understanding of Path MTU. I specifically wanted to understand the differences between IPv4 and IPv6, and the effect of VETH</a>, VLAN and Bridge virtual devices. Status update 2017-12-03T00:00:00+00:00 I’ll be speaking at linux.conf.au</a> 2018 in Sydney. Jittertrap</a> testing infrastructure is slowly improving. The idea is to include known test data so that the front-end rendering can be verified and refactored. I expect a sprint over the holidays and some new bugs for lca. :) Clean up of the tn40xx driver</a> is continuing - hopefully towards mainline inclusion. The driver project gets more visitors and clones than Jittertrap, which I find interesting, unexpected and somewhat sobering. Listing Network interface, MAC address and PCI bus/device/function 2016-03-28T00:00:00+00:00 New hardware for BufferUpr</a> has arrived! I’m working on adding Wifi capability into the product and with new hardware comes new challenges. Challenge: Find a simple way to list all the network interfaces in the system, including the PCIe slot and MAC address for each. Solution: find /sys/devices/pci0000* \ -wholename "*net/*/address" \ -printf "%h/%f " \ -exec cat '{}' \; </code></pre> This depends on a proper find</code>, of course, so Busybox won’t do. Example output: /sys/devices/pci0000:00/0000:00:03.0/0000:03:00.0/net/p3p0/address 90:e2:ba:7f:97:00 /sys/devices/pci0000:00/0000:00:03.0/0000:03:00.1/net/p3p1/address 90:e2:ba:7f:97:01 /sys/devices/pci0000:00/0000:00:03.0/0000:03:00.2/net/p3p2/address 90:e2:ba:7f:97:02 /sys/devices/pci0000:00/0000:00:03.0/0000:03:00.3/net/p3p3/address 90:e2:ba:7f:97:03 /sys/devices/pci0000:00/0000:00:19.0/net/em1/address 88:88:88:88:87:88 </code></pre> Notice from the path to the address files that some of the interfaces on this system are named according to their PCI BDF (bus/device/function) identifiers. For example, the p3p2</code> interface is bus 3, device 0, function 2. From these BDF numbers, we can guess that there is a 4-port card in PCIe slot 3. However, it may not be easy to tell which physical connector on the motherboard maps to slot 3. It might even involve screwdrivers and dust… shudder. There are some more examples and a better explanation of device naming rules over there</a>. If the disappearance of eth0</code> is news to you and you’re frustrated that nobody asked you whether you wanted interface naming fixed, you might want to read about why this change was made</a>. The Xen wiki</a> has a good page on BDF numbers and Wikipedia has more detail</a>. Installing SciPy on Fedora 23 2016-01-11T00:00:00+00:00 Fedora 23</a> (the current release at the time of writing) ships an outdated version of SciPy</a> that doesn’t include the spectrogram function. Installing the latest Scipy was kind-of a pain, so I thought I’d record some instructions for future-me and share it with you. These instructions install dependencies, set up a python virtualenv container and then install the latest matplotlib</a>, SciPy</a> and Numpy</a> on Fedora 23</a> Dependencies: python-virtualenv </li> libpng-devel freetype-devel pygobject2-devel </li> redhat-rpm-config </li> gcc-gfortran gcc-c++ </li> atlas-devel lapack-devel </li> </ul> Install these from Fedora repositories: sudo dnf install python-virtualenv libpng-devel freetype-devel pygobject2-devel redhat-rpm-config gcc-gfortran gcc-c++ atlas-devel lapack-devel </code></pre> And then the python packages: virtualenv env source env/bin/activate pip install -U pip && pip install numpy && pip install matplotlib && pip install scipy </code></pre> Installing numpy, matplotlib and scipy simultaneously fails. Thanks, pip. JitterTrap: Baby steps in DSP 2015-12-01T00:00:00+00:00 This is the story of my first expensive lesson in Digital Signal Processing. It is about JitterTrap</a>, the free software that powers BufferUpr</a>. The premise of BufferUpr is to combine commodity hardware with open source software to create a product that can measure data stream delays of 1-100 milliseconds. This is the range of delay that most multimedia developers and consumers are concerned with. To measure the delay, we count the packets and bytes as they fly past and look for changes in throughput. My naive logic was that if we look at consecutive intervals of 1ms, we’ll be able to measure the throughput over that interval, plot it and show any variation in throughput. This seemed like a very simple task, except that I had no previous experience in Digital Signal Processing and didn’t see why that would be relevant. Lightweight containers in Fedora using systemd 2015-04-02T00:00:00+00:00 It was the year of 2015 and people were still developing new applications in PHP… but for those who could no longer accept the idea of installing a system-wide LAMP stack, there was a new-old fassionable thing: Containers! This is a quick howto for creating a throw-away container for messing around with PHP apps on Fedora. Most of it is taken from the examples at the bottom of ‘man 1 systemd-nspawn’ Create the directory for the container filesystem: [user@host]$ mkdir web-dev-container </code></pre> Install a minimal Fedora image (including apache and php) into the new container root: [user@host]$ sudo yum -y --releasever=21 --nogpg --installroot=$(pwd)/web-dev-container --disablerepo='*' --enablerepo=fedora install systemd passwd yum fedora-release less vim iproute httpd php </code></pre> Update the SELinux context: [user@host]$ sudo semanage fcontext -a -t svirt_sandbox_file_t "web-dev-container(/.*)?" [user@host]$ sudo restorecon -R web-dev-container </code></pre> Spawn the new container: [user@host]$ sudo systemd-nspawn -D web-dev-container </code></pre> In the new container, change the root password: [root@container]# passwd [root@container]# logout </code></pre> Boot the new container: [user@host]$ sudo systemd-nspawn -bD web-dev-container </code></pre> Enable and start the web server and check that it’s running: [root@container]# systemctl enable httpd.service [root@container]# systemctl start httpd.service [root@container]# ss -lt </code></pre> Check http://localhost in a web browser. Add an index.html file: [root@container]# echo "Hello World" >> /var/www/html/index.html [root@container]# echo "<?php echo \"Hello PHP World\" ?>" > /var/www/html/index.php </code></pre> Refresh http://localhost Check http://localhost/index.php You can now logout of the container (it will keep running). Change the ownership and hack away! [user@host]$ sudo chown -R user.user web-dev-container/var/www/html [user@host]$ vim web-dev-container/var/www/html/index.php </code></pre> Login to the container, if needed: [user@host]$ sudo machinectl login web-dev-container </code></pre> Exit the container by pressing ^] three times in one second. Using policy routing to loopback over external interfaces 2015-04-01T00:00:00+00:00 Sometimes there is good reason to talk to yourself. You might be doing a sound check, for example. Likewise, it can be useful to route IP packets between two interfaces on the same machine using an external path. One reason to do this is to test other network devices like routers or switches. For many years this has been much trickier to do than it might seem, until Patrick McHardy’s introduction</a> of the ‘accept_local’ setting</a> in December 2009. A little more than a year after that, Kirill Smelkov replied with a functional example script</a> that shows how to achieve full loopback bi-directional IP traffic over physical interfaces. See also this GitHub gist</a> with a similar script. Even so, there are numerous red herring posts on StackOverflow and elsewhere that don’t really help: 1</a> 2</a> 3</a> 4</a> 5</a> So, how does it work? Policy Routing, that’s how! Basic preparation:</h2> If you have not already done so, step 0 is to tell NetworkManager to leave the interfaces alone! Select the interface names and addresses: IFACE_A=p3p1 IFACE_B=p3p4 IFACE_A_IP="10.10.3.1" IFACE_B_IP="10.10.3.4" </code></pre> Assign addresses: ip addr replace $IFACE_A_IP dev $IFACE_A ip addr replace $IFACE_B_IP dev $IFACE_B </code></pre> Ensure the links are up: ip link change $IFACE_A up ip link change $IFACE_B up </code></pre> Policy Routing</h2> Enable receiving of local source addresses on external ifaces (see Documentation/networking/ip-sysctl.txt in the kernel source tree). sysctl -w net.ipv4.conf.$IFACE_A.accept_local=1 sysctl -w net.ipv4.conf.$IFACE_B.accept_local=1 </code></pre> Decrease priority of the ‘local’ routing table (change preference from 0 to 1000): ip rule del pref 0 lookup local ip rule add pref 1000 lookup local </code></pre> Prevent routing loop by using preference rules to route incoming packets using ‘local’ table. ip rule add pref 100 iif $IFACE_A lookup local ip rule add pref 101 iif $IFACE_B lookup local </code></pre> Use a routing table for each interface, that is, destination policy routing: ip route add default dev $IFACE_A table 200 ip route add default dev $IFACE_B table 201 </code></pre> Note: There’s no need to create the tables first, adding a route entry is sufficient. Add preference rules to route each destination using a coresponding table: ip rule add pref 200 to ${IFACE_B_IP} lookup 200 # IFACE_A to IFACE_B ip rule add pref 201 to ${IFACE_A_IP} lookup 201 </code></pre> Note: Do not be confused by the numbers. ‘pref 200’ refers to the preference/priority of the rule, whereas ‘lookup 200’ refers to table number 200. The kernel only cares about the number, but ‘ip’ will use the optional number-to-name mapping defined in /etc/iproute2/rt_tables. Finally, flush the route cache to ensure the tables are reread: ip route flush cache </code></pre> Check the routing rules: ip rule show 100: from all iif p3p1 lookup local 101: from all iif p3p4 lookup local 200: from all to 10.10.3.4 lookup 200 201: from all to 10.10.3.1 lookup 201 1000: from all lookup local 32766: from all lookup main 32767: from all lookup default </code></pre> Test that the traffic is routed via the external interface using ‘tcpdump’ and ‘ping’: tcpdump: # tcpdump -nlq -i p3p1 -c 6 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on p3p1, link-type EN10MB (Ethernet), capture size 262144 bytes 15:51:00.965010 IP 10.10.3.1 > 10.10.3.4: ICMP echo request, id 27352, seq 1, length 64 15:51:00.965217 IP 10.10.3.4 > 10.10.3.1: ICMP echo reply, id 27352, seq 1, length 64 15:51:01.964694 IP 10.10.3.1 > 10.10.3.4: ICMP echo request, id 27352, seq 2, length 64 15:51:01.964825 IP 10.10.3.4 > 10.10.3.1: ICMP echo reply, id 27352, seq 2, length 64 15:51:02.964785 IP 10.10.3.1 > 10.10.3.4: ICMP echo request, id 27352, seq 3, length 64 15:51:02.964987 IP 10.10.3.4 > 10.10.3.1: ICMP echo reply, id 27352, seq 3, length 64 6 packets captured 6 packets received by filter 0 packets dropped by kernel </code></pre> ping: $ ping -c 3 10.10.3.4 PING 10.10.3.4 (10.10.3.4) 56(84) bytes of data. 64 bytes from 10.10.3.4: icmp_seq=1 ttl=64 time=0.247 ms 64 bytes from 10.10.3.4: icmp_seq=2 ttl=64 time=0.183 ms 64 bytes from 10.10.3.4: icmp_seq=3 ttl=64 time=0.243 ms --- 10.10.3.4 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 1999ms rtt min/avg/max/mdev = 0.183/0.224/0.247/0.031 ms </code></pre> Thats it! There’s no need to mess around with ARP or reverse path filtering! DSCP vs Linux socket priorities 2014-09-27T00:00:00+00:00 I received some encouraging comments on G+</a> from Jesper Dangaard Brouer about my previous post</a> on DSCP, Linux and VLAN priorities. Those comments and the work linked to (here</a>) points to a few long-standing (but minor) issues with the way DSCP priorities are handled in Linux. Some DSCP values, like Expedited Forwarding, are not currently (3.17 and earlier) handled correctly.</li> Linux Priorities, defined in include/uapi/linux/pkt_sched.h, are not documented particularly well, but forms part of the stable interface with userspace. Working with traffic classification (tc), queuing disciplines (qdisc) or VLANs requires at least a basic understanding of Linux socket priorities.</li> </ol> Looking into DSCP and IEEE 802.1p (VLAN priorities). 2014-09-17T00:00:00+00:00 I recently discovered a flaw in the VLAN implementation I did at work. It seemed that the normal TCP traffic had the correct VLAN priorities applied, but audio streaming UDP traffic did not. This was due to DSCP being applied to the streaming audio and the fact that the VLAN device’s egress-qos-map was incorrect. I had assumed, incorrectly, that VLAN priorities are applied to all traffic as long as we’re not using fancy queuing disciplines (qdiscs). After all, 802.1Q is strictly a layer 2 thing. Linux Network Configuration 2014-08-12T00:00:00+00:00 This concerns the proliferation of netlink libraries and a lack of direction and documentation. Background:</h1> I’ve configured a router with netem</a> (see Bandwidth Throttling with NetEM Network Emulation</a> and the tc-netem man page</a>) to test Tieline</a> devices under various delay and loss network conditions. It’s not really feasible for the tester to use ‘tc’ on the command line for the various tests, so my simple requirement is to build a little web interface. It’s really easy to hack something together quickly (parsing the output of iproute2), but I’ve been developing a bit of an allergy for this kind of hacking things together. I’d like to use a scripting language (Python or Perl, not C) for the webby stuff, so I started looking at what modules these languages provide for netlink, which seems like a better idea than parsing output of “tc” and “ip”. I couldn’t find anything that plays well with netem, but the python bindings for libnl seems to be closest to what I need. This led me to look at the larger landscape for userspace netlink libraries… Existing Netlink Libraries:</h1> There are a few different userspace netlink “libraries”, with somewhat different goals. The libnl documentation</a> makes the distinction between the different subsystems clear in this diagram: libnl</a>: seems to be aimed at general use,</li> includes Python bindings,</li> used by NetworkManager, libvirt, libpcap, etc.</li> decent documentation.</li> </ul> However, libnl is not the only netlink library in use. libmnl</a>: available for general use.</li> used by nftables</li> </ul> libnetlink</a>: internal to iproute2, not for general use.</li> </ul> sd-netlink</a>: internal to systemd-networkd, not for general use.</li> </ul> Network Configuration Managers</h1> Lets look at this from another angle: what do the typical network managers / configuration applications use? NetworkManager</a>: mature</li> maintained</li> uses libnl for direct netlink comms</li> C</li> provides DBus interface</li> </ul> ConnMan</a>: maturity / maintenance concerns</li> C</li> </ul> Wicd</a>: unmaintained</li> Python</li> calls external tools like ifconfig</li> provides DBus interface</li> </ul> netctl</a> (Arch package</a>): maintained</li> Bash</li> calls external tools like iproute2, etc.</li> </ul> netifd (OpenWRT)</a>: active / maintained</li> C</li> </ul> systemd-networkd</a>: active / maintained</li> C</li> uses internal sd-rtnl</li> </ul> vyconfd</a>: maintenance concerns after vyatta sold to brocade.</li> Python</li> external iproute2, etc. tools?</li> </ul> LNST</a>: Python</li> external tools or NetworkManager via DBus</li> </ul> netcf</a>: tries to bridge distro-specific network config files with its own and provides a C library.</li> does not seem to modify the state directly.</li> seems to be used by virsh</li> </ul> ncfg (slides no longer available): does it, or will it matter?</li> </ul> I also fell down the rabbit warren of looking what the various SDN and router OS projects are doing - and I’m still recovering from the experience - so I’ll write about that when I have a better understanding of it. Back to solving the problem:</h1> Remember that the initial problem was how to configure Netem. Option 1: the quick and easy hack. This is what I ended up doing at work, for now, because it’s that kind of place. exec tc</li> </ul> Option 2: Use a network manager with a dbus interface. Both NetworkManager and Wicd can do this.</li> </ul> Alternative 2: Use libnl’s python bindings.</li> </ul> Conclusion</h1> Basically, many projects for network configuration have not progressed past shell scripting and iproute2 - some kind of lowest common denominator - and make a large effort to tie individual tools together into something with a semi-coherent interface. I believe there is a need for a good netlink library for one of the popular scripting languages, so that we can move past the old crufty shell scripts and parsing output of stand-alone tools. On the other hand, many of these basic tools have implemented their own abstractions for using the netlink kernel interface. A Grand Unified Netlink Userspace certainly seems like it would be an improvement over the status quo, but each netlink library probably exists for good reason and a merger seems unlikely. libnl seems like the natural choice for mere mortals. I’ll post some examples as I learn more about it. See also</h1> Video: Can Linux network configuration suck less?</del> (link dead) Slides</a> OpenWRT comparison of network managers</a> Plain text email attachments. 2014-07-24T00:00:00+00:00 After finally completing task 01 of the Eudyptula Challenge</a>, I’d like to share a few things I’ve learned, without divulging any crucial details about the task or solution. tl;dr Pay attention. Patience…</h1> Maybe it’s just the timezone, but the turn-around time for a response to a submission meant that I could only send one message per day. That fits with my experience on the netdev and iommu mailing lists. Paying attention to detail.</h1> Having a longer turn-around time really discourages small, quick changes and “is it good enough now?” messages. This is a Good Thing. It makes you think about what you’re doing. It reinforces the concept that it’s good to take as long as you need to do things properly and double check your work, because submitting rubbish will waste a lot of your own time, not just reviewer time. Pay even more attention to detail, or, email clients suck.</h1> It took me four attempts to complete the first task. It’s embarrassing. The task involves sending plain text attachments, as opposed to base64 encoded attachments. To achieve this, I first went through some effort to get this domain up and running and get back some of the control over email that gmail doesn’t provide. Then, I thought I’d use Alpine as a safe bet for an email client, but it turns out Alpine always uses base64 encoding for attachments. Sigh… So, for the first attempt I submitted a patch instead of a simple attachment. I thought I could cheat, just a little, and avoid a problem with base64 encoded attachments. Unfortunately the script wasn’t expecting it and by submitting an unexpected result I wasted my time as well as my anonymous teacher’s time. So, I switched to Thunderbird. The second attempt was rejected, because I didn’t read the requirement properly. I won’t go into more details, but doh! The third attempt… this is where the email hiccup came in. For the second attempt I carefully tested that Thunderbird was creating plain text attachments, as follows: --------------040002060807090301030305 Content-Type: text/plain; charset=UTF-8; name="Makefile" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="Makefile" </code></pre> It turns out that the character encoding of the email influences what Thunderbird does with the attachment. Of course I didn’t know that and I didn’t think to check it. After all, the previous message had the correct attachment encoding. So, for the record, don’t use UTF-8 encoding for the message body, or your attachment will be base64 encoded. Conclusion</h1> The Eudyptula Challenge requires careful attention to detail, more than the standard expected of me in my day job. It encourages me to pursue a higher standard of work. Self-hosting email again. 2014-07-18T00:00:00+00:00 It’s been a few years since I set up or maintained a public mail server. It took much longer than expected to get everything working again, but here we are! acooks @ rationali.st Let’s see how long it takes before I need to tweak spam filters again. Hello World 2014-07-14T00:00:00+00:00 What’s this? This little bit of interweb detritus was created for a few reasons: Transparency. Employers want to know what you’ve done before.</li> To learn about “the cloud”. I’ve done plenty of sysadmin work in the past, but that was all on my own bare metal. This helps to keep my sysadmin skills current, since my day job doesn’t involve that at the moment and I might need those skills again.</li> The first task of the Eudyptula Challenge</a> suggests not using Gmail. This blog is another side-effect of getting my own infrastructure up again.</li> To document some projects I’ve been working on.</li> </ol> About me 2014-07-14T00:00:00+00:00 Hi! I’m a Software Engineer and have been a Linux and networking enthusiast since the late nineties. Born and raised in Pretoria, South Africa. I moved to Perth, Western Australia in December 2011, before settling in Brisbane in 2016. In 2015 I founded a startup producing test and measurement tools for real-time IP communications applications. As a business, it failed to gain any customers, but the Jittertrap</a> software is freely available. In 2017 I worked on the tn40xx Ethernet driver</a> (as a hobby) with the hope of upstreaming it. That ended after the chip vendor went out of business and my employement and family changed. Since 2018 I’ve been building software for software defined networks and communication systems</a>. I love working in the low level details, though recently I’ve been spending more time on collaborative work like sketching designs on white boards or Visio and finding less time for independent technical contributions. A couple of trivial patches of mine made it into the kernel</a> and I’m always talking about how I want to be a kernel developer when I grow up. That’s a Red Hat Linux 5</a> (Not RHEL) user’s guide circa 1998. TCP/IP Unleashed from 1996. Seems I started young and haven’t moved on.

rationali.st

When Packets Can't Wait: Comparing Protocols for Delay-Sensitive Data

Protocol Reference: Transport for Data with Deadlines

The Jitter Cliff: When TCP Goes Chaotic

Diagnosing Video Stuttering Over TCP: A JitterTrap Investigation

RFC 8362: Extended LSA Support for FRR ospf6d

Linux Network Configuration: A Decade Later

MCR: A High-Performance Userspace Multicast Relay

The Curious Case of the Disappearing Multicast Packet

The DSP Lens on Real-World Events, Part 1: A New Dimension for Your Data

The DSP Lens on Real-World Events, Part 2: A Framework for Analysis

The DSP Lens on Real-World Events, Part 3: Building an Honest Instrument

A fresh coat of paint for Jittertrap

Rapido - rapidly creating test VMs for driver development

PMTU weirdness

Status update

Listing Network interface, MAC address and PCI bus/device/function

Installing SciPy on Fedora 23

JitterTrap: Baby steps in DSP

Lightweight containers in Fedora using systemd

Using policy routing to loopback over external interfaces

DSCP vs Linux socket priorities

Looking into DSCP and IEEE 802.1p (VLAN priorities).

Linux Network Configuration

Network Configuration Managers</h1> Lets look at this from another angle: what do the typical network managers / configuration applications use?</p>

Back to solving the problem:</h1> Remember that the initial problem was how to configure Netem.</p> Option 1: the quick and easy hack. This is what I ended up doing at work, for now, because it’s that kind of place.</em></p> exec tc</li> </ul> Option 2:</em></p>

See also</h1> Video: Can Linux network configuration suck less?</del> (link dead)</em></p> Slides</a> OpenWRT comparison of network managers</a></p>

Plain text email attachments.

Conclusion</h1> The Eudyptula Challenge requires careful attention to detail, more than the standard expected of me in my day job. It encourages me to pursue a higher standard of work.</p>

Self-hosting email again.

Hello World

About me

See also</h1>
Video: Can Linux network configuration suck less?</del> (link dead)</em></p>
Slides</a> OpenWRT comparison of network managers</a></p>

Conclusion</h1>
The Eudyptula Challenge requires careful attention to detail, more than the standard expected of me in my day job. It encourages me to pursue a higher standard of work.</p>