Sat 25 November 2023

Filed under Linux Networking

Thanks to rapido, it's become much simpler to test Linux device drivers for real PCIe devices in VMs.

The advantages of this approach are:

  • the host is protected from memory corruption errors caused by buggy kernel drivers
  • the PCI peripheral can be physically installed in a multi-use machine, reducing hardware & lab requirements
  • debugging info is easily available
  • the development cycle is short and simple - rapid even :)

Here's a screen cast showing the workflow for a simple developer test: (thanks to termtosvg)

SeaBIOS (version 1.16.2-2.fc39 SeaBIOS (version 1.16.2-2.fc39)iPXE (https://ipxe.org) 00:02.0 CA00 PCI2.10 PnP PMM+0EFCBEB0+0EF0BEB0 CA00Press Ctrl-B to configu Booting from ROM.. [ 0.000000][ T0] Linux version 6.6.1 (acooks@BBB) (gcc (GCC) 13.2.1 20231011 (Red Hat 13.2.1-4), GNU ld version 2.40-13.fc39)3[ 0.000000][ T0] Command line: rapido.vm_num=1 rd.systemd.unit=dracut-cmdline.service console=ttyS [ 0.000000][ T0] Command line: rapido.vm_num=1 rd.systemd.unit=dracut-cmdline.service console=ttyS0 net.ifnames=0 systemd.mac1[ 0.000000][ T0] BIOS-provided physical RAM map:[ 0.000000][ T0] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable[ 0.000000][ T0] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved[ 0.000000][ T0] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved[ 0.000000][ T0] BIOS-e820: [mem 0x0000000000100000-0x000000000ffde [ 0.000000][ T0] BIOS-e820: [mem 0x0000000000100000-0x000000000ffdef [ 0.000000][ T0] BIOS-e820: [mem 0x0000000000100000-0x000000000ffdefff] usable[ 0.000000][ T0] BIOS-e820: [mem 0x000000000ffdf000-0x000000000fffffff] reserved[ 0.000000][ T0] BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved[ 0.000000][ T0] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved[ 0.000000][ T0] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved[ 0.000000][ T0] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved[ 0.000000][ T0] NX (Exe [ 0.000000][ T0] NX (Exec [ 0.000000][ T0] NX (Execute Disable) protection: active[ 0.000000][ T0] APIC: Static calls initialized[ 0.000000][ T0] SMBIOS 3.0.0 present.[ 0.000000][ T0] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-2.fc39 04/01/2014[ 0.000000][ T0] Hypervisor detected: KVM[ 0.000000][ T0] kvm-clock: Using msrs 4b564d01 and 4b564d00[ 0.000002][ T0] kvm-clock: using sched offset of 92642224 cycles[ 0.000006][ T0] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns[ 0.000014][ T0] tsc: Detected 3494.400 MHz processo [ 0.000014][ T0] tsc: Detected 3494.400 MHz processor[ 0.000866][ T0] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved[ 0.000875][ T0] e820: remove [mem 0x000a0000-0x000fffff] usable[ 0.000884][ T0] last_pfn = 0xffdf max_arch_pfn = 0x400000000[ 0.000904][ T0] MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs[ [ 0.000908][ T0] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.003475][ T0] found SMP MP-table at [mem 0x000f5bd0-0x000f5bdf][ 0.004955][ T0] RAMDISK: [mem 0x0de9d000-0x0ffcffff][ 0.004959][ T0] ACPI: Early table checksum verification disabled[ 0.004962][ T0] ACPI: RSDP 0x00000000000F5A10 000014 (v00 BOCHS )[ 0.004971][ T0] ACPI: RSDT 0x000000000FFE24C5 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)[ 0.004980][ T0] ACPI: FACP 0x000000000FFE2215 0000F4 (v03 BOCHS BXPC 00000001 BXPC 00000001)[ 0.004990][ T0] ACPI: DSDT 0x000000000FFE0040 0021D5 (v01 BOCHS BXPC 00000001 BXPC 00000001)[ 0.004997][ T0] ACPI: FACS 0x000000 [ 0.004997][ T0] ACPI: FACS 0x000000000FFE0000 000040[ 0.005004][ T0] ACPI: APIC 0x000000000FFE2309 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001)[ 0.005011][ T0] ACPI: HPET 0x000000000FFE2399 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001)[ 0.005018][ T0] ACPI: MCFG 0x000000000FFE23D1 00003C (v01 BOCHS BXPC 00000001 BXPC 00000001)[ 0.005025][ T0] ACPI: DMAR 0x000000000FFE240D 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001)[ 0.005032][ T0] ACPI: WAET 0x000000000FFE249D 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001)[ 0.005038][ T0] ACPI: Reserving FACP table memory at [mem 0xffe2215-0xffe2308][ 0.005041][ T0] ACPI: Reserving DSDT table memory at [mem 0xffe0040-0xff [ 0.005041][ T0] ACPI: Reserving DSDT table memory at [mem 0xffe0040-0xffe2214][ 0.005043][ T0] ACPI: Reserving FACS table memory at [mem 0xffe0000-0xffe003f][ 0.005046][ T0] ACPI: Reserving APIC table memory at [mem 0xffe2309-0xffe2398][ 0.005048][ T0] ACPI: Reserving HPET table memory at [mem 0xffe2399-0xffe23d0][ 0.005051][ T0] ACPI: Reserving MCFG table memory at [mem 0xffe23d1-0xffe240c][ 0.005053][ T0] ACPI: Reserving DMAR table memory at [mem 0xffe240d-0xffe249c][ 0.005055][ T0] ACPI: Reserving WAET table memory at [mem 0xffe249d-0xffe24c4][ 0.005090][ T0] Zone ranges:[ 0.005091][ T0] DMA [mem 0x0000000000001000-0x0000000000ffffff][ 0.005096][ T0] DMA32 [mem 0x0000000001000000-0x000000000ffdefff][ 0.005100][ T0] Normal empty[ 0.005103][ T0] Movable zone start for each node[ 0.005105][ T0] Early memory node ranges[ 0.005107][ T0] node 0: [mem 0x0000000000001000-0x000000000009efff][ 0.005109][ T0] node 0: [mem 0x0000000000100000 [ 0.005109][ T0] node 0: [mem 0x0000000000100000-0x000000000ffdefff][ 0.005112][ T0] Initmem setup node 0 [mem 0x0000000000001000-0x000000000ffdefff][ 0.005123][ T0] On node 0, zone DMA: 1 pages in unavailable ranges[ 0.005234][ T0] On node 0, zone DMA: 97 pages in unavailable ranges[ 0.006920][ T0] On node 0, zone DMA32: 33 pages in unavailable ranges[ 0.010804][ T0] kasan: KernelAddressSanitizer initialized[ 0.011013][ T0] ACPI: PM-Timer IO Port: 0x608[ 0.011022][ T0] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])[ 0.011064][ T0] IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23[ 0.011069][ T0] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)[ 0.011072][ T0] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)[ 0.011075][ T0] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)[ 0.011078][ T0] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)[ 0.011081][ T0] ACPI: INT_SRC_OVR (bus 0 bus_irq [ 0.011081][ T0] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)[ 0.011088][ T0] ACPI: Using ACPI (MADT) for SMP configuration information[ 0.011095][ T0] ACPI: HPET id: 0x8086a201 base: 0xfed00000[ 0.011100][ T0] TSC deadline timer available[ 0.011102][ T0] smpboot: Allowing 4 CPUs, 0 hotplug CPUs[ 0.011134][ T0] kvm-guest: APIC: eoi() replaced with kvm_guest_apic_eoi_write()[ 0.011143][ T0] kvm-guest: KVM setup pv remote TLB flush[ 0.011146][ T0] kvm-guest: setup PV sched yield[ 0.011175][ T0] [mem 0x10000000-0xafffffff] available for PCI devices[ 0.011178][ T0] Booting paravirtualized kernel on KVM[ 0.011181][ T0] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645519600211568 ns[ 0.015439][ T0] setup_percpu: NR_CPUS:64 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1[ 0.017472][ T0] percpu: Embedded 515 pages/cpu s2070824 r8192 d30424 u4194304[ 0.017485][ T0] pcpu-alloc: s2070824 r8192 d30424 [ 0.017485][ T0] pcpu-alloc: s2070824 r8192 d30424 u4194304 alloc=2*2097152[ 0.017491][ T0] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3 [ 0.017545][ T0] Kernel command line: rapido.vm_num=1 rd.systemd.unit=dracut-cmdline.service console=ttyS0 net.ifnames=0 syst1[ 0.017682][ T0] random: crng init done[ 0.017709][ T0] Dentry cache hash table entries: 32768 (order: 6, 262144 bytes, linear)[ 0.017725][ T0] Inode-cache hash table entries: 16384 (order: 5, 131072 bytes, linear)[ 0.017876][ T0] Built 1 zonelists, mobility grouping on. Total pages: 64223[ 0.018049][ T0] mem auto-init: stack:all(zero), heap alloc:off, heap free:off[ 0.018052][ T0] stackdepot: allocating hash table via alloc_large_system_hash[ 0.018689][ T0] stackdepot hash table entries: 1048576 (order: 11, 8388608 bytes, linear)[ 0.024444][ T0] Memory: 116924K/261620K available (12288K kernel code, 6239K rwdata, 4456K rodata, 3040K [ 0.024444][ T0] Memory: 116924K/261620K available (12288K kernel code, 6239K rwdata, 4456K rodata, 3040K init, 16516K bss, 1)[ 0.025160][ T0] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1[ 0.026890][ T0] Running RCU self tests[ 0.026892][ T0] Running RCU synchronous self tests[ 0.026910][ T0] rcu: Hierarchical RCU implementation.[ 0.026912][ T0] rcu: RCU lockdep checking is enabled.[ 0.026914][ T0] rcu: RCU restricting CPUs from NR_CPUS=64 to nr_cpu_ids=4.[ 0.026917][ T0] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.[ 0.026919][ T0] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4[ 0.027088][ T0] Running RCU synchronous self tests[ 0.043380][ T0] NR_IRQS: 4352, nr_irqs: 456, preallocated irqs: 16[ 0.044334][ T0] rcu: srcu_init: Setting srcu_struct sizes based on contention.[ 0.044448][ T0] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)[ 0.049395][ T0] Console: co [ 0.049395][ T0] Console: colour VGA+ 80x25[ 0.049454][ T0] printk: console [ttyS0] enabled[ 0.081844][ T0] Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., Ingo Molnar[ 0.082198][ T0] ... MAX_LOCKDEP_SUBCLASSES: 8[ 0.082401][ T0] ... MAX_LOCK_DEPTH: 48[ 0.082607][ T0] ... MAX_LOCKDEP_KEYS: 8192[ 0.082822][ T0] ... CLASSHASH_SIZE: 4096[ 0.083036][ T0] ... MAX_LOCKDEP_ENTRIES: 32768[ 0.083253][ T0] ... MAX_LOCKDEP_CHAINS: 65536[ 0.083471][ T0] ... CHAINHASH_SIZE: 32768[ 0.083687][ T0] memory used by lock dependency info: 7005 kB[ 0.083945][ T0] memory used for stack traces: 4224 kB[ 0.084175][ T0] per task-struct memory footprint: 2688 bytes[ 0.084558][ T0] ACPI: Core revision 20230628[ 0.085744][ T0] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns[ 0.086329][ T0] APIC: Switch to symmetric I/O mode setup[ [ 0.086571][ T0] DMAR: Host address width 39[ 0.086767][ T0] DMAR: DRHD base: 0x000000fed90000 flags: 0x0[ 0.087138][ T0] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap d2008c22260286 ecap f00f5e[ 0.087500][ T0] DMAR: ATSR flags: 0x1[ 0.087687][ T0] DMAR-IR: IOAPIC id 0 under DRHD base 0xfed90000 IOMMU 0[ 0.087986][ T0] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.[ 0.089006][ T0] DMAR-IR: Enabled IRQ remapping in x2apic mode[ 0.089273][ T0] x2apic enabled[ 0.089621][ T0] APIC: Switched APIC routing to: cluster x2apic[ 0.089890][ T0] kvm-guest: APIC: send_IPI_mask() replaced with kvm_send_ipi_mask()[ 0.090287][ T0] kvm-guest: APIC: send_IPI_mask_allbutself() replaced with kvm_send_ipi_mask_allbutself()[ 0.090696][ T0] kvm-guest: setup PV IPIs[ 0.093858][ T0] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1[ 0.094209][ T0] clocksource: tsc-early: m [ 0.094209][ T0] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x325ea749ca1, max_idle_ns: 440795373125 ns[ 0.094691][ T0] Calibrating delay loop (skipped) preset value.. 6988.80 BogoMIPS (lpj=13977600)[ 0.095139][ T0] x86/cpu: User Mode Instruction Prevention (UMIP) activated[ 0.095451][ T0] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0[ 0.095704][ T0] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0[ 0.095985][ T0] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization[ 0.096371][ T0] Spectre V2 : Mitigation: Enhanced / Automatic IBRS[ 0.096624][ T0] Spectre V2 : Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch[ 0.096970][ T0] Spectre V2 : Spectre v2 / PBRSB-eIBRS: Retire a single CALL on VMEXIT[ 0.097284][ T0] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier[ 0.097635][ T0] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl[ 0.0980 [ 0.098010][ T0] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'[ 0.098694][ T0] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'[ 0.098978][ T0] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'[ 0.099264][ T0] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'[ 0.099606][ T0] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256[ 0.099882][ T0] x86/fpu: xstate_offset[9]: 832, xstate_sizes[9]: 8[ 0.100161][ T0] x86/fpu: Enabled xstate features 0x207, context size is 840 bytes, using 'compacted' format.[ 0.102144][ T0] debug: unmapping init [mem 0xffffffff82ddb000-0xffffffff82ddefff][ 0.102488][ T0] pid_max: default: 32768 minimum: 301[ 0.103709][ T0] Mount-cache hash table entries: 512 (order: 0, 4096 bytes, linear)[ 0.104036][ T0] Mountpoint-cache hash table entries: 512 (order: 0, 4096 bytes, linear)[ 0.108060][ T0] Running RCU synchronous self tests[ 0.108277][ T0] Running RCU synchronous self tests[ 0.108877][ T1] smpboot: CPU0: Intel(R) Core(TM) i5-14600K (family: 0x6, model: 0xb7, stepping: 0x1)[ 0.110690][ [ 0.110690][ T1] Performance Events: unsupported p6 CPU model 183 no PMU driver, software events only.[ 0.110690][ T1] signal: max sigframe size: 3632[ 0.110690][ T1] rcu: Hierarchical SRCU implementation.[ 0.110690][ T1] rcu: Max phase no-delay instances is 1000.[ 0.110690][ T1] NMI watchdog: Perf NMI watchdog permanently disabled[ 0.111231][ T1] smp: Bringing up secondary CPUs ...[ 0.112450][ T1] smpboot: x86: Booting SMP configuration:[ 0.112697][ T1] .... node #0, CPUs: #1 #2 #3[ 0.114933][ T1] smp: Brought up 1 node, 4 CPUs[ 0.115112][ T1] smpboot: Max logical packages: 1[ 0.115318][ T1] smpboot: Total of 4 processors activated (27955.20 BogoMIPS)[ 0.119328][ T1] devtmpfs: initialized[ 0.124244][ T1] Running RCU synchronous self tests[ 0.124244][ T1] clocksource: jiffies: mask: 0xffffff [ 0.124244][ T1] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns[ 0.124302][ T1] futex hash table entries: 1024 (order: 5, 131072 bytes, linear)[ 0.128563][ T1] NET: Registered PF_NETLINK/PF_ROUTE protocol family[ [ 0.133151][ T1] thermal_sys: Registered thermal governor 'step_wise'[ 0.133155][ T1] thermal_sys: Registered thermal governor 'user_space'[ 0.133630][ T1] cpuidle: using governor menu[ 0.135390][ T1] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xb0000000-0xbfffffff] (base 0xb0000000)[ 0.135794][ T1] PCI: MMCONFIG at [mem 0xb0000000-0xbfffffff] reserved as E820 entry[ 0.136170][ T1] PCI: Using configuration type 1 for base access[ 0.139762][ T1] ACPI: Added _OSI(Module Device)[ 0.139787][ T1] ACPI: Added _OSI(Processor Device)[ 0.140002][ T1] ACPI: Added _OSI(3.0 _SCP Extensions)[ 0.140224][ T1] ACPI: Added _OSI(Processor Aggregator Device)[ 0.241334][ T1] ACPI: 1 ACPI AML tables successfully acquired and loaded[ 0.25157 [ 0.251572][ T1] ACPI: _OSC evaluation for CPUs failed, trying _PDC[ 0.261214] [ 0.261214][ T1] ACPI: Interpreter enabled[ 0.261523][ T1] ACPI: PM: (supports S0 S5)[ 0.261718][ T1] ACPI: Using IOAPIC for interrupt routing[ 0.262486][ T1] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug[ 0.262700][ T1] PCI: Using E820 reservations for host bridge windows[ 0.266847][ T1] ACPI: Enabled 2 GPEs in block 00 to 3F[ 0.362816][ T1] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])[ 0.363201][ T1] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI EDR HPX-Type3][ 0.367204][ T1] acpi PNP0A08:00: _OSC: platform does not support [PCIeHotplug LTR DPC][ 0.373937 [ 0.373937][ T1] acpi PNP0A08:00: _OSC: OS now controls [PME AER PCIeCapability][ 0.375702][ T1] PCI host bridge to bus 0000:00[ 0.375915][ T1] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window][ 0.376234][ T1] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window][ 0.376550][ T1] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window][ 0.376894][ T1] pci_bus 0000:00: root bus resource [mem 0x10000000-0xafffffff window][ 0.377240][ T1] pci_bus 0000:00: root bus resource [mem 0x [ 0.377240][ T1] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window][ 0.377584][ T1] pci_bus 0000:00: root bus resource [mem 0x100000000-0x8ffffffff window][ 0.377935][ T1] pci_bus 0000:00: root bus resource [bus 00-ff][ 0.378419][ T1] pci_bus 0000:00: scanning bus[ 0.378728][ T1] pci 0000:00:00.0: [8086:29c0] type 00 class 0x060000[ 0.379929][ T1] pci 0000:00:00.0: EDR: Notify handler installed[ 0.381968][ T1] pci 0000:00:01.0: [1234:1111] type 00 class 0x030000[ 0.383173][ T1] pci 0000:00:01.0: reg 0x10: [mem 0xfd000000-0xfdffffff pref][ 0.385105][ T1] pci 0000:00:01.0: reg 0x18: [mem 0xfebd0000-0xfebd0fff][ 0.388289][ T1] pci 0000:00:01.0: reg 0x30: [mem 0xfebc000 [ 0.388289][ T1] pci 0000:00:01.0: reg 0x30: [mem 0xfebc0000-0xfebcffff pref][ 0.388726][ T1] pci 0000:00:01.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff][ 0.389726][ T1] pci 0000:00:01.0: EDR: Notify handler installed[ 0.391726][ T1] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000[ 0.393112][ T1] pci 0000:00:02.0: reg 0x10: [io 0xc040-0xc05f][ 0.393833][ T1] pci 0000:00:02.0: reg 0x14: [mem 0xfebd1000-0xfebd1fff][ 0.395498][ T1] pci 0000:00: [ 0.395498][ T1] pci 0000:00:02.0: reg 0x20: [mem 0xfe010000-0xfe013fff 64bit pref][ 0.396612][ T1] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref][ 0.397840][ T1] pci 0000:00:02.0: EDR: Notify handler installed[ 0.399809][ T1] pci 0000:00:03.0: [1af4:1005] type 00 class 0x00ff00[ 0.400928][ T1] pci 0000:00:03.0: reg 0x10: [io 0xc060-0xc07f][ 0.401990][ T1] pci 0000:00:03.0: reg 0x14: [mem 0xfebd2000-0xfebd2fff][ 0.404249][ T1] pci 0000:00:03.0: reg 0x20: [mem 0xfe014000-0xfe017fff 64bit pref][ 0.405861][ T1] pci 0000:00:03.0: EDR: Notify handler installed[ 0.407861][ T1] pci 0000:00:04.0: [1fc9:4022] type 00 class 0x020000[ 0.408546][ T1] pci 0000:00:04.0: reg 0x10: [mem 0xfe000000-0xfe00ffff 64bit pref][ 0.410570][ T1] pci 0000:00:04.0: supports D1 D2[ 0.411402][ T1] pci 0000:00:04.0: EDR: Notify handler installed[ 0.413913][ T1] pci 0000:00:1f.0: [8086:2918] type 00 class 0x060100[ 0.414471][ T1] pci 0000:00:1f.0: quirk: [io 0x0600-0x067f] claimed by ICH6 ACPI/GPIO/TCO[ 0.415348][ T1] pci 0000:00:1f.0: EDR: Notify handler installed[ 0.417298][ T1] pci 0000:00:1f.2: [8086:2922] type 00 class 0x010601[ 0.418821][ T1] pci 0000:00:1f.2: reg 0x20: [io 0xc080-0xc09f][ 0.419329][ T1] pci 0000:00:1f.2: reg 0x24: [mem 0xfebd3000-0xfebd3fff][ 0.4 [ 0.420705][ T1] pci 0000:00:1f.2: EDR: Notify handler installed[ 0.422607][ T1] pci 0000:00:1f.3: [8086:2930] type 00 class 0x0c0500[ 0.424087][ T1] pci 0000:00:1f.3: reg 0x20: [io 0x0700-0x073f][ 0.425351][ T1] pci 0000:00:1f.3: EDR: Notify handler installed[ 0.427195][ T1] pci_bus 0000:00: fixups for bus[ 0.427398][ T1] pci_bus 0000:00: bus scan returning with max=00[ 0.427654][ T1] pci_bus 0000:00: on NUMA node 0[ 0.434545] [ 0.434545][ T1] ACPI: PCI: Interrupt link LNKA configured for IRQ 10[ 0.438055][ T1] ACPI: PCI: Interrupt link LNK [ 0.438055][ T1] ACPI: PCI: Interrupt link LNKB configured for IRQ 10[ 0.441613][ T1] ACPI: PCI: Interrupt link LNKC con [ 0.441613][ T1] ACPI: PCI: Interrupt link LNKC configured for IRQ 11[ 0.445176][ T1] ACPI: PCI: Interrupt link [ 0.445176][ T1] ACPI: PCI: Interrupt link LNKD configured for IRQ 11[ 0.448732][ T1] ACPI [ 0.448732][ T1] ACPI: PCI: Interrupt link LNKE configured for IRQ 10[ 0.452513][ T1] ACPI: PCI: Interrupt link LNKF configured for IRQ 10[ 0.456173][ T1] ACPI: PCI: Interrupt link LNKG configured for IRQ 11[ 0.459795][ T1] ACPI: PCI: Interrupt link LNKH configured for IRQ 11[ 0.460831][ T1] ACPI: PCI: Interrupt link GSIA configured for IRQ 16[ 0.461568][ T1] ACPI: PCI: Interrupt link GSIB configured for IRQ 17[ 0.462296][ T1] ACPI: PCI: Interrupt link GSIC configured for IRQ 18[ 0.463026][ T1] ACPI: PCI: Interrupt link GSID configured for IRQ 19[ 0.463754][ T1] ACPI: PCI: Interrupt link GSIE configured for IRQ 20[ 0.464485][ [ 0.464485][ T1] ACPI: PCI: Interrupt link GSIF configured for IRQ 21[ 0.465219][ T1] ACPI: PCI: Interrupt link GSIG configured for IRQ 22[ 0.465951][ T1] ACPI: PCI: Interrupt link GSIH configured for IRQ 23[ 0.479112 [ 0.479112][ T1] iommu: Default domain type: Translated[ 0.479355][ T1] iommu: DMA domain TLB invalidation policy: lazy mode[ 0.479973][ T1] pps_core: LinuxPPS API ver. 1 registered[ 0.480203][ T1] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>[ 0.480682][ T1] PTP clock support registered[ 0.484160][ T1] PCI: Using ACPI for IRQ routing[ 0.508292 [ 0.508292][ T1] PCI: pci_cache_line_size set to 64 bytes[ 0.508538][ T1] pci 0000:00:01.0: BAR 0: reserving [mem 0xfd000000-0xfdffffff flags 0x42208] (d=0, p=0)[ 0.508941][ T1] pci 0000:00:01.0: BAR 2: reserving [mem 0xfebd0000-0xfebd0fff flags 0x40200] (d=0, p=0)[ 0.509336][ T1] pci 0000:00:02.0: BAR 0: reserving [io 0xc040-0xc05f flags 0x40101] (d=0, p=0)[ 0.509709][ T1] pci 0000:00:02.0: BAR 1: reserving [mem 0xfebd1000-0xfebd1fff flags 0x40200] (d=0, p=0)[ 0.510100][ T1] pci 0000:00:02.0: BAR 4: reserving [mem 0xfe010000-0xfe013fff flags 0x14220c] (d=0, p=0)[ 0.510515][ T1] pci 0000:00:03.0: BAR 0: reserving [io 0xc060-0xc07f flags 0x40101] (d=0, p=0)[ 0.510699][ T1] pci 0000:00:03.0: BAR 1: reserving [mem 0xfebd2000-0xfebd2fff flags 0x40200] (d=0, p=0)[ 0.511168][ T1] pci 0000:00:03.0: BAR 4: reserving [mem 0xfe014000-0xfe017fff flags 0x14220c] (d=0, p=0)[ [ 0.511641][ T1] pci 0000:00:04.0: BAR 0: reserving [mem 0xfe000000-0xfe00ffff flags 0x14220c] (d=0, p=0)[ 0.512119][ T1] pci 0000:00:1f.2: BAR 4: reserving [io 0xc080-0xc09f flags 0x40101] (d=0, p=0)[ 0.512546][ T1] pci 0000:00:1f.2: BAR 5: reserving [mem 0xfebd3000-0xfebd3fff flags 0x40200] (d=0, p=0)[ 0.513015][ T1] pci 0000:00:1f.3: BAR 4: reserving [io 0x0700-0x073f flags 0x40101] (d=0, p=0)[ 0.513427][ T1] e820: reserve RAM buffer [mem 0x0009fc00-0x0009ffff][ 0.513739][ T1] e820: reserve RAM buffer [mem 0x0ffdf000-0x0fffffff][ 0.516545][ T1] pci 0000:00:01.0: vgaarb: setting as b [ 0.516545][ T1] pci 0000:00:01.0: vgaarb: setting as boot VGA device[ 0.516545][ T1] pci 0000:00:01.0: vgaarb: bridge control possible[ 0.516545][ T1] pci 0000:00:01.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none[ 0.516545][ T1] vgaarb: loaded[ 0.516545][ T1] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0[ 0.516545][ T1] hpet0: 3 comparators, 64-bit 100.000000 MHz counter[ 0.528677] [ 0.528677][ T1] clocksource: Switched to clocksource kvm-clock[ 0.530307][ T1] pnp: PnP ACPI init[ 0.534823][ T1] syste [ 0.534823][ T1] system 00:05: [mem 0xb0000000-0xbfffffff window] has been reserved[ 0.544608][ [ 0.544608][ T1] pnp: PnP ACPI: found 6 devices[ 0.578651][ T1] cloc [ 0.578651][ T1] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns[ 0.579484][ T1] NET: Registered PF_INET protocol family[ 0.579911][ T1] IP idents hash table entries: 4096 (order: 3, 32768 bytes, linear)[ 0.582346][ T1] tcp_listen_portaddr_hash [ 0.582346][ T1] tcp_listen_portaddr_hash hash table entries: 128 (order: 1, 10240 bytes, linear)[ 0.582730][ T1] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)[ 0.583150][ T1] TCP established hash table entries: 2048 (order: 2, 16384 bytes, linear)[ 0.583555][ T1] TCP bind hash table entries: 2048 (order: 6, 327680 bytes, linear)[ 0.583996][ T1] TCP: Hash tables configured (established 2048 bind 2048)[ 0.584536][ T1] UDP hash table entries: 256 (order: 3, 49152 bytes, linear)[ 0.584858][ T1] UDP-Lite hash table entries: 256 (order: 3, 49152 bytes, linear)[ 0.585878][ T1] NET: Registered PF_UNIX/PF_LOCAL protocol family[ 0.586224][ T1] pci [ 0.586224][ T1] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window][ 0.586498][ T1] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window][ 0.586761][ T1] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window][ 0.587050][ T1] pci_bus 0000:00: resource 7 [mem 0x10000000-0xafffffff window][ 0.587341][ T1] pci_bus 0000:00: resource 8 [mem 0xc0000000-0xfebfffff window][ 0.587630][ T1] pci_bus 0000:00: resource 9 [mem 0x100000000-0x8ffffffff window][ 0.588346][ T1] PCI: CLS 64 bytes, default 64[ 0.588650][ T1] DMAR: No RMRR found[ 0.588813][ T1] DMAR: No SATC found[ 0.588964][ T1] DMAR: dmar0: Using Queued invalidation[ 0.589259][ T1] DMAR: IOMMU batching disallowed due to virtualization[ 0.589700][ T11] Unpacking initramfs...[ 0.590396][ T1] pci 0000:00:00.0: Adding to iommu group 0[ 0.590880][ T1] pci 0000:00:01.0: Adding to iommu group 1[ 0.591336][ T1] pci 0000:00:02.0: Adding to iommu group 2[ 0.591789][ T1] pci 0000:00:03.0: Adding to iommu group 3[ 0.592244][ T1] pci 0000:00:04.0: Adding to iommu group 4[ 0.592728][ T1] pci 0000:00:1f.0: Adding to iommu group 5[ 0.593092][ T1] pci 0000:00:1f.2: Adding to iommu group 5[ 0.593456][ T1] pci 0000:00:1f.3: Adding to iommu group 5[ 0.617366] [ 0.617366][ T1] DMAR: Intel(R) Virtualization Technology for Directed I/O[ 0.618107][ T1] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer[ 0.618567][ T1] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x325ea749ca1, max_idle_ns: 440795373125 ns[ 0.62724 [ 0.627248][ T1] workingset: timestamp_bits=46 max_order=15 bucket_order=0[ 0.628145][ T1] 9p: Installing v9fs 9p2000 file system support[ 0.628600][ T1] io scheduler mq-deadline registered[ 0.628816][ T1] io scheduler kyber registered[ 0.629559][ T1] pciehp: pcie_port_service_register = 0[ 0.633521][ T1] virtio-pci 0000:00:02.0: vgaarb: pci_notify[ 0.633833][ T1] virtio-pci 0000:00:02.0: runtime IRQ mapping not provided by arch[ 0.659840 [ 0.659840][ T1] ACPI: \_SB_.GSIG: Enabled at IRQ 22[ 0.660995][ T1] virtio-pci 0000:00:02.0: enabling bus mastering[ 0.661925][ T1] virtio-pci 0000:00:02.0: vgaarb: pci_notify[ 0.662229][ T1] virtio-pci 0000:00:03.0: vgaarb: pci_notify[ 0.662525][ T1] virtio-pci 0000:00:03.0: runtime IRQ mapping not provided by arch[ 0.686909][ T11] deb [ 0.686909][ T11] debug: unmapping init [mem 0xffff88800de9d000-0xffff88800ffcffff][ 0.688516][ T1] ACPI: \_SB_.GSIH: Enabled at IRQ 23[ 0.689545][ T1] virtio-pci 0000:00:03.0: enabling bus mastering[ 0.690370][ T1] virtio-pci 0000:00:03.0: vgaarb: pci_notify[ 0.691418][ T1] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled[ 0.693077][ T1] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A[ [ 0.716991][ T1] VFIO - User Level meta-driver version: 0.3[ 0.717804][ T1] intel_pstate: CPU model not supported[ 0.718829][ T1] NET: Registered PF_INET6 protocol family[ 0.722187][ T1] Segment Routing with IPv6[ 0.722500][ T1] In-situ OAM (IOAM) with IPv6[ 0.722810][ T1] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver[ 0.725625][ T1] 8021q: 802.1 [ 0.725625][ T1] 8021q: 802.1Q VLAN Support v1.8[ 0.725927][ T1] 9pnet: Installing 9P2000 support[ 0.726338][ T1] IPI shorthand broadcast: enabled[ 0.753535][ T1] sched_clock: Marking stable (712002168, 38350193)->(754854527, -4502166)[ 0.810015][ T1] kmemle [ 0.810015][ T1] kmemleak: Kernel memory leak detector initialized (mem pool available: 14477)[ 0.810020][ T46] kmemleak: Automatic memory scanning thread started[ 0.810435][ T1] page_owner is disabled[ 0.812309][ T1] KTAP version 1[ 0.812454][ T1] 1..0[ 0.813368][ T1] debug: unmapping init [mem 0xffffffff82ae3000-0x [ 0.813368][ T1] debug: unmapping init [mem 0xffffffff82ae3000-0xffffffff82ddafff][ 0.834592] [ 0.834592][ T1] Write protecting the kernel read-only data: 18432k[ 0.836282][ T1] debug: unmapping init [mem 0xffffffff8205a000-0xffffffff821fffff][ 0.836875][ T1] Run /init as init process[ 0.837057][ T1] with arguments:[ 0.837206][ T1] /init[ 0.837333][ T1] with environment:[ 0.837488][ T1] HOME=/[ 0.837612][ T1] TERM=linux[ 0.854144][ T47 [ 0.854144][ T47] mount (47) used greatest stack depth: 27160 bytes left[ 1.233479][ T1 [ 1.233479][ T1] dracut: Fedora Linux-39[ 1.326719][ T112] NET: Re [ 1.326719][ T112] NET: Registered PF_PACKET protocol familyWaiting for network to come onli Waiting for network to come online...Enumeration comp Enumeration completedlo: Configuring with /rapido-rsc/net/vm1/lo.network.eth0: Configuring with /rapido-rsc/net/vm1/rapido-tap1.network.[ 1.550970][ T125] 8021q: adding VLAN 0 to HW filter on device eth0lo: Link UPlo: Gained carriereth0: Link UPeth0: Gai eth0: Gained carrierRapido: starting /rapido_autorun Rapido: starting /rapido_autorun/00-net_test_tn40xx.sh[ 1.780000][USPA [ 1.780000][USPACE] Starting udevd[ 1.780000][USPACE] Loading Driver[ 1.835177][ T1 [ 1.835177][ T132] tn40xx: loading out-of-tree module taints kernel.[ 1.837277][ T132] Tehuti Network Driver from https://github.com/acooks/tn40xx-driver, 004[ 1.837498][ T132] Supported phys : QT2025 TLK10232 AQR105 MUSTANG [ 1.837743][ T132] tn40xx 0000:00:04.0: vgaarb: pci_notify[ 1.83 [ 1.837939][ T132] tn40xx 0000:00:04.0: runtime IRQ mapping not provided by archUsing default in Using default interface naming scheme 'v253'.[ 1.867307][ T132 [ 1.867307][ T132] ACPI: \_SB_.GSIE: Enabled at IRQ 20[ 1.868064][ T132] tn40xx 0000:00:04.0: enabling bus mastering[ 1.868546][ T132] srom 0x0 HWver 16 build 0 lane# 0 max_pl 0x2 mrrs 0x2[ 2.122702][ T132] [ 2.122702][ T132] PHY detected on port 1 ID=43A400 - QT2025 10Gbps SFP+[ 3.886944][ T132 [ 3.886944][ T132] QT2025 FW version 2.0.3.3 module type 0x4eth1: Co eth1: Configuring with /rapido-rsc/net/vm1/foo.network.[ 3.996070][ T [ 3.996070][ T132] fw 0xe[ 3.996469][ T132] eth1, Port A[ 3.996586][ T132] 1 1fc9:4022:1fc9:3015[ 3.996771][ T132] detected 1 cards, 1 loaded[ 3.996950][ T132] tn40xx 0000:00:04.0: vgaarb: pci_notify[ 4.004835][ [ 4.004835][ T132] modprobe (132) used greatest stack depth: 24952 bytes left[ 3.960000][USPACE] Waiting for dynamic IP addresses...[ 4.036294][ T125] 8 [ 4.036294][ T125] 8021q: adding VLAN 0 to HW filter on device eth1eth1: Link UP[ 6.269004][ C1 [ 6.269004][ C1] eth1 Link Up 10Geth1: Gained carrier[ 6.637132][ T125] [ 6.637132][ T125] ================================================================================[ 6.638606][ T125] UBSAN: array-index-out-of-bounds in /home/acooks/tn40xx-driver/tn40.c:2952:32[ 6.639759][ T125] index 1 is out of range for type 'pbl [*]'[ 6.640532][ T125] CPU: 2 PID: 125 Comm: systemd-network Tainted: G O 6.6.1 #13[ 6.641686][ T125] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-2.fc39 04/01/2014[ 6.642875][ T125] Call Trace:[ 6.643319][ T125] <TASK>[ 6.643692][ T125] dump_stack_lvl+0x48/0x5b[ 6.643817][ T125] ubsan_epilogue+0x5/0x2c[ 6.643936][ T125] __ubsan_handle_out_of_bounds+0x64/0x6d[ 6.644086][ T125] bdx_tx_transmit+0x66c/0x921 [tn40xx][ 6.644238][ T125] netdev_start_xmit+0x42/0x8e[ 6.644366][ T125] dev_hard_start_xmit+0x109/0x1fe[ 6.644501][ T125] sch_direct_xmit+0x11c/0x34a[ 6.644629][ T125] ? pfifo_fast_reset+0xf9/0xf9[ 6.644757][ T125] ? do_raw_spin_loc [ 6.644757][ T125] ? do_raw_spin_lock+0x151/0x151[ 6.644891][ T125] ? rcu_read_lock_held_common+0x34/0x3c[ 6.645039][ T125] __dev_queue_xmit+0x93e/0xd28[ 6.645167][ T125] ? copy_page_from_iter_atomic+0x663/0x663[ 6.645324][ T125] ? netdev_core_pick_tx+0x10c/0x10c[ 6.645463][ T125] ? packet_parse_headers+0x2c5/0x2f6 [af_packet][ 6.645635][ T125] ? prb_retire_rx_blk_timer_expired+0x1ab/0x1ab [af_packet][ 6.645832][ T125] ? skb_copy_datagram_from_iter+0x7e/0x24c[ 6.645986][ T125] ? packet_sock_flag+0x18/0x26 [af_packet][ 6.646145][ T125] ? packet_xmit+0x17/0x12d [af_packet][ 6.646300][ T125] packet_sendmsg+0x17f6/0x187b [af_packet][ 6.646460][ T125] ? lock_acquire+0x19b/0x2e1[ 6.646584][ T125] ? __might_fault+0x72/0xba[ 6.646708][ T125] ? lock_time_inc+0x50/0x5f[ 6.646830][ T125] ? lock_release+0x292/0x37e[ 6.646955][ T125] ? packet_extra_vlan_len_allowed.isra.0+0x75/0x75 [af_packet][ 6.647156][ T125] ? reacquire_held_loc [ 6.647156][ T125] ? reacquire_held_locks+0x1bd/0x1bd[ 6.647299][ T125] ? packet_extra_vlan_len_allowed.isra.0+0x75/0x75 [af_packet][ 6.647495][ T125] sock_sendmsg_nosec+0x42/0x7e[ 6.647625][ T125] __sys_sendto+0x191/0x1ea[ 6.647746][ T125] ? __ia32_sys_getpeername+0x42/0x42[ 6.647887][ T125] ? rcu_is_watching+0x23/0x34[ 6.648013][ T125] ? __rseq_handle_notify_resume+0x609/0x654[ 6.648170][ T125] ? ksys_read+0xf8/0x11a[ 6.648289][ T125] __x64_sys_sendto+0x75/0x81[ 6.648413][ T125] do_syscall_64+0x8a/0xaa[ 6.648533][ T125] entry_SYSCALL_64_after_hwframe+0x46/0xb0[ 6.648688][ T125] RIP: 0033:0x7f52e1cb8a87[ 6.648806][ T125] Code: c7 c0 ff ff ff ff eb be 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 80 3d e5 88 0c 00 00 41 89 ca 74 0[ 6.649299][ T125] RSP: 002b:00007ffda3436398 EFLAGS: 00000202 ORIG_RAX: 000000000000002c[ 6.649517][ T125] RAX: fffffffffffff [ 6.649517][ T125] RAX: ffffffffffffffda RBX: 00005618cd2374bc RCX: 00007f52e1cb8a87[ 6.649721][ T125] RDX: 000000000000001c RSI: 00007ffda34363b0 RDI: 0000000000000010[ 6.649925][ T125] RBP: 00007ffda3436480 R08: 00007ffda34363d0 R09: 0000000000000014[ 6.650129][ T125] R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000000010[ 6.650335][ T125] R13: 00005618cd2374b8 R14: 0000000000000000 R15: 00007ffda3436540[ 6.650539][ T125] </TASK>[ 6.650630][ T125] ================================================================================eth1: Gained IPv eth1: Gained IPv6LLeth1: DHCPv6 address eth1: DHCPv6 address 2403:580a:d3e9::717/128 (valid for 1h 56min 35s, preferred for 56min 34s)eth1: DHCPv4 ad eth1: DHCPv4 address 10.1.4.116/24, gateway 10.1.4.1 acquired from 10.1.4.1[ 11.240000][U [ 11.240000][USPACE] Starting iperf3Connecting to ho Connecting to host 10.1.4.153, port 5201[ 5] local 10.1.4.116 port 41094 connected to 10.1.4.153 port 5201[ ID] Interval Transfer [ ID] Interval Transfer Bitrate Retr Cwnd[ 5] 0.00-1.00 sec 113 MBytes 948 Mbits/sec 2 300 KBytes [ 5] 1.00-2.0 [ 5] 1.00-2.00 sec 112 MBytes 941 Mbits/sec 47 305 KBytes [ 5] [ 5] 2.00-3.00 sec 112 MBytes 942 Mbits/sec 0 300 KBytes [ 5] 3.00-4.00 [ 5] 3.00-4.00 sec 112 MBytes 939 Mbits/sec 97 291 KBytes [ 5] 4.00-5.00 se [ 5] 4.00-5.00 sec 112 MBytes 941 Mbits/sec 2 303 KBytes [ 5] 5.00-6.0 [ 5] 5.00-6.00 sec 112 MBytes 943 Mbits/sec 0 297 KBytes [ 5] 6.00-7.00 sec [ 5] 6.00-7.00 sec 112 MBytes 941 Mbits/sec 19 303 KBytes [ 5] [ 5] 7.00-8.00 sec 112 MBytes 940 Mbits/sec 5 300 KBytes [ 5] [ 5] 8.00-9.00 sec 112 MBytes 942 Mbits/sec 6 300 KBytes [ 5] 9.00-10.00 sec 112 MB [ 5] 9.00-10.00 sec 112 MBytes 942 Mbits/sec 0 5.66 KBytes - - - - - - - - - - - - - - - - - - - - - - - - -[ ID] Interval Transfer Bitrate Retr[ 5] 0.00-10.00 sec 1.10 GBytes 942 Mbits/sec 178 sender[ 5] 0.00-10.00 sec 1.10 GBytes 941 Mbits/sec receiveriperf Done.[ 21.260000][USPACE] You can now do manual testing...rapido1:/# rapido1:/# e rapido1:/# et rapido1:/# eth rapido1:/# etht rapido1:/# ethto rapido1:/# ethtoo rapido1:/# ethtool rapido1:/# ethtool rapido1:/# ethtool - rapido1:/# ethtool -i rapido1:/# ethtool -i rapido1:/# ethtool -i e rapido1:/# ethtool -i et rapido1:/# ethtool -i eth rapido1:/# ethtool -i eth1 rapido1:/# ethtool -i eth1driver: tn40xxversion: 004 version: 004firmware-version: N/Aexpansion-rom-version: bus-info: 0000:00:04.0supports-statistics: yessupports-test: nosupports-eeprom-access: nosupports-register-dump: nosupports-priv-flags: norapido1:/# logout rapido1:/# logout[ 51.211864][ T1] sysrq: Power Offqemu-system-x86_64: vtd_interrupt_remap_msi: MSI address low 32 bit invalid: 0x0[ 51.522270][ T8] Device removed[ 51.522535][ T8] ACPI: PM: Preparing to enter system sleep state S5[ 51.523155][ T8] reboot: Power down 0:00/0:00

Here's what's happening:

  1. The test environement is started with one simple command: ./rapido cut net-test-tn40xx
  2. A new initramfs is created;
  3. Qemu boots a new VM, passing a real PCI network device from the host to the VM;
  4. The kernel boots and loads the initramfs containing the test tools;
  5. The developmental tn40xx driver loads;
  6. Ethernet carrier is negotiated and IP addresses are assigned;
  7. The Undefined Behavior Sanitizer (UBSAN) detects a bug in the tn40xx driver;
  8. iperf3 client performs a simple tcp goodput test;
  9. VM exits

Initramfs creation

Rapido takes care of building an initramfs (by calling dracut) that contains the software under test, test tools and scripted test steps.

Rapido uses a 'cut script' to define:

  • the name of the autorun test script (reminds me a bit of classic BSD-style init scripts, in a good way).
  • the VM resources (2 cores, 512M RAM)
  • additional drivers to install in the initramfs
  • userspace tools to install in the initramfs

Here is the 'cut' script used by rapido for generating the initramfs:

#!/bin/bash
# SPDX-License-Identifier: (LGPL-2.1 OR LGPL-3.0)
# Copyright (C) SUSE LLC 2022, all rights reserved.

RAPIDO_DIR="$(realpath -e ${0%/*})/.."
. "${RAPIDO_DIR}/runtime.vars"

_rt_require_dracut_args "$RAPIDO_DIR/autorun/net_test_tn40xx.sh" "$@"
_rt_require_networking
_rt_cpu_resources_set "2"
_rt_mem_resources_set "512M"

"$DRACUT" \
        --install "hostname lspci ip bridge ethtool iperf3 nc" \
        --add-drivers "tn40xx" \
        --modules "base" \
        "${DRACUT_RAPIDO_ARGS[@]}" \
        "$DRACUT_OUT" || _fail "dracut failed"

Virtual Machine definition

Rapido also starts qemu after creating the initramfs. I've configured it to add a virtual IOMMU and pass the physical PCIe NIC to the VM.

Here's the snippet of the QEMU extra arguments in my rapido.conf file:

QEMU_EXTRA_ARGS="\
 -M q35,accel=kvm,kernel-irqchip=split \
 -cpu host \
 -nographic\
 -device intel-iommu,intremap=on,caching-mode=on,device-iotlb=on \
 -device virtio-rng \
 -device vfio-pci,host=07:00.0 \
"

The passed-through vfio-pci device would be better in a cut script, because it doesn't apply to all VMs. TODO

Kernel boots

The kernel running here is the mainline 6.6.1 version. I've compiled it with debugging options appropriate for the driver development it's used for here. The ~1.5 second boot time is pretty good, although the debugging options makes the output more verbose and also has a noticeable performance cost.

Driver loads, tests happen

The test script in the screencast is this:

#!/bin/bash
# SPDX-License-Identifier: (LGPL-2.1 OR LGPL-3.0)
# Copyright (C) SUSE LLC 2022, all rights reserved.

_vm_ar_env_check || exit 1
_vm_ar_dyn_debug_enable

now() {
        declare -a _uptime
        readarray -d " " -t _uptime < /proc/uptime
        printf "[%12s][USPACE] " "${_uptime[0]}0000"
}

now; echo "Starting udevd"

/usr/lib/systemd/systemd-udevd &

now; echo "Loading Driver"
modprobe tn40xx || _fatal "Failed to load tn40xx module"

now; echo "Waiting for dynamic IP addresses..."
READY=$(ip -oneline -4 addr show dev eth1 scope global)

while [ -z "$READY" ] ; do
        sleep 1
        READY=$(ip -oneline -4 addr show dev eth1 scope global)
done

now; echo "Starting iperf3"
iperf3 -c 10.1.4.153

now; echo "You can now do your manual testing..."

I thought it would be nice to have timestamps on the userspace output as well, so I added a little function for it.

There are opportunities for improvement here too:

  • a number of hardcoded system attributes that should be moved out of the test logic, eg. eth1;
  • the test results should be stored for later analysis;

The actual test steps for a network interface card need some thought. Something like this:

  • Cover various ioctls using ethtool.
  • Add/remove change VLANs
  • Change MTU
  • Promiscuous mode
  • Receive Multicast groups
  • TSO / GSO
  • driver unload / module removal
  • ...

But this is a start :)

At some point I should also get a proper reference NIC like an Intel X520-DA2 as a benchmark. Update: Got an X550 T2 on loan from a friend. Thanks, Simeon!

Conclusion

An efficient, repeatable test workflow is essential for all development. This is my first meaningful progress towards developing the tn40xx driver in five years and I can now start thinking about what tests are useful, doing regular releases and tackling the major rework it needs.

It's also quite satisfying to be able to progress three topics I find interesting in one post:

  • development of the tn40xx driver
  • real hardware device passthrough to VMs, with IOMMU memory protection
  • ephemeral Linux-based micro VMs

Let the healing begin!


Sat 11 March 2023

Filed under Linux Networking

On a good day, my day job involves building networking tools in python. Too many python networking tools look like shell scripts, spawning subprocesses for basic tools like 'ping' or 'ip' - often resulting in a fragile mess due to poor, or inconsistent error handling.

I was quite excited to find …

Read More

Mon 28 March 2016

Filed under Linux Networking

Tags Linux Networking

New hardware for BufferUpr has arrived! I'm working on adding Wifi capability into the product and with new hardware comes new challenges.

Challenge: Find a simple way to list all the network interfaces in the system, including the PCIe slot and MAC address for each.

Solution:

find /sys/devices/pci0000 …
Read More

Sat 27 September 2014

Filed under Linux Networking

Tags Linux Networking

I received some encouraging comments on G+ from Jesper Dangaard Brouer about my previous post on DSCP, Linux and VLAN priorities. Those comments and the work linked to (here and here) points to a few long-standing (but minor) issues with the way DSCP priorities are handled in Linux.

  1. Some DSCP …
Read More

Wed 17 September 2014

Filed under Linux Networking

Tags Linux Networking

I recently discovered a flaw in the VLAN implementation I did at work. It seemed that the normal TCP traffic had the correct VLAN priorities applied, but audio streaming UDP traffic did not.

This was due to DSCP being applied to the streaming audio and the fact that the VLAN …

Read More

Tue 12 August 2014

Filed under Linux Networking

Tags Linux Networking

This concerns the proliferation of netlink libraries and a lack of direction and documentation.

Background:

I've configured a router with netem (see Bandwidth Throttling with NetEM Network Emulation and netem example rules) to test Tieline devices under various delay and loss network conditions.

It's not really feasible for the tester …

Read More

rationali.st © Andrew Cooks