(Analysis and solution significantly aided by ChatGPT)

After an upgrade of my pfSense from 2.7.2 to 2.8.0, the firewall started crashing about once every couple of days with a kernel panic. After updating to 2.8.1, the problem persisted.

The Problem

Checking /var/crash, I found multiple dumps, all reporting the same kernel panic:

panic: page fault

Digging through textdump.tar.* and info.*, the backtrace showed:

vm_reserv_level_iffullpop()
pmap_enter()
vm_fault()
vm_map_wire_locked()
vm_map_wire()
vslock()
sysctl_wire_old_buffer()
sysctl_kern_malloc_stats()
sysctl_root_handler_locked()
sysctl_root()
userland_sysctl()
kern___sysctlbyname()
sys___sysctlbyname()
--- syscall (570, __sysctlbyname) ---

Each crash was triggered by the process netstat—specifically when it queried system memory statistics using the -m flag.

In plain terms, any netstat -m call would instantly crash the kernel.

The Root Cause

This pfSense release (2.8.x-RELEASE) is based on FreeBSD 15.0-CURRENT, a development snapshot. The kernel panic originates from a bug in the FreeBSD memory subsystem, specifically in sysctl_kern_malloc_stats().

When a userland process (like netstat or pfSense’s PHP dashboard) calls this sysctl, the kernel incorrectly tries to wire an invalid memory page — resulting in a fatal page fault.

The Trigger: pfSense Dashboard and RRD

I confirmed the trigger with a simple search:

grep -R "netstat -m" /etc /usr/local/{pkg,bin,sbin,www}

These were the offenders:

/etc/inc/rrd.inc:        $rrdupdatesh .= "MBUF=`$netstat -m | ";
/usr/local/www/status.php:status_cmd_define("Network-Mbuf Usage", "/usr/bin/netstat -mb");
/usr/local/www/includes/functions.inc.php:  $mbufs_output=trim(`/usr/bin/netstat -mb | ...`);

So the web dashboard and RRD data collector were both invoking netstat -m automatically — each time crashing the firewall.

The Fix

The goal was to prevent netstat -m from running, without breaking the rest of pfSense or normal netstat commands.

  1. Save the Original Binary
cp /usr/bin/netstat /usr/bin/netstat.real
chgrp kmem /usr/bin/netstat.real
chmod 2555 /usr/bin/netstat.real
  1. Replace /usr/bin/netstat with a Wrapper

cat >/usr/bin/netstat <<'EOF'
#!/bin/sh
# pfSense 2.8.1 snapshot workaround: avoid kernel panic from kern.malloc_stats
for arg in "$@"; do
  case "$arg" in
    -m|-mb|-bm)
      # Exit success with empty output so callers don't crash the kernel
      exit 0
      ;;
  esac
done
exec /usr/bin/netstat.real "$@"
EOF
chmod 755 /usr/bin/netstat

Now:

netstat -m    # -> no output, safe
netstat -rn   # -> works as normal
  1. Verify
/usr/bin/netstat -m ; echo "exit=$?"
/usr/bin/netstat -rn | head -5

Expected result:

exit=0
Routing tables
Internet:
...

TL;DR

If your pfSense 2.8.x (FreeBSD 15.0-CURRENT) system keeps crashing with “panic: page fault” in sysctl_kern_malloc_stats, it’s caused by netstat -m.
Replace /usr/bin/netstat with a wrapper that ignores -m options, and your firewall will live happily ever after.

pfSense v2.8.x Repeatedly Crashing Due to a Kernel Panic
Tagged on: