Text of wiki.ubuntu.com › Debugging_kernel_panics_remotely_with_netconsole as Styx crawled it on Nov 18, 2026. The page may have changed since. Only its words are kept, without pictures or layout.

Open the live page

Debugging kernel panics remotely with netconsole - Ubuntu Wiki

Jump to content The new Ubuntu Wiki is live. Legacy content may be unavailable and page links may have changed. Read the announcement on Discourse . From Ubuntu Wiki Introduction Netconsole is a kernel module for logging console messages over UDP to a separate machine, useful for capturing failures remotely when capturing logs is difficult on the system under test. Netconsole has some limitations: A separate receiver machine is required in order to capture the UDP packets sent from the system under test.

If you have no other systems running Linux, you can boot Ubuntu from a live USB temporarily to complete the capture. Netconsole initializes when modules are loaded into the kernel. This doesn't allow capture of failures during early boot. The driver for the network card must support the netpoll API . Only IP networks, UDP packets, and Ethernet devices are supported. Set up netconsole This document will guide you through the steps to set up netconsole for Ubuntu.

Step 1: Determine remote MAC address We need to know the MAC address of the receiving machine. When the receiver is in same subnet To send the UDP packets to 192.168.1.103, which in this example is the same subnet as the sender, you would find the MAC address with the following: pet@sender:~$ ping -c 1 192 .168.1.103 > /dev/null pet@sender:~$ arp -n 192 .168.1.103 Address HWtype HWaddress Flags Mask Iface 192 .168.1.103 ether 08 :00:46:d4:1d:82 C eth0 Therefore, 08:00:46:d4:1d:82 is the MAC address of the receiver.

When the receiver isn't in the same subnet First, determine the default gateway IP address: pet@sender:~$ netstat -rn | grep ^0.0.0.0 0 .0.0.0 192 .168.1.1 0 .0.0.0 UG 0 0 0 eth0 In this case, that's 192.168.1.1 . Next, find the MAC address of the default gateway: pet@sender:~$ ping -c 1 192 .168.1.1 > /dev/null pet@sender:~$ arp -n 192 .168.1.1 Address HWtype HWaddress Flags Mask Iface 192 .168.1.1 ether 00 :0f:66:5b:2a:3c C eth0 Therefore, 00:0f:66:5b:2a:3c is the MAC address of the receiver.

Step 2: Change boot time kernel command line arguments For this guide it is assumed that the system under test uses GRUB as its bootloader. By default, GRUB will boot the kernel with command line arguments quiet splash . These need to be changed to capture as much information as possible. GRUB 2 instructions Create a backup of /etc/default/grub first: pet@sender:~$ sudo cp /etc/default/grub /etc/default/grub.backup Now, you will need to edit /etc/default/grub as root.

Locate the line GRUB_CMDLINE_LINUX_DEFAULT="quiet splash" and replace this line with GRUB_CMDLINE_LINUX_DEFAULT="debug ignore_loglevel" . --- /etc/default/grub.backup 2026-10-09 13:31:00.352048642 -0400 +++ /etc/default/grub 2026-10-09 13:31:18.271391144 -0400 @@ -7,7 +7,7 @@ GRUB_TIMEOUT_STYLE=hidden GRUB_TIMEOUT=0 GRUB_DISTRIBUTOR=`( . /etc/os-release; echo ${NAME:-Ubuntu} ) 2>/dev/null || echo Ubuntu` -GRUB_CMDLINE_LINUX_DEFAULT="quiet splash" +GRUB_CMDLINE_LINUX_DEFAULT="debug ignore_loglevel" GRUB_CMDLINE_LINUX="" # If your computer has multiple operating systems installed, then you Finally, update GRUB with the new config: pet@sender:~$ sudo update-grub Step 3: Initialize netconsole at boot time Update the list of modules to automatically load.

If /etc/modules-load.d exists pet@sender:~$ sudo sh -c 'echo netconsole > /etc/modules-load.d/netconsole.conf' If /etc/modules-load.d does not exist pet@sender:~$ sudo sh -c 'echo netconsole >> /etc/modules.conf' To configure netconsole, you will need the IP and MAC address of the receiver, as well as the IP address and network interface of the sender. The receiver's IP and MAC addresses should have been gathered in step 1.

To get the network interface and IP address of the sender, run: pet@sender:~$ ifconfig -a Next, fill out the netconsole config by creating/updating /etc/modprobe.d/netconsole.conf : pet@sender:~$ sudo sh -c 'echo options netconsole netconsole=6666@192.168.1.102/eth0,6666@192.168.1.103/08:00:46:d4:1d:82 > /etc/modprobe.d/netconsole.conf' The format for this configuration can be broken down this way: netconsole=<LOCAL_PORT>@<SENDER_IP_ADDRESS>/<SENDER_INTERFACE>,<REMOTE_PORT>@<RECEIVER_IP_ADDRESS>/<STEP_1_MAC_ADDRESS> The local and remote ports are up to you, the example uses 6666 for both.

Step 4: Set up receiver The receiver now needs to run a process to listen for packets. Syslog-ng Install syslog-ng: pet@receiver:~$ sudo apt-get install syslog-ng Edit /etc/syslog-ng/syslog-ng.conf and append these lines to the end of the file: source net { udp(ip("0.0.0.0") port(6666)); }; destination netconsole { file("/var/log/$HOST-netconsole.log"); }; log { source(net); destination(netconsole); }; Now restart syslog-ng: pet@receiver:~$ sudo /etc/init.d/syslog-ng restart Continue with Step 5: Capture logs .

Once you are done, the logging will be in /var/log/<SENDER_IP_ADDRESS>-netconsole.log . Netcat Netcat is an alternative if syslog-ng is not available for whatever reason. In this example the <RECEIVER_IP_ADDRESS> is 192.168.1.103. (see Step 3: Initialize netconsole at boot time ) pet@receiver:~$ netcat -l -u 192 .168.1.103 6666 | tee ~/netconsole.log Allow the process to run in the background as you continue with Step 5: Capture logs .

Once you are done, press "Control-C" to stop netcat. The messages received will be forwarded to ~/netconsole.log . You can also try without the IP address specifically listed if you are having issues connecting: pet@receiver:~$ netcat -l -u 6666 | tee ~/netconsole.log On very old versions of netcat, you may see UDP listen needs -p arg which means you need to use netcat-traditional syntax. For example: pet@receiver:~$ netcat -l -p 6666 -u | tee ~/netconsole.log Step 5: Capture logs Reboot the sender, then execute the command below. pet@sender:~$ dmesg | grep netcon [ 21 .048406 ] netconsole: local port 6666 [ 21 .048410 ] netconsole: local IP 192 .168.1.102 [ 21 .048411 ] netconsole: interface eth0 [ 21 .048413 ] netconsole: remote port 6666 [ 21 .048415 ] netconsole: remote IP 192 .168.1.103 [ 21 .048418 ] netconsole: remote ethernet address 08 :00:46:d4:1d:82 [ 21 .048421 ] netconsole: device eth0 not up yet, forcing it [ 22 .908106 ] console [ netcon0 ] enabled [ 22 .911536 ] netconsole: network logging started Seems to be working!

Check your receiver to confirm the kernel messages were received. If logs are appearing in your target file, you can continue to use the sender machine until it crashes. If you need to avoid rebooting the sender, you can manually load netconsole via: sudo modprobe netconsole netconsole = 6666 @192.168.1.102/eth0,6666@192.168.1.103/08:00:46:d4:1d:82 Substituting the example config for your own. To test logging before a kernel panic, one option is to insert removable media on the sender, like a CD or USB.

Remove netconsole Once you've retrieved the information you needed, these are the steps to clean up. Revert GRUB to its original configuration GRUB 2 instructions pet@sender:~$ sudo mv /etc/default/grub.backup /etc/default/grub pet@sender:~$ sudo update-grub Remove netconsole itself If /etc/modules-load.d exists pet@sender:~$ sudo rm -f /etc/modules-load.d/netconsole.conf pet@sender:~$ sudo rm -f /etc/modprobe.d/netconsole.conf If /etc/modules-load.d does not exist pet@sender:~$ sudo sed -i '/^netconsole/d' /etc/modules pet@sender:~$ sudo rm -f /etc/modprobe.d/netconsole.conf See also Netconsole documentation Another netconsole tutorial Credits Original writeup by ~petski .

Original writeup reviewed by Peter Veerman and Leann Ogasawara . Original GRUB 2 instructions by Bruno Girin . Updated for new wiki by Ian Whitfield . Retrieved from " https://wiki.ubuntu.com/w/index.php?title=Debugging_kernel_panics_remotely_with_netconsole&oldid=1300 " Categories : Debugging Kernel Debugging kernel panics remotely with netconsole Add topic