Using Kexec and Kdump For Linux System Troubleshooting on RHEL5/CentOS5

#### Overview: ####

http://download.swsoft.com/virtuozzo/virtuozzo4.0/docs/en/lin/VzLinuxUG/20027.htm - OpenVZ: Using Kexec and Kdump For System Troubleshooting
http://kbase.redhat.com/faq/docs/DOC-6039 - How do I configure kexec/kdump on Red Hat Enterprise Linux 5?

Kexec is a fastboot mechanism that allows booting a Linux kernel from the context of an already running kernel without going through the BIOS. The BIOS can be very time consuming, especially on big servers with numerous peripherals. This can save a lot of time for developers who end up booting a machine numerous times.

Kdump is a new kernel crash dumping mechanism and is very reliable. The crash dump is captured from the context of a freshly booted kernel and not from the context of the crashed kernel. Kdump uses kexec to boot into a second kernel whenever the system crashes. This second kernel, often called a capture kernel, boots with very little memory and captures the dump image.

How is it working? When the kernel crashed the new so called capture kernel is booted via kexec tool. The memory of previous crashed kernel is leaved intact and the capture kernel is able to capture it. In detail, first kernel needs to reserve some memory for capture kernel. It is used by capture kernel for booting. The consequence is the total system memory is lowered by reserverd memory size.

#### Save a crash dump: ####

Install kexec-tools:


yum install kexec-tools

Edit /boot/grub/grub.conf and append to the end of the kernel line (for 32-bit or x86-64-bit systems): "crashkernel=128M@16M"
The sample of kernel line:


kernel /boot/vmlinuz-2.6.18-128.1.16.el5 ro root=LABEL=/ rhgb quiet crashkernel=128M@16M

Enable kdump service and reboot the system:

chkconfig kdump on
reboot 


Now in the case of a system crash, Kexec will boot to the capture kernel without clearing the crashed kernel memory and then pass the control to this kernel. Kdump, in its turn, will capture the dump and put it to the /var/crash directory. This directory is used by Kdump by default for storing system crash dumps. You can leave the default location or redefine it by editing the Kdump configuration file (/etc/kdump.conf). Anyway, you should make sure that the corresponding directory has enough free space to store system dumps.

#### Dump analizing: ####

http://kbase.redhat.com/faq/docs/DOC-3431 - How do I view a vmcore file using Crash in Red Hat Enterprise Linux 4 or 5?

To view a core dump with the crash utility in RHEL5/CentOS5, two arguments must be passed to the crash command.
These are: the uncompressed vmlinux image and the uncompressed vmcore file itself. The uncompressed vmlinux image is in the kernel-debuginfo RPM. The kernel-debuginfo RPM must match the kernel version on the system from which the core dump was obtained.

For CentOS5 you should download and install the kernel-debuginfo and kernel-debuginfo-common packages.
They are available here: http://debuginfo.centos.org


wget http://debuginfo.centos.org/5/`uname -i`/kernel-debuginfo-`uname -r`.i686.rpm
wget http://debuginfo.centos.org/5/`uname -i`/kernel-debuginfo-common-`uname -r`.i686.rpm
rpm -Uvh kernel-debuginfo*

Execute crash like this:


crash /usr/lib/debug/lib/modules/crashed-kernel-version/vmlinux /path-to-the-vmcore

In my case:


crash  /usr/lib/debug/lib/modules/2.6.18-128.1.16.el5/vmlinux  /var/crash/2009-06-09-20\:18/vmcore

Also very useful is coping a standart output to the log file:


crash  /usr/lib/debug/lib/modules/`uname -r`/vmlinux  /var/crash/2009-06-09-20\:18/vmcore  | tee /var/crash/crash3.log 

This will provide a log that may be uploaded to Issue Tracker.

http://kbase.redhat.com/faq/docs/DOC-2122 - How can I use crash to send Red Hat some vmcore pre-analysis information before or while uploading the vmcore image?
http://kbase.redhat.com/faq/docs/DOC-2502 - My box panicked, and I have captured the kernel panic/oops messages. How can I quickly find out where the failure has occurred based on the EIP address?

The following crash utility commands have useful output:


crash> sys
crash> bt -a
crash> mod
crash> log

sys displays the system data:


crash> sys
      KERNEL: /usr/lib/debug/lib/modules/2.6.18-128.1.10.el5/vmlinux
    DUMPFILE: /var/crash/2009-06-12-20:40/vmcore
        CPUS: 2
        DATE: Fri Jun 12 20:40:07 2009
      UPTIME: 3 days, 00:20:29
LOAD AVERAGE: 11.95, 2.74, 1.15
       TASKS: 192
    NODENAME: baikonur
     RELEASE: 2.6.18-128.1.10.el5
     VERSION: #1 SMP Thu May 7 10:39:21 EDT 2009
     MACHINE: i686  (1862 Mhz)
      MEMORY: 2 GB
       PANIC: "Oops: 0000 [#1]" (check log for details)

bt -a displays the stack traces of the active task on each CPU:


crash> bt -a
PID: 3045   TASK: f7199550  CPU: 0   COMMAND: "hald"
 #0 [f77e1ddc] crash_kexec at c0442d32
 #1 [f77e1e20] die at c04064c6
 #2 [f77e1e50] do_page_fault at c06112df
 #3 [f77e1e88] error_code (via page_fault) at c0405a87
    EAX: 00006563  EBX: 00000001  ECX: f77e1000  EDX: f77e1000  EBP: ca15499c 
    DS:  007b      ESI: f5165c40  ES:  007b      EDI: 00000100
    CS:  0060      EIP: c04a9c34  ERR: ffffffff  EFLAGS: 00010206 
 #4 [f77e1ebc] sysfs_follow_link at c04a9c34
 #5 [f77e1ee0] generic_readlink at c047dda7
 #6 [f77e1f44] sys_readlinkat at c047a5ef
 #7 [f77e1fa4] sys_readlink at c047a619
 #8 [f77e1fb8] system_call at c0404f10
    EAX: ffffffda  EBX: bfdedc68  ECX: 087b1f78  EDX: 00000100 
    DS:  007b      ESI: 00000100  ES:  007b      EDI: 087b1f78
    SS:  007b      ESP: bfded5a4  EBP: bfded5e8
    CS:  0073      EIP: 0013a402  ERR: 00000055  EFLAGS: 00000206 

PID: 3238   TASK: ca123000  CPU: 1   COMMAND: "Xorg"
 #0 [f70b5f44] crash_nmi_callback at c04193c1
 #1 [f70b5f90] do_nmi at c0406828
 #2 [f70b5fb8] nmi at c0405b29
    EAX: 4800000b  EBX: 00178aa0  ECX: 0a0442f0  EDX: b7f64000 
    DS:  007b      ESI: 0a03f088  ES:  007b      EDI: 0a041d40
    SS:  007b      ESP: bfe37610  EBP: bfe37628
    CS:  0073      EIP: 00135876  ERR: 4800000b  EFLAGS: 00203202 

mod displays module information and loading of symbols and debugging data:


crash> mod
 MODULE   NAME                       SIZE  OBJECT FILE
f883f000  ehci_hcd                  33357  (not loaded)  [CONFIG_KALLSYMS]
f8842900  dm_message                 6977  (not loaded)  [CONFIG_KALLSYMS]
f8845600  dm_zero                    6209  (not loaded)  [CONFIG_KALLSYMS]
f8848980  pcspkr                     7105  (not loaded)  [CONFIG_KALLSYMS]
f884fe00  ohci_hcd                  24681  (not loaded)  [CONFIG_KALLSYMS]
f8858100  uhci_hcd                  25421  (not loaded)  [CONFIG_KALLSYMS]
f8860000  sd_mod                    25153  (not loaded)  [CONFIG_KALLSYMS]
....

log command dumps the kernel log buf contents in chronological order. This is useful when you want to look at the panic/oops message. An oops is triggered by some exception and is a dump of the CPU state and kernel stack at that instant:


crash> log
Linux version 2.6.18-128.1.10.el5 (mockbuild@builder16.centos.org) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-44)) 
#1 SMP Thu May 7 10:39:21 EDT 2009
BIOS-provided physical RAM map:
 BIOS-e820: 0000000000000000 - 000000000009f800 (usable)
 BIOS-e820: 000000000009f800 - 00000000000a0000 (reserved)
 BIOS-e820: 00000000000dc000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 000000007f680000 (usable)
 BIOS-e820: 000000007f680000 - 000000007f690000 (ACPI NVS)
 BIOS-e820: 000000007f690000 - 0000000080000000 (reserved)
 BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved)
 BIOS-e820: 00000000fec00000 - 00000000fec10000 (reserved)
 BIOS-e820: 00000000fed14000 - 00000000fed1a000 (reserved)
 BIOS-e820: 00000000fed1c000 - 00000000fed90000 (reserved)
 BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
 BIOS-e820: 00000000ff000000 - 0000000100000000 (reserved)
1142MB HIGHMEM available.
896MB LOWMEM available.
found SMP MP-table at 000f77a0
.....
.....
BUG: unable to handle kernel paging request at virtual address 00006587
 printing eip:
c04a9c34
*pde = 72823067
Oops: 0000 [#1]
SMP 
last sysfs file: /class/input/input8/event3/dev
Modules linked in: ppp_deflate zlib_deflate ppp_async crc_ccitt ppp_generic slhc sha256 aes_generic testmgr_cipher testmgr
......
CPU:    0
EIP:    0060:[<c04a9c34>]    Tainted: G      VLI
EFLAGS: 00010206   (2.6.18-128.1.10.el5 #1) 
EIP is at sysfs_follow_link+0xc2/0x1ab
eax: 00006563   ebx: 00000001   ecx: f77e1000   edx: f77e1000
esi: f5165c40   edi: 00000100   ebp: ca15499c   esp: f77e1ec0
ds: 007b   es: 007b   ss: 0068
Process hald (pid: 3045, ti=f77e1000 task=f7199550 task.ti=f77e1000)
Stack: f77e1ee4 e8ac7000 f0251c84 00000000 c0688fa0 e89862b0 00000100 087b1f78 
       c047ddaa f7199550 f77e1f34 e89862b0 c04c705c f77e1efc f77e1efc 00000001 
       00000000 00000000 ef4a1658 3b8d3635 4a3292f7 c04284f1 00000001 f7d6e200 
Call Trace:
 [<c047ddaa>] generic_readlink+0x25/0x68
 [<c04c705c>] selinux_inode_readlink+0x4c/0x54
 [<c04284f1>] current_fs_time+0x4a/0x55
 [<c0488f9b>] touch_atime+0x60/0x91
 [<c047a5f2>] sys_readlinkat+0x79/0x92
 [<c044a29f>] audit_syscall_entry+0x14b/0x17d
 [<c047a61e>] sys_readlink+0x13/0x17
 [<c0404f17>] syscall_call+0x7/0xb
 =======================
Code: c5 b0 01 86 05 80 e8 6e c0 85 ed 75 0e 8b 44 24 08 e8 02 ca 03 00 e9 c0 00 00 00 b8 00 8f 68 c0 31 db e8 51 d3 f8 
ff 8b 44 24 08 <8b> 40 24 43 85 c0 75 f8 89 ea be 01 00 00 00 8b 3a 31 c0 83 c9 
EIP: [<c04a9c34>] sysfs_follow_link+0xc2/0x1ab SS:ESP 0068:f77e1ec0

Take note of the EIP address: EIP: [<c04a9c34>]
From the line c04a9c34, this is the address at which the failure occurred. Issuing the following command can give more hints as to which function/source code/assembly statement in the kernel triggered that:


crash> dis -lr c04a9c34
/usr/src/debug/kernel-2.6.18/linux-2.6.18.i686/fs/sysfs/symlink.c: 157
0xc04a9b72 <sysfs_follow_link>: push   %ebp
0xc04a9b73 <sysfs_follow_link+1>:       push   %edi
0xc04a9b74 <sysfs_follow_link+2>:       push   %esi
0xc04a9b75 <sysfs_follow_link+3>:       push   %ebx
0xc04a9b76 <sysfs_follow_link+4>:       mov    %eax,%ebx
0xc04a9b78 <sysfs_follow_link+6>:       sub    $0x10,%esp
/usr/src/debug/kernel-2.6.18/linux-2.6.18.i686/fs/sysfs/symlink.c: 159
0xc04a9b7b <sysfs_follow_link+9>:       mov    $0xd0,%eax
/usr/src/debug/kernel-2.6.18/linux-2.6.18.i686/fs/sysfs/symlink.c: 157
0xc04a9b80 <sysfs_follow_link+14>:      mov    %edx,(%esp)
/usr/src/debug/kernel-2.6.18/linux-2.6.18.i686/fs/sysfs/symlink.c: 159
0xc04a9b83 <sysfs_follow_link+17>:      call   0xc045b200 <get_zeroed_page>
0xc04a9b88 <sysfs_follow_link+22>:      mov    %eax,0x4(%esp)
...

The full crash log of this examples may be found here:crash2.log
If you intrested in another crash log samples look crash1.log and crash3.log
This crashes were obtained on my laptop with CentOS5.3 after updating to the kernel 2.6.18-128.1.10.el5.
All faults occurred by pm-suspend.

I hope this may be helpful

Sergej A. Kandyla skandyla at gmail.com Any comments and suggestions are welcome.