crash dump analysis

3 replies [Last post]
vf
Offline
Joined: 06/17/2009

hi,

I'm looking at a crash
while running an application with DDD attached to the Java process
on exiting pcap loop (stop the capture).

What might be a reason for such a crash?

Thank you

(gdb) bt
#0 0xb8073430 in __kernel_vsyscall ()
#1 0xb7efa6d0 in raise () from /lib/tls/i686/cmov/libc.so.6
#2 0xb7efc098 in abort () from /lib/tls/i686/cmov/libc.so.6
#3 0xb78be9af in os::abort () from /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/i386/server/libjvm.so
#4 0xb79e1891 in VMError::report_and_die () from /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/i386/server/libjvm.so
#5 0xb78c51bc in JVM_handle_linux_signal () from /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/i386/server/libjvm.so
#6 0xb78c1434 in signalHandler () from /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/i386/server/libjvm.so
#7
#8 pcap_read_linux_mmap (handle=0x7bdcb178, max_packets=-1, callback=0x7abb3104 , user=0x7a9cce08 "x±Ü{\214\212]\n\020Mn|ÈÎ\234zÔÎ\234z") at ./pcap-linux.c:2363
#9 0x7aa8915b in pcap_loop (p=0x7bdcb178, cnt=-1, callback=0x7abb3104 , user=0x7a9cce08 "x±Ü{\214\212]\n\020Mn|ÈÎ\234zÔÎ\234z") at ./pcap.c:302
#10 0x7abab682 in Java_org_jnetpcap_Pcap_loop__IILorg_jnetpcap_packet_JPacketHandler_2Ljava_lang_Object_2Lorg_jnetpcap_packet_JPacket_2Lorg_jnetpcap_packet_JPacket_00024State_2Lorg_jnetpcap_PcapHeader_2Lorg_jnetpcap_packet_JScanner_2 (env=0x7c6e4d10, obj=0x7a9cced4, jcnt=-1, id=1, jhandler=0x7a9ccec8, juser=0x0, jpacket=0x7a9ccec0, jstate=0x7a9ccebc, jheader=0x7a9cceb8, jscanner=0x7a9cceb4) at /home/mark/prjs/jnetpcap/src/c/jnetpcap.cpp:717
#11 0xb42954aa in ?? ()
#12 0xb428df69 in ?? ()
#13 0xb428df69 in ?? ()
#14 0xb428e569 in ?? ()
#15 0xb428b34c in ?? ()
#16 0xb76fe850 in JavaCalls::call_helper () from /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/i386/server/libjvm.so
#17 0xb78c2638 in os::os_exception_wrapper () from /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/i386/server/libjvm.so
#18 0xb76fe057 in JavaCalls::call_virtual () from /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/i386/server/libjvm.so
#19 0xb76fe10a in JavaCalls::call_virtual () from /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/i386/server/libjvm.so
#20 0xb777b425 in thread_entry () from /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/i386/server/libjvm.so
#21 0xb79a268e in JavaThread::run () from /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/i386/server/libjvm.so
#22 0xb78c3ade in java_start () from /usr/lib/jvm/java-6-sun-1.6.0.14/jre/lib/i386/server/libjvm.so
#23 0xb80464ff in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
#24 0xb7fb349e in clone () from /lib/tls/i686/cmov/libc.so.6
(gdb)

jnetpcap-1.3.b0006-1.ubuntu.i386.deb

lrwxrwxrwx 1 root root 14 2009-05-07 15:16 /usr/lib/libpcap.so -> libpcap.so.0.8
lrwxrwxrwx 1 root root 16 2009-05-07 15:16 /usr/lib/libpcap.so.0.8 -> libpcap.so.1.0.0
-rw-r--r-- 1 root root 194884 2009-03-31 20:12 /usr/lib/libpcap.so.1.0.0

Mark Bednarczyk
Mark Bednarczyk's picture
Offline
Joined: 03/22/2008
What does dpkg-query --show

What does

dpkg-query --show --showformat ${Version} libpcap0.8
has to say about the version of libpcap?

Can you please provide me info about the ubuntu system (exact OS version) and libpcap version installed?

Does the crash happen immediately? After a certain time? Once in a blue moon?

Sly Technologies, Inc.
R&D

vf
Offline
Joined: 06/17/2009
Crash happens on exit from

Crash happens on exit from the pcap loop for openLive() irregularly,
maybe once in three runs,
on a couple of ubuntu boxes.

My system is a ubuntu generic

$ uname -a
Linux desktop 2.6.28-13-generic #45-Ubuntu SMP Tue Jun 30 19:49:51 UTC 2009 i686 GNU/Linux

$ dpkg -l libpcap0.8
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Cfg-files/Unpacked/Failed-cfg/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Hold/Reinst-required/X=both-problems (Status,Err: uppercase=bad)
||/ Name Version Description
+++-=============================================-=============================================-==========================================================================================================
ii libpcap0.8 1.0.0-1 system interface for user-level packet capture

$ dpkg -L libpcap0.8
/.
/usr
/usr/lib
/usr/lib/libpcap.so.1.0.0
/usr/share
/usr/share/doc
/usr/share/doc/libpcap0.8
/usr/share/doc/libpcap0.8/changelog.Debian.gz
/usr/share/doc/libpcap0.8/README.Debian
/usr/share/doc/libpcap0.8/copyright
/usr/share/doc/libpcap0.8/changelog.gz
/usr/share/doc/libpcap0.8/README.gz
/usr/share/doc/libpcap0.8/CREDITS.gz
/usr/lib/libpcap.so.0.8

I would install a custom (with debug info) libpcap in case this may help

Mark Bednarczyk
Mark Bednarczyk's picture
Offline
Joined: 03/22/2008
Lets see. #8

Lets see.


#8 pcap_read_linux_mmap (handle=0x7bdcb178, max_packets=-1, callback=0x7abb3104 , user=0x7a9cce08 "x±Ü{\214\212]\n\020Mn|ÈÎ\234zÔÎ\234z") at ./pcap-linux.c:2363
#9 0x7aa8915b in pcap_loop (p=0x7bdcb178, cnt=-1, callback=0x7abb3104 , user=0x7a9cce08 "x±Ü{\214\212]\n\020Mn|ÈÎ\234zÔÎ\234z") at ./pcap.c:302
#10 0x7abab682 in Java_org_jnetpcap_Pcap_loop__IILorg_jnetpcap_packet_JPacketHandler_2Ljava_lang_Object_2Lorg_jnetpcap_packet_JPacket_2Lorg_jnetpcap_packet_JPacket_00024State_2Lorg_jnetpcap_PcapHeader_2Lorg_jnetpcap_packet_JScanner_2 (env=0x7c6e4d10, obj=0x7a9cced4, jcnt=-1, id=1, jhandler=0x7a9ccec8, juser=0x0, jpacket=0x7a9ccec0, jstate=0x7a9ccebc, jheader=0x7a9cceb8, jscanner=0x7a9cceb4) at /home/mark/prjs/jnetpcap/src/c/jnetpcap.cpp:717

You can see #10 jNP called pcap_loop which is a libpcap call in libpcap-1.0.0. Then it fails at stack level #9 and #8 pcap_read_linux_mmap. And you say that this happens after you have processed the last packet and loop should be exiting.

So here is what could be happening:

1) there is a bug in libpcap 1.0.0 and memory mapper.

2) Something in your code is causing the memory mapped buffer to fail

3) Something else we can't see from the debugger

Not much we can do about #1 except try a different version of libpcap such as downgrade to 0.9.X.

As to #2. JPacketHandler copies the packets provided by libpcap into newly allocated memory block. If you could possibly just not do any processing with those packets as a test run and see if the crash still occurs. I'm wondering if the JPacketHandler somehow is holding up the libpcap memory block and preventing it from being unmapped by libpcap. If that is the case the problem would be in the cb_jpacket_dispatch method and the way that it copies the contents of the libpcap provided packet. Note that the exception doesn't occur in the cb_jpacket_dispatch native handler.

#3 we would have to set additional testcases. I would need to be able to duplicate this issue in my build environment.

Unfortunately I am going to be traveling for the next 5 days (be back tuesday of next week) and won't have access to my build lab.

In the meantime try running your loop with basically no code in it and see if that still causes problems. Check if somehow keeping references to packet objects resurfaces the crash problem. Also for #1 you try and downgrade libpcap version or atleast change the LD_LIBRARY_PATH so that jNP runs against an older libpcap.so file.

Sly Technologies, Inc.
R&D

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.