I've fixed the main memory leak, but I'm still seeing very slow-slow leak and I'm still investigating. I'm analyzing memory usage very carefully and I'm seeing a progressive slow leak. I think its actually due to memory thrashing, that is lots of small memory blocks allocated to hold copies of packet and quickly deallocated. This causes certain amount of overhead in allocation tables in the runtime library. At least that is what I think is I'm witnessing.
None the less I'm still working on it. I running long tests in different modes and carefully analyzing memory usage. The slow leak is about 200K per 1,000,000 packets processed. So its relatively very slow, none the less it could cause big problems for long running or never ending processes.
#10288 packets=741441 pps=17414.529312 bytes=4832Kb/s hdr=76649/s hdr=13us rm=26140Kb pm=36b vm=242812Kb #10288 packets=734100 pps=17147.061572 bytes=4757Kb/s hdr=75472/s hdr=13us rm=26056Kb pm=36b vm=242812Kb #10288 packets=734100 pps=17414.717465 bytes=4832Kb/s hdr=76650/s hdr=13us rm=26284Kb pm=36b vm=242812Kb #10288 packets=734100 pps=17743.027022 bytes=4923Kb/s hdr=78095/s hdr=13us rm=26264Kb pm=36b vm=242812Kb #10288 packets=734100 pps=17590.396089 bytes=4880Kb/s hdr=77423/s hdr=13us rm=26592Kb pm=37b vm=242812Kb #10288 packets=734100 pps=17683.191213 bytes=4906Kb/s hdr=77832/s hdr=13us rm=26736Kb pm=37b vm=242812Kb #10288 packets=734100 pps=17349.688032 bytes=4814Kb/s hdr=76364/s hdr=13us rm=26676Kb pm=37b vm=242812Kb #10288 packets=734100 pps=17610.228854 bytes=4886Kb/s hdr=77510/s hdr=13us rm=26800Kb pm=37b vm=242812Kb
Memory counters: rm = Resident Memory, pm = per packet memory, vm = virtual memory. These are system process memory statistics, not java VM. I'm using a 3rd party library called SIGAR to access system process statistics.
You can see that rm slowly increases. I would like to do a release tonight, but may postpone it until tomorrow so I can fix this properly once and for all. In either case a "relief" release is due, since the main memory leak bug is fixed and the builds released are pretty much unusable in their current state.
I'll keep everyone updated as often as I can on my progress.
I think this was a false alarm. Memory stabilizes eventually and doesn't increase anymore. This is after several million packets are processed. There are multiple factors impacting my tests.
1) number of pcap.openOffline calls which allocate and deallocate pcap_t structures
2) packet copies into new small buffers
Because the unexpected memory usage eventually levels out, it seems like its a normal C runtime issue under high memory allocation loads. Therefore I'm satisfied that its not an actual memory leak and all the holes have been plugged up.
Hi Mark,
Thanks for the updates.
What platform/OS were you running your tests on? Have you tried the tests on different platforms to see if you see the same behavior (memory usage increases slowly for awhile, then plateaus)?
Just curious.
Jim
Yes, on multiple platforms. The only thing is that these tests take a long time to complete (hours), that is why such a slow progress.
Hi,
Of the platforms you tested on, were they all *nix? Again, just curious, because I was thinking that all the *nixes would have the same sort of memory management, whereas Win32 might be different, so it'd be interesting to see if you get the same behavior on a *nix vs. Win32. Again, just curious.
Jim
Yes, tested on WinXP and FC11. The differences are in C runtime memory management. I also discovered that the memory thrashing occurs, even when I turn off memory copies and the scanner. Basically the JBufferHandler which doesn't use any resources of its own. The thrashing is a by product of Pcap.openOffline, loop and other unavoidable allocations out of explicit control of jNetPcap.
I'm fixing this bug, but I will also be replacing the entire memory allocation for packet copies in handlers in the near future. Memory allocation needs to be more efficient and minimize thrashing. The new method will allocate memory in larger blocks and sub-allocate out of it. For example a 16Kb block will be allocated once and copies will take place into this block until its used up. Then new block will be pre-allocated and used up and so forth. This has several advantages:
1) won't be calling malloc excessively especially for tiny packets
2) block allocations will take place on a hardware friendly boundary
3) block deallocations will be of exactly the same size and not variable length from tiny to large. This will make it easier for malloc to reuse previously allocated blocks and since freed because they are all the same size.
4) This type of memory allocation is needed for other things such as Analyzers and native header dissectors which need to allocate and attach memory to packet objects.
My priority is to fix this bug right now and make a release. Then I will work on improving the allocation. I'm already adding memory-leak tests and will test this new code thoroughly before releasing the new memory managment in handlers.
Hi,
What you're describing, except for the fixed size blocks of memory, kind of sounds like the way that the Java heap works?
Jim
Yes Hotspot preallocates large blocks upto the -Xmx size and sub-allocates internally. I guess jNetPcap's way will be similar. JMemory class is designed perfectly to handle sub-allocated memory. JMemoryPool class already works this way.
Ok, I'm satisfied that the memory leak problem is fixed. I have checked everything in and I'm working on the next dev build release (1.3.b0010). I will be done in a few hours.
I have the basic code done for jUnit testcases that will check for memory leaks. They are not properly implemented at this time to be fully automated, but I checked things thoroughly by hand against reports. I will automate the tests and make them part of the BuildTests.
1.3.b0010 is released. This release fixes the memory leak and the coredumps. Please verify that indeed both of these bugs have been fixed. Once confirmed, I can relax a bit and concentrate on producing a stable 1.2 release.
Hi Mark,
Are you not doing a Win32 release for b0010?
Also, the "jnetpcap-1.3.b0010-1.el4.i386.tgz" is for RHEL4/FC3, right?
Jim
Hi,
I test "jnetpcap-1.3.b0010-1.el4.i386.tgz" on FC3, and using the same PCAP file that was getting the error I reported in:
and, using the b0010, I was able to process the entire PCAP without encountering that exception.
Looks good, at least for what I tested 
!!
Jim
Win32 is there:
https://sourceforge.net/projects/jnetpcap/files/jnetpcap/dev/1.3/jnetpca...
And yes el4 is RHEL4.
Mark,
Ok, this is weird.
I was about to suggest that you were joking with me, but then I tried going to the downloads page using Firefox instead of IE6, and the Win32 files appeared (in Firefox, but not in IE6)!!
Jim