Greetings,
I'm new here, i just started trying to use jnetpcap on monday. I've run into a bunch of problems and so far i haven't been able to get past them. i've tried to do some searches on the forums, and i've read over just about all of the userguide, however, to be honest, i find this site very difficult to understand. It is hard to find information and simple code examples/snippets, and it is hard to understand what should work, and with what version. I'm unclear what examples are supposed to only work with 1.2, 1.3 or how jnetstream relates to all of this as well. I want to find the answer myself without having to bother anyone, but alas i need to post here.
I have a tcpdump file that i want to analyze and re-create the tcp stream a user application would receive from a tcp socket. I also then want to, for each byte of that stream, find the associated packet that byte was delivered in. I want to use this to do some performance analysis.
What i've tried so far:
* I first tried to apply the Mac OS X patch from Mardy but it had issues (not sure if it was for 1.2 or 1.3, and i tried 1.3, which might have been my problem). I then tried to re-create the changes by hand, but then i had a lot of compiler errors (i think related to lack of JAVA_HOME and proper pcap headers being available, but i could not find any help/docs on how to build and what the requirements are). I gave up on this, however i would really like to see a supported OS X distribution in the future
* Next i installed Ubuntu on vmware and was able to install the jnetpcap 1.3b3 deb package just fine. I pulled this into my Eclipse project (it would be nice if i could just download the .jar and .so files without a deb package however, the extra step seems needless for development) and was able to do an openOffline of my capture file. I was then able to specify a PcapPacketHandler and see my packets coming through (very nice textformatting btw). Then i wanted to use the TcpAnylizer/Sequencer/Assembler to get the full tcp stream. I looked everywhere for how to do this, but i did not see any example that actually told you how to use these Anylizers. There is one example in the UserGuide that says add a listener, but no where does it tell you how to actually cause these anylizers to kick into gear (please correct me if i'm wrong). So i looked into the unit tests and found that you need to pass the JController to the pcap.loop function to get this working. I tried this and it did start dumping events to TcpAnylizer correctly, and that did send events to the TcpSequencer, however every event was thrown away by the TcpSequencer when null was returned from getSequence(hash, false) in processAnalyzerEvent. This then of course doesn't call the TcpAssembler ever, so i never get the tcp stream i'm looking for.
* Thinking this might be a bug, and that i might have been a little over ambitious with trying 1.3, i removed 1.3 and installed 1.2. Now nothing worked. I wasn't getting events in the TcpAnyzlier at all now, so i started to wonder if this feature is in 1.2 or just in 1.3. Looking over the JavaDoc it looks like this should be working, so i reverted to using my PcapPacketHandler again and printing out text formatted packets. Now, strangely, in 1.2, i don't get any IP4 or TCP header information for my packets, only Frame/Eth/Data show up in the text formatter. As well, when i try to do a packet.hasHeader(ip) or tcp, they both return false. Is there something i need to do to enable the ip and tcp scanners in 1.2? this worked fine in 1.3 but again, i'm real confused by the site and not sure what is supposed to work where. I expected this packet to be fully decoded, but it doesn't seem to be, or some decoding is not in 1.2 that was in 1.3.
So now i need some help.
* What version of jnetpcap should i be using to get TcpAnylizer/Sequencer/Assembler to work?
* And how should i get it to work? Was i right in passing the controller to pcap.loop? Is there any example or code snippet anywhere that has the Assembler working and getting data?
* Has anyone else had success in getting something like this to work? or are there known issues with this and it won't work right now?
Thank you very much for your time, sorry for the long email, but it has been a long couple of days.
....
Kabe
OK, so after posting this, i just started reading posts in the forum and i saw the SLL "Linux Cooked" issue with the use of device "any". This is what i did and this is why it IP/TCP headers worked in 1.3 and not 1.2.
So any help on how to get TcpSequencer/TcpAssembler working in 1.3 would be great. If they don't work, then i can go back and re-capture with direct devices. Please advise on what version of the library i should be using, as well as examples on how to use these Anylizers. On a side note, i notice there is no TcpHandler class such as there is for other anylizers. Is this by design?
....
Kabe
Holly moly, that is quiet a lot you are trying to do there. Here are a few pointers. As to versions and features:
Version 1.2 only has scanner/decoder and NO analyzers. Version 1.3 adds analysis and reassemblers.
I realize that 1.2 release cycles are very confusing. That is why I have backed up and standardized on features that we can get working soon. Analyzer's are very involved and need a lot more time and work before I'm able to release them as something stable, but they continue to be released as 'dev' releases.
If you need TcpAnalyzer then you need to be using the dev builds labeled 1.3.bXXXX. There is no TcpHandler. All TcpAssembly and sequencing is done as a service for higher level protocols. I have plans for more tcp level API, but its a feature that has to wait until I re-release the stable 1.2 release which will clarify all the confusion.
Documentation wise, because analysis is a feature still under heavy development, I haven't sat down and written up documentation on it. Main for the fact that its still being developent and is likely to change.
Yes passing controller in to the loop is the right way. JController is the main piece of code that dispatches packets to all the other analyzers. I added a convenience method Pcap.analyzer in 1.3.dev as well.
I don't know of anyone assembling the entire raw tcp stream although I do plan on providing a java.io.InputStream adapter that would work with the tcp analyzer/sequencer/assembler to provide byte level access to the entire stream. Its actually quiet possible to do this right now. Todo it properly though, we need a "sliding window" buffer which works the same way a normal sliding window algorithm works. This allows never ending streams to be programatically read and the consumed data to be discarded thus providing a mechanism to go on forever if needed. InputStream adapter would be based on this new "sliding window" buffer, just consuming data out of the buffer 1 byte at a time. And to be clear, I don't have a "sliding window" buffer developed yet.
Here is a rough outline. If you listen to TcpAnalyzer events and wait for a start of a new stream which will create a TcpDuplexStream. From the duplex stream you can get at individual TcpStream (one in each direction). Then using TcpSequencer.setFragmentationBoundary you can start sequencing and reassembly will happen automatically. You have to add another listener to TcpAssembler to listen for reassembly complete events. What you will get from TcpAssembler is a single packet with fully reassembled stream. It will only have Ip4/Tcp header and if scanner was able to match any additional higher level headers. This isn't very portable and will break for very long streams (will run out of memory).
My primary focus right now is 1.2. Once that is re-released as 1.2.alpha1 without analyzer features, it will mature toward production/stable. After the 1.2 release I can get back to working on analyzers. I have a lot better idea how to make it all work very nicely with an easy API.
Lastly about jNetStream. jNetStream is built ontop of jNetPcap. So until jNetPcap is properly released, jNetStream won't move ahead. jNetStream adds a nicer higher level API and capture file manipulation logic, but for now its on the back burner.
Version 1.2 only has scanner/decoder and NO analyzers. Version 1.3 adds analysis and reassemblers.
This is the part that is confusing. Looking at the JavaDoc i see analyzers in 1.2. For example the TcpAnalyzer is here in the 1.2rc5 javadoc, and it is in the jar file as well (http://jnetpcap.com/docs/javadoc/jnetpcap-javadoc/org/jnetpcap/protocol/...).
My primary focus right now is 1.2. Once that is re-released as 1.2.alpha1 without analyzer features, it will mature toward production/stable. After the 1.2 release I can get back to working on analyzers. I have a lot better idea how to make it all work very nicely with an easy API.
Hm, so by this do mean that the current release of 1.2 does have analyzers, but it is not supposed to and they will be removed in the next 1.2 release?
Thanks for your other clarification about TCP streams. This does seem like the feature i want. It isn't something i can build just yet, so i'm going to see if i can get the information i need using the lower level, and less coordinated, data.
Thanks.
Yes, I will be removing analyzers from 1.2. They should not have been put into it in the first place. Here is the main story that explains what is happening with the releases:
This is exactly why I have to rerelease 1.2 and deprecate all the packages that were already released (rc1-rc5). THere is no way I can make any stable release without freezing features. I can't freeze analyzers since they are the most complex part of the entire jNetPcap API and I will be spending a lot of time working on it. So if we want a stable release that people can use in production environments, 1.2 has to be stripped down to the parts it was originally intended to have. At the same time the dev branch is still available with the latest and greatest features but they will be in flux for some time to come.