I've been sick last few days and haven't been working on jNetPcap. I'm feeling much better now and I'm ready to finish rc5 up.
Ip4 and Tcp analyzers and reassemblers are working. There are still few final things to finish up. I expect to make good progress over next couple of days to catch up and hopefully release rc5 this weekend.
Some progress on HttpAnalysis. Implemented the HttpHandler interface. Analyzer now spews out http request/responses. Just some preliminary stuff:
public void processHttp(Http http) {
try {
out.printf("\n\n#%d *** %s ***", http.getPacket().getFrameNumber(),
http.getMessageType());
out.format(http);
} catch (IOException e) {
e.printStackTrace();
}
}
and output
#4 *** REQUEST *** Http: ******* Http offset=54 (0x36) length=476 Http: Http: RequestMethod = GET Http: RequestUrl = / Http: RequestVersion = HTTP/1.1 Http: User-Agent = Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0) Opera 7.11 [en] Http: Host = 10.1.1.1 Http: Accept = application/x-shockwave-flash,text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,video/x-mng,image/png,image/jpeg,image/gif;q=0.2,text/css,*/*;q=0.1 Http: Accept-Charset = windows-1252, utf-8, utf-16, iso-8859-1;q=0.6, *;q=0.1 Http: Accept-Encoding = deflate, gzip, x-gzip, identity, *;q=0 Http: Connection = Keep-Alive Http: HttpAnalyzer::len=160 frag=false #6 *** RESPONSE *** Http: ******* Http offset=54 (0x36) length=275 Http: Http: RequestVersion = HTTP/1.1 Http: ResponseCode = 200 Http: ResponseCodeMsg = OK Http: Date = Sat, 20 Nov 2004 10:21:06 GMT Http: Accept-Ranges = bytes Http: Content-Length = 160 Http: Connection = close Http: Content-Type = text/html; charset=ISO-8859-1 Http: #31 *** REQUEST *** Http: ******* Http offset=54 (0x36) length=574 Http: Http: RequestMethod = GET Http: RequestUrl = /Websidan/index.html Http: RequestVersion = HTTP/1.1 Http: User-Agent = Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0) Opera 7.11 [en] Http: Host = 10.1.1.1
I'm putting everything together now. I finished up the Http header type. Most specifically finished JMappedHeader implementation which allows enum tables to be used as fields. This greatly saves on the amount of code that needs to go into the file.
I still haven't fully solved the threading issue, although I haven't looked at it too much, as I started working on JMappedHeader instead. Putting the primary thread to sleep for 100ms stops the other thread from exiting prematurely. I still don't understand why. I'll revisit the issue tomorrow with new perspective.
Added JRegistry methods for managing analyzers. Also added a rudimentatary HttpAnalyzer. This is what everything I've been doing is leading up to. HttpAnalyzer is in the position to tell tcp how to reassemble the stream. Anyway, I made some good progress.
Http is implemented in terms of JMappedHeader. A base class called AbstractMessageHeader implements most of the common header stuff and then Http header defines Http specific stuff. SIP header will also extend AbstractMessageHeader and add its own field types.
Here is the complete Http and Html header definitions:
package org.jnetpcap.packet.header;
import org.jnetpcap.newstuff.AbstractMessageHeader;
import org.jnetpcap.packet.JPacket;
import org.jnetpcap.packet.annotate.Bind;
import org.jnetpcap.packet.annotate.Field;
import org.jnetpcap.packet.annotate.Header;
/**
* @author Mark Bednarczyk
* @author Sly Technologies, Inc.
*/
@Header
public class Http
extends AbstractMessageHeader {
@Bind(to = Tcp.class, intValue = {
80,
8080 })
public static boolean bindToTcp(JPacket packet, Tcp tcp) {
return tcp.destination() == 80 || tcp.source() == 80
|| tcp.destination() == 8080 || tcp.source() == 8080;
}
/**
* HTTP Request fields
*
* @author Mark Bednarczyk
* @author Sly Technologies, Inc.
*/
@Field
public enum Request {
Accept,
I've been fighting a threading issue since yesterday. A BlockQueuePump, a new class based on BlockingQueue interface, (delegates all methods to one of JRE blocking queue implementations) is supposed to dish out packets that are put on its queue in a separate worker thread. The queue takes packets but the moment I start the worker thread which takes packets from the queue and pushes them out to listeners, the thread quietly dies. It dies at unpredictable times (shortly after start) and terminates the entire VM. No errors, no a single peep out of it. The debugger can't catch it either, it just exists too. There is nothing special about the thread or the queue.
The strange thing is that there is no errors or even indication of what the problem is. Even any outstanding stdout output gets interrupted in the middle of characters being displayed. Literally the VM just abruptly ends. With success as return code.
I'm designing 2 layers of API for tcp based protocols. The first layer is the normal packet based dispatch method. Each packet is analyzed, analysis information attached, including reassembled stuff, and the packet is returned to the user.
The second layer is slightly above that. You don't work with normal packets (although you can drill down to that level), and you work with tcp streams and reassembled higher level headers. The handler is not JPacket based but each type of analyzer provides a protocol specific listener interface. For example there is a MySQLListener, a HttpListener, HtmlListener, TelnetListener, etc... This way the user picks and chooses which protocols are of interest and gets nicely decoded, analyzed and protocol specific dispatched events.
For example if you were to register a TelnetListener, it would look like something like this:
TelnetAnalyzer ta = JRegistry.getAnalyzer(TelnetAnalyzer.class);
ta.addTelnetListener(new TelnetListener() {
/*
* Telnet is a normal JHeader based header. It is fully reassembled
*/
public void processTelnet(Telnet telnet) {
/*
* So we know who is talking to who
*/
TcpDuplexStream duplex = telnet.getDuplexStream();
if (telnet.isCommand()) {
Telnet response = telnet.getResponse();
} else if (telnet.isResponse()) {
Telnet command = telnet.getCommand();
long rtt = telnet.getRTT(); // Round Trip Time between cmd/reply
}
}
});
Here is a testcase I'm working on that demonstrates various ways that higher level protocols work with TcpAnalyzers to reassemble part of the stream. This is the syntax that provides access to the reassembled data:
public void nextPacket(JPacket packet, Object user) {
try {
if (packet.hasHeader(img) && img.type() == Image.Type.JPEG) {
/*
* Content can be reassembled completely into memory, depending how
* much memory we want to dedicate for this purpose. Remember the
* content could be very very large. The call to getContent()
* signals TcpReassembler to reassembly all related TCP segments
* into a single buffer.
*/
if (img.length() < 1024 * 1024) {
JBuffer buf = img.getReassembledBuffer();
JPacket asPacket = img.getReassembledPacket();
} else {
/*
* Or for larger content, read one byte at a time using IO stream.
* getInputStream signals TcpReassembler that we are going to be
* reading 1 byte at a time out of each TCP segment. This only
* requires the use of TcpFragmentationAnalyzer to tell us what is
* the next ACKed TCP segment in the chain.
*/
InputStream in = img.getInputStream();
/*
* Or for larger content, read a buffer full at a time while
* sliding left edge of buffer/window. getSlidingBuffer() signals
* TcpReassembler that we will be reassembling various portions of
* this part of the stream. We will be adding new segments on the
* right, while letting already processed segments on the left
* expire and be released. Both sequence analysis and reassembly
* will be required.
*/
SlidingBuffer window = img.getSlidingBuffer();
}
}
if (packet.hasHeader(http) && http.isResponse() && http.hasContent()) {
out.format(packet);
}
if (packet.hasHeader(img)) {
}
} catch (IOException e) {
// TODO Auto-generated catch block
I have the tcp reassembler working... The tcp analysis is broken up into 3 classes: TcpAnalyzer, TcpFragmentationAnalyzer and TcpReassembler. TcpAnalyzer analyzes the bi-directional stream of packets and tags each tcp packet with TcpDuplexStream object. It also fires off events.
Specifically TcpFragmentationAnalyzer listens to TcpAcked events. The user uses TcpFragmentationAnalyzer.setFragmentationBoundary method to tell it which part of the stream to reassemble. Then when it receives TcpAcked events, events for segments that have been acknoledged, it groups them into a fragmentation sequence. It fires off its own events, specifically SEQUENCE_START and SEQUENCE_COMPLETE events.
TcpReassembler listens for those events. On START event, it puts a hold on the main outbound queue, preventing packets from being prematurely released to the user, then it listens for COMPLETE event. When it sees that it then has a complete list of packets or tcp segments in our case to be reassembled. It creates a new memory buffer, copies a template IP and TCP headers into it then all the segment data from the fragmentation sequence.
After all the data has been copied, it builds a JMemoryPacket around it, scans the new packet, decoding all the headers, starting with Ip4 header (I decided to drop datalink header from the new packet, since packet may have taken different physical layer paths anyhow) and tags the new packet as FragmentReassembly. It contains the reference to the original FragmentationSequence which has a list of packets that were reassembled. Then as a last step put the new packet on top of the inbound queue and lets the packet be processed once again. This time this is potentially a much larger packet, containing multi-segment headers and data in it.
Now the higher level analyzers such as HttpAnalyzer can signal Tcp layer, which portions of the stream can be reassembled.