Packet decoding

I've been working on various aspects of jNetPcap. Besides the unix and MS extensions, I've also been working on packet decoding. The decoding isn't as involved and thorough as you will find in jNetStream but should do the job for most needs. If this feature ends up in 1.2 release it will be released as a beta feature stored in a separate JAR file, so not to interfere or be confused with the rest of jNetPcap that is very stable and production quality. The single DLL contains all the necessary JNI code since none of it overlaps with the production side of the API.

The architecture for packet decoding is much simpler than it is for decoding packets in jNetStream. It also relies heavily on native code for actual decoding of core protocols. This is one great benefit that jNetStream will also be able to take advantage of, the raw speed for decoding purposes.

The new extension is called: org.jnetpcap.packet and contains a sub package org.jnetpcap.packet.header which stores several CORE headers that are going to be standard with the release. The package also allows easy addition of new types of headers and a binding system. This all works with the native code.

The new functionality is mostly implemented in 2 new classes JScanner and JPacket. JScanner keeps a table of header types and their bindings, scans a buffer containing packet data and outputs its findings as a series of bits masks and an array of offsets into a JPacket class. JPacket class allows the user to query if a particular header is in the packet and then retrieve it. All this is done mainly in native code.

JScanner does not instantiate new JPacket objects, but requires the user to supply an object that it then re-references to new state. This means that JScanner, JPacket and subclasses JHeader are all reused. The storage area for all these classes and peered C structures is allocated out of a large round-robin buffer that JScanner manages. This is the same concept that libpcap uses with its capture buffer and the way it round-robins around that buffer. JScanner has a similar matching buffer for storing decoded packet state.

JScanner is extremely EFFICIENT in its design and runtime performance which is the key aspect of its design. This is accumulation of many years of experience designing and working on decoding packets in an Object oriented way

All this is done completely behind the scenes and efficiently. The user will simply use a new dispatcher method that will dispatch JPackets instead of ByteBuffers. The JPackets will already contain the decoded packet state.

The basic outline of how this is used I present in this jUnit test case:

/* Some reusable JHeader classes for our decoded packets */
private Ethernet ethernet = new Ethernet();
private Ip4 ip4 = new Ip4();
private Ip6 ip6 = new Ip6();

pcap = // Do the normal Pcap stuff to open up interface
pcap.dispatch(2, new JPacketHandler<String>() {
	public void nextPacket(PcapHeader header, JPacket packet, String user) {

		if (packet.hasHeader(ethernet)) {
			System.out.println("ethernet.dst=" + ethernet.destination());
		}

		if (packet.hasHeader(ip4)) {
			System.out.println("ip4.ver=" + ip4.version());
		}

		if (packet.hasHeader(ip6)) {
			System.out.println("ip6.ver=" + ip6.version());
		}
	}

}, "JPacket - testcase");

JPacket.hasHeader(<? extends JHeader>):boolean is a method that checks if the specified header is found in the packet, if yes, then that supplied header object is redirected to point at the packet data buffer at the right offset. Both of these operations are very fast and efficient since this state is already recorded in bit encoded unsigned integer and a native array of offsets. To test if header exists is a simple bitwise AND and comparison, while retrieval is a array lookup into an array. Both are fast and efficient.

There are other alternatives for methods, but that is main API interface the user will use to access decoded packets.

There are several other variations of the Pcap.dispatch method which allow the user to specify a different JScanner and a JPacket that is also reused to be dispatched to the user. The JPacket in the handler is reused not allocated on a per call basis.

The JScanner class maintains a global registry of default headers and their bindings. The default Pcap.dispatch call just uses the defaults, but the user can reconfigure the registry and the binding any way that is needed. The CORE protocols use a native hardcoded scanning routine, which can also be overriden by the user. Any protocol can provide a set of new header to header bindings.

This provides enough flexibility that multiple scanners can to be working with different set of bindings and even headers if needed.

The native scanning routine has all the CORE protocols hardcoded for super fast scanning. At the same time it will also use user supplied java bindings (using JBinding interface) if not match was made on a particular header. The user can also override those hardcoded bindings and replace them with a java binding.

That native scanner utilizes bitmasks in an unsigned long. There are 2 versions a JSmallScanner and a JLargeScanner. The small scanner is optimized for fewer that 64 headers total. While the large scanner can handle unlimited set of protocols. The scanner in both cases uses several bit maps and arrays to aid in decoding. For example presence of java bindings is encoded in a single unsigned long. Binding dependency, that is a list of other headers that must have already been decoded in the packet before the java binding should be attempted, is also encoded in a bit map. The CORE scanner bindings are also overridden in this way even though they are hard coded. In conclusion, all the heavy duty information is boiled down to essentially 3 unsigned longs that contain most of the configuration of a protocol registry, header mapping within a packet buffer and protocol bindings. The scanner makes most of its decisions just making a few bitwise ANDs. Like I said, super efficient.

The scanner utilizes a large memory block to store its configuration, tables and sub-allocated out of this large buffer for storage of a per packet state. As new packets arrive they are simply assigned next struct packet_t block within the large block. A single pointer is changed in JPacket to point at the new configuration state (struct packet_t) and thats it. Same thing for all the JHeader subclasses. They are simply repointed to a new struct header_t which contains the header state, all natively stored in memory.

The scanner C++ code is very portable and doesn't rely on any OS calls, so this should be available on all the platforms that jNetPcap is currently distributed for. The java source code is in the jnetpcap SVN module, but under a different source code tree within. The build scripts compile the source code into a seperate JAR file. All the user has to do is to include that secondary JAR file to have that functionality, but it seperates the production from the new beta feature. At some point this beta feature will be merged with the production source code tree. The package names between beta and production sources overlap and represent exactly where the new classes will be when moved to production. The only difference is that a new Pcap subclass exists called PcapBeta which subclasses Pcap class. The PcapBeta class adds all the new functions which will eventually be moved into the main Pcap class when in production.

We'll see how this model of beta vs. production source works. If it works well, I will utilize the approach to all new features in the future to separate production from beta and allow great flexibility in nailing down the final API.