Justniffer

justniffer is a TCP packet sniffer. It can log network traffic in a 'standard' (web server like) or in a customized way. It can also log response times, useful for tracking network services performances (e.g. web server, application server, etc.). It can capture HTTP traffic and rebuild HTTP file.

Main differences from other sniffers

Most of the sniffers are divided into two categories, packet an text sniffers. Both suffer from incompleteness of information that may be collected and analyzed

Packet sniffers collect too much data, such as packet headers details, and they make easy to analize low level network problem (such as tcp retrasmissions, or ip fragmentation) but make hard-working and time wasting to analize more high level details ( such as content correctness, keep-alive issues, connection timeouts, response time, etc)
Textmode sniffers usually rebuild TCP stream but cannot collect low level information such as timestamps. They often rebuild the tcp flow in a too simplistic way and fail when dealing with complex TCP/IP issues (reordering, retransmission, reassemlbying, etc). TCP reassembling and reordering is a complex exercise, and require a deep knowledge of TCP/IP protocol and long experience in the field. Usually,they are useful for pure grabbing content from network traffic.

Justniffer was born to help in toubleshooting perfomance in network tcp based services : HTTP, JDBC, RTSP, SIP, SMTP, IMAP, POP, LDAP, etc.

It can collect low and hight level protocol and performance info reconstructing the tcp flow in a reliable way using portions of the Linux Kernel code. Precisely, it uses a slightly modified version of the libnids libraries that already include a modified version of linux kernel code in a more reusable way.

You can extend traffic analisys with external scripts (bash, python, or any executable). An example is provided: justniffer-grab-http-trafficscript uses justniffer to saves files (images, text, html pages, javascript, flash, video, etc) captured from HTTP traffic.

Justniffer can generate logs in a customizable way. For example it can mimic the apache access_log

Justniffer features summary
TCP flow rebuild	very reliable: it can reorder, reassemble tcp segments and ip fragments using portions of the Linux kernel code
Logging	text mode: can be customized
Extensibility	by any executable, such as bash, python, perl scripts, ELF executable, etc.
Performance measurement	it can collect many information on performances: connection time, close time, request time , response time, close time, etc.

Install

Examples

Example 1

Retrieving http network traffic in access_log format

$ justniffer -i eth0

output:

192.168.2.2 - - [15/Apr/2009:17:19:57 +0200] "GET /sflogo.php?group_id=205860&type=2 HTTP/1.1" 200 0 "" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.8) Gecko/2009032711 Ubuntu/8.10 (intrepid) Firefox/3.0.8)"

192.168.2.2 - - [15/Apr/2009:17:20:18 +0200] "GET /search?q=subversion+tagging&ie=utf-8&oe=utf-8&aq=t&rls=com.ubuntu:en-US:unofficial&client=firefox-a HTTP/1.1" 200 0 "" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.8) Gecko/2009032711 Ubuntu/8.10 (intrepid) Firefox/3.0.8)"

192.168.2.2 - - [15/Apr/2009:17:20:07 +0200] "GET /sflogo.php?group_id=205860&type=2 HTTP/1.1" 200 0 "http://justniffer.sourceforge.net/" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.8) Gecko/2009032711 Ubuntu/8.10 (intrepid) Firefox/3.0.8)"

192.168.2.2 - - [15/Apr/2009:17:20:18 +0200] "GET /csi?v=3&s=web&action=&tran=undefined&ei=MvvlSdjOEciRsAbY0rGpCw&e=19592,20292&rt=prt.175,xjs.557,ol.558 HTTP/1.1" 204 0 "http://www.google.it/search?q=subversion+tagging&ie=utf-8&oe=utf-8&aq=t&rls=com.ubuntu:en-US:unofficial&client=firefox-a" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.8) Gecko/2009032711 Ubuntu/8.10 (intrepid) Firefox/3.0.8)"

192.168.2.2 - - [15/Apr/2009:17:20:07 +0200] "GET / HTTP/1.1" 200 0 "" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.8) Gecko/2009032711 Ubuntu/8.10 (intrepid) Firefox/3.0.8)"

Example 2

We can append fields. For example http response time (see man pagefor a complete keyword list)

$ justniffer -i eth0 -a " %response.time"

output:

Example 3

You can capture all tcp traffic (add -u or -x options to encode unprintable characters):

$ justniffer -i eth0 -r

output:

GET /doc/maint-guide/ch-upload.en.html HTTP/1.1
Host: www.debian.org
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.8)
Gecko/2009032711 Ubuntu/8.10 (intrepid) Firefox/3.0.8
Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en,it;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: UTF-8,*
Keep-Alive: 300
Connection: keep-alive
Referer: http://www.debian.org/doc/maint-guide/
If-Modified-Since: Wed, 22 Apr 2009 19:36:31 GMT
If-None-Match: "400d604-3014-46829e160adc0"
Cache-Control: max-age=0

HTTP/1.1 304 Not Modified
Date: Wed, 22 Apr 2009 20:38:51 GMT
Server: Apache
Connection: Keep-Alive
Keep-Alive: timeout=15, max=100
ETag: "400d604-3014-46829e160adc0"
Expires: Thu, 23 Apr 2009 20:38:51 GMT
Cache-Control: max-age=86400

Example 4

You can define a completely custom log format

$ justniffer -i eth0 -l " %request.timestamp %source.ip %dest.ip %request.header.host %request.url"

output:

06/28/09 13:30:48 192.168.2.2 72.14.221.118 i1.ytimg.com /vi/TjSk6CVN5LY/default.jpg

06/28/09 13:30:47 192.168.2.2 72.14.221.118 i2.ytimg.com /vi/iw_nzfm1Vts/default.jpg

06/28/09 13:30:47 192.168.2.2 216.34.181.71 static.sourceforge.net /css/phoneix/jquery.cluetip.php?secure=0

06/28/09 13:30:48 192.168.2.2 216.34.181.71 static.sourceforge.net /sfx.js

06/28/09 13:30:49 192.168.2.2 216.34.181.71 static.sourceforge.net /include/coremetrics/v40/eluminate.js

06/28/09 13:30:51 192.168.2.2 199.93.61.126 c.fsdn.com /sf/images/phoneix/grad_white_dual_100.png

Example 5

You can read from a capture file. NOTE: capture file must be performed with unlimited snaplen for catching whole packets. Justniffer can work only works on pcap files with whole packets.
tcpdump command example: tcpdump -w /tmp/file.cap -s 0 -i ath0

$ justniffer -f /file.cap

Example 6

Many keyword has paramaters for setting a more precise formatting:

$ justniffer -i eth0 -l "%request.timestamp %request.header.host %request.url %response.time"

output:

06/28/09 13:39:40 static.sourceforge.net /css/phoneix/print.css?secure=0&20080417-1657 0.162620

you can specifing timestamp formatting

$ justniffer -i eth0 -l "%request.timestamp (%B %d %Y %H:%M:%S)%request.header.host %request.url %response.time"

output:

June 28 2009 13:39:40static.sourceforge.net /css/phoneix/print.css?secure=0&20080417-1657 0.162620

or to print the string NoHostFound if the %request.header.host cannot be valued

$ justniffer -i eth0 -l "%request.timestamp %request.header.host (NoHostFound)%request.url %response.time"

output:

06/28/09 15:10:28 www.google.com /ig?hl=en 0.116146

06/28/09 15:10:28 NoHostFound/ig?hl=en 0.116146

many keywords have their own formatting string. A generic option ( -n) can be used for setting a default "not found" string: a string that must replace the keyword if it cannot be valued. for example if request.header.[headername] is not found or if connection.time cannot be applicable. Anyway, if a "no found" string is provided as keyword parameter (for those that expect it), it will override the -n option

$ justniffer -i eth0 -l "%request.timestamp %request.header.host(NoHostFound) %request.header.host%request.url %response.time" -n N/A

output:

06/28/09 15:10:28 www.google.com www.google.com/ig?hl=en 0.116146

06/28/09 15:10:28 NoHostFound N/A/ig?hl=en 0.116146

Example 7

the -p option let you to specify a tcpdump compatibile filter: "port 80 or port 8080" capture only http traffic (usually using tcp port 80 and 8080)

$ justniffer -i eth0 -r -p "port 80 or port 8080"

Example 8

the -e option let you to specify an external executable (usually a script) to which , for every log, the output will be redirect to. If you want to perform complex extraction operations, you can write your own script that will receive from the standard input all content specified by the -l option. A complete ad useful example is provided with justniffer-grab-http-traffic

$ justniffer -l "%response" -e ./myscript.sh -i ath0

#!/bin/bash
# myscript.sh 
# example script (print all lines containing "href" string)

while read inputline
do 
 anchors=`echo "$inputline" | grep href`
 if [ "$anchors" != "" ]; then
    echo $anchors;
 fi;
done

Example 9

capture smtp traffic (usually using tcp port 25)

$ justniffer -i eth0 -r -p "port 25"

output:

220 plecno.com ESMTP Postfix (Ubuntu)

EHLO unknown.localnet
250-plecno.com       
250-PIPELINING       
250-SIZE             
250-VRFY             
250-ETRN             
250-STARTTLS         
250-ENHANCEDSTATUSCODES
250-8BITMIME           
250 DSN                

MAIL FROM:<oreste.notelli@plecno.com> SIZE=1079
RCPT TO:<oreste.notelli@gmail.com>             
DATA                                           
250 2.1.0 Ok                                   
250 2.1.5 Ok                                   
354 End data with <CR><LF>.<CR><LF>        
   

From: Oreste Notelli <oreste.notelli@plecno.com>
Organization: Plecno
To: oreste.notelli@gmail.com
Subject: test
Date: Wed, 22 Apr 2009 22:46:16 +0200
User-Agent: KMail/1.11.2 (Linux/2.6.27-8-generic; KDE/4.2.2; i686;
; )
MIME-Version: 1.0
Content-Type: multipart/alternative;
  boundary="Boundary-00=_ZI47J3FTNXn+25g"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
Message-Id: <200904222246.17292.oreste.notelli@plecno.com>

--Boundary-00=_ZI47J3FTNXn+25g
Content-Type: text/plain;
  charset="us-ascii"
Content-Transfer-Encoding: 7bit

test

--Boundary-00=_ZI47J3FTNXn+25g
Content-Type: text/html;
  charset="us-ascii"
Content-Transfer-Encoding: 7bit

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0//EN"
"http://www.w3.org/TR/REC-html40/strict.dtd">
<html>
<head>
<meta name="qrichtext" content="1" />
<style type="text/css">p, li { white-space: pre-wrap;
}</style>
</head>
<body style=" font-family:'DejaVu Sans'; font-size:8pt;
font-weight:400; font-style:normal;">
<p>
test
</p>
</body>
</html>
--Boundary-00=_ZI47J3FTNXn+25g--
.
250 2.0.0 Ok: queued as 33E7235C21A

QUIT
221 2.0.0 Bye

Example 10

The "grep keywords"can be used to capture portions of text using regular expressions. In this example we want to collect:

the url from the request header by the regular expression [^\s]*[\s]*([^\s]*)
and the content type from the response header by the regular expression Content-Type:(\s)*([^\r]*)

$ sudo justniffer -l "%request.header.grep([^\s]*[\s]*([^\s]*)) %response.header.grep(Content-Type:(\s)*([^\r]*)) %source.ip" -i eth0

output:

/ text/html 192.168.10.2

/plecno_res/src/effects.js application/javascript 192.168.10.2

/plecno_res/src/slider.js application/javascript 192.168.10.2

/plecno_res/src/effects.js application/javascript 192.168.10.2

/plecno_res/src/builder.js application/javascript 192.168.10.2

/plecno_res/src/effects.js application/javascript 192.168.10.2

Install

On Ubuntu ( 11.04 / 11.10 / 12.04)

    $ sudo apt-get install add-apt-repository
    $ sudo add-apt-repository ppa:oreste-notelli/ppa
    $ sudo apt-get update
    $ sudo apt-get install justniffer

Other Ubuntus

Download the .deb file from the file repository and install it:

    $ sudo apt-get install gdebi-core
    $ sudo gdebi justniffer-x.x.deb

Other Distributions

be sure you have installed third-party tools and libraries:

    patch
    tar
    autotools
    make
    libc6
    libpcap0.8
    g++                         
    gcc 
    libboost-iostreams          
    libboost-program-options
    libboost-regex

unpacked the source package, type:

    $ ./configure
    $ make
    $ make install

Copyright (c) 2007-2011 Plecno s.r.l. All Rights Reserved 
info@plecno.com
via Giovio 8, 20144 Milano, Italy
Released under the terms of the GPLv3 or later
Author: Oreste Notelli < oreste.notelli at plecno.com >