Justniffer

Network TCP Packet Sniffer

Justniffer is a network protocol analyzer that captures network traffic and produces logs in a customized way, can emulate Apache web server log files, track response times and extract all "intercepted" files from the HTTP traffic.

It lets you interactively trace tcp traffic from a live network or from a previously saved capture file. Justniffer's native capture file format is libpcap format, which is also the format used by tcpdump and various other tools.

Reliable TCP Flow Rebuilding

The main Justniffer's feature is the ability to handle all those complex low level protocol issues and retrieve the correct flow of the TCP/IP traffic: IP fragmentation, TCP retransmission, reordering. etc. It uses portions of Linux kernel source code for handling all TCP/IP stuff. Precisely, it uses a slightly modified version of the libnids libraries that already include a modified version of Linux code in a more reusable way.

Optimized for "Request / Response" protocols. It is able to track server response time

Justniffer was born as tool for helping in analyzing performance problem in complex network environment when it becomes impractical to analyze network captures solely using wireshark. It will help you to quickly identify the most significant bottlenecks analyzing the performance at "application" protocol level.

In very complex and distributed systems is often useful to understand how communication takes place between different components, and when this is implemented as a network protocol based on TCP/IP (HTTP, JDBC, RTSP, SIP, SMTP, IMAP, POP, LDAP, REST, XML-RPC, IIOP, SOAP, etc.), justniffer comes in handy. Often the logging level and monitoring systems of these systems does not report important information to determine performance issues such as the response time of each network request. Because they are in a "production" environment and cannot be too much verbose or they are in-house developed applications and do not provide such logging.

Other times it is desirable to collect access logs from web services implemented on different environments (various web servers, application servers, python web frameworks, etc.) or web services that are not accessible and therefore traceable only on client side.

Justniffer can capture traffic in promiscuous mode so it can be installed on dedicated and independent station within the same network "collision domain" of the gateway of the systems that must be analyzed, collecting all traffic without affecting the system performances and requiring invasive installation of new software in production environments.

Can rebuild and save HTTP content on files

The robust implementation for the reconstruction of the TCP flow turns it in a multipurpose sniffer.

  • HTTP sniffer
  • LDAP sniffer
  • SMTP sniffer
  • SIP sniffer
  • password sniffer
justniffer can also be used to retrieve files sent over the network.

It is extensible

Can be extended by external scripts. A python script has been developed to recover all files sent via HTTP (images, text, html, javascript, etc.).

Features Summary

  • Reliable TCP flow rebuilding: it can reorder, reassemble tcp segments and ip fragments using portions of the Linux kernel code
  • Logging text mode can be customized
  • Extensibility by any executable, such as bash, python, perl scripts, ELF executable, etc.
  • Performance measurement it can collect many information on performances: connection time, close time, request time , response time, close time, etc.

Install

On Ubuntu ( 11.04 / 11.10 / 12.04 / 14.04 )

$ sudo add-apt-repository ppa:oreste-notelli/ppa 
$ sudo apt-get update
$ sudo apt-get install justniffer

Other Ubuntus

Download the .deb file from the Ubuntu file repository and install it:

$ sudo apt-get install gdebi-core 
$ sudo gdebi justniffer-x.x.deb

Other Distributions

be sure you have installed third-party tools and libraries:

patch
tar
autotools
make
libc6
libpcap0.8
g++                         
gcc 
libboost-iostreams          
libboost-program-options
libboost-regex  

unpacked the source package, type:

$ ./configure 
$ make 
$ make install

JUSTNIFFER

Section: (8)
Updated: September 12, 2011

NAME

justniffer - tcp flow sniffer  

SYNOPSIS

justniffer [ [-i interface] or [-f <tcpdump file>] ] [-F]
[-p <packet filter>] [-u or -x] 
[ [-r] or [-l <log format>] or [-a <log format>]  ] 
[-c <config file>]  [-e <external program>]  [-U <user> ]   
[-n <not-found> ]  [-s <max concurrent tcp streams> ]  
[-d <max concurrent IP fragments> ]

Examples: 
Logging network traffic in Apache like format:
  justniffer -i eth0 

Tracing tcpdump dump in Apache like format:
  justniffer -f file.cap 

Tracing TCP streams from the network:
  justniffer -i eth0 -r 

Logging network traffic in Apache like format appending response times:
  justniffer -i eth0 -a " %response.time" 

Logging network traffic in a customized format:
  justniffer -i eth0 -l "%source.ip%tab%dest.ip%tab%request.header.host%tab%response.time" 


 

DESCRIPTION


 justniffer captures reassembles and reorders TCP packets, performs IP packet defragmentation 
and displays the tcp flow in the standard output.
 It is useful for logging network traffic in a 'standard' (web server like) or in a customized way.
  It can log network services performances (e.g. web server response time , application server keep alive behaviour, etc.) .
 Output format can be customized using the log format option -l (see FORMAT KEYWORDS). 
 Most of them can be used for retrieving HTTP protocol informations. See EXAMPLES

 

TRACKING PERFORMANCES


 The main feature of justniffer is to analize network traffic to monitor performances. The performances related keywords are:
  %connection.time
  %idle.time.0
  %request.time
  %response.time
    %response.time.begin
    %response.time.end
  %idle.time.1


+---------+ +---------+
| | | |
| Client | | Server |
| | | |
+---------+ +---------+
| |
| ----- connect syn --------> |----+
| | |
| <------ syn/ack ---------> | | %connection.time
| | |
| ------- ack ----------> | |
| ESTABLISHED |----+
| | | %idle.time.0
| | |(after connection, before
| | | request)
| | |
| --- request/first packet ---> |----+
| <------ ack ----------- | |
| | |
| --- request/.... ---> | | %request.time
| <------ ack ----------- | |
| | |
| --- request/last packet ---> | |
| <------ ack ----------- |----+--------------------+
| | | |
| | | |
| | |%response.time.begin |
| | | |
| <-- response/first packet ---- |----+ | response
| ------- ack ----------> | | | time
| | | |
| <-- response/.... ---- | |%response.time.end |
| ------- ack ----------> | | |
| | | |
| <-- response/last packet ---- | | |
| ------- ack ----------> |----+--------------------+
| | |
| | |
| | | %idle.time.1 (after response,
| | | before new request or close)
| | |
| <------ close ---------> |----+
| | |
| | |

 

OPTIONS

-i or --interface=<interface>interface to listen on (e.g. eth0, en1, etc.)
Example: justniffer -i eth0
-f or --filecap=<file>
tcpdump file to read from (for offline network traffic processing). It must be a pcap file (produced by network capture programs such as tcpdump or wireshark) WARNING: justniffer needs a complete dump, usually, sniffers collect just the few first (96) bytes per packet. (when using tcpdump you must specify "-s 0" option. Example: tcpdump -i eth0 -s 0 -w /tmp/file.cap)
Example: justniffer -f /tmp/file.cap
-F or --force-read-pcap
Force the reading of a pcap file also if captured with truncated packets (for example when using tcpdump without the "-s 0" option)
-p or --packet-filter=<filter>
packet filter (pcap filter syntax) (see pcap-filter(7))
Examples:


  sniffing proxy traffic
    justniffer -i eth0 -p "tcp port 8080"


  sniffing row telnet traffic
    justniffer -i eth0 -r -p "tcp port 23"


  sniffing row pop3 traffic
    justniffer -i eth0 -r -p "tcp port 110"


  sniffing  traffic from/to a specific host
    justniffer -i eth0 -r -p "host 10.10.10.2"


  sniffing  HTTP traffic from/to a specific host and port
    justniffer -i eth0 -r -p "host 10.10.10.2 and tcp port 80"

-l or --log-format=<format>
log format. You can specify the output string format containing reserved keyword that will be replaced with the proper value (see FORMAT KEYWORDS). If missing, the CLF (Common Log Format, used by almost all web servers) is used as default.
Example: justniffer -i eth0 -l "\"%request.line\"%tab%response.time"


    "POST /v2/rss/alerts?src=ffbmext2.1.034 HTTP/1.1" 0.139011
    "POST /v2/rss/network/oreste.notelli?src=ffbmext2.1.034 HTTP/1.1" 0.623382
    "GET /man_page_howto.html HTTP/1.1"       0.024437
    "GET /ig?hl=en HTTP/1.1"  0.764945
    "GET /?hl=en&tab=wv HTTP/1.1"   0.242342
    "GET /s/-yCdCsgUnsI/css/homepage_c.css HTTP/1.1"        0.071942
    "GET /vi/YUvWcegtqik/default.jpg HTTP/1.1"      0.821472

-r or --raw
show raw stream. it is a shortcut for -l %request%response
Example: justniffer -i eth0 -r
-s or --max-tcp-streams
max concurrent TCP stream. (default= 65536) excess will be discarded
-d or --max-fragmented-ip
max concurrent fragmented IP. (default= 65536) excess will be discarded
-u or --unprintable
encode as dots (.) unprintable characters ( for more control on character you should use pipelining to cat -v.
Example:

  justniffer -i eth0 -l "%request%newline%response"| cat -v
  justniffer -i eth0 -ru
-x or --hex-encode
encode unprintable characters in [<char hex code>] format
Example:

  justniffer -i eth0 -rx
-n or --not-found=<not found string>
Not found string. It is used to replace a specified keyword when it cannot be valued because it is not found. All request.header.* and response.header.* keywords can override the "not found string" passing it as parameter. For example:
 %request.header.host() will be replaced by the Host header value, or an empty string if Host header not found
 %request.header.host(UNKNOWN) will be replaced by the Host header 
value, or the UNKNOWN string if Host header not found
 %request.header.host will be replaced by the Host header value, or , if Host header not found, with 
the string specified by the -n option 
Examples:

  justniffer -i eth0 -l "%request.header.connection" -n N/A  


  will produce such logs:
    N/A
    N/A
    keep-alive
    close

Note: each keyword can override the "not found string" value:


  justniffer -i eth0 -l "%request.header.connection(None) %request.header.connection(-) %request.header.connection() %request.header.connection" -n N/A  


  will produce such logs:
    None -  N/A
    None -  N/A
    keep-alive keep-alive keep-alive keep-alive
    close close close close

-e or --execute=<external program>
call the specified external program/shell script pipelining the standard output for each request/response phase You can write shell script for handling, for example, HTTP traffic
Example:

  justniffer -i eth0 -l "%request%newline%response" -e "tail -2 "
-U or --user=<user>
User to impersonate when executing the program specified with the -e option, used to avoid to security exploits when running justniffer with root privileges
Example:

  justniffer -i eth0 -l "%request%newline%response" -e "grep password >> /tmp/passwords.txt"  -U guest
-c or --config=<config file>
configuration file. You can specify options in a configuration file (command line options override file configuration options) using the following format specifications:
   <option> = <value>

   configuration file example:
log-format = "%request.url %request.header.host %response.code %%response.time"

packet-filter = "tcp port 80 or tcp port 8080 or tcp port 3526"

 

FORMAT KEYWORDS

List of all recognized keywords:
%close.time([not applicable string])
elapsed time from last response and when the connection is closed. the "not applicable" string is replaced in case the keyword value cannot be applicable. if not provided the -n value or the default value "-" is used
%close.timestamp([format])
is replaced by the close timestamp. You can use optional format specification (see TIMESTAMP FORMAT)
%close.timestamp2([not applicable string])
is replaced by the connection timestamp using format "seconds.microseconds" the "not applicable" string is replaced in case the keyword value cannot be applicable. if not provided the -n value or the default value "-" is used
%connection
 connection persistence indicator:
    unique: the request/response is the unique in the tcp connection
    start: the request/response is the first in the tcp connection
    last : the request/response is the last in the tcp connection
    continue : the request/response is the middle in the tcp connection
%connection.time([not applicable string])
elapsed time for establishing the TCP connection. The "not applicable" string is replaced in case the keyword value cannot be applicable. if not provided the -n value or the default value "-" is used
%connection.timestamp([format])
is replaced by the connection timestamp. You can use optional format specification (see TIMESTAMP FORMAT)
%connection.timestamp2([not applicable string])
is replaced by the connection timestamp using format "seconds.microseconds" the "not applicable" string is replaced in case the keyword value cannot be applicable. if not provided the -n value or the default value "-" is used
%idle.time.0
elapsed time form when the connection is established and the request is started the "not applicable" string is replaced in case the keyword value cannot be applicable. if not provided the -n value or the default value "-" is used
%idle.time.1
elapsed time form when the last response and the next request (or the connection closing) the "not applicable" string is replaced in case the keyword value cannot be applicable. if not provided the -n value or the default value "-" is used
%response.time.begin
elapsed time form when the request and the response start the "not applicable" string is replaced in case the keyword value cannot be applicable. if not provided the -n value or the default value "-" is used
%response.time.end
elapsed time form the response start and the response end the "not applicable" string is replaced in case the keyword value cannot be applicable. if not provided the -n value or the default value "-" is used
%response.timestamp([format])
elapsed time for the whole response the "not applicable" string is replaced in case the keyword value cannot be applicable. if not provided the -n value or the default value "-" is used
%dest.ip
is replaced by the destination IP address
%dest.port
is replaced by the destination TCP port
%source.ip
is replaced by the source IP address
%source.port
is replaced by the source TCP port
%request
is replaced by the the whole request ( (it is multiline and may contain unprintable characters)
%request.timestamp([format])
is replaced by the request timestamp. You can use optional format specification (see TIMESTAMP FORMAT)
%request.timestamp2([not applicable string])
is replaced by the request timestamp using format "seconds.microseconds" the "not applicable" string is replaced in case the keyword value cannot be applicable. if not provided the -n value or the default value "-" is used
%request.size
is replaced by the request size (including request header size)
%request.line
is replaced by the request line (e.g. "GET /index.html HTTP/1.1")
%request.method
is replaced by the request method (e.g. GET, POST, HEAD)
%request.url
is replaced by the url
%request.protocol
is replaced by the protocol (e.g. HTTP/1.0, HTTP/1.1)
%request.grep(<regular-expression>)
is replaced by the result of the specified regular expression applied on the whole request [Perl regular expression syntax, see perlre(1) or perl(1)]. The most nested subgroup is returned
%request.header
is replaced by the request header (it is multiline)
%request.header.host([not found string])
is replaced by the request Host header value. The optional "not found" string is replaced in case the keyword value was not found. if not provided the -n value or the default value "-" is used
%request.header.user-agent([not found string])
is replaced by the request User-Agent header value. The optional "not found" string is replaced in case the keyword value was not found. if not provided the -n value or the default value "-" is used
%request.header.accept([not found string])
is replaced by the request Accept header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used
%request.header.accept-language([not found string])
is replaced by the request Accept-Language header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used
%request.header.accept-charset([not found string])
is replaced by the request Accept-Charset header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used
%request.header.accept-encoding([not found string])
is replaced by the request Accept-Encoding header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used
%request.header.authorization([not found string])
is replaced by the request Authorization header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used
%request.header.connection([not found string])
is replaced by the request Connection header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used
%request.header.content-encoding([not found string])
is replaced by the request Content-Encoding header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used
%request.header.content-length([not found string])
is replaced by the request Content-Length header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used
%request.header.content-md5([not found string])
is replaced by the request Content-MD5 header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used
%request.header.cookie([not found string])
is replaced by the request Cookie header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used
%request.header.range([not found string])
is replaced by the request Range header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used
%request.header.referer([not found string])
is replaced by the request Referer header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used
%request.header.keep-alive([not found string])
is replaced by the request Keep-Alive header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used
%request.header.value(<header-name>
is replaced by the request header value (e.g. "%request.header.value(Cookie)") The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used
%request.header.transfer-encoding([not found string])
is replaced by the request Transfer-Encoding header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used
%request.header.via([not found string])
is replaced by the request Via header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used
%request.header.grep(<regular-expression>)
is replaced by the result of the specified regular expression applied on the request header [Perl regular expression syntax, see perlre(1) or perl(1)]. The most nested subgroup is returned (e.g. to obtain the request URL: "%request.header.grep(^[^\s]*\s*([^\s]*))"
%response
is replaced by the while response (it is multiline and may contain unprintable characters)
%response.timestamp([format])
is replaced by the response timestamp. You can use optional format specification (see TIMESTAMP FORMAT)
%request.timestamp2([not applicable string]))
is replaced by the response timestamp using format "seconds.microseconds" the "not applicable" string is replaced in case the keyword value cannot be applicable. if not provided the -n value or the default value "-" is used
%response.size
is replaced by the response size (including response the header size)
%response.time
is replaced by the response time (difference from the request time and the time the response finish)
%response.line
is replaced by the response line
%response.protocol
is replaced by the response protocol
%response.code
is replaced by the response code (e.g. 200, 404, 500, etc.)
%response.message
is replaced by response message (e.g. OK, Not Found, Internal Server Error, etc.)
%response.grep(<regular-expression>)
is replaced by the result of the specified regular expression applied on the whole response [Perl regular expression syntax, see perlre(1) or perl(1)]. The most nested subgroup is returned
%response.header
is replaced by the response header (it is multiline)
%request.header.allow([not found string])
is replaced by the request Allow header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used
%response.header.server([not found string])
is replaced by the response Server header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used
%response.header.date([not found string])
is replaced by the response Date header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used
%response.header.content-length([not found string])
is replaced by the response Content-Length header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used
%response.header.content-type([not found string])
is replaced by the response Content-Type header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used
%request.header.content-md5([not found string])
is replaced by the request Content-MD5 header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used
%request.header.content-range([not found string])
is replaced by the request Content-Range header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used
%response.header.content-encoding([not found string])
is replaced by the response Content-Encoding header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used
%response.header.content-language([not found string])
is replaced by the response Content-Language header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used
%response.header.transfer-encoding([not found string])
is replaced by the response Transfer-Encoding header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used
%%response.header.expires is replaced by the response Expires header value
%response.header.etag([not found string])
is replaced by the response ETag header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used
%response.header.cache-control([not found string])
is replaced by the response Cache-Control header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used
%response.header.last-modified([not found string])
is replaced by the response Last-Modified header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used
%response.header.pragma([not found string])
is replaced by the response Pragma header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used
%response.header.age([not found string])
is replaced by the response Age header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used
%response.header.connection([not found string])
is replaced by the response Connection header value. The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used
%response.header.keep-alive([not found string])
is replaced by the response Keep-Alive header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used
%response.header.via([not found string])
is replaced by the response Via header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used
%response.header.vary([not found string])
is replaced by the response Vary header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used
%request.header.www-authenticate([not found string])
is replaced by the request WWW-Authenticate header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used
%response.header.accept-ranges([not found string])
is replaced by the response Accept-Ranges header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used
%response.header.set-cookie([not found string])
is replaced by the response Set-Cookie header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used
%request.header.value(<header-name>)
%%response.header.value(<header-name>) is replaced by the response header value (e.g. "%request.header.value(Set-Cookie)")
%response.header.grep(<regular-expression>)
is replaced by the result of the specified regular expression applied on the response header [Perl regular expression syntax, see perlre(1) or perl(1)]. The most nested subgroup is returned (e.g. to obtain the request URL: "%request.header.grep(^[^\s]*\s*([^\s]*))"
%tab
is replaced by a tab
%-
break (used for breaking keywords). For example, if you want to obtain output like this:

"0.234342                  seconds"

you must use the break keyword (%-) to mark the %tab keyword end:


 "%response.time%tab%-seconds" 

%%
is replaced by the '%' character
%newline
is replaced by a newline
 

TIMESTAMP FORMAT

Timestamp format keywords (see strftime(3) ) :
%A
is replaced by national representation of the full weekday name.
%a
is replaced by national representation of the abbreviated weekday name.
%B
is replaced by national representation of the full month name.
%b
is replaced by national representation of the abbreviated month name.
%C
is replaced by (year / 100) as decimal number; single digits are preceded by a zero.
%c
is replaced by national representation of time and date.
%D
is equivalent to ``%m/%d/%y''.
%d
is replaced by the day of the month as a decimal number (01-31).
%E* %O*
POSIX locale extensions. The sequences %Ec %EC %Ex %EX %Ey %EY %%Od %Oe %OH %OI %Om %OM %OS %Ou %OU %OV %Ow %OW %Oy are supposed to provide alternate representations.

Additionally %OB implemented to represent alternative months names (used standalone, without day mentioned).

%e
is replaced by the day of month as a decimal number (1-31); single digits are preceded by a blank.
%F
is equivalent to ``%Y-%m-%d''.
%G
is replaced by a year as a decimal number with century. This year is the one that contains the greater part of the week (Monday as the first day of the week).
%g
is replaced by the same year as in ``%G'', but as a decimal number without century (00-99).
%H
is replaced by the hour (24-hour clock) as a decimal number (00-23).
%h
the same as %b.
%I
is replaced by the hour (12-hour clock) as a decimal number (01-12).
%j
is replaced by the day of the year as a decimal number (001-366).
%k
is replaced by the hour (24-hour clock) as a decimal number (0-23); single digits are preceded by a blank.
%l
is replaced by the hour (12-hour clock) as a decimal number (1-12); single digits are preceded by a blank.
%M
is replaced by the minute as a decimal number (00-59).
%m
is replaced by the month as a decimal number (01-12).
%n
is replaced by a newline.
%O*
the same as %E*.
%p
is replaced by national representation of either "ante meridiem" or "post meridiem" as appropriate.
%R
is equivalent to ``%H:%M''.
%r
is equivalent to ``%I:%M:%S %p''.
%S
is replaced by the second as a decimal number (00-60).
%s
is replaced by the number of seconds since the Epoch, UTC (see mktime(3)).
%T
is equivalent to ``%H:%M:%S''.
%t
is replaced by a tab.
%U
is replaced by the week number of the year (Sunday as the first day of the week) as a decimal number (00-53).
%u
is replaced by the weekday (Monday as the first day of the week) as a decimal number (1-7).
%V
is replaced by the week number of the year (Monday as the first day
of the week) as a decimal number (01-53).
If the week containing January 1 has four or more days in the new year, then it is week 1; otherwise it is the last week of the previous year, and the next week is week 1.
%v
is equivalent to ``%e-%b-%Y''.
%W
is replaced by the week number of the year (Monday as the first day of the week) as a decimal number (00-53).
%w
is replaced by the weekday (Sunday as the first day of the week) as a decimal number (0-6).
%X
is replaced by national representation of the time.
%x
is replaced by national representation of the date.
%Y
is replaced by the year with century as a decimal number.
%y
is replaced by the year without century as a decimal number (00-99).
%Z
is replaced by the time zone name.
%z
is replaced by the time zone offset from UTC; a leading plus sign stands for east of UTC, a minus sign for west of UTC, hours and minutes follow with two digits each and no delimiter between them (common form for RFC 822 date headers).
%+
is replaced by national representation of the date and time (the format is similar to that produced by date(1)).
%%
is replaced by `%'.
 

EXAMPLES

Some examples:
sudo justsfniffer -i eth0 > /tmp/test.log
sudo justsfniffer -i eth0 -l "%request.timestamp(%T %%D) - %request.header.host - %response.code - %response.time" > /tmp/test.log
sudo justniffer -i eth0 -c config
justniffer -f ./test.cap
 

justniffer-grab-http-traffic

The output format of the traffic can be easily customized. An example written in python is the http_parser.py It stores the transferred contents in an output directory separated by domains. That means that the transferred files like html, css, javascript, images, sounds, etc. can be saved to a directory. justniffer-grab-http-traffic is a python script that wraps justniffer and properly set the command line options in order to easily use with the http_parser.py script

Example: save all http contents into /tmp/nav directory:

$ sudo ./justniffer-grab-http-traffic -d /tmp/nav -U nobody -i eth0

Note that you MUST specify a user without root privileges, that will be user when saving files, in order to avoid to save files with root privileges that could be very dangerous.

Synopsis

Usage: justniffer-grab-http-traffic [options] Options:
--version show program's version number and exit
-h, --help show this help message and exit
-d DIRECTORY, --directory=DIRECTORY MANDATORY: directory where to save files
-p PACKET_FILTER, --packet-filter=PACKET_FILTER packet filter (tcpdump filter syntax), default ='port 80'
-U USER, --user=USER user to impersonate when saving files, cannot be the root user
-i INTERFACE, --interface=INTERFACE network interface to listen on (e.g. eth0, en1, etc.)
-f FILECAP, --filecap=FILECAP input file in 'tcpdump capture file format'
-P PARSER_SCRIPT, --parser_script=PARSER_SCRIPT parser script to execute, default is /usr/share/justniffer/scripts/http_parser.py

Example of some files captured from http traffic

Examples

Example 1 Retrieving http network traffic in access_log format

$ justniffer -i eth0
output:
192.168.2.2 - - [15/Apr/2009:17:19:57 +0200] "GET /sflogo.php?group_id=205860&type=2 HTTP/1.1" 200 0 "" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.8) Gecko/2009032711 Ubuntu/8.10 (intrepid) Firefox/3.0.8)"
192.168.2.2 - - [15/Apr/2009:17:20:18 +0200] "GET /search?q=subversion+tagging&ie=utf-8&oe=utf-8&aq=t&rls=com.ubuntu:en-US:unofficial&client=firefox-a HTTP/1.1" 200 0 "" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.8) Gecko/2009032711 Ubuntu/8.10 (intrepid) Firefox/3.0.8)"
192.168.2.2 - - [15/Apr/2009:17:20:07 +0200] "GET /sflogo.php?group_id=205860&type=2 HTTP/1.1" 200 0 "http://justniffer.sourceforge.net/" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.8) Gecko/2009032711 Ubuntu/8.10 (intrepid)Firefox/3.0.8)"
192.168.2.2 - - [15/Apr/2009:17:20:18 +0200] "GET /csi?v=3&s=web&action=&tran=undefined&ei=MvvlSdjOEciRsAbY0rGpCw&e=19592,20292&rt=prt.175,xjs.557,ol.558 HTTP/1.1" 204 0 "http://www.google.it/search?q=subversion+tagging&ie=utf-8&oe=utf-8&aq=t&rls=com.ubuntu:en-US:unofficial&client=firefox-a" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.8 Gecko/2009032711 Ubuntu/8.10 (intrepid) Firefox/3.0.8)"
192.168.2.2 - - [15/Apr/2009:17:20:07 +0200] "GET /HTTP/1.1" 200 0 "" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.8) Gecko/2009032711 Ubuntu/8.10 (intrepid) Firefox/3.0.8)"

Example 2 Like Example 1 but appending other fields

For example http response time (see man page for a complete keyword list)
$ justniffer -i eth0 -a " %response.time"
output:
192.168.2.5 - - [22/Apr/2009:22:27:36 +0200] "GET /sflogo.php?group_id=205860&type=2 HTTP/1.1" 200 0 "http://justniffer.sourceforge.net/" "Mozilla/5.0 (X11;U; Linux i686; en-US; rv:1.9.0.8) Gecko/2009032711 Ubuntu/8.10 (intrepid) Firefox/3.0.8)" 0.427993 
192.168.2.5 - - [22/Apr/2009:22:27:50 +0200] "GET /complete/search?output=firefox&client=firefox&hl=en-US&q=add+e HTTP/1.1" 200 140 "" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.8) Gecko/2009032711 Ubuntu/8.10 (intrepid) Firefox/3.0.8)"0.294897 
192.168.2.5 - - [22/Apr/2009:22:27:51 +0200] "GET /complete/search?output=firefox&client=firefox&hl=en-US&q=add+a HTTP/1.1" 200 128 "" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.8) Gecko/2009032711 Ubuntu/8.10 (intrepid) Firefox/3.0.8)"0.266929 
192.168.2.5 - - [22/Apr/2009:22:27:21 +0200] "GET /extern_js/f/CgJlbiswCjgVLCswDjgFLCswFjgJLCswFzgBLCswGDgDLCswITgWLCswJTjJiAEsKzAmOAQsKzAnOAAs/-wB3HvFrpXA.js HTTP/1.1" 304 0 "http://www.google.com/search?q=gnusticker&hl=en&safe=off&start=20&sa=N" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.8) Gecko/2009032711 Ubuntu/8.10 (intrepid) Firefox/3.0.8)" 2.025879

Example 3 Capture all tcp traffic

(add -u or -x options to encode unprintable characters):
$ justniffer -i eth0 -r
output:
GET /doc/maint-guide/ch-upload.en.html HTTP/1.1
Host: www.debian.org
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.8)
Gecko/2009032711 Ubuntu/8.10 (intrepid) Firefox/3.0.8
Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en,it;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: UTF-8,*
Keep-Alive: 300
Connection: keep-alive
Referer: http://www.debian.org/doc/maint-guide/
If-Modified-Since: Wed, 22 Apr 2009 19:36:31 GMT
If-None-Match: "400d604-3014-46829e160adc0"
Cache-Control: max-age=0

HTTP/1.1 304 Not Modified
Date: Wed, 22 Apr 2009 20:38:51 GMT
Server: Apache
Connection: Keep-Alive
Keep-Alive: timeout=15, max=100
ETag: "400d604-3014-46829e160adc0"
Expires: Thu, 23 Apr 2009 20:38:51 GMT
Cache-Control: max-age=86400

Example 4 Define a completely custom log format

$ justniffer -i eth0 -l "%request.timestamp %source.ip %dest.ip %request.header.host %request.url"
output:
06/28/11 13:30:48 192.168.2.2 72.14.221.118 i1.ytimg.com /vi/TjSk6CVN5LY/default.jpg 
06/28/11 13:30:47 192.168.2.2 72.14.221.118 i2.ytimg.com /vi/iw_nzfm1Vts/default.jpg 
06/28/11 13:30:47 192.168.2.2 216.34.181.71 static.sourceforge.net /css/phoneix/jquery.cluetip.php?secure=0 
06/28/11 13:30:48 192.168.2.2 216.34.181.71 static.sourceforge.net /sfx.js 
06/28/11 13:30:49 192.168.2.2 216.34.181.71 static.sourceforge.net /include/coremetrics/v40/eluminate.js 
06/28/11 13:30:51 192.168.2.2 199.93.61.126 c.fsdn.com /sf/images/phoneix/grad_white_dual_100.png 

Example 5 Read from a capture file

NOTE: capture file must be performed with unlimited snaplen for catching whole packets. Justniffer can work only works on pcap files with whole packets.
tcpdump command example: tcpdump -w /tmp/file.cap -s0 -i ath0
$ justniffer -f /file.cap

Example 6 Parameters for setting a more precise formatting

Many keyword has parameters for setting a more precise formatting:
$ justniffer -i eth0 -l "%request.timestamp %request.header.host %request.url %response.time"
output:
06/28/11 13:39:40 static.sourceforge.net /css/phoneix/print.css?secure=0&20080417-1657 0.162620
you can specifying timestamp formatting
$ justniffer -i eth0 -l "%request.timestamp (%B %d %Y %H:%M:%S) %request.header.host %request.url %response.time"
output:
June 28 2009 13:39:40 static.sourceforge.net /css/phoneix/print.css?secure=0&20080417-1657 0.162620
or to print the string NoHostFound if the %request.header.host cannot be valued
$ justniffer -i eth0 -l "%request.timestamp %request.header.host (NoHostFound)%request.url %response.time"
output:
06/28/11 15:10:28 www.google.com /ig?hl=en 0.116146
06/28/11 15:10:28 NoHostFound/ig?hl=en 0.116146
many keywords have their own formatting string. A generic option ( -n) can be used for setting a default "not found" string: a string that must replace the keyword if it cannot be valued. for example if request.header.[headername] is not found or if connection.time cannot be applicable. Anyway, if a "no found" string is provided as keyword parameter (for those that expect it), it will override the -n option
$ justniffer -i eth0 -l "%request.timestamp %request.header.host(NoHostFound) %request.header.host %request.url %response.time" -n N/A
output:
06/28/11 15:10:28 www.google.com www.google.com/ig?hl=en 0.116146
06/28/11 15:10:28 NoHostFound N/A/ig?hl=en 0.116146 

Example 7 Capture only http traffic

the -p option let you to specify a tcpdump compatible filter (see pcap-filter(7)): "port 80 or port 8080" capture only http traffic (usually using tcp port 80 and 8080)
$ justniffer -i eth0 -r -p "port 80 or port 8080"

Example 8 Extend with an external executable

the -e option let you to specify an external executable (usually a script) to which , for every log, the output will be redirect to. If you want to perform complex extraction operations, you can write your own script that will receive from the standard input all content specified by the -l option. A complete ad useful example is provided with justniffer-grab-http-traffic
$ justniffer -l "%response" -e ./myscript.sh -i ath0
myscript.sh
      
#!/bin/bash
# myscript.sh 
# example script (print all lines containing "href" string)

while read inputline
do 
anchors=`echo "$inputline" | grep href`
if [ "$anchors" != "" ]; then
echo $anchors;
fi;
done

Example 9 Capture smtp traffic (usually using tcp port 25)

$ justniffer -i eth0 -r -p "port 25"
output:
      
220 plecno.com ESMTP Postfix (Ubuntu)

EHLO unknown.localnet
250-plecno.com       
250-PIPELINING       
250-SIZE             
250-VRFY             
250-ETRN             
250-STARTTLS         
250-ENHANCEDSTATUSCODES
250-8BITMIME           
250 DSN                

MAIL FROM:<oreste.notelli@plecno.com> SIZE=1079
RCPT TO:<oreste.notelli@gmail.com>             
DATA                                           
250 2.1.0 Ok                                   
250 2.1.5 Ok                                   
354 End data with <CR><LF>.<CR><LF>        


From: Oreste Notelli <oreste.notelli@plecno.com>
Organization: Plecno
To: oreste.notelli@gmail.com
Subject: test
Date: Wed, 22 Apr 2009 22:46:16 +0200
User-Agent: KMail/1.11.2 (Linux/2.6.27-8-generic; KDE/4.2.2; i686;
; )
MIME-Version: 1.0
Content-Type: multipart/alternative;
boundary="Boundary-00=_ZI47J3FTNXn+25g"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
Message-Id: <200904222246.17292.oreste.notelli@plecno.com>

--Boundary-00=_ZI47J3FTNXn+25g
Content-Type: text/plain;
charset="us-ascii"
Content-Transfer-Encoding: 7bit

test

--Boundary-00=_ZI47J3FTNXn+25g
Content-Type: text/html;
charset="us-ascii"
Content-Transfer-Encoding: 7bit

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0//EN"
"http://www.w3.org/TR/REC-html40/strict.dtd">
<html>
<head>
<meta name="qrichtext" content="1" />
<style type="text/css">p, li { white-space: pre-wrap;
}</style>
</head>
<body style=" font-family:'DejaVu Sans'; font-size:8pt;
font-weight:400; font-style:normal;">
<p>
test
</p>
</body>
</html>
--Boundary-00=_ZI47J3FTNXn+25g--
.
250 2.0.0 Ok: queued as 33E7235C21A

QUIT
221 2.0.0 Bye

Example 10 Trace performances

The following keywords are used to obtain logs that give an overview on the performance of services based on HTTP protocol measuring connection time, response time, tcp connection timeouts, keep alive requests, etc. (see man for more information)
$ sudo justniffer -i eth0 -u -p "port 80 or port 8080" -l "%request.header.host %request.url %connection.time %idle.time.0  %request.time %response.time.begin %response.time.end %idle.time.1 %connection %close.originator"

Example 12 Grep keywords

The "grep keywords"can be used to capture portions of text using regular expressions. In this example we want to collect:

the url from the request header by the regular expression [^\s]*[\s]*([^\s]*)
and the content type from the response header by the regular expression Content-Type:(\s)*([^\r]*)
$ sudo justniffer -l "%request.header.grep([^\s]*[\s]*([^\s]*)) %response.header.grep(Content-Type:(\s)*([^\r]*)) %source.ip" -i eth0
output:
/ text/html 192.168.10.2
/plecno_res/src/effects.js application/javascript 192.168.10.2
/plecno_res/src/slider.js application/javascript 192.168.10.2
/plecno_res/src/effects.js application/javascript 192.168.10.2
/plecno_res/src/builder.js application/javascript 192.168.10.2
/plecno_res/src/effects.js application/javascript 192.168.10.2

Upload one ore more .cap file

You can upload your .cap files (obtained with wireshark or tcpdump) and run justniffer analysis without the need to install it

NOTE: capture file must be performed with unlimited snaplen for capturing whole packets. Justniffer can only work on pcap files with whole packets. (Example: tcpdump -w /tmp/file.cap -s0 -i eth0)

Add files...
 

Drop you files here