Justniffer

Justniffer

Justniffer is a network protocol analyzer that captures network traffic and produces logs in a customized way, can emulate Apache web server log files, track response times and extract all "intercepted" files from the HTTP traffic.

It lets you interactively trace tcp traffic from a live network or from a previously saved capture file. Justniffer's native capture file format is libpcap format, which is also the format used by tcpdump and various other tools.

The main Justniffer's feature is the ability to handle all those complex low level protocol issues and retrieve the correct flow of the TCP/IP traffic: IP fragmentation, TCP retransmission, reordering. etc. It uses portions of Linux kernel source code for handling all TCP/IP stuff. Precisely, it uses a slightly modified version of the libnids libraries that already include a modified version of Linux code in a more reusable way.

Justniffer was born as tool for helping in analyzing performance problem in complex network environment when it becomes impractical to analyze network captures solely using wireshark. It will help you to quickly identify the most significant bottlenecks analyzing the performance at "application" protocol level.

In very complex and distributed systems is often useful to understand how communication takes place between different components, and when this is implemented as a network protocol based on TCP/IP (HTTP, JDBC, RTSP, SIP, SMTP, IMAP, POP, LDAP, REST, XML-RPC, IIOP, SOAP, etc.), justniffer becomes very useful. Often the logging level and monitoring systems of these systems does not report important information to determine performance issues such as the response time of each network request. Because they are in a "production" environment and cannot be too much verbose or they are in-house developed applications and do not provide such logging.

Other times it is desirable to collect access logs from web services implemented on different environments (various web servers, application servers, python web frameworks, etc.) or web services that are not accessible and therefore traceable only on client side.

Justniffer can capture traffic in promiscuous mode so it can be installed on dedicated and independent station within the same network "collision domain" of the gateway of the systems that must be analyzed, collecting all traffic without affecting the system performances and requiring invasive installation of new software in production environments.

Thanks to the robust implementation for the reconstruction of the TCP flow, justniffer can also be used to retrieve files sent over the network.

A python script has been developed to recover all files sent via HTTP (images, text, html, javascript, etc.).

Justniffer features summary

• TCP flow rebuild very reliable: it can reorder, reassemble tcp segments and ip fragments using portions of the Linux kernel code
• Logging text mode can be customized
• Extensibility by any executable, such as bash, python, perl scripts, ELF executable, etc.
• Performance measurement it can collect many information on performances: connection time, close time, request time , response time, close time, etc.

Install

On Ubuntu ( 11.04 / 11.10 / 12.04)

$ sudo add-apt-repository ppa:oreste-notelli/ppa 
$ sudo apt-get update
$ sudo apt-get install justniffer

Other Ubuntus

Download the .deb file from the Ubuntu file repository and install it:

$ sudo apt-get install gdebi-core 
$ sudo gdebi justniffer-x.x.deb

Other Distributions

be sure you have installed third-party tools and libraries:

patch
tar
autotools
make
libc6
libpcap0.8
g++                         
gcc 
libboost-iostreams          
libboost-program-options
libboost-regex  

unpacked the source package, type:

$ ./configure 
$ make 
$ make install

JUSTNIFFER

NAME

SYNOPSIS

justniffer [ [-i interface] or [-f <tcpdump file>] ] [-F]
[-p <packet filter>] [-u or -x] 
[ [-r] or [-l <log format>] or [-a <log format>]  ] 
[-c <config file>]  [-e <external program>]  [-U <user> ]   
[-n <not-found> ]  [-s <max concurrent tcp streams> ]  
[-d <max concurrent IP fragments> ]

Examples: 
Logging network traffic in Apache like format:
  justniffer -i eth0 

Tracing tcpdump dump in Apache like format:
  justniffer -f file.cap 

Tracing TCP streams from the network:
  justniffer -i eth0 -r 

Logging network traffic in Apache like format appending response times:
  justniffer -i eth0 -a " %response.time" 

Logging network traffic in a customized format:
  justniffer -i eth0 -l "%source.ip%tab%dest.ip%tab%request.header.host%tab%response.time" 

DESCRIPTION

TRACKING PERFORMANCES

 The main feature of justniffer is to analize network traffic to monitor performances. The performances related keywords are:
  %connection.time
  %idle.time.0
  %request.time
  %response.time
    %response.time.begin
    %response.time.end
  %idle.time.1

 +---------+                           +---------+

 |         |                           |         |

 |  Client |                           | Server  |

 |         |                           |         |

 +---------+                           +---------+

      |                                     |

      |  -----   connect syn   -------->    |----+

      |                                     |    |

      |  <------   syn/ack    --------->    |    | %connection.time

      |                                     |    |

      |  -------     ack     ---------->    |    |

      |           ESTABLISHED               |----+

      |                                     |    | %idle.time.0 

      |                                     |    |(after connection, before 

      |                                     |    | request)

      |                                     |    |

      |  ---  request/first packet  --->    |----+

      |  <------     ack     -----------    |    |

      |                                     |    |

      |  ---  request/....          --->    |    | %request.time

      |  <------     ack     -----------    |    |

      |                                     |    |

      |  ---   request/last packet  --->    |    |

      |  <------     ack     -----------    |----+--------------------+

      |                                     |    |                    |

      |                                     |    |                    |

      |                                     |    |%response.time.begin |   

      |                                     |    |                    |

      |  <--  response/first packet ----    |----+                    | response 

      |  -------     ack     ---------->    |    |                    | time

      |                                     |    |                    |

      |  <--  response/....         ----    |    |%response.time.end   |

      |  -------     ack     ---------->    |    |                    |

      |                                     |    |                    |

      |  <--  response/last packet  ----    |    |                    |

      |  -------     ack     ---------->    |----+--------------------+

      |                                     |    |

      |                                     |    |

      |                                     |    | %idle.time.1 (after response, 

      |                                     |    | before new request or close)

      |                                     |    |

      |  <------   close      --------->    |----+

      |                                     |    |

      |                                     |    |

OPTIONS

Example: justniffer -i eth0

-f or --filecap=<file>

tcpdump file to read from (for offline network traffic processing). It must be a pcap file (produced by network capture programs such as tcpdump or wireshark) WARNING: justniffer needs a complete dump, usually, sniffers collect just the few first (96) bytes per packet. (when using tcpdump you must specify "-s 0" option. Example: tcpdump -i eth0 -s 0 -w /tmp/file.cap)

Example: justniffer -f /tmp/file.cap

-F or --force-read-pcap

Force the reading of a pcap file also if captured with truncated packets (for example when using tcpdump without the "-s 0" option)

-p or --packet-filter=<filter>

packet filter (pcap filter syntax) (see pcap-filter(7))

Examples:

  sniffing proxy traffic
    justniffer -i eth0 -p "tcp port 8080"

  sniffing row telnet traffic
    justniffer -i eth0 -r -p "tcp port 23"

  sniffing row pop3 traffic
    justniffer -i eth0 -r -p "tcp port 110"

  sniffing  traffic from/to a specific host
    justniffer -i eth0 -r -p "host 10.10.10.2"

  sniffing  HTTP traffic from/to a specific host and port
    justniffer -i eth0 -r -p "host 10.10.10.2 and tcp port 80"

-l or --log-format=<format>

log format. You can specify the output string format containing reserved keyword that will be replaced with the proper value (see FORMAT KEYWORDS). If missing, the CLF (Common Log Format, used by almost all web servers) is used as default.

Example: justniffer -i eth0 -l "\"%request.line\"%tab%response.time"

    "POST /v2/rss/alerts?src=ffbmext2.1.034 HTTP/1.1" 0.139011
    "POST /v2/rss/network/oreste.notelli?src=ffbmext2.1.034 HTTP/1.1" 0.623382
    "GET /man_page_howto.html HTTP/1.1"       0.024437
    "GET /ig?hl=en HTTP/1.1"  0.764945
    "GET /?hl=en&tab=wv HTTP/1.1"   0.242342
    "GET /s/-yCdCsgUnsI/css/homepage_c.css HTTP/1.1"        0.071942
    "GET /vi/YUvWcegtqik/default.jpg HTTP/1.1"      0.821472

-r or --raw

show raw stream. it is a shortcut for -l %request%response

Example: justniffer -i eth0 -r

-s or --max-tcp-streams

max concurrent TCP stream. (default= 65536) excess will be discarded

-d or --max-fragmented-ip

max concurrent fragmented IP. (default= 65536) excess will be discarded

-u or --unprintable

encode as dots (.) unprintable characters ( for more control on character you should use pipelining to cat -v.

Example:

  justniffer -i eth0 -l "%request%newline%response"| cat -v
  justniffer -i eth0 -ru

-x or --hex-encode

encode unprintable characters in [<char hex code>] format

Example:

  justniffer -i eth0 -rx

-n or --not-found=<not found string>

Not found string. It is used to replace a specified keyword when it cannot be valued because it is not found. All request.header.* and response.header.* keywords can override the "not found string" passing it as parameter. For example:
 %request.header.host() will be replaced by the Host header value, or an empty string if Host header not found
 %request.header.host(UNKNOWN) will be replaced by the Host header 
value, or the UNKNOWN string if Host header not found
 %request.header.host will be replaced by the Host header value, or , if Host header not found, with 
the string specified by the -n option 

Examples:

  justniffer -i eth0 -l "%request.header.connection" -n N/A  

  will produce such logs:
    N/A
    N/A
    keep-alive
    close

Note: each keyword can override the "not found string" value:

  justniffer -i eth0 -l "%request.header.connection(None) %request.header.connection(-) %request.header.connection() %request.header.connection" -n N/A  

  will produce such logs:
    None -  N/A
    None -  N/A
    keep-alive keep-alive keep-alive keep-alive
    close close close close

-e or --execute=<external program>

call the specified external program/shell script pipelining the standard output for each request/response phase You can write shell script for handling, for example, HTTP traffic

Example:

  justniffer -i eth0 -l "%request%newline%response" -e "tail -2 "

-U or --user=<user>

User to impersonate when executing the program specified with the -e option, used to avoid to security exploits when running justniffer with root privileges

Example:

  justniffer -i eth0 -l "%request%newline%response" -e "grep password >> /tmp/passwords.txt"  -U guest

-c or --config=<config file>

configuration file. You can specify options in a configuration file (command line options override file configuration options) using the following format specifications:

   <option> = <value>

   configuration file example:

log-format = "%request.url %request.header.host %response.code %%response.time"

packet-filter = "tcp port 80 or tcp port 8080 or tcp port 3526"

FORMAT KEYWORDS

%close.time([not applicable string])

elapsed time from last response and when the connection is closed. the "not applicable" string is replaced in case the keyword value cannot be applicable. if not provided the -n value or the default value "-" is used

%close.timestamp([format])

is replaced by the close timestamp. You can use optional format specification (see TIMESTAMP FORMAT)

%close.timestamp2([not applicable string])

is replaced by the connection timestamp using format "seconds.microseconds" the "not applicable" string is replaced in case the keyword value cannot be applicable. if not provided the -n value or the default value "-" is used

%connection

 connection persistence indicator:
    unique: the request/response is the unique in the tcp connection
    start: the request/response is the first in the tcp connection
    last : the request/response is the last in the tcp connection
    continue : the request/response is the middle in the tcp connection

%connection.time([not applicable string])

elapsed time for establishing the TCP connection. The "not applicable" string is replaced in case the keyword value cannot be applicable. if not provided the -n value or the default value "-" is used

%connection.timestamp([format])

is replaced by the connection timestamp. You can use optional format specification (see TIMESTAMP FORMAT)

%connection.timestamp2([not applicable string])

is replaced by the connection timestamp using format "seconds.microseconds" the "not applicable" string is replaced in case the keyword value cannot be applicable. if not provided the -n value or the default value "-" is used

%idle.time.0

elapsed time form when the connection is established and the request is started the "not applicable" string is replaced in case the keyword value cannot be applicable. if not provided the -n value or the default value "-" is used

%idle.time.1

elapsed time form when the last response and the next request (or the connection closing) the "not applicable" string is replaced in case the keyword value cannot be applicable. if not provided the -n value or the default value "-" is used

%response.time.begin

elapsed time form when the request and the response start the "not applicable" string is replaced in case the keyword value cannot be applicable. if not provided the -n value or the default value "-" is used

%response.time.end

elapsed time form the response start and the response end the "not applicable" string is replaced in case the keyword value cannot be applicable. if not provided the -n value or the default value "-" is used

%response.timestamp([format])

elapsed time for the whole response the "not applicable" string is replaced in case the keyword value cannot be applicable. if not provided the -n value or the default value "-" is used

%dest.ip

is replaced by the destination IP address

%dest.port

is replaced by the destination TCP port

%source.ip

is replaced by the source IP address

%source.port

is replaced by the source TCP port

%request

is replaced by the the whole request ( (it is multiline and may contain unprintable characters)

%request.timestamp([format])

is replaced by the request timestamp. You can use optional format specification (see TIMESTAMP FORMAT)

%request.timestamp2([not applicable string])

is replaced by the request timestamp using format "seconds.microseconds" the "not applicable" string is replaced in case the keyword value cannot be applicable. if not provided the -n value or the default value "-" is used

%request.size

is replaced by the request size (including request header size)

%request.line

is replaced by the request line (e.g. "GET /index.html HTTP/1.1")

%request.method

is replaced by the request method (e.g. GET, POST, HEAD)

%request.url

is replaced by the url

%request.protocol

is replaced by the protocol (e.g. HTTP/1.0, HTTP/1.1)

%request.grep(<regular-expression>)

is replaced by the result of the specified regular expression applied on the whole request [Perl regular expression syntax, see perlre(1) or perl(1)]. The most nested subgroup is returned

%request.header

is replaced by the request header (it is multiline)

%request.header.host([not found string])

is replaced by the request Host header value. The optional "not found" string is replaced in case the keyword value was not found. if not provided the -n value or the default value "-" is used

%request.header.user-agent([not found string])

is replaced by the request User-Agent header value. The optional "not found" string is replaced in case the keyword value was not found. if not provided the -n value or the default value "-" is used

%request.header.accept([not found string])

is replaced by the request Accept header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used

%request.header.accept-language([not found string])

is replaced by the request Accept-Language header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used

%request.header.accept-charset([not found string])

is replaced by the request Accept-Charset header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used

%request.header.accept-encoding([not found string])

is replaced by the request Accept-Encoding header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used

%request.header.authorization([not found string])

is replaced by the request Authorization header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used

%request.header.connection([not found string])

is replaced by the request Connection header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used

%request.header.content-encoding([not found string])

is replaced by the request Content-Encoding header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used

%request.header.content-length([not found string])

is replaced by the request Content-Length header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used

%request.header.content-md5([not found string])

is replaced by the request Content-MD5 header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used

%request.header.cookie([not found string])

is replaced by the request Cookie header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used

%request.header.range([not found string])

is replaced by the request Range header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used

%request.header.referer([not found string])

is replaced by the request Referer header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used

%request.header.keep-alive([not found string])

is replaced by the request Keep-Alive header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used

%request.header.value(<header-name>

is replaced by the request header value (e.g. "%request.header.value(Cookie)") The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used

%request.header.transfer-encoding([not found string])

is replaced by the request Transfer-Encoding header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used

%request.header.via([not found string])

is replaced by the request Via header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used

%request.header.grep(<regular-expression>)

is replaced by the result of the specified regular expression applied on the request header [Perl regular expression syntax, see perlre(1) or perl(1)]. The most nested subgroup is returned (e.g. to obtain the request URL: "%request.header.grep(^[^\s]*\s*([^\s]*))"

%response

is replaced by the while response (it is multiline and may contain unprintable characters)

%response.timestamp([format])

is replaced by the response timestamp. You can use optional format specification (see TIMESTAMP FORMAT)

%request.timestamp2([not applicable string]))

is replaced by the response timestamp using format "seconds.microseconds" the "not applicable" string is replaced in case the keyword value cannot be applicable. if not provided the -n value or the default value "-" is used

%response.size

is replaced by the response size (including response the header size)

%response.time

is replaced by the response time (difference from the request time and the time the response finish)

%response.line

is replaced by the response line

%response.protocol

is replaced by the response protocol

%response.code

is replaced by the response code (e.g. 200, 404, 500, etc.)

%response.message

is replaced by response message (e.g. OK, Not Found, Internal Server Error, etc.)

%response.grep(<regular-expression>)

is replaced by the result of the specified regular expression applied on the whole response [Perl regular expression syntax, see perlre(1) or perl(1)]. The most nested subgroup is returned

%response.header

is replaced by the response header (it is multiline)

%request.header.allow([not found string])

is replaced by the request Allow header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used

%response.header.server([not found string])

is replaced by the response Server header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used

%response.header.date([not found string])

is replaced by the response Date header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used

%response.header.content-length([not found string])

is replaced by the response Content-Length header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used

%response.header.content-type([not found string])

is replaced by the response Content-Type header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used

%request.header.content-md5([not found string])

is replaced by the request Content-MD5 header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used

%request.header.content-range([not found string])

is replaced by the request Content-Range header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used

%response.header.content-encoding([not found string])

is replaced by the response Content-Encoding header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used

%response.header.content-language([not found string])

is replaced by the response Content-Language header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used

%response.header.transfer-encoding([not found string])

is replaced by the response Transfer-Encoding header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used

%%response.header.expires is replaced by the response Expires header value

%response.header.etag([not found string])

is replaced by the response ETag header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used

%response.header.cache-control([not found string])

is replaced by the response Cache-Control header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used

%response.header.last-modified([not found string])

is replaced by the response Last-Modified header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used

%response.header.pragma([not found string])

is replaced by the response Pragma header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used

%response.header.age([not found string])

is replaced by the response Age header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used

%response.header.connection([not found string])

is replaced by the response Connection header value. The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used

%response.header.keep-alive([not found string])

is replaced by the response Keep-Alive header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used

%response.header.via([not found string])

is replaced by the response Via header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used

%response.header.vary([not found string])

is replaced by the response Vary header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used

%request.header.www-authenticate([not found string])

is replaced by the request WWW-Authenticate header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used

%response.header.accept-ranges([not found string])

is replaced by the response Accept-Ranges header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used

%response.header.set-cookie([not found string])

is replaced by the response Set-Cookie header value The optional "not found" string is replaced in case the keyword value was not found. If not provided the -n value or the default value "-" is used

%request.header.value(<header-name>)

%%response.header.value(<header-name>) is replaced by the response header value (e.g. "%request.header.value(Set-Cookie)")

%response.header.grep(<regular-expression>)

is replaced by the result of the specified regular expression applied on the response header [Perl regular expression syntax, see perlre(1) or perl(1)]. The most nested subgroup is returned (e.g. to obtain the request URL: "%request.header.grep(^[^\s]*\s*([^\s]*))"

%tab

is replaced by a tab

%-

break (used for breaking keywords). For example, if you want to obtain output like this:

"0.234342                  seconds"

you must use the break keyword (%-) to mark the %tab keyword end:

 "%response.time%tab%-seconds" 

%%

is replaced by the '%' character

%newline

is replaced by a newline

TIMESTAMP FORMAT

%A

is replaced by national representation of the full weekday name.

%a

is replaced by national representation of the abbreviated weekday name.

%B

is replaced by national representation of the full month name.

%b

is replaced by national representation of the abbreviated month name.

%C

is replaced by (year / 100) as decimal number; single digits are preceded by a zero.

%c

is replaced by national representation of time and date.

%D

is equivalent to ``%m/%d/%y''.

%d

is replaced by the day of the month as a decimal number (01-31).

%E* %O*

POSIX locale extensions. The sequences %Ec %EC %Ex %EX %Ey %EY %%Od %Oe %OH %OI %Om %OM %OS %Ou %OU %OV %Ow %OW %Oy are supposed to provide alternate representations.

Additionally %OB implemented to represent alternative months names (used standalone, without day mentioned).

%e

is replaced by the day of month as a decimal number (1-31); single digits are preceded by a blank.

%F

is equivalent to ``%Y-%m-%d''.

%G

is replaced by a year as a decimal number with century. This year is the one that contains the greater part of the week (Monday as the first day of the week).

%g

is replaced by the same year as in ``%G'', but as a decimal number without century (00-99).

%H

is replaced by the hour (24-hour clock) as a decimal number (00-23).

%h

the same as %b.

%I

is replaced by the hour (12-hour clock) as a decimal number (01-12).

%j

is replaced by the day of the year as a decimal number (001-366).

%k

is replaced by the hour (24-hour clock) as a decimal number (0-23); single digits are preceded by a blank.

%l

is replaced by the hour (12-hour clock) as a decimal number (1-12); single digits are preceded by a blank.

%M

is replaced by the minute as a decimal number (00-59).

%m

is replaced by the month as a decimal number (01-12).

%n

is replaced by a newline.

%O*

the same as %E*.

%p

is replaced by national representation of either "ante meridiem" or "post meridiem" as appropriate.

%R

is equivalent to ``%H:%M''.

%r

is equivalent to ``%I:%M:%S %p''.

%S

is replaced by the second as a decimal number (00-60).

%s

is replaced by the number of seconds since the Epoch, UTC (see mktime(3)).

%T

is equivalent to ``%H:%M:%S''.

%t

is replaced by a tab.

%U

is replaced by the week number of the year (Sunday as the first day of the week) as a decimal number (00-53).

%u

is replaced by the weekday (Monday as the first day of the week) as a decimal number (1-7).

%V

is replaced by the week number of the year (Monday as the first day

of the week) as a decimal number (01-53).

If the week containing January 1 has four or more days in the new year, then it is week 1; otherwise it is the last week of the previous year, and the next week is week 1.

%v

is equivalent to ``%e-%b-%Y''.

%W

is replaced by the week number of the year (Monday as the first day of the week) as a decimal number (00-53).

%w

is replaced by the weekday (Sunday as the first day of the week) as a decimal number (0-6).

%X

is replaced by national representation of the time.

%x

is replaced by national representation of the date.

%Y

is replaced by the year with century as a decimal number.

%y

is replaced by the year without century as a decimal number (00-99).

%Z

is replaced by the time zone name.

%z

is replaced by the time zone offset from UTC; a leading plus sign stands for east of UTC, a minus sign for west of UTC, hours and minutes follow with two digits each and no delimiter between them (common form for RFC 822 date headers).

%+

is replaced by national representation of the date and time (the format is similar to that produced by date(1)).

%%

is replaced by `%'.

EXAMPLES

sudo justsfniffer -i eth0 > /tmp/test.log

sudo justsfniffer -i eth0 -l "%request.timestamp(%T %%D) - %request.header.host - %response.code - %response.time" > /tmp/test.log

sudo justniffer -i eth0 -c config

justniffer -f ./test.cap

justniffer-grab-http-traffic

The output format of the traffic can be easily customized. An example written in python is the http_parser.py It stores the transferred contents in an output directory separated by domains. This means that the transferred files like html, css, javascript, images, sounds, etc. can be saved to a directory. justniffer-grab-http-traffic is a python script that wraps justniffer and properly set the command line options in order to easily use with the http_parser.py script

Example: save all http contents into /tmp/nav directory:

Note that you MUST specify a user without root privileges, that will be user when saving files, in order to avoid to save files with root privileges that could be very dangerous.

Synopsis

Example of some files captured from http traffic

Examples

192.168.2.2 - - [15/Apr/2009:17:19:57 +0200] "GET /sflogo.php?group_id=205860&type=2 HTTP/1.1" 200 0 "" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.8) Gecko/2009032711 Ubuntu/8.10 (intrepid) Firefox/3.0.8)"

192.168.2.2 - - [15/Apr/2009:17:20:18 +0200] "GET /search?q=subversion+tagging&ie=utf-8&oe=utf-8&aq=t&rls=com.ubuntu:en-US:unofficial&client=firefox-a HTTP/1.1" 200 0 "" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.8) Gecko/2009032711 Ubuntu/8.10 (intrepid) Firefox/3.0.8)"

192.168.2.2 - - [15/Apr/2009:17:20:07 +0200] "GET /sflogo.php?group_id=205860&type=2 HTTP/1.1" 200 0 "http://justniffer.sourceforge.net/" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.8) Gecko/2009032711 Ubuntu/8.10 (intrepid)Firefox/3.0.8)"

192.168.2.2 - - [15/Apr/2009:17:20:18 +0200] "GET /csi?v=3&s=web&action=&tran=undefined&ei=MvvlSdjOEciRsAbY0rGpCw&e=19592,20292&rt=prt.175,xjs.557,ol.558 HTTP/1.1" 204 0 "http://www.google.it/search?q=subversion+tagging&ie=utf-8&oe=utf-8&aq=t&rls=com.ubuntu:en-US:unofficial&client=firefox-a" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.8 Gecko/2009032711 Ubuntu/8.10 (intrepid) Firefox/3.0.8)"

192.168.2.2 - - [15/Apr/2009:17:20:07 +0200] "GET /HTTP/1.1" 200 0 "" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.8) Gecko/2009032711 Ubuntu/8.10 (intrepid) Firefox/3.0.8)"

192.168.2.5 - - [22/Apr/2009:22:27:36 +0200] "GET /sflogo.php?group_id=205860&type=2 HTTP/1.1" 200 0 "http://justniffer.sourceforge.net/" "Mozilla/5.0 (X11;U; Linux i686; en-US; rv:1.9.0.8) Gecko/2009032711 Ubuntu/8.10 (intrepid) Firefox/3.0.8)" 0.427993 

192.168.2.5 - - [22/Apr/2009:22:27:50 +0200] "GET /complete/search?output=firefox&client=firefox&hl=en-US&q=add+e HTTP/1.1" 200 140 "" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.8) Gecko/2009032711 Ubuntu/8.10 (intrepid) Firefox/3.0.8)"0.294897 

192.168.2.5 - - [22/Apr/2009:22:27:51 +0200] "GET /complete/search?output=firefox&client=firefox&hl=en-US&q=add+a HTTP/1.1" 200 128 "" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.8) Gecko/2009032711 Ubuntu/8.10 (intrepid) Firefox/3.0.8)"0.266929 

192.168.2.5 - - [22/Apr/2009:22:27:21 +0200] "GET /extern_js/f/CgJlbiswCjgVLCswDjgFLCswFjgJLCswFzgBLCswGDgDLCswITgWLCswJTjJiAEsKzAmOAQsKzAnOAAs/-wB3HvFrpXA.js HTTP/1.1" 304 0 "http://www.google.com/search?q=gnusticker&hl=en&safe=off&start=20&sa=N" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.8) Gecko/2009032711 Ubuntu/8.10 (intrepid) Firefox/3.0.8)" 2.025879

GET /doc/maint-guide/ch-upload.en.html HTTP/1.1
Host: www.debian.org
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.8)
Gecko/2009032711 Ubuntu/8.10 (intrepid) Firefox/3.0.8
Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en,it;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: UTF-8,*
Keep-Alive: 300
Connection: keep-alive
Referer: http://www.debian.org/doc/maint-guide/
If-Modified-Since: Wed, 22 Apr 2009 19:36:31 GMT
If-None-Match: "400d604-3014-46829e160adc0"
Cache-Control: max-age=0

HTTP/1.1 304 Not Modified
Date: Wed, 22 Apr 2009 20:38:51 GMT
Server: Apache
Connection: Keep-Alive
Keep-Alive: timeout=15, max=100
ETag: "400d604-3014-46829e160adc0"
Expires: Thu, 23 Apr 2009 20:38:51 GMT
Cache-Control: max-age=86400

06/28/11 13:30:48 192.168.2.2 72.14.221.118 i1.ytimg.com /vi/TjSk6CVN5LY/default.jpg 

06/28/11 13:30:47 192.168.2.2 72.14.221.118 i2.ytimg.com /vi/iw_nzfm1Vts/default.jpg 

06/28/11 13:30:47 192.168.2.2 216.34.181.71 static.sourceforge.net /css/phoneix/jquery.cluetip.php?secure=0 

06/28/11 13:30:48 192.168.2.2 216.34.181.71 static.sourceforge.net /sfx.js 

06/28/11 13:30:49 192.168.2.2 216.34.181.71 static.sourceforge.net /include/coremetrics/v40/eluminate.js 

06/28/11 13:30:51 192.168.2.2 199.93.61.126 c.fsdn.com /sf/images/phoneix/grad_white_dual_100.png 

06/28/11 13:39:40 static.sourceforge.net /css/phoneix/print.css?secure=0&20080417-1657 0.162620

June 28 2009 13:39:40 static.sourceforge.net /css/phoneix/print.css?secure=0&20080417-1657 0.162620

06/28/11 15:10:28 www.google.com /ig?hl=en 0.116146

06/28/11 15:10:28 NoHostFound/ig?hl=en 0.116146

06/28/11 15:10:28 www.google.com www.google.com/ig?hl=en 0.116146

06/28/11 15:10:28 NoHostFound N/A/ig?hl=en 0.116146 

      

        
#!/bin/bash
# myscript.sh 
# example script (print all lines containing "href" string)

while read inputline
do 
anchors=`echo "$inputline" | grep href`
if [ "$anchors" != "" ]; then
echo $anchors;
fi;
done

      
    

      

        
220 plecno.com ESMTP Postfix (Ubuntu)

EHLO unknown.localnet
250-plecno.com       
250-PIPELINING       
250-SIZE             
250-VRFY             
250-ETRN             
250-STARTTLS         
250-ENHANCEDSTATUSCODES
250-8BITMIME           
250 DSN                

MAIL FROM:<oreste.notelli@plecno.com> SIZE=1079
RCPT TO:<oreste.notelli@gmail.com>             
DATA                                           
250 2.1.0 Ok                                   
250 2.1.5 Ok                                   
354 End data with <CR><LF>.<CR><LF>        


From: Oreste Notelli <oreste.notelli@plecno.com>
Organization: Plecno
To: oreste.notelli@gmail.com
Subject: test
Date: Wed, 22 Apr 2009 22:46:16 +0200
User-Agent: KMail/1.11.2 (Linux/2.6.27-8-generic; KDE/4.2.2; i686;
; )
MIME-Version: 1.0
Content-Type: multipart/alternative;
boundary="Boundary-00=_ZI47J3FTNXn+25g"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
Message-Id: <200904222246.17292.oreste.notelli@plecno.com>

--Boundary-00=_ZI47J3FTNXn+25g
Content-Type: text/plain;
charset="us-ascii"
Content-Transfer-Encoding: 7bit

test

--Boundary-00=_ZI47J3FTNXn+25g
Content-Type: text/html;
charset="us-ascii"
Content-Transfer-Encoding: 7bit

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0//EN"
"http://www.w3.org/TR/REC-html40/strict.dtd">
<html>
<head>
<meta name="qrichtext" content="1" />
<style type="text/css">p, li { white-space: pre-wrap;
}</style>
</head>
<body style=" font-family:'DejaVu Sans'; font-size:8pt;
font-weight:400; font-style:normal;">
<p>
test
</p>
</body>
</html>
--Boundary-00=_ZI47J3FTNXn+25g--
.
250 2.0.0 Ok: queued as 33E7235C21A

QUIT
221 2.0.0 Bye

      
    

$ sudo justniffer -i eth0 -u -p "port 80 or port 8080" -l "%request.header.host %request.url %connection.time %idle.time.0  %request.time %response.time.begin %response.time.end %idle.time.1 %connection %close.originator"

$ sudo justniffer -l "%request.header.grep([^\s]*[\s]*([^\s]*)) %response.header.grep(Content-Type:(\s)*([^\r]*)) %source.ip" -i eth0

/ text/html 192.168.10.2

/plecno_res/src/effects.js application/javascript 192.168.10.2

/plecno_res/src/slider.js application/javascript 192.168.10.2

/plecno_res/src/effects.js application/javascript 192.168.10.2

/plecno_res/src/builder.js application/javascript 192.168.10.2

/plecno_res/src/effects.js application/javascript 192.168.10.2

Upload one ore more .cap file

You can upload your .cap files (obtained with wireshark or tcpdump) and run justniffer analysis without the need to install it

NOTE: capture file must be performed with unlimited snaplen for capturing whole packets. Justniffer can only work on pcap files with whole packets. (Example: tcpdump -w /tmp/file.cap -s0 -i eth0)