Output File Format
NetSniff parses captured traffic at multiple layers and outputs any logged
information to a series of log files. On this page we list all the protocols
that are parsed and formally describe the format of the subsequent logged
output and the filenames to which this information is written.
Applications which make use of a stream protocol such as TCP are handled a
little differently. NetSniff reconstructs the TCP data bitstream in the
TCPStream class (which also calculates TCP level statistics) and passes
the regenerated data stream to an Application Layer Parser for further
processing.
Structure
NetSniff is programmed in C++ using an Object-Oriented
Methodology. Due to its design, output of logged information to log files
is performed in layers. While a particular protocol parser may not output to a
specific file, it does perform output the same information - formatted in the
same way - for all higher-layered protocols contained within the packet under
consideration.
What this means in practice is that (for example) the IP Packet parser will
output source and destination IP address information in a particular format.
A UDP Packet Parser will output formatted IP Packet information followed by
UDP Packet information such as port numbers. A DNS Packet Parser will output
all IP and UDP Packet information followed by the logged contents of the
DNS Query. This keeps both the formatting of the logged output consistent,
and ensures that addition of new Protocol parsers can leverage existing
logging code.
A similar approach is used with Stream Type protocols. Each Application
bitstream parser will output the TCP level statistics in a common format
along with extra logged information about the application being parsed.
Packet Level Protocols
This section details the output format of any parsed packet level protocols.
All captured packets are logged as a single entry in the corresponding log
output file. When output is logged, each entry contains the output of all
encapsulating packets from the lowest level packet encountered.
Base Level Information
Output Specification: timestamp Headers(n) Payload(n)
- timestamp - The timestamp of the captured packet
in the format "YYYY-MM-DD hh:mm:ss.xxxxxx"
- Headers(n) - The amount of overhead in the packet
data transfer (n is equal to the number of bytes in all
protocol layer packet headers - e.g. Ethernet header + IP Header +
UDP Header)
- Payload(n) - The actual payload size, where n
is equal to the total captured packet size minus the header/overhead
size.
Ethernet Header Information
Output Specification: No output of Ethernet Level Information
ARP Protocol
Note: At present only IPv4 ARP resolution packets are logged.
Output File: arp.log
Output Specification: hrd pro hln pln op sha spa tha tpa
- hrd - The hardware type field of the ARP Packet.
- pro - The protocol type field of the ARP Packet.
- hln - The length in bytes of a hardware address.
- pln - The length in bytes of a protocol address.
- op - The ARP operation of the packet, as defined by
RFC 826.
- sha - The sender hardware address. If anonymisation
is enabled, this field is anonymised.
- spa - The sender protocol address. If anonymisation
is enabled, this field (an IP address) is anonymised.
- tha - The target hardware address. If anonymisation
is enabled, this field is anonymised.
- tpa - The target protocol address. If anonymisation
is enabled, this field (an IP address) is anonymised.
IP Protocol
Output Specification: src_ip src_port dst_ip dst_port hops
- src_ip - The IP address of the source of the IP
datagram. This value is anonymised if anonymisation is enabled in
Netsniff.
- src_port - The source port number is extracted from
the encapsulated Transport Layer Protocol parser and output here.
This form of output is more consistent with human readable IP/Port
number pairs.
- dst_ip - The IP address of the destination of the IP
datagram. This value is anonymised if specified.
- dst_port - The destination port number is extracted from
the encapsulated Transport Layer Protocol parser and output here.
- hops - The estimated number of hops visited by this
packet prior to capture.
ICMP Protocol
Output File: icmp.log
Output Specification: [type (code)]
- type - The ICMP type is extracted from the ICMP packet
and output as a numerical output.
- code - This value is only output IF the ICMP type is
unreachable. Again this value is output numerically as a value
extracted from the ICMP packet.
UDP Protocol
Output Specification: No output of UDP Level Information
- Note that source and destination port numbers are retrieved from this
layer for output by the IP Protocol parser.
DNS Protocol
Output File: dns.log
Output Specification: dnstype(id) result query_list rr_list
- dnstype - The type of DNS packet (either Query or
Response).
- id - The DNS Identification number. This allows us
to match corresponding DNS Queries and Responses.
- result - The DNS result code (can be one of the string
values NO_ERROR, FORMAT_ERROR, SERVER_FAILURE,
NAME_ERROR, NOT_IMPLEMENTED or UNKNOWN code, where
Code is the unknown numerical result code.
- query_list - A list (possibly consisting of zero entries)
of all DNS queries in the DNS packet. Each list entry is separated
by a <space>i; and is formatted as Q(desc - type/class)
where:
- desc - The actual query value, could be a URL or an IP
address. This value is anonymised if necessary.
- type - Possible values are A for a IP address
lookup request, PTR for a reverse lookup request, and
the numerical type value for other request types.
- class - Possible values are IN for a IPv4 request,
IN6 for an IPv6 request, and the numerical type value
for other request types.
- rr_list - A list (possibly consisting of zero entries)
of all DNS responses in the DNS packet. Each list entry is separated
by a <space> and is formatted as R(desc - TTL val
seconds) where:
- desc - The actual query response, could be a URL or an IP
address. This value is anonymised if necessary.
- val - The TTL of the DNS response in seconds.
Stream Based Protocols
This section details the output format of any parsed stream-based application
protocols. All application streams are logged as a single entry in the
corresponding log output file. As entries in these files can consist of
multiple lines, each entry is separated by a blank line in the corresponding
log file. When output is logged, each entry contains the output of all
encapsulating protocols beginning from the Stream Definition protocol -
currently TCP is the only Stream Definition Protocol supported.
TCP Stream
Output Specification: Each output entry is over five lines
timestamp duration src_ip src_port dst_ip dst_port c2sdata s2cdata c2sohead s2cohead hop rtt jit c2splr s2cplr
Hops Client:,hcmean,hc_histo_list
Hops Server:,hsmean,hs_histo_list
RTT:,rtt_mean,rtt_list
Jitter:,jit_mean, jit_list
- timestamp - The timestamp when the first packet of
the TCP Stream was captured in the format "YYYY-MM-DD hh:mm:ss.xxxxxx"
- duration - The estimated duration of the TCP stream
in seconds.
- src_ip - The (possibly anonymised) IP address of the
TCP Stream initiator.
- src_port - The Port Number used by the TCP Stream
Initiator.
- dst_ip - The (possibly anonymised) IP address of the
TCP Stream receiver.
- dst_port - The Port Number used by the TCP Stream
Receiver.
- c2sdata - TCP Payload delivered from the TCP Client to
the TCP Server.
- s2cdata - TCP Payload delivered from the TCP Server to
the TCP Client.
- c2sohead - TCP Overhead from the TCP Client to the TCP
Server.
- s2cohead - TCP Overhead from the TCP Server to the TCP
Client.
- hop - Mean hop count estimation between the Client and
Server over the duration of the TCP Stream.
- rtt - Mean RTT estimation between the Client and Server
over the duration of the TCP Stream.
- jit - Mean Jitter estimation between the Client and Server
over the duration of the TCP Stream.
- c2splr - Estimated Packet Loss Rate for the data flow
from the TCP Client to the TCP Server.
- s2cplr - Estimated Packet Loss Rate for the data flow
from the TCP Server to the TCP Client.
- hcmean - Mean hop count estimation between the packet
capture device and the TCP Client over the duration of the TCP Stream.
- hc_histo_list - Histogram values containing the count
of all hop estimations between the packet capture device and the TCP
Client. The first value is the number of times a hop count of zero
was estimated, the next value the number of times a hop count of one
was estimated, and so on. These values can be used to see changes in
network routes.
- hsmean - Mean hop count estimation between the packet
capture device and the TCP Server over the duration of the TCP Stream.
- hs_histo_list - Histogram values containing the count
of all hop estimations between the packet capture device and the TCP
Server.
- rtt_mean - Mean RTT estimation between the Client and
Server over the duration of the TCP Stream.
- rtt_list - A list of the RTT estimates at intervals
throughout the duration of the TCP Stream. The sample interval is
determined using the -s command line parameter to NetSniff,
the default value is one sample every 10ms.
- jit_mean - Mean Jitter estimation between the Client
and Server over the duration of the TCP Stream.
- jit_list - A list of the Jitter estimates at intervals
throughout the duration of the TCP Stream.
HTTP
Output File: http.log
Output Specification: [transaction_list]
- transaction_list - A list (possibly consisting
of zero entries) of all HTTP transactions in the HTTP connection.
Each list entry consists of a single line and is formatted as
cmd [url host ref] [up_len] [c_type] rcode [dl_len] [cache]
where:
- cmd - The type of HTTP request, possible values are
GET, HEAD, POST, PUT,
DELETE, TRACE, CONNECT OR
?(unknown command type).
- url - The URL specified in the HTTP request, value
is only present when netsniff is executing in NON-Anonymised
mode.
- host - The host specified in the HTTP request, value
is only present when netsniff is executing in NON-Anonymised
mode.
- ref - The referrer specified in the HTTP request, value
is only present when netsniff is executing in NON-Anonymised
mode.
- up_len - The number of bytes uploaded, value
is only present if the HTTP request type is POST.
- c_type - The content type of the HTTP transaction, value
is only present if the HTTP request type is not CONNECT.
- rcode - The HTTP response code from the server..
- dl_len - The number of bytes downloaded to the client,
value is only present if the HTTP request type is not
CONNECT.
- cache - Is the delivered content cacheable or not.
This field can have the possible values NOT_Cacheable
or Cacheable. This field is only present if the
HTTP transaction was successful.
FTP
Output File: ftp.log
Output Specification: username OSstr file_op err_file_op unimpl err_resp un_cmd anonm login data_cnct
- username - The user name used to login to the FTP server.
If anonymisation is enables, consists of the hashed value of the user name,
or common values of anonymous, guest or ftp.
- OSstr - The string representing the advertised Operating System
running on the FTP server.
- file_op - The number of successful file operations on the FTP
server.
- err_file_op - The number of file operations that produced an error.
- unimpl - The number of unimplemented commands that occured during the FTP
connection (can be helpful to decide if additional commands are
necessary).
- err_resp - The number of times the server replied with an error
status.
- un_cmd - The number of unknown (by netsniff) commands sent to the
server.
- anonm - Set if the login was anonymous.
- login - Indicates if the FTP connection contained a successful login.
- data_cnct - A list of connections opened from server to client
and respectively from client to server, and the operation the connection
was used for (LIST, RETR or SEND).
SMTP
Output File: smtp.log
Output Specification: [sender] adv_size cmd [mail_size]
- sender - The sender email addresses used in the connection.
If anonymisation is enabled, this is the secure hash of the sender address.
- adv_size - The maximum advertised size that the SMTP server is
willing to send.
- cmd - The number of commands sent from SMTP client to SMTP
server.
- mail_size - The list of sizes of sent e-mails.
POP3
Output File: pop3.log
Output Specification: user del_mails err prot_err [rec_mails] [inbox]
- user - The user name used to connect to the POP3 server.
If anonymisation is enabled, a secure hash of the user name.
- del_mails - The number of mails deleted from the
server.
- err - The number of errors encountered during the session
(number of times the server responded with ERR).
- prot_err - The number of times netsniff encountered
behaviour not conformant to the RFC.
- rec_mails - List of the sizes of mails downloaded from
the server.
- inbox - List of mail sizes on the server. Only present
if the client issued a LIST command.
IMAP
Output File: imap.log
Output Specification: user aut_mtd [part_ml] [dld_ml]
- user - The user name used to connect to the IMAP server. If
anonymisation is enabled, a secure hash of the user name. If the
authentication method differs from "username/password", then the value
is UNKNOWN.
- aut_mtd - The method used to authenticate the user
(currently password or CRAM-MD5).
- part_ml - The sizes of the emails partially downloaded.
Gives a distribution of e-mail sizes kept in the inbox.
- dld_ml - The sizes of the mails completely downloaded.
TLS Session
Output File: tlssession.log
Output Specification: sessid haderr vs ciph comp pld_lth ohead tcpstr sdur tcptm
- sessid - The session ID created by the TLS server and
sent to the client. Uniquely defines the TLS session in a certain
time frame.
- haderr - Flag set when the session wasn't completely captured.
- vs - TLS version number (0x301).
- ciph - Cipher method used.
- comp - Compression method used. 0x0 if no compression.
- pld_lth - Number of bytes transported through the TLS session.
- ohead - Overhead produced by using TLS.
- tcpstr - Number of TCP streams associated with the TLS session.
- sdur - Session duration. Amount of time elapsed from the
creation of the session until the end of the last TCP stream belonging
to the TLS session.
- tcptm - List of time stamps, uniquely identify the TCP streams
associated to this TLS session.
TLS Stream
Output File: tlsstream.log
Output Specification: As per output for TCP Stream
- Each individual TLS Stream captured is logged in this file. The streams
can be correlated back to the TLS Session for which they belong. This file
contains the TCP information of the TLS Stream.
All Remaining TCP Streams
All TCP Streams analysed whose application type is not parsed by one of
netsniff's existing Application Layer parsers will have its TCP statistics
logged to the tcpstream.log file. This allows partial analysis
of other TCP streams captured as part of the traffic trace.
Output File: tcpstream.log
Output Specification: As per output for TCP Stream
Non-Parsed Traffic
Netsniff captures and analyses all traffic that passes through its configured
filter, this may include traffic that is not parsed by one of the application
filters listed above. In this case, these packets - including TCP Packets of
non-parsed TCP Streams logged in tcpstream.log - are logged in the
notparsed.dump file as described below.
Output File: notparsed.dump
Output Specification: tcpdump
formatted file with the following modifications:
- Only the first 68 bytes of the captured packet are stored.
- If anonymisation is enabled in netsniff, the IP addresses within the
logged packets will be anonymised consistently with all anonymised
IP addresses logged by netsniff. Further all checksum fields will
be modified to reflect the new IP addresses.
|
