Stephen Reese

Detecting Tor network traffic with SiLK

January 9, 2014 · By Stephen Reese

This article continues a series on identifying Tor network traffic. It demonstrates how to build a SiLK IP set containing known Tor relay addresses and compare that set with collected flow records. This article assumes that you have downloaded, compiled, and installed SiLK, YAF, and libfixbuf.

We need to obtain the current list of Tor servers and place them in a file. We then extract the destination IP addresses and use rwsetbuild to store them in a SiLK IP set. The resulting IP set can be supplied to rwfilter to identify outbound flows whose destination addresses match known Tor relays.

A destination-address match shows that a system communicated with an IP address associated with a Tor relay. It does not prove that the connection used the Tor protocol. Relay hosts may provide unrelated services, and relay assignments change over time.

Historical note: The following script reflects the relay-list collection method used when this article was written. A modern implementation should retrieve maintained Tor relay data from a current authoritative source instead of relying on a hard-coded directory-server address.

#!/usr/bin/perl
#
# Fetch the list of known Tor servers (from an existing Tor server) and
# display some of the basic info for each router. use LWP::Simple; # Hostname of an existing Tor router. We use one of the directory authorities
# since that's pretty much what they're for.
$INITIAL_TOR_SERVER = "193.23.244.244"; # http://dannenberg.ccc.de/tor/status/all
$DIR_PORT = 80; # Fetch the list of servers
$content = get("http://$INITIAL_TOR_SERVER:$DIR_PORT/tor/status/all");
@lines = split /\n/,$content; foreach $router (@lines) { if($router =~ m/^r\s+(\S+)\s+(\S+)\s+(\S+)\s+(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})\s+(\S+)\s+(\d+)\s+(\d+)$/) { ($name, $address, $or_port, $directory_port, $update_time) = ($1, $5, $6, $7, $4); print "$name | $address | $or_port | $directory_port | $update_time\n"; }
}

Now that we have the current Tor server list, we can parse the Tor IP addresses. Although the Perl script could be modified to print only relay IP addresses, the following pipeline extracts the address field, removes leading whitespace, sorts the results, and removes duplicates. You could also specify what type of Tor IP addresses you would like, i.e. exit, active, etc. Retaining the complete relay data can also be useful when later queries need to associate addresses with relay ports or attributes.

$ awk -F "|" '{ print $2 }' exit-addresses \ | awk '{ sub(/^[ \t]+/, "") }; 1' \ | sort \ | uniq > tor.txt

We process the packet capture with YAF and write the resulting flow records to a file:

$ rwsetbuild tor.txt tor-servers.set

In a production environment, flow records would typically already exist for retrospective analysis. For this example, however, we use a packet capture containing historical Tor traffic. We first process the packet capture with YAF and write the resulting flow records to a file.

$ /usr/local/bin/yaf --in tor.pcap --out ~/tor.yaf --filter="port 443" --applabel --applabel-rules=/usr/local/etc/yafApplabelRules.conf --max-payload=4000 --plugin-name=/usr/local/lib/yaf/dpacketplugin.la --plugin-opts="443" --lock &

Next, convert the YAF-generated IPFIX records into SiLK flow records:

$ rwipfix2silk --silk-output=tor.rw tor.yaf

The following rwfilter command selects flows whose destination addresses appear in the Tor relay set and writes the matching records to a smaller binary SiLK file. We could write the results to standard output but I usually end up running additional queries using tools such as rwcut and rwstats. Writing the matches to a smaller binary file makes subsequent rwcut and rwstats queries faster than repeatedly filtering the original dataset.

$ rwfilter --start-date=2013/12/30 --end-date=2013/12/30 --dipset=tor-servers.set --proto=0- --type=all --pass=tor2.bin tor.rw

We parse the SiLK records we are interested in seeing to standard out via the rwcut command. Note the use of the cut command to minimize the whitespace prefixing the output.

$ rwcut tor2.bin | cut -c26- sIP| dIP|sPort|dPort|pro| packets| bytes| flags| sTime| duration| eTime|sen|
10.0.0.126| 198.27.97.223|38946| 443| 6| 30| 8497|FS PA |2013/12/30T20:20:21.336| 76.182|2013/12/30T20:21:37.518| S0|
198.27.97.223| 10.0.0.126| 443|38946| 6| 32| 28802|FS PA |2013/12/30T20:20:21.381| 76.137|2013/12/30T20:21:37.518| S0|
10.0.0.126| 96.127.153.58|42529| 443| 6| 27| 8341|FS PA |2013/12/30T20:20:22.190| 75.341|2013/12/30T20:21:37.531| S0|
96.127.153.58| 10.0.0.126| 443|42529| 6| 30| 26678|FS PA |2013/12/30T20:20:22.232| 75.299|2013/12/30T20:21:37.531| S0|
10.0.0.126| 192.151.147.5|44384| 443| 6| 14| 3502|FS PA |2013/12/30T20:20:26.486| 71.052|2013/12/30T20:21:37.538| S0|
192.151.147.5| 10.0.0.126| 443|44384| 6| 14| 4819|FS PA |2013/12/30T20:20:26.535| 71.003|2013/12/30T20:21:37.538| S0|

Additional matching records were omitted for readability.

With the next query, we adjust the type of traffic we want to look at to only outgoing traffic to the Tor servers instead of the previously displayed bidirectional traffic.

$ rwfilter --dipset=tor-servers.set --proto=0- --type=out --pass=tor.bin tor.rw

We then parse the outbound records with rwcut. The cut command removes the leading whitespace reserved for undisplayed columns. The reason for this is there are additional columns of data not displayed by default. Check the rwcut manual page for other data columns that may be of useful.

$ rwcut tor.bin | cut -c30- sIP| dIP|sPort|dPort|pro| packets| bytes| flags| sTime| duration| eTime|sen| 10.0.0.126| 198.27.97.223|38946| 443| 6| 30| 8497|FS PA |2013/12/30T20:20:21.336| 76.182|2013/12/30T20:21:37.518| S0|
10.0.0.126| 96.127.153.58|42529| 443| 6| 27| 8341|FS PA |2013/12/30T20:20:22.190| 75.341|2013/12/30T20:21:37.531| S0|
10.0.0.126| 192.151.147.5|44384| 443| 6| 14| 3502|FS PA |2013/12/30T20:20:26.486| 71.052|2013/12/30T20:21:37.538| S0|
10.0.0.126| 66.18.12.197|49341| 443| 6| 28| 8475|FS PA |2013/12/30T20:20:21.426| 76.125|2013/12/30T20:21:37.551| S0|
10.0.0.126| 64.62.249.222|40742| 443| 6| 30| 8159|FS PA |2013/12/30T20:20:21.375| 76.208|2013/12/30T20:21:37.583| S0|
10.0.0.126| 212.83.158.173|40825| 443| 6| 28| 8394|FS PA |2013/12/30T20:20:22.079| 75.506|2013/12/30T20:21:37.585| S0|

Additional matching records were omitted for readability.

Some reverse-DNS results contain tor in the hostname, providing an additional—but weak—indication that the destination may operate a Tor relay.

$ rwcut tor.bin | rwresolve | cut -c30- sIP| dIP|sPort|dPort|pro| packets| bytes| flags| sTime| duration| eTime|sen| 10.0.0.126|198.27.97.223.vpsrealm.com|38946| 443| 6| 30| 8497|FS PA |2013/12/30T20:20:21.336| 76.182|2013/12/30T20:21:37.518| S0|
10.0.0.126|xxviii.example.tld|42529| 443| 6| 27| 8341|FS PA |2013/12/30T20:20:22.190| 75.341|2013/12/30T20:21:37.531| S0|
10.0.0.126|tor.koehn.com|44384| 443| 6| 14| 3502|FS PA |2013/12/30T20:20:26.486| 71.052|2013/12/30T20:21:37.538| S0|
10.0.0.126| 66.18.12.197|49341| 443| 6| 28| 8475|FS PA |2013/12/30T20:20:21.426| 76.125|2013/12/30T20:21:37.551| S0|
10.0.0.126|hecustomer.10gigabitethernet8-1.core1.pao1.he.net|40742| 443| 6| 30| 8159|FS PA |2013/12/30T20:20:21.375| 76.208|2013/12/30T20:21:37.583| S0|
10.0.0.126|n5.servbr.net|40825| 443| 6| 28| 8394|FS PA |2013/12/30T20:20:22.079| 75.506|2013/12/30T20:21:37.585| S0|

Additional matching records were omitted for readability.

Or we can use the rwuniq command to list the unique destinations, again piping through rwresolve:

$ rwuniq --fields=2 --no-columns tor.bin | rwresolve
dIP|Records|
luftgitarr.mooo.se|1|
tor.b0red.de|1|
junis.mooo.se|1|
31.7.186.228|1|
tor21.anonymizer.ccc.de|1|
xxviii.example.tld|1|
tor.koehn.com|1|
n15.servbr.net|1|
a80-100-45-156.adsl.xs4all.nl|1|
n13.servbr.net|1|
120-20-159-88.business.edutel.nl|1|
91.143.91.174|1|
195.ab4.interhost.co.il|1|
37-59-150-178.static-ip.hostplanet.me|1|
sa0111.azar-a.net|1|
static.188-40-98-96.clients.your-server.de|1|
n5.servbr.net|1|
torsrvl.snydernet.net|1|
198.27.97.223.vpsrealm.com|1|
66.18.12.197|1|
v37433.1blu.de|1|
hecustomer.10gigabitethernet1-2.core1.ams1.he.net|1|
212-83-140-45.rev.poneytelecom.eu|1|
kimya.mooo.se|1|
85.17.122.80|1|
n12.servbr.net|1|
greendale.badexample.net|1|
n10.servbr.net|1|
hecustomer.10gigabitethernet8-1.core1.pao1.he.net|1|

This method is not definitive. An IP-address match can produce false positives because a Tor relay host may also provide legitimate non-Tor services. Nevertheless, the technique can quickly identify flows that warrant additional protocol-level investigation.