Decoding an XOR-Encoded Payload Using Its First Few Bytes as the Key

I recently came across the need to decode an exclusive or (XOR) payload. In my case, the key to de-obfuscating the traffic was the first three bytes of each packet's payload. Although decoding a single payload is straightforward, manually processing a large packet capture is impractical.

For testing purposes, create a packet:

$ scapy
Welcome to Scapy (2.1.0)
>>> p = (IP(ttl=10)/TCP(sport=1024,dport=443,flags="S")/" WHATSTHESECRET0000ABCD0000ABCD0000ABCD")
>>> wrpcap("p.pcap", p)
>>> quit()

You should see output similar to the following:

04:29:31.255470 IP 127.0.0.1.1024 > 127.0.0.1.443: Flags [S], seq 0:41, win 8192, length 41 0x0000: 4500 0051 0001 0000 0a06 b2a4 7f00 0001 E..Q............ 0x0010: 7f00 0001 0400 01bb 0000 0000 0000 0000 ................ 0x0020: 5002 2000 751d 0000 2020 2057 4841 5453 P...u......WHATS 0x0030: 5448 4553 4543 5245 5430 3030 3041 4243 THESECRET0000ABC 0x0040: 4430 3030 3041 4243 4430 3030 3041 4243 D0000ABCD0000ABC 0x0050: 44 D

Next, the payload is XOR-encoded using its first three bytes as a repeating key. In the first tcpdump output, the initial three payload bytes were spaces. In the next packet, those bytes contain the key used to XOR the remainder of the payload.

The payload has been obfuscated using the key 'the'.

Next we can use the script to decode all of the packets. The script does not identify encoded packets automatically. Filter the relevant packets into a separate PCAP before running it. Secondly, the script requires Scapy to be installed.

#!/usr/bin/python
# Script to parse a PCAP and XOR data based on a byte offset
# Requires Scapy
# 0.1 - 07172012
# Default is two bytes, change at line 35
# Stephen Reese and Chris Gragsone
#
# todo: add two more args, offset length and static offset option from scapy.all import *
import sys # Get input and output files from command line
if len(sys.argv) < 2: print "Usage: decodexorpayload.py [input pcap file]" sys.exit(1) # Assign variable names for input and output files
infile = sys.argv[1] def many_byte_xor(buf, key): buf = bytearray(buf) key = bytearray(key) key_len = len(key) for i, bufbyte in enumerate(buf): buf[i] = bufbyte ^ key[i % key_len] return str(buf) def process_packets(): pkts = rdpcap(infile) cooked=[] for p in pkts: # You may have to adjust the payload depth here: # i.e. p.payload.payload.payload pkt_payload = str(p.payload.payload) pkt_offset = str(p.payload.payload)[:3] if pkt_payload and pkt_offset: pmod=p # You may have to adjust the payload depth here: p.payload.payload=many_byte_xor(pkt_payload, pkt_offset) cooked.append(pmod) wrpcap("dump.pcap", cooked) process_packets()

After script completion, viewing the packet does indeed show the de-obfuscated packet:

reading from file dump.pcap, link-type RAW (Raw IP)
04:24:44.415262 IP 127.0.0.1.1024 > 127.0.0.1.443: Flags [S], seq 0:41, win 8192, length 41 0x0000: 4500 0051 0001 0000 0a06 b2a4 7f00 0001 E..Q............ 0x0010: 7f00 0001 0400 01bb 0000 0000 0000 0000 ................ 0x0020: 5002 2000 751d 0000 0000 0057 4841 5453 P...u......WHATS 0x0030: 5448 4553 4543 5245 5430 3030 3041 4243 THESECRET0000ABC 0x0040: 4430 3030 3041 4243 4430 3030 3041 4243 D0000ABCD0000ABC 0x0050: 44 D

Although simple, this technique is still encountered in modern malware families and remains a useful addition to a reverse engineer's toolkit.