Reverse-Engineering and Fuzzing a Custom Network Protocol

#Overview

The communication protocol deﬁnes the format and semantics of message exchange between applications. In modern times there are a myriad of proprietary application protocols like Skype Protocol, Dropbox Protocol, etc which applications use to achieve various goals like bandwidth eﬃciency, custom encryption/compression, etc. These protocols could have security vulnerabilities. Protocol Reverse Engineering (PRE) is not only useful for oﬀensive purposes but also used by modern Intrusion Detection Systems(IDS), they use the knowledge of protocol speciﬁcation to do Deep Packet Inspection(DPI) which can enhance its capabilities, where it earlier relied just based on pattern matching which may produce lots of false positives. Custom protocols are not only used by legitimate applications but also by malware and botnets like Zeus, Emotet, etc. By reversing malware protocol you can connect to malware servers and track their campaigns.

Protocol Reverse Engineering(PRE) is an art and science of recovering the protocol speciﬁcation of the obscure/proprietary protocol whose documentation is unavailable or poorly documented. There are eﬀorts to develop automated PRE tools but they are largely academic and are not mature enough to be usable, and can’t give the accuracy a human analyst can oﬀer. Automated tools face the challenges of heterogeneous protocol data which is often a mixture of text and binary, and it has diﬀerent data types and variable-length ﬁelds and this is the reason I have created this training, it is to help you understand these challenges and learn to recover protocol speciﬁcation.

This training is divided into two parts, in the ﬁrst part we will learn about Protocol Reverse Engineering principles. We will look at some of the common data formats and other protocol structures and with that understanding we will write a protocol dissector using Scapy framework for a target Desktop game Minetest (open source implementation of Minecraft). Minetest is online multiplayer game in which diﬀerent players can connect to the server and play with other players, there are also many public servers which you can connect and play. Once we have written the decoder we will sniﬀ the connection and look at the communication ﬂow between the client and the server which we will capture and re-analyze the traﬃc to improve the dissector further, using this newly improved dissector we will implement a custom game client/bot which will connect to the server and play as a Bot player.

In the second part, with a decent understanding of the Minetest Protocol we will move on to the oﬀensive side of the training and try to fuzz the game server to ﬁnd some security vulnerabilities, we will start with basic Fuzzer and try to do incremental improvement such that we have good code coverage. Leveraging their reverse-engineered understanding of the protocol, participants will employ Generational Fuzzing by defining the protocol specification in the Boofuzz fuzzing framework and subsequently fuzzing the application. The training will also explore Mutation Fuzzing as an alternative approach to identify potential crashes or vulnerabilities.

This hands-on game hacking training is a takes project-based learning approach, ensuring a comprehensive and practical understanding of Protocol Reverse Engineering. In summary, this training aims to equip you with the knowledge and skills to reverse engineer and understand obscure protocols, enhance IDS capabilities, and explore oﬀensive techniques such as protocol fuzzing to uncover potential security weaknesses. The ultimate goal is to empower participants to create their own tools in the realm of protocol security.

#Why should you take this course?

Understand the structure of the requests and response especially useful for a malware analyst.
Construction of protocol decoders useful for writing gaming clients or to add support for a third-party proprietary product.
Reverse engineer communication of online games like MMORPG which can help you do security testing of multi-player online games.
Create Network signatures for malware communication that can be integrated with IDS and IPS, understanding the protocol specification can help you to do deep packet inspection.
Write a protocol fuzzer to feed the remote server with crafted randomness in the data to crash the data processing part of the application with the intent of finding security vulnerabilities.
Identify security vulnerabilities in protocol implementation like authentication bypass, replay attack, information disclosure, DOS, RCE etc. PRE can also help you to do deeper black-box testing of the application.
Build a protocol specification for a vaguely/undocumented protocol.
Audit the privacy and security of an application running on your phone/computer by looking at what data is it exporting.

#Who should take this course?

DFIR practitioner - to investigation malicious activity in the network
Reverse Engineer - write a custom client that fully replicates the existing client software/game.
Bug Hunter - Write protocol fuzzer for Black Box testing for application processing remote data, for example, lots of IoT Devices use custom protocol for efficient communication.
Malware Analyst - To decode C&C server commands and the data which is exfiltrated
Threat Hunting - write network signatures for new emerging APT threats or it could be an intruder in your network, this course will help you decode network and analyze network traffic.
Developers - who don’t have access to source code or protocol documentation, it usually happens when you are dealing with a legacy system which is too old and the company cannot find any documentation and you intend to migrate the system to new technology.
While debugging software over the network, writing a protocol dissector can help you to get a deeper understanding of network communication done by your software.
Helps you to do network debugging/diagnostics of application layer data.
It helps you understand what is really transmitted over the network.
RED Team - take advantage of what the Security Operation Center (SOC) doesn’t know. Look for data leaks, do attacks like inject, replay and spoofing.
Vulnerability Researcher/Exploit Developer - this will also help exploit developer and vulnerability research to reproduce remote vulnerability and find zero-day bugs.

#Course Outline

Networking Basics
Capturing Network Traffic
1. Passive analysis
2. Active analysis
Protocol Reversing
1. Protocol Structure
2. Protocol Flow
Protocol Dissector (targeting Minetest game)
1. Scapy 101
2. Implementing protocol dissector in scapy for Minetent game.
  1. Protocol decoding TLV format
  2. Packet decompression
  3. Packet Reassembly
Custom Client (Bot Player for Minetest Game)
1. Brief Understanding of Application
2. Authenticate the client
3. Establish a valid session
4. Some game hacks like making the player fly
5. Create A Bot Army (if time permits)
Protocol Fuzzing (targeting Minetest game)
1. What is fuzzing?
2. Implement Mutation Fuzzer
3. Implement Dumb Fuzzer
4. Implement Generation Fuzzing (Protocol Aware Fuzzing)
5. Vulnerability Triaging (Harness)

#Tools of the Trade

Below are some of the tools that you will learn in this training that will make you Protocol Reversing experience more fun.

Protocol Reversing tools
1. Wireshark
2. Scapy
3. strace
Protocol Fuzzing Tool
1. Boofuzz (Sulley) fuzzing framework

#Prerequisite

Knowledge of security concepts
Basic understanding of networking concepts
Knowledge of Linux OS
Basic Python programming language

#What attendees should bring

Laptop with at least 50 GB free space
8+ GB minimum RAM (4+GB for the VM)
Administrative privileges on the system
Virtualization software – Latest VirtualBox (including Virtualbox extension pack)
Virtualization (Vx-t) option enabled in the BIOS settings for VirtualBox to work

#What attendees will be provided with

Virtual Machine with all the needed software pre-installed.
Training Material/slides.
Lab Manual

#What to expect

Hands-on Labs
The joy of Reverse Engineer (looking under the hood)
Getting familiar with Network Protocol Analysis
Unlimited Email Support.

#What not to expect

Become a Protocol Reversing Ninja.
Use the knowledge gained in this training to start exploring some Open and Close Protocol to improve your understanding of this topic. That will help you to get a deeper understanding of some underlying issues more closely.

#Trainer - Bio

Munawwar Hussain Shelia is employed as a Vulnerability Researcher at Qualcomm. His primary role involves diligently searching for and resolving bugs in embedded products, as well as developing tools to automate the process of bug detection. With a background in Computer Science and software development experience, Munawwar possesses a unique perspective on product design and creation, enabling him to eﬀectively identify vulnerabilities. In 2019, he conducted the “Practical IoT Hacking” Training at Nullcon and other conferences. Additionally, Munawwar delivered a talk at the diﬀerent conferences. His areas of expertise include Reverse Engineering, Binary Analysis, Malware Analysis, and Software Exploitation, topics on which he frequently shares insights through his blog, https://taintedbits.com. Munawwar has conducted training sessions for various governmental and private organizations worldwide. Notably, he has discovered and reported vulnerabilities in IoT devices and published a paper on Android Malware.

#Training History

This training has been delivered in various conferences

Reverse-Engineering and Fuzzing Custom Network Protocol - Blueteam Con 2024
Reverse-Engineering and Fuzzing Custom Network Protocol - Nullcon Goa 2023
Reverse-Engineering and Fuzzing Custom Network Protocol - Nullcon Goa 2020

#Conference Talks & Workshops

Below are some of the conference talks and workshops which I delivery in the past.