Payload Analysis of Adversaries' Tooling: Automated Identification of Fuzzers

API fuzzing, a technique widely used to uncover vulnerabilities in web applications, poses significant security risks when exploited maliciously, leading to service disruptions and data breaches. While firewalls can block unauthorized fuzzing attempts, they limit defenders' ability to gather data on attacker methodologies, reducing actionable cyber threat intelligence. Identifying the responsible fuzzers enables defenders to trace the attacker, uncover their motives, and assess the potential impact, which helps security teams prepare more effectively, mitigate attacks, and develop targeted countermeasures to enhance the security of web APIs. However, analyzing the payloads generated by fuzzers remains largely unexplored and presents significant challenges. For instance, fuzzers often generate similar payloads due to shared initial seeds and similar fuzzing strategies, making accurate fuzzer identification more complex. To analyze this, we experimented with four well-known API fuzzers; APIFuzzer, Kiterunner, RESTler, and Schemathesis, and created a comprehensive dataset of their payloads targeting five different web APIs. Our thorough analysis reveals that the overlapping payloads, i.e., the identical generated payloads across these fuzzers, can be substantially large. For instance, ≈17% of payloads generated with Schemathesis overlapped with ≈12% of the payloads generated with RESTler across different web APIs. As a result, defining distinctive payload features that machine learning models can learn to differentiate and identify their fuzzer accurately becomes more difficult. Alternatively, deep learning techniques, known for their ability to automatically extract features, present a compelling alternative. To evaluate this, we experimented with an architecture combining a bidirectional Transformers-based encoder-decoder and a machine learning classifier to classify fuzzers based on their payloads. Rigorous evaluation using k-fold cross-validation demonstrated high precision and recall, averaging 89%, showcasing this combinatorial architecture's robustness and effectiveness. Our findings demonstrate the potential of combining deep learning and machine learning for fuzzer identification and enhancing web API security.