AMOS2: adaptive multi-objective seed schedule in gray-box fuzzing

Abstract Coverage-based graybox fuzzing is one of the most effective methods for identifying vulnerabilities in the field of software security testing. To maximize performance, fuzzers need to assess the quality of seeds and make two decisions appropriately: (1) which seed (Parent test case) has more potential for fuzzing, i.e. the prioritization problem? (2) How many new inputs (Child test cases) are generated by mutating a seed, i.e. energy schedule problem? However, existing studies do not rationally utilize metrics of seeds to make the above two decisions. To cope with the above problems, we implement our fuzzer AMOS2. We propose the concepts of state-dependent and state-independent metrics of seeds for the first time. We prioritize the seeds based on state-dependent metrics and balance these metrics according to the idea of multi-objective optimization. To make the energy schedule more rational, we utilize the Analytic Hierarchy Process (AHP) and Technique for Order Preference by Similarity to Ideal Solution (TOPSIS) to evaluate the state-independent metrics of seeds and guide the energy schedule. We evaluate AMOS2’s effectiveness on 10 real-world programs and the ground-truth fuzzing benchmark called MAGMA. The results show that AMOS2 achieves the highest path coverage in 9 of 10 real-world programs and finds a 0-day Floating Point Exception vulnerability (CVE-2024-57598) in mp4dump. In addition, AMOS2 found more bugs in MAGMA than other baseline fuzzers.