However, dealing with the rapidly expanding scale of modern network infrastructure, the current RL-based methods have shown some limitations. They often struggle with the huge action spaces in large networks, resulting in less effective and monotonous penetration testing strategies.
To solve this challenges, a research team led by Dr. Xin Liu published a new research paper on the subject that introduces a new approach to help solve the problem.
Their research introduces an innovative RL agent called Coverage-Based Reinforcement Learning to Automate Penetration Testing (CLAP), aims to change how we do network security assessments. CLAP deals with the challenge by proposing the coverage mechanism, a novel neural network, to deal with large action spaces in large networks.
Furthermore, CLAP provides comprehensive network security assessments with diverse adversary testing behaviors on a massive scale. Moreover, CLAP employs a novel neural network, namely the coverage mechanism, to address the enormous and growing action spaces in large networks.
Additionally, the proposed agent is also able to discover pareto-dominant strategies that are diverse and effective in achieving multiple objectives.
Furthermore, it employs a Chebyshev decomposition critic to identify diverse adversary strategies and strike a balance between them.
Experimental results across various scenarios demonstrate CLAP outperforms state-of-the-art methods, by further reducing attack operations by nearly 35%. CLAP also provides enhanced training efficiency and stability and can effectively perform pen-testing over large-scale networks with up to 500 hosts.

Penetration testing performed by a reinforcement learning agent.
Source: Yizhou Yang, Longde Chen, Sha Liu, Lanning Wang, Haohuan Fu, Xin Liu, Zuoning Chen
The agent also excels in discovering pareto-dominant strategies that are both diverse and effective in achieving multiple objectives. CLAP’s unique approach and impressive results mark a significant leap forward in cybersecurity research, addressing the challenges posed by the expanding scale of modern network environments.

