Learning to detect malicious bots in computer networks via heterophily-aware isotropic out-of-distribution detection

Malicious bots typically operate within networks through peer-to-peer (P2P) communication structures, leading to the emergence of graph neural networks (GNNs) as a promising bot detection method. However, communications graphs representing bot-infected networks often exhibit an inherent imbalance, coupled with a high degree of heterophily. Graph oversampling techniques, employed to address class imbalance on graphs, are burdened with downsides, such as the creation of complex and noisy topological structures or further amplification of heterophily in a graph. Out-of-distribution detection (ODD) is considered as an alternative solution to address data imbalance issues, but when applied to graphs, this belief is built on an assumption that the underlying graph structure does not interfere with the learning of data distributions. In this paper, we propose a new ODD model HistNet which implements Heterophily-aware isotropic out-of-distribution detection to explore how to leverage ODD for malicious bot detection in a Network. HistNet proceeds with heterophily-aware node embedding that facilitates enhanced isotropic distance calculation and homophily-augmented distance-based belief propagation, which is further regularized by implicit clustering. These technical designs enable HistNet to overcome performance issues caused by imbalance and heterophily in graphs and improve isotropic ODD for bot detection. We validate our claims through extensive experiments on 10 computer networks derived from TON IoT datasets, which comprise real captured bot data. The experimental results demonstrate that HistNet achieves state-of-the-art performance in malicious bot detection on graphs with high graph heterophily and extreme class imbalance.