[Paper Review] DeepDGA: Adversarially-Tuned Domain Generation and Detection

Hyrum S. Anderson, Jonathan Woodbridge+1 more|arXiv (Cornell University)|Oct 6, 2016

Advanced Malware Detection Techniques45 citations

TL;DR

This paper proposes DeepDGA, a generative adversarial network (GAN)-based DGA that adversarially tunes domain names to evade deep learning-based DGA detectors. By iteratively training a generator to produce stealthy domains and a detector to distinguish them, the framework produces adversarial examples that generalize to other models—demonstrating that augmenting training data with these examples significantly improves detection of unseen DGA families, including on a random forest classifier with hand-crafted features.

ABSTRACT

Many malware families utilize domain generation algorithms (DGAs) to establish command and control (C&C) connections. While there are many methods to pseudorandomly generate domains, we focus in this paper on detecting (and generating) domains on a per-domain basis which provides a simple and flexible means to detect known DGA families. Recent machine learning approaches to DGA detection have been successful on fairly simplistic DGAs, many of which produce names of fixed length. However, models trained on limited datasets are somewhat blind to new DGA variants. In this paper, we leverage the concept of generative adversarial networks to construct a deep learning based DGA that is designed to intentionally bypass a deep learning based detector. In a series of adversarial rounds, the generator learns to generate domain names that are increasingly more difficult to detect. In turn, a detector model updates its parameters to compensate for the adversarially generated domains. We test the hypothesis of whether adversarially generated domains may be used to augment training sets in order to harden other machine learning models against yet-to-be-observed DGAs. We detail solutions to several challenges in training this character-based generative adversarial network (GAN). In particular, our deep learning architecture begins as a domain name auto-encoder (encoder + decoder) trained on domains in the Alexa one million. Then the encoder and decoder are reassembled competitively in a generative adversarial network (detector + generator), with novel neural architectures and training strategies to improve convergence.

Motivation & Objective

• To develop a deep learning-based DGA that explicitly evades detection by a deep learning classifier through adversarial training.
• To investigate whether adversarially generated domains can serve as effective data augmentation to harden machine learning models against previously unseen DGA families.
• To address training challenges in character-level GANs for domain name generation, including mode collapse and poor convergence.
• To evaluate the generalization of adversarial examples across different model architectures, such as random forests with hand-engineered features.
• To demonstrate that adversarial examples from a GAN can improve detection performance on out-of-distribution DGA families.

Proposed method

• A character-level deep learning model is pre-trained as an autoencoder on Alexa’s top 1 million domains to learn domain name representations.
• The encoder and decoder are re-purposed into a GAN: the generator produces domain names, and the discriminator acts as a DGA detector.
• Novel training strategies include history regularization, box layers, and principal axis box layers to stabilize training and improve convergence.
• The generator is trained to produce domains that fool the discriminator, while the discriminator is updated to better distinguish real from adversarial domains in adversarial rounds.
• Adversarial examples from the GAN are used to augment a training dataset for a separate random forest classifier to improve its robustness.
• The framework uses a leave-one-family-out cross-validation strategy to evaluate generalization to unseen DGA families.

Experimental results

Research questions

• RQ1 Can a GAN be trained to generate domain names that are adversarial to a deep learning DGA detector, thereby evading detection?
• RQ2 Do adversarial examples generated for one model architecture (e.g., deep neural network) also evade detection by a different model architecture (e.g., random forest with hand-crafted features)?
• RQ3 Can adversarial examples from a GAN be used to augment training data and improve detection performance on previously unseen DGA families?
• RQ4 What novel training techniques are necessary to stabilize training of a character-level GAN for domain name generation?
• RQ5 To what extent does adversarial data augmentation improve the true positive rate at a fixed false positive rate for out-of-distribution DGA families?

Key findings

• The DeepDGA generator successfully produced domains that evaded detection by its own discriminator, achieving a detection rate of only 48% on a 20% holdout set.
• Adversarial domains from the GAN also evaded a random forest classifier trained on hand-engineered features, indicating generalization across model types.
• After augmenting the training set with 10,000 adversarial DeepDGA samples, the random forest classifier’s average true positive rate at 1% FPR improved from 68% to 70% on held-out DGA families.
• The hardened model showed particular improvement on previously low-performing families, such as pykspa (from 67% to 71% TPR) and ramdo (from 54% to 54% TPR), with most families seeing gains or stable performance.
• The approach demonstrated that GAN-generated adversarial examples can effectively harden models against unseen DGA variants, outperforming traditional perturbation-based adversarial training in distributional realism.
• The use of autoencoder pre-training and novel layers (box layer, principal axis box layer) significantly improved GAN training stability and convergence.