[Paper Review] Limitless HTTP in an HTTPS World: Inferring the Semantics of the HTTPS Protocol without Decryption

Blake Anderson, Andrew Chi+2 more|arXiv (Cornell University)|May 29, 2018

Internet Traffic Analysis and Secure E-voting20 references3 citations

TL;DR

This paper presents a novel framework to infer HTTP semantics—such as method, status code, Content-Type, and header presence—from passive HTTPS traffic without decryption. Using iterative multi-class and binary classification on TLS-encrypted traffic from browsers and malware sandboxes, the method achieves F₁ scores above 0.900 for most fields, demonstrating that significant protocol-level information can be extracted despite encryption, with implications for malware detection and privacy trade-offs.

ABSTRACT

We present new analytic techniques for inferring HTTP semantics from passive observations of HTTPS that can infer the value of important fields including the status-code, Content-Type, and Server, and the presence or absence of several additional HTTP header fields, e.g., Cookie and Referer. Our goals are twofold: to better understand the limitations of the confidentiality of HTTPS, and to explore benign uses of traffic analysis such as application troubleshooting and malware detection that could replace HTTPS interception and static private keys in some scenarios. We found that our techniques improve the efficacy of malware detection, but they do not enable more powerful website fingerprinting attacks against Tor. Our broader set of results raises concerns about the confidentiality goals of TLS relative to a user's expectation of privacy, warranting future research. We apply our methods to the semantics of both HTTP/1.1 and HTTP/2 on data collected from automated runs of Firefox 58.0, Chrome 63.0, and Tor Browser 7.0.11 in a lab setting, and from applications running in a malware sandbox. We obtain ground truth plaintext for a diverse set of applications from the malware sandbox by extracting the key material needed for decryption from RAM post-execution. We developed an iterative approach to simultaneously solve several multi-class (field values) and binary (field presence) classification problems, and we show that our inference algorithm achieves an unweighted $F_1$ score greater than 0.900 for most HTTP fields examined.

Motivation & Objective

• To investigate whether meaningful HTTP protocol semantics can be inferred from passive HTTPS traffic without decryption.
• To evaluate the feasibility of replacing TLS decryption with traffic analysis for network monitoring and security tasks.
• To assess the impact of these inference techniques on privacy, particularly in adversarial contexts like website fingerprinting and de-anonymization.
• To explore practical applications in malware detection, network debugging, and policy enforcement using encrypted traffic analysis.
• To evaluate the robustness of these techniques against privacy-preserving systems like Tor, which employ defenses such as fixed-length cells and multiplexing.

Proposed method

• The authors frame the problem as a set of disjoint multi-class and binary classification tasks for HTTP fields such as method, status-code, Content-Type, and header presence (e.g., Cookie, Referer).
• They use an iterative classification strategy that leverages inferences from related transactions in the same TLS connection to improve accuracy.
• The method is trained on encrypted HTTPS traffic collected from Firefox 58.0, Chrome 63.0, Tor Browser 7.0.11, and a malware sandbox, with data split into training (week 1) and testing (week 2) sets.
• Ground truth plaintext for malware data is obtained by extracting TLS key material from RAM post-execution, enabling supervised learning on real-world encrypted traffic.
• The system first identifies HTTP/1.1 and HTTP/2 records within TLS streams with F₁ > 0.99 and F₁ > 0.98, respectively, before classifying field values and presence.
• Feature importance and model robustness are evaluated over time, acknowledging that feature relevance may shift, necessitating continual retraining.

Experimental results

Research questions

• RQ1 Can HTTP protocol semantics such as method, status code, and header fields be accurately inferred from passive HTTPS traffic without decryption?
• RQ2 To what extent do these inference techniques improve malware detection in encrypted traffic, and can they be evaded by common obfuscation techniques?
• RQ3 Do these inference methods enhance website fingerprinting attacks on the Tor network, and if not, why?
• RQ4 How does the performance of the inference system vary across different browsers and network environments, including dynamic and sandboxed traffic?
• RQ5 What are the ethical and privacy implications of inferring HTTP semantics from encrypted traffic, particularly regarding user de-anonymization and policy enforcement?

Key findings

• The inference framework achieves an unweighted F₁ score greater than 0.900 for most HTTP fields examined, including Content-Type, Server, and status-code, with the method field achieving 0.717 and the status-code field 0.492.
• The system identifies HTTP/1.1 and HTTP/2 records within TLS streams with F₁ scores of 0.99 and 0.98, respectively, indicating high accuracy in record boundary detection.
• The technique improves malware detection performance by leveraging distinct distributions of HTTP header fields in malicious traffic, though evasion strategies can reduce its effectiveness.
• Website fingerprinting attacks on Tor were not improved, as Tor’s use of fixed-length cells and multiplexing effectively mitigates the inference techniques.
• The method enables passive network monitoring and debugging without active scanning or TLS termination, offering a scalable alternative for network administrators.
• The framework can be used to de-anonymize NAT’d users by correlating Referer/Origin/Cookie field presence with timestamps, and to detect out-of-policy file transfers by inferring file sizes from HTTP/2 frame types.