[Paper Review] Narcissus: Deriving Correct-By-Construction Decoders and Encoders from Binary Formats

Benjamin Delaware, Sorawit Suriyakarn+3 more|arXiv (Cornell University)|Mar 13, 2018

Advanced Malware Detection Techniques42 references3 citations

TL;DR

Narcissus is a Coq-based framework that automatically derives correct-by-construction decoders and encoders for complex, non-context-free binary formats using parser combinator-style syntax. By leveraging proof automation and bidirectional derivation, it ensures encoder-decoder invertibility with machine-checked correctness proofs, enabling secure, efficient implementation in systems like MirageOS with minimal performance overhead.

ABSTRACT

It is a neat result from functional programming that libraries of parser combinators can support rapid construction of decoders for quite a range of formats. With a little more work, the same combinator program can denote both a decoder and an encoder. Unfortunately, the real world is full of gnarly formats, as with the packet formats that make up the standard Internet protocol stack. Most past parser-combinator approaches cannot handle these formats, and the few exceptions require redundancy -- one part of the natural grammar needs to be hand-translated into hints in multiple parts of a parser program. We show how to recover very natural and nonredundant format specifications, covering all popular network packet formats and generating both decoders and encoders automatically. The catch is that we use the Coq proof assistant to derive both kinds of artifacts using tactics, automatically, in a way that guarantees that they form inverses of each other. We used our approach to reimplement packet processing for a full Internet protocol stack, inserting our replacement into the OCaml-based MirageOS unikernel, resulting in minimal performance degradation.

Motivation & Objective

• To address the lack of correct-by-construction, non-redundant specification and implementation of encoders and decoders for complex binary formats in real-world systems.
• To eliminate user-induced bugs in encoder/decoder generation by integrating formal verification via Coq's proof automation.
• To support fine-grained, extensible format specifications that handle context-sensitive, non-context-free protocols common in networking.
• To enable seamless integration into verified systems by producing machine-checked proofs of correctness for both encoding and decoding logic.
• To demonstrate practical viability by replacing packet processing in the MirageOS unikernel with minimal performance impact.

Proposed method

• Uses a domain-specific embedded language (EDSL) in Coq to specify binary formats using combinators like `format_word`, `++` (sequence), and `format_unused_word` for padding.
• Employs a custom tactic, `derive_encoder_decoder_pair`, to automatically generate both encoder and decoder functions from a single format specification.
• Maintains bidirectional consistency by threading type and bit-level information through the derivation, ensuring that encoding and decoding are exact inverses.
• Generates code in a stateful error monad with `GetCurrentByte` and `SetCurrentByte` operations to model byte-stream I/O.
• Applies bit-level manipulation primitives such as `high_bits` and `low_bits` to handle multi-byte word serialization and reassembly.
• Uses Coq’s proof automation to generate and discharge correctness proofs, guaranteeing that every generated encoder-decoder pair is mathematically sound.

Experimental results

Research questions

• RQ1 Can a functional programming approach to format specification produce correct-by-construction encoders and decoders without redundancy?
• RQ2 How can a single format specification be used to derive both encoder and decoder functions with formal guarantees of correctness?
• RQ3 Can the framework handle complex, non-context-free binary formats such as those found in standard networking protocols?
• RQ4 What is the performance overhead of using a formally verified encoder-decoder stack in a real-world unikernel system?
• RQ5 To what extent can the framework be extended with user-defined format combinators and implementation strategies?

Key findings

• The Narcissus framework successfully derives correct-by-construction encoders and decoders for all major network packet formats, including those with alignment, padding, and complex field dependencies.
• The framework’s use of Coq’s proof automation ensures that every generated encoder-decoder pair is mathematically proven to be inverses, eliminating the possibility of encoding-decoding mismatches.
• The approach achieves minimal performance degradation when replacing packet processing in the MirageOS unikernel, demonstrating practical efficiency.
• The framework supports extensibility through user-defined format combinators and can interoperate with arbitrary Coq data types, not just fixed C-style structs.
• The generated code is free of redundancy, as the same format specification directly drives both encoder and decoder generation without duplication or manual translation.
• The framework outperforms existing code-generation and parser-combinator tools in correctness guarantees, as it provides machine-checked proofs rather than informal or ad hoc verification.