[Paper Review] Van Wijngaarden grammars, metamorphism and K-ary malwares

Gueguen Geoffroy|arXiv (Cornell University)|Sep 21, 2010

semigroups and automata theory7 references3 citations

TL;DR

This paper proposes using Van Wijngaarden grammars to formally generate K-ary malware—malicious code split across K files that act together as a single virus—by leveraging their expressive power to encode semantic consistency and mutual dependencies across components. The method enables constructive, undetectable metamorphic generation of K-ary viruses through context-sensitive rewriting and shared information propagation.

ABSTRACT

Grammars are used to describe sentences structure, thanks to some sets of rules, which depends on the grammar type. A classification of grammars has been made by Noam Chomsky, which led to four well-known types. Yet, there are other types of grammars, which do not exactly fit in Chomsky's classification, such as the two-level grammars. As their name suggests it, the main idea behind these grammars is that they are composed of two grammars. Van Wijngaarden grammars, particularly, are such grammars. They are interesting by their power (expressiveness), which can be the same, under some hypotheses, as the most powerful grammars of Chomsky's classification, i.e. Type 0 grammars. Another point of interest is their relative conciseness and readability. Van Wijngaarden grammars can describe static and dynamic semantic of a language. So, by using them as a generative engine, it is possible to generate a possibly infinite set of words, while assuring us that they all have the same semantic. Moreover, they can describe K-ary codes, by describing the semantic of each components of a code.

Motivation & Objective

• To formalize the generation of K-ary malware using Van Wijngaarden grammars, which support complex semantic constraints across distributed code parts.
• To overcome limitations of traditional polymorphic and metamorphic malware by enabling automatic, constructive generation of K-ary viruses with shared semantic information.
• To provide a formal framework that supports undecidable or NP-hard recognition problems, making detection inherently difficult.
• To model the distributed nature of K-ary viruses as a unified grammar system where each component is generated with shared context.
• To demonstrate that Van Wijngaarden grammars can serve as a powerful generative engine for advanced malware families beyond Chomsky’s hierarchy.

Proposed method

• Use Van Wijngaarden grammars, which consist of two context-free grammars and a uniform replacement rule, to define the structure and semantics of K-ary malware components.
• Define the starting symbol S to generate all K parts of the virus, with each part produced via a grammar that depends on shared contextual information (INFOS).
• Apply hyperrules to bind each component (e.g., PART1 INFOS, PART2 INFOS) to a grammar that generates syntactically distinct but semantically equivalent code.
• Use the ⊕ operator to select subwords from input files and check if any form a valid virus word in the language L(Gv), establishing the K-ary virus relation Rv.
• Leverage the uniform replacement rule to enforce context-sensitive conditions, allowing semantic consistency across components during generation.
• Constructively generate K-ary malware by recursively applying grammar rules with shared INFOS, enabling dynamic mutation and metamorphism per component.

Experimental results

Research questions

• RQ1 Can Van Wijngaarden grammars be used to formally define and generate K-ary malware with distributed, semantically coordinated components?
• RQ2 How can the expressive power of Van Wijngaarden grammars be harnessed to model metamorphic transformations across multiple code files?
• RQ3 What formal mechanism allows K-ary viruses to maintain semantic equivalence while being syntactically diverse across components?
• RQ4 In what way do Van Wijngaarden grammars support undecidability or NP-hardness in malware recognition, enhancing evasion capabilities?
• RQ5 How can shared contextual information (INFOS) be consistently propagated across K distinct files to ensure functional coherence of the full virus?

Key findings

• Van Wijngaarden grammars can generate an infinite set of syntactically distinct yet semantically equivalent malware variants, including K-ary viruses.
• The use of a shared INFOS context across components enables consistent semantic behavior despite syntactic mutation in each file.
• The formalism supports the constructive generation of K-ary malware, providing a proof-of-concept for automatic, undetectable code generation.
• The grammar’s ability to express context-sensitive conditions via uniform replacement allows modeling complex dependencies between distributed virus parts.
• K-ary viruses defined via Van Wijngaarden grammars are not subject to Cohen’s original virus model, as they are not single sequences but distributed programs.
• The recognition problem for such generated malware is inherently hard (NP-hard or undecidable), making signature-based detection ineffective.