[Paper Review] Transparent Live Code Offloading on FPGA

Roberto Rigamonti, Baptiste Delporte+2 more|arXiv (Cornell University)|Sep 1, 2016

Parallel Computing and Optimization Techniques13 references3 citations

TL;DR

This paper presents a transparent, runtime framework for dynamically offloading computationally intensive code fragments to FPGAs without requiring code modifications or pre-deployed bitstreams. By analyzing LLVM Intermediate Representation at runtime, the system identifies hotspots, reconfigures a pre-programmed data flow overlay on the FPGA via JIT compilation, and achieves on-the-fly acceleration with minimal overhead, enabling adaptive, application-aware hardware acceleration.

ABSTRACT

Even though it seems that FPGAs have finally made the transition from research labs to the consumer devices' market, programming them remains challenging. Despite the improvements made by High-Level Synthesis (HLS), which removed the language and paradigm barriers that prevented many computer scientists from working with them, producing a new design typically requires at least several hours, making data- and context-dependent adaptations virtually impossible. In this paper we present a new framework that off-loads, on-the-fly and transparently to both the user and the developer, computationally-intensive code fragments to FPGAs. While the performance should not surpass that of hand-crafted HDL code, or even code produced by HLS, our results come with no additional development costs and do not require producing and deploying a new bit-stream to the FPGA each time a change is made. Moreover, since optimizations are made at run-time, they may fit particular datasets or usage scenarios, something which is rarely foreseeable at design or compile time. Our proposal revolves around an overlay architecture that is pre-programmed on the FPGA and dynamically reconfigured by our framework to execute code fragments extracted from the Data Flow Graph (DFG) of computational intensive routines. We validated our solution using standard benchmarks and proved we are able to off-load to FPGAs without developer's intervention.

Motivation & Objective

• To address the high development cost and inflexibility of traditional FPGA programming, especially with High-Level Synthesis (HLS), which requires long compilation and deployment cycles.
• To enable automatic, transparent offloading of computationally intensive code fragments to FPGAs at runtime, without requiring changes to application code or pragma annotations.
• To allow runtime adaptation of FPGA functionality based on actual workload and input data, overcoming the limitations of static optimization and pre-compiled bitstreams.
• To reduce the barrier to FPGA adoption by making hardware acceleration language-agnostic and transparent to both developers and end users.
• To demonstrate a fully dynamic, on-the-fly reconfiguration system that can identify, analyze, and accelerate performance-critical code regions in real time.

Proposed method

• The framework monitors application execution at runtime using a low-overhead performance monitor to detect code fragments consuming the most resources (execution time or memory accesses).
• It extracts the Control Flow Graph (CFG) and Data Flow Graph (DFG) of identified critical code regions from the LLVM Intermediate Representation (IR).
• A Just-In-Time (JIT) compiler generates stub code and compiles the DFG into a hardware configuration for a pre-programmed data flow overlay (DFE) on the FPGA.
• The DFE is reconfigured on-the-fly via the PCIe interface using DMA transfers, with data transfer protocols designed for simplicity and flexibility, though with a 75% bandwidth overhead due to lack of compression.
• The system uses asynchronous, non-blocking data transfers to minimize CPU involvement, with configuration and data transfer handled independently of the main application flow.
• The framework is generic and supports all FPGAs, with the DFE designed to be parametrically sized to fit available resources.

Experimental results

Research questions

• RQ1 Can computationally intensive code fragments be automatically detected and offloaded to FPGAs at runtime without developer intervention?
• RQ2 Can on-the-fly FPGA reconfiguration be achieved with minimal performance overhead and without pre-deploying new bitstreams?
• RQ3 To what extent can dynamic, data-aware optimization improve performance compared to static HLS compilation?
• RQ4 How does the system handle the trade-off between runtime reconfiguration cost and performance gain in real-world workloads?
• RQ5 Can a language-agnostic, transparent acceleration framework be built that operates entirely at the LLVM IR level?

Key findings

• The system successfully offloads code fragments to the FPGA in real time, with the entire reconfiguration process (including place-and-route) taking approximately 1.18 seconds in the evaluated case.
• The JIT compilation phase completed in 16.7ms, and the final DFE configuration download took only 2.1ms, demonstrating low runtime reconfiguration latency.
• The data transfer protocol introduced a 75% overhead due to sending 128 bits for every 32 bits of data, limiting the effective PCIe bandwidth to approximately 230MB/s (equivalent to ~57.5MB/s after accounting for compression inefficiency).
• Despite these limitations, the system achieved a frame rate of 31 frames per second on a video processing benchmark, compared to 83 frames per second in pure software, indicating significant room for improvement with optimized data transfer.
• The framework successfully identified and offloaded a convolution kernel with 17 inputs, 1 output, and 16 computing nodes, demonstrating the feasibility of dynamic code offloading at the DFG level.
• The system operates transparently: no code changes, no pragmas, and no developer awareness of the FPGA are required, enabling seamless hardware acceleration.