[Paper Review] Sketch2CAD: Sequential CAD Modeling by Sketching in Context

Changjian Li, Hao Pan+2 more|arXiv (Cornell University)|Sep 10, 2020

3D Shape Modeling and Analysis38 references4 citations

TL;DR

Sketch2CAD introduces a deep learning-based system that translates sequential freehand sketches into precise, editable CAD models by interpreting each sketch stroke in the context of a growing partial CAD model. Using a context-aware neural network, it maps sketch strokes to parametric CAD operations (e.g., extrude, bevel, add/subtract), achieving high accuracy in operation recognition and parameter fitting, enabling novice users to rapidly create diverse, manufacturable 3D models without CAD expertise.

ABSTRACT

We present a sketch-based CAD modeling system, where users create objects incrementally by sketching the desired shape edits, which our system automatically translates to CAD operations. Our approach is motivated by the close similarities between the steps industrial designers follow to draw 3D shapes, and the operations CAD modeling systems offer to create similar shapes. To overcome the strong ambiguity with parsing 2D sketches, we observe that in a sketching sequence, each step makes sense and can be interpreted in the \emph{context} of what has been drawn before. In our system, this context corresponds to a partial CAD model, inferred in the previous steps, which we feed along with the input sketch to a deep neural network in charge of interpreting how the model should be modified by that sketch. Our deep network architecture then recognizes the intended CAD operation and segments the sketch accordingly, such that a subsequent optimization estimates the parameters of the operation that best fit the segmented sketch strokes. Since there exists no datasets of paired sketching and CAD modeling sequences, we train our system by generating synthetic sequences of CAD operations that we render as line drawings. We present a proof of concept realization of our algorithm supporting four frequently used CAD operations. Using our system, participants are able to quickly model a large and diverse set of objects, demonstrating Sketch2CAD to be an alternate way of interacting with current CAD modeling systems.

Motivation & Objective

• To bridge the gap between sketch-based ideation and precise CAD modeling by aligning industrial design sketching workflows with standard CAD operations.
• To overcome ambiguity in 2D sketch interpretation by leveraging contextual understanding from prior modeling steps.
• To enable novice users to create accurate, editable CAD models through intuitive sketching, bypassing complex CAD software.
• To develop a system that supports common CAD operations (e.g., extrude, bevel, add/subtract, sweep) with high recognition accuracy and parameter fitting.
• To generate a synthetic dataset of paired sketch sequences and CAD operations for training a context-aware neural network.

Proposed method

• The system uses a deep neural network that takes both the current partial CAD model and the new sketch stroke as input to interpret the intended CAD operation.
• Each CAD operation is parameterized to map to specific sketch strokes (e.g., two parallel curves for a bevel, a closed loop for a subtract operation).
• The network performs instance segmentation of sketch strokes to identify components corresponding to operation parameters.
• A subsequent optimization step estimates the precise geometric parameters of the CAD operation that best fit the segmented strokes.
• The system is trained on a synthetic dataset generated by applying random sequences of CAD operations and rendering the results as line drawings.
• The architecture includes a U-Net-based segmentation head with multiple output maps (strokes, base, profile, offset, face) to guide operation recognition and fitting.

Experimental results

Research questions

• RQ1 Can a deep neural network accurately interpret freehand sketch strokes as specific CAD operations when conditioned on the current partial CAD model?
• RQ2 To what extent does contextual modeling—using prior sketching steps—improve the accuracy of CAD operation recognition compared to isolated stroke interpretation?
• RQ3 Can a system trained on synthetic data generalize to real-world sketching by novices to produce accurate, editable CAD models?
• RQ4 How well can the system recognize and parameterize complex operations like beveling, adding/subtracting parts, and sweeping across diverse object types?
• RQ5 Can the system support interactive, sequential modeling that mirrors the natural workflow of industrial designers?

Key findings

• The system achieves high accuracy in operation recognition, with 98.12% top-1 accuracy for extrude and 98.12% for bevel at level 1 (most confident predictions).
• For complex operations like add/subtract and sweep, the system maintains strong performance, with 93.54% and 95.57% top-1 accuracy at level 1, respectively.
• The network generalizes well to sketch perturbations, maintaining high performance even with noisy or imprecise strokes, as shown in qualitative results across three noise levels.
• User studies demonstrate that novices can efficiently model a wide variety of complex, diverse objects using only sketching, producing precise and editable CAD outputs.
• The system enables a novel workflow where CAD models can be procedurally edited and then re-rendered into sketches, supporting bidirectional design iteration.
• The synthetic training data generation pipeline successfully produces diverse, geometrically valid sketch-CAD pairs, enabling end-to-end training without real paired data.