[논문 리뷰] SpArSe: Sparse Architecture Search for CNNs on Resource-Constrained Microcontrollers

Igor Fedorov, Ryan P. Adams외 2명|arXiv (Cornell University)|2019. 05. 28.

Advanced Neural Network Applications참고 문헌 44인용 수 82

한 줄 요약

SpArSe는 기억 제한이 있는 마이크로컨트롤러에 맞는 CNN을 자동으로 설계하기 위해 신경망 아키텍처 검색과 가지치기를 결합하여 정확도를 유지하면서 새로운 MCU 방법들보다 더 작은 모델을 최대 4.35배까지 만들고 더 높은 정확도를 달성합니다.

ABSTRACT

The vast majority of processors in the world are actually microcontroller units (MCUs), which find widespread use performing simple control tasks in applications ranging from automobiles to medical devices and office equipment. The Internet of Things (IoT) promises to inject machine learning into many of these every-day objects via tiny, cheap MCUs. However, these resource-impoverished hardware platforms severely limit the complexity of machine learning models that can be deployed. For example, although convolutional neural networks (CNNs) achieve state-of-the-art results on many visual recognition tasks, CNN inference on MCUs is challenging due to severe finite memory limitations. To circumvent the memory challenge associated with CNNs, various alternatives have been proposed that do fit within the memory budget of an MCU, albeit at the cost of prediction accuracy. This paper challenges the idea that CNNs are not suitable for deployment on MCUs. We demonstrate that it is possible to automatically design CNNs which generalize well, while also being small enough to fit onto memory-limited MCUs. Our Sparse Architecture Search method combines neural architecture search with pruning in a single, unified approach, which learns superior models on four popular IoT datasets. The CNNs we find are more accurate and up to $4.35 imes$ smaller than previous approaches, while meeting the strict MCU working memory constraint.

연구 동기 및 목표

• 메모리 한정 MCU에서 CNN을 배치하도록 동기를 부여하고 엄격한 RAM/ROM 제약(C1, C2)을 다룹니다.
• 메모리 한계를 충족하기 위해 아키텍처를 탐색하고 가중치를 가지치기하는 통합 프레임워크(SpArSe)를 개발합니다.
• 검증 정확도, 모델 크기, 작동 메모리의 균형을 맞추기 위한 다중 목표 최적화를 시연합니다.
• 아키텍처 탐색과 함께 가지치기가 여러 데이터셋에서 더 작고 더 정확한 MCU CNN을 만들어낸다는 것을 보여 줍니다.

제안 방법

• 아키텍처, 연산, 가중치, 훈련/가지치기 하이퍼파라미터를 나타내는 다중 목표 설계 공간 Omega = {alpha, theta, omega, theta}를 정의합니다.
• 목표를 공식화합니다: f1 = 1 - ValidationAccuracy, f2 = ModelSize (omega), f3 = max over layers of WorkingMemory(Omega).
• 정확도, 크기 및 메모리의 균형을 맞추는 구성 Omega*를 식별하기 위해 파레토 프런티어 접근법을 사용합니다.
• 탐색 공간은 가변 깊이/너비, 컨볼루션 타입, 다운샘플링, 잔차, 다중 스케일 출력 및 훈련/가지치기 하이퍼파라미터를 포함합니다.
• Sparse Variational Dropout (unstructured)와 Bayesian Compression (structured)를 통해 omega와 층 입력을 줄이기 위한 가지치기를 적용합니다.
• 무작위 스칼라화가 있는 3단계 다중 목표 베이지안 옵티마이저(MOBO)를 사용하여 구성을 샘플링하고 형태 변환(network morphism)을 통해 모프 간에 학습된 가중치를 재사용합니다.
• 최적화의 일부로 가지치기를 활용하여 고성능 서브그래프를 드러내고 수렴 속도를 가속화합니다.

실험 결과

연구 질문

• RQ1 Can CNNs be designed to meet strict MCU memory constraints (RAM/ROM) while maintaining competitive accuracy?
• RQ2 Does jointly optimizing architecture and weight sparsity via SpArSe outperform pruning or NAS in isolation for MCU deployment?
• RQ3 What are the effects of memory-oriented objectives (ModelSize and WorkingMemory) on resulting architectures and performance?
• RQ4 How does pruning influence the relationship between network edges and nonzero weights in ultra-small CNNs?

주요 결과

• SpArSe discovers CNNs that are more accurate and smaller than previous MCU-specific methods across MNIST, CIFAR10-binary, CUReT-binary, and Chars4k-binary.
• When optimized for memory footprint, SpArSe yields models that exceed Bonsai in accuracy while using less memory, and can achieve working memory well below 2 KB in some cases.
• Pruning significantly reduces parameter counts (e.g., up to ~80x fewer parameters) and enables uncovering high-performing subgraphs not evident from edge count alone.
• Structured pruning and quantization enable memory measurements (MS and WM) to be realistic for MCU deployment; SpArSe-produced models outperform Bonsai on most datasets under WM/ MS constraints.
• Network morphism and staged search considerably accelerate convergence, with pruning providing substantial gains in search efficiency and final model quality.