Cheerful: Hardware-Oriented Early Exits for Ubiquitous Computing

Variations in the computational capabilities of edge devices and runtime resource availability pose considerable challenges in developing efficient ubiquitous inference. Dynamic Neural Networks (DyNNs) have been developed to address device heterogeneity by adapting network structures or parameters to complex deployment environments. However, existing DyNN methods are largely insensitive to runtime resource variations and hardware-specific configurations, and their reliance on extensive profiling complicates deployment. In this work, we present Cheerful, a hardware-oriented, ubiquitous early exit framework that enables efficient inference on heterogeneous devices by combining hardware-parameterized performance modeling and optimization with online adaptation to resource variations. In the offline stage, Cheerful models key performance metrics by integrating comprehensive analytical and data-driven methods, formulates an optimization problem to achieve the trade-off between inference accuracy and energy cost, and incorporates hardware-specific parameters to accommodate device heterogeneity. In the online stage, Cheerful integrates a lightweight hardware-aware inference engine (HIE) that jointly adjusts early-exit configurations and CPU frequency based on run-time resource availability, using Dynamic Voltage and Frequency Scaling (DVFS). Extensive experiments on heterogeneous devices demonstrate that Cheerful outperforms state-of-the-art methods on the accuracy-energy Pareto frontier in resource-constrained environments while maintaining low deployment overhead.