T2C: A Multi-User System for Deploying DNNs in a Thing-to-Cloud Continuum

Abstract

The importance of IoT analytics in smart deploy-ments has resulted in an increased use of powerful Deep Neural Network (DNN) models to extract insights from the growing amount of IoT sensor data. Traditional approaches that entirely offload computation and model deployment to cloud servers have been shown to be inefficient due to network congestion and latency concerns. However, with the improved capabilities of IoT devices, it has now become possible to distribute and host DNNs across IoT devices, edge servers and the cloud. In this paper, we propose a multi-user system, called T2C, to dynamically choose, deploy, monitor and control DNN-driven IoT analytics in a thing-to-cloud continuum. T2C leverages strategies such as multi-task learning, hitchhiking, early exit, and dynamic reconfiguration, to maximize the number of served user requests while simultaneously satisfying accuracy and latency requirements. We propose a suite of deployment planning and reconfiguration algorithms to dynamically deploy and migrate DNN layers between IoT devices, edge servers, and the cloud. We implement T2C in a prototype testbed and show that our system: (i) achieves 6.8X throughput boost compared to baseline algorithms in the planning phase, and (ii) improves the satisfied ratio by up to 35% in the operation and reconfiguration phase.