Evaluation of 'Edge Computing' architectures for the inference of large language models

This project investigates the deployment of small language models on edge devices by systematically evaluating inference performance across CPU, GPU, and NPU backends, with the goal of understanding the trade-offs in latency, throughput, memory usage, and energy efficiency under realistic on-device constraints. It focuses on transformer-based models in the sub–few-billion parameter range and examines how different optimization strategies, including quantization, operator fusion, and backend-specific compilation pipelines, affect execution efficiency across heterogeneous hardware platforms commonly found in mobile and embedded systems. The study is designed to reflect practical edge conditions such as batch size one inference, memory bandwidth limitations, and runtime overhead introduced by model conversion toolchains. By comparing performance across architectures, the project aims to identify when general-purpose CPUs are sufficient, when GPUs provide advantages in flexibility or throughput, and when NPUs offer superior energy efficiency and latency for real-time applications. The outcome is intended to provide a consolidated set of empirical insights and deployment guidelines for selecting the most appropriate hardware backend depending on application constraints, while also highlighting key system-level bottlenecks that limit efficient edge inference of language models.