HyPIM: LLM Acceleration with A Hybrid ReRAM/SRAM 3D-PIM Architecture

Transformer-based Large Language Models (LLMs) have made significant progress in the fields of computer vision and natural language processing. The use of LLMs in embedded and real-time systems to provide more powerful intelligence capabilities is also an inevitable trend. However, the traditional von Neumann architecture encounters great challenges with transformer-based LLMs due to huge data movement overheads. Processing-in-Memory (PIM) emerges as a great potential solution by significantly closing the gap between computation and storage. Nevertheless, transformer-based LLMs usually consist of multiple transformer blocks, and the operations within each transformer block vary significantly on computation and memory access characteristics. Existing PIM designs that rely solely on SRAM, DRAM, or ReRAM cannot adapt well to transformer-based LLMs. To address these challenges, we introduce HyPIM, a 3D hybrid PIM architecture that integrates SRAM and ReRAM slices to process through their cooperation. However, HyPIM does not change the scenario where the inference latency of the attention mechanism continues to dominate the overall inference latency in the transformer-based LLMs when dealing with long sequences of data. Exploiting these insights, we then design a hardware-friendly early termination approximation algorithm and a sparse matrix multiplication strategy for the HyPIM architecture, reducing the overhead through a software and hardware co-design. Compared with the state-of-the-art Newton, TransPIM, HAIMA, HARDSEA, and H3DAtten architectures, the proposed HyPIM architecture achieves up to a 1.27×–1.67× latency improvement.