QuadPipe: A Low-Error and High-Throughput Softmax Accelerator for Large Model Inference on FPGA

YU Yuxin; ZHOU Haiyang; GUO Tao; WANG Shuo

doi:10.11999/JEIT260610

Article Contents

Article Navigation > Journal of Electronics & Information Technology > 2025 >

YU Yuxin, ZHOU Haiyang, GUO Tao, WANG Shuo. QuadPipe: A Low-Error and High-Throughput Softmax Accelerator for Large Model Inference on FPGA[J]. Journal of Electronics & Information Technology. doi: 10.11999/JEIT260610

Citation:

YU Yuxin, ZHOU Haiyang, GUO Tao, WANG Shuo. QuadPipe: A Low-Error and High-Throughput Softmax Accelerator for Large Model Inference on FPGA[J]. Journal of Electronics & Information Technology. doi: 10.11999/JEIT260610

Citation:

PDF( 1460 KB)

QuadPipe: A Low-Error and High-Throughput Softmax Accelerator for Large Model Inference on FPGA

doi: 10.11999/JEIT260610 cstr: 32379.14.JEIT260610

YU Yuxin^1
,,
ZHOU Haiyang^{1
,
,},
GUO Tao¹,
WANG Shuo^{1, 2}

1.
Beijing Microelectronics Technology Institute, Beijing 100076, China
2.
Northwestern Polytechnical University, Xi’an 710072, China

Received Date: 2026-05-15
Accepted Date: 2026-09-14
Rev Recd Date: 2026-08-31

Available Online: 2026-09-21

Abstract

Abstract

Objective With the large-scale deployment of large model inference in cloud and edge scenarios, FPGA has emerged as an important heterogeneous acceleration platform that complements GPU due to its reconfigurable computing capability, high resource efficiency, and low inference latency. However, efficient Softmax implementation on FPGAs still faces three major challenges: (1) rounding-error accumulation and numerical drift caused by long-sequence denominator summation, (2) significant utilization degradation of fixed-parallelism pipelines when processing misaligned tails of variable-length vectors, and (3) throughput limitations caused by the imbalance between memory bandwidth and computational density. Methods This paper proposes QuadPipe, a high-throughput Softmax hardware pipeline that integrates three complementary design strategies. (1) A Q48.47-format 96-bit extended-precision accumulator is introduced into the denominator summation path. This design reduces the upper bound of per-addition rounding error from O(2^–24) under FP32 to O(2^–47), thereby suppressing error propagation during long-sequence denominator accumulation. (2) A hardware-level boundary-agnostic processing mechanism based on negative-infinity padding is developed. By exploiting the property $ \mathrm{e}^{-\mathrm{\infty}}=0 $, the mechanism automatically inserts NEG_INF into invalid tail slots within the compute core, eliminating software-side pre-padding overhead for variable-length vectors. (3) An X-Y dual-dimensional batch command scheduler and a multi-level backpressure control loop are designed to coordinate throughput between the compute core and the AXI interface. The proposed cross-precision data path, consisting of INT8 input loading, FP32 computation, and FP32 output writeback, enables continuous batch processing of up to Y=65 536 vectors while reducing input bandwidth requirements to one quarter of those of the FP32 scheme. Results and Discussions The proposed QuadPipe accelerator is implemented on a Xilinx Kintex-7 325T development board. At 150 MHz, it achieves a 59-cycle pipeline depth, a steady-state throughput of 4 FP32 results per cycle, and a peak throughput of 600 M elements/s, while consuming only 4.72% of the device LUT resources and 8 DSPs (Table 1). The Figure Of Merit (FOM) reaches 1.14×10⁶ bit·s^–1·(LUT+FF)^–1 ( Table 1). Board-level measurements show that the Mean Absolute Error (MAE) remains within the 10^–11～10^–10 range across all tested sequence lengths (Table 2). At N=2048, the Q48.47 extended-precision accumulation reduces the denominator error by 87.59% compared with standard FP32 accumulation (Table 2). Compared with the software pre-padding scheme, the hardware boundary-agnostic processing mechanism improves the effective element throughput of misaligned vectors by up to 6.49% at N=1 025 and reduces batch-processing latency by up to 5.98% (Table 3). Conclusions The proposed QuadPipe accelerator integrates extended-precision accumulation, boundary-agnostic processing, and dual-dimensional batch scheduling within a unified pipeline framework to address three major challenges in FPGA-based Softmax acceleration: long-sequence numerical stability, variable-length input processing, and system throughput matching. The experimental results demonstrate that QuadPipe achieves low numerical error, efficient resource utilization, and reliable system-level integration, providing a feasible hardware solution for large model inference on FPGA platforms.
- Softmax,
- FPGA,
- Extended-precision accumulation,
- Boundary-agnostic processing,
- Backpressure flow control

FullText(HTML)

References(28)

References

[1]	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]. The 31st International Conference on Neural Information Processing Systems, Long Beach, USA, 2017: 5998–6008.
[2]	YUAN Bo. Efficient hardware architecture of softmax layer in deep neural network[C]. The 29th IEEE International System-on-Chip Conference, Seattle, USA, 2016: 323–326. doi: 10.1109/SOCC.2016.7905501.
[3]	WANG Meiqi, LU Siyuan, ZHU Danyang, et al. A high-speed and low-complexity architecture for softmax function in deep learning[C]. 2018 IEEE Asia Pacific Conference on Circuits and Systems, Chengdu, China, 2018: 223–226. doi: 10.1109/APCCAS.2018.8605654.
[4]	GAO Yue, LIU Weiqiang, and LOMBARDI F. Design and implementation of an approximate softmax layer for deep neural networks[C]. 2020 IEEE International Symposium on Circuits and Systems, Seville, Spain, 2020: 1–5. doi: 10.1109/ISCAS45731.2020.9180870.
[5]	KIM J, KIM S, CHOI K, et al. Hardware-efficient softmax architecture with bit-wise exponentiation and reciprocal calculation[J]. IEEE Transactions on Circuits and Systems I: Regular Papers, 2024, 71(10): 4574–4585. doi: 10.1109/TCSI.2024.3443270.
[6]	TARASOV I and POTEKHIN D. Calculation of activation functions in FPGA-based neuroprocessors using the CORDIC algorithm[C]. The 11th International Conference on High-Performance Computing Systems and Technologies in Scientific Research, Automation of Control and Production, Barnaul, Russia, 2022: 16–30. doi: 10.1007/978-3-030-94141-3_2.
[7]	SUDHA J, HANUMANTHARAJU M C, VENKATESWARULU V, et al. A novel method for computing exponential function using CORDIC algorithm[J]. Procedia Engineering, 2012, 30: 519–528. doi: 10.1016/j.proeng.2012.01.893.
[8]	STEVENS J R, VENKATESAN R, DAI S, et al. Softermax: Hardware/software co-design of an efficient softmax for transformers[C]. The 58th ACM/IEEE Design Automation Conference, San Francisco, USA, 2021: 469–474. doi: 10.1109/DAC18074.2021.9586134.
[9]	ZHANG Huan, ZHANG Yonggang, PENG Lele, et al. Base-2 softmax function: Suitability for training and efficient hardware implementation[J]. IEEE Transactions on Circuits and Systems I: Regular Papers, 2022, 69(9): 3605–3618. doi: 10.1109/TCSI.2022.3175534.
[10]	SPAGNOLO F, PERRI S, and CORSONELLO P. Aggressive approximation of the softmax function for power-efficient hardware implementations[J]. IEEE Transactions on Circuits and Systems II: Express Briefs, 2022, 69(3): 1652–1656. doi: 10.1109/TCSII.2021.3120495.
[11]	ZHU Danyang, LU Siyuan, WANG Meiqi, et al. Efficient precision-adjustable architecture for softmax function in deep learning[J]. IEEE Transactions on Circuits and Systems II: Express Briefs, 2020, 67(12): 3382–3386. doi: 10.1109/TCSII.2020.3002564.
[12]	XIA Tianhua and ZHANG Saiqian. Hyft: A reconfigurable softmax accelerator with hybrid numeric format for both training and inference[C]. The 29th ACM/IEEE International Symposium on Low Power Electronics and Design, Newport Beach, USA, 2024. doi: 10.1145/3665314.3670816.
[13]	WU Yuanchen, XIE Zhiheng, PAN Hongbing, et al. MBS: A high-precision approximation method for softmax and efficient hardware implementation[J]. IEEE Transactions on Circuits and Systems I: Regular Papers, 2025, 72(7): 3366–3375. doi: 10.1109/TCSI.2025.3559069.
[14]	CUI Zhanhao, MA Yuhao, CHENG Yuan, et al. TEA-SPS: A tiny and efficient architecture for softmax with parallelism and sparsity adaptability[J]. IEEE Transactions on Circuits and Systems I: Regular Papers, 2026, 73(2): 1245–1258. doi: 10.1109/TCSI.2025.3610210.
[15]	KOCA N A, DO A T, and CHANG C H. Hardware-efficient softmax approximation for self-attention networks[C]. 2023 IEEE International Symposium on Circuits and Systems, Monterey, USA, 2023. doi: 10.1109/ISCAS46773.2023.10181465.
[16]	DAO T, FU D Y, ERMON S, et al. FLASHATTENTION: Fast and memory-efficient exact attention with IO-awareness[C]. The 36th International Conference on Neural Information Processing Systems, New Orleans, USA, 2022: 1189.
[17]	YE Wenhua, ZHOU Xu, ZHOU J, et al. Accelerating attention mechanism on FPGAs based on efficient reconfigurable systolic array[J]. ACM Transactions on Embedded Computing Systems, 2023, 22(6): 93. doi: 10.1145/3549937.
[18]	KABIR E, KABIR A, DOWNEY A R J, et al. FAMOUS: Flexible accelerator for the attention mechanism of transformer on Ultrascale+ FPGAs[C]. 2024 International Conference on Field Programmable Technology, Sydney, Australia, 2024. doi: 10.1109/ICFPT64416.2024.11113430.
[19]	ISLAMOGLU G, SCHERER M, PAULIN G, et al. ITA: An energy-efficient attention and softmax accelerator for quantized transformers[C]. 2023 IEEE/ACM International Symposium on Low Power Electronics and Design, Vienna, Austria, 2023. doi: 10.1109/ISLPED58423.2023.10244348.
[20]	LI Tianyang, ZHANG Fan, FAN Xitian, et al. Unified accelerator for attention and convolution in inference based on FPGA[C]. 2023 IEEE International Symposium on Circuits and Systems, Monterey, USA, 2023. doi: 10.1109/ISCAS46773.2023.10182145.
[21]	李天阳, 张帆, 王松, 等. 基于FPGA的卷积神经网络和视觉Transformer通用加速器[J]. 电子与信息学报, 2024, 46(6): 2663–2672. doi: 10.11999/JEIT230713. LI Tianyang, ZHANG Fan, WANG Song, et al. FPGA-based unified accelerator for convolutional neural network and vision Transformer[J]. Journal of Electronics & Information Technology, 2024, 46(6): 2663–2672. doi: 10.11999/JEIT230713.
[22]	姜小波, 邓晗珂, 莫志杰, 等. 规则压缩模型和灵活架构的Transformer加速器设计[J]. 电子与信息学报, 2024, 46(3): 1079–1088. doi: 10.11999/JEIT230188. JIANG Xiaobo, DENG Hanke, MO Zhijie, et al. Design of Transformer accelerator with regular compression model and flexible architecture[J]. Journal of Electronics & Information Technology, 2024, 46(3): 1079–1088. doi: 10.11999/JEIT230188.
[23]	ZENG Shulin, LIU Jun, DAI Guohao, et al. FlightLLM: Efficient large language model inference with a complete mapping flow on FPGAs[C]. 2024 ACM/SIGDA International Symposium on Field Programmable Gate Arrays, Monterey, USA, 2024. doi: 10.1145/3626202.3637562.
[24]	CHEN Hongzheng, ZHANG Jiahao, DU Yixiao, et al. Understanding the potential of FPGA-based spatial acceleration for large language model inference[J]. ACM Transactions on Reconfigurable Technology and Systems, 2025, 18(1): 5. doi: 10.1145/3656177.
[25]	LI Cong, YIN Yihan, WU Xintong, et al. H²-LLM: Hardware-dataflow co-exploration for heterogeneous hybrid-bonding-based low-batch LLM inference[C]. The 52nd Annual International Symposium on Computer Architecture, Tokyo, Japan, 2025. doi: 10.1145/3695053.3731008.
[26]	王泽昊, 朱振华, 谢童欣, 等. 混合专家大语言模型的系统与架构优化技术综述[J]. 电子与信息学报, 2025, 47(11): 4055–4078. doi: 10.11999/JEIT250407. WANG Zehao, ZHU Zhenhua, XIE Tongxin, et al. A survey on system and architecture optimization techniques for mixture-of-experts large language models[J]. Journal of Electronics & Information Technology, 2025, 47(11): 4055–4078. doi: 10.11999/JEIT250407.
[27]	林志平, 肖亮, 陈宏毅, 等. 面向大语言模型的抗干扰协同推断技术[J]. 电子与信息学报, 2025, 47(11): 4572–4582. doi: 10.11999/JEIT250675. LIN Zhiping, XIAO Liang, CHEN Hongyi, et al. Collaborative inference for large language models against jamming attacks[J]. Journal of Electronics & Information Technology, 2025, 47(11): 4572–4582. doi: 10.11999/JEIT250675.
[28]	WANG Hui, CHENG Yuan, HAN Xiaomeng, et al. Pushing the limits of BFP on narrow precision LLM inference[C]. The 39th AAAI Conference on Artificial Intelligence, Philadelphia, USA, 2025: 21099–21107. doi: 10.1609/aaai.v39i20.35407.