SkipSync：面向大模型负载的指令采样加速方法

蔡洛姗; 周耀阳; 王凯帆; 刘天义; 孙凝晖; 包云岗

doi:10.11999/JEIT260397

SkipSync：面向大模型负载的指令采样加速方法

doi: 10.11999/JEIT260397 cstr: 32379.14.JEIT260397

蔡洛姗^{1, 2},
周耀阳³,
王凯帆^{1, 2},
刘天义^{1, 2},
孙凝晖^{1, 2},
包云岗^{1, 2, ,}

1.
中国科学院计算技术研究所处理器芯片全国重点实验室北京 100190
2.
中国科学院大学北京 100049
3.
北京开源芯片研究院北京 100080

基金项目: 中国科学院计算所创新课题 (E661150)，北京市自然科学基金-小米创新联合基金(L2603004)

详细信息

作者简介:
蔡洛姗：女，博士生，研究方向为开源处理器微架构、AI加速器芯片敏捷设计与评估

周耀阳：男，博士，研究方向为可扩展处理器设计、负载采样和性能评测方法

王凯帆：男，博士生，研究方向为计算机体系结构、开源处理器敏捷性能分析

刘天义：男，博士，副研究员，研究方向为计算机系统结构、开源处理器敏捷设计、云渲染和AI基础设施

孙凝晖：男，博士，中国工程院院士，CCF会士，研究方向为计算机系统结构、高性能计算

包云岗：男，博士，研究员，研究方向为数据中心体系结构、处理器芯片敏捷设计方法论、开源处理器芯片生态

通讯作者:
包云岗　baoyg@ict.ac.cn

中图分类号: TP391.9
计量
- 文章访问数: 60
- HTML全文浏览量: 34
- PDF下载量: 3
- 被引次数: 0
出版历程
- 收稿日期: 2026-04-07
- 修回日期: 2026-07-21
- 录用日期: 2026-07-28
- 网络出版日期: 2026-08-07

SkipSync: Accelerating Instruction Sampling for LLM Workload

CAI Luoshan^{1, 2},
ZHOU Yaoyang³,
WANG Kaifan^{1, 2},
LIU Tianyi^{1, 2},
SUN Ninghui^{1, 2},
BAO Yungang^{1, 2
, ,}

1.
National Key Laboratory of Processor Chips, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China
2.
University of Chinese Academy of Sciences, Beijing 100049, China
3.
Beijing Institute of Open Source Chip, Beijing 100080, China

Funds: Supported by the Innovation Funding of ICT, CAS under Grant No. E661150, Beijing Natural Science Foundation – Xiaomi Joint Innovation Fund under Grant No. L2603004

摘要

摘要: 指令采样是一种广泛应用于硬件设计评估的技术，能够在加快评估速度的同时保证评估精度。然而在大语言模型(LLM)推理场景下，模型结构、软件框架和专用加速器指令等软硬件技术快速演进，使得每一次加速器迭代评估时都需要在指令集模拟器(ISS)上重新执行完整的应用负载以获得最新指令流完成采样。此外，加速器指令在ISS中通常以解释执行的方式运行，导致指令采样时间长，因此ISS上的执行效率逐渐成为评估流程的主要瓶颈。针对上述问题，该文提出一种面向LLM推理负载的指令采样加速方法SkipSync。该方法基于如下观察：LLM算子内部的程序控制流通常与张量计算结果无关，因此可以在这类算子中跳过高开销的加速器指令执行过程，从而提升ISS的执行速度。同时为保障程序的后续执行与采样准确性，SkipSync通过轻量级数据同步机制用宿主机的正确计算结果校正模拟器上的运行状态。基于QEMU指令集模拟器的实验结果表明，在多种模型规模及推理阶段下，SkipSync可获得平均5.08倍的加速效果，显著提升了基于指令采样的LLM加速器性能评估效率，且不影响采样的准确性。另外，该方法无需针对新型加速器指令进行定制实现，具有良好的可扩展性，能够有效支撑快速演进背景下的LLM加速器性能评估与设计迭代。
- 大语言模型 /
- 采样评估方法 /
- 指令集模拟器
Abstract: Objective The rapid evolution of Large Language Models (LLMs) has significantly increased the demand for efficient design and evaluation of domain-specific accelerators. While sampling-based performance evaluation method effectively reduces the cost of cycle-accurate simulation, they rely on Instruction Set Simulators (ISS) to execute full workloads for profiling and checkpoint generation. In LLM scenarios, frequent changes in model structures, inference frameworks, and accelerator instruction sets invalidate checkpoint reuse and substantially increase the overhead of ISS-based functional simulation, which becomes the dominant bottleneck in the evaluation workflow. Existing ISS acceleration techniques, such as virtualization and dynamic binary translation (DBT), either require ISA compatibility or incur high implementation complexity, making them unsuitable for rapidly evolving LLM workloads. Therefore, a flexible and efficient ISS acceleration approach is required to support fast and accurate sampling-based evaluation for LLM accelerators. Methods This paper proposes SkipSync, an ISS acceleration method for LLM inference workload sampling. The key insight is that the control flow of most LLM operators is independent of intermediate tensor values and determined by static parameters. Therefore, SkipSync introduces Skip Execution to bypass the execution stage of accelerator instructions within selected operators while preserving instruction fetch and decode stages to maintain profiling correctness. To guarantee correct control flow of subsequent execution, a lightweight Host–Simulator Synchronization mechanism is further introduced to synchronize host-computed results back to the ISS. Lightweight synchronization primitives and custom instructions are designed to integrate SkipSync into existing sampling workflows with minimal implementation effort. Results and Discussions SkipSync is implemented on QEMU and supports RISC-V matrix and vector extensions as accelerator instructions. Experimental results show SkipSync significantly reduces ISS simulation overhead while preserving accuracy. Compared with the baseline, it achieves an average speedup of 4.64× in the Prefill stage and 5.51× in the Decode stage (Fig. 5), as it eliminates the dominant cost of vector and matrix instructions execution, which accounts for over 80% of runtime (Table 1). SkipSync outperforms DBT-based approaches (Fig. 6), while offering better extensibility to support new accelerator instructions. Notably, the synchronization mechanism introduces only limited overhead, adding approximately 8.5% additional runtime relative to ideal skip-only execution (Fig. 8). This confirms that the data synchronization does not negate the acceleration benefits. Meanwhile, SkipSync maintains high sampling fidelity, with an average error of approximately 2.55% and performance results nearly identical to baseline execution (Fig. 9). Conclusions This paper presents SkipSync, an extensible ISS acceleration method for sampling-based performance evaluation of LLM inference workloads. By combining Skip Execution and Host–Simulator Synchronization, SkipSync effectively alleviates the ISS bottleneck in LLM sampling workflows while preserving correctness and sampling fidelity. Experimental results demonstrate that SkipSync achieves significant speedup with low synchronization overhead and negligible impact on sampling accuracy. The proposed method provides a practical solution for efficient accelerator evaluation required by rapidly evolving LLM ecosystems. Future work will extend SkipSync to more complex LLM inference optimization scenarios and explore automatic identification of skippable operators to further improve usability.
- Large Language Model (LLM) /
- Workload Sampling /
- Instruction Set Simulator

HTML全文

图 1 LLM加速器设计：流程瓶颈从性能仿真转向功能仿真

下载: 全尺寸图片幻灯片

图 2 跳过指令执行的加速方法示意图

下载: 全尺寸图片幻灯片

图 3 主机-模拟器间同步流程

下载: 全尺寸图片幻灯片

图 4 SkipSync 采样加速流程总览

下载: 全尺寸图片幻灯片

图 5 SkipSync对比原始QEMU在RISC-V向量和矩阵扩展中的延迟与加速比

下载: 全尺寸图片幻灯片

图 6 用户态模式下SkipSync对比SIMD_DBT在ARM NEON扩展中的延迟与加速比

下载: 全尺寸图片幻灯片

图 7 全系统模式下SkipSync对比SIMD_DBT理想上界的加速比

下载: 全尺寸图片幻灯片

图 8 Skip vs Skip+Sync 归一化运行时间对比

下载: 全尺寸图片幻灯片

图 9 SimPoint 采样误差对比

下载: 全尺寸图片幻灯片

表 1 QEMU运行LLM负载的时间占比分析

操作类别		向量/矩阵访存	向量/矩阵计算	标量指令执行	其他开销
时间占比(%)	Prefill阶段	34.93	43.74	7.53	13.80
时间占比(%)	Decode阶段	68.55	13.40	4.02	14.03

下载: 导出CSV

表 2 LLM算子控制流特性分析

算子类型	是否控制流独立	原因说明
Matmul	√	仅由矩阵尺寸和分块方式决定
LayerNorm/RMSNorm	√	循环次数固定
Softmax	√	Reduction 范围由张量维度决定
SiLU/GELU/ReLU	√	逐元素计算范围固定
Add/Mul/Residual Add	√	逐元素计算范围固定
Top-k Sampling	×	Top-k选择由运行时数据决定
Dynamic Sparse Attention	×	需要根据实际数据确定非零元素位置

下载: 导出CSV

表 3 多核系统下SkipSync的加速比

核心数量	1	2	4	8	16
Prefill阶段加速比	4.43	4.51	4.45	4.68	4.59
Decode阶段加速比	5.48	5.63	5.26	5.66	5.46

下载: 导出CSV

参考文献(25)

[1]	SHERWOOD T, PERELMAN E, HAMERLY G, et al. Automatically characterizing large scale program behavior[J]. ACM SIGPLAN Notices, 2002, 37(10): 45–57. doi: 10.1145/605432.605403.
[2]	WUNDERLICH R E, WENISCH T F, FALSAFI B, et al. SMARTS: Accelerating microarchitecture simulation via rigorous statistical sampling[C]. Proceedings of the 2003 30th Annual International Symposium on Computer Architecture (ISCA), San Diego, USA, 2003: 84–95. doi: 10.1109/ISCA.2003.1206991.
[3]	BELLARD F. QEMU, a fast and portable dynamic translator[C]. Proceedings of the Annual Conference on USENIX Annual Technical Conference, Anaheim, USA, 2005: 41–46.
[4]	BINKERT N, BECKMANN B, BLACK G, et al. The gem5 simulator[J]. ACM SIGARCH Computer Architecture News, 2011, 39(2): 1–7. doi: 10.1145/2024716.2024718.
[5]	SANDBERG A, NIKOLERIS N, CARLSON T E, et al. Full speed ahead: Detailed architectural simulation at near-native speed[C]. 2015 IEEE International Symposium on Workload Characterization (IISWC), Atlanta, USA, 2015: 183–192. doi: 10.1109/IISWC.2015.29.
[6]	FU Shengyu, HONG Dingyong, LIU Yuping, et al. Efficient and retargetable SIMD translation in a dynamic binary translator[J]. Software: Practice and Experience, 2018, 48(6): 1312–1330. doi: 10.1002/spe.2573.
[7]	JIANG Jinhu, LIANG Chaoyi, DONG Rongchao, et al. A system-level dynamic binary translator using automatically-learned translation rules[C]. 2024 IEEE/ACM International Symposium on Code Generation and Optimization (CGO), Edinburgh, UK, 2024: 423–434. doi: 10.1109/CGO57630.2024.10444850.
[8]	卢迪, 王振发. 脉动阵列协同层融合的卷积神经网络加速器设计[J]. 电子与信息学报, 2026, 48(3): 1280–1291. doi: 10.11999/JEIT250867. LU Di and WANG Zhenfa. Design of a CNN accelerator based on systolic array collaboration with inter-layer fusion[J]. Journal of Electronics & Information Technology, 2026, 48(3): 1280–1291. doi: 10.11999/JEIT250867.
[9]	李天阳, 张帆, 王松, 等. 基于FPGA的卷积神经网络和视觉Transformer通用加速器[J]. 电子与信息学报, 2024, 46(6): 2663–2672. doi: 10.11999/JEIT230713. LI Tianyang, ZHANG Fan, WANG Song, et al. FPGA-based unified accelerator for convolutional neural network and vision Transformer[J]. Journal of Electronics & Information Technology, 2024, 46(6): 2663–2672. doi: 10.11999/JEIT230713.
[10]	于斌, 闵玉新, 张自豪, 等. 基于RISC-V指令扩展的双线性对协处理器设计[J]. 电子与信息学报, 2025, 47(9): 3137–3145. doi: 10.11999/JEIT250367. YU Bin, MIN Yuxin, ZHANG Zihao, et al. Design of a bilinear pairing coprocessor based on RISC-V instruction extension[J]. Journal of Electronics & Information Technology, 2025, 47(9): 3137–3145. doi: 10.11999/JEIT250367.
[11]	KIM H, YE Gaohan, WANG Nachuan, et al. Exploiting Intel Advanced Matrix Extensions (AMX) for large language model inference[J]. IEEE Computer Architecture Letters, 2024, 23(1): 117–120. doi: 10.1109/LCA.2024.3397747.
[12]	ILIESCU D A and PETROGALLI F. Arm scalable vector extension and application to machine learning[R]. 2017. (查阅网上资料, 未找到本条文献报告编号信息, 请确认).
[13]	PARK D and EGGER B. Improving throughput-oriented LLM inference with CPU computations[C]. Proceedings of the 2024 International Conference on Parallel Architectures and Compilation Techniques (PACT), Long Beach, USA, 2024: 233–245. doi: 10.1145/3656019.3676949.
[14]	LUO Xinhao, LIU Zihan, ZHOU Yangjie, et al. ClusterFusion: Expanding operator fusion scope for LLM inference via cluster-level collective primitive[C]. 39th Conference on Neural Information Processing Systems, San Diago, USA, 2025.
[15]	SHAH J, BIKSHANDI G, ZHANG Ying, et al. Flashattention-3: Fast and accurate attention with asynchrony and low-precision[C]. Proceedings of the 38th International Conference on Neural Information Processing Systems, Vancouver, Canada, 2024: 2193.
[16]	LIU Zihan, LUO Xinhao, GUO Junxian, et al. VQ-LLM: High-performance code generation for vector quantization augmented LLM inference[C]. 2025 IEEE International Symposium on High Performance Computer Architecture (HPCA), Las Vegas, USA, 2025: 1496–1509. doi: 10.1109/HPCA61900.2025.00112.
[17]	HAN Chenji, LI Xinyu, XUE Feng, et al. MultiPoint: Enabling scalable pre-silicon performance evaluation for multi-task workloads[J]. BenchCouncil Transactions on Benchmarks, Standards and Evaluations, 2024, 4(3): 100189. doi: 10.1016/j.tbench.2025.100189.
[18]	LIU Changxi, SABU A, CHAUDHARI A, et al. Pac-Sim: Simulation of multi-threaded workloads using intelligent, live sampling[J]. ACM Transactions on Architecture and Code Optimization, 2024, 21(4): 81. doi: 10.1145/3680548.
[19]	LIU Changxi, SUN Yifan, and CARLSON T E. Photon: A fine-grained sampled simulation methodology for GPU workloads[C]. 2023 56th IEEE/ACM International Symposium on Microarchitecture (MICRO), Toronto, Canada, 2023: 1227–1241. doi: 10.1145/3613424.3623773.
[20]	LIU Yuping, HONG Dingyong, WU Jan-jan, et al. Exploiting SIMD asymmetry in ARM-to-x86 dynamic binary translation[J]. ACM Transactions on Architecture and Code Optimization, 2019, 16(1): 2. doi: 10.1145/3301488.
[21]	RISC-V International. The RISC-V vector extension, version 1.0[EB/OL]. https://github.com/riscv/riscv-v-spec. 2026. (查阅网上资料,未找到本条文献信息,请确认).
[22]	XUANTIE-RV. RISC-V matrix extension specification proposal V0.6. 0[EB/OL]. https://github.com/XUANTIE-RV/riscv-matrix-extension-spec, 2026.
[23]	LEVIATHAN Y, KALMAN M, and MATIAS Y. Fast inference from transformers via speculative decoding[C]. Proceedings of the 40th International Conference on Machine Learning (ICML), Honolulu, USA, 2023: 795.
[24]	QEMU. QEMU v9.0. 0[EB/OL]. https://github.com/qemu/qemu/releases/tag/v9.0.0, 2024.
[25]	GRYGIER A. Llama. cpp[EB/OL]. https://github.com/ggml-org/llama.cpp, 2026.