SkipSync: Accelerating Instruction Sampling for LLM Workload

CAI Luoshan; ZHOU Yaoyang; WANG Kaifan; LIU Tianyi; SUN Ninghui; BAO Yungang

doi:10.11999/JEIT260397

Article Contents

Article Navigation > Journal of Electronics & Information Technology > 2026 >

CAI Luoshan, ZHOU Yaoyang, WANG Kaifan, LIU Tianyi, SUN Ninghui, BAO Yungang. SkipSync: Accelerating Instruction Sampling for LLM Workload[J]. Journal of Electronics & Information Technology. doi: 10.11999/JEIT260397

Citation:

CAI Luoshan, ZHOU Yaoyang, WANG Kaifan, LIU Tianyi, SUN Ninghui, BAO Yungang. SkipSync: Accelerating Instruction Sampling for LLM Workload[J]. Journal of Electronics & Information Technology. doi: 10.11999/JEIT260397

Citation:

PDF( 3235 KB)

SkipSync: Accelerating Instruction Sampling for LLM Workload

doi: 10.11999/JEIT260397 cstr: 32379.14.JEIT260397

CAI Luoshan^{1, 2},
ZHOU Yaoyang³,
WANG Kaifan^{1, 2},
LIU Tianyi^{1, 2},
SUN Ninghui^{1, 2},
BAO Yungang^{1, 2
,
,}

1.
National Key Laboratory of Processor Chips, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China
2.
University of Chinese Academy of Sciences, Beijing 100049, China
3.
Beijing Institute of Open Source Chip, Beijing 100080, China

Funds: Supported by the Innovation Funding of ICT, CAS under Grant No. E661150, Beijing Natural Science Foundation – Xiaomi Joint Innovation Fund under Grant No. L2603004

Received Date: 2026-04-07
Accepted Date: 2026-07-28
Rev Recd Date: 2026-07-21

Available Online: 2026-08-07

Abstract

Abstract

Objective The rapid evolution of Large Language Models (LLMs) has significantly increased the demand for efficient design and evaluation of domain-specific accelerators. While sampling-based performance evaluation method effectively reduces the cost of cycle-accurate simulation, they rely on Instruction Set Simulators (ISS) to execute full workloads for profiling and checkpoint generation. In LLM scenarios, frequent changes in model structures, inference frameworks, and accelerator instruction sets invalidate checkpoint reuse and substantially increase the overhead of ISS-based functional simulation, which becomes the dominant bottleneck in the evaluation workflow. Existing ISS acceleration techniques, such as virtualization and dynamic binary translation (DBT), either require ISA compatibility or incur high implementation complexity, making them unsuitable for rapidly evolving LLM workloads. Therefore, a flexible and efficient ISS acceleration approach is required to support fast and accurate sampling-based evaluation for LLM accelerators. Methods This paper proposes SkipSync, an ISS acceleration method for LLM inference workload sampling. The key insight is that the control flow of most LLM operators is independent of intermediate tensor values and determined by static parameters. Therefore, SkipSync introduces Skip Execution to bypass the execution stage of accelerator instructions within selected operators while preserving instruction fetch and decode stages to maintain profiling correctness. To guarantee correct control flow of subsequent execution, a lightweight Host–Simulator Synchronization mechanism is further introduced to synchronize host-computed results back to the ISS. Lightweight synchronization primitives and custom instructions are designed to integrate SkipSync into existing sampling workflows with minimal implementation effort. Results and Discussions SkipSync is implemented on QEMU and supports RISC-V matrix and vector extensions as accelerator instructions. Experimental results show SkipSync significantly reduces ISS simulation overhead while preserving accuracy. Compared with the baseline, it achieves an average speedup of 4.64× in the Prefill stage and 5.51× in the Decode stage (Fig. 5), as it eliminates the dominant cost of vector and matrix instructions execution, which accounts for over 80% of runtime (Table 1). SkipSync outperforms DBT-based approaches (Fig. 6), while offering better extensibility to support new accelerator instructions. Notably, the synchronization mechanism introduces only limited overhead, adding approximately 8.5% additional runtime relative to ideal skip-only execution (Fig. 8). This confirms that the data synchronization does not negate the acceleration benefits. Meanwhile, SkipSync maintains high sampling fidelity, with an average error of approximately 2.55% and performance results nearly identical to baseline execution (Fig. 9). Conclusions This paper presents SkipSync, an extensible ISS acceleration method for sampling-based performance evaluation of LLM inference workloads. By combining Skip Execution and Host–Simulator Synchronization, SkipSync effectively alleviates the ISS bottleneck in LLM sampling workflows while preserving correctness and sampling fidelity. Experimental results demonstrate that SkipSync achieves significant speedup with low synchronization overhead and negligible impact on sampling accuracy. The proposed method provides a practical solution for efficient accelerator evaluation required by rapidly evolving LLM ecosystems. Future work will extend SkipSync to more complex LLM inference optimization scenarios and explore automatic identification of skippable operators to further improve usability.
- Large Language Model (LLM),
- Workload Sampling,
- Instruction Set Simulator

FullText(HTML)

References(25)

References

[1]	SHERWOOD T, PERELMAN E, HAMERLY G, et al. Automatically characterizing large scale program behavior[J]. ACM SIGPLAN Notices, 2002, 37(10): 45–57. doi: 10.1145/605432.605403.
[2]	WUNDERLICH R E, WENISCH T F, FALSAFI B, et al. SMARTS: Accelerating microarchitecture simulation via rigorous statistical sampling[C]. Proceedings of the 2003 30th Annual International Symposium on Computer Architecture (ISCA), San Diego, USA, 2003: 84–95. doi: 10.1109/ISCA.2003.1206991.
[3]	BELLARD F. QEMU, a fast and portable dynamic translator[C]. Proceedings of the Annual Conference on USENIX Annual Technical Conference, Anaheim, USA, 2005: 41–46.
[4]	BINKERT N, BECKMANN B, BLACK G, et al. The gem5 simulator[J]. ACM SIGARCH Computer Architecture News, 2011, 39(2): 1–7. doi: 10.1145/2024716.2024718.
[5]	SANDBERG A, NIKOLERIS N, CARLSON T E, et al. Full speed ahead: Detailed architectural simulation at near-native speed[C]. 2015 IEEE International Symposium on Workload Characterization (IISWC), Atlanta, USA, 2015: 183–192. doi: 10.1109/IISWC.2015.29.
[6]	FU Shengyu, HONG Dingyong, LIU Yuping, et al. Efficient and retargetable SIMD translation in a dynamic binary translator[J]. Software: Practice and Experience, 2018, 48(6): 1312–1330. doi: 10.1002/spe.2573.
[7]	JIANG Jinhu, LIANG Chaoyi, DONG Rongchao, et al. A system-level dynamic binary translator using automatically-learned translation rules[C]. 2024 IEEE/ACM International Symposium on Code Generation and Optimization (CGO), Edinburgh, UK, 2024: 423–434. doi: 10.1109/CGO57630.2024.10444850.
[8]	卢迪, 王振发. 脉动阵列协同层融合的卷积神经网络加速器设计[J]. 电子与信息学报, 2026, 48(3): 1280–1291. doi: 10.11999/JEIT250867. LU Di and WANG Zhenfa. Design of a CNN accelerator based on systolic array collaboration with inter-layer fusion[J]. Journal of Electronics & Information Technology, 2026, 48(3): 1280–1291. doi: 10.11999/JEIT250867.
[9]	李天阳, 张帆, 王松, 等. 基于FPGA的卷积神经网络和视觉Transformer通用加速器[J]. 电子与信息学报, 2024, 46(6): 2663–2672. doi: 10.11999/JEIT230713. LI Tianyang, ZHANG Fan, WANG Song, et al. FPGA-based unified accelerator for convolutional neural network and vision Transformer[J]. Journal of Electronics & Information Technology, 2024, 46(6): 2663–2672. doi: 10.11999/JEIT230713.
[10]	于斌, 闵玉新, 张自豪, 等. 基于RISC-V指令扩展的双线性对协处理器设计[J]. 电子与信息学报, 2025, 47(9): 3137–3145. doi: 10.11999/JEIT250367. YU Bin, MIN Yuxin, ZHANG Zihao, et al. Design of a bilinear pairing coprocessor based on RISC-V instruction extension[J]. Journal of Electronics & Information Technology, 2025, 47(9): 3137–3145. doi: 10.11999/JEIT250367.
[11]	KIM H, YE Gaohan, WANG Nachuan, et al. Exploiting Intel Advanced Matrix Extensions (AMX) for large language model inference[J]. IEEE Computer Architecture Letters, 2024, 23(1): 117–120. doi: 10.1109/LCA.2024.3397747.
[12]	ILIESCU D A and PETROGALLI F. Arm scalable vector extension and application to machine learning[R]. 2017. (查阅网上资料, 未找到本条文献报告编号信息, 请确认).
[13]	PARK D and EGGER B. Improving throughput-oriented LLM inference with CPU computations[C]. Proceedings of the 2024 International Conference on Parallel Architectures and Compilation Techniques (PACT), Long Beach, USA, 2024: 233–245. doi: 10.1145/3656019.3676949.
[14]	LUO Xinhao, LIU Zihan, ZHOU Yangjie, et al. ClusterFusion: Expanding operator fusion scope for LLM inference via cluster-level collective primitive[C]. 39th Conference on Neural Information Processing Systems, San Diago, USA, 2025.
[15]	SHAH J, BIKSHANDI G, ZHANG Ying, et al. Flashattention-3: Fast and accurate attention with asynchrony and low-precision[C]. Proceedings of the 38th International Conference on Neural Information Processing Systems, Vancouver, Canada, 2024: 2193.
[16]	LIU Zihan, LUO Xinhao, GUO Junxian, et al. VQ-LLM: High-performance code generation for vector quantization augmented LLM inference[C]. 2025 IEEE International Symposium on High Performance Computer Architecture (HPCA), Las Vegas, USA, 2025: 1496–1509. doi: 10.1109/HPCA61900.2025.00112.
[17]	HAN Chenji, LI Xinyu, XUE Feng, et al. MultiPoint: Enabling scalable pre-silicon performance evaluation for multi-task workloads[J]. BenchCouncil Transactions on Benchmarks, Standards and Evaluations, 2024, 4(3): 100189. doi: 10.1016/j.tbench.2025.100189.
[18]	LIU Changxi, SABU A, CHAUDHARI A, et al. Pac-Sim: Simulation of multi-threaded workloads using intelligent, live sampling[J]. ACM Transactions on Architecture and Code Optimization, 2024, 21(4): 81. doi: 10.1145/3680548.
[19]	LIU Changxi, SUN Yifan, and CARLSON T E. Photon: A fine-grained sampled simulation methodology for GPU workloads[C]. 2023 56th IEEE/ACM International Symposium on Microarchitecture (MICRO), Toronto, Canada, 2023: 1227–1241. doi: 10.1145/3613424.3623773.
[20]	LIU Yuping, HONG Dingyong, WU Jan-jan, et al. Exploiting SIMD asymmetry in ARM-to-x86 dynamic binary translation[J]. ACM Transactions on Architecture and Code Optimization, 2019, 16(1): 2. doi: 10.1145/3301488.
[21]	RISC-V International. The RISC-V vector extension, version 1.0[EB/OL]. https://github.com/riscv/riscv-v-spec. 2026. (查阅网上资料,未找到本条文献信息,请确认).
[22]	XUANTIE-RV. RISC-V matrix extension specification proposal V0.6. 0[EB/OL]. https://github.com/XUANTIE-RV/riscv-matrix-extension-spec, 2026.
[23]	LEVIATHAN Y, KALMAN M, and MATIAS Y. Fast inference from transformers via speculative decoding[C]. Proceedings of the 40th International Conference on Machine Learning (ICML), Honolulu, USA, 2023: 795.
[24]	QEMU. QEMU v9.0. 0[EB/OL]. https://github.com/qemu/qemu/releases/tag/v9.0.0, 2024.
[25]	GRYGIER A. Llama. cpp[EB/OL]. https://github.com/ggml-org/llama.cpp, 2026.