* [PATCH v3 0/3] LoongArch: BPF: Refine move_imm() and optimize 32-bit operations
@ 2026-10-16 8:10 Tiezhu Yang
2026-10-16 8:10 ` [PATCH v3 1/3] LoongArch: Add CPUCFG2 DIV32 feature detection Tiezhu Yang
` (3 more replies)
0 siblings, 4 replies; 15+ messages in thread
From: Tiezhu Yang @ 2026-10-16 8:10 UTC (permalink / raw)
To: Huacai Chen, Hengqi Chen; +Cc: loongarch, bpf, linux-kernel
v3:
-- Add a prerequisite patch to detect CPUCFG2_DIV32 feature.
-- Fix a hardware compatibility bug by checking cpu_has_div32
before emitting native 32-bit div/mod instructions.
This version is based on v7.3-rc7 and has been tested on Loongson-3A6000
and Loongson-3A5000 physical machines.
All of the testcases of test_bpf passed on Loongson-3A6000 (with DIV32)
and Loongson-3A5000 (without DIV32), here are the test results:
$ sudo modprobe test_bpf
$ dmesg -t | grep Summary
test_bpf: Summary: 1061 PASSED, 0 FAILED, [1049/1049 JIT'ed]
test_bpf: test_tail_calls: Summary: 10 PASSED, 0 FAILED, [10/10 JIT'ed]
test_bpf: test_skb_segment: Summary: 2 PASSED, 0 FAILED
For the BPF selftests test_verifier and test_progs, there are no any
regressions.
Tiezhu Yang (3):
LoongArch: Add CPUCFG2 DIV32 feature detection
LoongArch: BPF: Clean up and decouple is32 parameter from move_imm()
LoongArch: BPF: Optimize 32-bit operations using native instructions
arch/loongarch/include/asm/cpu-features.h | 1 +
arch/loongarch/include/asm/cpu.h | 2 +
arch/loongarch/include/asm/inst.h | 6 +
arch/loongarch/include/uapi/asm/hwcap.h | 1 +
arch/loongarch/kernel/cpu-probe.c | 4 +
arch/loongarch/kernel/proc.c | 2 +
arch/loongarch/net/bpf_jit.c | 381 ++++++++++++----------
arch/loongarch/net/bpf_jit.h | 25 +-
8 files changed, 234 insertions(+), 188 deletions(-)
--
2.42.0
^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH v3 1/3] LoongArch: Add CPUCFG2 DIV32 feature detection
2026-10-16 8:10 [PATCH v3 0/3] LoongArch: BPF: Refine move_imm() and optimize 32-bit operations Tiezhu Yang
@ 2026-10-16 8:10 ` Tiezhu Yang
2026-10-20 10:48 ` Huacai Chen
2026-10-16 8:10 ` [PATCH v3 2/3] LoongArch: BPF: Clean up and decouple is32 parameter from move_imm() Tiezhu Yang
` (2 subsequent siblings)
3 siblings, 1 reply; 15+ messages in thread
From: Tiezhu Yang @ 2026-10-16 8:10 UTC (permalink / raw)
To: Huacai Chen, Hengqi Chen; +Cc: loongarch, bpf, linux-kernel
The LoongArch Architecture Manual defines a hardware capability bit
CPUCFG2_DIV32 (bit 26 of CPUCFG 0x2). When this bit is set to 1, it
indicates that on 64-bit machines, the division and mod instructions
(DIV.W[U] and MOD.W[U]) will ignore the upper 32 bits of the input
registers and only compute based on the lower 32 bits.
On certain CPU cores without this feature, these 32-bit instructions
require both input operands to be properly sign-extended. Otherwise,
the hardware execution results can be randomly meaningless values.
Add CPU_FEATURE_DIV32 to detect and export this hardware capability
via elf_hwcap and cpu_has_div32. This provides the infrastructure
for subsystems like BPF JIT to safely choose either native 32-bit
instructions or fallback instruction sequences based on the actual
CPU hardware capabilities.
Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
---
arch/loongarch/include/asm/cpu-features.h | 1 +
arch/loongarch/include/asm/cpu.h | 2 ++
arch/loongarch/include/uapi/asm/hwcap.h | 1 +
arch/loongarch/kernel/cpu-probe.c | 4 ++++
arch/loongarch/kernel/proc.c | 2 ++
5 files changed, 10 insertions(+)
diff --git a/arch/loongarch/include/asm/cpu-features.h b/arch/loongarch/include/asm/cpu-features.h
index 62059c5551b9..3e7cf1987171 100644
--- a/arch/loongarch/include/asm/cpu-features.h
+++ b/arch/loongarch/include/asm/cpu-features.h
@@ -69,5 +69,6 @@
#define cpu_has_msgint cpu_opt(LOONGARCH_CPU_MSGINT)
#define cpu_has_avecint cpu_opt(LOONGARCH_CPU_AVECINT)
#define cpu_has_redirectint cpu_opt(LOONGARCH_CPU_REDIRECTINT)
+#define cpu_has_div32 cpu_opt(LOONGARCH_CPU_DIV32)
#endif /* __ASM_CPU_FEATURES_H */
diff --git a/arch/loongarch/include/asm/cpu.h b/arch/loongarch/include/asm/cpu.h
index 91b96938861e..d1cf5dd5eceb 100644
--- a/arch/loongarch/include/asm/cpu.h
+++ b/arch/loongarch/include/asm/cpu.h
@@ -127,6 +127,7 @@ static inline char *id_to_core_name(unsigned int id)
#define CPU_FEATURE_MSGINT 31 /* CPU has MSG interrupt */
#define CPU_FEATURE_AVECINT 32 /* CPU has AVEC interrupt */
#define CPU_FEATURE_REDIRECTINT 33 /* CPU has interrupt remapping */
+#define CPU_FEATURE_DIV32 34 /* CPU has DIV32 feature */
#define LOONGARCH_CPU_CPUCFG BIT_ULL(CPU_FEATURE_CPUCFG)
#define LOONGARCH_CPU_LAM BIT_ULL(CPU_FEATURE_LAM)
@@ -162,5 +163,6 @@ static inline char *id_to_core_name(unsigned int id)
#define LOONGARCH_CPU_MSGINT BIT_ULL(CPU_FEATURE_MSGINT)
#define LOONGARCH_CPU_AVECINT BIT_ULL(CPU_FEATURE_AVECINT)
#define LOONGARCH_CPU_REDIRECTINT BIT_ULL(CPU_FEATURE_REDIRECTINT)
+#define LOONGARCH_CPU_DIV32 BIT_ULL(CPU_FEATURE_DIV32)
#endif /* _ASM_CPU_H */
diff --git a/arch/loongarch/include/uapi/asm/hwcap.h b/arch/loongarch/include/uapi/asm/hwcap.h
index 90e96113ba51..00bf93683c74 100644
--- a/arch/loongarch/include/uapi/asm/hwcap.h
+++ b/arch/loongarch/include/uapi/asm/hwcap.h
@@ -20,5 +20,6 @@
#define HWCAP_LOONGARCH_LSPW (1 << 14)
#define HWCAP_LOONGARCH_SCQ (1 << 15)
#define HWCAP_LOONGARCH_LAM_BH (1 << 16)
+#define HWCAP_LOONGARCH_DIV32 (1 << 17)
#endif /* _UAPI_ASM_HWCAP_H */
diff --git a/arch/loongarch/kernel/cpu-probe.c b/arch/loongarch/kernel/cpu-probe.c
index 74d31f260dfd..c50fcba805f7 100644
--- a/arch/loongarch/kernel/cpu-probe.c
+++ b/arch/loongarch/kernel/cpu-probe.c
@@ -236,6 +236,10 @@ static void cpu_probe_common(struct cpuinfo_loongarch *c)
elf_hwcap |= HWCAP_LOONGARCH_LBT_MIPS;
}
#endif
+ if (config & CPUCFG2_DIV32) {
+ c->options |= LOONGARCH_CPU_DIV32;
+ elf_hwcap |= HWCAP_LOONGARCH_DIV32;
+ }
config = read_cpucfg(LOONGARCH_CPUCFG6);
if (config & CPUCFG6_PMP)
diff --git a/arch/loongarch/kernel/proc.c b/arch/loongarch/kernel/proc.c
index d4ce5b585453..c8d6f397de55 100644
--- a/arch/loongarch/kernel/proc.c
+++ b/arch/loongarch/kernel/proc.c
@@ -94,6 +94,8 @@ static int show_cpuinfo(struct seq_file *m, void *v)
seq_puts(m, " lbt_arm");
if (cpu_has_lbt_mips)
seq_puts(m, " lbt_mips");
+ if (cpu_has_div32)
+ seq_puts(m, " div32");
seq_puts(m, "\n");
seq_printf(m, "Hardware Watchpoint\t: %s", str_yes_no(cpu_has_watch));
--
2.42.0
^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH v3 2/3] LoongArch: BPF: Clean up and decouple is32 parameter from move_imm()
2026-10-16 8:10 [PATCH v3 0/3] LoongArch: BPF: Refine move_imm() and optimize 32-bit operations Tiezhu Yang
2026-10-16 8:10 ` [PATCH v3 1/3] LoongArch: Add CPUCFG2 DIV32 feature detection Tiezhu Yang
@ 2026-10-16 8:10 ` Tiezhu Yang
2026-10-27 1:36 ` Hengqi Chen
2026-10-16 8:10 ` [PATCH v3 3/3] LoongArch: BPF: Optimize 32-bit operations using native instructions Tiezhu Yang
2026-10-22 7:43 ` [PATCH v3 0/3] LoongArch: BPF: Refine move_imm() and optimize 32-bit operations Tiezhu Yang
3 siblings, 1 reply; 15+ messages in thread
From: Tiezhu Yang @ 2026-10-16 8:10 UTC (permalink / raw)
To: Huacai Chen, Hengqi Chen; +Cc: loongarch, bpf, linux-kernel
The current move_imm() relies on the 'is32' parameter for ALU32 and JMP32,
forcing the helper to internally clear the upper 32 bits. However, this is
redundant for 64-bit immediate loads. Furthermore, it is even undesirable
for load/store signed address offset computations.
Clean up move_imm() by removing the 'is32' parameter which allows it to
focus on loading values, while moving the zero-extension logic outward to
the explicit callers. For 64-bit and load-store operations, just drop the
parameter.
This is preparation for later patch, no functional changes.
Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
---
arch/loongarch/net/bpf_jit.c | 102 +++++++++++++++++++----------------
arch/loongarch/net/bpf_jit.h | 7 +--
2 files changed, 58 insertions(+), 51 deletions(-)
diff --git a/arch/loongarch/net/bpf_jit.c b/arch/loongarch/net/bpf_jit.c
index e78eb582c300..77cd96a6f84c 100644
--- a/arch/loongarch/net/bpf_jit.c
+++ b/arch/loongarch/net/bpf_jit.c
@@ -194,7 +194,7 @@ static void build_prologue(struct jit_ctx *ctx)
ctx->stack_size = stack_adjust;
if (ctx->arena_vm_start)
- move_imm(ctx, REG_ARENA, ctx->arena_vm_start, false);
+ move_imm(ctx, REG_ARENA, ctx->arena_vm_start);
}
static void __build_epilogue(struct jit_ctx *ctx, bool is_tail_call)
@@ -339,7 +339,7 @@ static int emit_bpf_tail_call(struct jit_ctx *ctx, int insn)
static void emit_store_stack_imm64(struct jit_ctx *ctx, int reg, int stack_off, u64 imm64)
{
- move_imm(ctx, reg, imm64, false);
+ move_imm(ctx, reg, imm64);
emit_insn(ctx, std, reg, LOONGARCH_GPR_FP, stack_off);
}
@@ -355,7 +355,7 @@ static int emit_atomic_rmw(const struct bpf_insn *insn, struct jit_ctx *ctx)
const s32 imm = insn->imm;
const bool isdw = BPF_SIZE(insn->code) == BPF_DW;
- move_imm(ctx, t1, off, false);
+ move_imm(ctx, t1, off);
emit_insn(ctx, addd, t1, dst, t1);
move_reg(ctx, t3, src);
@@ -528,7 +528,7 @@ static int emit_atomic_ld_st(const struct bpf_insn *insn, struct jit_ctx *ctx)
if (is_signed_imm12(off)) {
emit_insn(ctx, ldbu, dst, src, off);
} else {
- move_imm(ctx, t1, off, false);
+ move_imm(ctx, t1, off);
emit_insn(ctx, ldxbu, dst, src, t1);
}
break;
@@ -536,7 +536,7 @@ static int emit_atomic_ld_st(const struct bpf_insn *insn, struct jit_ctx *ctx)
if (is_signed_imm12(off)) {
emit_insn(ctx, ldhu, dst, src, off);
} else {
- move_imm(ctx, t1, off, false);
+ move_imm(ctx, t1, off);
emit_insn(ctx, ldxhu, dst, src, t1);
}
break;
@@ -544,7 +544,7 @@ static int emit_atomic_ld_st(const struct bpf_insn *insn, struct jit_ctx *ctx)
if (is_signed_imm12(off)) {
emit_insn(ctx, ldwu, dst, src, off);
} else {
- move_imm(ctx, t1, off, false);
+ move_imm(ctx, t1, off);
emit_insn(ctx, ldxwu, dst, src, t1);
}
break;
@@ -552,7 +552,7 @@ static int emit_atomic_ld_st(const struct bpf_insn *insn, struct jit_ctx *ctx)
if (is_signed_imm12(off)) {
emit_insn(ctx, ldd, dst, src, off);
} else {
- move_imm(ctx, t1, off, false);
+ move_imm(ctx, t1, off);
emit_insn(ctx, ldxd, dst, src, t1);
}
break;
@@ -567,7 +567,7 @@ static int emit_atomic_ld_st(const struct bpf_insn *insn, struct jit_ctx *ctx)
if (is_signed_imm12(off)) {
emit_insn(ctx, stb, src, dst, off);
} else {
- move_imm(ctx, t1, off, false);
+ move_imm(ctx, t1, off);
emit_insn(ctx, stxb, src, dst, t1);
}
break;
@@ -575,7 +575,7 @@ static int emit_atomic_ld_st(const struct bpf_insn *insn, struct jit_ctx *ctx)
if (is_signed_imm12(off)) {
emit_insn(ctx, sth, src, dst, off);
} else {
- move_imm(ctx, t1, off, false);
+ move_imm(ctx, t1, off);
emit_insn(ctx, stxh, src, dst, t1);
}
break;
@@ -583,7 +583,7 @@ static int emit_atomic_ld_st(const struct bpf_insn *insn, struct jit_ctx *ctx)
if (is_signed_imm12(off)) {
emit_insn(ctx, stw, src, dst, off);
} else {
- move_imm(ctx, t1, off, false);
+ move_imm(ctx, t1, off);
emit_insn(ctx, stxw, src, dst, t1);
}
break;
@@ -591,7 +591,7 @@ static int emit_atomic_ld_st(const struct bpf_insn *insn, struct jit_ctx *ctx)
if (is_signed_imm12(off)) {
emit_insn(ctx, std, src, dst, off);
} else {
- move_imm(ctx, t1, off, false);
+ move_imm(ctx, t1, off);
emit_insn(ctx, stxd, src, dst, t1);
}
break;
@@ -716,7 +716,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
if (insn_is_cast_user(insn)) {
move_reg(ctx, t1, src);
emit_zext_32(ctx, t1, true);
- move_imm(ctx, dst, (ctx->user_vm_start >> 32) << 32, false);
+ move_imm(ctx, dst, (ctx->user_vm_start >> 32) << 32);
emit_insn(ctx, beq, t1, LOONGARCH_GPR_ZERO, 2);
emit_insn(ctx, or, t1, dst, t1);
move_reg(ctx, dst, t1);
@@ -753,7 +753,8 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
/* dst = imm */
case BPF_ALU | BPF_MOV | BPF_K:
case BPF_ALU64 | BPF_MOV | BPF_K:
- move_imm(ctx, dst, imm, is32);
+ move_imm(ctx, dst, imm);
+ emit_zext_32(ctx, dst, is32);
break;
/* dst = dst + src */
@@ -769,7 +770,8 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
if (is_signed_imm12(imm)) {
emit_insn(ctx, addid, dst, dst, imm);
} else {
- move_imm(ctx, t1, imm, is32);
+ move_imm(ctx, t1, imm);
+ emit_zext_32(ctx, t1, is32);
emit_insn(ctx, addd, dst, dst, t1);
}
emit_zext_32(ctx, dst, is32);
@@ -788,7 +790,8 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
if (is_signed_imm12(-imm)) {
emit_insn(ctx, addid, dst, dst, -imm);
} else {
- move_imm(ctx, t1, imm, is32);
+ move_imm(ctx, t1, imm);
+ emit_zext_32(ctx, t1, is32);
emit_insn(ctx, subd, dst, dst, t1);
}
emit_zext_32(ctx, dst, is32);
@@ -804,7 +807,8 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
/* dst = dst * imm */
case BPF_ALU | BPF_MUL | BPF_K:
case BPF_ALU64 | BPF_MUL | BPF_K:
- move_imm(ctx, t1, imm, is32);
+ move_imm(ctx, t1, imm);
+ emit_zext_32(ctx, t1, is32);
emit_insn(ctx, muld, dst, dst, t1);
emit_zext_32(ctx, dst, is32);
break;
@@ -831,12 +835,13 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
case BPF_ALU | BPF_DIV | BPF_K:
case BPF_ALU64 | BPF_DIV | BPF_K:
if (!off) {
- move_imm(ctx, t1, imm, is32);
+ move_imm(ctx, t1, imm);
+ emit_zext_32(ctx, t1, is32);
emit_zext_32(ctx, dst, is32);
emit_insn(ctx, divdu, dst, dst, t1);
emit_zext_32(ctx, dst, is32);
} else {
- move_imm(ctx, t1, imm, false);
+ move_imm(ctx, t1, imm);
emit_sext_32(ctx, t1, is32);
emit_sext_32(ctx, dst, is32);
emit_insn(ctx, divd, dst, dst, t1);
@@ -866,12 +871,13 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
case BPF_ALU | BPF_MOD | BPF_K:
case BPF_ALU64 | BPF_MOD | BPF_K:
if (!off) {
- move_imm(ctx, t1, imm, is32);
+ move_imm(ctx, t1, imm);
+ emit_zext_32(ctx, t1, is32);
emit_zext_32(ctx, dst, is32);
emit_insn(ctx, moddu, dst, dst, t1);
emit_zext_32(ctx, dst, is32);
} else {
- move_imm(ctx, t1, imm, false);
+ move_imm(ctx, t1, imm);
emit_sext_32(ctx, t1, is32);
emit_sext_32(ctx, dst, is32);
emit_insn(ctx, modd, dst, dst, t1);
@@ -899,7 +905,8 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
if (is_unsigned_imm12(imm)) {
emit_insn(ctx, andi, dst, dst, imm);
} else {
- move_imm(ctx, t1, imm, is32);
+ move_imm(ctx, t1, imm);
+ emit_zext_32(ctx, t1, is32);
emit_insn(ctx, and, dst, dst, t1);
}
emit_zext_32(ctx, dst, is32);
@@ -918,7 +925,8 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
if (is_unsigned_imm12(imm)) {
emit_insn(ctx, ori, dst, dst, imm);
} else {
- move_imm(ctx, t1, imm, is32);
+ move_imm(ctx, t1, imm);
+ emit_zext_32(ctx, t1, is32);
emit_insn(ctx, or, dst, dst, t1);
}
emit_zext_32(ctx, dst, is32);
@@ -937,7 +945,8 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
if (is_unsigned_imm12(imm)) {
emit_insn(ctx, xori, dst, dst, imm);
} else {
- move_imm(ctx, t1, imm, is32);
+ move_imm(ctx, t1, imm);
+ emit_zext_32(ctx, t1, is32);
emit_insn(ctx, xor, dst, dst, t1);
}
emit_zext_32(ctx, dst, is32);
@@ -1097,7 +1106,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
case BPF_JMP32 | BPF_JSLE | BPF_K:
jmp_offset = bpf2la_offset(i, off, ctx);
if (imm) {
- move_imm(ctx, t1, imm, false);
+ move_imm(ctx, t1, imm);
tm = t1;
} else {
/* If imm is 0, simply use zero register. */
@@ -1129,7 +1138,8 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
case BPF_JMP | BPF_JSET | BPF_K:
case BPF_JMP32 | BPF_JSET | BPF_K:
jmp_offset = bpf2la_offset(i, off, ctx);
- move_imm(ctx, t1, imm, is32);
+ move_imm(ctx, t1, imm);
+ emit_zext_32(ctx, t1, is32);
emit_insn(ctx, and, t1, dst, t1);
emit_zext_32(ctx, t1, is32);
if (emit_cond_jmp(ctx, cond, t1, LOONGARCH_GPR_ZERO, jmp_offset) < 0)
@@ -1227,7 +1237,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
if (bpf_pseudo_func(insn))
move_addr(ctx, dst, imm64);
else
- move_imm(ctx, dst, imm64, is32);
+ move_imm(ctx, dst, imm64);
return 1;
}
@@ -1268,7 +1278,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
else
emit_insn(ctx, ldbu, dst, src, off);
} else {
- move_imm(ctx, t1, off, is32);
+ move_imm(ctx, t1, off);
if (sign_extend)
emit_insn(ctx, ldxb, dst, src, t1);
else
@@ -1282,7 +1292,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
else
emit_insn(ctx, ldhu, dst, src, off);
} else {
- move_imm(ctx, t1, off, is32);
+ move_imm(ctx, t1, off);
if (sign_extend)
emit_insn(ctx, ldxh, dst, src, t1);
else
@@ -1296,7 +1306,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
else
emit_insn(ctx, ldwu, dst, src, off);
} else {
- move_imm(ctx, t1, off, is32);
+ move_imm(ctx, t1, off);
if (sign_extend)
emit_insn(ctx, ldxw, dst, src, t1);
else
@@ -1304,7 +1314,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
}
break;
case BPF_DW:
- move_imm(ctx, t1, off, is32);
+ move_imm(ctx, t1, off);
emit_insn(ctx, ldxd, dst, src, t1);
break;
}
@@ -1331,42 +1341,42 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
switch (BPF_SIZE(code)) {
case BPF_B:
- move_imm(ctx, t1, imm, is32);
+ move_imm(ctx, t1, imm);
if (is_signed_imm12(off)) {
emit_insn(ctx, stb, t1, dst, off);
} else {
- move_imm(ctx, t2, off, is32);
+ move_imm(ctx, t2, off);
emit_insn(ctx, stxb, t1, dst, t2);
}
break;
case BPF_H:
- move_imm(ctx, t1, imm, is32);
+ move_imm(ctx, t1, imm);
if (is_signed_imm12(off)) {
emit_insn(ctx, sth, t1, dst, off);
} else {
- move_imm(ctx, t2, off, is32);
+ move_imm(ctx, t2, off);
emit_insn(ctx, stxh, t1, dst, t2);
}
break;
case BPF_W:
- move_imm(ctx, t1, imm, is32);
+ move_imm(ctx, t1, imm);
if (is_signed_imm12(off)) {
emit_insn(ctx, stw, t1, dst, off);
} else if (is_signed_imm14(off)) {
emit_insn(ctx, stptrw, t1, dst, off);
} else {
- move_imm(ctx, t2, off, is32);
+ move_imm(ctx, t2, off);
emit_insn(ctx, stxw, t1, dst, t2);
}
break;
case BPF_DW:
- move_imm(ctx, t1, imm, is32);
+ move_imm(ctx, t1, imm);
if (is_signed_imm12(off)) {
emit_insn(ctx, std, t1, dst, off);
} else if (is_signed_imm14(off)) {
emit_insn(ctx, stptrd, t1, dst, off);
} else {
- move_imm(ctx, t2, off, is32);
+ move_imm(ctx, t2, off);
emit_insn(ctx, stxd, t1, dst, t2);
}
break;
@@ -1397,7 +1407,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
if (is_signed_imm12(off)) {
emit_insn(ctx, stb, src, dst, off);
} else {
- move_imm(ctx, t1, off, is32);
+ move_imm(ctx, t1, off);
emit_insn(ctx, stxb, src, dst, t1);
}
break;
@@ -1405,7 +1415,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
if (is_signed_imm12(off)) {
emit_insn(ctx, sth, src, dst, off);
} else {
- move_imm(ctx, t1, off, is32);
+ move_imm(ctx, t1, off);
emit_insn(ctx, stxh, src, dst, t1);
}
break;
@@ -1415,7 +1425,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
} else if (is_signed_imm14(off)) {
emit_insn(ctx, stptrw, src, dst, off);
} else {
- move_imm(ctx, t1, off, is32);
+ move_imm(ctx, t1, off);
emit_insn(ctx, stxw, src, dst, t1);
}
break;
@@ -1425,7 +1435,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
} else if (is_signed_imm14(off)) {
emit_insn(ctx, stptrd, src, dst, off);
} else {
- move_imm(ctx, t1, off, is32);
+ move_imm(ctx, t1, off);
emit_insn(ctx, stxd, src, dst, t1);
}
break;
@@ -1537,7 +1547,7 @@ static int emit_jump_and_link(struct jit_ctx *ctx, u8 rd, u64 target)
return -EFAULT;
}
- move_imm(ctx, LOONGARCH_GPR_T1, target, false);
+ move_imm(ctx, LOONGARCH_GPR_T1, target);
emit_insn(ctx, jirl, rd, LOONGARCH_GPR_T1, 0);
return 0;
@@ -1734,7 +1744,7 @@ static int invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_image *im,
emit_insn(ctx, std, LOONGARCH_GPR_ZERO, LOONGARCH_GPR_FP, -run_ctx_off + cookie_off);
/* arg1: prog */
- move_imm(ctx, LOONGARCH_GPR_A0, (const s64)p, false);
+ move_imm(ctx, LOONGARCH_GPR_A0, (const s64)p);
/* arg2: &run_ctx */
emit_insn(ctx, addid, LOONGARCH_GPR_A1, LOONGARCH_GPR_FP, -run_ctx_off);
ret = emit_call(ctx, (const u64)bpf_trampoline_enter(p));
@@ -1755,7 +1765,7 @@ static int invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_image *im,
/* arg1: &args_off */
emit_insn(ctx, addid, LOONGARCH_GPR_A0, LOONGARCH_GPR_FP, -args_off);
if (!p->jited)
- move_imm(ctx, LOONGARCH_GPR_A1, (const s64)p->insnsi, false);
+ move_imm(ctx, LOONGARCH_GPR_A1, (const s64)p->insnsi);
ret = emit_call(ctx, (const u64)p->bpf_func);
if (ret)
return ret;
@@ -1772,7 +1782,7 @@ static int invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_image *im,
}
/* arg1: prog */
- move_imm(ctx, LOONGARCH_GPR_A0, (const s64)p, false);
+ move_imm(ctx, LOONGARCH_GPR_A0, (const s64)p);
/* arg2: prog start time */
move_reg(ctx, LOONGARCH_GPR_A1, LOONGARCH_GPR_S1);
/* arg3: &run_ctx */
diff --git a/arch/loongarch/net/bpf_jit.h b/arch/loongarch/net/bpf_jit.h
index bb58c42c2f2a..57a98f8a055b 100644
--- a/arch/loongarch/net/bpf_jit.h
+++ b/arch/loongarch/net/bpf_jit.h
@@ -137,7 +137,7 @@ static inline void move_addr(struct jit_ctx *ctx, enum loongarch_gpr rd, u64 add
emit_insn(ctx, lu52id, rd, rd, imm_63_52);
}
-static inline void move_imm(struct jit_ctx *ctx, enum loongarch_gpr rd, long imm, bool is32)
+static inline void move_imm(struct jit_ctx *ctx, enum loongarch_gpr rd, long imm)
{
long imm_11_0, imm_31_12, imm_51_32, imm_63_52, imm_51_0, imm_51_31;
@@ -150,7 +150,7 @@ static inline void move_imm(struct jit_ctx *ctx, enum loongarch_gpr rd, long imm
/* addiw rd, $zero, imm_11_0 */
if (is_signed_imm12(imm)) {
emit_insn(ctx, addiw, rd, LOONGARCH_GPR_ZERO, imm);
- goto zext;
+ return;
}
/* ori rd, $zero, imm_11_0 */
@@ -195,9 +195,6 @@ static inline void move_imm(struct jit_ctx *ctx, enum loongarch_gpr rd, long imm
if (!is_signed_imm52(imm))
emit_insn(ctx, lu52id, rd, rd, imm_63_52);
}
-
-zext:
- emit_zext_32(ctx, rd, is32);
}
static inline void move_reg(struct jit_ctx *ctx, enum loongarch_gpr rd,
--
2.42.0
^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH v3 3/3] LoongArch: BPF: Optimize 32-bit operations using native instructions
2026-10-16 8:10 [PATCH v3 0/3] LoongArch: BPF: Refine move_imm() and optimize 32-bit operations Tiezhu Yang
2026-10-16 8:10 ` [PATCH v3 1/3] LoongArch: Add CPUCFG2 DIV32 feature detection Tiezhu Yang
2026-10-16 8:10 ` [PATCH v3 2/3] LoongArch: BPF: Clean up and decouple is32 parameter from move_imm() Tiezhu Yang
@ 2026-10-16 8:10 ` Tiezhu Yang
2026-10-16 15:18 ` Alexei Starovoitov
` (2 more replies)
2026-10-22 7:43 ` [PATCH v3 0/3] LoongArch: BPF: Refine move_imm() and optimize 32-bit operations Tiezhu Yang
3 siblings, 3 replies; 15+ messages in thread
From: Tiezhu Yang @ 2026-10-16 8:10 UTC (permalink / raw)
To: Huacai Chen, Hengqi Chen; +Cc: loongarch, bpf, linux-kernel
Currently, the LoongArch BPF JIT emits numerous redundant zero and sign
extensions across 32-bit operations, bloating the final JIT image size
and also leading to runtime overhead, particularly in the division and
modulo operations. Thus, it is necessary to optimize 32-bit operations
by emitting LoongArch native 32-bit word instructions.
For general ALU32 operations, utilize native 32-bit word instructions
for core calculations. Given that the BPF spec requires the upper 32
bits of the destination register to be zero-extended upon completion,
use a unified 'bstrpickd' instruction to clear the upper 32 bits at
the common exit of build_insn(), while immediate loads are optimized
via (u32) casting to bypass the trailing extension.
According to the LoongArch Reference Manual, native 32-bit division
and modulo instructions (DIV.W[U] and MOD.W[U]) require the upper
bits [63:31] of both input operands to be all 0s or all 1s (0x0 or
0x1ffffffff), that is to say, they need to be properly sign-extended
by the software on CPU cores without the DIV32 feature. Otherwise,
the 32-bit division and modulo instructions could trigger undefined
hardware behaviors and return unexpected values.
To ensure both performance and hardware compatibility, leverage the
newly introduced cpu_has_div32 helper:
(1) on newer CPUs with DIV32, emit native 32-bit div/mod instructions
directly for optimal performance;
(2) on older CPUs without DIV32, sign-extend the destination register
in place and the source register into a temporary register via 'addw'
before executing the native 32-bit div/mod instructions. Note that the
JIT compiler must never alter the state of the src register because it
remains read-only for the duration of the instruction.
For JMP32 conditional branches, the source operands may originate from
64-bit contexts where the upper 32 bits contain arbitrary data, so use
6 independent case structures to prevent 64-bit comparison inversions.
Furthermore, use 'bstrpickd' to mask out upper bits for unsigned JMP32,
and 'addw' with GPR_ZERO to re-align sign bits for signed JMP32.
With this patch, the emit_{zext, sext}_32() and is_signed_bpf_cond()
helpers are completely wiped out from the header and source files,
delivering a native execution stream with a reduced JIT image size,
especially for division and mod operations.
All of the testcases of test_bpf passed on Loongson-3A6000 (with DIV32)
and Loongson-3A5000 (without DIV32), here are the test results:
$ sudo modprobe test_bpf
$ dmesg -t | grep Summary
test_bpf: Summary: 1061 PASSED, 0 FAILED, [1049/1049 JIT'ed]
test_bpf: test_tail_calls: Summary: 10 PASSED, 0 FAILED, [10/10 JIT'ed]
test_bpf: test_skb_segment: Summary: 2 PASSED, 0 FAILED
For the BPF selftests test_verifier and test_progs, there are no any
regressions.
Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
---
arch/loongarch/include/asm/inst.h | 6 +
arch/loongarch/net/bpf_jit.c | 313 +++++++++++++++++-------------
arch/loongarch/net/bpf_jit.h | 18 --
3 files changed, 183 insertions(+), 154 deletions(-)
diff --git a/arch/loongarch/include/asm/inst.h b/arch/loongarch/include/asm/inst.h
index 585667e361c2..b55aae325300 100644
--- a/arch/loongarch/include/asm/inst.h
+++ b/arch/loongarch/include/asm/inst.h
@@ -752,11 +752,17 @@ static inline void emit_##NAME(union loongarch_instruction *insn, \
DEF_EMIT_REG3_FORMAT(addw, addw_op)
DEF_EMIT_REG3_FORMAT(addd, addd_op)
+DEF_EMIT_REG3_FORMAT(subw, subw_op)
DEF_EMIT_REG3_FORMAT(subd, subd_op)
+DEF_EMIT_REG3_FORMAT(mulw, mulw_op)
DEF_EMIT_REG3_FORMAT(muld, muld_op)
+DEF_EMIT_REG3_FORMAT(divw, divw_op)
DEF_EMIT_REG3_FORMAT(divd, divd_op)
+DEF_EMIT_REG3_FORMAT(modw, modw_op)
DEF_EMIT_REG3_FORMAT(modd, modd_op)
+DEF_EMIT_REG3_FORMAT(divwu, divwu_op)
DEF_EMIT_REG3_FORMAT(divdu, divdu_op)
+DEF_EMIT_REG3_FORMAT(modwu, modwu_op)
DEF_EMIT_REG3_FORMAT(moddu, moddu_op)
DEF_EMIT_REG3_FORMAT(maskeqz, maskeqz_op)
DEF_EMIT_REG3_FORMAT(masknez, masknez_op)
diff --git a/arch/loongarch/net/bpf_jit.c b/arch/loongarch/net/bpf_jit.c
index 77cd96a6f84c..1a7c8effc37b 100644
--- a/arch/loongarch/net/bpf_jit.c
+++ b/arch/loongarch/net/bpf_jit.c
@@ -288,7 +288,7 @@ static int emit_bpf_tail_call(struct jit_ctx *ctx, int insn)
* if (index >= array->map.max_entries)
* goto out;
*/
- emit_zext_32(ctx, a2, true);
+ emit_insn(ctx, bstrpickd, a2, a2, 31, 0);
off = offsetof(struct bpf_array, map.max_entries);
emit_insn(ctx, ldwu, t1, a1, off);
@@ -412,7 +412,6 @@ static int emit_atomic_rmw(const struct bpf_insn *insn, struct jit_ctx *ctx)
return -EINVAL;
}
emit_insn(ctx, amadddbb, src, t1, t3);
- emit_zext_32(ctx, src, true);
break;
case BPF_H:
if (!cpu_has_lam_bh) {
@@ -420,23 +419,23 @@ static int emit_atomic_rmw(const struct bpf_insn *insn, struct jit_ctx *ctx)
return -EINVAL;
}
emit_insn(ctx, amadddbh, src, t1, t3);
- emit_zext_32(ctx, src, true);
break;
case BPF_W:
emit_insn(ctx, amadddbw, src, t1, t3);
- emit_zext_32(ctx, src, true);
break;
case BPF_DW:
emit_insn(ctx, amadddbd, src, t1, t3);
break;
}
+ if (!isdw)
+ emit_insn(ctx, bstrpickd, src, src, 31, 0);
break;
case BPF_AND | BPF_FETCH:
if (isdw) {
emit_insn(ctx, amanddbd, src, t1, t3);
} else {
emit_insn(ctx, amanddbw, src, t1, t3);
- emit_zext_32(ctx, src, true);
+ emit_insn(ctx, bstrpickd, src, src, 31, 0);
}
break;
case BPF_OR | BPF_FETCH:
@@ -444,7 +443,7 @@ static int emit_atomic_rmw(const struct bpf_insn *insn, struct jit_ctx *ctx)
emit_insn(ctx, amordbd, src, t1, t3);
} else {
emit_insn(ctx, amordbw, src, t1, t3);
- emit_zext_32(ctx, src, true);
+ emit_insn(ctx, bstrpickd, src, src, 31, 0);
}
break;
case BPF_XOR | BPF_FETCH:
@@ -452,7 +451,7 @@ static int emit_atomic_rmw(const struct bpf_insn *insn, struct jit_ctx *ctx)
emit_insn(ctx, amxordbd, src, t1, t3);
} else {
emit_insn(ctx, amxordbw, src, t1, t3);
- emit_zext_32(ctx, src, true);
+ emit_insn(ctx, bstrpickd, src, src, 31, 0);
}
break;
/* src = atomic_xchg(dst + off, src); */
@@ -464,7 +463,6 @@ static int emit_atomic_rmw(const struct bpf_insn *insn, struct jit_ctx *ctx)
return -EINVAL;
}
emit_insn(ctx, amswapdbb, src, t1, t3);
- emit_zext_32(ctx, src, true);
break;
case BPF_H:
if (!cpu_has_lam_bh) {
@@ -472,16 +470,16 @@ static int emit_atomic_rmw(const struct bpf_insn *insn, struct jit_ctx *ctx)
return -EINVAL;
}
emit_insn(ctx, amswapdbh, src, t1, t3);
- emit_zext_32(ctx, src, true);
break;
case BPF_W:
emit_insn(ctx, amswapdbw, src, t1, t3);
- emit_zext_32(ctx, src, true);
break;
case BPF_DW:
emit_insn(ctx, amswapdbd, src, t1, t3);
break;
}
+ if (!isdw)
+ emit_insn(ctx, bstrpickd, src, src, 31, 0);
break;
/* r0 = atomic_cmpxchg(dst + off, r0, src); */
case BPF_CMPXCHG:
@@ -494,13 +492,13 @@ static int emit_atomic_rmw(const struct bpf_insn *insn, struct jit_ctx *ctx)
emit_insn(ctx, beq, t3, LOONGARCH_GPR_ZERO, -4);
} else {
emit_insn(ctx, llw, r0, t1, 0);
- emit_zext_32(ctx, t2, true);
- emit_zext_32(ctx, r0, true);
+ emit_insn(ctx, bstrpickd, t2, t2, 31, 0);
+ emit_insn(ctx, bstrpickd, r0, r0, 31, 0);
emit_insn(ctx, bne, t2, r0, 4);
move_reg(ctx, t3, src);
emit_insn(ctx, scw, t3, t1, 0);
emit_insn(ctx, beq, t3, LOONGARCH_GPR_ZERO, -6);
- emit_zext_32(ctx, r0, true);
+ emit_insn(ctx, bstrpickd, r0, r0, 31, 0);
}
emit_insn(ctx, dbar, DBAR_LLSC_MB);
break;
@@ -605,12 +603,6 @@ static int emit_atomic_ld_st(const struct bpf_insn *insn, struct jit_ctx *ctx)
return 0;
}
-static bool is_signed_bpf_cond(u8 cond)
-{
- return cond == BPF_JSGT || cond == BPF_JSLT ||
- cond == BPF_JSGE || cond == BPF_JSLE;
-}
-
#define BPF_FIXUP_REG_MASK GENMASK(31, 27)
#define BPF_FIXUP_OFFSET_MASK GENMASK(26, 0)
#define REG_DONT_CLEAR_MARKER 0
@@ -707,7 +699,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
u8 dst = regmap[insn->dst_reg];
const s16 off = insn->off;
const s32 imm = insn->imm;
- const bool is32 = BPF_CLASS(insn->code) == BPF_ALU || BPF_CLASS(insn->code) == BPF_JMP32;
+ const bool is32 = BPF_CLASS(insn->code) == BPF_ALU && BPF_OP(insn->code) != BPF_END;
switch (code) {
/* dst = src */
@@ -715,7 +707,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
case BPF_ALU64 | BPF_MOV | BPF_X:
if (insn_is_cast_user(insn)) {
move_reg(ctx, t1, src);
- emit_zext_32(ctx, t1, true);
+ emit_insn(ctx, bstrpickd, t1, t1, 31, 0);
move_imm(ctx, dst, (ctx->user_vm_start >> 32) << 32);
emit_insn(ctx, beq, t1, LOONGARCH_GPR_ZERO, 2);
emit_insn(ctx, or, t1, dst, t1);
@@ -734,15 +726,12 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
switch (off) {
case 0:
move_reg(ctx, dst, src);
- emit_zext_32(ctx, dst, is32);
break;
case 8:
emit_insn(ctx, extwb, dst, src);
- emit_zext_32(ctx, dst, is32);
break;
case 16:
emit_insn(ctx, extwh, dst, src);
- emit_zext_32(ctx, dst, is32);
break;
case 32:
emit_insn(ctx, addw, dst, src, LOONGARCH_GPR_ZERO);
@@ -752,151 +741,188 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
/* dst = imm */
case BPF_ALU | BPF_MOV | BPF_K:
+ move_imm(ctx, dst, (u32)imm);
+ return 0;
case BPF_ALU64 | BPF_MOV | BPF_K:
move_imm(ctx, dst, imm);
- emit_zext_32(ctx, dst, is32);
break;
/* dst = dst + src */
case BPF_ALU | BPF_ADD | BPF_X:
+ emit_insn(ctx, addw, dst, dst, src);
+ break;
case BPF_ALU64 | BPF_ADD | BPF_X:
emit_insn(ctx, addd, dst, dst, src);
- emit_zext_32(ctx, dst, is32);
break;
/* dst = dst + imm */
case BPF_ALU | BPF_ADD | BPF_K:
+ if (is_signed_imm12(imm)) {
+ emit_insn(ctx, addiw, dst, dst, imm);
+ } else {
+ move_imm(ctx, t1, (u32)imm);
+ emit_insn(ctx, addw, dst, dst, t1);
+ }
+ break;
case BPF_ALU64 | BPF_ADD | BPF_K:
if (is_signed_imm12(imm)) {
emit_insn(ctx, addid, dst, dst, imm);
} else {
move_imm(ctx, t1, imm);
- emit_zext_32(ctx, t1, is32);
emit_insn(ctx, addd, dst, dst, t1);
}
- emit_zext_32(ctx, dst, is32);
break;
/* dst = dst - src */
case BPF_ALU | BPF_SUB | BPF_X:
+ emit_insn(ctx, subw, dst, dst, src);
+ break;
case BPF_ALU64 | BPF_SUB | BPF_X:
emit_insn(ctx, subd, dst, dst, src);
- emit_zext_32(ctx, dst, is32);
break;
/* dst = dst - imm */
case BPF_ALU | BPF_SUB | BPF_K:
+ if (is_signed_imm12(-imm)) {
+ emit_insn(ctx, addiw, dst, dst, -imm);
+ } else {
+ move_imm(ctx, t1, (u32)imm);
+ emit_insn(ctx, subw, dst, dst, t1);
+ }
+ break;
case BPF_ALU64 | BPF_SUB | BPF_K:
if (is_signed_imm12(-imm)) {
emit_insn(ctx, addid, dst, dst, -imm);
} else {
move_imm(ctx, t1, imm);
- emit_zext_32(ctx, t1, is32);
emit_insn(ctx, subd, dst, dst, t1);
}
- emit_zext_32(ctx, dst, is32);
break;
/* dst = dst * src */
case BPF_ALU | BPF_MUL | BPF_X:
+ emit_insn(ctx, mulw, dst, dst, src);
+ break;
case BPF_ALU64 | BPF_MUL | BPF_X:
emit_insn(ctx, muld, dst, dst, src);
- emit_zext_32(ctx, dst, is32);
break;
/* dst = dst * imm */
case BPF_ALU | BPF_MUL | BPF_K:
+ move_imm(ctx, t1, (u32)imm);
+ emit_insn(ctx, mulw, dst, dst, t1);
+ break;
case BPF_ALU64 | BPF_MUL | BPF_K:
move_imm(ctx, t1, imm);
- emit_zext_32(ctx, t1, is32);
emit_insn(ctx, muld, dst, dst, t1);
- emit_zext_32(ctx, dst, is32);
break;
/* dst = dst / src */
case BPF_ALU | BPF_DIV | BPF_X:
- case BPF_ALU64 | BPF_DIV | BPF_X:
- if (!off) {
- emit_zext_32(ctx, dst, is32);
- move_reg(ctx, t1, src);
- emit_zext_32(ctx, t1, is32);
- emit_insn(ctx, divdu, dst, dst, t1);
- emit_zext_32(ctx, dst, is32);
- } else {
- emit_sext_32(ctx, dst, is32);
- move_reg(ctx, t1, src);
- emit_sext_32(ctx, t1, is32);
- emit_insn(ctx, divd, dst, dst, t1);
- emit_zext_32(ctx, dst, is32);
+ if (cpu_has_div32) {
+ if (!off)
+ emit_insn(ctx, divwu, dst, dst, src);
+ else
+ emit_insn(ctx, divw, dst, dst, src);
+ break;
}
+ /*
+ * Legacy fallback for !div32: src must be extended into a
+ * temporary register to keep the original src register intact.
+ */
+ emit_insn(ctx, addw, dst, dst, LOONGARCH_GPR_ZERO);
+ emit_insn(ctx, addw, t1, src, LOONGARCH_GPR_ZERO);
+ if (!off)
+ emit_insn(ctx, divwu, dst, dst, t1);
+ else
+ emit_insn(ctx, divw, dst, dst, t1);
+ break;
+ case BPF_ALU64 | BPF_DIV | BPF_X:
+ if (!off)
+ emit_insn(ctx, divdu, dst, dst, src);
+ else
+ emit_insn(ctx, divd, dst, dst, src);
break;
/* dst = dst / imm */
case BPF_ALU | BPF_DIV | BPF_K:
+ move_imm(ctx, t1, (u32)imm);
+ if (!cpu_has_div32) {
+ emit_insn(ctx, addw, dst, dst, LOONGARCH_GPR_ZERO);
+ emit_insn(ctx, addw, t1, t1, LOONGARCH_GPR_ZERO);
+ }
+ if (!off)
+ emit_insn(ctx, divwu, dst, dst, t1);
+ else
+ emit_insn(ctx, divw, dst, dst, t1);
+ break;
case BPF_ALU64 | BPF_DIV | BPF_K:
- if (!off) {
- move_imm(ctx, t1, imm);
- emit_zext_32(ctx, t1, is32);
- emit_zext_32(ctx, dst, is32);
+ move_imm(ctx, t1, imm);
+ if (!off)
emit_insn(ctx, divdu, dst, dst, t1);
- emit_zext_32(ctx, dst, is32);
- } else {
- move_imm(ctx, t1, imm);
- emit_sext_32(ctx, t1, is32);
- emit_sext_32(ctx, dst, is32);
+ else
emit_insn(ctx, divd, dst, dst, t1);
- emit_zext_32(ctx, dst, is32);
- }
break;
/* dst = dst % src */
case BPF_ALU | BPF_MOD | BPF_X:
- case BPF_ALU64 | BPF_MOD | BPF_X:
- if (!off) {
- emit_zext_32(ctx, dst, is32);
- move_reg(ctx, t1, src);
- emit_zext_32(ctx, t1, is32);
- emit_insn(ctx, moddu, dst, dst, t1);
- emit_zext_32(ctx, dst, is32);
- } else {
- emit_sext_32(ctx, dst, is32);
- move_reg(ctx, t1, src);
- emit_sext_32(ctx, t1, is32);
- emit_insn(ctx, modd, dst, dst, t1);
- emit_zext_32(ctx, dst, is32);
+ if (cpu_has_div32) {
+ if (!off)
+ emit_insn(ctx, modwu, dst, dst, src);
+ else
+ emit_insn(ctx, modw, dst, dst, src);
+ break;
}
+ /*
+ * Legacy fallback for !div32: src must be extended into a
+ * temporary register to keep the original src register intact.
+ */
+ emit_insn(ctx, addw, dst, dst, LOONGARCH_GPR_ZERO);
+ emit_insn(ctx, addw, t1, src, LOONGARCH_GPR_ZERO);
+ if (!off)
+ emit_insn(ctx, modwu, dst, dst, t1);
+ else
+ emit_insn(ctx, modw, dst, dst, t1);
+ break;
+ case BPF_ALU64 | BPF_MOD | BPF_X:
+ if (!off)
+ emit_insn(ctx, moddu, dst, dst, src);
+ else
+ emit_insn(ctx, modd, dst, dst, src);
break;
/* dst = dst % imm */
case BPF_ALU | BPF_MOD | BPF_K:
+ move_imm(ctx, t1, (u32)imm);
+ if (!cpu_has_div32) {
+ emit_insn(ctx, addw, dst, dst, LOONGARCH_GPR_ZERO);
+ emit_insn(ctx, addw, t1, t1, LOONGARCH_GPR_ZERO);
+ }
+ if (!off)
+ emit_insn(ctx, modwu, dst, dst, t1);
+ else
+ emit_insn(ctx, modw, dst, dst, t1);
+ break;
case BPF_ALU64 | BPF_MOD | BPF_K:
- if (!off) {
- move_imm(ctx, t1, imm);
- emit_zext_32(ctx, t1, is32);
- emit_zext_32(ctx, dst, is32);
+ move_imm(ctx, t1, imm);
+ if (!off)
emit_insn(ctx, moddu, dst, dst, t1);
- emit_zext_32(ctx, dst, is32);
- } else {
- move_imm(ctx, t1, imm);
- emit_sext_32(ctx, t1, is32);
- emit_sext_32(ctx, dst, is32);
+ else
emit_insn(ctx, modd, dst, dst, t1);
- emit_zext_32(ctx, dst, is32);
- }
break;
/* dst = -dst */
case BPF_ALU | BPF_NEG:
+ emit_insn(ctx, subw, dst, LOONGARCH_GPR_ZERO, dst);
+ break;
case BPF_ALU64 | BPF_NEG:
emit_insn(ctx, subd, dst, LOONGARCH_GPR_ZERO, dst);
- emit_zext_32(ctx, dst, is32);
break;
/* dst = dst & src */
case BPF_ALU | BPF_AND | BPF_X:
case BPF_ALU64 | BPF_AND | BPF_X:
emit_insn(ctx, and, dst, dst, src);
- emit_zext_32(ctx, dst, is32);
break;
/* dst = dst & imm */
@@ -905,18 +931,18 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
if (is_unsigned_imm12(imm)) {
emit_insn(ctx, andi, dst, dst, imm);
} else {
- move_imm(ctx, t1, imm);
- emit_zext_32(ctx, t1, is32);
+ if (is32)
+ move_imm(ctx, t1, (u32)imm);
+ else
+ move_imm(ctx, t1, imm);
emit_insn(ctx, and, dst, dst, t1);
}
- emit_zext_32(ctx, dst, is32);
break;
/* dst = dst | src */
case BPF_ALU | BPF_OR | BPF_X:
case BPF_ALU64 | BPF_OR | BPF_X:
emit_insn(ctx, or, dst, dst, src);
- emit_zext_32(ctx, dst, is32);
break;
/* dst = dst | imm */
@@ -925,18 +951,18 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
if (is_unsigned_imm12(imm)) {
emit_insn(ctx, ori, dst, dst, imm);
} else {
- move_imm(ctx, t1, imm);
- emit_zext_32(ctx, t1, is32);
+ if (is32)
+ move_imm(ctx, t1, (u32)imm);
+ else
+ move_imm(ctx, t1, imm);
emit_insn(ctx, or, dst, dst, t1);
}
- emit_zext_32(ctx, dst, is32);
break;
/* dst = dst ^ src */
case BPF_ALU | BPF_XOR | BPF_X:
case BPF_ALU64 | BPF_XOR | BPF_X:
emit_insn(ctx, xor, dst, dst, src);
- emit_zext_32(ctx, dst, is32);
break;
/* dst = dst ^ imm */
@@ -945,19 +971,18 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
if (is_unsigned_imm12(imm)) {
emit_insn(ctx, xori, dst, dst, imm);
} else {
- move_imm(ctx, t1, imm);
- emit_zext_32(ctx, t1, is32);
+ if (is32)
+ move_imm(ctx, t1, (u32)imm);
+ else
+ move_imm(ctx, t1, imm);
emit_insn(ctx, xor, dst, dst, t1);
}
- emit_zext_32(ctx, dst, is32);
break;
/* dst = dst << src (logical) */
case BPF_ALU | BPF_LSH | BPF_X:
emit_insn(ctx, sllw, dst, dst, src);
- emit_zext_32(ctx, dst, is32);
break;
-
case BPF_ALU64 | BPF_LSH | BPF_X:
emit_insn(ctx, slld, dst, dst, src);
break;
@@ -965,9 +990,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
/* dst = dst << imm (logical) */
case BPF_ALU | BPF_LSH | BPF_K:
emit_insn(ctx, slliw, dst, dst, imm);
- emit_zext_32(ctx, dst, is32);
break;
-
case BPF_ALU64 | BPF_LSH | BPF_K:
emit_insn(ctx, sllid, dst, dst, imm);
break;
@@ -975,9 +998,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
/* dst = dst >> src (logical) */
case BPF_ALU | BPF_RSH | BPF_X:
emit_insn(ctx, srlw, dst, dst, src);
- emit_zext_32(ctx, dst, is32);
break;
-
case BPF_ALU64 | BPF_RSH | BPF_X:
emit_insn(ctx, srld, dst, dst, src);
break;
@@ -985,9 +1006,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
/* dst = dst >> imm (logical) */
case BPF_ALU | BPF_RSH | BPF_K:
emit_insn(ctx, srliw, dst, dst, imm);
- emit_zext_32(ctx, dst, is32);
break;
-
case BPF_ALU64 | BPF_RSH | BPF_K:
emit_insn(ctx, srlid, dst, dst, imm);
break;
@@ -995,9 +1014,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
/* dst = dst >> src (arithmetic) */
case BPF_ALU | BPF_ARSH | BPF_X:
emit_insn(ctx, sraw, dst, dst, src);
- emit_zext_32(ctx, dst, is32);
break;
-
case BPF_ALU64 | BPF_ARSH | BPF_X:
emit_insn(ctx, srad, dst, dst, src);
break;
@@ -1005,9 +1022,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
/* dst = dst >> imm (arithmetic) */
case BPF_ALU | BPF_ARSH | BPF_K:
emit_insn(ctx, sraiw, dst, dst, imm);
- emit_zext_32(ctx, dst, is32);
break;
-
case BPF_ALU64 | BPF_ARSH | BPF_K:
emit_insn(ctx, sraid, dst, dst, imm);
break;
@@ -1016,12 +1031,10 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
case BPF_ALU | BPF_END | BPF_FROM_LE:
switch (imm) {
case 16:
- /* zero-extend 16 bits into 64 bits */
emit_insn(ctx, bstrpickd, dst, dst, 15, 0);
break;
case 32:
- /* zero-extend 32 bits into 64 bits */
- emit_zext_32(ctx, dst, is32);
+ emit_insn(ctx, bstrpickd, dst, dst, 31, 0);
break;
case 64:
/* do nothing */
@@ -1034,13 +1047,11 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
switch (imm) {
case 16:
emit_insn(ctx, revb2h, dst, dst);
- /* zero-extend 16 bits into 64 bits */
emit_insn(ctx, bstrpickd, dst, dst, 15, 0);
break;
case 32:
emit_insn(ctx, revb2w, dst, dst);
- /* clear the upper 32 bits */
- emit_zext_32(ctx, dst, true);
+ emit_insn(ctx, bstrpickd, dst, dst, 31, 0);
break;
case 64:
emit_insn(ctx, revbd, dst, dst);
@@ -1048,7 +1059,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
}
break;
- /* PC += off if dst cond src */
+ /* PC += off if dst cond src (64-bit) */
case BPF_JMP | BPF_JEQ | BPF_X:
case BPF_JMP | BPF_JNE | BPF_X:
case BPF_JMP | BPF_JGT | BPF_X:
@@ -1059,31 +1070,38 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
case BPF_JMP | BPF_JSGE | BPF_X:
case BPF_JMP | BPF_JSLT | BPF_X:
case BPF_JMP | BPF_JSLE | BPF_X:
+ jmp_offset = bpf2la_offset(i, off, ctx);
+ if (emit_cond_jmp(ctx, cond, dst, src, jmp_offset) < 0)
+ goto toofar;
+ break;
+
+ /* PC += off if dst cond src (32-bit unsigned) */
case BPF_JMP32 | BPF_JEQ | BPF_X:
case BPF_JMP32 | BPF_JNE | BPF_X:
case BPF_JMP32 | BPF_JGT | BPF_X:
case BPF_JMP32 | BPF_JGE | BPF_X:
case BPF_JMP32 | BPF_JLT | BPF_X:
case BPF_JMP32 | BPF_JLE | BPF_X:
+ jmp_offset = bpf2la_offset(i, off, ctx);
+ emit_insn(ctx, bstrpickd, t1, dst, 31, 0);
+ emit_insn(ctx, bstrpickd, t2, src, 31, 0);
+ if (emit_cond_jmp(ctx, cond, t1, t2, jmp_offset) < 0)
+ goto toofar;
+ break;
+
+ /* PC += off if dst cond src (32-bit signed) */
case BPF_JMP32 | BPF_JSGT | BPF_X:
case BPF_JMP32 | BPF_JSGE | BPF_X:
case BPF_JMP32 | BPF_JSLT | BPF_X:
case BPF_JMP32 | BPF_JSLE | BPF_X:
jmp_offset = bpf2la_offset(i, off, ctx);
- move_reg(ctx, t1, dst);
- move_reg(ctx, t2, src);
- if (is_signed_bpf_cond(BPF_OP(code))) {
- emit_sext_32(ctx, t1, is32);
- emit_sext_32(ctx, t2, is32);
- } else {
- emit_zext_32(ctx, t1, is32);
- emit_zext_32(ctx, t2, is32);
- }
+ emit_insn(ctx, addw, t1, dst, LOONGARCH_GPR_ZERO);
+ emit_insn(ctx, addw, t2, src, LOONGARCH_GPR_ZERO);
if (emit_cond_jmp(ctx, cond, t1, t2, jmp_offset) < 0)
goto toofar;
break;
- /* PC += off if dst cond imm */
+ /* PC += off if dst cond imm (64-bit) */
case BPF_JMP | BPF_JEQ | BPF_K:
case BPF_JMP | BPF_JNE | BPF_K:
case BPF_JMP | BPF_JGT | BPF_K:
@@ -1094,12 +1112,38 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
case BPF_JMP | BPF_JSGE | BPF_K:
case BPF_JMP | BPF_JSLT | BPF_K:
case BPF_JMP | BPF_JSLE | BPF_K:
+ jmp_offset = bpf2la_offset(i, off, ctx);
+ if (imm) {
+ move_imm(ctx, t1, imm);
+ tm = t1;
+ } else {
+ tm = LOONGARCH_GPR_ZERO;
+ }
+ if (emit_cond_jmp(ctx, cond, dst, tm, jmp_offset) < 0)
+ goto toofar;
+ break;
+
+ /* PC += off if dst cond imm (32-bit unsigned) */
case BPF_JMP32 | BPF_JEQ | BPF_K:
case BPF_JMP32 | BPF_JNE | BPF_K:
case BPF_JMP32 | BPF_JGT | BPF_K:
case BPF_JMP32 | BPF_JGE | BPF_K:
case BPF_JMP32 | BPF_JLT | BPF_K:
case BPF_JMP32 | BPF_JLE | BPF_K:
+ jmp_offset = bpf2la_offset(i, off, ctx);
+ if (imm) {
+ move_imm(ctx, t1, imm);
+ emit_insn(ctx, bstrpickd, t1, t1, 31, 0);
+ tm = t1;
+ } else {
+ tm = LOONGARCH_GPR_ZERO;
+ }
+ emit_insn(ctx, bstrpickd, t2, dst, 31, 0);
+ if (emit_cond_jmp(ctx, cond, t2, tm, jmp_offset) < 0)
+ goto toofar;
+ break;
+
+ /* PC += off if dst cond imm (32-bit signed) */
case BPF_JMP32 | BPF_JSGT | BPF_K:
case BPF_JMP32 | BPF_JSGE | BPF_K:
case BPF_JMP32 | BPF_JSLT | BPF_K:
@@ -1107,19 +1151,12 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
jmp_offset = bpf2la_offset(i, off, ctx);
if (imm) {
move_imm(ctx, t1, imm);
+ emit_insn(ctx, addw, t1, t1, LOONGARCH_GPR_ZERO);
tm = t1;
} else {
- /* If imm is 0, simply use zero register. */
tm = LOONGARCH_GPR_ZERO;
}
- move_reg(ctx, t2, dst);
- if (is_signed_bpf_cond(BPF_OP(code))) {
- emit_sext_32(ctx, tm, is32);
- emit_sext_32(ctx, t2, is32);
- } else {
- emit_zext_32(ctx, tm, is32);
- emit_zext_32(ctx, t2, is32);
- }
+ emit_insn(ctx, addw, t2, dst, LOONGARCH_GPR_ZERO);
if (emit_cond_jmp(ctx, cond, t2, tm, jmp_offset) < 0)
goto toofar;
break;
@@ -1129,7 +1166,8 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
case BPF_JMP32 | BPF_JSET | BPF_X:
jmp_offset = bpf2la_offset(i, off, ctx);
emit_insn(ctx, and, t1, dst, src);
- emit_zext_32(ctx, t1, is32);
+ if (BPF_CLASS(code) == BPF_JMP32)
+ emit_insn(ctx, bstrpickd, t1, t1, 31, 0);
if (emit_cond_jmp(ctx, cond, t1, LOONGARCH_GPR_ZERO, jmp_offset) < 0)
goto toofar;
break;
@@ -1139,9 +1177,9 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
case BPF_JMP32 | BPF_JSET | BPF_K:
jmp_offset = bpf2la_offset(i, off, ctx);
move_imm(ctx, t1, imm);
- emit_zext_32(ctx, t1, is32);
emit_insn(ctx, and, t1, dst, t1);
- emit_zext_32(ctx, t1, is32);
+ if (BPF_CLASS(code) == BPF_JMP32)
+ emit_insn(ctx, bstrpickd, t1, t1, 31, 0);
if (emit_cond_jmp(ctx, cond, t1, LOONGARCH_GPR_ZERO, jmp_offset) < 0)
goto toofar;
break;
@@ -1468,6 +1506,9 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
return -EINVAL;
}
+ if (is32)
+ emit_insn(ctx, bstrpickd, dst, dst, 31, 0);
+
return 0;
toofar:
diff --git a/arch/loongarch/net/bpf_jit.h b/arch/loongarch/net/bpf_jit.h
index 57a98f8a055b..2bd0dc5b911c 100644
--- a/arch/loongarch/net/bpf_jit.h
+++ b/arch/loongarch/net/bpf_jit.h
@@ -72,24 +72,6 @@ static inline int epilogue_offset(const struct jit_ctx *ctx)
return (to - from);
}
-/* Zero-extend 32 bits into 64 bits */
-static inline void emit_zext_32(struct jit_ctx *ctx, enum loongarch_gpr reg, bool is32)
-{
- if (!is32)
- return;
-
- emit_insn(ctx, lu32id, reg, 0);
-}
-
-/* Signed-extend 32 bits into 64 bits */
-static inline void emit_sext_32(struct jit_ctx *ctx, enum loongarch_gpr reg, bool is32)
-{
- if (!is32)
- return;
-
- emit_insn(ctx, addiw, reg, reg, 0);
-}
-
/* Emit proper extension according to ABI requirements.
* Note that it requires a value of size `size` already resides in register `reg`.
*/
--
2.42.0
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v3 3/3] LoongArch: BPF: Optimize 32-bit operations using native instructions
2026-10-16 8:10 ` [PATCH v3 3/3] LoongArch: BPF: Optimize 32-bit operations using native instructions Tiezhu Yang
@ 2026-10-16 15:18 ` Alexei Starovoitov
2026-10-17 10:58 ` Tiezhu Yang
2026-10-27 2:02 ` Hengqi Chen
2026-10-28 4:19 ` Huacai Chen
2 siblings, 1 reply; 15+ messages in thread
From: Alexei Starovoitov @ 2026-10-16 15:18 UTC (permalink / raw)
To: Tiezhu Yang, Huacai Chen, Hengqi Chen; +Cc: loongarch, bpf, linux-kernel
On Fri, Oct 16, 2026 at 04:10 PM Tiezhu Yang <yangtiezhu@loongson.cn> wrote:
> All of the testcases of test_bpf passed on Loongson-3A6000 (with DIV32)
> and Loongson-3A5000 (without DIV32), here are the test results:
fyi test_bpf is gone. See
commit 4532e525b286 ("selftests/bpf: Remove the old test_bpf module").
Pls run test_progs
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v3 3/3] LoongArch: BPF: Optimize 32-bit operations using native instructions
2026-10-16 15:18 ` Alexei Starovoitov
@ 2026-10-17 10:58 ` Tiezhu Yang
0 siblings, 0 replies; 15+ messages in thread
From: Tiezhu Yang @ 2026-10-17 10:58 UTC (permalink / raw)
To: Alexei Starovoitov, Huacai Chen, Hengqi Chen; +Cc: loongarch, bpf, linux-kernel
On 2026/10/16 下午11:18, Alexei Starovoitov wrote:
> On Fri, Oct 16, 2026 at 04:10 PM Tiezhu Yang <yangtiezhu@loongson.cn> wrote:
>
>> All of the testcases of test_bpf passed on Loongson-3A6000 (with DIV32)
>> and Loongson-3A5000 (without DIV32), here are the test results:
>
> fyi test_bpf is gone. See
> commit 4532e525b286 ("selftests/bpf: Remove the old test_bpf module").
> Pls run test_progs
I tested this series based on bpf-next tree with "./test_progs -a jit_insns"
and "./test_progs -a jit_insns_blinded", there are 3 pre-existing failures:
(1) #191/1 jit_insns/arena access with zero offset:FAIL
(2) #191/25 jit_insns/ldst_offsets:FAIL
(3) #192/11 jit_insns_blinded/ldst_offsets:FAIL
These 3 failures are entirely unrelated to this series; they still fail on
the clean, original bpf-next tree without this series.
The root cause is that LoongArch BPF JIT support for arena atomics and
sign-extending loads from arena is still a work-in-progress.
IIUC, there are already two patches to fix them, but need revisions:
https://lore.kernel.org/loongarch/20260702022322.51033-8-dongtai.guo@linux.dev/
https://lore.kernel.org/loongarch/20260702022322.51033-7-dongtai.guo@linux.dev/
Bypassing these known architectural gaps using the blacklist (-b) option
allows the remaining subtests under jit_insns and jit_insns_blinded to pass:
$ sudo ./test_progs -a jit_insns -b "jit_insns/arena access with zero
offset,jit_insns/ldst_offsets"
#191/2 jit_insns/cast to a user address:OK
#191/3 jit_insns/alu_imm:OK
#191/4 jit_insns/alu_reg:OK
#191/5 jit_insns/alu_imm_regs:OK
#191/6 jit_insns/div_const:OK
#191/7 jit_insns/alu_reg_pairs:OK
#191/8 jit_insns/alu_edges:OK
#191/9 jit_insns/shifts:OK
#191/10 jit_insns/shift_counts:OK
#191/11 jit_insns/atomics:OK
#191/12 jit_insns/atomic_reg_pairs:OK
#191/13 jit_insns/xadd_chain:OK
#191/14 jit_insns/ld_imm64:OK
#191/15 jit_insns/jmp_imm:OK
#191/16 jit_insns/jmp_reg:OK
#191/17 jit_insns/jmp_reg_pairs:OK
#191/18 jit_insns/jmp_edges:OK
#191/19 jit_insns/staggered_jumps:OK
#191/20 jit_insns/max_jmp:OK
#191/21 jit_insns/convergence:OK
#191/22 jit_insns/calls:OK
#191/23 jit_insns/helper_calls:OK
#191/24 jit_insns/ldst_zero_off:OK
#191/26 jit_insns/acq_rel:OK
#191/27 jit_insns/movsx:OK
#191/28 jit_insns/jset_imm:OK
#191/29 jit_insns/cmp_imm:OK
#191/30 jit_insns/alu_consts:OK
#191/31 jit_insns/bswap:OK
#191/32 jit_insns/probe_mem:OK
#191 jit_insns:OK
Summary: 1/30 PASSED, 0 SKIPPED, 0/0 FAILED
$ sudo ./test_progs -a jit_insns_blinded -b jit_insns_blinded/ldst_offsets
#192/1 jit_insns_blinded/alu_imm:OK
#192/2 jit_insns_blinded/alu_imm_regs:OK
#192/3 jit_insns_blinded/alu_edges:OK
#192/4 jit_insns_blinded/div_const:OK
#192/5 jit_insns_blinded/shifts:OK
#192/6 jit_insns_blinded/xadd_chain:OK
#192/7 jit_insns_blinded/ld_imm64:OK
#192/8 jit_insns_blinded/jmp_imm:OK
#192/9 jit_insns_blinded/jmp_edges:OK
#192/10 jit_insns_blinded/ldst_zero_off:OK
#192/12 jit_insns_blinded/jset_imm:OK
#192/13 jit_insns_blinded/cmp_imm:OK
#192/14 jit_insns_blinded/alu_consts:OK
#192 jit_insns_blinded:OK
Summary: 1/13 PASSED, 0 SKIPPED, 0/0 FAILED
Thanks,
Tiezhu
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v3 1/3] LoongArch: Add CPUCFG2 DIV32 feature detection
2026-10-16 8:10 ` [PATCH v3 1/3] LoongArch: Add CPUCFG2 DIV32 feature detection Tiezhu Yang
@ 2026-10-20 10:48 ` Huacai Chen
0 siblings, 0 replies; 15+ messages in thread
From: Huacai Chen @ 2026-10-20 10:48 UTC (permalink / raw)
To: Tiezhu Yang; +Cc: Hengqi Chen, loongarch, bpf, linux-kernel
This one is applied, thanks.
Huacai
On Fri, Oct 16, 2026 at 4:10 PM Tiezhu Yang <yangtiezhu@loongson.cn> wrote:
>
> The LoongArch Architecture Manual defines a hardware capability bit
> CPUCFG2_DIV32 (bit 26 of CPUCFG 0x2). When this bit is set to 1, it
> indicates that on 64-bit machines, the division and mod instructions
> (DIV.W[U] and MOD.W[U]) will ignore the upper 32 bits of the input
> registers and only compute based on the lower 32 bits.
>
> On certain CPU cores without this feature, these 32-bit instructions
> require both input operands to be properly sign-extended. Otherwise,
> the hardware execution results can be randomly meaningless values.
>
> Add CPU_FEATURE_DIV32 to detect and export this hardware capability
> via elf_hwcap and cpu_has_div32. This provides the infrastructure
> for subsystems like BPF JIT to safely choose either native 32-bit
> instructions or fallback instruction sequences based on the actual
> CPU hardware capabilities.
>
> Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
> ---
> arch/loongarch/include/asm/cpu-features.h | 1 +
> arch/loongarch/include/asm/cpu.h | 2 ++
> arch/loongarch/include/uapi/asm/hwcap.h | 1 +
> arch/loongarch/kernel/cpu-probe.c | 4 ++++
> arch/loongarch/kernel/proc.c | 2 ++
> 5 files changed, 10 insertions(+)
>
> diff --git a/arch/loongarch/include/asm/cpu-features.h b/arch/loongarch/include/asm/cpu-features.h
> index 62059c5551b9..3e7cf1987171 100644
> --- a/arch/loongarch/include/asm/cpu-features.h
> +++ b/arch/loongarch/include/asm/cpu-features.h
> @@ -69,5 +69,6 @@
> #define cpu_has_msgint cpu_opt(LOONGARCH_CPU_MSGINT)
> #define cpu_has_avecint cpu_opt(LOONGARCH_CPU_AVECINT)
> #define cpu_has_redirectint cpu_opt(LOONGARCH_CPU_REDIRECTINT)
> +#define cpu_has_div32 cpu_opt(LOONGARCH_CPU_DIV32)
>
> #endif /* __ASM_CPU_FEATURES_H */
> diff --git a/arch/loongarch/include/asm/cpu.h b/arch/loongarch/include/asm/cpu.h
> index 91b96938861e..d1cf5dd5eceb 100644
> --- a/arch/loongarch/include/asm/cpu.h
> +++ b/arch/loongarch/include/asm/cpu.h
> @@ -127,6 +127,7 @@ static inline char *id_to_core_name(unsigned int id)
> #define CPU_FEATURE_MSGINT 31 /* CPU has MSG interrupt */
> #define CPU_FEATURE_AVECINT 32 /* CPU has AVEC interrupt */
> #define CPU_FEATURE_REDIRECTINT 33 /* CPU has interrupt remapping */
> +#define CPU_FEATURE_DIV32 34 /* CPU has DIV32 feature */
>
> #define LOONGARCH_CPU_CPUCFG BIT_ULL(CPU_FEATURE_CPUCFG)
> #define LOONGARCH_CPU_LAM BIT_ULL(CPU_FEATURE_LAM)
> @@ -162,5 +163,6 @@ static inline char *id_to_core_name(unsigned int id)
> #define LOONGARCH_CPU_MSGINT BIT_ULL(CPU_FEATURE_MSGINT)
> #define LOONGARCH_CPU_AVECINT BIT_ULL(CPU_FEATURE_AVECINT)
> #define LOONGARCH_CPU_REDIRECTINT BIT_ULL(CPU_FEATURE_REDIRECTINT)
> +#define LOONGARCH_CPU_DIV32 BIT_ULL(CPU_FEATURE_DIV32)
>
> #endif /* _ASM_CPU_H */
> diff --git a/arch/loongarch/include/uapi/asm/hwcap.h b/arch/loongarch/include/uapi/asm/hwcap.h
> index 90e96113ba51..00bf93683c74 100644
> --- a/arch/loongarch/include/uapi/asm/hwcap.h
> +++ b/arch/loongarch/include/uapi/asm/hwcap.h
> @@ -20,5 +20,6 @@
> #define HWCAP_LOONGARCH_LSPW (1 << 14)
> #define HWCAP_LOONGARCH_SCQ (1 << 15)
> #define HWCAP_LOONGARCH_LAM_BH (1 << 16)
> +#define HWCAP_LOONGARCH_DIV32 (1 << 17)
>
> #endif /* _UAPI_ASM_HWCAP_H */
> diff --git a/arch/loongarch/kernel/cpu-probe.c b/arch/loongarch/kernel/cpu-probe.c
> index 74d31f260dfd..c50fcba805f7 100644
> --- a/arch/loongarch/kernel/cpu-probe.c
> +++ b/arch/loongarch/kernel/cpu-probe.c
> @@ -236,6 +236,10 @@ static void cpu_probe_common(struct cpuinfo_loongarch *c)
> elf_hwcap |= HWCAP_LOONGARCH_LBT_MIPS;
> }
> #endif
> + if (config & CPUCFG2_DIV32) {
> + c->options |= LOONGARCH_CPU_DIV32;
> + elf_hwcap |= HWCAP_LOONGARCH_DIV32;
> + }
>
> config = read_cpucfg(LOONGARCH_CPUCFG6);
> if (config & CPUCFG6_PMP)
> diff --git a/arch/loongarch/kernel/proc.c b/arch/loongarch/kernel/proc.c
> index d4ce5b585453..c8d6f397de55 100644
> --- a/arch/loongarch/kernel/proc.c
> +++ b/arch/loongarch/kernel/proc.c
> @@ -94,6 +94,8 @@ static int show_cpuinfo(struct seq_file *m, void *v)
> seq_puts(m, " lbt_arm");
> if (cpu_has_lbt_mips)
> seq_puts(m, " lbt_mips");
> + if (cpu_has_div32)
> + seq_puts(m, " div32");
> seq_puts(m, "\n");
>
> seq_printf(m, "Hardware Watchpoint\t: %s", str_yes_no(cpu_has_watch));
> --
> 2.42.0
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v3 0/3] LoongArch: BPF: Refine move_imm() and optimize 32-bit operations
2026-10-16 8:10 [PATCH v3 0/3] LoongArch: BPF: Refine move_imm() and optimize 32-bit operations Tiezhu Yang
` (2 preceding siblings ...)
2026-10-16 8:10 ` [PATCH v3 3/3] LoongArch: BPF: Optimize 32-bit operations using native instructions Tiezhu Yang
@ 2026-10-22 7:43 ` Tiezhu Yang
2026-10-23 8:42 ` Huacai Chen
3 siblings, 1 reply; 15+ messages in thread
From: Tiezhu Yang @ 2026-10-22 7:43 UTC (permalink / raw)
To: Huacai Chen, Hengqi Chen; +Cc: loongarch, bpf, linux-kernel
On 2026/10/16 下午4:10, Tiezhu Yang wrote:
> v3:
> -- Add a prerequisite patch to detect CPUCFG2_DIV32 feature.
> -- Fix a hardware compatibility bug by checking cpu_has_div32
> before emitting native 32-bit div/mod instructions.
>
> This version is based on v7.3-rc7 and has been tested on Loongson-3A6000
> and Loongson-3A5000 physical machines.
>
> All of the testcases of test_bpf passed on Loongson-3A6000 (with DIV32)
> and Loongson-3A5000 (without DIV32), here are the test results:
>
> $ sudo modprobe test_bpf
> $ dmesg -t | grep Summary
> test_bpf: Summary: 1061 PASSED, 0 FAILED, [1049/1049 JIT'ed]
> test_bpf: test_tail_calls: Summary: 10 PASSED, 0 FAILED, [10/10 JIT'ed]
> test_bpf: test_skb_segment: Summary: 2 PASSED, 0 FAILED
>
> For the BPF selftests test_verifier and test_progs, there are no any
> regressions.
>
> Tiezhu Yang (3):
> LoongArch: Add CPUCFG2 DIV32 feature detection
> LoongArch: BPF: Clean up and decouple is32 parameter from move_imm()
> LoongArch: BPF: Optimize 32-bit operations using native instructions
Hi Huacai, Hengqi,
A gentle ping on the last 2 patches of this series since patch 1
has been applied.
There are no any reports from bot+bpf-ci and sashiko-bot,
and also "test_bpf" and "./test_progs -t jit_insns" tests passed.
Just to clarify the testing status: the 3 failures mentioned in
the previous test report are entirely pre-existing on the clean
bpf-next tree and completely unrelated to this 32-bit optimization
series.
The core ALU32/JMP32 paths introduced here have been fully verified
and passed without any regressions on both 3A6000 and 3A5000 physical
machines.
Do you have any further comments or objections to patches 2 and 3?
If these look good to you, it would be great to have them merged to
avoid rebasing and resending these patches after this merge window.
Thanks,
Tiezhu
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v3 0/3] LoongArch: BPF: Refine move_imm() and optimize 32-bit operations
2026-10-22 7:43 ` [PATCH v3 0/3] LoongArch: BPF: Refine move_imm() and optimize 32-bit operations Tiezhu Yang
@ 2026-10-23 8:42 ` Huacai Chen
0 siblings, 0 replies; 15+ messages in thread
From: Huacai Chen @ 2026-10-23 8:42 UTC (permalink / raw)
To: Tiezhu Yang; +Cc: Hengqi Chen, loongarch, bpf, linux-kernel
On Thu, Oct 22, 2026 at 3:44 PM Tiezhu Yang <yangtiezhu@loongson.cn> wrote:
>
> On 2026/10/16 下午4:10, Tiezhu Yang wrote:
> > v3:
> > -- Add a prerequisite patch to detect CPUCFG2_DIV32 feature.
> > -- Fix a hardware compatibility bug by checking cpu_has_div32
> > before emitting native 32-bit div/mod instructions.
> >
> > This version is based on v7.3-rc7 and has been tested on Loongson-3A6000
> > and Loongson-3A5000 physical machines.
> >
> > All of the testcases of test_bpf passed on Loongson-3A6000 (with DIV32)
> > and Loongson-3A5000 (without DIV32), here are the test results:
> >
> > $ sudo modprobe test_bpf
> > $ dmesg -t | grep Summary
> > test_bpf: Summary: 1061 PASSED, 0 FAILED, [1049/1049 JIT'ed]
> > test_bpf: test_tail_calls: Summary: 10 PASSED, 0 FAILED, [10/10 JIT'ed]
> > test_bpf: test_skb_segment: Summary: 2 PASSED, 0 FAILED
> >
> > For the BPF selftests test_verifier and test_progs, there are no any
> > regressions.
> >
> > Tiezhu Yang (3):
> > LoongArch: Add CPUCFG2 DIV32 feature detection
> > LoongArch: BPF: Clean up and decouple is32 parameter from move_imm()
> > LoongArch: BPF: Optimize 32-bit operations using native instructions
>
> Hi Huacai, Hengqi,
>
> A gentle ping on the last 2 patches of this series since patch 1
> has been applied.
>
> There are no any reports from bot+bpf-ci and sashiko-bot,
> and also "test_bpf" and "./test_progs -t jit_insns" tests passed.
>
> Just to clarify the testing status: the 3 failures mentioned in
> the previous test report are entirely pre-existing on the clean
> bpf-next tree and completely unrelated to this 32-bit optimization
> series.
>
> The core ALU32/JMP32 paths introduced here have been fully verified
> and passed without any regressions on both 3A6000 and 3A5000 physical
> machines.
>
> Do you have any further comments or objections to patches 2 and 3?
> If these look good to you, it would be great to have them merged to
> avoid rebasing and resending these patches after this merge window.
I hope Hengqi can take a look before applying.
Huacai
>
> Thanks,
> Tiezhu
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v3 2/3] LoongArch: BPF: Clean up and decouple is32 parameter from move_imm()
2026-10-16 8:10 ` [PATCH v3 2/3] LoongArch: BPF: Clean up and decouple is32 parameter from move_imm() Tiezhu Yang
@ 2026-10-27 1:36 ` Hengqi Chen
0 siblings, 0 replies; 15+ messages in thread
From: Hengqi Chen @ 2026-10-27 1:36 UTC (permalink / raw)
To: Tiezhu Yang; +Cc: Huacai Chen, loongarch, bpf, linux-kernel
On Fri, Oct 16, 2026 at 4:10 PM Tiezhu Yang <yangtiezhu@loongson.cn> wrote:
>
> The current move_imm() relies on the 'is32' parameter for ALU32 and JMP32,
> forcing the helper to internally clear the upper 32 bits. However, this is
> redundant for 64-bit immediate loads. Furthermore, it is even undesirable
> for load/store signed address offset computations.
>
> Clean up move_imm() by removing the 'is32' parameter which allows it to
> focus on loading values, while moving the zero-extension logic outward to
> the explicit callers. For 64-bit and load-store operations, just drop the
> parameter.
>
> This is preparation for later patch, no functional changes.
>
> Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
> ---
Acked-by: Hengqi Chen <hengqi.chen@gmail.com>
> arch/loongarch/net/bpf_jit.c | 102 +++++++++++++++++++----------------
> arch/loongarch/net/bpf_jit.h | 7 +--
> 2 files changed, 58 insertions(+), 51 deletions(-)
>
> diff --git a/arch/loongarch/net/bpf_jit.c b/arch/loongarch/net/bpf_jit.c
> index e78eb582c300..77cd96a6f84c 100644
> --- a/arch/loongarch/net/bpf_jit.c
> +++ b/arch/loongarch/net/bpf_jit.c
> @@ -194,7 +194,7 @@ static void build_prologue(struct jit_ctx *ctx)
> ctx->stack_size = stack_adjust;
>
> if (ctx->arena_vm_start)
> - move_imm(ctx, REG_ARENA, ctx->arena_vm_start, false);
> + move_imm(ctx, REG_ARENA, ctx->arena_vm_start);
> }
>
> static void __build_epilogue(struct jit_ctx *ctx, bool is_tail_call)
> @@ -339,7 +339,7 @@ static int emit_bpf_tail_call(struct jit_ctx *ctx, int insn)
>
> static void emit_store_stack_imm64(struct jit_ctx *ctx, int reg, int stack_off, u64 imm64)
> {
> - move_imm(ctx, reg, imm64, false);
> + move_imm(ctx, reg, imm64);
> emit_insn(ctx, std, reg, LOONGARCH_GPR_FP, stack_off);
> }
>
> @@ -355,7 +355,7 @@ static int emit_atomic_rmw(const struct bpf_insn *insn, struct jit_ctx *ctx)
> const s32 imm = insn->imm;
> const bool isdw = BPF_SIZE(insn->code) == BPF_DW;
>
> - move_imm(ctx, t1, off, false);
> + move_imm(ctx, t1, off);
> emit_insn(ctx, addd, t1, dst, t1);
> move_reg(ctx, t3, src);
>
> @@ -528,7 +528,7 @@ static int emit_atomic_ld_st(const struct bpf_insn *insn, struct jit_ctx *ctx)
> if (is_signed_imm12(off)) {
> emit_insn(ctx, ldbu, dst, src, off);
> } else {
> - move_imm(ctx, t1, off, false);
> + move_imm(ctx, t1, off);
> emit_insn(ctx, ldxbu, dst, src, t1);
> }
> break;
> @@ -536,7 +536,7 @@ static int emit_atomic_ld_st(const struct bpf_insn *insn, struct jit_ctx *ctx)
> if (is_signed_imm12(off)) {
> emit_insn(ctx, ldhu, dst, src, off);
> } else {
> - move_imm(ctx, t1, off, false);
> + move_imm(ctx, t1, off);
> emit_insn(ctx, ldxhu, dst, src, t1);
> }
> break;
> @@ -544,7 +544,7 @@ static int emit_atomic_ld_st(const struct bpf_insn *insn, struct jit_ctx *ctx)
> if (is_signed_imm12(off)) {
> emit_insn(ctx, ldwu, dst, src, off);
> } else {
> - move_imm(ctx, t1, off, false);
> + move_imm(ctx, t1, off);
> emit_insn(ctx, ldxwu, dst, src, t1);
> }
> break;
> @@ -552,7 +552,7 @@ static int emit_atomic_ld_st(const struct bpf_insn *insn, struct jit_ctx *ctx)
> if (is_signed_imm12(off)) {
> emit_insn(ctx, ldd, dst, src, off);
> } else {
> - move_imm(ctx, t1, off, false);
> + move_imm(ctx, t1, off);
> emit_insn(ctx, ldxd, dst, src, t1);
> }
> break;
> @@ -567,7 +567,7 @@ static int emit_atomic_ld_st(const struct bpf_insn *insn, struct jit_ctx *ctx)
> if (is_signed_imm12(off)) {
> emit_insn(ctx, stb, src, dst, off);
> } else {
> - move_imm(ctx, t1, off, false);
> + move_imm(ctx, t1, off);
> emit_insn(ctx, stxb, src, dst, t1);
> }
> break;
> @@ -575,7 +575,7 @@ static int emit_atomic_ld_st(const struct bpf_insn *insn, struct jit_ctx *ctx)
> if (is_signed_imm12(off)) {
> emit_insn(ctx, sth, src, dst, off);
> } else {
> - move_imm(ctx, t1, off, false);
> + move_imm(ctx, t1, off);
> emit_insn(ctx, stxh, src, dst, t1);
> }
> break;
> @@ -583,7 +583,7 @@ static int emit_atomic_ld_st(const struct bpf_insn *insn, struct jit_ctx *ctx)
> if (is_signed_imm12(off)) {
> emit_insn(ctx, stw, src, dst, off);
> } else {
> - move_imm(ctx, t1, off, false);
> + move_imm(ctx, t1, off);
> emit_insn(ctx, stxw, src, dst, t1);
> }
> break;
> @@ -591,7 +591,7 @@ static int emit_atomic_ld_st(const struct bpf_insn *insn, struct jit_ctx *ctx)
> if (is_signed_imm12(off)) {
> emit_insn(ctx, std, src, dst, off);
> } else {
> - move_imm(ctx, t1, off, false);
> + move_imm(ctx, t1, off);
> emit_insn(ctx, stxd, src, dst, t1);
> }
> break;
> @@ -716,7 +716,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> if (insn_is_cast_user(insn)) {
> move_reg(ctx, t1, src);
> emit_zext_32(ctx, t1, true);
> - move_imm(ctx, dst, (ctx->user_vm_start >> 32) << 32, false);
> + move_imm(ctx, dst, (ctx->user_vm_start >> 32) << 32);
> emit_insn(ctx, beq, t1, LOONGARCH_GPR_ZERO, 2);
> emit_insn(ctx, or, t1, dst, t1);
> move_reg(ctx, dst, t1);
> @@ -753,7 +753,8 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> /* dst = imm */
> case BPF_ALU | BPF_MOV | BPF_K:
> case BPF_ALU64 | BPF_MOV | BPF_K:
> - move_imm(ctx, dst, imm, is32);
> + move_imm(ctx, dst, imm);
> + emit_zext_32(ctx, dst, is32);
> break;
>
> /* dst = dst + src */
> @@ -769,7 +770,8 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> if (is_signed_imm12(imm)) {
> emit_insn(ctx, addid, dst, dst, imm);
> } else {
> - move_imm(ctx, t1, imm, is32);
> + move_imm(ctx, t1, imm);
> + emit_zext_32(ctx, t1, is32);
> emit_insn(ctx, addd, dst, dst, t1);
> }
> emit_zext_32(ctx, dst, is32);
> @@ -788,7 +790,8 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> if (is_signed_imm12(-imm)) {
> emit_insn(ctx, addid, dst, dst, -imm);
> } else {
> - move_imm(ctx, t1, imm, is32);
> + move_imm(ctx, t1, imm);
> + emit_zext_32(ctx, t1, is32);
> emit_insn(ctx, subd, dst, dst, t1);
> }
> emit_zext_32(ctx, dst, is32);
> @@ -804,7 +807,8 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> /* dst = dst * imm */
> case BPF_ALU | BPF_MUL | BPF_K:
> case BPF_ALU64 | BPF_MUL | BPF_K:
> - move_imm(ctx, t1, imm, is32);
> + move_imm(ctx, t1, imm);
> + emit_zext_32(ctx, t1, is32);
> emit_insn(ctx, muld, dst, dst, t1);
> emit_zext_32(ctx, dst, is32);
> break;
> @@ -831,12 +835,13 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> case BPF_ALU | BPF_DIV | BPF_K:
> case BPF_ALU64 | BPF_DIV | BPF_K:
> if (!off) {
> - move_imm(ctx, t1, imm, is32);
> + move_imm(ctx, t1, imm);
> + emit_zext_32(ctx, t1, is32);
> emit_zext_32(ctx, dst, is32);
> emit_insn(ctx, divdu, dst, dst, t1);
> emit_zext_32(ctx, dst, is32);
> } else {
> - move_imm(ctx, t1, imm, false);
> + move_imm(ctx, t1, imm);
> emit_sext_32(ctx, t1, is32);
> emit_sext_32(ctx, dst, is32);
> emit_insn(ctx, divd, dst, dst, t1);
> @@ -866,12 +871,13 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> case BPF_ALU | BPF_MOD | BPF_K:
> case BPF_ALU64 | BPF_MOD | BPF_K:
> if (!off) {
> - move_imm(ctx, t1, imm, is32);
> + move_imm(ctx, t1, imm);
> + emit_zext_32(ctx, t1, is32);
> emit_zext_32(ctx, dst, is32);
> emit_insn(ctx, moddu, dst, dst, t1);
> emit_zext_32(ctx, dst, is32);
> } else {
> - move_imm(ctx, t1, imm, false);
> + move_imm(ctx, t1, imm);
> emit_sext_32(ctx, t1, is32);
> emit_sext_32(ctx, dst, is32);
> emit_insn(ctx, modd, dst, dst, t1);
> @@ -899,7 +905,8 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> if (is_unsigned_imm12(imm)) {
> emit_insn(ctx, andi, dst, dst, imm);
> } else {
> - move_imm(ctx, t1, imm, is32);
> + move_imm(ctx, t1, imm);
> + emit_zext_32(ctx, t1, is32);
> emit_insn(ctx, and, dst, dst, t1);
> }
> emit_zext_32(ctx, dst, is32);
> @@ -918,7 +925,8 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> if (is_unsigned_imm12(imm)) {
> emit_insn(ctx, ori, dst, dst, imm);
> } else {
> - move_imm(ctx, t1, imm, is32);
> + move_imm(ctx, t1, imm);
> + emit_zext_32(ctx, t1, is32);
> emit_insn(ctx, or, dst, dst, t1);
> }
> emit_zext_32(ctx, dst, is32);
> @@ -937,7 +945,8 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> if (is_unsigned_imm12(imm)) {
> emit_insn(ctx, xori, dst, dst, imm);
> } else {
> - move_imm(ctx, t1, imm, is32);
> + move_imm(ctx, t1, imm);
> + emit_zext_32(ctx, t1, is32);
> emit_insn(ctx, xor, dst, dst, t1);
> }
> emit_zext_32(ctx, dst, is32);
> @@ -1097,7 +1106,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> case BPF_JMP32 | BPF_JSLE | BPF_K:
> jmp_offset = bpf2la_offset(i, off, ctx);
> if (imm) {
> - move_imm(ctx, t1, imm, false);
> + move_imm(ctx, t1, imm);
> tm = t1;
> } else {
> /* If imm is 0, simply use zero register. */
> @@ -1129,7 +1138,8 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> case BPF_JMP | BPF_JSET | BPF_K:
> case BPF_JMP32 | BPF_JSET | BPF_K:
> jmp_offset = bpf2la_offset(i, off, ctx);
> - move_imm(ctx, t1, imm, is32);
> + move_imm(ctx, t1, imm);
> + emit_zext_32(ctx, t1, is32);
> emit_insn(ctx, and, t1, dst, t1);
> emit_zext_32(ctx, t1, is32);
> if (emit_cond_jmp(ctx, cond, t1, LOONGARCH_GPR_ZERO, jmp_offset) < 0)
> @@ -1227,7 +1237,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> if (bpf_pseudo_func(insn))
> move_addr(ctx, dst, imm64);
> else
> - move_imm(ctx, dst, imm64, is32);
> + move_imm(ctx, dst, imm64);
> return 1;
> }
>
> @@ -1268,7 +1278,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> else
> emit_insn(ctx, ldbu, dst, src, off);
> } else {
> - move_imm(ctx, t1, off, is32);
> + move_imm(ctx, t1, off);
> if (sign_extend)
> emit_insn(ctx, ldxb, dst, src, t1);
> else
> @@ -1282,7 +1292,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> else
> emit_insn(ctx, ldhu, dst, src, off);
> } else {
> - move_imm(ctx, t1, off, is32);
> + move_imm(ctx, t1, off);
> if (sign_extend)
> emit_insn(ctx, ldxh, dst, src, t1);
> else
> @@ -1296,7 +1306,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> else
> emit_insn(ctx, ldwu, dst, src, off);
> } else {
> - move_imm(ctx, t1, off, is32);
> + move_imm(ctx, t1, off);
> if (sign_extend)
> emit_insn(ctx, ldxw, dst, src, t1);
> else
> @@ -1304,7 +1314,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> }
> break;
> case BPF_DW:
> - move_imm(ctx, t1, off, is32);
> + move_imm(ctx, t1, off);
> emit_insn(ctx, ldxd, dst, src, t1);
> break;
> }
> @@ -1331,42 +1341,42 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
>
> switch (BPF_SIZE(code)) {
> case BPF_B:
> - move_imm(ctx, t1, imm, is32);
> + move_imm(ctx, t1, imm);
> if (is_signed_imm12(off)) {
> emit_insn(ctx, stb, t1, dst, off);
> } else {
> - move_imm(ctx, t2, off, is32);
> + move_imm(ctx, t2, off);
> emit_insn(ctx, stxb, t1, dst, t2);
> }
> break;
> case BPF_H:
> - move_imm(ctx, t1, imm, is32);
> + move_imm(ctx, t1, imm);
> if (is_signed_imm12(off)) {
> emit_insn(ctx, sth, t1, dst, off);
> } else {
> - move_imm(ctx, t2, off, is32);
> + move_imm(ctx, t2, off);
> emit_insn(ctx, stxh, t1, dst, t2);
> }
> break;
> case BPF_W:
> - move_imm(ctx, t1, imm, is32);
> + move_imm(ctx, t1, imm);
> if (is_signed_imm12(off)) {
> emit_insn(ctx, stw, t1, dst, off);
> } else if (is_signed_imm14(off)) {
> emit_insn(ctx, stptrw, t1, dst, off);
> } else {
> - move_imm(ctx, t2, off, is32);
> + move_imm(ctx, t2, off);
> emit_insn(ctx, stxw, t1, dst, t2);
> }
> break;
> case BPF_DW:
> - move_imm(ctx, t1, imm, is32);
> + move_imm(ctx, t1, imm);
> if (is_signed_imm12(off)) {
> emit_insn(ctx, std, t1, dst, off);
> } else if (is_signed_imm14(off)) {
> emit_insn(ctx, stptrd, t1, dst, off);
> } else {
> - move_imm(ctx, t2, off, is32);
> + move_imm(ctx, t2, off);
> emit_insn(ctx, stxd, t1, dst, t2);
> }
> break;
> @@ -1397,7 +1407,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> if (is_signed_imm12(off)) {
> emit_insn(ctx, stb, src, dst, off);
> } else {
> - move_imm(ctx, t1, off, is32);
> + move_imm(ctx, t1, off);
> emit_insn(ctx, stxb, src, dst, t1);
> }
> break;
> @@ -1405,7 +1415,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> if (is_signed_imm12(off)) {
> emit_insn(ctx, sth, src, dst, off);
> } else {
> - move_imm(ctx, t1, off, is32);
> + move_imm(ctx, t1, off);
> emit_insn(ctx, stxh, src, dst, t1);
> }
> break;
> @@ -1415,7 +1425,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> } else if (is_signed_imm14(off)) {
> emit_insn(ctx, stptrw, src, dst, off);
> } else {
> - move_imm(ctx, t1, off, is32);
> + move_imm(ctx, t1, off);
> emit_insn(ctx, stxw, src, dst, t1);
> }
> break;
> @@ -1425,7 +1435,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> } else if (is_signed_imm14(off)) {
> emit_insn(ctx, stptrd, src, dst, off);
> } else {
> - move_imm(ctx, t1, off, is32);
> + move_imm(ctx, t1, off);
> emit_insn(ctx, stxd, src, dst, t1);
> }
> break;
> @@ -1537,7 +1547,7 @@ static int emit_jump_and_link(struct jit_ctx *ctx, u8 rd, u64 target)
> return -EFAULT;
> }
>
> - move_imm(ctx, LOONGARCH_GPR_T1, target, false);
> + move_imm(ctx, LOONGARCH_GPR_T1, target);
> emit_insn(ctx, jirl, rd, LOONGARCH_GPR_T1, 0);
>
> return 0;
> @@ -1734,7 +1744,7 @@ static int invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_image *im,
> emit_insn(ctx, std, LOONGARCH_GPR_ZERO, LOONGARCH_GPR_FP, -run_ctx_off + cookie_off);
>
> /* arg1: prog */
> - move_imm(ctx, LOONGARCH_GPR_A0, (const s64)p, false);
> + move_imm(ctx, LOONGARCH_GPR_A0, (const s64)p);
> /* arg2: &run_ctx */
> emit_insn(ctx, addid, LOONGARCH_GPR_A1, LOONGARCH_GPR_FP, -run_ctx_off);
> ret = emit_call(ctx, (const u64)bpf_trampoline_enter(p));
> @@ -1755,7 +1765,7 @@ static int invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_image *im,
> /* arg1: &args_off */
> emit_insn(ctx, addid, LOONGARCH_GPR_A0, LOONGARCH_GPR_FP, -args_off);
> if (!p->jited)
> - move_imm(ctx, LOONGARCH_GPR_A1, (const s64)p->insnsi, false);
> + move_imm(ctx, LOONGARCH_GPR_A1, (const s64)p->insnsi);
> ret = emit_call(ctx, (const u64)p->bpf_func);
> if (ret)
> return ret;
> @@ -1772,7 +1782,7 @@ static int invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_image *im,
> }
>
> /* arg1: prog */
> - move_imm(ctx, LOONGARCH_GPR_A0, (const s64)p, false);
> + move_imm(ctx, LOONGARCH_GPR_A0, (const s64)p);
> /* arg2: prog start time */
> move_reg(ctx, LOONGARCH_GPR_A1, LOONGARCH_GPR_S1);
> /* arg3: &run_ctx */
> diff --git a/arch/loongarch/net/bpf_jit.h b/arch/loongarch/net/bpf_jit.h
> index bb58c42c2f2a..57a98f8a055b 100644
> --- a/arch/loongarch/net/bpf_jit.h
> +++ b/arch/loongarch/net/bpf_jit.h
> @@ -137,7 +137,7 @@ static inline void move_addr(struct jit_ctx *ctx, enum loongarch_gpr rd, u64 add
> emit_insn(ctx, lu52id, rd, rd, imm_63_52);
> }
>
> -static inline void move_imm(struct jit_ctx *ctx, enum loongarch_gpr rd, long imm, bool is32)
> +static inline void move_imm(struct jit_ctx *ctx, enum loongarch_gpr rd, long imm)
> {
> long imm_11_0, imm_31_12, imm_51_32, imm_63_52, imm_51_0, imm_51_31;
>
> @@ -150,7 +150,7 @@ static inline void move_imm(struct jit_ctx *ctx, enum loongarch_gpr rd, long imm
> /* addiw rd, $zero, imm_11_0 */
> if (is_signed_imm12(imm)) {
> emit_insn(ctx, addiw, rd, LOONGARCH_GPR_ZERO, imm);
> - goto zext;
> + return;
> }
>
> /* ori rd, $zero, imm_11_0 */
> @@ -195,9 +195,6 @@ static inline void move_imm(struct jit_ctx *ctx, enum loongarch_gpr rd, long imm
> if (!is_signed_imm52(imm))
> emit_insn(ctx, lu52id, rd, rd, imm_63_52);
> }
> -
> -zext:
> - emit_zext_32(ctx, rd, is32);
> }
>
> static inline void move_reg(struct jit_ctx *ctx, enum loongarch_gpr rd,
> --
> 2.42.0
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v3 3/3] LoongArch: BPF: Optimize 32-bit operations using native instructions
2026-10-16 8:10 ` [PATCH v3 3/3] LoongArch: BPF: Optimize 32-bit operations using native instructions Tiezhu Yang
2026-10-16 15:18 ` Alexei Starovoitov
@ 2026-10-27 2:02 ` Hengqi Chen
2026-10-27 4:10 ` Tiezhu Yang
2026-10-28 4:19 ` Huacai Chen
2 siblings, 1 reply; 15+ messages in thread
From: Hengqi Chen @ 2026-10-27 2:02 UTC (permalink / raw)
To: Tiezhu Yang; +Cc: Huacai Chen, loongarch, bpf, linux-kernel
On Fri, Oct 16, 2026 at 4:10 PM Tiezhu Yang <yangtiezhu@loongson.cn> wrote:
>
> Currently, the LoongArch BPF JIT emits numerous redundant zero and sign
> extensions across 32-bit operations, bloating the final JIT image size
> and also leading to runtime overhead, particularly in the division and
> modulo operations. Thus, it is necessary to optimize 32-bit operations
> by emitting LoongArch native 32-bit word instructions.
>
> For general ALU32 operations, utilize native 32-bit word instructions
> for core calculations. Given that the BPF spec requires the upper 32
> bits of the destination register to be zero-extended upon completion,
> use a unified 'bstrpickd' instruction to clear the upper 32 bits at
> the common exit of build_insn(), while immediate loads are optimized
> via (u32) casting to bypass the trailing extension.
>
> According to the LoongArch Reference Manual, native 32-bit division
> and modulo instructions (DIV.W[U] and MOD.W[U]) require the upper
> bits [63:31] of both input operands to be all 0s or all 1s (0x0 or
> 0x1ffffffff), that is to say, they need to be properly sign-extended
> by the software on CPU cores without the DIV32 feature. Otherwise,
> the 32-bit division and modulo instructions could trigger undefined
> hardware behaviors and return unexpected values.
>
> To ensure both performance and hardware compatibility, leverage the
> newly introduced cpu_has_div32 helper:
> (1) on newer CPUs with DIV32, emit native 32-bit div/mod instructions
> directly for optimal performance;
> (2) on older CPUs without DIV32, sign-extend the destination register
> in place and the source register into a temporary register via 'addw'
> before executing the native 32-bit div/mod instructions. Note that the
> JIT compiler must never alter the state of the src register because it
> remains read-only for the duration of the instruction.
>
> For JMP32 conditional branches, the source operands may originate from
> 64-bit contexts where the upper 32 bits contain arbitrary data, so use
> 6 independent case structures to prevent 64-bit comparison inversions.
> Furthermore, use 'bstrpickd' to mask out upper bits for unsigned JMP32,
> and 'addw' with GPR_ZERO to re-align sign bits for signed JMP32.
>
> With this patch, the emit_{zext, sext}_32() and is_signed_bpf_cond()
> helpers are completely wiped out from the header and source files,
> delivering a native execution stream with a reduced JIT image size,
> especially for division and mod operations.
>
> All of the testcases of test_bpf passed on Loongson-3A6000 (with DIV32)
> and Loongson-3A5000 (without DIV32), here are the test results:
>
> $ sudo modprobe test_bpf
> $ dmesg -t | grep Summary
> test_bpf: Summary: 1061 PASSED, 0 FAILED, [1049/1049 JIT'ed]
> test_bpf: test_tail_calls: Summary: 10 PASSED, 0 FAILED, [10/10 JIT'ed]
> test_bpf: test_skb_segment: Summary: 2 PASSED, 0 FAILED
>
> For the BPF selftests test_verifier and test_progs, there are no any
> regressions.
>
> Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
> ---
> arch/loongarch/include/asm/inst.h | 6 +
> arch/loongarch/net/bpf_jit.c | 313 +++++++++++++++++-------------
> arch/loongarch/net/bpf_jit.h | 18 --
> 3 files changed, 183 insertions(+), 154 deletions(-)
>
> diff --git a/arch/loongarch/include/asm/inst.h b/arch/loongarch/include/asm/inst.h
> index 585667e361c2..b55aae325300 100644
> --- a/arch/loongarch/include/asm/inst.h
> +++ b/arch/loongarch/include/asm/inst.h
> @@ -752,11 +752,17 @@ static inline void emit_##NAME(union loongarch_instruction *insn, \
>
> DEF_EMIT_REG3_FORMAT(addw, addw_op)
> DEF_EMIT_REG3_FORMAT(addd, addd_op)
> +DEF_EMIT_REG3_FORMAT(subw, subw_op)
> DEF_EMIT_REG3_FORMAT(subd, subd_op)
> +DEF_EMIT_REG3_FORMAT(mulw, mulw_op)
> DEF_EMIT_REG3_FORMAT(muld, muld_op)
> +DEF_EMIT_REG3_FORMAT(divw, divw_op)
> DEF_EMIT_REG3_FORMAT(divd, divd_op)
> +DEF_EMIT_REG3_FORMAT(modw, modw_op)
> DEF_EMIT_REG3_FORMAT(modd, modd_op)
> +DEF_EMIT_REG3_FORMAT(divwu, divwu_op)
> DEF_EMIT_REG3_FORMAT(divdu, divdu_op)
> +DEF_EMIT_REG3_FORMAT(modwu, modwu_op)
> DEF_EMIT_REG3_FORMAT(moddu, moddu_op)
> DEF_EMIT_REG3_FORMAT(maskeqz, maskeqz_op)
> DEF_EMIT_REG3_FORMAT(masknez, masknez_op)
> diff --git a/arch/loongarch/net/bpf_jit.c b/arch/loongarch/net/bpf_jit.c
> index 77cd96a6f84c..1a7c8effc37b 100644
> --- a/arch/loongarch/net/bpf_jit.c
> +++ b/arch/loongarch/net/bpf_jit.c
> @@ -288,7 +288,7 @@ static int emit_bpf_tail_call(struct jit_ctx *ctx, int insn)
> * if (index >= array->map.max_entries)
> * goto out;
> */
> - emit_zext_32(ctx, a2, true);
> + emit_insn(ctx, bstrpickd, a2, a2, 31, 0);
>
> off = offsetof(struct bpf_array, map.max_entries);
> emit_insn(ctx, ldwu, t1, a1, off);
> @@ -412,7 +412,6 @@ static int emit_atomic_rmw(const struct bpf_insn *insn, struct jit_ctx *ctx)
> return -EINVAL;
> }
> emit_insn(ctx, amadddbb, src, t1, t3);
> - emit_zext_32(ctx, src, true);
> break;
> case BPF_H:
> if (!cpu_has_lam_bh) {
> @@ -420,23 +419,23 @@ static int emit_atomic_rmw(const struct bpf_insn *insn, struct jit_ctx *ctx)
> return -EINVAL;
> }
> emit_insn(ctx, amadddbh, src, t1, t3);
> - emit_zext_32(ctx, src, true);
> break;
> case BPF_W:
> emit_insn(ctx, amadddbw, src, t1, t3);
> - emit_zext_32(ctx, src, true);
> break;
> case BPF_DW:
> emit_insn(ctx, amadddbd, src, t1, t3);
> break;
> }
> + if (!isdw)
> + emit_insn(ctx, bstrpickd, src, src, 31, 0);
> break;
> case BPF_AND | BPF_FETCH:
> if (isdw) {
> emit_insn(ctx, amanddbd, src, t1, t3);
> } else {
> emit_insn(ctx, amanddbw, src, t1, t3);
> - emit_zext_32(ctx, src, true);
> + emit_insn(ctx, bstrpickd, src, src, 31, 0);
> }
> break;
> case BPF_OR | BPF_FETCH:
> @@ -444,7 +443,7 @@ static int emit_atomic_rmw(const struct bpf_insn *insn, struct jit_ctx *ctx)
> emit_insn(ctx, amordbd, src, t1, t3);
> } else {
> emit_insn(ctx, amordbw, src, t1, t3);
> - emit_zext_32(ctx, src, true);
> + emit_insn(ctx, bstrpickd, src, src, 31, 0);
> }
> break;
> case BPF_XOR | BPF_FETCH:
> @@ -452,7 +451,7 @@ static int emit_atomic_rmw(const struct bpf_insn *insn, struct jit_ctx *ctx)
> emit_insn(ctx, amxordbd, src, t1, t3);
> } else {
> emit_insn(ctx, amxordbw, src, t1, t3);
> - emit_zext_32(ctx, src, true);
> + emit_insn(ctx, bstrpickd, src, src, 31, 0);
> }
> break;
> /* src = atomic_xchg(dst + off, src); */
> @@ -464,7 +463,6 @@ static int emit_atomic_rmw(const struct bpf_insn *insn, struct jit_ctx *ctx)
> return -EINVAL;
> }
> emit_insn(ctx, amswapdbb, src, t1, t3);
> - emit_zext_32(ctx, src, true);
> break;
> case BPF_H:
> if (!cpu_has_lam_bh) {
> @@ -472,16 +470,16 @@ static int emit_atomic_rmw(const struct bpf_insn *insn, struct jit_ctx *ctx)
> return -EINVAL;
> }
> emit_insn(ctx, amswapdbh, src, t1, t3);
> - emit_zext_32(ctx, src, true);
> break;
> case BPF_W:
> emit_insn(ctx, amswapdbw, src, t1, t3);
> - emit_zext_32(ctx, src, true);
> break;
> case BPF_DW:
> emit_insn(ctx, amswapdbd, src, t1, t3);
> break;
> }
> + if (!isdw)
> + emit_insn(ctx, bstrpickd, src, src, 31, 0);
> break;
> /* r0 = atomic_cmpxchg(dst + off, r0, src); */
> case BPF_CMPXCHG:
> @@ -494,13 +492,13 @@ static int emit_atomic_rmw(const struct bpf_insn *insn, struct jit_ctx *ctx)
> emit_insn(ctx, beq, t3, LOONGARCH_GPR_ZERO, -4);
> } else {
> emit_insn(ctx, llw, r0, t1, 0);
> - emit_zext_32(ctx, t2, true);
> - emit_zext_32(ctx, r0, true);
> + emit_insn(ctx, bstrpickd, t2, t2, 31, 0);
> + emit_insn(ctx, bstrpickd, r0, r0, 31, 0);
> emit_insn(ctx, bne, t2, r0, 4);
> move_reg(ctx, t3, src);
> emit_insn(ctx, scw, t3, t1, 0);
> emit_insn(ctx, beq, t3, LOONGARCH_GPR_ZERO, -6);
> - emit_zext_32(ctx, r0, true);
> + emit_insn(ctx, bstrpickd, r0, r0, 31, 0);
> }
> emit_insn(ctx, dbar, DBAR_LLSC_MB);
> break;
> @@ -605,12 +603,6 @@ static int emit_atomic_ld_st(const struct bpf_insn *insn, struct jit_ctx *ctx)
> return 0;
> }
>
> -static bool is_signed_bpf_cond(u8 cond)
> -{
> - return cond == BPF_JSGT || cond == BPF_JSLT ||
> - cond == BPF_JSGE || cond == BPF_JSLE;
> -}
> -
> #define BPF_FIXUP_REG_MASK GENMASK(31, 27)
> #define BPF_FIXUP_OFFSET_MASK GENMASK(26, 0)
> #define REG_DONT_CLEAR_MARKER 0
> @@ -707,7 +699,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> u8 dst = regmap[insn->dst_reg];
> const s16 off = insn->off;
> const s32 imm = insn->imm;
> - const bool is32 = BPF_CLASS(insn->code) == BPF_ALU || BPF_CLASS(insn->code) == BPF_JMP32;
> + const bool is32 = BPF_CLASS(insn->code) == BPF_ALU && BPF_OP(insn->code) != BPF_END;
>
> switch (code) {
> /* dst = src */
> @@ -715,7 +707,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> case BPF_ALU64 | BPF_MOV | BPF_X:
> if (insn_is_cast_user(insn)) {
> move_reg(ctx, t1, src);
> - emit_zext_32(ctx, t1, true);
> + emit_insn(ctx, bstrpickd, t1, t1, 31, 0);
> move_imm(ctx, dst, (ctx->user_vm_start >> 32) << 32);
> emit_insn(ctx, beq, t1, LOONGARCH_GPR_ZERO, 2);
> emit_insn(ctx, or, t1, dst, t1);
> @@ -734,15 +726,12 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> switch (off) {
> case 0:
> move_reg(ctx, dst, src);
> - emit_zext_32(ctx, dst, is32);
> break;
> case 8:
> emit_insn(ctx, extwb, dst, src);
> - emit_zext_32(ctx, dst, is32);
> break;
> case 16:
> emit_insn(ctx, extwh, dst, src);
> - emit_zext_32(ctx, dst, is32);
> break;
> case 32:
> emit_insn(ctx, addw, dst, src, LOONGARCH_GPR_ZERO);
> @@ -752,151 +741,188 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
>
> /* dst = imm */
> case BPF_ALU | BPF_MOV | BPF_K:
> + move_imm(ctx, dst, (u32)imm);
> + return 0;
> case BPF_ALU64 | BPF_MOV | BPF_K:
> move_imm(ctx, dst, imm);
> - emit_zext_32(ctx, dst, is32);
> break;
>
> /* dst = dst + src */
> case BPF_ALU | BPF_ADD | BPF_X:
> + emit_insn(ctx, addw, dst, dst, src);
> + break;
> case BPF_ALU64 | BPF_ADD | BPF_X:
> emit_insn(ctx, addd, dst, dst, src);
> - emit_zext_32(ctx, dst, is32);
> break;
>
> /* dst = dst + imm */
> case BPF_ALU | BPF_ADD | BPF_K:
> + if (is_signed_imm12(imm)) {
> + emit_insn(ctx, addiw, dst, dst, imm);
> + } else {
> + move_imm(ctx, t1, (u32)imm);
> + emit_insn(ctx, addw, dst, dst, t1);
> + }
> + break;
> case BPF_ALU64 | BPF_ADD | BPF_K:
> if (is_signed_imm12(imm)) {
> emit_insn(ctx, addid, dst, dst, imm);
> } else {
> move_imm(ctx, t1, imm);
> - emit_zext_32(ctx, t1, is32);
> emit_insn(ctx, addd, dst, dst, t1);
> }
> - emit_zext_32(ctx, dst, is32);
> break;
>
> /* dst = dst - src */
> case BPF_ALU | BPF_SUB | BPF_X:
> + emit_insn(ctx, subw, dst, dst, src);
> + break;
> case BPF_ALU64 | BPF_SUB | BPF_X:
> emit_insn(ctx, subd, dst, dst, src);
> - emit_zext_32(ctx, dst, is32);
> break;
>
> /* dst = dst - imm */
> case BPF_ALU | BPF_SUB | BPF_K:
> + if (is_signed_imm12(-imm)) {
> + emit_insn(ctx, addiw, dst, dst, -imm);
> + } else {
> + move_imm(ctx, t1, (u32)imm);
> + emit_insn(ctx, subw, dst, dst, t1);
> + }
> + break;
> case BPF_ALU64 | BPF_SUB | BPF_K:
> if (is_signed_imm12(-imm)) {
> emit_insn(ctx, addid, dst, dst, -imm);
> } else {
> move_imm(ctx, t1, imm);
> - emit_zext_32(ctx, t1, is32);
> emit_insn(ctx, subd, dst, dst, t1);
> }
> - emit_zext_32(ctx, dst, is32);
> break;
>
> /* dst = dst * src */
> case BPF_ALU | BPF_MUL | BPF_X:
> + emit_insn(ctx, mulw, dst, dst, src);
> + break;
> case BPF_ALU64 | BPF_MUL | BPF_X:
> emit_insn(ctx, muld, dst, dst, src);
> - emit_zext_32(ctx, dst, is32);
> break;
>
> /* dst = dst * imm */
> case BPF_ALU | BPF_MUL | BPF_K:
> + move_imm(ctx, t1, (u32)imm);
> + emit_insn(ctx, mulw, dst, dst, t1);
> + break;
> case BPF_ALU64 | BPF_MUL | BPF_K:
> move_imm(ctx, t1, imm);
> - emit_zext_32(ctx, t1, is32);
> emit_insn(ctx, muld, dst, dst, t1);
> - emit_zext_32(ctx, dst, is32);
> break;
>
> /* dst = dst / src */
> case BPF_ALU | BPF_DIV | BPF_X:
> - case BPF_ALU64 | BPF_DIV | BPF_X:
> - if (!off) {
> - emit_zext_32(ctx, dst, is32);
> - move_reg(ctx, t1, src);
> - emit_zext_32(ctx, t1, is32);
> - emit_insn(ctx, divdu, dst, dst, t1);
> - emit_zext_32(ctx, dst, is32);
> - } else {
> - emit_sext_32(ctx, dst, is32);
> - move_reg(ctx, t1, src);
> - emit_sext_32(ctx, t1, is32);
> - emit_insn(ctx, divd, dst, dst, t1);
> - emit_zext_32(ctx, dst, is32);
> + if (cpu_has_div32) {
> + if (!off)
> + emit_insn(ctx, divwu, dst, dst, src);
> + else
> + emit_insn(ctx, divw, dst, dst, src);
For ALU32 SDIV, off != 0, it seems like there is a overflow in S32_MIN / -1.
verifier_sdiv.c has related test cases but annotate with
__arch_x86_64, so it is not run on LoongArch.
> + break;
> }
> + /*
> + * Legacy fallback for !div32: src must be extended into a
> + * temporary register to keep the original src register intact.
> + */
> + emit_insn(ctx, addw, dst, dst, LOONGARCH_GPR_ZERO);
> + emit_insn(ctx, addw, t1, src, LOONGARCH_GPR_ZERO);
> + if (!off)
> + emit_insn(ctx, divwu, dst, dst, t1);
> + else
> + emit_insn(ctx, divw, dst, dst, t1);
> + break;
> + case BPF_ALU64 | BPF_DIV | BPF_X:
> + if (!off)
> + emit_insn(ctx, divdu, dst, dst, src);
> + else
> + emit_insn(ctx, divd, dst, dst, src);
> break;
>
> /* dst = dst / imm */
> case BPF_ALU | BPF_DIV | BPF_K:
> + move_imm(ctx, t1, (u32)imm);
> + if (!cpu_has_div32) {
> + emit_insn(ctx, addw, dst, dst, LOONGARCH_GPR_ZERO);
> + emit_insn(ctx, addw, t1, t1, LOONGARCH_GPR_ZERO);
> + }
> + if (!off)
> + emit_insn(ctx, divwu, dst, dst, t1);
> + else
> + emit_insn(ctx, divw, dst, dst, t1);
> + break;
> case BPF_ALU64 | BPF_DIV | BPF_K:
> - if (!off) {
> - move_imm(ctx, t1, imm);
> - emit_zext_32(ctx, t1, is32);
> - emit_zext_32(ctx, dst, is32);
> + move_imm(ctx, t1, imm);
> + if (!off)
> emit_insn(ctx, divdu, dst, dst, t1);
> - emit_zext_32(ctx, dst, is32);
> - } else {
> - move_imm(ctx, t1, imm);
> - emit_sext_32(ctx, t1, is32);
> - emit_sext_32(ctx, dst, is32);
> + else
> emit_insn(ctx, divd, dst, dst, t1);
> - emit_zext_32(ctx, dst, is32);
> - }
> break;
>
> /* dst = dst % src */
> case BPF_ALU | BPF_MOD | BPF_X:
> - case BPF_ALU64 | BPF_MOD | BPF_X:
> - if (!off) {
> - emit_zext_32(ctx, dst, is32);
> - move_reg(ctx, t1, src);
> - emit_zext_32(ctx, t1, is32);
> - emit_insn(ctx, moddu, dst, dst, t1);
> - emit_zext_32(ctx, dst, is32);
> - } else {
> - emit_sext_32(ctx, dst, is32);
> - move_reg(ctx, t1, src);
> - emit_sext_32(ctx, t1, is32);
> - emit_insn(ctx, modd, dst, dst, t1);
> - emit_zext_32(ctx, dst, is32);
> + if (cpu_has_div32) {
> + if (!off)
> + emit_insn(ctx, modwu, dst, dst, src);
> + else
> + emit_insn(ctx, modw, dst, dst, src);
> + break;
> }
> + /*
> + * Legacy fallback for !div32: src must be extended into a
> + * temporary register to keep the original src register intact.
> + */
> + emit_insn(ctx, addw, dst, dst, LOONGARCH_GPR_ZERO);
> + emit_insn(ctx, addw, t1, src, LOONGARCH_GPR_ZERO);
> + if (!off)
> + emit_insn(ctx, modwu, dst, dst, t1);
> + else
> + emit_insn(ctx, modw, dst, dst, t1);
> + break;
> + case BPF_ALU64 | BPF_MOD | BPF_X:
> + if (!off)
> + emit_insn(ctx, moddu, dst, dst, src);
> + else
> + emit_insn(ctx, modd, dst, dst, src);
> break;
>
> /* dst = dst % imm */
> case BPF_ALU | BPF_MOD | BPF_K:
> + move_imm(ctx, t1, (u32)imm);
> + if (!cpu_has_div32) {
> + emit_insn(ctx, addw, dst, dst, LOONGARCH_GPR_ZERO);
> + emit_insn(ctx, addw, t1, t1, LOONGARCH_GPR_ZERO);
> + }
> + if (!off)
> + emit_insn(ctx, modwu, dst, dst, t1);
> + else
> + emit_insn(ctx, modw, dst, dst, t1);
> + break;
> case BPF_ALU64 | BPF_MOD | BPF_K:
> - if (!off) {
> - move_imm(ctx, t1, imm);
> - emit_zext_32(ctx, t1, is32);
> - emit_zext_32(ctx, dst, is32);
> + move_imm(ctx, t1, imm);
> + if (!off)
> emit_insn(ctx, moddu, dst, dst, t1);
> - emit_zext_32(ctx, dst, is32);
> - } else {
> - move_imm(ctx, t1, imm);
> - emit_sext_32(ctx, t1, is32);
> - emit_sext_32(ctx, dst, is32);
> + else
> emit_insn(ctx, modd, dst, dst, t1);
> - emit_zext_32(ctx, dst, is32);
> - }
> break;
>
> /* dst = -dst */
> case BPF_ALU | BPF_NEG:
> + emit_insn(ctx, subw, dst, LOONGARCH_GPR_ZERO, dst);
> + break;
> case BPF_ALU64 | BPF_NEG:
> emit_insn(ctx, subd, dst, LOONGARCH_GPR_ZERO, dst);
> - emit_zext_32(ctx, dst, is32);
> break;
>
> /* dst = dst & src */
> case BPF_ALU | BPF_AND | BPF_X:
> case BPF_ALU64 | BPF_AND | BPF_X:
> emit_insn(ctx, and, dst, dst, src);
> - emit_zext_32(ctx, dst, is32);
> break;
>
> /* dst = dst & imm */
> @@ -905,18 +931,18 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> if (is_unsigned_imm12(imm)) {
> emit_insn(ctx, andi, dst, dst, imm);
> } else {
> - move_imm(ctx, t1, imm);
> - emit_zext_32(ctx, t1, is32);
> + if (is32)
> + move_imm(ctx, t1, (u32)imm);
> + else
> + move_imm(ctx, t1, imm);
> emit_insn(ctx, and, dst, dst, t1);
> }
> - emit_zext_32(ctx, dst, is32);
> break;
>
> /* dst = dst | src */
> case BPF_ALU | BPF_OR | BPF_X:
> case BPF_ALU64 | BPF_OR | BPF_X:
> emit_insn(ctx, or, dst, dst, src);
> - emit_zext_32(ctx, dst, is32);
> break;
>
> /* dst = dst | imm */
> @@ -925,18 +951,18 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> if (is_unsigned_imm12(imm)) {
> emit_insn(ctx, ori, dst, dst, imm);
> } else {
> - move_imm(ctx, t1, imm);
> - emit_zext_32(ctx, t1, is32);
> + if (is32)
> + move_imm(ctx, t1, (u32)imm);
> + else
> + move_imm(ctx, t1, imm);
> emit_insn(ctx, or, dst, dst, t1);
> }
> - emit_zext_32(ctx, dst, is32);
> break;
>
> /* dst = dst ^ src */
> case BPF_ALU | BPF_XOR | BPF_X:
> case BPF_ALU64 | BPF_XOR | BPF_X:
> emit_insn(ctx, xor, dst, dst, src);
> - emit_zext_32(ctx, dst, is32);
> break;
>
> /* dst = dst ^ imm */
> @@ -945,19 +971,18 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> if (is_unsigned_imm12(imm)) {
> emit_insn(ctx, xori, dst, dst, imm);
> } else {
> - move_imm(ctx, t1, imm);
> - emit_zext_32(ctx, t1, is32);
> + if (is32)
> + move_imm(ctx, t1, (u32)imm);
> + else
> + move_imm(ctx, t1, imm);
> emit_insn(ctx, xor, dst, dst, t1);
> }
> - emit_zext_32(ctx, dst, is32);
> break;
>
> /* dst = dst << src (logical) */
> case BPF_ALU | BPF_LSH | BPF_X:
> emit_insn(ctx, sllw, dst, dst, src);
> - emit_zext_32(ctx, dst, is32);
> break;
> -
> case BPF_ALU64 | BPF_LSH | BPF_X:
> emit_insn(ctx, slld, dst, dst, src);
> break;
> @@ -965,9 +990,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> /* dst = dst << imm (logical) */
> case BPF_ALU | BPF_LSH | BPF_K:
> emit_insn(ctx, slliw, dst, dst, imm);
> - emit_zext_32(ctx, dst, is32);
> break;
> -
> case BPF_ALU64 | BPF_LSH | BPF_K:
> emit_insn(ctx, sllid, dst, dst, imm);
> break;
> @@ -975,9 +998,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> /* dst = dst >> src (logical) */
> case BPF_ALU | BPF_RSH | BPF_X:
> emit_insn(ctx, srlw, dst, dst, src);
> - emit_zext_32(ctx, dst, is32);
> break;
> -
> case BPF_ALU64 | BPF_RSH | BPF_X:
> emit_insn(ctx, srld, dst, dst, src);
> break;
> @@ -985,9 +1006,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> /* dst = dst >> imm (logical) */
> case BPF_ALU | BPF_RSH | BPF_K:
> emit_insn(ctx, srliw, dst, dst, imm);
> - emit_zext_32(ctx, dst, is32);
> break;
> -
> case BPF_ALU64 | BPF_RSH | BPF_K:
> emit_insn(ctx, srlid, dst, dst, imm);
> break;
> @@ -995,9 +1014,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> /* dst = dst >> src (arithmetic) */
> case BPF_ALU | BPF_ARSH | BPF_X:
> emit_insn(ctx, sraw, dst, dst, src);
> - emit_zext_32(ctx, dst, is32);
> break;
> -
> case BPF_ALU64 | BPF_ARSH | BPF_X:
> emit_insn(ctx, srad, dst, dst, src);
> break;
> @@ -1005,9 +1022,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> /* dst = dst >> imm (arithmetic) */
> case BPF_ALU | BPF_ARSH | BPF_K:
> emit_insn(ctx, sraiw, dst, dst, imm);
> - emit_zext_32(ctx, dst, is32);
> break;
> -
> case BPF_ALU64 | BPF_ARSH | BPF_K:
> emit_insn(ctx, sraid, dst, dst, imm);
> break;
> @@ -1016,12 +1031,10 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> case BPF_ALU | BPF_END | BPF_FROM_LE:
> switch (imm) {
> case 16:
> - /* zero-extend 16 bits into 64 bits */
> emit_insn(ctx, bstrpickd, dst, dst, 15, 0);
> break;
> case 32:
> - /* zero-extend 32 bits into 64 bits */
> - emit_zext_32(ctx, dst, is32);
> + emit_insn(ctx, bstrpickd, dst, dst, 31, 0);
> break;
> case 64:
> /* do nothing */
> @@ -1034,13 +1047,11 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> switch (imm) {
> case 16:
> emit_insn(ctx, revb2h, dst, dst);
> - /* zero-extend 16 bits into 64 bits */
> emit_insn(ctx, bstrpickd, dst, dst, 15, 0);
> break;
> case 32:
> emit_insn(ctx, revb2w, dst, dst);
> - /* clear the upper 32 bits */
> - emit_zext_32(ctx, dst, true);
> + emit_insn(ctx, bstrpickd, dst, dst, 31, 0);
> break;
> case 64:
> emit_insn(ctx, revbd, dst, dst);
> @@ -1048,7 +1059,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> }
> break;
>
> - /* PC += off if dst cond src */
> + /* PC += off if dst cond src (64-bit) */
> case BPF_JMP | BPF_JEQ | BPF_X:
> case BPF_JMP | BPF_JNE | BPF_X:
> case BPF_JMP | BPF_JGT | BPF_X:
> @@ -1059,31 +1070,38 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> case BPF_JMP | BPF_JSGE | BPF_X:
> case BPF_JMP | BPF_JSLT | BPF_X:
> case BPF_JMP | BPF_JSLE | BPF_X:
> + jmp_offset = bpf2la_offset(i, off, ctx);
> + if (emit_cond_jmp(ctx, cond, dst, src, jmp_offset) < 0)
> + goto toofar;
> + break;
> +
> + /* PC += off if dst cond src (32-bit unsigned) */
> case BPF_JMP32 | BPF_JEQ | BPF_X:
> case BPF_JMP32 | BPF_JNE | BPF_X:
> case BPF_JMP32 | BPF_JGT | BPF_X:
> case BPF_JMP32 | BPF_JGE | BPF_X:
> case BPF_JMP32 | BPF_JLT | BPF_X:
> case BPF_JMP32 | BPF_JLE | BPF_X:
> + jmp_offset = bpf2la_offset(i, off, ctx);
> + emit_insn(ctx, bstrpickd, t1, dst, 31, 0);
> + emit_insn(ctx, bstrpickd, t2, src, 31, 0);
> + if (emit_cond_jmp(ctx, cond, t1, t2, jmp_offset) < 0)
> + goto toofar;
> + break;
> +
> + /* PC += off if dst cond src (32-bit signed) */
> case BPF_JMP32 | BPF_JSGT | BPF_X:
> case BPF_JMP32 | BPF_JSGE | BPF_X:
> case BPF_JMP32 | BPF_JSLT | BPF_X:
> case BPF_JMP32 | BPF_JSLE | BPF_X:
> jmp_offset = bpf2la_offset(i, off, ctx);
> - move_reg(ctx, t1, dst);
> - move_reg(ctx, t2, src);
> - if (is_signed_bpf_cond(BPF_OP(code))) {
> - emit_sext_32(ctx, t1, is32);
> - emit_sext_32(ctx, t2, is32);
> - } else {
> - emit_zext_32(ctx, t1, is32);
> - emit_zext_32(ctx, t2, is32);
> - }
> + emit_insn(ctx, addw, t1, dst, LOONGARCH_GPR_ZERO);
> + emit_insn(ctx, addw, t2, src, LOONGARCH_GPR_ZERO);
> if (emit_cond_jmp(ctx, cond, t1, t2, jmp_offset) < 0)
> goto toofar;
> break;
>
> - /* PC += off if dst cond imm */
> + /* PC += off if dst cond imm (64-bit) */
> case BPF_JMP | BPF_JEQ | BPF_K:
> case BPF_JMP | BPF_JNE | BPF_K:
> case BPF_JMP | BPF_JGT | BPF_K:
> @@ -1094,12 +1112,38 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> case BPF_JMP | BPF_JSGE | BPF_K:
> case BPF_JMP | BPF_JSLT | BPF_K:
> case BPF_JMP | BPF_JSLE | BPF_K:
> + jmp_offset = bpf2la_offset(i, off, ctx);
> + if (imm) {
> + move_imm(ctx, t1, imm);
> + tm = t1;
> + } else {
> + tm = LOONGARCH_GPR_ZERO;
> + }
> + if (emit_cond_jmp(ctx, cond, dst, tm, jmp_offset) < 0)
> + goto toofar;
> + break;
> +
> + /* PC += off if dst cond imm (32-bit unsigned) */
> case BPF_JMP32 | BPF_JEQ | BPF_K:
> case BPF_JMP32 | BPF_JNE | BPF_K:
> case BPF_JMP32 | BPF_JGT | BPF_K:
> case BPF_JMP32 | BPF_JGE | BPF_K:
> case BPF_JMP32 | BPF_JLT | BPF_K:
> case BPF_JMP32 | BPF_JLE | BPF_K:
> + jmp_offset = bpf2la_offset(i, off, ctx);
> + if (imm) {
> + move_imm(ctx, t1, imm);
> + emit_insn(ctx, bstrpickd, t1, t1, 31, 0);
> + tm = t1;
> + } else {
> + tm = LOONGARCH_GPR_ZERO;
> + }
> + emit_insn(ctx, bstrpickd, t2, dst, 31, 0);
> + if (emit_cond_jmp(ctx, cond, t2, tm, jmp_offset) < 0)
> + goto toofar;
> + break;
> +
> + /* PC += off if dst cond imm (32-bit signed) */
> case BPF_JMP32 | BPF_JSGT | BPF_K:
> case BPF_JMP32 | BPF_JSGE | BPF_K:
> case BPF_JMP32 | BPF_JSLT | BPF_K:
> @@ -1107,19 +1151,12 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> jmp_offset = bpf2la_offset(i, off, ctx);
> if (imm) {
> move_imm(ctx, t1, imm);
> + emit_insn(ctx, addw, t1, t1, LOONGARCH_GPR_ZERO);
> tm = t1;
> } else {
> - /* If imm is 0, simply use zero register. */
> tm = LOONGARCH_GPR_ZERO;
> }
> - move_reg(ctx, t2, dst);
> - if (is_signed_bpf_cond(BPF_OP(code))) {
> - emit_sext_32(ctx, tm, is32);
> - emit_sext_32(ctx, t2, is32);
> - } else {
> - emit_zext_32(ctx, tm, is32);
> - emit_zext_32(ctx, t2, is32);
> - }
> + emit_insn(ctx, addw, t2, dst, LOONGARCH_GPR_ZERO);
> if (emit_cond_jmp(ctx, cond, t2, tm, jmp_offset) < 0)
> goto toofar;
> break;
> @@ -1129,7 +1166,8 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> case BPF_JMP32 | BPF_JSET | BPF_X:
> jmp_offset = bpf2la_offset(i, off, ctx);
> emit_insn(ctx, and, t1, dst, src);
> - emit_zext_32(ctx, t1, is32);
> + if (BPF_CLASS(code) == BPF_JMP32)
> + emit_insn(ctx, bstrpickd, t1, t1, 31, 0);
> if (emit_cond_jmp(ctx, cond, t1, LOONGARCH_GPR_ZERO, jmp_offset) < 0)
> goto toofar;
> break;
> @@ -1139,9 +1177,9 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> case BPF_JMP32 | BPF_JSET | BPF_K:
> jmp_offset = bpf2la_offset(i, off, ctx);
> move_imm(ctx, t1, imm);
> - emit_zext_32(ctx, t1, is32);
> emit_insn(ctx, and, t1, dst, t1);
> - emit_zext_32(ctx, t1, is32);
> + if (BPF_CLASS(code) == BPF_JMP32)
> + emit_insn(ctx, bstrpickd, t1, t1, 31, 0);
> if (emit_cond_jmp(ctx, cond, t1, LOONGARCH_GPR_ZERO, jmp_offset) < 0)
> goto toofar;
> break;
> @@ -1468,6 +1506,9 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> return -EINVAL;
> }
>
> + if (is32)
> + emit_insn(ctx, bstrpickd, dst, dst, 31, 0);
> +
> return 0;
>
> toofar:
> diff --git a/arch/loongarch/net/bpf_jit.h b/arch/loongarch/net/bpf_jit.h
> index 57a98f8a055b..2bd0dc5b911c 100644
> --- a/arch/loongarch/net/bpf_jit.h
> +++ b/arch/loongarch/net/bpf_jit.h
> @@ -72,24 +72,6 @@ static inline int epilogue_offset(const struct jit_ctx *ctx)
> return (to - from);
> }
>
> -/* Zero-extend 32 bits into 64 bits */
> -static inline void emit_zext_32(struct jit_ctx *ctx, enum loongarch_gpr reg, bool is32)
> -{
> - if (!is32)
> - return;
> -
> - emit_insn(ctx, lu32id, reg, 0);
> -}
> -
> -/* Signed-extend 32 bits into 64 bits */
> -static inline void emit_sext_32(struct jit_ctx *ctx, enum loongarch_gpr reg, bool is32)
> -{
> - if (!is32)
> - return;
> -
> - emit_insn(ctx, addiw, reg, reg, 0);
> -}
> -
> /* Emit proper extension according to ABI requirements.
> * Note that it requires a value of size `size` already resides in register `reg`.
> */
> --
> 2.42.0
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v3 3/3] LoongArch: BPF: Optimize 32-bit operations using native instructions
2026-10-27 2:02 ` Hengqi Chen
@ 2026-10-27 4:10 ` Tiezhu Yang
2026-10-27 8:13 ` Hengqi Chen
0 siblings, 1 reply; 15+ messages in thread
From: Tiezhu Yang @ 2026-10-27 4:10 UTC (permalink / raw)
To: Hengqi Chen; +Cc: Huacai Chen, loongarch, bpf, linux-kernel
On 2026/10/27 上午10:02, Hengqi Chen wrote:
> On Fri, Oct 16, 2026 at 4:10 PM Tiezhu Yang <yangtiezhu@loongson.cn> wrote:
>>
>> Currently, the LoongArch BPF JIT emits numerous redundant zero and sign
>> extensions across 32-bit operations, bloating the final JIT image size
>> and also leading to runtime overhead, particularly in the division and
>> modulo operations. Thus, it is necessary to optimize 32-bit operations
>> by emitting LoongArch native 32-bit word instructions.
...
>> + if (cpu_has_div32) {
>> + if (!off)
>> + emit_insn(ctx, divwu, dst, dst, src);
>> + else
>> + emit_insn(ctx, divw, dst, dst, src);
>
> For ALU32 SDIV, off != 0, it seems like there is a overflow in S32_MIN / -1.
> verifier_sdiv.c has related test cases but annotate with
> __arch_x86_64, so it is not run on LoongArch.
After enabling the signed division and modulo overflow tests for
LoongArch, here are the test results:
$ sudo ./test_progs -t verifier_sdiv | grep "MIN/-1"
#731/121 verifier_sdiv/SDIV64, overflow r/r, LLONG_MIN/-1:OK
#731/123 verifier_sdiv/SDIV64, overflow r/i, LLONG_MIN/-1:OK
#731/125 verifier_sdiv/SDIV32, overflow r/r, INT_MIN/-1:OK
#731/127 verifier_sdiv/SDIV32, overflow r/i, INT_MIN/-1:OK
#731/129 verifier_sdiv/SMOD64, overflow r/r, LLONG_MIN/-1:OK
#731/131 verifier_sdiv/SMOD64, overflow r/i, LLONG_MIN/-1:OK
#731/133 verifier_sdiv/SMOD32, overflow r/r, INT_MIN/-1:OK
#731/135 verifier_sdiv/SMOD32, overflow r/i, INT_MIN/-1:OK
That is to say, the current jit code is no problem.
Thanks,
Tiezhu
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v3 3/3] LoongArch: BPF: Optimize 32-bit operations using native instructions
2026-10-27 4:10 ` Tiezhu Yang
@ 2026-10-27 8:13 ` Hengqi Chen
0 siblings, 0 replies; 15+ messages in thread
From: Hengqi Chen @ 2026-10-27 8:13 UTC (permalink / raw)
To: Tiezhu Yang; +Cc: Huacai Chen, loongarch, bpf, linux-kernel
On Tue, Oct 27, 2026 at 12:10 PM Tiezhu Yang <yangtiezhu@loongson.cn> wrote:
>
> On 2026/10/27 上午10:02, Hengqi Chen wrote:
> > On Fri, Oct 16, 2026 at 4:10 PM Tiezhu Yang <yangtiezhu@loongson.cn> wrote:
> >>
> >> Currently, the LoongArch BPF JIT emits numerous redundant zero and sign
> >> extensions across 32-bit operations, bloating the final JIT image size
> >> and also leading to runtime overhead, particularly in the division and
> >> modulo operations. Thus, it is necessary to optimize 32-bit operations
> >> by emitting LoongArch native 32-bit word instructions.
>
> ...
>
> >> + if (cpu_has_div32) {
> >> + if (!off)
> >> + emit_insn(ctx, divwu, dst, dst, src);
> >> + else
> >> + emit_insn(ctx, divw, dst, dst, src);
> >
> > For ALU32 SDIV, off != 0, it seems like there is a overflow in S32_MIN / -1.
> > verifier_sdiv.c has related test cases but annotate with
> > __arch_x86_64, so it is not run on LoongArch.
>
> After enabling the signed division and modulo overflow tests for
> LoongArch, here are the test results:
>
> $ sudo ./test_progs -t verifier_sdiv | grep "MIN/-1"
> #731/121 verifier_sdiv/SDIV64, overflow r/r, LLONG_MIN/-1:OK
> #731/123 verifier_sdiv/SDIV64, overflow r/i, LLONG_MIN/-1:OK
> #731/125 verifier_sdiv/SDIV32, overflow r/r, INT_MIN/-1:OK
> #731/127 verifier_sdiv/SDIV32, overflow r/i, INT_MIN/-1:OK
> #731/129 verifier_sdiv/SMOD64, overflow r/r, LLONG_MIN/-1:OK
> #731/131 verifier_sdiv/SMOD64, overflow r/i, LLONG_MIN/-1:OK
> #731/133 verifier_sdiv/SMOD32, overflow r/r, INT_MIN/-1:OK
> #731/135 verifier_sdiv/SMOD32, overflow r/i, INT_MIN/-1:OK
>
> That is to say, the current jit code is no problem.
>
Great.
Acked-by: Hengqi Chen <hengqi.chen@gmail.com>
> Thanks,
> Tiezhu
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v3 3/3] LoongArch: BPF: Optimize 32-bit operations using native instructions
2026-10-16 8:10 ` [PATCH v3 3/3] LoongArch: BPF: Optimize 32-bit operations using native instructions Tiezhu Yang
2026-10-16 15:18 ` Alexei Starovoitov
2026-10-27 2:02 ` Hengqi Chen
@ 2026-10-28 4:19 ` Huacai Chen
2026-10-28 6:55 ` Hengqi Chen
2 siblings, 1 reply; 15+ messages in thread
From: Huacai Chen @ 2026-10-28 4:19 UTC (permalink / raw)
To: Tiezhu Yang; +Cc: Hengqi Chen, loongarch, bpf, linux-kernel
Hi, Hengqi,
On Fri, Oct 16, 2026 at 4:10 PM Tiezhu Yang <yangtiezhu@loongson.cn> wrote:
>
> Currently, the LoongArch BPF JIT emits numerous redundant zero and sign
> extensions across 32-bit operations, bloating the final JIT image size
> and also leading to runtime overhead, particularly in the division and
> modulo operations. Thus, it is necessary to optimize 32-bit operations
> by emitting LoongArch native 32-bit word instructions.
>
> For general ALU32 operations, utilize native 32-bit word instructions
> for core calculations. Given that the BPF spec requires the upper 32
> bits of the destination register to be zero-extended upon completion,
> use a unified 'bstrpickd' instruction to clear the upper 32 bits at
> the common exit of build_insn(), while immediate loads are optimized
> via (u32) casting to bypass the trailing extension.
>
> According to the LoongArch Reference Manual, native 32-bit division
> and modulo instructions (DIV.W[U] and MOD.W[U]) require the upper
> bits [63:31] of both input operands to be all 0s or all 1s (0x0 or
> 0x1ffffffff), that is to say, they need to be properly sign-extended
> by the software on CPU cores without the DIV32 feature. Otherwise,
> the 32-bit division and modulo instructions could trigger undefined
> hardware behaviors and return unexpected values.
>
> To ensure both performance and hardware compatibility, leverage the
> newly introduced cpu_has_div32 helper:
> (1) on newer CPUs with DIV32, emit native 32-bit div/mod instructions
> directly for optimal performance;
> (2) on older CPUs without DIV32, sign-extend the destination register
> in place and the source register into a temporary register via 'addw'
> before executing the native 32-bit div/mod instructions. Note that the
> JIT compiler must never alter the state of the src register because it
> remains read-only for the duration of the instruction.
>
> For JMP32 conditional branches, the source operands may originate from
> 64-bit contexts where the upper 32 bits contain arbitrary data, so use
> 6 independent case structures to prevent 64-bit comparison inversions.
> Furthermore, use 'bstrpickd' to mask out upper bits for unsigned JMP32,
> and 'addw' with GPR_ZERO to re-align sign bits for signed JMP32.
>
> With this patch, the emit_{zext, sext}_32() and is_signed_bpf_cond()
> helpers are completely wiped out from the header and source files,
> delivering a native execution stream with a reduced JIT image size,
> especially for division and mod operations.
>
> All of the testcases of test_bpf passed on Loongson-3A6000 (with DIV32)
> and Loongson-3A5000 (without DIV32), here are the test results:
>
> $ sudo modprobe test_bpf
> $ dmesg -t | grep Summary
> test_bpf: Summary: 1061 PASSED, 0 FAILED, [1049/1049 JIT'ed]
> test_bpf: test_tail_calls: Summary: 10 PASSED, 0 FAILED, [10/10 JIT'ed]
> test_bpf: test_skb_segment: Summary: 2 PASSED, 0 FAILED
>
> For the BPF selftests test_verifier and test_progs, there are no any
> regressions.
>
> Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
> ---
> arch/loongarch/include/asm/inst.h | 6 +
> arch/loongarch/net/bpf_jit.c | 313 +++++++++++++++++-------------
> arch/loongarch/net/bpf_jit.h | 18 --
> 3 files changed, 183 insertions(+), 154 deletions(-)
>
> diff --git a/arch/loongarch/include/asm/inst.h b/arch/loongarch/include/asm/inst.h
> index 585667e361c2..b55aae325300 100644
> --- a/arch/loongarch/include/asm/inst.h
> +++ b/arch/loongarch/include/asm/inst.h
> @@ -752,11 +752,17 @@ static inline void emit_##NAME(union loongarch_instruction *insn, \
>
> DEF_EMIT_REG3_FORMAT(addw, addw_op)
> DEF_EMIT_REG3_FORMAT(addd, addd_op)
> +DEF_EMIT_REG3_FORMAT(subw, subw_op)
> DEF_EMIT_REG3_FORMAT(subd, subd_op)
> +DEF_EMIT_REG3_FORMAT(mulw, mulw_op)
> DEF_EMIT_REG3_FORMAT(muld, muld_op)
> +DEF_EMIT_REG3_FORMAT(divw, divw_op)
> DEF_EMIT_REG3_FORMAT(divd, divd_op)
> +DEF_EMIT_REG3_FORMAT(modw, modw_op)
> DEF_EMIT_REG3_FORMAT(modd, modd_op)
> +DEF_EMIT_REG3_FORMAT(divwu, divwu_op)
> DEF_EMIT_REG3_FORMAT(divdu, divdu_op)
> +DEF_EMIT_REG3_FORMAT(modwu, modwu_op)
> DEF_EMIT_REG3_FORMAT(moddu, moddu_op)
> DEF_EMIT_REG3_FORMAT(maskeqz, maskeqz_op)
> DEF_EMIT_REG3_FORMAT(masknez, masknez_op)
> diff --git a/arch/loongarch/net/bpf_jit.c b/arch/loongarch/net/bpf_jit.c
> index 77cd96a6f84c..1a7c8effc37b 100644
> --- a/arch/loongarch/net/bpf_jit.c
> +++ b/arch/loongarch/net/bpf_jit.c
> @@ -288,7 +288,7 @@ static int emit_bpf_tail_call(struct jit_ctx *ctx, int insn)
> * if (index >= array->map.max_entries)
> * goto out;
> */
> - emit_zext_32(ctx, a2, true);
> + emit_insn(ctx, bstrpickd, a2, a2, 31, 0);
>
> off = offsetof(struct bpf_array, map.max_entries);
> emit_insn(ctx, ldwu, t1, a1, off);
> @@ -412,7 +412,6 @@ static int emit_atomic_rmw(const struct bpf_insn *insn, struct jit_ctx *ctx)
> return -EINVAL;
> }
> emit_insn(ctx, amadddbb, src, t1, t3);
> - emit_zext_32(ctx, src, true);
> break;
> case BPF_H:
> if (!cpu_has_lam_bh) {
> @@ -420,23 +419,23 @@ static int emit_atomic_rmw(const struct bpf_insn *insn, struct jit_ctx *ctx)
> return -EINVAL;
> }
> emit_insn(ctx, amadddbh, src, t1, t3);
> - emit_zext_32(ctx, src, true);
> break;
> case BPF_W:
> emit_insn(ctx, amadddbw, src, t1, t3);
> - emit_zext_32(ctx, src, true);
> break;
> case BPF_DW:
> emit_insn(ctx, amadddbd, src, t1, t3);
> break;
> }
> + if (!isdw)
> + emit_insn(ctx, bstrpickd, src, src, 31, 0);
> break;
> case BPF_AND | BPF_FETCH:
> if (isdw) {
> emit_insn(ctx, amanddbd, src, t1, t3);
> } else {
> emit_insn(ctx, amanddbw, src, t1, t3);
> - emit_zext_32(ctx, src, true);
> + emit_insn(ctx, bstrpickd, src, src, 31, 0);
> }
> break;
> case BPF_OR | BPF_FETCH:
> @@ -444,7 +443,7 @@ static int emit_atomic_rmw(const struct bpf_insn *insn, struct jit_ctx *ctx)
> emit_insn(ctx, amordbd, src, t1, t3);
> } else {
> emit_insn(ctx, amordbw, src, t1, t3);
> - emit_zext_32(ctx, src, true);
> + emit_insn(ctx, bstrpickd, src, src, 31, 0);
> }
> break;
> case BPF_XOR | BPF_FETCH:
> @@ -452,7 +451,7 @@ static int emit_atomic_rmw(const struct bpf_insn *insn, struct jit_ctx *ctx)
> emit_insn(ctx, amxordbd, src, t1, t3);
> } else {
> emit_insn(ctx, amxordbw, src, t1, t3);
> - emit_zext_32(ctx, src, true);
> + emit_insn(ctx, bstrpickd, src, src, 31, 0);
> }
> break;
> /* src = atomic_xchg(dst + off, src); */
> @@ -464,7 +463,6 @@ static int emit_atomic_rmw(const struct bpf_insn *insn, struct jit_ctx *ctx)
> return -EINVAL;
> }
> emit_insn(ctx, amswapdbb, src, t1, t3);
> - emit_zext_32(ctx, src, true);
> break;
> case BPF_H:
> if (!cpu_has_lam_bh) {
> @@ -472,16 +470,16 @@ static int emit_atomic_rmw(const struct bpf_insn *insn, struct jit_ctx *ctx)
> return -EINVAL;
> }
> emit_insn(ctx, amswapdbh, src, t1, t3);
> - emit_zext_32(ctx, src, true);
> break;
> case BPF_W:
> emit_insn(ctx, amswapdbw, src, t1, t3);
> - emit_zext_32(ctx, src, true);
> break;
> case BPF_DW:
> emit_insn(ctx, amswapdbd, src, t1, t3);
> break;
> }
> + if (!isdw)
> + emit_insn(ctx, bstrpickd, src, src, 31, 0);
> break;
> /* r0 = atomic_cmpxchg(dst + off, r0, src); */
> case BPF_CMPXCHG:
> @@ -494,13 +492,13 @@ static int emit_atomic_rmw(const struct bpf_insn *insn, struct jit_ctx *ctx)
> emit_insn(ctx, beq, t3, LOONGARCH_GPR_ZERO, -4);
> } else {
> emit_insn(ctx, llw, r0, t1, 0);
> - emit_zext_32(ctx, t2, true);
> - emit_zext_32(ctx, r0, true);
> + emit_insn(ctx, bstrpickd, t2, t2, 31, 0);
> + emit_insn(ctx, bstrpickd, r0, r0, 31, 0);
> emit_insn(ctx, bne, t2, r0, 4);
> move_reg(ctx, t3, src);
> emit_insn(ctx, scw, t3, t1, 0);
> emit_insn(ctx, beq, t3, LOONGARCH_GPR_ZERO, -6);
> - emit_zext_32(ctx, r0, true);
> + emit_insn(ctx, bstrpickd, r0, r0, 31, 0);
> }
> emit_insn(ctx, dbar, DBAR_LLSC_MB);
> break;
> @@ -605,12 +603,6 @@ static int emit_atomic_ld_st(const struct bpf_insn *insn, struct jit_ctx *ctx)
> return 0;
> }
>
> -static bool is_signed_bpf_cond(u8 cond)
> -{
> - return cond == BPF_JSGT || cond == BPF_JSLT ||
> - cond == BPF_JSGE || cond == BPF_JSLE;
> -}
> -
> #define BPF_FIXUP_REG_MASK GENMASK(31, 27)
> #define BPF_FIXUP_OFFSET_MASK GENMASK(26, 0)
> #define REG_DONT_CLEAR_MARKER 0
> @@ -707,7 +699,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> u8 dst = regmap[insn->dst_reg];
> const s16 off = insn->off;
> const s32 imm = insn->imm;
> - const bool is32 = BPF_CLASS(insn->code) == BPF_ALU || BPF_CLASS(insn->code) == BPF_JMP32;
> + const bool is32 = BPF_CLASS(insn->code) == BPF_ALU && BPF_OP(insn->code) != BPF_END;
>
> switch (code) {
> /* dst = src */
> @@ -715,7 +707,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> case BPF_ALU64 | BPF_MOV | BPF_X:
> if (insn_is_cast_user(insn)) {
> move_reg(ctx, t1, src);
> - emit_zext_32(ctx, t1, true);
> + emit_insn(ctx, bstrpickd, t1, t1, 31, 0);
> move_imm(ctx, dst, (ctx->user_vm_start >> 32) << 32);
> emit_insn(ctx, beq, t1, LOONGARCH_GPR_ZERO, 2);
> emit_insn(ctx, or, t1, dst, t1);
> @@ -734,15 +726,12 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> switch (off) {
> case 0:
> move_reg(ctx, dst, src);
> - emit_zext_32(ctx, dst, is32);
> break;
> case 8:
> emit_insn(ctx, extwb, dst, src);
> - emit_zext_32(ctx, dst, is32);
> break;
> case 16:
> emit_insn(ctx, extwh, dst, src);
> - emit_zext_32(ctx, dst, is32);
> break;
> case 32:
> emit_insn(ctx, addw, dst, src, LOONGARCH_GPR_ZERO);
> @@ -752,151 +741,188 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
>
> /* dst = imm */
> case BPF_ALU | BPF_MOV | BPF_K:
> + move_imm(ctx, dst, (u32)imm);
> + return 0;
> case BPF_ALU64 | BPF_MOV | BPF_K:
> move_imm(ctx, dst, imm);
> - emit_zext_32(ctx, dst, is32);
> break;
dst = imm is the simplest case, I think the 64bit version can also
"return 0" here. If you agree, I will modify it when apply.
Huacai
>
> /* dst = dst + src */
> case BPF_ALU | BPF_ADD | BPF_X:
> + emit_insn(ctx, addw, dst, dst, src);
> + break;
> case BPF_ALU64 | BPF_ADD | BPF_X:
> emit_insn(ctx, addd, dst, dst, src);
> - emit_zext_32(ctx, dst, is32);
> break;
>
> /* dst = dst + imm */
> case BPF_ALU | BPF_ADD | BPF_K:
> + if (is_signed_imm12(imm)) {
> + emit_insn(ctx, addiw, dst, dst, imm);
> + } else {
> + move_imm(ctx, t1, (u32)imm);
> + emit_insn(ctx, addw, dst, dst, t1);
> + }
> + break;
> case BPF_ALU64 | BPF_ADD | BPF_K:
> if (is_signed_imm12(imm)) {
> emit_insn(ctx, addid, dst, dst, imm);
> } else {
> move_imm(ctx, t1, imm);
> - emit_zext_32(ctx, t1, is32);
> emit_insn(ctx, addd, dst, dst, t1);
> }
> - emit_zext_32(ctx, dst, is32);
> break;
>
> /* dst = dst - src */
> case BPF_ALU | BPF_SUB | BPF_X:
> + emit_insn(ctx, subw, dst, dst, src);
> + break;
> case BPF_ALU64 | BPF_SUB | BPF_X:
> emit_insn(ctx, subd, dst, dst, src);
> - emit_zext_32(ctx, dst, is32);
> break;
>
> /* dst = dst - imm */
> case BPF_ALU | BPF_SUB | BPF_K:
> + if (is_signed_imm12(-imm)) {
> + emit_insn(ctx, addiw, dst, dst, -imm);
> + } else {
> + move_imm(ctx, t1, (u32)imm);
> + emit_insn(ctx, subw, dst, dst, t1);
> + }
> + break;
> case BPF_ALU64 | BPF_SUB | BPF_K:
> if (is_signed_imm12(-imm)) {
> emit_insn(ctx, addid, dst, dst, -imm);
> } else {
> move_imm(ctx, t1, imm);
> - emit_zext_32(ctx, t1, is32);
> emit_insn(ctx, subd, dst, dst, t1);
> }
> - emit_zext_32(ctx, dst, is32);
> break;
>
> /* dst = dst * src */
> case BPF_ALU | BPF_MUL | BPF_X:
> + emit_insn(ctx, mulw, dst, dst, src);
> + break;
> case BPF_ALU64 | BPF_MUL | BPF_X:
> emit_insn(ctx, muld, dst, dst, src);
> - emit_zext_32(ctx, dst, is32);
> break;
>
> /* dst = dst * imm */
> case BPF_ALU | BPF_MUL | BPF_K:
> + move_imm(ctx, t1, (u32)imm);
> + emit_insn(ctx, mulw, dst, dst, t1);
> + break;
> case BPF_ALU64 | BPF_MUL | BPF_K:
> move_imm(ctx, t1, imm);
> - emit_zext_32(ctx, t1, is32);
> emit_insn(ctx, muld, dst, dst, t1);
> - emit_zext_32(ctx, dst, is32);
> break;
>
> /* dst = dst / src */
> case BPF_ALU | BPF_DIV | BPF_X:
> - case BPF_ALU64 | BPF_DIV | BPF_X:
> - if (!off) {
> - emit_zext_32(ctx, dst, is32);
> - move_reg(ctx, t1, src);
> - emit_zext_32(ctx, t1, is32);
> - emit_insn(ctx, divdu, dst, dst, t1);
> - emit_zext_32(ctx, dst, is32);
> - } else {
> - emit_sext_32(ctx, dst, is32);
> - move_reg(ctx, t1, src);
> - emit_sext_32(ctx, t1, is32);
> - emit_insn(ctx, divd, dst, dst, t1);
> - emit_zext_32(ctx, dst, is32);
> + if (cpu_has_div32) {
> + if (!off)
> + emit_insn(ctx, divwu, dst, dst, src);
> + else
> + emit_insn(ctx, divw, dst, dst, src);
> + break;
> }
> + /*
> + * Legacy fallback for !div32: src must be extended into a
> + * temporary register to keep the original src register intact.
> + */
> + emit_insn(ctx, addw, dst, dst, LOONGARCH_GPR_ZERO);
> + emit_insn(ctx, addw, t1, src, LOONGARCH_GPR_ZERO);
> + if (!off)
> + emit_insn(ctx, divwu, dst, dst, t1);
> + else
> + emit_insn(ctx, divw, dst, dst, t1);
> + break;
> + case BPF_ALU64 | BPF_DIV | BPF_X:
> + if (!off)
> + emit_insn(ctx, divdu, dst, dst, src);
> + else
> + emit_insn(ctx, divd, dst, dst, src);
> break;
>
> /* dst = dst / imm */
> case BPF_ALU | BPF_DIV | BPF_K:
> + move_imm(ctx, t1, (u32)imm);
> + if (!cpu_has_div32) {
> + emit_insn(ctx, addw, dst, dst, LOONGARCH_GPR_ZERO);
> + emit_insn(ctx, addw, t1, t1, LOONGARCH_GPR_ZERO);
> + }
> + if (!off)
> + emit_insn(ctx, divwu, dst, dst, t1);
> + else
> + emit_insn(ctx, divw, dst, dst, t1);
> + break;
> case BPF_ALU64 | BPF_DIV | BPF_K:
> - if (!off) {
> - move_imm(ctx, t1, imm);
> - emit_zext_32(ctx, t1, is32);
> - emit_zext_32(ctx, dst, is32);
> + move_imm(ctx, t1, imm);
> + if (!off)
> emit_insn(ctx, divdu, dst, dst, t1);
> - emit_zext_32(ctx, dst, is32);
> - } else {
> - move_imm(ctx, t1, imm);
> - emit_sext_32(ctx, t1, is32);
> - emit_sext_32(ctx, dst, is32);
> + else
> emit_insn(ctx, divd, dst, dst, t1);
> - emit_zext_32(ctx, dst, is32);
> - }
> break;
>
> /* dst = dst % src */
> case BPF_ALU | BPF_MOD | BPF_X:
> - case BPF_ALU64 | BPF_MOD | BPF_X:
> - if (!off) {
> - emit_zext_32(ctx, dst, is32);
> - move_reg(ctx, t1, src);
> - emit_zext_32(ctx, t1, is32);
> - emit_insn(ctx, moddu, dst, dst, t1);
> - emit_zext_32(ctx, dst, is32);
> - } else {
> - emit_sext_32(ctx, dst, is32);
> - move_reg(ctx, t1, src);
> - emit_sext_32(ctx, t1, is32);
> - emit_insn(ctx, modd, dst, dst, t1);
> - emit_zext_32(ctx, dst, is32);
> + if (cpu_has_div32) {
> + if (!off)
> + emit_insn(ctx, modwu, dst, dst, src);
> + else
> + emit_insn(ctx, modw, dst, dst, src);
> + break;
> }
> + /*
> + * Legacy fallback for !div32: src must be extended into a
> + * temporary register to keep the original src register intact.
> + */
> + emit_insn(ctx, addw, dst, dst, LOONGARCH_GPR_ZERO);
> + emit_insn(ctx, addw, t1, src, LOONGARCH_GPR_ZERO);
> + if (!off)
> + emit_insn(ctx, modwu, dst, dst, t1);
> + else
> + emit_insn(ctx, modw, dst, dst, t1);
> + break;
> + case BPF_ALU64 | BPF_MOD | BPF_X:
> + if (!off)
> + emit_insn(ctx, moddu, dst, dst, src);
> + else
> + emit_insn(ctx, modd, dst, dst, src);
> break;
>
> /* dst = dst % imm */
> case BPF_ALU | BPF_MOD | BPF_K:
> + move_imm(ctx, t1, (u32)imm);
> + if (!cpu_has_div32) {
> + emit_insn(ctx, addw, dst, dst, LOONGARCH_GPR_ZERO);
> + emit_insn(ctx, addw, t1, t1, LOONGARCH_GPR_ZERO);
> + }
> + if (!off)
> + emit_insn(ctx, modwu, dst, dst, t1);
> + else
> + emit_insn(ctx, modw, dst, dst, t1);
> + break;
> case BPF_ALU64 | BPF_MOD | BPF_K:
> - if (!off) {
> - move_imm(ctx, t1, imm);
> - emit_zext_32(ctx, t1, is32);
> - emit_zext_32(ctx, dst, is32);
> + move_imm(ctx, t1, imm);
> + if (!off)
> emit_insn(ctx, moddu, dst, dst, t1);
> - emit_zext_32(ctx, dst, is32);
> - } else {
> - move_imm(ctx, t1, imm);
> - emit_sext_32(ctx, t1, is32);
> - emit_sext_32(ctx, dst, is32);
> + else
> emit_insn(ctx, modd, dst, dst, t1);
> - emit_zext_32(ctx, dst, is32);
> - }
> break;
>
> /* dst = -dst */
> case BPF_ALU | BPF_NEG:
> + emit_insn(ctx, subw, dst, LOONGARCH_GPR_ZERO, dst);
> + break;
> case BPF_ALU64 | BPF_NEG:
> emit_insn(ctx, subd, dst, LOONGARCH_GPR_ZERO, dst);
> - emit_zext_32(ctx, dst, is32);
> break;
>
> /* dst = dst & src */
> case BPF_ALU | BPF_AND | BPF_X:
> case BPF_ALU64 | BPF_AND | BPF_X:
> emit_insn(ctx, and, dst, dst, src);
> - emit_zext_32(ctx, dst, is32);
> break;
>
> /* dst = dst & imm */
> @@ -905,18 +931,18 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> if (is_unsigned_imm12(imm)) {
> emit_insn(ctx, andi, dst, dst, imm);
> } else {
> - move_imm(ctx, t1, imm);
> - emit_zext_32(ctx, t1, is32);
> + if (is32)
> + move_imm(ctx, t1, (u32)imm);
> + else
> + move_imm(ctx, t1, imm);
> emit_insn(ctx, and, dst, dst, t1);
> }
> - emit_zext_32(ctx, dst, is32);
> break;
>
> /* dst = dst | src */
> case BPF_ALU | BPF_OR | BPF_X:
> case BPF_ALU64 | BPF_OR | BPF_X:
> emit_insn(ctx, or, dst, dst, src);
> - emit_zext_32(ctx, dst, is32);
> break;
>
> /* dst = dst | imm */
> @@ -925,18 +951,18 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> if (is_unsigned_imm12(imm)) {
> emit_insn(ctx, ori, dst, dst, imm);
> } else {
> - move_imm(ctx, t1, imm);
> - emit_zext_32(ctx, t1, is32);
> + if (is32)
> + move_imm(ctx, t1, (u32)imm);
> + else
> + move_imm(ctx, t1, imm);
> emit_insn(ctx, or, dst, dst, t1);
> }
> - emit_zext_32(ctx, dst, is32);
> break;
>
> /* dst = dst ^ src */
> case BPF_ALU | BPF_XOR | BPF_X:
> case BPF_ALU64 | BPF_XOR | BPF_X:
> emit_insn(ctx, xor, dst, dst, src);
> - emit_zext_32(ctx, dst, is32);
> break;
>
> /* dst = dst ^ imm */
> @@ -945,19 +971,18 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> if (is_unsigned_imm12(imm)) {
> emit_insn(ctx, xori, dst, dst, imm);
> } else {
> - move_imm(ctx, t1, imm);
> - emit_zext_32(ctx, t1, is32);
> + if (is32)
> + move_imm(ctx, t1, (u32)imm);
> + else
> + move_imm(ctx, t1, imm);
> emit_insn(ctx, xor, dst, dst, t1);
> }
> - emit_zext_32(ctx, dst, is32);
> break;
>
> /* dst = dst << src (logical) */
> case BPF_ALU | BPF_LSH | BPF_X:
> emit_insn(ctx, sllw, dst, dst, src);
> - emit_zext_32(ctx, dst, is32);
> break;
> -
> case BPF_ALU64 | BPF_LSH | BPF_X:
> emit_insn(ctx, slld, dst, dst, src);
> break;
> @@ -965,9 +990,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> /* dst = dst << imm (logical) */
> case BPF_ALU | BPF_LSH | BPF_K:
> emit_insn(ctx, slliw, dst, dst, imm);
> - emit_zext_32(ctx, dst, is32);
> break;
> -
> case BPF_ALU64 | BPF_LSH | BPF_K:
> emit_insn(ctx, sllid, dst, dst, imm);
> break;
> @@ -975,9 +998,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> /* dst = dst >> src (logical) */
> case BPF_ALU | BPF_RSH | BPF_X:
> emit_insn(ctx, srlw, dst, dst, src);
> - emit_zext_32(ctx, dst, is32);
> break;
> -
> case BPF_ALU64 | BPF_RSH | BPF_X:
> emit_insn(ctx, srld, dst, dst, src);
> break;
> @@ -985,9 +1006,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> /* dst = dst >> imm (logical) */
> case BPF_ALU | BPF_RSH | BPF_K:
> emit_insn(ctx, srliw, dst, dst, imm);
> - emit_zext_32(ctx, dst, is32);
> break;
> -
> case BPF_ALU64 | BPF_RSH | BPF_K:
> emit_insn(ctx, srlid, dst, dst, imm);
> break;
> @@ -995,9 +1014,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> /* dst = dst >> src (arithmetic) */
> case BPF_ALU | BPF_ARSH | BPF_X:
> emit_insn(ctx, sraw, dst, dst, src);
> - emit_zext_32(ctx, dst, is32);
> break;
> -
> case BPF_ALU64 | BPF_ARSH | BPF_X:
> emit_insn(ctx, srad, dst, dst, src);
> break;
> @@ -1005,9 +1022,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> /* dst = dst >> imm (arithmetic) */
> case BPF_ALU | BPF_ARSH | BPF_K:
> emit_insn(ctx, sraiw, dst, dst, imm);
> - emit_zext_32(ctx, dst, is32);
> break;
> -
> case BPF_ALU64 | BPF_ARSH | BPF_K:
> emit_insn(ctx, sraid, dst, dst, imm);
> break;
> @@ -1016,12 +1031,10 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> case BPF_ALU | BPF_END | BPF_FROM_LE:
> switch (imm) {
> case 16:
> - /* zero-extend 16 bits into 64 bits */
> emit_insn(ctx, bstrpickd, dst, dst, 15, 0);
> break;
> case 32:
> - /* zero-extend 32 bits into 64 bits */
> - emit_zext_32(ctx, dst, is32);
> + emit_insn(ctx, bstrpickd, dst, dst, 31, 0);
> break;
> case 64:
> /* do nothing */
> @@ -1034,13 +1047,11 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> switch (imm) {
> case 16:
> emit_insn(ctx, revb2h, dst, dst);
> - /* zero-extend 16 bits into 64 bits */
> emit_insn(ctx, bstrpickd, dst, dst, 15, 0);
> break;
> case 32:
> emit_insn(ctx, revb2w, dst, dst);
> - /* clear the upper 32 bits */
> - emit_zext_32(ctx, dst, true);
> + emit_insn(ctx, bstrpickd, dst, dst, 31, 0);
> break;
> case 64:
> emit_insn(ctx, revbd, dst, dst);
> @@ -1048,7 +1059,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> }
> break;
>
> - /* PC += off if dst cond src */
> + /* PC += off if dst cond src (64-bit) */
> case BPF_JMP | BPF_JEQ | BPF_X:
> case BPF_JMP | BPF_JNE | BPF_X:
> case BPF_JMP | BPF_JGT | BPF_X:
> @@ -1059,31 +1070,38 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> case BPF_JMP | BPF_JSGE | BPF_X:
> case BPF_JMP | BPF_JSLT | BPF_X:
> case BPF_JMP | BPF_JSLE | BPF_X:
> + jmp_offset = bpf2la_offset(i, off, ctx);
> + if (emit_cond_jmp(ctx, cond, dst, src, jmp_offset) < 0)
> + goto toofar;
> + break;
> +
> + /* PC += off if dst cond src (32-bit unsigned) */
> case BPF_JMP32 | BPF_JEQ | BPF_X:
> case BPF_JMP32 | BPF_JNE | BPF_X:
> case BPF_JMP32 | BPF_JGT | BPF_X:
> case BPF_JMP32 | BPF_JGE | BPF_X:
> case BPF_JMP32 | BPF_JLT | BPF_X:
> case BPF_JMP32 | BPF_JLE | BPF_X:
> + jmp_offset = bpf2la_offset(i, off, ctx);
> + emit_insn(ctx, bstrpickd, t1, dst, 31, 0);
> + emit_insn(ctx, bstrpickd, t2, src, 31, 0);
> + if (emit_cond_jmp(ctx, cond, t1, t2, jmp_offset) < 0)
> + goto toofar;
> + break;
> +
> + /* PC += off if dst cond src (32-bit signed) */
> case BPF_JMP32 | BPF_JSGT | BPF_X:
> case BPF_JMP32 | BPF_JSGE | BPF_X:
> case BPF_JMP32 | BPF_JSLT | BPF_X:
> case BPF_JMP32 | BPF_JSLE | BPF_X:
> jmp_offset = bpf2la_offset(i, off, ctx);
> - move_reg(ctx, t1, dst);
> - move_reg(ctx, t2, src);
> - if (is_signed_bpf_cond(BPF_OP(code))) {
> - emit_sext_32(ctx, t1, is32);
> - emit_sext_32(ctx, t2, is32);
> - } else {
> - emit_zext_32(ctx, t1, is32);
> - emit_zext_32(ctx, t2, is32);
> - }
> + emit_insn(ctx, addw, t1, dst, LOONGARCH_GPR_ZERO);
> + emit_insn(ctx, addw, t2, src, LOONGARCH_GPR_ZERO);
> if (emit_cond_jmp(ctx, cond, t1, t2, jmp_offset) < 0)
> goto toofar;
> break;
>
> - /* PC += off if dst cond imm */
> + /* PC += off if dst cond imm (64-bit) */
> case BPF_JMP | BPF_JEQ | BPF_K:
> case BPF_JMP | BPF_JNE | BPF_K:
> case BPF_JMP | BPF_JGT | BPF_K:
> @@ -1094,12 +1112,38 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> case BPF_JMP | BPF_JSGE | BPF_K:
> case BPF_JMP | BPF_JSLT | BPF_K:
> case BPF_JMP | BPF_JSLE | BPF_K:
> + jmp_offset = bpf2la_offset(i, off, ctx);
> + if (imm) {
> + move_imm(ctx, t1, imm);
> + tm = t1;
> + } else {
> + tm = LOONGARCH_GPR_ZERO;
> + }
> + if (emit_cond_jmp(ctx, cond, dst, tm, jmp_offset) < 0)
> + goto toofar;
> + break;
> +
> + /* PC += off if dst cond imm (32-bit unsigned) */
> case BPF_JMP32 | BPF_JEQ | BPF_K:
> case BPF_JMP32 | BPF_JNE | BPF_K:
> case BPF_JMP32 | BPF_JGT | BPF_K:
> case BPF_JMP32 | BPF_JGE | BPF_K:
> case BPF_JMP32 | BPF_JLT | BPF_K:
> case BPF_JMP32 | BPF_JLE | BPF_K:
> + jmp_offset = bpf2la_offset(i, off, ctx);
> + if (imm) {
> + move_imm(ctx, t1, imm);
> + emit_insn(ctx, bstrpickd, t1, t1, 31, 0);
> + tm = t1;
> + } else {
> + tm = LOONGARCH_GPR_ZERO;
> + }
> + emit_insn(ctx, bstrpickd, t2, dst, 31, 0);
> + if (emit_cond_jmp(ctx, cond, t2, tm, jmp_offset) < 0)
> + goto toofar;
> + break;
> +
> + /* PC += off if dst cond imm (32-bit signed) */
> case BPF_JMP32 | BPF_JSGT | BPF_K:
> case BPF_JMP32 | BPF_JSGE | BPF_K:
> case BPF_JMP32 | BPF_JSLT | BPF_K:
> @@ -1107,19 +1151,12 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> jmp_offset = bpf2la_offset(i, off, ctx);
> if (imm) {
> move_imm(ctx, t1, imm);
> + emit_insn(ctx, addw, t1, t1, LOONGARCH_GPR_ZERO);
> tm = t1;
> } else {
> - /* If imm is 0, simply use zero register. */
> tm = LOONGARCH_GPR_ZERO;
> }
> - move_reg(ctx, t2, dst);
> - if (is_signed_bpf_cond(BPF_OP(code))) {
> - emit_sext_32(ctx, tm, is32);
> - emit_sext_32(ctx, t2, is32);
> - } else {
> - emit_zext_32(ctx, tm, is32);
> - emit_zext_32(ctx, t2, is32);
> - }
> + emit_insn(ctx, addw, t2, dst, LOONGARCH_GPR_ZERO);
> if (emit_cond_jmp(ctx, cond, t2, tm, jmp_offset) < 0)
> goto toofar;
> break;
> @@ -1129,7 +1166,8 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> case BPF_JMP32 | BPF_JSET | BPF_X:
> jmp_offset = bpf2la_offset(i, off, ctx);
> emit_insn(ctx, and, t1, dst, src);
> - emit_zext_32(ctx, t1, is32);
> + if (BPF_CLASS(code) == BPF_JMP32)
> + emit_insn(ctx, bstrpickd, t1, t1, 31, 0);
> if (emit_cond_jmp(ctx, cond, t1, LOONGARCH_GPR_ZERO, jmp_offset) < 0)
> goto toofar;
> break;
> @@ -1139,9 +1177,9 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> case BPF_JMP32 | BPF_JSET | BPF_K:
> jmp_offset = bpf2la_offset(i, off, ctx);
> move_imm(ctx, t1, imm);
> - emit_zext_32(ctx, t1, is32);
> emit_insn(ctx, and, t1, dst, t1);
> - emit_zext_32(ctx, t1, is32);
> + if (BPF_CLASS(code) == BPF_JMP32)
> + emit_insn(ctx, bstrpickd, t1, t1, 31, 0);
> if (emit_cond_jmp(ctx, cond, t1, LOONGARCH_GPR_ZERO, jmp_offset) < 0)
> goto toofar;
> break;
> @@ -1468,6 +1506,9 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> return -EINVAL;
> }
>
> + if (is32)
> + emit_insn(ctx, bstrpickd, dst, dst, 31, 0);
> +
> return 0;
>
> toofar:
> diff --git a/arch/loongarch/net/bpf_jit.h b/arch/loongarch/net/bpf_jit.h
> index 57a98f8a055b..2bd0dc5b911c 100644
> --- a/arch/loongarch/net/bpf_jit.h
> +++ b/arch/loongarch/net/bpf_jit.h
> @@ -72,24 +72,6 @@ static inline int epilogue_offset(const struct jit_ctx *ctx)
> return (to - from);
> }
>
> -/* Zero-extend 32 bits into 64 bits */
> -static inline void emit_zext_32(struct jit_ctx *ctx, enum loongarch_gpr reg, bool is32)
> -{
> - if (!is32)
> - return;
> -
> - emit_insn(ctx, lu32id, reg, 0);
> -}
> -
> -/* Signed-extend 32 bits into 64 bits */
> -static inline void emit_sext_32(struct jit_ctx *ctx, enum loongarch_gpr reg, bool is32)
> -{
> - if (!is32)
> - return;
> -
> - emit_insn(ctx, addiw, reg, reg, 0);
> -}
> -
> /* Emit proper extension according to ABI requirements.
> * Note that it requires a value of size `size` already resides in register `reg`.
> */
> --
> 2.42.0
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v3 3/3] LoongArch: BPF: Optimize 32-bit operations using native instructions
2026-10-28 4:19 ` Huacai Chen
@ 2026-10-28 6:55 ` Hengqi Chen
0 siblings, 0 replies; 15+ messages in thread
From: Hengqi Chen @ 2026-10-28 6:55 UTC (permalink / raw)
To: Huacai Chen; +Cc: Tiezhu Yang, loongarch, bpf, linux-kernel
On Wed, Oct 28, 2026 at 12:19 PM Huacai Chen <chenhuacai@kernel.org> wrote:
>
> Hi, Hengqi,
>
> On Fri, Oct 16, 2026 at 4:10 PM Tiezhu Yang <yangtiezhu@loongson.cn> wrote:
> >
> > Currently, the LoongArch BPF JIT emits numerous redundant zero and sign
> > extensions across 32-bit operations, bloating the final JIT image size
> > and also leading to runtime overhead, particularly in the division and
> > modulo operations. Thus, it is necessary to optimize 32-bit operations
> > by emitting LoongArch native 32-bit word instructions.
> >
> > For general ALU32 operations, utilize native 32-bit word instructions
> > for core calculations. Given that the BPF spec requires the upper 32
> > bits of the destination register to be zero-extended upon completion,
> > use a unified 'bstrpickd' instruction to clear the upper 32 bits at
> > the common exit of build_insn(), while immediate loads are optimized
> > via (u32) casting to bypass the trailing extension.
> >
> > According to the LoongArch Reference Manual, native 32-bit division
> > and modulo instructions (DIV.W[U] and MOD.W[U]) require the upper
> > bits [63:31] of both input operands to be all 0s or all 1s (0x0 or
> > 0x1ffffffff), that is to say, they need to be properly sign-extended
> > by the software on CPU cores without the DIV32 feature. Otherwise,
> > the 32-bit division and modulo instructions could trigger undefined
> > hardware behaviors and return unexpected values.
> >
> > To ensure both performance and hardware compatibility, leverage the
> > newly introduced cpu_has_div32 helper:
> > (1) on newer CPUs with DIV32, emit native 32-bit div/mod instructions
> > directly for optimal performance;
> > (2) on older CPUs without DIV32, sign-extend the destination register
> > in place and the source register into a temporary register via 'addw'
> > before executing the native 32-bit div/mod instructions. Note that the
> > JIT compiler must never alter the state of the src register because it
> > remains read-only for the duration of the instruction.
> >
> > For JMP32 conditional branches, the source operands may originate from
> > 64-bit contexts where the upper 32 bits contain arbitrary data, so use
> > 6 independent case structures to prevent 64-bit comparison inversions.
> > Furthermore, use 'bstrpickd' to mask out upper bits for unsigned JMP32,
> > and 'addw' with GPR_ZERO to re-align sign bits for signed JMP32.
> >
> > With this patch, the emit_{zext, sext}_32() and is_signed_bpf_cond()
> > helpers are completely wiped out from the header and source files,
> > delivering a native execution stream with a reduced JIT image size,
> > especially for division and mod operations.
> >
> > All of the testcases of test_bpf passed on Loongson-3A6000 (with DIV32)
> > and Loongson-3A5000 (without DIV32), here are the test results:
> >
> > $ sudo modprobe test_bpf
> > $ dmesg -t | grep Summary
> > test_bpf: Summary: 1061 PASSED, 0 FAILED, [1049/1049 JIT'ed]
> > test_bpf: test_tail_calls: Summary: 10 PASSED, 0 FAILED, [10/10 JIT'ed]
> > test_bpf: test_skb_segment: Summary: 2 PASSED, 0 FAILED
> >
> > For the BPF selftests test_verifier and test_progs, there are no any
> > regressions.
> >
> > Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
> > ---
> > arch/loongarch/include/asm/inst.h | 6 +
> > arch/loongarch/net/bpf_jit.c | 313 +++++++++++++++++-------------
> > arch/loongarch/net/bpf_jit.h | 18 --
> > 3 files changed, 183 insertions(+), 154 deletions(-)
> >
> > diff --git a/arch/loongarch/include/asm/inst.h b/arch/loongarch/include/asm/inst.h
> > index 585667e361c2..b55aae325300 100644
> > --- a/arch/loongarch/include/asm/inst.h
> > +++ b/arch/loongarch/include/asm/inst.h
> > @@ -752,11 +752,17 @@ static inline void emit_##NAME(union loongarch_instruction *insn, \
> >
> > DEF_EMIT_REG3_FORMAT(addw, addw_op)
> > DEF_EMIT_REG3_FORMAT(addd, addd_op)
> > +DEF_EMIT_REG3_FORMAT(subw, subw_op)
> > DEF_EMIT_REG3_FORMAT(subd, subd_op)
> > +DEF_EMIT_REG3_FORMAT(mulw, mulw_op)
> > DEF_EMIT_REG3_FORMAT(muld, muld_op)
> > +DEF_EMIT_REG3_FORMAT(divw, divw_op)
> > DEF_EMIT_REG3_FORMAT(divd, divd_op)
> > +DEF_EMIT_REG3_FORMAT(modw, modw_op)
> > DEF_EMIT_REG3_FORMAT(modd, modd_op)
> > +DEF_EMIT_REG3_FORMAT(divwu, divwu_op)
> > DEF_EMIT_REG3_FORMAT(divdu, divdu_op)
> > +DEF_EMIT_REG3_FORMAT(modwu, modwu_op)
> > DEF_EMIT_REG3_FORMAT(moddu, moddu_op)
> > DEF_EMIT_REG3_FORMAT(maskeqz, maskeqz_op)
> > DEF_EMIT_REG3_FORMAT(masknez, masknez_op)
> > diff --git a/arch/loongarch/net/bpf_jit.c b/arch/loongarch/net/bpf_jit.c
> > index 77cd96a6f84c..1a7c8effc37b 100644
> > --- a/arch/loongarch/net/bpf_jit.c
> > +++ b/arch/loongarch/net/bpf_jit.c
> > @@ -288,7 +288,7 @@ static int emit_bpf_tail_call(struct jit_ctx *ctx, int insn)
> > * if (index >= array->map.max_entries)
> > * goto out;
> > */
> > - emit_zext_32(ctx, a2, true);
> > + emit_insn(ctx, bstrpickd, a2, a2, 31, 0);
> >
> > off = offsetof(struct bpf_array, map.max_entries);
> > emit_insn(ctx, ldwu, t1, a1, off);
> > @@ -412,7 +412,6 @@ static int emit_atomic_rmw(const struct bpf_insn *insn, struct jit_ctx *ctx)
> > return -EINVAL;
> > }
> > emit_insn(ctx, amadddbb, src, t1, t3);
> > - emit_zext_32(ctx, src, true);
> > break;
> > case BPF_H:
> > if (!cpu_has_lam_bh) {
> > @@ -420,23 +419,23 @@ static int emit_atomic_rmw(const struct bpf_insn *insn, struct jit_ctx *ctx)
> > return -EINVAL;
> > }
> > emit_insn(ctx, amadddbh, src, t1, t3);
> > - emit_zext_32(ctx, src, true);
> > break;
> > case BPF_W:
> > emit_insn(ctx, amadddbw, src, t1, t3);
> > - emit_zext_32(ctx, src, true);
> > break;
> > case BPF_DW:
> > emit_insn(ctx, amadddbd, src, t1, t3);
> > break;
> > }
> > + if (!isdw)
> > + emit_insn(ctx, bstrpickd, src, src, 31, 0);
> > break;
> > case BPF_AND | BPF_FETCH:
> > if (isdw) {
> > emit_insn(ctx, amanddbd, src, t1, t3);
> > } else {
> > emit_insn(ctx, amanddbw, src, t1, t3);
> > - emit_zext_32(ctx, src, true);
> > + emit_insn(ctx, bstrpickd, src, src, 31, 0);
> > }
> > break;
> > case BPF_OR | BPF_FETCH:
> > @@ -444,7 +443,7 @@ static int emit_atomic_rmw(const struct bpf_insn *insn, struct jit_ctx *ctx)
> > emit_insn(ctx, amordbd, src, t1, t3);
> > } else {
> > emit_insn(ctx, amordbw, src, t1, t3);
> > - emit_zext_32(ctx, src, true);
> > + emit_insn(ctx, bstrpickd, src, src, 31, 0);
> > }
> > break;
> > case BPF_XOR | BPF_FETCH:
> > @@ -452,7 +451,7 @@ static int emit_atomic_rmw(const struct bpf_insn *insn, struct jit_ctx *ctx)
> > emit_insn(ctx, amxordbd, src, t1, t3);
> > } else {
> > emit_insn(ctx, amxordbw, src, t1, t3);
> > - emit_zext_32(ctx, src, true);
> > + emit_insn(ctx, bstrpickd, src, src, 31, 0);
> > }
> > break;
> > /* src = atomic_xchg(dst + off, src); */
> > @@ -464,7 +463,6 @@ static int emit_atomic_rmw(const struct bpf_insn *insn, struct jit_ctx *ctx)
> > return -EINVAL;
> > }
> > emit_insn(ctx, amswapdbb, src, t1, t3);
> > - emit_zext_32(ctx, src, true);
> > break;
> > case BPF_H:
> > if (!cpu_has_lam_bh) {
> > @@ -472,16 +470,16 @@ static int emit_atomic_rmw(const struct bpf_insn *insn, struct jit_ctx *ctx)
> > return -EINVAL;
> > }
> > emit_insn(ctx, amswapdbh, src, t1, t3);
> > - emit_zext_32(ctx, src, true);
> > break;
> > case BPF_W:
> > emit_insn(ctx, amswapdbw, src, t1, t3);
> > - emit_zext_32(ctx, src, true);
> > break;
> > case BPF_DW:
> > emit_insn(ctx, amswapdbd, src, t1, t3);
> > break;
> > }
> > + if (!isdw)
> > + emit_insn(ctx, bstrpickd, src, src, 31, 0);
> > break;
> > /* r0 = atomic_cmpxchg(dst + off, r0, src); */
> > case BPF_CMPXCHG:
> > @@ -494,13 +492,13 @@ static int emit_atomic_rmw(const struct bpf_insn *insn, struct jit_ctx *ctx)
> > emit_insn(ctx, beq, t3, LOONGARCH_GPR_ZERO, -4);
> > } else {
> > emit_insn(ctx, llw, r0, t1, 0);
> > - emit_zext_32(ctx, t2, true);
> > - emit_zext_32(ctx, r0, true);
> > + emit_insn(ctx, bstrpickd, t2, t2, 31, 0);
> > + emit_insn(ctx, bstrpickd, r0, r0, 31, 0);
> > emit_insn(ctx, bne, t2, r0, 4);
> > move_reg(ctx, t3, src);
> > emit_insn(ctx, scw, t3, t1, 0);
> > emit_insn(ctx, beq, t3, LOONGARCH_GPR_ZERO, -6);
> > - emit_zext_32(ctx, r0, true);
> > + emit_insn(ctx, bstrpickd, r0, r0, 31, 0);
> > }
> > emit_insn(ctx, dbar, DBAR_LLSC_MB);
> > break;
> > @@ -605,12 +603,6 @@ static int emit_atomic_ld_st(const struct bpf_insn *insn, struct jit_ctx *ctx)
> > return 0;
> > }
> >
> > -static bool is_signed_bpf_cond(u8 cond)
> > -{
> > - return cond == BPF_JSGT || cond == BPF_JSLT ||
> > - cond == BPF_JSGE || cond == BPF_JSLE;
> > -}
> > -
> > #define BPF_FIXUP_REG_MASK GENMASK(31, 27)
> > #define BPF_FIXUP_OFFSET_MASK GENMASK(26, 0)
> > #define REG_DONT_CLEAR_MARKER 0
> > @@ -707,7 +699,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> > u8 dst = regmap[insn->dst_reg];
> > const s16 off = insn->off;
> > const s32 imm = insn->imm;
> > - const bool is32 = BPF_CLASS(insn->code) == BPF_ALU || BPF_CLASS(insn->code) == BPF_JMP32;
> > + const bool is32 = BPF_CLASS(insn->code) == BPF_ALU && BPF_OP(insn->code) != BPF_END;
> >
> > switch (code) {
> > /* dst = src */
> > @@ -715,7 +707,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> > case BPF_ALU64 | BPF_MOV | BPF_X:
> > if (insn_is_cast_user(insn)) {
> > move_reg(ctx, t1, src);
> > - emit_zext_32(ctx, t1, true);
> > + emit_insn(ctx, bstrpickd, t1, t1, 31, 0);
> > move_imm(ctx, dst, (ctx->user_vm_start >> 32) << 32);
> > emit_insn(ctx, beq, t1, LOONGARCH_GPR_ZERO, 2);
> > emit_insn(ctx, or, t1, dst, t1);
> > @@ -734,15 +726,12 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> > switch (off) {
> > case 0:
> > move_reg(ctx, dst, src);
> > - emit_zext_32(ctx, dst, is32);
> > break;
> > case 8:
> > emit_insn(ctx, extwb, dst, src);
> > - emit_zext_32(ctx, dst, is32);
> > break;
> > case 16:
> > emit_insn(ctx, extwh, dst, src);
> > - emit_zext_32(ctx, dst, is32);
> > break;
> > case 32:
> > emit_insn(ctx, addw, dst, src, LOONGARCH_GPR_ZERO);
> > @@ -752,151 +741,188 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> >
> > /* dst = imm */
> > case BPF_ALU | BPF_MOV | BPF_K:
> > + move_imm(ctx, dst, (u32)imm);
> > + return 0;
> > case BPF_ALU64 | BPF_MOV | BPF_K:
> > move_imm(ctx, dst, imm);
> > - emit_zext_32(ctx, dst, is32);
> > break;
> dst = imm is the simplest case, I think the 64bit version can also
> "return 0" here. If you agree, I will modify it when apply.
>
I am fine with the change. Cheers.
> Huacai
>
> >
> > /* dst = dst + src */
> > case BPF_ALU | BPF_ADD | BPF_X:
> > + emit_insn(ctx, addw, dst, dst, src);
> > + break;
> > case BPF_ALU64 | BPF_ADD | BPF_X:
> > emit_insn(ctx, addd, dst, dst, src);
> > - emit_zext_32(ctx, dst, is32);
> > break;
> >
> > /* dst = dst + imm */
> > case BPF_ALU | BPF_ADD | BPF_K:
> > + if (is_signed_imm12(imm)) {
> > + emit_insn(ctx, addiw, dst, dst, imm);
> > + } else {
> > + move_imm(ctx, t1, (u32)imm);
> > + emit_insn(ctx, addw, dst, dst, t1);
> > + }
> > + break;
> > case BPF_ALU64 | BPF_ADD | BPF_K:
> > if (is_signed_imm12(imm)) {
> > emit_insn(ctx, addid, dst, dst, imm);
> > } else {
> > move_imm(ctx, t1, imm);
> > - emit_zext_32(ctx, t1, is32);
> > emit_insn(ctx, addd, dst, dst, t1);
> > }
> > - emit_zext_32(ctx, dst, is32);
> > break;
> >
> > /* dst = dst - src */
> > case BPF_ALU | BPF_SUB | BPF_X:
> > + emit_insn(ctx, subw, dst, dst, src);
> > + break;
> > case BPF_ALU64 | BPF_SUB | BPF_X:
> > emit_insn(ctx, subd, dst, dst, src);
> > - emit_zext_32(ctx, dst, is32);
> > break;
> >
> > /* dst = dst - imm */
> > case BPF_ALU | BPF_SUB | BPF_K:
> > + if (is_signed_imm12(-imm)) {
> > + emit_insn(ctx, addiw, dst, dst, -imm);
> > + } else {
> > + move_imm(ctx, t1, (u32)imm);
> > + emit_insn(ctx, subw, dst, dst, t1);
> > + }
> > + break;
> > case BPF_ALU64 | BPF_SUB | BPF_K:
> > if (is_signed_imm12(-imm)) {
> > emit_insn(ctx, addid, dst, dst, -imm);
> > } else {
> > move_imm(ctx, t1, imm);
> > - emit_zext_32(ctx, t1, is32);
> > emit_insn(ctx, subd, dst, dst, t1);
> > }
> > - emit_zext_32(ctx, dst, is32);
> > break;
> >
> > /* dst = dst * src */
> > case BPF_ALU | BPF_MUL | BPF_X:
> > + emit_insn(ctx, mulw, dst, dst, src);
> > + break;
> > case BPF_ALU64 | BPF_MUL | BPF_X:
> > emit_insn(ctx, muld, dst, dst, src);
> > - emit_zext_32(ctx, dst, is32);
> > break;
> >
> > /* dst = dst * imm */
> > case BPF_ALU | BPF_MUL | BPF_K:
> > + move_imm(ctx, t1, (u32)imm);
> > + emit_insn(ctx, mulw, dst, dst, t1);
> > + break;
> > case BPF_ALU64 | BPF_MUL | BPF_K:
> > move_imm(ctx, t1, imm);
> > - emit_zext_32(ctx, t1, is32);
> > emit_insn(ctx, muld, dst, dst, t1);
> > - emit_zext_32(ctx, dst, is32);
> > break;
> >
> > /* dst = dst / src */
> > case BPF_ALU | BPF_DIV | BPF_X:
> > - case BPF_ALU64 | BPF_DIV | BPF_X:
> > - if (!off) {
> > - emit_zext_32(ctx, dst, is32);
> > - move_reg(ctx, t1, src);
> > - emit_zext_32(ctx, t1, is32);
> > - emit_insn(ctx, divdu, dst, dst, t1);
> > - emit_zext_32(ctx, dst, is32);
> > - } else {
> > - emit_sext_32(ctx, dst, is32);
> > - move_reg(ctx, t1, src);
> > - emit_sext_32(ctx, t1, is32);
> > - emit_insn(ctx, divd, dst, dst, t1);
> > - emit_zext_32(ctx, dst, is32);
> > + if (cpu_has_div32) {
> > + if (!off)
> > + emit_insn(ctx, divwu, dst, dst, src);
> > + else
> > + emit_insn(ctx, divw, dst, dst, src);
> > + break;
> > }
> > + /*
> > + * Legacy fallback for !div32: src must be extended into a
> > + * temporary register to keep the original src register intact.
> > + */
> > + emit_insn(ctx, addw, dst, dst, LOONGARCH_GPR_ZERO);
> > + emit_insn(ctx, addw, t1, src, LOONGARCH_GPR_ZERO);
> > + if (!off)
> > + emit_insn(ctx, divwu, dst, dst, t1);
> > + else
> > + emit_insn(ctx, divw, dst, dst, t1);
> > + break;
> > + case BPF_ALU64 | BPF_DIV | BPF_X:
> > + if (!off)
> > + emit_insn(ctx, divdu, dst, dst, src);
> > + else
> > + emit_insn(ctx, divd, dst, dst, src);
> > break;
> >
> > /* dst = dst / imm */
> > case BPF_ALU | BPF_DIV | BPF_K:
> > + move_imm(ctx, t1, (u32)imm);
> > + if (!cpu_has_div32) {
> > + emit_insn(ctx, addw, dst, dst, LOONGARCH_GPR_ZERO);
> > + emit_insn(ctx, addw, t1, t1, LOONGARCH_GPR_ZERO);
> > + }
> > + if (!off)
> > + emit_insn(ctx, divwu, dst, dst, t1);
> > + else
> > + emit_insn(ctx, divw, dst, dst, t1);
> > + break;
> > case BPF_ALU64 | BPF_DIV | BPF_K:
> > - if (!off) {
> > - move_imm(ctx, t1, imm);
> > - emit_zext_32(ctx, t1, is32);
> > - emit_zext_32(ctx, dst, is32);
> > + move_imm(ctx, t1, imm);
> > + if (!off)
> > emit_insn(ctx, divdu, dst, dst, t1);
> > - emit_zext_32(ctx, dst, is32);
> > - } else {
> > - move_imm(ctx, t1, imm);
> > - emit_sext_32(ctx, t1, is32);
> > - emit_sext_32(ctx, dst, is32);
> > + else
> > emit_insn(ctx, divd, dst, dst, t1);
> > - emit_zext_32(ctx, dst, is32);
> > - }
> > break;
> >
> > /* dst = dst % src */
> > case BPF_ALU | BPF_MOD | BPF_X:
> > - case BPF_ALU64 | BPF_MOD | BPF_X:
> > - if (!off) {
> > - emit_zext_32(ctx, dst, is32);
> > - move_reg(ctx, t1, src);
> > - emit_zext_32(ctx, t1, is32);
> > - emit_insn(ctx, moddu, dst, dst, t1);
> > - emit_zext_32(ctx, dst, is32);
> > - } else {
> > - emit_sext_32(ctx, dst, is32);
> > - move_reg(ctx, t1, src);
> > - emit_sext_32(ctx, t1, is32);
> > - emit_insn(ctx, modd, dst, dst, t1);
> > - emit_zext_32(ctx, dst, is32);
> > + if (cpu_has_div32) {
> > + if (!off)
> > + emit_insn(ctx, modwu, dst, dst, src);
> > + else
> > + emit_insn(ctx, modw, dst, dst, src);
> > + break;
> > }
> > + /*
> > + * Legacy fallback for !div32: src must be extended into a
> > + * temporary register to keep the original src register intact.
> > + */
> > + emit_insn(ctx, addw, dst, dst, LOONGARCH_GPR_ZERO);
> > + emit_insn(ctx, addw, t1, src, LOONGARCH_GPR_ZERO);
> > + if (!off)
> > + emit_insn(ctx, modwu, dst, dst, t1);
> > + else
> > + emit_insn(ctx, modw, dst, dst, t1);
> > + break;
> > + case BPF_ALU64 | BPF_MOD | BPF_X:
> > + if (!off)
> > + emit_insn(ctx, moddu, dst, dst, src);
> > + else
> > + emit_insn(ctx, modd, dst, dst, src);
> > break;
> >
> > /* dst = dst % imm */
> > case BPF_ALU | BPF_MOD | BPF_K:
> > + move_imm(ctx, t1, (u32)imm);
> > + if (!cpu_has_div32) {
> > + emit_insn(ctx, addw, dst, dst, LOONGARCH_GPR_ZERO);
> > + emit_insn(ctx, addw, t1, t1, LOONGARCH_GPR_ZERO);
> > + }
> > + if (!off)
> > + emit_insn(ctx, modwu, dst, dst, t1);
> > + else
> > + emit_insn(ctx, modw, dst, dst, t1);
> > + break;
> > case BPF_ALU64 | BPF_MOD | BPF_K:
> > - if (!off) {
> > - move_imm(ctx, t1, imm);
> > - emit_zext_32(ctx, t1, is32);
> > - emit_zext_32(ctx, dst, is32);
> > + move_imm(ctx, t1, imm);
> > + if (!off)
> > emit_insn(ctx, moddu, dst, dst, t1);
> > - emit_zext_32(ctx, dst, is32);
> > - } else {
> > - move_imm(ctx, t1, imm);
> > - emit_sext_32(ctx, t1, is32);
> > - emit_sext_32(ctx, dst, is32);
> > + else
> > emit_insn(ctx, modd, dst, dst, t1);
> > - emit_zext_32(ctx, dst, is32);
> > - }
> > break;
> >
> > /* dst = -dst */
> > case BPF_ALU | BPF_NEG:
> > + emit_insn(ctx, subw, dst, LOONGARCH_GPR_ZERO, dst);
> > + break;
> > case BPF_ALU64 | BPF_NEG:
> > emit_insn(ctx, subd, dst, LOONGARCH_GPR_ZERO, dst);
> > - emit_zext_32(ctx, dst, is32);
> > break;
> >
> > /* dst = dst & src */
> > case BPF_ALU | BPF_AND | BPF_X:
> > case BPF_ALU64 | BPF_AND | BPF_X:
> > emit_insn(ctx, and, dst, dst, src);
> > - emit_zext_32(ctx, dst, is32);
> > break;
> >
> > /* dst = dst & imm */
> > @@ -905,18 +931,18 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> > if (is_unsigned_imm12(imm)) {
> > emit_insn(ctx, andi, dst, dst, imm);
> > } else {
> > - move_imm(ctx, t1, imm);
> > - emit_zext_32(ctx, t1, is32);
> > + if (is32)
> > + move_imm(ctx, t1, (u32)imm);
> > + else
> > + move_imm(ctx, t1, imm);
> > emit_insn(ctx, and, dst, dst, t1);
> > }
> > - emit_zext_32(ctx, dst, is32);
> > break;
> >
> > /* dst = dst | src */
> > case BPF_ALU | BPF_OR | BPF_X:
> > case BPF_ALU64 | BPF_OR | BPF_X:
> > emit_insn(ctx, or, dst, dst, src);
> > - emit_zext_32(ctx, dst, is32);
> > break;
> >
> > /* dst = dst | imm */
> > @@ -925,18 +951,18 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> > if (is_unsigned_imm12(imm)) {
> > emit_insn(ctx, ori, dst, dst, imm);
> > } else {
> > - move_imm(ctx, t1, imm);
> > - emit_zext_32(ctx, t1, is32);
> > + if (is32)
> > + move_imm(ctx, t1, (u32)imm);
> > + else
> > + move_imm(ctx, t1, imm);
> > emit_insn(ctx, or, dst, dst, t1);
> > }
> > - emit_zext_32(ctx, dst, is32);
> > break;
> >
> > /* dst = dst ^ src */
> > case BPF_ALU | BPF_XOR | BPF_X:
> > case BPF_ALU64 | BPF_XOR | BPF_X:
> > emit_insn(ctx, xor, dst, dst, src);
> > - emit_zext_32(ctx, dst, is32);
> > break;
> >
> > /* dst = dst ^ imm */
> > @@ -945,19 +971,18 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> > if (is_unsigned_imm12(imm)) {
> > emit_insn(ctx, xori, dst, dst, imm);
> > } else {
> > - move_imm(ctx, t1, imm);
> > - emit_zext_32(ctx, t1, is32);
> > + if (is32)
> > + move_imm(ctx, t1, (u32)imm);
> > + else
> > + move_imm(ctx, t1, imm);
> > emit_insn(ctx, xor, dst, dst, t1);
> > }
> > - emit_zext_32(ctx, dst, is32);
> > break;
> >
> > /* dst = dst << src (logical) */
> > case BPF_ALU | BPF_LSH | BPF_X:
> > emit_insn(ctx, sllw, dst, dst, src);
> > - emit_zext_32(ctx, dst, is32);
> > break;
> > -
> > case BPF_ALU64 | BPF_LSH | BPF_X:
> > emit_insn(ctx, slld, dst, dst, src);
> > break;
> > @@ -965,9 +990,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> > /* dst = dst << imm (logical) */
> > case BPF_ALU | BPF_LSH | BPF_K:
> > emit_insn(ctx, slliw, dst, dst, imm);
> > - emit_zext_32(ctx, dst, is32);
> > break;
> > -
> > case BPF_ALU64 | BPF_LSH | BPF_K:
> > emit_insn(ctx, sllid, dst, dst, imm);
> > break;
> > @@ -975,9 +998,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> > /* dst = dst >> src (logical) */
> > case BPF_ALU | BPF_RSH | BPF_X:
> > emit_insn(ctx, srlw, dst, dst, src);
> > - emit_zext_32(ctx, dst, is32);
> > break;
> > -
> > case BPF_ALU64 | BPF_RSH | BPF_X:
> > emit_insn(ctx, srld, dst, dst, src);
> > break;
> > @@ -985,9 +1006,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> > /* dst = dst >> imm (logical) */
> > case BPF_ALU | BPF_RSH | BPF_K:
> > emit_insn(ctx, srliw, dst, dst, imm);
> > - emit_zext_32(ctx, dst, is32);
> > break;
> > -
> > case BPF_ALU64 | BPF_RSH | BPF_K:
> > emit_insn(ctx, srlid, dst, dst, imm);
> > break;
> > @@ -995,9 +1014,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> > /* dst = dst >> src (arithmetic) */
> > case BPF_ALU | BPF_ARSH | BPF_X:
> > emit_insn(ctx, sraw, dst, dst, src);
> > - emit_zext_32(ctx, dst, is32);
> > break;
> > -
> > case BPF_ALU64 | BPF_ARSH | BPF_X:
> > emit_insn(ctx, srad, dst, dst, src);
> > break;
> > @@ -1005,9 +1022,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> > /* dst = dst >> imm (arithmetic) */
> > case BPF_ALU | BPF_ARSH | BPF_K:
> > emit_insn(ctx, sraiw, dst, dst, imm);
> > - emit_zext_32(ctx, dst, is32);
> > break;
> > -
> > case BPF_ALU64 | BPF_ARSH | BPF_K:
> > emit_insn(ctx, sraid, dst, dst, imm);
> > break;
> > @@ -1016,12 +1031,10 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> > case BPF_ALU | BPF_END | BPF_FROM_LE:
> > switch (imm) {
> > case 16:
> > - /* zero-extend 16 bits into 64 bits */
> > emit_insn(ctx, bstrpickd, dst, dst, 15, 0);
> > break;
> > case 32:
> > - /* zero-extend 32 bits into 64 bits */
> > - emit_zext_32(ctx, dst, is32);
> > + emit_insn(ctx, bstrpickd, dst, dst, 31, 0);
> > break;
> > case 64:
> > /* do nothing */
> > @@ -1034,13 +1047,11 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> > switch (imm) {
> > case 16:
> > emit_insn(ctx, revb2h, dst, dst);
> > - /* zero-extend 16 bits into 64 bits */
> > emit_insn(ctx, bstrpickd, dst, dst, 15, 0);
> > break;
> > case 32:
> > emit_insn(ctx, revb2w, dst, dst);
> > - /* clear the upper 32 bits */
> > - emit_zext_32(ctx, dst, true);
> > + emit_insn(ctx, bstrpickd, dst, dst, 31, 0);
> > break;
> > case 64:
> > emit_insn(ctx, revbd, dst, dst);
> > @@ -1048,7 +1059,7 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> > }
> > break;
> >
> > - /* PC += off if dst cond src */
> > + /* PC += off if dst cond src (64-bit) */
> > case BPF_JMP | BPF_JEQ | BPF_X:
> > case BPF_JMP | BPF_JNE | BPF_X:
> > case BPF_JMP | BPF_JGT | BPF_X:
> > @@ -1059,31 +1070,38 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> > case BPF_JMP | BPF_JSGE | BPF_X:
> > case BPF_JMP | BPF_JSLT | BPF_X:
> > case BPF_JMP | BPF_JSLE | BPF_X:
> > + jmp_offset = bpf2la_offset(i, off, ctx);
> > + if (emit_cond_jmp(ctx, cond, dst, src, jmp_offset) < 0)
> > + goto toofar;
> > + break;
> > +
> > + /* PC += off if dst cond src (32-bit unsigned) */
> > case BPF_JMP32 | BPF_JEQ | BPF_X:
> > case BPF_JMP32 | BPF_JNE | BPF_X:
> > case BPF_JMP32 | BPF_JGT | BPF_X:
> > case BPF_JMP32 | BPF_JGE | BPF_X:
> > case BPF_JMP32 | BPF_JLT | BPF_X:
> > case BPF_JMP32 | BPF_JLE | BPF_X:
> > + jmp_offset = bpf2la_offset(i, off, ctx);
> > + emit_insn(ctx, bstrpickd, t1, dst, 31, 0);
> > + emit_insn(ctx, bstrpickd, t2, src, 31, 0);
> > + if (emit_cond_jmp(ctx, cond, t1, t2, jmp_offset) < 0)
> > + goto toofar;
> > + break;
> > +
> > + /* PC += off if dst cond src (32-bit signed) */
> > case BPF_JMP32 | BPF_JSGT | BPF_X:
> > case BPF_JMP32 | BPF_JSGE | BPF_X:
> > case BPF_JMP32 | BPF_JSLT | BPF_X:
> > case BPF_JMP32 | BPF_JSLE | BPF_X:
> > jmp_offset = bpf2la_offset(i, off, ctx);
> > - move_reg(ctx, t1, dst);
> > - move_reg(ctx, t2, src);
> > - if (is_signed_bpf_cond(BPF_OP(code))) {
> > - emit_sext_32(ctx, t1, is32);
> > - emit_sext_32(ctx, t2, is32);
> > - } else {
> > - emit_zext_32(ctx, t1, is32);
> > - emit_zext_32(ctx, t2, is32);
> > - }
> > + emit_insn(ctx, addw, t1, dst, LOONGARCH_GPR_ZERO);
> > + emit_insn(ctx, addw, t2, src, LOONGARCH_GPR_ZERO);
> > if (emit_cond_jmp(ctx, cond, t1, t2, jmp_offset) < 0)
> > goto toofar;
> > break;
> >
> > - /* PC += off if dst cond imm */
> > + /* PC += off if dst cond imm (64-bit) */
> > case BPF_JMP | BPF_JEQ | BPF_K:
> > case BPF_JMP | BPF_JNE | BPF_K:
> > case BPF_JMP | BPF_JGT | BPF_K:
> > @@ -1094,12 +1112,38 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> > case BPF_JMP | BPF_JSGE | BPF_K:
> > case BPF_JMP | BPF_JSLT | BPF_K:
> > case BPF_JMP | BPF_JSLE | BPF_K:
> > + jmp_offset = bpf2la_offset(i, off, ctx);
> > + if (imm) {
> > + move_imm(ctx, t1, imm);
> > + tm = t1;
> > + } else {
> > + tm = LOONGARCH_GPR_ZERO;
> > + }
> > + if (emit_cond_jmp(ctx, cond, dst, tm, jmp_offset) < 0)
> > + goto toofar;
> > + break;
> > +
> > + /* PC += off if dst cond imm (32-bit unsigned) */
> > case BPF_JMP32 | BPF_JEQ | BPF_K:
> > case BPF_JMP32 | BPF_JNE | BPF_K:
> > case BPF_JMP32 | BPF_JGT | BPF_K:
> > case BPF_JMP32 | BPF_JGE | BPF_K:
> > case BPF_JMP32 | BPF_JLT | BPF_K:
> > case BPF_JMP32 | BPF_JLE | BPF_K:
> > + jmp_offset = bpf2la_offset(i, off, ctx);
> > + if (imm) {
> > + move_imm(ctx, t1, imm);
> > + emit_insn(ctx, bstrpickd, t1, t1, 31, 0);
> > + tm = t1;
> > + } else {
> > + tm = LOONGARCH_GPR_ZERO;
> > + }
> > + emit_insn(ctx, bstrpickd, t2, dst, 31, 0);
> > + if (emit_cond_jmp(ctx, cond, t2, tm, jmp_offset) < 0)
> > + goto toofar;
> > + break;
> > +
> > + /* PC += off if dst cond imm (32-bit signed) */
> > case BPF_JMP32 | BPF_JSGT | BPF_K:
> > case BPF_JMP32 | BPF_JSGE | BPF_K:
> > case BPF_JMP32 | BPF_JSLT | BPF_K:
> > @@ -1107,19 +1151,12 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> > jmp_offset = bpf2la_offset(i, off, ctx);
> > if (imm) {
> > move_imm(ctx, t1, imm);
> > + emit_insn(ctx, addw, t1, t1, LOONGARCH_GPR_ZERO);
> > tm = t1;
> > } else {
> > - /* If imm is 0, simply use zero register. */
> > tm = LOONGARCH_GPR_ZERO;
> > }
> > - move_reg(ctx, t2, dst);
> > - if (is_signed_bpf_cond(BPF_OP(code))) {
> > - emit_sext_32(ctx, tm, is32);
> > - emit_sext_32(ctx, t2, is32);
> > - } else {
> > - emit_zext_32(ctx, tm, is32);
> > - emit_zext_32(ctx, t2, is32);
> > - }
> > + emit_insn(ctx, addw, t2, dst, LOONGARCH_GPR_ZERO);
> > if (emit_cond_jmp(ctx, cond, t2, tm, jmp_offset) < 0)
> > goto toofar;
> > break;
> > @@ -1129,7 +1166,8 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> > case BPF_JMP32 | BPF_JSET | BPF_X:
> > jmp_offset = bpf2la_offset(i, off, ctx);
> > emit_insn(ctx, and, t1, dst, src);
> > - emit_zext_32(ctx, t1, is32);
> > + if (BPF_CLASS(code) == BPF_JMP32)
> > + emit_insn(ctx, bstrpickd, t1, t1, 31, 0);
> > if (emit_cond_jmp(ctx, cond, t1, LOONGARCH_GPR_ZERO, jmp_offset) < 0)
> > goto toofar;
> > break;
> > @@ -1139,9 +1177,9 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> > case BPF_JMP32 | BPF_JSET | BPF_K:
> > jmp_offset = bpf2la_offset(i, off, ctx);
> > move_imm(ctx, t1, imm);
> > - emit_zext_32(ctx, t1, is32);
> > emit_insn(ctx, and, t1, dst, t1);
> > - emit_zext_32(ctx, t1, is32);
> > + if (BPF_CLASS(code) == BPF_JMP32)
> > + emit_insn(ctx, bstrpickd, t1, t1, 31, 0);
> > if (emit_cond_jmp(ctx, cond, t1, LOONGARCH_GPR_ZERO, jmp_offset) < 0)
> > goto toofar;
> > break;
> > @@ -1468,6 +1506,9 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
> > return -EINVAL;
> > }
> >
> > + if (is32)
> > + emit_insn(ctx, bstrpickd, dst, dst, 31, 0);
> > +
> > return 0;
> >
> > toofar:
> > diff --git a/arch/loongarch/net/bpf_jit.h b/arch/loongarch/net/bpf_jit.h
> > index 57a98f8a055b..2bd0dc5b911c 100644
> > --- a/arch/loongarch/net/bpf_jit.h
> > +++ b/arch/loongarch/net/bpf_jit.h
> > @@ -72,24 +72,6 @@ static inline int epilogue_offset(const struct jit_ctx *ctx)
> > return (to - from);
> > }
> >
> > -/* Zero-extend 32 bits into 64 bits */
> > -static inline void emit_zext_32(struct jit_ctx *ctx, enum loongarch_gpr reg, bool is32)
> > -{
> > - if (!is32)
> > - return;
> > -
> > - emit_insn(ctx, lu32id, reg, 0);
> > -}
> > -
> > -/* Signed-extend 32 bits into 64 bits */
> > -static inline void emit_sext_32(struct jit_ctx *ctx, enum loongarch_gpr reg, bool is32)
> > -{
> > - if (!is32)
> > - return;
> > -
> > - emit_insn(ctx, addiw, reg, reg, 0);
> > -}
> > -
> > /* Emit proper extension according to ABI requirements.
> > * Note that it requires a value of size `size` already resides in register `reg`.
> > */
> > --
> > 2.42.0
> >
^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2026-10-28 6:55 UTC | newest]
Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-10-16 8:10 [PATCH v3 0/3] LoongArch: BPF: Refine move_imm() and optimize 32-bit operations Tiezhu Yang
2026-10-16 8:10 ` [PATCH v3 1/3] LoongArch: Add CPUCFG2 DIV32 feature detection Tiezhu Yang
2026-10-20 10:48 ` Huacai Chen
2026-10-16 8:10 ` [PATCH v3 2/3] LoongArch: BPF: Clean up and decouple is32 parameter from move_imm() Tiezhu Yang
2026-10-27 1:36 ` Hengqi Chen
2026-10-16 8:10 ` [PATCH v3 3/3] LoongArch: BPF: Optimize 32-bit operations using native instructions Tiezhu Yang
2026-10-16 15:18 ` Alexei Starovoitov
2026-10-17 10:58 ` Tiezhu Yang
2026-10-27 2:02 ` Hengqi Chen
2026-10-27 4:10 ` Tiezhu Yang
2026-10-27 8:13 ` Hengqi Chen
2026-10-28 4:19 ` Huacai Chen
2026-10-28 6:55 ` Hengqi Chen
2026-10-22 7:43 ` [PATCH v3 0/3] LoongArch: BPF: Refine move_imm() and optimize 32-bit operations Tiezhu Yang
2026-10-23 8:42 ` Huacai Chen
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®