[PATCH v3 0/9] KVM: SVM: Enable FRED support

[PATCH v3 0/9] KVM: SVM: Enable FRED support

[Shivansh Dhiman]

This series adds SVM support for FRED (Flexible Return and Event Delivery)
virtualization in KVM.

Background
----------
FRED introduces simplified privilege level transitions to replace IDT-based
event delivery and IRET returns, providing lower latency event handling while
ensuring complete supervisor context on delivery and full user context on
return. FRED defines event delivery for both ring 3->0 and ring 0->0
transitions, and introduces ERETU for returning to ring 3 and ERETS for
remaining in ring 0.

AMD hardware extends the VMCB to support FRED virtualization [1] with
dedicated save area fields for FRED MSRs (RSP1-3, SSP1-3, STKLVLS, CONFIG;
RSP0 only in the SEV-ES VMSA) and control fields for event injection data
(EXITINTDATA, EVENTINJDATA).

The implementation spans nine patches. The important changes are:

1) Extend VMCB structures with the FRED fields mentioned above, disable MSR
   interception for FRED-enabled guests to avoid unnecessary VM exits, and
   context switch FRED_RSP0, which hardware doesn't save/restore for
   non-SEV-ES guests.

2) Support for nested exceptions, where we populate event injection data
   when delivering exceptions like page faults and debug traps.

3) Let userspace save/restore the FRED MSRs, and don't intercept ICEBP
   while the guest has FRED enabled.

Major changes in v3
-------------------
 * Rebased on v10 of the FRED VMX series [2].
 * New patch to save/restore the FRED MSRs for live migration.
 * New patch to disable interception of ICEBP when the guest enables FRED.
 * Update the FRED fields and their offsets in the VMCB to match the latest
   APM.
 * Clear FRED from the KVM capabilities when vNMI is unavailable.

The full changelog is at the end of this letter.

Dependencies
------------
This series applies on top of:
 * v10 of the FRED VMX series [2], which adds the common x86 FRED support
   (CR4.FRED, the FRED MSRs, exception event data and nested exception
   tracking).
 * "KVM: SVM: Clear VMCB save area instead of entire VMCB on shutdown
   intercept" [3]. It keeps FRED_VIRT_ENABLE set across an intercepted
   shutdown.

Testing
-------
Testing was done on a Zen 6 machine in the following scenarios:
 * Booting FRED-enabled guests and checking that FRED is exposed
 * Running SEV, SEV-ES and SEV-SNP guests with FRED enabled
 * Live migration of a guest with FRED enabled
 * KVM selftests

The msrs_test modified in the VMX series is passing, however, fred_test
doesn't pass on SVM yet. A fix, if needed, will be posted in upcoming
versions.

Opens
-----
 * INT1 event type: As I understand it, SVM intercepts ICEBP only because
   an ICEBP-induced #DB delivered through a task gate otherwise ends up
   with the wrong RIP [4]. FRED has no task gates, so patch 9 drops the
   intercept for FRED-enabled guests. The CPU then delivers INT1 itself,
   and the guest sees the correct event type 5. David, does this look
   like a reasonable approach to you?

   However, when KVM itself injects an INT1, the guest still sees event
   type 3 instead of 5, even though the hardware supports it. I think a
   generic fix in KVM may be required. Any thoughts would be appreciated.

 * Nested FRED: FRED for L2 under nested SVM isn't part of this series
   and will be posted in upcoming versions.

Links
-----
[1]: https://docs.amd.com/v/u/en-US/69191-PUB
[2]: https://lore.kernel.org/all/20260911213659.2025974-1-sohil.mehta@intel.com/
[3]: https://lore.kernel.org/all/20260824131824.6040-1-shivansh.dhiman@amd.com/
[4]: https://lore.kernel.org/kvm/e03f092dfbb7d391a6bf2797ba01e122ba080bcd.camel@infradead.org/

Previous versions
-----------------
v2: https://lore.kernel.org/all/20260402184240.1939480-1-shivansh.dhiman@amd.com/
v1: https://lore.kernel.org/kvm/20260129063653.3553076-1-shivansh.dhiman@amd.com/

Regards,
Shivansh

---
Changelog:

v2 -> v3:
 * Rebased on v10 of the FRED VMX series.
 * New patch to save/restore FRED MSRs via KVM_{GET,SET}_MSRS (patch 4).
 * New patch to stop intercepting ICEBP for FRED guests (patch 9).
 * Keep the guest's FRED_RSP0 in vcpu_svm for non-SEV-ES guests.
 * Updated the FRED fields and offsets in the VMCB per the latest APM.
 * Clear the FRED capability when vNMI is unavailable (Andrew Cooper).
 * Renamed FRED_VIRT_ENABLE_MASK to SVM_MISC2_ENABLE_V_FRED.
 * Use ex->is_nested to check for nested exceptions.
 * Updated the FRED prints in dump_vmcb().
 * Updated the commit messages.

v1 -> v2:
 * Modified the zeroing of FRED MSRs from INIT to RESET and updated the
   commit message (Sean Christopherson).
 * Moved FRED MSRs save/restore logic from svm_vcpu_enter_exit() to
   svm_prepare_[host_switch/switch_to_guest]() to reduce some MSR
   accesses.
 * Confined the enabling of vFRED to svm_vcpu_after_set_cpuid() only
   (Sean Christopherson).
 * Removed the need for a new function parameter 'reinject_on_vmexit' in
   svm_complete_interrupts() (Paolo Bonzini).
 * Used guest_cpu_cap_has() instead of checking VMCB bit in recalc
   intercepts (Sean Christopherson).
 * Gate all the FRED MSRs through guest_cpu_cap_has() while dumping VMCB
   (Sean Christopherson).
 * While switching to host, added a safety check on guest_state_loaded.
 * Variable rename from 'fred_enable' to 'intercept' (Sean Christopherson).
 * Replace variable 'nested' with 'is_nested' (Sean Christopherson).
 * Formatted variables in reverse fir-tree order (Sean Christopherson).

---
Neeraj Upadhyay (2):
  KVM: SVM: Populate FRED event data on event injection
  KVM: SVM: Support FRED nested exception injection

Shivansh Dhiman (7):
  KVM: SVM: Initialize FRED VMCB fields
  KVM: SVM: Disable interception of FRED MSRs for FRED supported guests
  KVM: SVM: Save/restore FRED_RSP0 for FRED supported guests
  KVM: SVM: Add support for saving and restoring FRED MSRs
  KVM: SVM: Dump FRED context in dump_vmcb()
  KVM: SVM: Enable FRED virtualization
  KVM: SVM: Don't intercept ICEBP while the guest has FRED enabled

 arch/x86/include/asm/svm.h |  33 ++++++-
 arch/x86/kvm/svm/svm.c     | 180 +++++++++++++++++++++++++++++++++++--
 arch/x86/kvm/svm/svm.h     |   2 +
 3 files changed, 207 insertions(+), 8 deletions(-)

-- 
2.43.0

[PATCH v3 1/9] KVM: SVM: Initialize FRED VMCB fields

[Shivansh Dhiman]

The AMD FRED (Flexible Return and Event Delivery) feature introduces
several new fields to the VMCB save area. These fields include
FRED-specific stack pointers (fred_rsp[1-3], fred_ssp[1-3]), stack level
tracking (fred_stklvls), and configuration (fred_config), and the
control-area fields exit_int_data and event_inj_data. Note that
fred_rsp0 is only saved/restored for SEV-ES guests.

All FRED MSRs are zeroed on RESET. Reproduce the HW behavior here to
ensure that vCPU starts with a valid FRED state. Also update the size
of save areas of VMCB.

Co-developed-by: Neeraj Upadhyay <Neeraj.Upadhyay@amd.com>
Signed-off-by: Neeraj Upadhyay <Neeraj.Upadhyay@amd.com>
Signed-off-by: Shivansh Dhiman <shivansh.dhiman@amd.com>
Reviewed-by: Nikunj A Dadhania <nikunj@amd.com>
---

Changes in v3:
* Update the fields and offsets of FRED MSRs according to the latest APM.
* Dropped fred_rsp0 from vmcb_save_area. Hardware doesn't save/restore
  it for non-SEV-ES guests. Move its handling to patch 3.
* Updated the commit message.

Changes in v2:
* Modified the zeroing of FRED MSRs from INIT to RESET and updated the commit
  message (Sean Christopherson).

---
 arch/x86/include/asm/svm.h | 31 ++++++++++++++++++++++++++++---
 arch/x86/kvm/svm/svm.c     | 11 +++++++++++
 2 files changed, 39 insertions(+), 3 deletions(-)

diff --git a/arch/x86/include/asm/svm.h b/arch/x86/include/asm/svm.h
index aa63431ba92c..0570581e251a 100644
--- a/arch/x86/include/asm/svm.h
+++ b/arch/x86/include/asm/svm.h
@@ -165,7 +165,10 @@ struct __attribute__ ((__packed__)) vmcb_control_area {
 	u8 reserved_9[22];
 	u64 allowed_sev_features;	/* Offset 0x138 */
 	u64 guest_sev_features;		/* Offset 0x140 */
-	u8 reserved_10[664];
+	u8 reserved_10[40];
+	u64 exit_int_data;		/* Offset 0x170 */
+	u64 event_inj_data;
+	u8 reserved_11[608];
 	/*
 	 * Offset 0x3e0, 32 bytes reserved
 	 * for use by hypervisor/software.
@@ -370,6 +373,15 @@ struct vmcb_save_area {
 	u64 last_excp_to;
 	u8 reserved_0x298[72];
 	u64 spec_ctrl;		/* Guest version of SPEC_CTRL at 0x2E0 */
+	u8 reserved_0x2e8[1496];
+	u64 fred_rsp1;
+	u64 fred_rsp2;
+	u64 fred_rsp3;
+	u64 fred_stklvls;
+	u64 fred_ssp1;
+	u64 fred_ssp2;
+	u64 fred_ssp3;
+	u64 fred_config;
 } __packed;
 
 /* Save area definition for SEV-ES and SEV-SNP guests */
@@ -482,6 +494,18 @@ struct sev_es_save_area {
 	u8 fpreg_x87[80];
 	u8 fpreg_xmm[256];
 	u8 fpreg_ymm[256];
+	u8 reserved_0x670[568];
+	u64 guest_exit_int_data;
+	u64 guest_event_inj_data;
+	u64 fred_rsp0;
+	u64 fred_rsp1;
+	u64 fred_rsp2;
+	u64 fred_rsp3;
+	u64 fred_stklvls;
+	u64 fred_ssp1;
+	u64 fred_ssp2;
+	u64 fred_ssp3;
+	u64 fred_config;
 } __packed;
 
 struct ghcb_save_area {
@@ -552,9 +576,9 @@ struct vmcb {
 	};
 } __packed;
 
-#define EXPECTED_VMCB_SAVE_AREA_SIZE		744
+#define EXPECTED_VMCB_SAVE_AREA_SIZE		2304
 #define EXPECTED_GHCB_SAVE_AREA_SIZE		1032
-#define EXPECTED_SEV_ES_SAVE_AREA_SIZE		1648
+#define EXPECTED_SEV_ES_SAVE_AREA_SIZE		2304
 #define EXPECTED_VMCB_CONTROL_AREA_SIZE		1024
 #define EXPECTED_GHCB_SIZE			PAGE_SIZE
 
@@ -578,6 +602,7 @@ static inline void __unused_size_checks(void)
 	BUILD_BUG_RESERVED_OFFSET(vmcb_save_area, 0x180);
 	BUILD_BUG_RESERVED_OFFSET(vmcb_save_area, 0x248);
 	BUILD_BUG_RESERVED_OFFSET(vmcb_save_area, 0x298);
+	BUILD_BUG_RESERVED_OFFSET(vmcb_save_area, 0x2e8);
 
 	BUILD_BUG_RESERVED_OFFSET(sev_es_save_area, 0xc8);
 	BUILD_BUG_RESERVED_OFFSET(sev_es_save_area, 0xcc);
diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index dd19c7b4e904..6d55b0a576d9 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -1217,6 +1217,17 @@ static void init_vmcb(struct kvm_vcpu *vcpu, bool init_event)
 	save->idtr.base = 0;
 	save->idtr.limit = 0xffff;
 
+	if (!init_event) {
+		save->fred_rsp1 = 0;
+		save->fred_rsp2 = 0;
+		save->fred_rsp3 = 0;
+		save->fred_stklvls = 0;
+		save->fred_ssp1 = 0;
+		save->fred_ssp2 = 0;
+		save->fred_ssp3 = 0;
+		save->fred_config = 0;
+	}
+
 	init_sys_seg(&save->ldtr, SEG_TYPE_LDT);
 	init_sys_seg(&save->tr, SEG_TYPE_BUSY_TSS16);
 
-- 
2.43.0

[PATCH v3 2/9] KVM: SVM: Disable interception of FRED MSRs for FRED supported guests

[Shivansh Dhiman]

The FRED (Flexible Return and Event Delivery) feature introduces a new set
of MSRs for managing its state, such as MSR_IA32_FRED_CONFIG and the
various stack pointer MSRs.

For a guest that has FRED enabled via its CPUID bits, the guest OS
expects to be able to directly read and write these MSRs. Intercepting
these accesses would cause unnecessary VM-Exits and performance overhead.
In addition, the state of the MSRs at any point should always correspond
to the context (host or guest) which is running. Otherwise, the event
delivery could refer to wrong MSR values.

Co-developed-by: Neeraj Upadhyay <Neeraj.Upadhyay@amd.com>
Signed-off-by: Neeraj Upadhyay <Neeraj.Upadhyay@amd.com>
Signed-off-by: Shivansh Dhiman <shivansh.dhiman@amd.com>
---

Changes in v3:
* Rebased, no functional changes.

Changes in v2:
* Used guest_cpu_cap_has() instead of checking VMCB bit (Sean Christopherson).
* Variable rename from 'fred_enable' to 'intercept' (Sean Christopherson).

---
 arch/x86/kvm/svm/svm.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index 6d55b0a576d9..e212f53d262a 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -788,6 +788,21 @@ static void svm_recalc_pmu_msr_intercepts(struct kvm_vcpu *vcpu)
 				  MSR_TYPE_RW, intercept);
 }
 
+static void svm_recalc_fred_msr_intercepts(struct kvm_vcpu *vcpu)
+{
+	bool intercept = guest_cpu_cap_has(vcpu, X86_FEATURE_FRED);
+
+	svm_set_intercept_for_msr(vcpu, MSR_IA32_FRED_RSP0, MSR_TYPE_RW, !intercept);
+	svm_set_intercept_for_msr(vcpu, MSR_IA32_FRED_RSP1, MSR_TYPE_RW, !intercept);
+	svm_set_intercept_for_msr(vcpu, MSR_IA32_FRED_RSP2, MSR_TYPE_RW, !intercept);
+	svm_set_intercept_for_msr(vcpu, MSR_IA32_FRED_RSP3, MSR_TYPE_RW, !intercept);
+	svm_set_intercept_for_msr(vcpu, MSR_IA32_FRED_STKLVLS, MSR_TYPE_RW, !intercept);
+	svm_set_intercept_for_msr(vcpu, MSR_IA32_FRED_SSP1, MSR_TYPE_RW, !intercept);
+	svm_set_intercept_for_msr(vcpu, MSR_IA32_FRED_SSP2, MSR_TYPE_RW, !intercept);
+	svm_set_intercept_for_msr(vcpu, MSR_IA32_FRED_SSP3, MSR_TYPE_RW, !intercept);
+	svm_set_intercept_for_msr(vcpu, MSR_IA32_FRED_CONFIG, MSR_TYPE_RW, !intercept);
+}
+
 static void svm_recalc_msr_intercepts(struct kvm_vcpu *vcpu)
 {
 	struct vcpu_svm *svm = to_svm(vcpu);
@@ -857,6 +872,7 @@ static void svm_recalc_msr_intercepts(struct kvm_vcpu *vcpu)
 		sev_es_recalc_msr_intercepts(vcpu);
 
 	svm_recalc_pmu_msr_intercepts(vcpu);
+	svm_recalc_fred_msr_intercepts(vcpu);
 
 	/*
 	 * x2APIC intercepts are modified on-demand and cannot be filtered by
-- 
2.43.0

[PATCH v3 3/9] KVM: SVM: Save/restore FRED_RSP0 for FRED supported guests

[Shivansh Dhiman]

Hardware does not save/restore FRED_RSP0 for Non-SEV-ES guests. Save it
in svm_prepare_host_switch() and restore it in svm_prepare_switch_to_guest()
so that the correct physical CPU state is updated.

Also, synchronize the current value of MSR_IA32_FRED_RSP0 in hardware to the
kernel's local cache. Note that the desired host's RSP0 will be set when the
CPU exits to userspace for servicing vCPU tasks.

Co-developed-by: Neeraj Upadhyay <Neeraj.Upadhyay@amd.com>
Signed-off-by: Neeraj Upadhyay <Neeraj.Upadhyay@amd.com>
Signed-off-by: Shivansh Dhiman <shivansh.dhiman@amd.com>
---

Changes in v3:
* Keep the guest's FRED_RSP0 in a new vcpu_svm field, msr_guest_fred_rsp0.
  The VMCB has no FRED_RSP0 field for non-SEV-ES guests.
* Zero msr_guest_fred_rsp0 on RESET.
* Added the svm_manages_fred_rsp0() helper.

Changes in v2:
* Moved FRED MSRs save/restore logic from svm_vcpu_enter_exit() to
  svm_prepare_[host_switch/switch_to_guest]() to reduce some MSR accesses.
* While switching to host, added a safety check on guest_state_loaded.

---
 arch/x86/kvm/svm/svm.c | 28 +++++++++++++++++++++++++++-
 arch/x86/kvm/svm/svm.h |  2 ++
 2 files changed, 29 insertions(+), 1 deletion(-)

diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index e212f53d262a..c1263a3b3e73 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -1234,6 +1234,7 @@ static void init_vmcb(struct kvm_vcpu *vcpu, bool init_event)
 	save->idtr.limit = 0xffff;
 
 	if (!init_event) {
+		svm->msr_guest_fred_rsp0 = 0;
 		save->fred_rsp1 = 0;
 		save->fred_rsp2 = 0;
 		save->fred_rsp3 = 0;
@@ -1458,6 +1459,11 @@ static void svm_srso_vm_init(void) { }
 static void svm_srso_vm_destroy(void) { }
 #endif
 
+static bool svm_manages_fred_rsp0(struct kvm_vcpu *vcpu)
+{
+	return !is_sev_es_guest(vcpu) && guest_cpu_cap_has(vcpu, X86_FEATURE_FRED);
+}
+
 static void svm_prepare_switch_to_guest(struct kvm_vcpu *vcpu)
 {
 	struct vcpu_svm *svm = to_svm(vcpu);
@@ -1495,12 +1501,32 @@ static void svm_prepare_switch_to_guest(struct kvm_vcpu *vcpu)
 		sd->bp_spec_reduce_set = true;
 		msr_set_bit(MSR_ZEN4_BP_CFG, MSR_ZEN4_BP_CFG_BP_SPEC_REDUCE_BIT);
 	}
+
+	/* Hardware does not save/restore FRED_RSP0 for Non-SEV-ES guests. */
+	if (svm_manages_fred_rsp0(vcpu))
+		wrmsrq(MSR_IA32_FRED_RSP0, svm->msr_guest_fred_rsp0);
+
 	svm->guest_state_loaded = true;
 }
 
 static void svm_prepare_host_switch(struct kvm_vcpu *vcpu)
 {
-	to_svm(vcpu)->guest_state_loaded = false;
+	struct vcpu_svm *svm = to_svm(vcpu);
+
+	if (!svm->guest_state_loaded)
+		return;
+
+	/*
+	 * Hardware does not save/restore FRED_RSP0 for Non-SEV-ES guests.
+	 * Also, sync hardware MSR value to per-CPU cache. This helps in
+	 * restoring Host RSP0 when exiting to userspace in fred_update_rsp0().
+	 */
+	if (svm_manages_fred_rsp0(vcpu)) {
+		rdmsrq(MSR_IA32_FRED_RSP0, svm->msr_guest_fred_rsp0);
+		fred_sync_rsp0(svm->msr_guest_fred_rsp0);
+	}
+
+	svm->guest_state_loaded = false;
 }
 
 static void svm_vcpu_load(struct kvm_vcpu *vcpu, int cpu)
diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
index e958943b8162..581289cc6e0b 100644
--- a/arch/x86/kvm/svm/svm.h
+++ b/arch/x86/kvm/svm/svm.h
@@ -362,6 +362,8 @@ struct vcpu_svm {
 
 	bool guest_state_loaded;
 
+	u64 msr_guest_fred_rsp0;
+
 	bool avic_irq_window;
 	bool x2avic_msrs_intercepted;
 	bool lbr_msrs_intercepted;
-- 
2.43.0

[PATCH v3 4/9] KVM: SVM: Add support for saving and restoring FRED MSRs

[Shivansh Dhiman]

Handle the FRED MSRs in svm_get_msr()/svm_set_msr() so they can be read
and written by userspace via KVM_{GET,SET}_MSRS, as required for VM
save/restore and live migration.

FRED_RSP1..FRED_CONFIG are backed by their VMCB save-area fields.
FRED_RSP0 has no VMCB field and is served from the KVM-managed cache
(svm->msr_guest_fred_rsp0).

Signed-off-by: Shivansh Dhiman <shivansh.dhiman@amd.com>
---

Changes in v3:
* New patch.
* Lets userspace save/restore the FRED MSRs through KVM_{GET,SET}_MSRS.

---
 arch/x86/kvm/svm/svm.c | 45 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 45 insertions(+)

diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index c1263a3b3e73..72d49c6daee7 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -2877,6 +2877,39 @@ static bool svm_pat_accesses_gpat(struct kvm_vcpu *vcpu, bool from_host)
 	return !from_host && is_guest_mode(vcpu) && l2_has_separate_pat(vcpu);
 }
 
+#ifdef CONFIG_X86_64
+static_assert(offsetof(struct vmcb_save_area, fred_config) -
+	      offsetof(struct vmcb_save_area, fred_rsp1) ==
+	      (MSR_IA32_FRED_CONFIG - MSR_IA32_FRED_RSP1) * sizeof(u64));
+
+static u64 *fred_msr_to_vmcb_field(struct vcpu_svm *svm, u32 msr)
+{
+	return &svm->vmcb->save.fred_rsp1 + (msr - MSR_IA32_FRED_RSP1);
+}
+
+static u64 svm_read_guest_fred_rsp0(struct vcpu_svm *svm)
+{
+	preempt_disable();
+	if (svm->guest_state_loaded && svm_manages_fred_rsp0(&svm->vcpu))
+		rdmsrq(MSR_IA32_FRED_RSP0, svm->msr_guest_fred_rsp0);
+	preempt_enable();
+
+	return svm->msr_guest_fred_rsp0;
+}
+
+static void svm_write_guest_fred_rsp0(struct vcpu_svm *svm, u64 data)
+{
+	preempt_disable();
+	if (svm->guest_state_loaded && svm_manages_fred_rsp0(&svm->vcpu))
+		wrmsrq(MSR_IA32_FRED_RSP0, data);
+	preempt_enable();
+
+	svm->msr_guest_fred_rsp0 = data;
+}
+#else
+static u64 svm_read_guest_fred_rsp0(struct vcpu_svm *svm) { return 0; }
+#endif
+
 static int svm_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
 {
 	struct vcpu_svm *svm = to_svm(vcpu);
@@ -2915,6 +2948,12 @@ static int svm_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
 	case MSR_SYSCALL_MASK:
 		msr_info->data = svm->vmcb01.ptr->save.sfmask;
 		break;
+	case MSR_IA32_FRED_RSP0:
+		msr_info->data = svm_read_guest_fred_rsp0(svm);
+		break;
+	case MSR_IA32_FRED_RSP1 ... MSR_IA32_FRED_CONFIG:
+		msr_info->data = *fred_msr_to_vmcb_field(svm, msr_info->index);
+		break;
 #endif
 	case MSR_IA32_SYSENTER_CS:
 		msr_info->data = svm->vmcb01.ptr->save.sysenter_cs;
@@ -3160,6 +3199,12 @@ static int svm_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr)
 	case MSR_SYSCALL_MASK:
 		svm->vmcb01.ptr->save.sfmask = data;
 		break;
+	case MSR_IA32_FRED_RSP0:
+		svm_write_guest_fred_rsp0(svm, data);
+		break;
+	case MSR_IA32_FRED_RSP1 ... MSR_IA32_FRED_CONFIG:
+		*fred_msr_to_vmcb_field(svm, ecx) = data;
+		break;
 #endif
 	case MSR_IA32_SYSENTER_CS:
 		svm->vmcb01.ptr->save.sysenter_cs = data;
-- 
2.43.0

[PATCH v3 5/9] KVM: SVM: Populate FRED event data on event injection

[Shivansh Dhiman]

From: Neeraj Upadhyay <Neeraj.Upadhyay@amd.com>

Set injected-event data (in EVENTINJDATA) when injecting an event,
use EXITINTDATA for populating the injected-event data during
reinjection.

Unlike IDT using some extra CPU register as part of an event
context, e.g., %cr2 for #PF, FRED saves a complete event context
in its stack frame, e.g., FRED saves the faulting linear address
of a #PF into the event data field defined in its stack frame.

Populate the EVENTINJDATA during event injection. The event data
will be pushed into a FRED stack frame for VM entries that inject
an event using FRED event delivery.

Signed-off-by: Neeraj Upadhyay <Neeraj.Upadhyay@amd.com>
Co-developed-by: Shivansh Dhiman <shivansh.dhiman@amd.com>
Signed-off-by: Shivansh Dhiman <shivansh.dhiman@amd.com>
---

Changes in v3:
* Rebased, no changes.

Changes in v2:
* Removed the need for a new function parameter 'reinject_on_vmexit' in
  svm_complete_interrupts() (Paolo Bonzini).

---
 arch/x86/kvm/svm/svm.c | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index 72d49c6daee7..6c15e8480d25 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -401,6 +401,10 @@ static void svm_inject_exception(struct kvm_vcpu *vcpu)
 		| SVM_EVTINJ_VALID
 		| (ex->has_error_code ? SVM_EVTINJ_VALID_ERR : 0)
 		| SVM_EVTINJ_TYPE_EXEPT;
+
+	if (is_fred_enabled(vcpu))
+		svm->vmcb->control.event_inj_data = ex->event_data;
+
 	svm->vmcb->control.event_inj_err = ex->error_code;
 }
 
@@ -4478,6 +4482,7 @@ static void svm_complete_interrupts(struct kvm_vcpu *vcpu)
 		break;
 	case SVM_EXITINTINFO_TYPE_EXEPT: {
 		u32 error_code = 0;
+		u64 event_data = 0;
 
 		/*
 		 * Never re-inject a #VC exception.
@@ -4488,9 +4493,16 @@ static void svm_complete_interrupts(struct kvm_vcpu *vcpu)
 		if (exitintinfo & SVM_EXITINTINFO_VALID_ERR)
 			error_code = svm->vmcb->control.exit_int_info_err;
 
+		/*
+		 * FRED requires an additional field to pass injected-event
+		 * data to the guest.
+		 */
+		if (is_fred_enabled(vcpu) && (vector == PF_VECTOR || vector == DB_VECTOR))
+			event_data = svm->vmcb->control.exit_int_data;
+
 		kvm_requeue_exception(vcpu, vector,
 				      exitintinfo & SVM_EXITINTINFO_VALID_ERR,
-				      error_code, false, 0);
+				      error_code, false, event_data);
 		break;
 	}
 	case SVM_EXITINTINFO_TYPE_INTR:
@@ -4512,6 +4524,7 @@ static void svm_cancel_injection(struct kvm_vcpu *vcpu)
 
 	control->exit_int_info = control->event_inj;
 	control->exit_int_info_err = control->event_inj_err;
+	control->exit_int_data = control->event_inj_data;
 	control->event_inj = 0;
 	svm_complete_interrupts(vcpu);
 }
-- 
2.43.0

[PATCH v3 6/9] KVM: SVM: Support FRED nested exception injection

[Shivansh Dhiman]

From: Neeraj Upadhyay <Neeraj.Upadhyay@amd.com>

Set the SVM nested exception bit in EVENT_INJECTION_CTL when
injecting a nested exception using FRED event delivery to
ensure:
  1) A nested exception is injected on a correct stack level.
  2) The nested bit defined in FRED stack frame is set.

The event stack level used by FRED event delivery depends on whether
the event was a nested exception encountered during delivery of an
earlier event, because a nested exception is "regarded" as happening
on ring 0.  E.g., when #PF is configured to use stack level 1 in
IA32_FRED_STKLVLS MSR:
  - nested #PF will be delivered on the stack pointed by FRED_RSP1
    MSR when encountered in ring 3 and ring 0.
  - normal #PF will be delivered on the stack pointed by FRED_RSP0
    MSR when encountered in ring 3.

The SVM nested-exception support ensures a correct event stack level is
chosen when a VM entry injects a nested exception.

Signed-off-by: Neeraj Upadhyay <Neeraj.Upadhyay@amd.com>
Co-developed-by: Shivansh Dhiman <shivansh.dhiman@amd.com>
Signed-off-by: Shivansh Dhiman <shivansh.dhiman@amd.com>
Reviewed-by: Nikunj A Dadhania <nikunj@amd.com>
---

Changes in v3:
* Use ex->is_nested to check for nested exceptions.

Changes in v2:
* Replace variable 'nested' with 'is_nested' (Sean Christopherson).
* Formatted variables in reverse fir-tree order (Sean Christopherson).

---
 arch/x86/include/asm/svm.h | 1 +
 arch/x86/kvm/svm/svm.c     | 5 ++++-
 2 files changed, 5 insertions(+), 1 deletion(-)

diff --git a/arch/x86/include/asm/svm.h b/arch/x86/include/asm/svm.h
index 0570581e251a..1a6d4509b05b 100644
--- a/arch/x86/include/asm/svm.h
+++ b/arch/x86/include/asm/svm.h
@@ -664,6 +664,7 @@ static inline void __unused_size_checks(void)
 
 #define SVM_EVTINJ_VALID (1 << 31)
 #define SVM_EVTINJ_VALID_ERR (1 << 11)
+#define SVM_EVTINJ_NESTED_EXCEPTION    (1 << 13)
 
 #define SVM_EVTINJ_RESERVED_BITS ~(SVM_EVTINJ_VEC_MASK | SVM_EVTINJ_TYPE_MASK | \
 				   SVM_EVTINJ_VALID_ERR | SVM_EVTINJ_VALID)
diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index 6c15e8480d25..fb6b94405f88 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -389,6 +389,7 @@ static int svm_update_soft_interrupt_rip(struct kvm_vcpu *vcpu, u8 vector)
 static void svm_inject_exception(struct kvm_vcpu *vcpu)
 {
 	struct kvm_queued_exception *ex = &vcpu->arch.exception;
+	bool is_nested = is_fred_enabled(vcpu) && ex->is_nested;
 	struct vcpu_svm *svm = to_svm(vcpu);
 
 	kvm_deliver_exception_payload(vcpu, ex);
@@ -400,6 +401,7 @@ static void svm_inject_exception(struct kvm_vcpu *vcpu)
 	svm->vmcb->control.event_inj = ex->vector
 		| SVM_EVTINJ_VALID
 		| (ex->has_error_code ? SVM_EVTINJ_VALID_ERR : 0)
+		| (is_nested ? SVM_EVTINJ_NESTED_EXCEPTION : 0)
 		| SVM_EVTINJ_TYPE_EXEPT;
 
 	if (is_fred_enabled(vcpu))
@@ -4502,7 +4504,8 @@ static void svm_complete_interrupts(struct kvm_vcpu *vcpu)
 
 		kvm_requeue_exception(vcpu, vector,
 				      exitintinfo & SVM_EXITINTINFO_VALID_ERR,
-				      error_code, false, event_data);
+				      error_code, exitintinfo & SVM_EVTINJ_NESTED_EXCEPTION,
+				      event_data);
 		break;
 	}
 	case SVM_EXITINTINFO_TYPE_INTR:
-- 
2.43.0

[PATCH v3 7/9] KVM: SVM: Dump FRED context in dump_vmcb()

[Shivansh Dhiman]

Add fields related to FRED to dump_vmcb() to dump FRED context.

Signed-off-by: Shivansh Dhiman <shivansh.dhiman@amd.com>
---

Changes in v3:
* Update the print statements to align with fields in VMCB save area.

Changes in v2:
* Gate all the FRED MSRs through guest_cpu_cap_has() (Sean Christopherson).

---
 arch/x86/kvm/svm/svm.c | 43 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 43 insertions(+)

diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index fb6b94405f88..86a3ba54aab2 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -3589,6 +3589,11 @@ static void dump_vmcb(struct kvm_vcpu *vcpu)
 	pr_err("%-20s%016llx\n", "allowed_sev_features:", control->allowed_sev_features);
 	pr_err("%-20s%016llx\n", "guest_sev_features:", control->guest_sev_features);
 
+	if (guest_cpu_cap_has(vcpu, X86_FEATURE_FRED)) {
+		pr_err("%-20s%016llx\n", "exit_int_data:", control->exit_int_data);
+		pr_err("%-20s%016llx\n", "event_inj_data:", control->event_inj_data);
+	}
+
 	if (is_sev_es_guest(vcpu)) {
 		save = sev_decrypt_vmsa(vcpu);
 		if (!save)
@@ -3703,6 +3708,27 @@ static void dump_vmcb(struct kvm_vcpu *vcpu)
 		       "r14:", vmsa->r14, "r15:", vmsa->r15);
 		pr_err("%-15s %016llx %-13s %016llx\n",
 		       "xcr0:", vmsa->xcr0, "xss:", vmsa->xss);
+
+		if (guest_cpu_cap_has(vcpu, X86_FEATURE_FRED)) {
+			pr_err("%-27s %d %-18s%016llx\n",
+			       "is_fred_enabled:", is_fred_enabled(vcpu),
+			       "guest_evntinjdata:", vmsa->guest_event_inj_data);
+			pr_err("%-12s %016llx %-18s%016llx\n",
+			       "fred_config:", vmsa->fred_config,
+			       "guest_exitintdata:", vmsa->guest_exit_int_data);
+			pr_err("%-15s %016llx %-13s %016llx\n",
+			       "fred_rsp0:", vmsa->fred_rsp0,
+			       "fred_rsp1:", vmsa->fred_rsp1);
+			pr_err("%-15s %016llx %-13s %016llx\n",
+			       "fred_rsp2:", vmsa->fred_rsp2,
+			       "fred_rsp3:", vmsa->fred_rsp3);
+			pr_err("%-15s %016llx %-13s %016llx\n",
+			       "fred_stklvls:", vmsa->fred_stklvls,
+			       "fred_ssp1:", vmsa->fred_ssp1);
+			pr_err("%-15s %016llx %-13s %016llx\n",
+			       "fred_ssp2:", vmsa->fred_ssp2,
+			       "fred_ssp3:", vmsa->fred_ssp3);
+		}
 	} else {
 		pr_err("%-15s %016llx %-13s %016lx\n",
 		       "rax:", save->rax, "rbx:",
@@ -3730,6 +3756,23 @@ static void dump_vmcb(struct kvm_vcpu *vcpu)
 		       "r14:", vcpu->arch.regs[VCPU_REGS_R14],
 		       "r15:", vcpu->arch.regs[VCPU_REGS_R15]);
 #endif
+		if (guest_cpu_cap_has(vcpu, X86_FEATURE_FRED)) {
+			pr_err("%-15s %-15d %-13s %016llx\n",
+			       "is_fred_enabled:", is_fred_enabled(vcpu),
+			       "fred_config:", save->fred_config);
+			pr_err("%-15s %016llx %-13s %016llx\n",
+			       "fred_rsp0:", svm_read_guest_fred_rsp0(svm),
+			       "fred_rsp1:", save->fred_rsp1);
+			pr_err("%-15s %016llx %-13s %016llx\n",
+			       "fred_rsp2:", save->fred_rsp2,
+			       "fred_rsp3:", save->fred_rsp3);
+			pr_err("%-15s %016llx %-13s %016llx\n",
+			       "fred_stklvls:", save->fred_stklvls,
+			       "fred_ssp1:", save->fred_ssp1);
+			pr_err("%-15s %016llx %-13s %016llx\n",
+			       "fred_ssp2:", save->fred_ssp2,
+			       "fred_ssp3:", save->fred_ssp3);
+		}
 	}
 
 no_vmsa:
-- 
2.43.0

[PATCH v3 8/9] KVM: SVM: Enable FRED virtualization

[Shivansh Dhiman]

Set the FRED_VIRT_ENABLE bit (bit 4) in the misc_ctl2 field of the VMCB to
enable FRED virtualization. This enables hardware to automatically save and
restore the FRED MSRs on VMRUN and #VMEXIT.

Set the bit in svm_vcpu_after_set_cpuid() when FRED is exposed to the
guest, and clear it otherwise.

FRED virtualization requires vNMI to handle guest NMIs correctly, so, clear
FRED when vNMI is unavailable.

Signed-off-by: Shivansh Dhiman <shivansh.dhiman@amd.com>
---

Changes in v3:
* Renamed FRED_VIRT_ENABLE_MASK to SVM_MISC2_ENABLE_V_FRED.
* Set the bit in misc_ctl2, the current name of the VMCB field.
* Removed the unconditional kvm_cpu_cap_clear(X86_FEATURE_FRED).
* Clear FRED when vNMI is unavailable (Andrew Cooper).

Changes in v2:
* Confined the enabling of vFRED to svm_vcpu_after_set_cpuid() only
  (Sean Christopherson).

---
 arch/x86/include/asm/svm.h |  1 +
 arch/x86/kvm/svm/svm.c     | 11 ++++++++---
 2 files changed, 9 insertions(+), 3 deletions(-)

diff --git a/arch/x86/include/asm/svm.h b/arch/x86/include/asm/svm.h
index 1a6d4509b05b..288a8a78f1c7 100644
--- a/arch/x86/include/asm/svm.h
+++ b/arch/x86/include/asm/svm.h
@@ -250,6 +250,7 @@ struct __attribute__ ((__packed__)) vmcb_control_area {
 
 #define SVM_MISC2_ENABLE_V_LBR	BIT_ULL(0)
 #define SVM_MISC2_ENABLE_V_VMLOAD_VMSAVE	BIT_ULL(1)
+#define SVM_MISC2_ENABLE_V_FRED BIT_ULL(4)
 
 #define SVM_TSC_RATIO_RSVD	0xffffff0000000000ULL
 #define SVM_TSC_RATIO_MIN	0x0000000000000001ULL
diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index 86a3ba54aab2..25dd379725d4 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -4912,6 +4912,11 @@ static void svm_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu)
 	if (guest_cpuid_is_intel_compatible(vcpu))
 		guest_cpu_cap_clear(vcpu, X86_FEATURE_V_VMSAVE_VMLOAD);
 
+	if (guest_cpu_cap_has(vcpu, X86_FEATURE_FRED))
+		svm->vmcb->control.misc_ctl2 |= SVM_MISC2_ENABLE_V_FRED;
+	else
+		svm->vmcb->control.misc_ctl2 &= ~SVM_MISC2_ENABLE_V_FRED;
+
 	if (is_sev_guest(vcpu))
 		sev_vcpu_after_set_cpuid(svm);
 }
@@ -5675,9 +5680,6 @@ static __init void svm_set_cpu_caps(void)
 
 	kvm_cpu_cap_clear(X86_FEATURE_IBT);
 
-	/* SVM FRED virtualization not implemented yet */
-	kvm_cpu_cap_clear(X86_FEATURE_FRED);
-
 	/* CPUID 0x80000001 and 0x8000000A (SVM features) */
 	if (nested) {
 		kvm_cpu_cap_set(X86_FEATURE_SVM);
@@ -5751,6 +5753,9 @@ static __init void svm_set_cpu_caps(void)
 	/* CPUID 0x8000001F (SME/SEV features) */
 	sev_set_cpu_caps();
 
+	if (!vnmi)
+		kvm_cpu_cap_clear(X86_FEATURE_FRED);
+
 	/*
 	 * Clear capabilities that are automatically configured by common code,
 	 * but that require explicit SVM support (that isn't yet implemented).
-- 
2.43.0

[PATCH v3 9/9] KVM: SVM: Don't intercept ICEBP while the guest has FRED enabled

[Shivansh Dhiman]

Commit ec9a16c6aeba ("KVM: SVM: Always intercept ICEBP to workaround AMD
ICEBP+TASK_SWITCH flaws") makes KVM intercept ICEBP and inject the
resulting #DB as a hardware exception. A FRED guest therefore sees INT1
as event type 3 instead of 5. FRED has no task gates, so the workaround
isn't needed while CR4.FRED is set.

Stop intercepting ICEBP while CR4.FRED is set, and recalculate the
intercepts whenever svm_set_cr4() changes CR4.FRED.

Signed-off-by: Shivansh Dhiman <shivansh.dhiman@amd.com>
---

Changes in v3:
* New patch.

---
 arch/x86/kvm/svm/svm.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index 25dd379725d4..9b2cab844823 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -1137,6 +1137,11 @@ static void svm_recalc_instruction_intercepts(struct kvm_vcpu *vcpu)
 		svm_set_intercept(svm, INTERCEPT_RDPMC);
 	else
 		svm_clr_intercept(svm, INTERCEPT_RDPMC);
+
+	if (is_fred_enabled(vcpu))
+		svm_clr_intercept(svm, INTERCEPT_ICEBP);
+	else
+		svm_set_intercept(svm, INTERCEPT_ICEBP);
 }
 
 static void svm_recalc_intercepts(struct kvm_vcpu *vcpu)
@@ -1911,6 +1916,9 @@ void svm_set_cr4(struct kvm_vcpu *vcpu, unsigned long cr4)
 
 	if ((cr4 ^ old_cr4) & (X86_CR4_OSXSAVE | X86_CR4_PKE))
 		vcpu->arch.cpuid_dynamic_bits_dirty = true;
+
+	if ((cr4 ^ old_cr4) & X86_CR4_FRED)
+		kvm_make_request(KVM_REQ_RECALC_INTERCEPTS, vcpu);
 }
 
 static void svm_set_segment(struct kvm_vcpu *vcpu,
-- 
2.43.0