Small refactor that replaces array of v0.1 function IDs indexed by an
enum of function-name constants with a struct of function IDs "indexed"
by field names. This is done in preparation for exposing the IDs to
other parts of the kernel. Exposing a struct avoids the need for
bounds checking.
Signed-off-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Acked-by: Mark Rutland <mark.rutland@arm.com>
Link: https://lore.kernel.org/r/20201202184122.26046-6-dbrazdil@google.com
(cherry picked from commit 82ac62d1658b42392282550078a189ccd3f50214)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I647ba9d5de29a20fea51d57752dbaacc72102690
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
Refactor implementation of v0.1+ functions (CPU_SUSPEND, CPU_OFF,
CPU_ON, MIGRATE) to have two functions psci_0_1_foo / psci_0_2_foo that
select the function ID and call a common helper __psci_foo.
This is a small cleanup so that the function ID array is only used for
v0.1 configurations.
Signed-off-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Acked-by: Mark Rutland <mark.rutland@arm.com>
Link: https://lore.kernel.org/r/20201202184122.26046-5-dbrazdil@google.com
(cherry picked from commit 0bc7474fb7673422b134e88feb49cde54b22bb75)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I831b00e9c48bcca9a9fff66d613b99229ee91dee
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
KVM's host PSCI SMC filter needs to be aware of the PSCI version of the
system but currently it is impossible to distinguish between v0.1 and
PSCI disabled because both have get_version == NULL.
Populate get_version for v0.1 with a function that returns a constant.
psci_opt.get_version is currently unused so this has no effect on
existing functionality.
Signed-off-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Acked-by: Mark Rutland <mark.rutland@arm.com>
Link: https://lore.kernel.org/r/20201202184122.26046-4-dbrazdil@google.com
(cherry picked from commit e6dd9d89a64e30b25339d0dbe5c5aa589db8d530)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: Ied02571cdf87ec7b86dc1e0f6605a136d9fe26bd
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
Expose the boolean value whether the system is running with KVM in
protected mode (nVHE + kernel param). CPU capability was selected over
a global variable to allow use in alternatives.
Signed-off-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201202184122.26046-3-dbrazdil@google.com
(cherry picked from commit 3eb681fba2bf8b67b65ce92d0ebfd7cbfc263da9)
[will: Fix conflict in cpucaps definition, as ARM64_HAS_LDAPR is not present]
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I8bc16ffe7c024855ad1ae282749d660c7d48de90
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
Add an early parameter that allows users to select the mode of operation
for KVM/arm64.
For now, the only supported value is "protected". By passing this flag
users opt into the hypervisor placing additional restrictions on the
host kernel. These allow the hypervisor to spawn guests whose state is
kept private from the host. Restrictions will include stage-2 address
translation to prevent host from accessing guest memory, filtering its
SMC calls, etc.
Without this parameter, the default behaviour remains selecting VHE/nVHE
based on hardware support and CONFIG_ARM64_VHE.
Signed-off-by: David Brazdil <dbrazdil@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201202184122.26046-2-dbrazdil@google.com
(cherry picked from commit d8b369c4e31430a4746571bcae45a98933827232)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I1fe46bb18b40d0a1df41a600a07b848f82988ac6
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
As with SCTLR_ELx and other control registers, some PSTATE bits are
UNKNOWN out-of-reset, and we may not be able to rely on hardware or
firmware to initialize them to our liking prior to entry to the kernel,
e.g. in the primary/secondary boot paths and return from idle/suspend.
It would be more robust (and easier to reason about) if we consistently
initialized PSTATE to a default value, as we do with control registers.
This will ensure that the kernel is not adversely affected by bits it is
not aware of, e.g. when support for a feature such as PAN/UAO is
disabled.
This patch ensures that PSTATE is consistently initialized at boot time
via an ERET. This is not intended to relax the existing requirements
(e.g. DAIF bits must still be set prior to entering the kernel). For
features detected dynamically (which may require system-wide support),
it is still necessary to subsequently modify PSTATE.
As ERET is not always a Context Synchronization Event, an ISB is placed
before each exception return to ensure updates to control registers have
taken effect. This handles the kernel being entered with SCTLR_ELx.EOS
clear (or any future control bits being in an UNKNOWN state).
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Cc: Christoph Hellwig <hch@lst.de>
Cc: James Morse <james.morse@arm.com>
Cc: Will Deacon <will@kernel.org>
Link: https://lore.kernel.org/r/20201113124937.20574-6-mark.rutland@arm.com
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit d87a8e65b5101123a24cddeb7a8a2c7b45f7b60c)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: Ib189dca3244847e2f500fb7b4ca4ee1643480765
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
Let's make SCTLR_ELx initialization a bit clearer by using meaningful
names for the initialization values, following the same scheme for
SCTLR_EL1 and SCTLR_EL2.
These definitions will be used more widely in subsequent patches.
There should be no functional change as a result of this patch.
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Cc: Christoph Hellwig <hch@lst.de>
Cc: James Morse <james.morse@arm.com>
Cc: Will Deacon <will@kernel.org>
Link: https://lore.kernel.org/r/20201113124937.20574-5-mark.rutland@arm.com
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit 2ffac9e3fdbd54be953e773f9deb08fc6a488a9f)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: Icc3118148e8b0bb93ce221e64d5aaa95f6f76195
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
For a while now el2_setup has performed some basic initialization of EL1
even when the kernel is booted at EL1, so the name is a little
misleading. Further, some comments are stale as with VHE it doesn't drop
the CPU to EL1.
To clarify things, rename el2_setup to init_kernel_el, and update
comments to be clearer as to the function's purpose.
There should be no functional change as a result of this patch.
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Cc: Christoph Hellwig <hch@lst.de>
Cc: James Morse <james.morse@arm.com>
Cc: Will Deacon <will@kernel.org>
Link: https://lore.kernel.org/r/20201113124937.20574-4-mark.rutland@arm.com
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit ecbb11ab3ebc02763ec53489c9b1f983be9dc882)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: Idcff6f2e73aafe79a8e3605463639326dd313848
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
To make callsites easier to read, add trivial C wrappers for the
SET_PSTATE_*() helpers, and convert trivial uses over to these. The new
wrappers will be used further in subsequent patches.
There should be no functional change as a result of this patch.
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Cc: Christoph Hellwig <hch@lst.de>
Cc: James Morse <james.morse@arm.com>
Cc: Will Deacon <will@kernel.org>
Link: https://lore.kernel.org/r/20201113124937.20574-3-mark.rutland@arm.com
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit 515d5c8a1374b307225e41b3e66b0aad08585f53)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I0b45ca0bab8610eaf9b649d04fdb101e05ce3ada
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
For consistency, all tasks have a pt_regs reserved at the highest
portion of their task stack. Among other things, this ensures that a
task's SP is always pointing within its stack rather than pointing
immediately past the end.
While it is never legitimate to ERET from a kthread, we take pains to
initialize pt_regs for kthreads as if this were legitimate. As this is
never legitimate, the effects of an erroneous return are rarely tested.
Let's simplify things by initializing a kthread's pt_regs such that an
ERET is caught as an illegal exception return, and removing the explicit
initialization of other exception context. Note that as
spectre_v4_enable_task_mitigation() only manipulates the PSTATE within
the unused regs this is safe to remove.
As user tasks will have their exception context initialized via
start_thread() or start_compat_thread(), this should only impact cases
where something has gone very wrong and we'd like that to be clearly
indicated.
Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Cc: Christoph Hellwig <hch@lst.de>
Cc: James Morse <james.morse@arm.com>
Cc: Will Deacon <will@kernel.org>
Link: https://lore.kernel.org/r/20201113124937.20574-2-mark.rutland@arm.com
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit f80d034086d5bfcfd3bf4ab6f52b2df78c3ad2fa)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: Id6fca836e7830efb4df34350c01d44e7317b2dbd
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
Cores that predate the introduction of ID_AA64PFR0_EL1.CSV3 to
the ARMv8 architecture have this field set to 0, even of some of
them are not affected by the vulnerability.
The kernel maintains a list of unaffected cores (A53, A55 and a few
others) so that it doesn't impose an expensive mitigation uncessarily.
As we do for CSV2, let's expose the CSV3 property to guests that run
on HW that is effectively not vulnerable. This can be reset to zero
by writing to the ID register from userspace, ensuring that VMs can
be migrated despite the new property being set.
Reported-by: Will Deacon <will@kernel.org>
Signed-off-by: Marc Zyngier <maz@kernel.org>
(cherry picked from commit 4f1df628d4ec22b04f67e068e6d02538d3dd557b)
[will: Fix conflict with asymmetric 32-bit code in read_id_reg()]
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: Ifed7427d58bd57d98dce35d0abacdd2c25b98911
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
In order to reduce the impact of the VPT parsing happening on the GIC,
we can split the vcpu reseidency in two phases:
- programming GICR_VPENDBASER: this still happens in vcpu_load()
- checking for the VPT parsing to be complete: this can happen
on vcpu entry (in kvm_vgic_flush_hwstate())
This allows the GIC and the CPU to work in parallel, rewmoving some
of the entry overhead.
Suggested-by: Marc Zyngier <maz@kernel.org>
Signed-off-by: Shenming Lu <lushenming@huawei.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201128141857.983-3-lushenming@huawei.com
(cherry picked from commit 57e3cebd022fbc035dcf190ac789fd2ffc747f5b)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I5d3b5db2db817e99780e009caf8efe0f6692e0b1
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
Our Meltdown mitigation state isn't exposed outside of cpufeature.c,
contrary to the rest of the Spectre mitigation state. As we are going
to use it in KVM, expose a arm64_get_meltdown_state() helper which
returns the same possible values as arm64_get_spectre_v?_state().
Signed-off-by: Marc Zyngier <maz@kernel.org>
(cherry picked from commit 7f43c2014fa03bb8718569ae628acf2089683bff)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: If31ecb6978a131696dd1cb1aaf8dfbb4b2cf93f9
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
DEMUX register presence depends on the host's hardware (the
CLIDR_EL1 register to be precise). This means there's no set
of them that we can bless and that it's possible to encounter
new ones when running on different hardware (which would
generate "Consider adding them ..." messages, but we'll never
want to add them.)
Remove the ones we have in the blessed list and filter them
out of the new list, but also provide a new command line switch
to list them if one so desires.
Signed-off-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201126134641.35231-3-drjones@redhat.com
(cherry picked from commit c6232bd40b2eda3819d108e6e3f621ec604e15d8)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I4b4a088b986837f7ef50451d616e6090a07e5032
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
Not counting TnD, which KVM doesn't currently consider, CSSELR_EL1
can have a maximum value of 0b1101 (13), which corresponds to an
instruction cache at level 7. With CSSELR_MAX set to 12 we can
only select up to cache level 6. Change it to 14.
Signed-off-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201126134641.35231-2-drjones@redhat.com
(cherry picked from commit c73a44161776f6e60d933717f3b34084b0a0eba0)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I3030c34b51fc7e596d723fe071459a3719924455
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
__extended_idmap_trampoline() was removed a long time ago by
3421e9d88d ("arm64: KVM: Simplify HYP init/teardown") so remove the
unused function prototype.
Signed-off-by: Will Deacon <will@kernel.org>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Cc: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201118194402.2892-4-will@kernel.org
(cherry picked from commit bf118a5cb7e6d17e7ec9492e4dc676e7e7b69d01)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: Ib8a27cd58a82ce133ebaabee6572f88c5238dab4
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
kvm_arch_vm_ioctl_check_extension() is only called from
kvm_vm_ioctl_check_extension(), so we can inline it and remove the extra
function.
Signed-off-by: Will Deacon <will@kernel.org>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Cc: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201118194402.2892-3-will@kernel.org
(cherry picked from commit 36fb4cd55f626dff0f6e76bed14707fa00147b7f)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I41b82f4316871a4366d568eb9ec0a8fd4de6615f
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
'struct kvm_arch_memory_slot' isn't part of the user ABI, so move it out
of the uapi/ headers in case we start using it in future and accidentally
back ourselves into a corner.
Signed-off-by: Will Deacon <will@kernel.org>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201118194402.2892-2-will@kernel.org
(cherry picked from commit 8d14797b53f044fda3ed42b5b6357c7622b8af58)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: Id24327a02b210ed8efa65086bd9173558111e3a1
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
The PMU ready state has no user left. Goodbye.
Reviewed-by: Alexandru Elisei <alexandru.elisei@arm.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
(cherry picked from commit 7521c3a9e63041602d531e36c07a340f188dc1fa)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: Ic2ac0745d8e41aa439f560e5b385df8493f34620
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
We currently gate the update of the PMU state on the PMU being "ready".
The "ready" state is only set to true when the first vcpu run is
successful, and if it isn't, we never reach the update code.
So the "ready" state is never the right thing to check for, and it
should instead be the presence of the PMU feature, which makes
a bit more sense.
Reviewed-by: Alexandru Elisei <alexandru.elisei@arm.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
(cherry picked from commit 46acf89de499b2db07e120c62a796e8a0efbad8d)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: If51c248249d72b1256339c864a3eff0b1681e9f2
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
The handling of traps in access_pmu_evcntr() has a couple of
omminous "else return false;" statements that don't make any sense:
the decoding tree coverse all the registers that trap to this handler,
and returning false implies that we change PC, which we don't.
Get rid of what is evidently dead code.
Reviewed-by: Alexandru Elisei <alexandru.elisei@arm.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
(cherry picked from commit a3da93580202ac9075d4e96f73c8435b9d7262c1)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I70e709ef540eb13ff7b3f79d2ab507619c0797d4
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
There is no RAZ/WI handling allowed for the PMU registers in the
ARMv8 architecture. Nobody can remember how we cam to the conclusion
that we could do this, but the ARMv8 ARM is pretty clear that we cannot.
Remove the RAZ/WI handling of the PMU system registers when it is
not configured.
Reviewed-by: Alexandru Elisei <alexandru.elisei@arm.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
(cherry picked from commit f975ccb08d6530e58bac660c7a938f98bae5a651)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: Ic0b5bdfdc4cd0afb2189392de77c424067d1ed31
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
The ARMv8 architecture says that in the absence of FEAT_PMUv3,
all the PMU-related register generate an UNDEF. Let's make
sure that all our PMU handers catch this case by hooking into
check_pmu_access_disabled(), and add checks in a couple of
other places.
Note that we still cannot deliver an exception into the guest
as the offending cases are already caught by the RAZ/WI handling.
But this puts us one step away to architectural compliance.
Reviewed-by: Alexandru Elisei <alexandru.elisei@arm.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
(cherry picked from commit b0737e999ec0af007b10ac0b7db97932394a248f)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I72648165dd84156f5002a84f08be9c34c44d7747
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
We accept to configure a PMU when a vcpu is created, even if the
HW (or the host) doesn't support it. This results in failures
when attributes get set, which is a bit odd as we should have
failed the vcpu creation the first place.
Move the check to the point where we check the vcpu feature set,
and fail early if we cannot support a PMU. This further simplifies
the attribute handling.
Reviewed-by: Alexandru Elisei <alexandru.elisei@arm.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
(cherry picked from commit 77da43039ab5cfc9631159fd87fe38d4c34cdaf5)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: Ieefc9cd0f3f0d56a642ac2fa563f8e711f282c8e
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
We always expose the HW view of PMU in ID_AA64FDR0_EL1.PMUver,
even when the PMU feature is disabled, while the architecture
says that FEAT_PMUv3 not being implemented should result in this
field being zero.
Let's follow the architecture's guidance.
Reviewed-by: Alexandru Elisei <alexandru.elisei@arm.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
(cherry picked from commit 04355e41a60338206d6498fe463a86131d5ca06b)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I3fdccaa1461828b53bd82dc89a11f10753d4c104
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
When enabling the PMU in kvm_arm_pmu_v3_enable(), KVM returns early if the
PMU flag created is false and skips any other checks. Because PMU emulation
is gated only on the VCPU feature being set, this makes it possible for
userspace to get away with setting the VCPU feature but not doing any
initialization for the PMU. Fix it by returning an error when trying to run
the VCPU if the PMU hasn't been initialized correctly.
The PMU is marked as created only if the interrupt ID has been set when
using an in-kernel irqchip. This means the same check in
kvm_arm_pmu_v3_enable() is redundant, remove it.
Signed-off-by: Alexandru Elisei <alexandru.elisei@arm.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201126144916.164075-1-alexandru.elisei@arm.com
(cherry picked from commit 9bbfa4b565379eeb2fb8fdbcc9979549ae0e48d9)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: Ia11578ad1b6e12a3afcb22e0e589a4f74ceb0278
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
There are a number of places where we check for the KVM_ARM_VCPU_PMU_V3
feature. Wrap this check into a new kvm_vcpu_has_pmu(), and use
it at the existing locations.
No functional change.
Reviewed-by: Alexandru Elisei <alexandru.elisei@arm.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
(cherry picked from commit 14bda7a927336055d7c0deb1483f9cdb687c2080)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: Ia09ef62f014c55aaee381ca77e212501a83a9883
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
Registers x0/x1 get repeateadly pushed and poped during a host
HVC call. Instead, leave the registers on the stack, trading
a store instruction on the fast path for an add on the slow path.
Reviewed-by: Alexandru Elisei <alexandru.elisei@arm.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
(cherry picked from commit 83fa381f66ccb025f9e182d0b2ef9cd53c1f33ab)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: Idf1e4d523de3fd54ff99c3e8eb486256232c6ea4
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
Move the setting of SSBS directly into the HVC handler, using
the C helpers rather than the inline asssembly code.
Reviewed-by: Alexandru Elisei <alexandru.elisei@arm.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
(cherry picked from commit 29052f1b92f2bcb0419c544459f4c919bfb20898)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I0aa6e026e812bdbe52d5dd307f21d6e96d1aa76b
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
Directly using the kimage_voffset variable is fine for now, but
will become more problematic as we start distrusting EL1.
Instead, patch the kimage_voffset into the HYP text, ensuring
we don't have to load an untrusted value later on.
Signed-off-by: Marc Zyngier <maz@kernel.org>
(cherry picked from commit 68b824e428c5fb5c3dc5ef80b1543e767534b58e)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I775f901c7fece35bd7bd23ff15aeaa3ae5c9b32a
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
The hyp vectors entry corresponding to HYP_VECTOR_DIRECT (i.e. when
neither Spectre-v2 nor Spectre-v3a are present) is unused, as we can
simply dispatch straight to __kvm_hyp_vector in this case.
Remove the redundant vector, and massage the logic for resolving a slot
to a vectors entry.
Reported-by: Marc Zyngier <maz@kernel.org>
Signed-off-by: Will Deacon <will@kernel.org>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20201113113847.21619-11-will@kernel.org
(cherry picked from commit 4f6a36fed71dfe51df0ae9a282dc87c76d629bff)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I2fdfcbace784838d57ff0742fb980410d76c9f4d
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
The spectre-v3a mitigation is split between cpu_errata.c and spectre.c,
with the former handling detection of the problem and the latter handling
enabling of the workaround.
Move the detection logic alongside the enabling logic, like we do for the
other spectre mitigations.
Signed-off-by: Will Deacon <will@kernel.org>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Quentin Perret <qperret@google.com>
Link: https://lore.kernel.org/r/20201113113847.21619-10-will@kernel.org
(cherry picked from commit cd1f56b930e857c170d8a04f0f989bfb8a1b5ac1)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I01c54f09a8491c883666935bed50476c2f694823
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
Since ARM64_HARDEN_EL2_VECTORS is really a mitigation for Spectre-v3a,
rename it accordingly for consistency with the v2 and v4 mitigation.
Signed-off-by: Will Deacon <will@kernel.org>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Quentin Perret <qperret@google.com>
Link: https://lore.kernel.org/r/20201113113847.21619-9-will@kernel.org
(cherry picked from commit c4792b6dbc5070fe67f4cdcfdad39416333acbe0)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I6c026fa775b858cca5360aeabb25aeab209f72f7
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
The EL2 vectors installed when a guest is running point at one of the
following configurations for a given CPU:
- Straight at __kvm_hyp_vector
- A trampoline containing an SMC sequence to mitigate Spectre-v2 and
then a direct branch to __kvm_hyp_vector
- A dynamically-allocated trampoline which has an indirect branch to
__kvm_hyp_vector
- A dynamically-allocated trampoline containing an SMC sequence to
mitigate Spectre-v2 and then an indirect branch to __kvm_hyp_vector
The indirect branches mean that VA randomization at EL2 isn't trivially
bypassable using Spectre-v3a (where the vector base is readable by the
guest).
Rather than populate these vectors dynamically, configure everything
statically and use an enumerated type to identify the vector "slot"
corresponding to one of the configurations above. This both simplifies
the code, but also makes it much easier to implement at EL2 later on.
Signed-off-by: Will Deacon <will@kernel.org>
[maz: fixed double call to kvm_init_vector_slots() on nVHE]
Signed-off-by: Marc Zyngier <maz@kernel.org>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Quentin Perret <qperret@google.com>
Link: https://lore.kernel.org/r/20201113113847.21619-8-will@kernel.org
(cherry picked from commit b881cdce77b48bd488f268041f32951bab89bb0f)
[will: fixed hyp/Makefile conflict with LTO+CFI flags]
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I72ae8cfefa18c318bd69de986923e05159e64df4
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
The hardened hyp vectors are not used on systems running with VHE or CPUs
without the ARM64_HARDEN_EL2_VECTORS capability.
Re-jig the checking logic slightly in kvm_patch_vector_branch() so that
it's a bit clearer what we're looking for. This is purely cosmetic.
Signed-off-by: Will Deacon <will@kernel.org>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Quentin Perret <qperret@google.com>
Link: https://lore.kernel.org/r/20201113113847.21619-7-will@kernel.org
(cherry picked from commit da592e68a5a333b81111bd6336838764732f723e)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I6b9585ba1b0487a86a7af8f4f50c63d03c6f132a
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
The BP hardening helpers are an integral part of the Spectre-v2
mitigation, so move them into asm/spectre.h and inline the
arm64_get_bp_hardening_data() function at the same time.
Signed-off-by: Will Deacon <will@kernel.org>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Quentin Perret <qperret@google.com>
Link: https://lore.kernel.org/r/20201113113847.21619-6-will@kernel.org
(cherry picked from commit 6279017e807708a07db5edace462713a93625da3)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I61c8a5b9c451f6f60038b902d4c6f68ca0817ffc
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
Branch predictor hardening of the hyp vectors is partially driven by a
couple of global variables ('__kvm_bp_vect_base' and
'__kvm_harden_el2_vector_slot'). However, these are only used within a
single compilation unit, so internalise them there instead.
Signed-off-by: Will Deacon <will@kernel.org>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Quentin Perret <qperret@google.com>
Link: https://lore.kernel.org/r/20201113113847.21619-5-will@kernel.org
(cherry picked from commit 07cf8aa922db7747cd6e100d2e3f7ca839c7a419)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: Id89bad6b3c1f190d04ba6230658854e39e74e6db
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
kvm_get_hyp_vector() has only one caller, so move it out of kvm_mmu.h
and inline it into a new function, cpu_set_hyp_vector(), for setting
the vector.
Signed-off-by: Will Deacon <will@kernel.org>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Quentin Perret <qperret@google.com>
Link: https://lore.kernel.org/r/20201113113847.21619-4-will@kernel.org
(cherry picked from commit 042c76a9502bf281befc0ae2793ef1de55b65544)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I7480b810f73e33ca0d6ec8fe1cffbd345162a4ac
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
The bulk of the work in kvm_map_vector() is conditional on the
ARM64_HARDEN_EL2_VECTORS capability, so return early if that is not set
and make the code a bit easier to read.
Signed-off-by: Will Deacon <will@kernel.org>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Quentin Perret <qperret@google.com>
Link: https://lore.kernel.org/r/20201113113847.21619-3-will@kernel.org
(cherry picked from commit de5bcdb48498abeb019ae075d139850c52661627)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I671d9f8a5a7aa37cff1c43d373926b805427d759
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
'__kvm_bp_vect_base' is only used when dealing with the hardened vectors
so remove the redundant assignments in kvm_map_vectors().
Signed-off-by: Will Deacon <will@kernel.org>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Quentin Perret <qperret@google.com>
Link: https://lore.kernel.org/r/20201113113847.21619-2-will@kernel.org
(cherry picked from commit 8934c8454064757efd8d3fb0a729db7eb2d0e5f5)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I332ed153c64f8e31194f854e9d2b35f5fa504165
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
kvm_coproc.h used to serve as a compatibility layer for the files
shared between the 32 and 64 bit ports.
Another one bites the dust...
Signed-off-by: Marc Zyngier <maz@kernel.org>
(cherry picked from commit 6ac4a5ac50d1d25a61aa00e660eebb21a2ff9b96)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I535cd17d964725eb9ce5f549f567f25957336136
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
Finally remove one of the biggest 32bit legacy: the copro shadow
mapping. We won't missit.
Signed-off-by: Marc Zyngier <maz@kernel.org>
(cherry picked from commit 5f7e02aebdf0c8d255f3ff2df8595fd220e7d5ce)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: If7a26e294bdb3ae4d0d5a02e93b91a0094aaa275
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
is_aarch32 is only used once, and can be trivially replaced by
testing Op0 instead. Drop it.
Signed-off-by: Marc Zyngier <maz@kernel.org>
(cherry picked from commit 50f304532770c19a127b1e1b6769c0538abda58f)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: Iaf861445c29101d0adbe7788db3da19f2d06ffb6
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
The is_32bit attribute is now completely unused, drop it.
Signed-off-by: Marc Zyngier <maz@kernel.org>
(cherry picked from commit 2d27fd784893a767ec4162afc6d8c86eec2d1bfe)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I4d90edc8528efa4d8cb05d12e6b86cdd79717ac8
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
Similarly to what has been done on the cp15 front, repaint the
debug registers to use their AArch64 counterparts. This results
in some simplification as we can remove the 32bit-specific
accessors.
Signed-off-by: Marc Zyngier <maz@kernel.org>
(cherry picked from commit 1da42c34d7c42fe2840bfe3de83cd0b5aa374859)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: Ifda7d49fd448530cf24cf1520404bdfca6f106fe
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
Move all the cp15 registers over to their AArch64 counterpart.
This requires the annotation of a few of them (such as the usual
DFAR/IFAR vs FAR_EL1), and a new helper that generates mask/shift
pairs for the various configurations.
Signed-off-by: Marc Zyngier <maz@kernel.org>
(cherry picked from commit b1ea1d760d3331da19e33650bf8c09ce028a0a49)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: Iaae22dad0f62628595db26d9ac15ca5e593263bb
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
In order to deal with the few AArch32 system registers that map to
only a particular half of their AArch64 counterpart (such as DFAR
and IFAR being colocated in FAR_EL1), let's add an optional annotation
to the sysreg descriptor structure, indicating whether a register
maps to the upper or lower 32bits of a register.
Nothing is using these annotation yet.
Signed-off-by: Marc Zyngier <maz@kernel.org>
(cherry picked from commit 6ed6750f2b6d4a51f27615f3323d1850449299e3)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: Ibeda8db427f22e7a4b3d53a86c99680325b45de9
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
The use of the AArch32-specific accessors have always been a bit
annoying on 64bit, and it is time for a change.
Let's move the AArch32 exception injection over to the AArch64 encoding,
which requires us to split the two halves of FAR_EL1 into DFAR and IFAR.
This enables us to drop the preempt_disable() games on VHE, and to kill
the last user of the vcpu_cp15() macro.
Signed-off-by: Marc Zyngier <maz@kernel.org>
(cherry picked from commit 4ff3fc316d78daa2ed6de2f13616fb33a2926d8e)
Signed-off-by: Will Deacon <willdeacon@google.com>
Change-Id: I1ae87d54b17b1b897952b37d2c7e239994f4b197
Bug: 178098380
Test: atest VirtualizationHostTestCases on an EL2-enabled device
