TLB maintenance is currently performed while a buffer is being unmapped.
For large buffers, doing so is suboptimal, as opposed to invalidating
the entire TLB for a particular context after the buffer has been
unmapped.
Thus, defer TLB maintenance until a buffer is unmapped, and
iommu_iotlb_sync() is invoked. This shows a significant amount
of improvement in the latency incurred by unmapping a buffer.
Without this patch, we observe the following latencies:
(average over 10 iterations)
size iommu_map iommu_unmap
4K 1.265 us 651.619 us
64K 7.666 us 678.968 us
1M 90.979 us 1152.072 us
2M 179.885 us 2303.020 us
12M 1082.140 us 5537.349 us
24M 2159.463 us 9415.588 us
32M 2878.609 us 12001.406 us
size iommu_map_sg iommu_unmap
4K 1.088 us 647.921 us
64K 7.208 us 680.312 us
1M 103.505 us 1153.520 us
2M 200.885 us 2302.593 us
12M 1159.146 us 5534.989 us
24M 2300.744 us 9411.614 us
32M 3057.343 us 12000.468 us
While applying this patch yields the following latencies:
(average over 10 iterations)
size iommu_map iommu_unmap
4K 1.172 us 5.218 us
64K 6.229 us 9.338 us
1M 91.812 us 77.828 us
2M 179.500 us 154.156 us
12M 1077.927 us 154.572 us
24M 2159.630 us 157.453 us
32M 2883.953 us 157.921 us
size iommu_map_sg iommu_unmap
4K 1.041 us 5.005 us
64K 6.781 us 9.364 us
1M 102.390 us 79.515 us
2M 200.328 us 152.270 us
12M 1161.000 us 154.515 us
24M 2304.369 us 157.822 us
32M 3059.416 us 160.734 us.
Change-Id: I7aecf559746eb65d2543ce9b16ad12492eb70fa1
Signed-off-by: Isaac J. Manjarres <isaacm@codeaurora.org>
Page table memory is currently freed when an empty table is encountered
and we are about to replace the table entry with a block mapping.
Removing the table entry implies that TLB maintenance must be performed
to get rid of TLB entries that may refer to the now empty table as part
of the page table walk.
In preparation for deferring TLB maintenance to only when a buffer has
been completely unmapped, add support for freeing empty page tables when
memory is unmapped, so no TLB maintenance is required when mapping memory.
Change-Id: Ic2ffc8ed38d1df2443844fe69a50e2c06484f648
Signed-off-by: Isaac J. Manjarres <isaacm@codeaurora.org>
Add support for mapping memory with the attributes required
for it to be cached in the system cache, with a NWA policy:
MAIR: 0xe4: inner non-cacheable, outer write-back read allocate.
Change-Id: I4baa9bc32e20c2736867bb9871b3fcdedab0bafb
Signed-off-by: Isaac J. Manjarres <isaacm@codeaurora.org>
Add support for mapping memory with the attributes required
for it to be cached in the system cache:
MAIR: 0xf4: inner non-cacheable, outer write-back read/write allocate.
Change-Id: I1fb59d272223cc2a0d34250e7442fafb7190475d
Signed-off-by: Isaac J. Manjarres <isaacm@codeaurora.org>
The IO_PGTABLE_QUIRK_QCOM_USE_LLC_NWA quirk is used to ensure that
the IOMMU page tables are cached in the system cache with a no write
allocation cache policy. Add support for it by setting up the TCR
with the following memory attributes for the page table walker:
TCR.SH = Outer-shareable
TCR.IRGN = Non-cacheable normal memory
TCR.ORGN = Write-back, no write-allocate cacheable.
Change-Id: Ifa88b673de3b756e5b03bc36e89db84bc013346a
Signed-off-by: Isaac J. Manjarres <isaacm@codeaurora.org>
The IO_PGTABLE_QUIRK_QCOM_USE_UPSTREAM_HINT is used to ensure that the
IOMMU page tables are cached in the system cache. Add support for it
by setting up the TCR with the following memory attributes for the
page table walker:
TCR.SH = Outer Shareable
TCR.IRGN = Non-cacheable normal memory
TCR.ORGN = Write-back, write-allocate cacheable.
Change-Id: Iafb16fdee078af746a66821bb50192198beba5bc
Signed-off-by: Isaac J. Manjarres <isaacm@codeaurora.org>
Add the initial implementation for an IOMMU driver to allocate
and free page table memory. The IOMMU driver implementation of these
hooks will take care of preparing the page tables prior to use.
Change-Id: I1c3ef02fc9464a31d0e0ce65627692d53aa0f976
Signed-off-by: Isaac J. Manjarres <isaacm@codeaurora.org>
Add support for using this submodule.
Change-Id: I3658589b1d38ddcdf8bc9d2c01f6042cfeb964d5
Signed-off-by: Patrick Daly <pdaly@codeaurora.org>
Signed-off-by: Isaac J. Manjarres <isaacm@codeaurora.org>
Create a fork of io-pgtable-arm as of android12-5.10
commit 19057a6a6b ("Merge 5.10.4 into android12-5.10").
This is done in order to support qcom value added features such as:
qcom secure memory model.
refcounting & freeing page table memory.
map_sg operation
intelligent tlb invalidate operations.
Some of the above feature may be outdated or no longer required
with new hardware, or may have upstream alternatives. However,
proving that these features are unnessary may require extensive
testing. Therefore, port them to the GKI model so that this
testing may take place.
Change-Id: I95112cc260d3d254e7703513818b21e066d69978
Signed-off-by: Patrick Daly <pdaly@codeaurora.org>
Signed-off-by: Isaac J. Manjarres <isaacm@codeaurora.org>
