Changes in 5.10.47
module: limit enabling module.sig_enforce
Revert "drm/amdgpu/gfx9: fix the doorbell missing when in CGPG issue."
Revert "drm/amdgpu/gfx10: enlarge CP_MEC_DOORBELL_RANGE_UPPER to cover full doorbell."
drm: add a locked version of drm_is_current_master
drm/nouveau: wait for moving fence after pinning v2
drm/radeon: wait for moving fence after pinning
drm/amdgpu: wait for moving fence after pinning
ARM: 9081/1: fix gcc-10 thumb2-kernel regression
mmc: meson-gx: use memcpy_to/fromio for dram-access-quirk
MIPS: generic: Update node names to avoid unit addresses
arm64: Ignore any DMA offsets in the max_zone_phys() calculation
arm64: Force NO_BLOCK_MAPPINGS if crashkernel reservation is required
spi: spi-nxp-fspi: move the register operation after the clock enable
Revert "PCI: PM: Do not read power state in pci_enable_device_flags()"
drm/vc4: hdmi: Move the HSM clock enable to runtime_pm
drm/vc4: hdmi: Make sure the controller is powered in detect
x86/entry: Fix noinstr fail in __do_fast_syscall_32()
x86/xen: Fix noinstr fail in exc_xen_unknown_trap()
locking/lockdep: Improve noinstr vs errors
perf/x86/lbr: Remove cpuc->lbr_xsave allocation from atomic context
perf/x86/intel/lbr: Zero the xstate buffer on allocation
dmaengine: zynqmp_dma: Fix PM reference leak in zynqmp_dma_alloc_chan_resourc()
dmaengine: stm32-mdma: fix PM reference leak in stm32_mdma_alloc_chan_resourc()
dmaengine: xilinx: dpdma: Add missing dependencies to Kconfig
dmaengine: xilinx: dpdma: Limit descriptor IDs to 16 bits
mac80211: remove warning in ieee80211_get_sband()
mac80211_hwsim: drop pending frames on stop
cfg80211: call cfg80211_leave_ocb when switching away from OCB
dmaengine: rcar-dmac: Fix PM reference leak in rcar_dmac_probe()
dmaengine: mediatek: free the proper desc in desc_free handler
dmaengine: mediatek: do not issue a new desc if one is still current
dmaengine: mediatek: use GFP_NOWAIT instead of GFP_ATOMIC in prep_dma
net: ipv4: Remove unneed BUG() function
mac80211: drop multicast fragments
net: ethtool: clear heap allocations for ethtool function
inet: annotate data race in inet_send_prepare() and inet_dgram_connect()
ping: Check return value of function 'ping_queue_rcv_skb'
net: annotate data race in sock_error()
inet: annotate date races around sk->sk_txhash
net/packet: annotate data race in packet_sendmsg()
net: phy: dp83867: perform soft reset and retain established link
riscv32: Use medany C model for modules
net: caif: fix memory leak in ldisc_open
net/packet: annotate accesses to po->bind
net/packet: annotate accesses to po->ifindex
r8152: Avoid memcpy() over-reading of ETH_SS_STATS
sh_eth: Avoid memcpy() over-reading of ETH_SS_STATS
r8169: Avoid memcpy() over-reading of ETH_SS_STATS
KVM: selftests: Fix kvm_check_cap() assertion
net: qed: Fix memcpy() overflow of qed_dcbx_params()
mac80211: reset profile_periodicity/ema_ap
mac80211: handle various extensible elements correctly
recordmcount: Correct st_shndx handling
PCI: Add AMD RS690 quirk to enable 64-bit DMA
net: ll_temac: Add memory-barriers for TX BD access
net: ll_temac: Avoid ndo_start_xmit returning NETDEV_TX_BUSY
perf/x86: Track pmu in per-CPU cpu_hw_events
pinctrl: stm32: fix the reported number of GPIO lines per bank
i2c: i801: Ensure that SMBHSTSTS_INUSE_STS is cleared when leaving i801_access
gpiolib: cdev: zero padding during conversion to gpioline_info_changed
scsi: sd: Call sd_revalidate_disk() for ioctl(BLKRRPART)
nilfs2: fix memory leak in nilfs_sysfs_delete_device_group
s390/stack: fix possible register corruption with stack switch helper
KVM: do not allow mapping valid but non-reference-counted pages
i2c: robotfuzz-osif: fix control-request directions
ceph: must hold snap_rwsem when filling inode for async create
kthread_worker: split code for canceling the delayed work timer
kthread: prevent deadlock when kthread_mod_delayed_work() races with kthread_cancel_delayed_work_sync()
x86/fpu: Preserve supervisor states in sanitize_restored_user_xstate()
x86/fpu: Make init_fpstate correct with optimized XSAVE
mm: add VM_WARN_ON_ONCE_PAGE() macro
mm/rmap: remove unneeded semicolon in page_not_mapped()
mm/rmap: use page_not_mapped in try_to_unmap()
mm, thp: use head page in __migration_entry_wait()
mm/thp: fix __split_huge_pmd_locked() on shmem migration entry
mm/thp: make is_huge_zero_pmd() safe and quicker
mm/thp: try_to_unmap() use TTU_SYNC for safe splitting
mm/thp: fix vma_address() if virtual address below file offset
mm/thp: fix page_address_in_vma() on file THP tails
mm/thp: unmap_mapping_page() to fix THP truncate_cleanup_page()
mm: thp: replace DEBUG_VM BUG with VM_WARN when unmap fails for split
mm: page_vma_mapped_walk(): use page for pvmw->page
mm: page_vma_mapped_walk(): settle PageHuge on entry
mm: page_vma_mapped_walk(): use pmde for *pvmw->pmd
mm: page_vma_mapped_walk(): prettify PVMW_MIGRATION block
mm: page_vma_mapped_walk(): crossing page table boundary
mm: page_vma_mapped_walk(): add a level of indentation
mm: page_vma_mapped_walk(): use goto instead of while (1)
mm: page_vma_mapped_walk(): get vma_address_end() earlier
mm/thp: fix page_vma_mapped_walk() if THP mapped by ptes
mm/thp: another PVMW_SYNC fix in page_vma_mapped_walk()
mm, futex: fix shared futex pgoff on shmem huge page
KVM: SVM: Call SEV Guest Decommission if ASID binding fails
swiotlb: manipulate orig_addr when tlb_addr has offset
netfs: fix test for whether we can skip read when writing beyond EOF
Revert "drm: add a locked version of drm_is_current_master"
certs: Add EFI_CERT_X509_GUID support for dbx entries
certs: Move load_system_certificate_list to a common function
certs: Add ability to preload revocation certs
integrity: Load mokx variables into the blacklist keyring
Linux 5.10.47
Signed-off-by: Greg Kroah-Hartman <gregkh@google.com>
Change-Id: I68f731ad78a5db003c41093e4faf59f6f9f2e446
commit fe19bd3dae3d15d2fbfdb3de8839a6ea0fe94264 upstream.
If more than one futex is placed on a shmem huge page, it can happen
that waking the second wakes the first instead, and leaves the second
waiting: the key's shared.pgoff is wrong.
When 3.11 commit 13d60f4b6a ("futex: Take hugepages into account when
generating futex_key"), the only shared huge pages came from hugetlbfs,
and the code added to deal with its exceptional page->index was put into
hugetlb source. Then that was missed when 4.8 added shmem huge pages.
page_to_pgoff() is what others use for this nowadays: except that, as
currently written, it gives the right answer on hugetlbfs head, but
nonsense on hugetlbfs tails. Fix that by calling hugetlbfs-specific
hugetlb_basepage_index() on PageHuge tails as well as on head.
Yes, it's unconventional to declare hugetlb_basepage_index() there in
pagemap.h, rather than in hugetlb.h; but I do not expect anything but
page_to_pgoff() ever to need it.
[akpm@linux-foundation.org: give hugetlb_basepage_index() prototype the correct scope]
Link: https://lkml.kernel.org/r/b17d946b-d09-326e-b42a-52884c36df32@google.com
Fixes: 800d8c63b2 ("shmem: add huge pages support")
Reported-by: Neel Natu <neelnatu@google.com>
Signed-off-by: Hugh Dickins <hughd@google.com>
Reviewed-by: Matthew Wilcox (Oracle) <willy@infradead.org>
Acked-by: Thomas Gleixner <tglx@linutronix.de>
Cc: "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>
Cc: Zhang Yi <wetpzy@gmail.com>
Cc: Mel Gorman <mgorman@techsingularity.net>
Cc: Mike Kravetz <mike.kravetz@oracle.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Darren Hart <dvhart@infradead.org>
Cc: Davidlohr Bueso <dave@stgolabs.net>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Changes in 5.10.27
mm/memcg: rename mem_cgroup_split_huge_fixup to split_page_memcg and add nr_pages argument
mm/memcg: set memcg when splitting page
mt76: fix tx skb error handling in mt76_dma_tx_queue_skb
net: stmmac: fix dma physical address of descriptor when display ring
net: fec: ptp: avoid register access when ipg clock is disabled
powerpc/4xx: Fix build errors from mfdcr()
atm: eni: dont release is never initialized
atm: lanai: dont run lanai_dev_close if not open
Revert "r8152: adjust the settings about MAC clock speed down for RTL8153"
ALSA: hda: ignore invalid NHLT table
ixgbe: Fix memleak in ixgbe_configure_clsu32
scsi: ufs: ufs-qcom: Disable interrupt in reset path
blk-cgroup: Fix the recursive blkg rwstat
net: tehuti: fix error return code in bdx_probe()
net: intel: iavf: fix error return code of iavf_init_get_resources()
sun/niu: fix wrong RXMAC_BC_FRM_CNT_COUNT count
gianfar: fix jumbo packets+napi+rx overrun crash
cifs: ask for more credit on async read/write code paths
gfs2: fix use-after-free in trans_drain
cpufreq: blacklist Arm Vexpress platforms in cpufreq-dt-platdev
gpiolib: acpi: Add missing IRQF_ONESHOT
nfs: fix PNFS_FLEXFILE_LAYOUT Kconfig default
NFS: Correct size calculation for create reply length
net: hisilicon: hns: fix error return code of hns_nic_clear_all_rx_fetch()
net: wan: fix error return code of uhdlc_init()
net: davicom: Use platform_get_irq_optional()
net: enetc: set MAC RX FIFO to recommended value
atm: uPD98402: fix incorrect allocation
atm: idt77252: fix null-ptr-dereference
cifs: change noisy error message to FYI
irqchip/ingenic: Add support for the JZ4760
kbuild: add image_name to no-sync-config-targets
kbuild: dummy-tools: fix inverted tests for gcc
umem: fix error return code in mm_pci_probe()
sparc64: Fix opcode filtering in handling of no fault loads
habanalabs: Call put_pid() when releasing control device
staging: rtl8192e: fix kconfig dependency on CRYPTO
u64_stats,lockdep: Fix u64_stats_init() vs lockdep
kselftest: arm64: Fix exit code of sve-ptrace
regulator: qcom-rpmh: Correct the pmic5_hfsmps515 buck
block: Fix REQ_OP_ZONE_RESET_ALL handling
drm/amd/display: Revert dram_clock_change_latency for DCN2.1
drm/amdgpu: fb BO should be ttm_bo_type_device
drm/radeon: fix AGP dependency
nvme: simplify error logic in nvme_validate_ns()
nvme: add NVME_REQ_CANCELLED flag in nvme_cancel_request()
nvme-fc: set NVME_REQ_CANCELLED in nvme_fc_terminate_exchange()
nvme-fc: return NVME_SC_HOST_ABORTED_CMD when a command has been aborted
nvme-core: check ctrl css before setting up zns
nvme-rdma: Fix a use after free in nvmet_rdma_write_data_done
nvme-pci: add the DISABLE_WRITE_ZEROES quirk for a Samsung PM1725a
nfs: we don't support removing system.nfs4_acl
block: Suppress uevent for hidden device when removed
mm/fork: clear PASID for new mm
ia64: fix ia64_syscall_get_set_arguments() for break-based syscalls
ia64: fix ptrace(PTRACE_SYSCALL_INFO_EXIT) sign
static_call: Pull some static_call declarations to the type headers
static_call: Allow module use without exposing static_call_key
static_call: Fix the module key fixup
static_call: Fix static_call_set_init()
KVM: x86: Protect userspace MSR filter with SRCU, and set atomically-ish
btrfs: fix sleep while in non-sleep context during qgroup removal
selinux: don't log MAC_POLICY_LOAD record on failed policy load
selinux: fix variable scope issue in live sidtab conversion
netsec: restore phy power state after controller reset
platform/x86: intel-vbtn: Stop reporting SW_DOCK events
psample: Fix user API breakage
z3fold: prevent reclaim/free race for headless pages
squashfs: fix inode lookup sanity checks
squashfs: fix xattr id and id lookup sanity checks
hugetlb_cgroup: fix imbalanced css_get and css_put pair for shared mappings
kasan: fix per-page tags for non-page_alloc pages
gcov: fix clang-11+ support
ACPI: video: Add missing callback back for Sony VPCEH3U1E
ACPICA: Always create namespace nodes using acpi_ns_create_node()
arm64: stacktrace: don't trace arch_stack_walk()
arm64: dts: ls1046a: mark crypto engine dma coherent
arm64: dts: ls1012a: mark crypto engine dma coherent
arm64: dts: ls1043a: mark crypto engine dma coherent
ARM: dts: at91: sam9x60: fix mux-mask for PA7 so it can be set to A, B and C
ARM: dts: at91: sam9x60: fix mux-mask to match product's datasheet
ARM: dts: at91-sama5d27_som1: fix phy address to 7
integrity: double check iint_cache was initialized
drm/etnaviv: Use FOLL_FORCE for userptr
drm/amd/pm: workaround for audio noise issue
drm/amdgpu/display: restore AUX_DPHY_TX_CONTROL for DCN2.x
drm/amdgpu: Add additional Sienna Cichlid PCI ID
drm/i915: Fix the GT fence revocation runtime PM logic
dm verity: fix DM_VERITY_OPTS_MAX value
dm ioctl: fix out of bounds array access when no devices
bus: omap_l3_noc: mark l3 irqs as IRQF_NO_THREAD
ARM: OMAP2+: Fix smartreflex init regression after dropping legacy data
soc: ti: omap-prm: Fix occasional abort on reset deassert for dra7 iva
veth: Store queue_mapping independently of XDP prog presence
bpf: Change inode_storage's lookup_elem return value from NULL to -EBADF
libbpf: Fix INSTALL flag order
net/mlx5e: RX, Mind the MPWQE gaps when calculating offsets
net/mlx5e: When changing XDP program without reset, take refs for XSK RQs
net/mlx5e: Don't match on Geneve options in case option masks are all zero
ipv6: fix suspecious RCU usage warning
drop_monitor: Perform cleanup upon probe registration failure
macvlan: macvlan_count_rx() needs to be aware of preemption
net: sched: validate stab values
net: dsa: bcm_sf2: Qualify phydev->dev_flags based on port
igc: reinit_locked() should be called with rtnl_lock
igc: Fix Pause Frame Advertising
igc: Fix Supported Pause Frame Link Setting
igc: Fix igc_ptp_rx_pktstamp()
e1000e: add rtnl_lock() to e1000_reset_task
e1000e: Fix error handling in e1000_set_d0_lplu_state_82571
net/qlcnic: Fix a use after free in qlcnic_83xx_get_minidump_template
net: phy: broadcom: Add power down exit reset state delay
ftgmac100: Restart MAC HW once
clk: qcom: gcc-sc7180: Use floor ops for the correct sdcc1 clk
net: ipa: terminate message handler arrays
net: qrtr: fix a kernel-infoleak in qrtr_recvmsg()
flow_dissector: fix byteorder of dissected ICMP ID
selftests/bpf: Set gopt opt_class to 0 if get tunnel opt failed
netfilter: ctnetlink: fix dump of the expect mask attribute
net: hdlc_x25: Prevent racing between "x25_close" and "x25_xmit"/"x25_rx"
net: phylink: Fix phylink_err() function name error in phylink_major_config
tipc: better validate user input in tipc_nl_retrieve_key()
tcp: relookup sock for RST+ACK packets handled by obsolete req sock
can: isotp: isotp_setsockopt(): only allow to set low level TX flags for CAN-FD
can: isotp: TX-path: ensure that CAN frame flags are initialized
can: peak_usb: add forgotten supported devices
can: flexcan: flexcan_chip_freeze(): fix chip freeze for missing bitrate
can: kvaser_pciefd: Always disable bus load reporting
can: c_can_pci: c_can_pci_remove(): fix use-after-free
can: c_can: move runtime PM enable/disable to c_can_platform
can: m_can: m_can_do_rx_poll(): fix extraneous msg loss warning
can: m_can: m_can_rx_peripheral(): fix RX being blocked by errors
mac80211: fix rate mask reset
mac80211: Allow HE operation to be longer than expected.
selftests/net: fix warnings on reuseaddr_ports_exhausted
nfp: flower: fix unsupported pre_tunnel flows
nfp: flower: add ipv6 bit to pre_tunnel control message
nfp: flower: fix pre_tun mask id allocation
ftrace: Fix modify_ftrace_direct.
drm/msm/dsi: fix check-before-set in the 7nm dsi_pll code
ionic: linearize tso skb with too many frags
net/sched: cls_flower: fix only mask bit check in the validate_ct_state
netfilter: nftables: report EOPNOTSUPP on unsupported flowtable flags
netfilter: nftables: allow to update flowtable flags
netfilter: flowtable: Make sure GC works periodically in idle system
libbpf: Fix error path in bpf_object__elf_init()
libbpf: Use SOCK_CLOEXEC when opening the netlink socket
ARM: dts: imx6ull: fix ubi filesystem mount failed
ipv6: weaken the v4mapped source check
octeontx2-af: Formatting debugfs entry rsrc_alloc.
octeontx2-af: Modify default KEX profile to extract TX packet fields
octeontx2-af: Remove TOS field from MKEX TX
octeontx2-af: Fix irq free in rvu teardown
octeontx2-pf: Clear RSS enable flag on interace down
octeontx2-af: fix infinite loop in unmapping NPC counter
net: check all name nodes in __dev_alloc_name
net: cdc-phonet: fix data-interface release on probe failure
igb: check timestamp validity
r8152: limit the RX buffer size of RTL8153A for USB 2.0
net: stmmac: dwmac-sun8i: Provide TX and RX fifo sizes
selinux: vsock: Set SID for socket returned by accept()
selftests: forwarding: vxlan_bridge_1d: Fix vxlan ecn decapsulate value
libbpf: Fix BTF dump of pointer-to-array-of-struct
bpf: Fix umd memory leak in copy_process()
can: isotp: tx-path: zero initialize outgoing CAN frames
drm/msm: fix shutdown hook in case GPU components failed to bind
drm/msm: Fix suspend/resume on i.MX5
arm64: kdump: update ppos when reading elfcorehdr
PM: runtime: Defer suspending suppliers
net/mlx5: Add back multicast stats for uplink representor
net/mlx5e: Allow to match on MPLS parameters only for MPLS over UDP
net/mlx5e: Offload tuple rewrite for non-CT flows
net/mlx5e: Fix error path for ethtool set-priv-flag
PM: EM: postpone creating the debugfs dir till fs_initcall
net: bridge: don't notify switchdev for local FDB addresses
octeontx2-af: Fix memory leak of object buf
xen/x86: make XEN_BALLOON_MEMORY_HOTPLUG_LIMIT depend on MEMORY_HOTPLUG
RDMA/cxgb4: Fix adapter LE hash errors while destroying ipv6 listening server
bpf: Don't do bpf_cgroup_storage_set() for kuprobe/tp programs
net: Consolidate common blackhole dst ops
net, bpf: Fix ip6ip6 crash with collect_md populated skbs
igb: avoid premature Rx buffer reuse
net: axienet: Properly handle PCS/PMA PHY for 1000BaseX mode
net: axienet: Fix probe error cleanup
net: phy: introduce phydev->port
net: phy: broadcom: Avoid forward for bcm54xx_config_clock_delay()
net: phy: broadcom: Set proper 1000BaseX/SGMII interface mode for BCM54616S
net: phy: broadcom: Fix RGMII delays for BCM50160 and BCM50610M
Revert "netfilter: x_tables: Switch synchronization to RCU"
netfilter: x_tables: Use correct memory barriers.
dm table: Fix zoned model check and zone sectors check
mm/mmu_notifiers: ensure range_end() is paired with range_start()
Revert "netfilter: x_tables: Update remaining dereference to RCU"
ACPI: scan: Rearrange memory allocation in acpi_device_add()
ACPI: scan: Use unique number for instance_no
perf auxtrace: Fix auxtrace queue conflict
perf synthetic events: Avoid write of uninitialized memory when generating PERF_RECORD_MMAP* records
io_uring: fix provide_buffers sign extension
block: recalculate segment count for multi-segment discards correctly
scsi: Revert "qla2xxx: Make sure that aborted commands are freed"
scsi: qedi: Fix error return code of qedi_alloc_global_queues()
scsi: mpt3sas: Fix error return code of mpt3sas_base_attach()
smb3: fix cached file size problems in duplicate extents (reflink)
cifs: Adjust key sizes and key generation routines for AES256 encryption
locking/mutex: Fix non debug version of mutex_lock_io_nested()
x86/mem_encrypt: Correct physical address calculation in __set_clr_pte_enc()
mm/memcg: fix 5.10 backport of splitting page memcg
fs/cachefiles: Remove wait_bit_key layout dependency
ch_ktls: fix enum-conversion warning
can: dev: Move device back to init netns on owning netns delete
r8169: fix DMA being used after buffer free if WoL is enabled
net: dsa: b53: VLAN filtering is global to all users
mac80211: fix double free in ibss_leave
ext4: add reclaim checks to xattr code
fs/ext4: fix integer overflow in s_log_groups_per_flex
Revert "xen: fix p2m size in dom0 for disabled memory hotplug case"
Revert "net: bonding: fix error return code of bond_neigh_init()"
nvme: fix the nsid value to print in nvme_validate_or_alloc_ns
can: peak_usb: Revert "can: peak_usb: add forgotten supported devices"
xen-blkback: don't leak persistent grants from xen_blkbk_map()
Linux 5.10.27
Signed-off-by: Greg Kroah-Hartman <gregkh@google.com>
Change-Id: I7eafe976fd6bf33db6db4adb8ebf2ff087294a23
Provide infrastructure to do a speculative fault (not holding
mmap_sem).
The not holding of mmap_sem means we can race against VMA
change/removal and page-table destruction. We use the SRCU VMA freeing
to keep the VMA around. We use the VMA seqcount to detect change
(including umapping / page-table deletion) and we use gup_fast() style
page-table walking to deal with page-table races.
Once we've obtained the page and are ready to update the PTE, we
validate if the state we started the fault with is still valid, if
not, we'll fail the fault with VM_FAULT_RETRY, otherwise we update the
PTE and we're done.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
[Manage the newly introduced pte_spinlock() for speculative page
fault to fail if the VMA is touched in our back]
[Rename vma_is_dead() to vma_has_changed() and declare it here]
[Fetch p4d and pud]
[Set vmd.sequence in __handle_mm_fault()]
[Abort speculative path when handle_userfault() has to be called]
[Add additional VMA's flags checks in handle_speculative_fault()]
[Clear FAULT_FLAG_ALLOW_RETRY in handle_speculative_fault()]
[Don't set vmf->pte and vmf->ptl if pte_map_lock() failed]
[Remove warning comment about waiting for !seq&1 since we don't want
to wait]
[Remove warning about no huge page support, mention it explictly]
[Don't call do_fault() in the speculative path as __do_fault() calls
vma->vm_ops->fault() which may want to release mmap_sem]
[Only vm_fault pointer argument for vma_has_changed()]
[Fix check against huge page, calling pmd_trans_huge()]
[Use READ_ONCE() when reading VMA's fields in the speculative path]
[Explicitly check for __HAVE_ARCH_PTE_SPECIAL as we can't support for
processing done in vm_normal_page()]
[Check that vma->anon_vma is already set when starting the speculative
path]
[Check for memory policy as we can't support MPOL_INTERLEAVE case due to
the processing done in mpol_misplaced()]
[Don't support VMA growing up or down]
[Move check on vm_sequence just before calling handle_pte_fault()]
[Don't build SPF services if !CONFIG_SPECULATIVE_PAGE_FAULT]
[Add mem cgroup oom check]
[Use READ_ONCE to access p*d entries]
[Replace deprecated ACCESS_ONCE() by READ_ONCE() in vma_has_changed()]
[Don't fetch pte again in handle_pte_fault() when running the speculative
path]
[Check PMD against concurrent collapsing operation]
[Try spin lock the pte during the speculative path to avoid deadlock with
other CPU's invalidating the TLB and requiring this CPU to catch the
inter processor's interrupt]
[Move define of FAULT_FLAG_SPECULATIVE here]
[Introduce __handle_speculative_fault() and add a check against
mm->mm_users in handle_speculative_fault() defined in mm.h]
Signed-off-by: Laurent Dufour <ldufour@linux.vnet.ibm.com>
Link: https://lore.kernel.org/lkml/1523975611-15978-19-git-send-email-ldufour@linux.vnet.ibm.com/
Bug: 161210518
Change-Id: I6a29e6edd9779bd34a9f7f4f6034e041a8487f30
Signed-off-by: Vinayak Menon <vinmenon@codeaurora.org>
Signed-off-by: Charan Teja Reddy <charante@codeaurora.org>
Create a vendor hook inside of gfp_zone() to modify which allocations
get to enter ZONE_MOVABLE, by zeroing out __GFP_HIGHMEM inside of the
trace hook based on certain conditions.
Separately, create separate trace hooks in the swap-in and readahead
paths to affect the behavior of the tracehook in gfp_zone().
Bug: 158645321
Change-Id: I4a4f0b724267ee120a1e5661f6da5d43d7ef6fc6
Signed-off-by: Chris Goldsworthy <cgoldswo@codeaurora.org>
trace_sched_blocked_trace in CFS is really useful for debugging via
trace because it tell where the process was stuck on callstack.
For example,
<...>-6143 ( 6136) [005] d..2 50.278987: sched_blocked_reason: pid=6136 iowait=0 caller=SyS_mprotect+0x88/0x208
<...>-6136 ( 6136) [005] d..2 50.278990: sched_blocked_reason: pid=6142 iowait=0 caller=do_page_fault+0x1f4/0x3b0
<...>-6142 ( 6136) [006] d..2 50.278996: sched_blocked_reason: pid=6144 iowait=0 caller=SyS_prctl+0x52c/0xb58
<...>-6144 ( 6136) [006] d..2 50.279007: sched_blocked_reason: pid=6136 iowait=0 caller=vm_mmap_pgoff+0x74/0x104
However, sometime it gives pointless information like this.
RenderThread-2322 ( 1805) [006] d.s3 50.319046: sched_blocked_reason: pid=6136 iowait=1 caller=__lock_page_killable+0x17c/0x220
logd.writer-594 ( 587) [002] d.s3 50.334011: sched_blocked_reason: pid=6126 iowait=1 caller=wait_on_page_bit+0x194/0x208
kworker/u16:13-333 ( 333) [007] d.s4 50.343161: sched_blocked_reason: pid=6136 iowait=1 caller=__lock_page_killable+0x17c/0x220
Such wait_on_page_bit, __lock_page_killable are pointless because it doesn't
carry on higher information to identify the callstack.
The reason is page_lock and waitqueue are special synchronization method unlike
other normal locks(mutex, spinlock).
Let's mark them as "__sched" so get_wchan which used in trace_sched_blocked_trace
could detect it and skip them. It will produce more meaningful callstack
function like this.
<...>-2867 ( 1068) [002] d.h4 124.209701: sched_blocked_reason: pid=329 iowait=0 caller=worker_thread+0x378/0x470
<...>-2867 ( 1068) [002] d.s3 124.209763: sched_blocked_reason: pid=8454 iowait=1 caller=__filemap_fdatawait_range+0xa0/0x104
<...>-2867 ( 1068) [002] d.s4 124.209803: sched_blocked_reason: pid=869 iowait=0 caller=worker_thread+0x378/0x470
ScreenDecoratio-2364 ( 1867) [002] d.s3 124.209973: sched_blocked_reason: pid=8454 iowait=1 caller=f2fs_wait_on_page_writeback+0x84/0xcc
ScreenDecoratio-2364 ( 1867) [002] d.s4 124.209986: sched_blocked_reason: pid=869 iowait=0 caller=worker_thread+0x378/0x470
<...>-329 ( 329) [000] d..3 124.210435: sched_blocked_reason: pid=538 iowait=0 caller=worker_thread+0x378/0x470
kworker/u16:13-538 ( 538) [007] d..3 124.210450: sched_blocked_reason: pid=6 iowait=0 caller=worker_thread+0x378/0x470
Test: build pass and boot to home.
Bug: 144961676
Bug: 144713689
Bug: 172212772
Signed-off-by: Minchan Kim <minchan@google.com>
Signed-off-by: Jimmy Shiu <jimmyshiu@google.com>
Change-Id: I9c738802a16941ca767dcc37ae4463070b3fabf4
(cherry picked from commit 1e4de875d9e0cfaccf5131bcc709ae8646cdc168)
Signed-off-by: Will McVicker <willmcvicker@google.com>
Patch series "Readahead patches for 5.9/5.10".
These are infrastructure for both the THP patchset and for the fscache
rewrite,
For both pieces of infrastructure being build on top of this patchset, we
want the ractl to be available higher in the call-stack.
For David's work, he wants to add the 'critical page' to the ractl so that
he knows which page NEEDS to be brought in from storage, and which ones
are nice-to-have. We might want something similar in block storage too.
It used to be simple -- the first page was the critical one, but then mmap
added fault-around and so for that usecase, the middle page is the
critical one. Anyway, I don't have any code to show that yet, we just
know that the lowest point in the callchain where we have that information
is do_sync_mmap_readahead() and so the ractl needs to start its life
there.
For THP, we havew the code that needs it. It's actually the apex patch to
the series; the one which finally starts to allocate THPs and present them
to consenting filesystems:
798bcf30ab
This patch (of 8):
Allow for a more concise definition of a struct readahead_control.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Cc: Eric Biggers <ebiggers@google.com>
Cc: David Howells <dhowells@redhat.com>
Link: https://lkml.kernel.org/r/20200903140844.14194-1-willy@infradead.org
Link: https://lkml.kernel.org/r/20200903140844.14194-3-willy@infradead.org
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Pull iomap updates from Darrick Wong:
"There's not a lot of new stuff going on here -- a little bit of code
refactoring to make iomap workable with btrfs' fsync locking model,
cleanups in preparation for adding THP support for filesystems, and
fixing a data corruption issue for blocksize < pagesize filesystems.
Summary:
- Don't WARN_ON weird states that unprivileged users can create.
- Don't invalidate page cache when direct writes want to fall back to
buffered.
- Fix some problems when readahead ios fail.
- Fix a problem where inline data pages weren't getting flushed
during an unshare operation.
- Rework iomap to support arbitrarily many blocks per page in
preparation to support THP for the page cache.
- Fix a bug in the blocksize < pagesize buffered io path where we
could fail to initialize the many-blocks-per-page uptodate bitmap
correctly when the backing page is actually up to date. This could
cause us to forget to write out dirty pages.
- Split out the generic_write_sync at the end of the directio write
path so that btrfs can drop the inode lock before sync'ing the
file.
- Call inode_dio_end before trying to sync the file after a O_DSYNC
direct write (instead of afterwards) to match the behavior of the
old directio code"
* tag 'iomap-5.10-merge-4' of git://git.kernel.org/pub/scm/fs/xfs/xfs-linux:
iomap: Call inode_dio_end() before generic_write_sync()
iomap: Allow filesystem to call iomap_dio_complete without i_rwsem
iomap: Set all uptodate bits for an Uptodate page
iomap: Change calling convention for zeroing
iomap: Convert iomap_write_end types
iomap: Convert write_count to write_bytes_pending
iomap: Convert read_count to read_bytes_pending
iomap: Support arbitrarily many blocks per page
iomap: Use bitmap ops to set uptodate bits
iomap: Use kzalloc to allocate iomap_page
fs: Introduce i_blocks_per_page
iomap: Fix misplaced page flushing
iomap: Use round_down/round_up macros in __iomap_write_begin
iomap: Mark read blocks uptodate in write_begin
iomap: Clear page error before beginning a write
iomap: Fix direct I/O write consistency check
iomap: fix WARN_ON_ONCE() from unprivileged users
This helper is useful for both THPs and for supporting block size larger
than page size. Convert all users that I could find (we have a few
different ways of writing this idiom, and I may have missed some).
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Darrick J. Wong <darrick.wong@oracle.com>
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Acked-by: Dave Kleikamp <dave.kleikamp@oracle.com>
The thp prefix is more frequently used than hpage and we should be
consistent between the various functions.
[akpm@linux-foundation.org: fix mm/migrate.c]
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: William Kucharski <william.kucharski@oracle.com>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Cc: Mike Kravetz <mike.kravetz@oracle.com>
Cc: David Hildenbrand <david@redhat.com>
Cc: "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>
Link: http://lkml.kernel.org/r/20200629151959.15779-6-willy@infradead.org
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Pull io_uring updates from Jens Axboe:
"Lots of cleanups in here, hardening the code and/or making it easier
to read and fixing bugs, but a core feature/change too adding support
for real async buffered reads. With the latter in place, we just need
buffered write async support and we're done relying on kthreads for
the fast path. In detail:
- Cleanup how memory accounting is done on ring setup/free (Bijan)
- sq array offset calculation fixup (Dmitry)
- Consistently handle blocking off O_DIRECT submission path (me)
- Support proper async buffered reads, instead of relying on kthread
offload for that. This uses the page waitqueue to drive retries
from task_work, like we handle poll based retry. (me)
- IO completion optimizations (me)
- Fix race with accounting and ring fd install (me)
- Support EPOLLEXCLUSIVE (Jiufei)
- Get rid of the io_kiocb unionizing, made possible by shrinking
other bits (Pavel)
- Completion side cleanups (Pavel)
- Cleanup REQ_F_ flags handling, and kill off many of them (Pavel)
- Request environment grabbing cleanups (Pavel)
- File and socket read/write cleanups (Pavel)
- Improve kiocb_set_rw_flags() (Pavel)
- Tons of fixes and cleanups (Pavel)
- IORING_SQ_NEED_WAKEUP clear fix (Xiaoguang)"
* tag 'for-5.9/io_uring-20200802' of git://git.kernel.dk/linux-block: (127 commits)
io_uring: flip if handling after io_setup_async_rw
fs: optimise kiocb_set_rw_flags()
io_uring: don't touch 'ctx' after installing file descriptor
io_uring: get rid of atomic FAA for cq_timeouts
io_uring: consolidate *_check_overflow accounting
io_uring: fix stalled deferred requests
io_uring: fix racy overflow count reporting
io_uring: deduplicate __io_complete_rw()
io_uring: de-unionise io_kiocb
io-wq: update hash bits
io_uring: fix missing io_queue_linked_timeout()
io_uring: mark ->work uninitialised after cleanup
io_uring: deduplicate io_grab_files() calls
io_uring: don't do opcode prep twice
io_uring: clear IORING_SQ_NEED_WAKEUP after executing task works
io_uring: batch put_task_struct()
tasks: add put_task_struct_many()
io_uring: return locked and pinned page accounting
io_uring: don't miscount pinned memory
io_uring: don't open-code recv kbuf managment
...
Checks if the file supports it, and initializes the values that we need.
Caller passes in 'data' pointer, if any, and the callback function to
be used.
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Normally waiting for a page to become unlocked, or locking the page,
requires waiting for IO to complete. Add support for lock_page_async()
and wait_on_page_locked_async(), which are callback based instead. This
allows a caller to get notified when a page becomes unlocked, rather
than wait for it.
We add a new iocb field, ki_waitq, to pass in the necessary data for this
to happen. We can unionize this with ki_cookie, since that is only used
for polled IO. Polled IO can never co-exist with async callbacks, as it is
(by definition) polled completions. struct wait_page_key is made public,
and we define struct wait_page_async as the interface between the caller
and the core.
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
No functional changes in this patch, just in preparation for allowing
more callers.
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Patch series "Change readahead API", v11.
This series adds a readahead address_space operation to replace the
readpages operation. The key difference is that pages are added to the
page cache as they are allocated (and then looked up by the filesystem)
instead of passing them on a list to the readpages operation and having
the filesystem add them to the page cache. It's a net reduction in code
for each implementation, more efficient than walking a list, and solves
the direct-write vs buffered-read problem reported by yu kuai at
http://lkml.kernel.org/r/20200116063601.39201-1-yukuai3@huawei.com
The only unconverted filesystems are those which use fscache. Their
conversion is pending Dave Howells' rewrite which will make the
conversion substantially easier. This should be completed by the end of
the year.
I want to thank the reviewers/testers; Dave Chinner, John Hubbard, Eric
Biggers, Johannes Thumshirn, Dave Sterba, Zi Yan, Christoph Hellwig and
Miklos Szeredi have done a marvellous job of providing constructive
criticism.
These patches pass an xfstests run on ext4, xfs & btrfs with no
regressions that I can tell (some of the tests seem a little flaky
before and remain flaky afterwards).
This patch (of 25):
The readahead code is part of the page cache so should be found in the
pagemap.h file. force_page_cache_readahead is only used within mm, so
move it to mm/internal.h instead. Remove the parameter names where they
add no value, and rename the ones which were actively misleading.
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: John Hubbard <jhubbard@nvidia.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: William Kucharski <william.kucharski@oracle.com>
Reviewed-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
Cc: Chao Yu <yuchao0@huawei.com>
Cc: Cong Wang <xiyou.wangcong@gmail.com>
Cc: Darrick J. Wong <darrick.wong@oracle.com>
Cc: Dave Chinner <dchinner@redhat.com>
Cc: Eric Biggers <ebiggers@google.com>
Cc: Gao Xiang <gaoxiang25@huawei.com>
Cc: Jaegeuk Kim <jaegeuk@kernel.org>
Cc: Joseph Qi <joseph.qi@linux.alibaba.com>
Cc: Junxiao Bi <junxiao.bi@oracle.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Zi Yan <ziy@nvidia.com>
Cc: Miklos Szeredi <mszeredi@redhat.com>
Link: http://lkml.kernel.org/r/20200414150233.24495-1-willy@infradead.org
Link: http://lkml.kernel.org/r/20200414150233.24495-2-willy@infradead.org
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Patch series "vfs: have syncfs() return error when there are writeback
errors", v6.
Currently, syncfs does not return errors when one of the inodes fails to
be written back. It will return errors based on the legacy AS_EIO and
AS_ENOSPC flags when syncing out the block device fails, but that's not
particularly helpful for filesystems that aren't backed by a blockdev.
It's also possible for a stray sync to lose those errors.
The basic idea in this set is to track writeback errors at the
superblock level, so that we can quickly and easily check whether
something bad happened without having to fsync each file individually.
syncfs is then changed to reliably report writeback errors after they
occur, much in the same fashion as fsync does now.
This patch (of 2):
Usually we suggest that applications call fsync when they want to ensure
that all data written to the file has made it to the backing store, but
that can be inefficient when there are a lot of open files.
Calling syncfs on the filesystem can be more efficient in some
situations, but the error reporting doesn't currently work the way most
people expect. If a single inode on a filesystem reports a writeback
error, syncfs won't necessarily return an error. syncfs only returns an
error if __sync_blockdev fails, and on some filesystems that's a no-op.
It would be better if syncfs reported an error if there were any
writeback failures. Then applications could call syncfs to see if there
are any errors on any open files, and could then call fsync on all of
the other descriptors to figure out which one failed.
This patch adds a new errseq_t to struct super_block, and has
mapping_set_error also record writeback errors there.
To report those errors, we also need to keep an errseq_t in struct file
to act as a cursor. This patch adds a dedicated field for that purpose,
which slots nicely into 4 bytes of padding at the end of struct file on
x86_64.
An earlier version of this patch used an O_PATH file descriptor to cue
the kernel that the open file should track the superblock error and not
the inode's writeback error.
I think that API is just too weird though. This is simpler and should
make syncfs error reporting "just work" even if someone is multiplexing
fsync and syncfs on the same fds.
Signed-off-by: Jeff Layton <jlayton@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Jan Kara <jack@suse.cz>
Cc: Andres Freund <andres@anarazel.de>
Cc: Matthew Wilcox <willy@infradead.org>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Cc: Christoph Hellwig <hch@infradead.org>
Cc: Dave Chinner <david@fromorbit.com>
Cc: David Howells <dhowells@redhat.com>
Link: http://lkml.kernel.org/r/20200428135155.19223-1-jlayton@kernel.org
Link: http://lkml.kernel.org/r/20200428135155.19223-2-jlayton@kernel.org
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
When backporting commit 9c4e6b1a70 ("mm, mlock, vmscan: no more skipping
pagevecs") to our 4.9 kernel, our test bench noticed around 10% down with
a couple of vm-scalability's test cases (lru-file-readonce,
lru-file-readtwice and lru-file-mmap-read). I didn't see that much down
on my VM (32c-64g-2nodes). It might be caused by the test configuration,
which is 32c-256g with NUMA disabled and the tests were run in root memcg,
so the tests actually stress only one inactive and active lru. It sounds
not very usual in mordern production environment.
That commit did two major changes:
1. Call page_evictable()
2. Use smp_mb to force the PG_lru set visible
It looks they contribute the most overhead. The page_evictable() is a
function which does function prologue and epilogue, and that was used by
page reclaim path only. However, lru add is a very hot path, so it sounds
better to make it inline. However, it calls page_mapping() which is not
inlined either, but the disassemble shows it doesn't do push and pop
operations and it sounds not very straightforward to inline it.
Other than this, it sounds smp_mb() is not necessary for x86 since
SetPageLRU is atomic which enforces memory barrier already, replace it
with smp_mb__after_atomic() in the following patch.
With the two fixes applied, the tests can get back around 5% on that test
bench and get back normal on my VM. Since the test bench configuration is
not that usual and I also saw around 6% up on the latest upstream, so it
sounds good enough IMHO.
The below is test data (lru-file-readtwice throughput) against the v5.6-rc4:
mainline w/ inline fix
150MB 154MB
With this patch the throughput gets 2.67% up. The data with using
smp_mb__after_atomic() is showed in the following patch.
Shakeel Butt did the below test:
On a real machine with limiting the 'dd' on a single node and reading 100
GiB sparse file (less than a single node). Just ran a single instance to
not cause the lru lock contention. The cmdline used is "dd if=file-100GiB
of=/dev/null bs=4k". Ran the cmd 10 times with drop_caches in between and
measured the time it took.
Without patch: 56.64143 +- 0.672 sec
With patches: 56.10 +- 0.21 sec
[akpm@linux-foundation.org: move page_evictable() to internal.h]
Fixes: 9c4e6b1a70 ("mm, mlock, vmscan: no more skipping pagevecs")
Signed-off-by: Yang Shi <yang.shi@linux.alibaba.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Tested-by: Shakeel Butt <shakeelb@google.com>
Reviewed-by: Shakeel Butt <shakeelb@google.com>
Reviewed-by: Matthew Wilcox (Oracle) <willy@infradead.org>
Acked-by: Vlastimil Babka <vbabka@suse.cz>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Link: http://lkml.kernel.org/r/1584500541-46817-1-git-send-email-yang.shi@linux.alibaba.com
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
The check in block_page_mkwrite that is meant to determine whether an
offset is within the inode size is off by one. This bug has been copied
into iomap_page_mkwrite and several filesystems (ubifs, ext4, f2fs,
ceph).
Fix that by introducing a new page_mkwrite_check_truncate helper that
checks for truncate and computes the bytes in the page up to EOF. Use
the helper in iomap.
NOTE from Darrick: The original patch fixed a number of filesystems, but
then there were merge conflicts with the f2fs for-next tree; a
subsequent re-submission of the patch had different btrfs changes with
no explanation; and Christoph complained that each per-fs fix should be
a separate patch. In my view that's too much risk to take on, so I
decided to drop all the hunks except for iomap, since I've actually QA'd
XFS.
Signed-off-by: Andreas Gruenbacher <agruenba@redhat.com>
Reviewed-by: Darrick J. Wong <darrick.wong@oracle.com>
[darrick: drop everything but the iomap parts]
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Patch series "drop the mmap_sem when doing IO in the fault path", v6.
Now that we have proper isolation in place with cgroups2 we have started
going through and fixing the various priority inversions. Most are all
gone now, but this one is sort of weird since it's not necessarily a
priority inversion that happens within the kernel, but rather because of
something userspace does.
We have giant applications that we want to protect, and parts of these
giant applications do things like watch the system state to determine how
healthy the box is for load balancing and such. This involves running
'ps' or other such utilities. These utilities will often walk
/proc/<pid>/whatever, and these files can sometimes need to
down_read(&task->mmap_sem). Not usually a big deal, but we noticed when
we are stress testing that sometimes our protected application has latency
spikes trying to get the mmap_sem for tasks that are in lower priority
cgroups.
This is because any down_write() on a semaphore essentially turns it into
a mutex, so even if we currently have it held for reading, any new readers
will not be allowed on to keep from starving the writer. This is fine,
except a lower priority task could be stuck doing IO because it has been
throttled to the point that its IO is taking much longer than normal. But
because a higher priority group depends on this completing it is now stuck
behind lower priority work.
In order to avoid this particular priority inversion we want to use the
existing retry mechanism to stop from holding the mmap_sem at all if we
are going to do IO. This already exists in the read case sort of, but
needed to be extended for more than just grabbing the page lock. With
io.latency we throttle at submit_bio() time, so the readahead stuff can
block and even page_cache_read can block, so all these paths need to have
the mmap_sem dropped.
The other big thing is ->page_mkwrite. btrfs is particularly shitty here
because we have to reserve space for the dirty page, which can be a very
expensive operation. We use the same retry method as the read path, and
simply cache the page and verify the page is still setup properly the next
pass through ->page_mkwrite().
I've tested these patches with xfstests and there are no regressions.
This patch (of 3):
If we do not have a page at filemap_fault time we'll do this weird forced
page_cache_read thing to populate the page, and then drop it again and
loop around and find it. This makes for 2 ways we can read a page in
filemap_fault, and it's not really needed. Instead add a FGP_FOR_MMAP
flag so that pagecache_get_page() will return a unlocked page that's in
pagecache. Then use the normal page locking and readpage logic already in
filemap_fault. This simplifies the no page in page cache case
significantly.
[akpm@linux-foundation.org: fix comment text]
[josef@toxicpanda.com: don't unlock null page in FGP_FOR_MMAP case]
Link: http://lkml.kernel.org/r/20190312201742.22935-1-josef@toxicpanda.com
Link: http://lkml.kernel.org/r/20181211173801.29535-2-josef@toxicpanda.com
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Reviewed-by: Jan Kara <jack@suse.cz>
Reviewed-by: Andrew Morton <akpm@linux-foundation.org>
Cc: Tejun Heo <tj@kernel.org>
Cc: Dave Chinner <david@fromorbit.com>
Cc: Rik van Riel <riel@redhat.com>
Cc: "Kirill A. Shutemov" <kirill@shutemov.name>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
From: John Hubbard <jhubbard@nvidia.com>
This combines the common elements of these routines:
page_cache_get_speculative()
page_cache_add_speculative()
This was anticipated by the original author, as shown by the comment in
commit ce0ad7f095 ("powerpc/mm: Lockless get_user_pages_fast() for
64-bit (v3)"):
"Same as above, but add instead of inc (could just be merged)"
There is no intention to introduce any behavioral change, but there is a
small risk of that, due to slightly differing ways of expressing the
TINY_RCU and related configurations.
This also removes the VM_BUG_ON(in_interrupt()) that was in
page_cache_add_speculative(), but not in page_cache_get_speculative().
This provides slightly less detection of such bugs, but it given that it
was only there on the "add" path anyway, we can likely do without it
just fine.
And it removes the
VM_BUG_ON_PAGE(PageCompound(page) && page != compound_head(page), page);
that page_cache_add_speculative() had.
Link: http://lkml.kernel.org/r/20190206231016.22734-2-jhubbard@nvidia.com
Signed-off-by: John Hubbard <jhubbard@nvidia.com>
Reviewed-by: Andrew Morton <akpm@linux-foundation.org>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Dave Kleikamp <shaggy@linux.vnet.ibm.com>
Cc: Hugh Dickins <hughd@google.com>
Cc: Jeff Layton <jlayton@kernel.org>
Cc: Matthew Wilcox <willy@infradead.org>
Cc: Nicholas Piggin <npiggin@gmail.com>
Cc: Vlastimil Babka <vbabka@suse.cz>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Waiting on a page migration entry has used wait_on_page_locked() all along
since 2006: but you cannot safely wait_on_page_locked() without holding a
reference to the page, and that extra reference is enough to make
migrate_page_move_mapping() fail with -EAGAIN, when a racing task faults
on the entry before migrate_page_move_mapping() gets there.
And that failure is retried nine times, amplifying the pain when trying to
migrate a popular page. With a single persistent faulter, migration
sometimes succeeds; with two or three concurrent faulters, success becomes
much less likely (and the more the page was mapped, the worse the overhead
of unmapping and remapping it on each try).
This is especially a problem for memory offlining, where the outer level
retries forever (or until terminated from userspace), because a heavy
refault workload can trigger an endless loop of migration failures.
wait_on_page_locked() is the wrong tool for the job.
David Herrmann (but was he the first?) noticed this issue in 2014:
https://marc.info/?l=linux-mm&m=140110465608116&w=2
Tim Chen started a thread in August 2017 which appears relevant:
https://marc.info/?l=linux-mm&m=150275941014915&w=2 where Kan Liang went
on to implicate __migration_entry_wait():
https://marc.info/?l=linux-mm&m=150300268411980&w=2 and the thread ended
up with the v4.14 commits: 2554db9165 ("sched/wait: Break up long wake
list walk") 11a19c7b09 ("sched/wait: Introduce wakeup boomark in
wake_up_page_bit")
Baoquan He reported "Memory hotplug softlock issue" 14 November 2018:
https://marc.info/?l=linux-mm&m=154217936431300&w=2
We have all assumed that it is essential to hold a page reference while
waiting on a page lock: partly to guarantee that there is still a struct
page when MEMORY_HOTREMOVE is configured, but also to protect against
reuse of the struct page going to someone who then holds the page locked
indefinitely, when the waiter can reasonably expect timely unlocking.
But in fact, so long as wait_on_page_bit_common() does the put_page(), and
is careful not to rely on struct page contents thereafter, there is no
need to hold a reference to the page while waiting on it. That does mean
that this case cannot go back through the loop: but that's fine for the
page migration case, and even if used more widely, is limited by the "Stop
walking if it's locked" optimization in wake_page_function().
Add interface put_and_wait_on_page_locked() to do this, using "behavior"
enum in place of "lock" arg to wait_on_page_bit_common() to implement it.
No interruptible or killable variant needed yet, but they might follow: I
have a vague notion that reporting -EINTR should take precedence over
return from wait_on_page_bit_common() without knowing the page state, so
arrange it accordingly - but that may be nothing but pedantic.
__migration_entry_wait() still has to take a brief reference to the page,
prior to calling put_and_wait_on_page_locked(): but now that it is dropped
before waiting, the chance of impeding page migration is very much
reduced. Should we perhaps disable preemption across this?
shrink_page_list()'s __ClearPageLocked(): that was a surprise! This
survived a lot of testing before that showed up. PageWaiters may have
been set by wait_on_page_bit_common(), and the reference dropped, just
before shrink_page_list() succeeds in freezing its last page reference: in
such a case, unlock_page() must be used. Follow the suggestion from
Michal Hocko, just revert a978d6f521 ("mm: unlockless reclaim") now:
that optimization predates PageWaiters, and won't buy much these days; but
we can reinstate it for the !PageWaiters case if anyone notices.
It does raise the question: should vmscan.c's is_page_cache_freeable() and
__remove_mapping() now treat a PageWaiters page as if an extra reference
were held? Perhaps, but I don't think it matters much, since
shrink_page_list() already had to win its trylock_page(), so waiters are
not very common there: I noticed no difference when trying the bigger
change, and it's surely not needed while put_and_wait_on_page_locked() is
only used for page migration.
[willy@infradead.org: add put_and_wait_on_page_locked() kerneldoc]
Link: http://lkml.kernel.org/r/alpine.LSU.2.11.1811261121330.1116@eggly.anvils
Signed-off-by: Hugh Dickins <hughd@google.com>
Reported-by: Baoquan He <bhe@redhat.com>
Tested-by: Baoquan He <bhe@redhat.com>
Reviewed-by: Andrea Arcangeli <aarcange@redhat.com>
Acked-by: Michal Hocko <mhocko@suse.com>
Acked-by: Linus Torvalds <torvalds@linux-foundation.org>
Acked-by: Vlastimil Babka <vbabka@suse.cz>
Cc: Matthew Wilcox <willy@infradead.org>
Cc: Baoquan He <bhe@redhat.com>
Cc: David Hildenbrand <david@redhat.com>
Cc: Mel Gorman <mgorman@techsingularity.net>
Cc: David Herrmann <dh.herrmann@gmail.com>
Cc: Tim Chen <tim.c.chen@linux.intel.com>
Cc: Kan Liang <kan.liang@intel.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: Davidlohr Bueso <dave@stgolabs.net>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Christoph Lameter <cl@linux.com>
Cc: Nick Piggin <npiggin@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
The 'end' parameter of the xas_for_each iterator avoids a useless
iteration at the end of the range.
Signed-off-by: Matthew Wilcox <willy@infradead.org>
The page cache offers the ability to search for a miss in the previous or
next N locations. Rather than teach the XArray about the page cache's
definition of a miss, use xas_prev() and xas_next() to search the page
array. This should be more efficient as it does not have to start the
lookup from the top for each index.
Signed-off-by: Matthew Wilcox <willy@infradead.org>
