Export swapcache_free_entries to be used in the alternative function
android_vh_drain_slots_cache_cpu to swap entries in swap slot cache,
it's usage is similar to the usage in drain_slots_cache_cpu.
Bug: 234214858
Signed-off-by: Bing Han <bing.han@transsion.com>
Change-Id: Ia89b1728d540c5cc8995a939a918e12c23057266
3 symbols are exported to be used in vendor hook android_vh_get_swap_page:
1)check_cache_active, used to get swap page from the specified
swap location, it's usage is similar to the usage in get_swap_page
2)scan_swap_map_slots, used to get swap page from the specified swap,
it's usage is similar to get_swap_pages
3)swap_alloc_cluster, used to get swap page from the specified swap,
it's usage is similar to get_swap_pages
Bug: 234214858
Signed-off-by: Bing Han <bing.han@transsion.com>
Change-Id: Ie24c5d32a16c7cb87905d034095ec8fb070dbe0f
The function swap_type_to_swap_info is exported to access the
swap_info_struct of the specified swap, which is regarded as
reserved extended memory.
Bug: 234214858
Signed-off-by: Bing Han <bing.han@transsion.com>
Change-Id: I0107e7d561150f1945a4c161e886e9e03383fff6
Provide a vendor hook android_vh_si_swapinf to replace the
process of updating nr_to_be_unused. When the page is swapped
to a specified swap location, nr_to_be_unused should not be
updated. Because the specified swap is regarded as a reserved
extended memory.
Bug: 234214858
Signed-off-by: Bing Han <bing.han@transsion.com>
Change-Id: Ie41caec345658589bf908fb0f96d038d1fba21f3
Two vendor hooks are added to extend the struct swap_info_struct:
android_vh_alloc_si, extend the allocation of struct swap_info_struct,
adding data to record the information of specified reclaimed location;
android_vh_init_swap_info_struct, adding initializing the extension of
struct swap_info_struct;
Bug: 234214858
Signed-off-by: Bing Han <bing.han@transsion.com>
Change-Id: I0e1d8e38ba7dfd52b609b1c14eb78f8b0ef0f9e6
When the page is unused, a vendor hook android_vh_unuse_swap_page
should be called to specify that the page should not be swapped
to the specified swap location any more.
Bug: 234214858
Signed-off-by: Bing Han <bing.han@transsion.com>
Change-Id: I3fc3675020517f7cc69c76a06150dfb2380dae21
The specified swap is regarded as reserved extended memory.
So nr_swap_pages and total_swap_pages should not be affected
by the specified swap.
Provide a vendor hook android_vh_account_swap_pages to replace
the updating process of nr_swap_pages and total_swap_pages.
When the page is swapped to the specified swap location,
nr_swap_pages and total_swap_pages should not be updated.
Bug: 234214858
Signed-off-by: Bing Han <bing.han@transsion.com>
Change-Id: Ib8dfb355d190399a037b9d9eda478a81c436e224
Export walk_page_range and swp_swap_info for reading swap
from backing device to zram.
Bug: 225273514
Signed-off-by: Liujie Xie <xieliujie@oppo.com>
Change-Id: If888cfc2823d8003b62bdb177740643696cf6f7e
Inline mem_cgroup_try_charge_swap, mem_cgroup_uncharge_swap and
cgroup_throttle_swaprate functions to perform mem_cgroup_disabled static
key check inline before calling the main body of the function. This
minimizes the memcg overhead in the pagefault and exit_mmap paths when
memcgs are disabled using cgroup_disable=memory command-line option.
This change results in ~1% overhead reduction when running PFT test [1]
comparing {CONFIG_MEMCG=n} against {CONFIG_MEMCG=y, cgroup_disable=memory}
configuration on an 8-core ARM64 Android device.
[1] https://lkml.org/lkml/2006/8/29/294 also used in mmtests suite
Signed-off-by: Suren Baghdasaryan <surenb@google.com>
Reviewed-by: Shakeel Butt <shakeelb@google.com>
Reviewed-by: Muchun Song <songmuchun@bytedance.com>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Acked-by: Michal Hocko <mhocko@suse.com>
Link: https://lore.kernel.org/patchwork/patch/1458908/
Bug: 191223209
Signed-off-by: Suren Baghdasaryan <surenb@google.com>
Change-Id: I18d59090ec908037b39324d1f1bb511d06e9c690
Add mem_cgroup_disabled check in vmpressure, mem_cgroup_uncharge_swap and
cgroup_throttle_swaprate functions. This minimizes the memcg overhead in
the pagefault and exit_mmap paths when memcgs are disabled using
cgroup_disable=memory command-line option.
This change results in ~2.1% overhead reduction when running PFT test [1]
comparing {CONFIG_MEMCG=n, CONFIG_MEMCG_SWAP=n} against {CONFIG_MEMCG=y,
CONFIG_MEMCG_SWAP=y, cgroup_disable=memory} configuration on an 8-core
ARM64 Android device.
[1] https://lkml.org/lkml/2006/8/29/294 also used in mmtests suite
Signed-off-by: Suren Baghdasaryan <surenb@google.com>
Reviewed-by: Shakeel Butt <shakeelb@google.com>
Reviewed-by: Muchun Song <songmuchun@bytedance.com>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Acked-by: Michal Hocko <mhocko@suse.com>
Link: https://lore.kernel.org/patchwork/patch/1458906/
Bug: 191223209
Signed-off-by: Suren Baghdasaryan <surenb@google.com>
Change-Id: Ic1fc75eb1e4d7a9848cf641b9f232ad3262c490b
Changes in 5.10.46
dmaengine: idxd: add missing dsa driver unregister
dmaengine: fsl-dpaa2-qdma: Fix error return code in two functions
dmaengine: xilinx: dpdma: initialize registers before request_irq
dmaengine: ALTERA_MSGDMA depends on HAS_IOMEM
dmaengine: QCOM_HIDMA_MGMT depends on HAS_IOMEM
dmaengine: SF_PDMA depends on HAS_IOMEM
dmaengine: stedma40: add missing iounmap() on error in d40_probe()
afs: Fix an IS_ERR() vs NULL check
mm/memory-failure: make sure wait for page writeback in memory_failure
kvm: LAPIC: Restore guard to prevent illegal APIC register access
fanotify: fix copy_event_to_user() fid error clean up
batman-adv: Avoid WARN_ON timing related checks
mac80211: fix skb length check in ieee80211_scan_rx()
mlxsw: reg: Spectrum-3: Enforce lowest max-shaper burst size of 11
mlxsw: core: Set thermal zone polling delay argument to real value at init
libbpf: Fixes incorrect rx_ring_setup_done
net: ipv4: fix memory leak in netlbl_cipsov4_add_std
vrf: fix maximum MTU
net: rds: fix memory leak in rds_recvmsg
net: dsa: felix: re-enable TX flow control in ocelot_port_flush()
net: lantiq: disable interrupt before sheduling NAPI
netfilter: nft_fib_ipv6: skip ipv6 packets from any to link-local
ice: add ndo_bpf callback for safe mode netdev ops
ice: parameterize functions responsible for Tx ring management
udp: fix race between close() and udp_abort()
rtnetlink: Fix regression in bridge VLAN configuration
net/sched: act_ct: handle DNAT tuple collision
net/mlx5e: Remove dependency in IPsec initialization flows
net/mlx5e: Fix page reclaim for dead peer hairpin
net/mlx5: Consider RoCE cap before init RDMA resources
net/mlx5: DR, Allow SW steering for sw_owner_v2 devices
net/mlx5: DR, Don't use SW steering when RoCE is not supported
net/mlx5e: Block offload of outer header csum for UDP tunnels
netfilter: synproxy: Fix out of bounds when parsing TCP options
mptcp: Fix out of bounds when parsing TCP options
sch_cake: Fix out of bounds when parsing TCP options and header
mptcp: try harder to borrow memory from subflow under pressure
mptcp: do not warn on bad input from the network
selftests: mptcp: enable syncookie only in absence of reorders
alx: Fix an error handling path in 'alx_probe()'
cxgb4: fix endianness when flashing boot image
cxgb4: fix sleep in atomic when flashing PHY firmware
cxgb4: halt chip before flashing PHY firmware image
net: stmmac: dwmac1000: Fix extended MAC address registers definition
net: make get_net_ns return error if NET_NS is disabled
net: qualcomm: rmnet: Update rmnet device MTU based on real device
net: qualcomm: rmnet: don't over-count statistics
ethtool: strset: fix message length calculation
qlcnic: Fix an error handling path in 'qlcnic_probe()'
netxen_nic: Fix an error handling path in 'netxen_nic_probe()'
cxgb4: fix wrong ethtool n-tuple rule lookup
ipv4: Fix device used for dst_alloc with local routes
net: qrtr: fix OOB Read in qrtr_endpoint_post
bpf: Fix leakage under speculation on mispredicted branches
ptp: improve max_adj check against unreasonable values
net: cdc_ncm: switch to eth%d interface naming
lantiq: net: fix duplicated skb in rx descriptor ring
net: usb: fix possible use-after-free in smsc75xx_bind
net: fec_ptp: fix issue caused by refactor the fec_devtype
net: ipv4: fix memory leak in ip_mc_add1_src
net/af_unix: fix a data-race in unix_dgram_sendmsg / unix_release_sock
net/mlx5: E-Switch, Read PF mac address
net/mlx5: E-Switch, Allow setting GUID for host PF vport
net/mlx5: Reset mkey index on creation
be2net: Fix an error handling path in 'be_probe()'
net: hamradio: fix memory leak in mkiss_close
net: cdc_eem: fix tx fixup skb leak
cxgb4: fix wrong shift.
bnxt_en: Rediscover PHY capabilities after firmware reset
bnxt_en: Fix TQM fastpath ring backing store computation
bnxt_en: Call bnxt_ethtool_free() in bnxt_init_one() error path
icmp: don't send out ICMP messages with a source address of 0.0.0.0
net: ethernet: fix potential use-after-free in ec_bhf_remove
regulator: cros-ec: Fix error code in dev_err message
regulator: bd70528: Fix off-by-one for buck123 .n_voltages setting
platform/x86: thinkpad_acpi: Add X1 Carbon Gen 9 second fan support
ASoC: rt5659: Fix the lost powers for the HDA header
phy: phy-mtk-tphy: Fix some resource leaks in mtk_phy_init()
ASoC: fsl-asoc-card: Set .owner attribute when registering card.
regulator: rtmv20: Fix to make regcache value first reading back from HW
spi: spi-zynq-qspi: Fix some wrong goto jumps & missing error code
sched/pelt: Ensure that *_sum is always synced with *_avg
ASoC: tas2562: Fix TDM_CFG0_SAMPRATE values
spi: stm32-qspi: Always wait BUSY bit to be cleared in stm32_qspi_wait_cmd()
regulator: rt4801: Fix NULL pointer dereference if priv->enable_gpios is NULL
ASoC: rt5682: Fix the fast discharge for headset unplugging in soundwire mode
pinctrl: ralink: rt2880: avoid to error in calls is pin is already enabled
drm/sun4i: dw-hdmi: Make HDMI PHY into a platform device
ASoC: qcom: lpass-cpu: Fix pop noise during audio capture begin
radeon: use memcpy_to/fromio for UVD fw upload
hwmon: (scpi-hwmon) shows the negative temperature properly
mm: relocate 'write_protect_seq' in struct mm_struct
irqchip/gic-v3: Workaround inconsistent PMR setting on NMI entry
bpf: Inherit expanded/patched seen count from old aux data
bpf: Do not mark insn as seen under speculative path verification
can: bcm: fix infoleak in struct bcm_msg_head
can: bcm/raw/isotp: use per module netdevice notifier
can: j1939: fix Use-after-Free, hold skb ref while in use
can: mcba_usb: fix memory leak in mcba_usb
usb: core: hub: Disable autosuspend for Cypress CY7C65632
usb: chipidea: imx: Fix Battery Charger 1.2 CDP detection
tracing: Do not stop recording cmdlines when tracing is off
tracing: Do not stop recording comms if the trace file is being read
tracing: Do no increment trace_clock_global() by one
PCI: Mark TI C667X to avoid bus reset
PCI: Mark some NVIDIA GPUs to avoid bus reset
PCI: aardvark: Fix kernel panic during PIO transfer
PCI: Add ACS quirk for Broadcom BCM57414 NIC
PCI: Work around Huawei Intelligent NIC VF FLR erratum
KVM: x86: Immediately reset the MMU context when the SMM flag is cleared
KVM: x86/mmu: Calculate and check "full" mmu_role for nested MMU
KVM: X86: Fix x86_emulator slab cache leak
s390/mcck: fix calculation of SIE critical section size
s390/ap: Fix hanging ioctl caused by wrong msg counter
ARCv2: save ABI registers across signal handling
x86/mm: Avoid truncating memblocks for SGX memory
x86/process: Check PF_KTHREAD and not current->mm for kernel threads
x86/ioremap: Map EFI-reserved memory as encrypted for SEV
x86/pkru: Write hardware init value to PKRU when xstate is init
x86/fpu: Prevent state corruption in __fpu__restore_sig()
x86/fpu: Invalidate FPU state after a failed XRSTOR from a user buffer
x86/fpu: Reset state for all signal restore failures
crash_core, vmcoreinfo: append 'SECTION_SIZE_BITS' to vmcoreinfo
dmaengine: pl330: fix wrong usage of spinlock flags in dma_cyclc
mac80211: Fix NULL ptr deref for injected rate info
cfg80211: make certificate generation more robust
cfg80211: avoid double free of PMSR request
drm/amdgpu/gfx10: enlarge CP_MEC_DOORBELL_RANGE_UPPER to cover full doorbell.
drm/amdgpu/gfx9: fix the doorbell missing when in CGPG issue.
net: ll_temac: Make sure to free skb when it is completely used
net: ll_temac: Fix TX BD buffer overwrite
net: bridge: fix vlan tunnel dst null pointer dereference
net: bridge: fix vlan tunnel dst refcnt when egressing
mm/swap: fix pte_same_as_swp() not removing uffd-wp bit when compare
mm/slub: clarify verification reporting
mm/slub: fix redzoning for small allocations
mm/slub: actually fix freelist pointer vs redzoning
mm/slub.c: include swab.h
net: stmmac: disable clocks in stmmac_remove_config_dt()
net: fec_ptp: add clock rate zero check
tools headers UAPI: Sync linux/in.h copy with the kernel sources
perf beauty: Update copy of linux/socket.h with the kernel sources
usb: dwc3: debugfs: Add and remove endpoint dirs dynamically
usb: dwc3: core: fix kernel panic when do reboot
Linux 5.10.46
Signed-off-by: Greg Kroah-Hartman <gregkh@google.com>
Change-Id: I99f37c9f257f90ccdb091306f3d4cfb7c32e3880
Changes in 5.10.21
net: usb: qmi_wwan: support ZTE P685M modem
Input: elantech - fix protocol errors for some trackpoints in SMBus mode
Input: elan_i2c - add new trackpoint report type 0x5F
drm/virtio: use kvmalloc for large allocations
x86/build: Treat R_386_PLT32 relocation as R_386_PC32
JFS: more checks for invalid superblock
sched/core: Allow try_invoke_on_locked_down_task() with irqs disabled
udlfb: Fix memory leak in dlfb_usb_probe
media: mceusb: sanity check for prescaler value
erofs: fix shift-out-of-bounds of blkszbits
media: v4l2-ctrls.c: fix shift-out-of-bounds in std_validate
xfs: Fix assert failure in xfs_setattr_size()
net/af_iucv: remove WARN_ONCE on malformed RX packets
smackfs: restrict bytes count in smackfs write functions
tomoyo: ignore data race while checking quota
net: fix up truesize of cloned skb in skb_prepare_for_shift()
riscv: Get rid of MAX_EARLY_MAPPING_SIZE
nbd: handle device refs for DESTROY_ON_DISCONNECT properly
mm/hugetlb.c: fix unnecessary address expansion of pmd sharing
RDMA/rtrs: Do not signal for heatbeat
RDMA/rtrs-clt: Use bitmask to check sess->flags
RDMA/rtrs-srv: Do not signal REG_MR
tcp: fix tcp_rmem documentation
mptcp: do not wakeup listener for MPJ subflows
net: bridge: use switchdev for port flags set through sysfs too
net/sched: cls_flower: Reject invalid ct_state flags rules
net: dsa: tag_rtl4_a: Support also egress tags
net: ag71xx: remove unnecessary MTU reservation
net: hsr: add support for EntryForgetTime
net: psample: Fix netlink skb length with tunnel info
net: fix dev_ifsioc_locked() race condition
dt-bindings: ethernet-controller: fix fixed-link specification
dt-bindings: net: btusb: DT fix s/interrupt-name/interrupt-names/
ASoC: qcom: Remove useless debug print
rsi: Fix TX EAPOL packet handling against iwlwifi AP
rsi: Move card interrupt handling to RX thread
EDAC/amd64: Do not load on family 0x15, model 0x13
staging: fwserial: Fix error handling in fwserial_create
x86/reboot: Add Zotac ZBOX CI327 nano PCI reboot quirk
vt/consolemap: do font sum unsigned
wlcore: Fix command execute failure 19 for wl12xx
Bluetooth: hci_h5: Set HCI_QUIRK_SIMULTANEOUS_DISCOVERY for btrtl
Bluetooth: btusb: fix memory leak on suspend and resume
mt76: mt7615: reset token when mac_reset happens
pktgen: fix misuse of BUG_ON() in pktgen_thread_worker()
ath10k: fix wmi mgmt tx queue full due to race condition
net: sfp: add mode quirk for GPON module Ubiquiti U-Fiber Instant
Bluetooth: Add new HCI_QUIRK_NO_SUSPEND_NOTIFIER quirk
Bluetooth: Fix null pointer dereference in amp_read_loc_assoc_final_data
staging: most: sound: add sanity check for function argument
staging: bcm2835-audio: Replace unsafe strcpy() with strscpy()
brcmfmac: Add DMI nvram filename quirk for Predia Basic tablet
brcmfmac: Add DMI nvram filename quirk for Voyo winpad A15 tablet
drm/hisilicon: Fix use-after-free
crypto: tcrypt - avoid signed overflow in byte count
fs: make unlazy_walk() error handling consistent
drm/amdgpu: Add check to prevent IH overflow
PCI: Add a REBAR size quirk for Sapphire RX 5600 XT Pulse
ASoC: Intel: bytcr_rt5640: Add new BYT_RT5640_NO_SPEAKERS quirk-flag
drm/amd/display: Guard against NULL pointer deref when get_i2c_info fails
drm/amd/amdgpu: add error handling to amdgpu_virt_read_pf2vf_data
media: uvcvideo: Allow entities with no pads
f2fs: handle unallocated section and zone on pinned/atgc
f2fs: fix to set/clear I_LINKABLE under i_lock
nvme-core: add cancel tagset helpers
nvme-rdma: add clean action for failed reconnection
nvme-tcp: add clean action for failed reconnection
ASoC: Intel: Add DMI quirk table to soc_intel_is_byt_cr()
btrfs: fix error handling in commit_fs_roots
perf/x86/kvm: Add Cascade Lake Xeon steppings to isolation_ucodes[]
ASoC: Intel: sof-sdw: indent and add quirks consistently
ASoC: Intel: sof_sdw: detect DMIC number based on mach params
parisc: Bump 64-bit IRQ stack size to 64 KB
sched/features: Fix hrtick reprogramming
ASoC: Intel: bytcr_rt5640: Add quirk for the Estar Beauty HD MID 7316R tablet
ASoC: Intel: bytcr_rt5640: Add quirk for the Voyo Winpad A15 tablet
ASoC: Intel: bytcr_rt5651: Add quirk for the Jumper EZpad 7 tablet
ASoC: Intel: bytcr_rt5640: Add quirk for the Acer One S1002 tablet
scsi: iscsi: Restrict sessions and handles to admin capabilities
scsi: iscsi: Ensure sysfs attributes are limited to PAGE_SIZE
scsi: iscsi: Verify lengths on passthrough PDUs
Xen/gnttab: handle p2m update errors on a per-slot basis
xen-netback: respect gnttab_map_refs()'s return value
xen: fix p2m size in dom0 for disabled memory hotplug case
zsmalloc: account the number of compacted pages correctly
remoteproc/mediatek: Fix kernel test robot warning
swap: fix swapfile read/write offset
powerpc/sstep: Check instruction validity against ISA version before emulation
powerpc/sstep: Fix incorrect return from analyze_instr()
tty: fix up iterate_tty_read() EOVERFLOW handling
tty: fix up hung_up_tty_read() conversion
tty: clean up legacy leftovers from n_tty line discipline
tty: teach n_tty line discipline about the new "cookie continuations"
tty: teach the n_tty ICANON case about the new "cookie continuations" too
media: v4l: ioctl: Fix memory leak in video_usercopy
ALSA: hda/realtek: Add quirk for Clevo NH55RZQ
ALSA: hda/realtek: Add quirk for Intel NUC 10
ALSA: hda/realtek: Apply dual codec quirks for MSI Godlike X570 board
net: sfp: VSOL V2801F / CarlitoxxPro CPGOS03-0490 v2.0 workaround
net: sfp: add workaround for Realtek RTL8672 and RTL9601C chips
Linux 5.10.21
Signed-off-by: Greg Kroah-Hartman <gregkh@google.com>
Change-Id: I52b1105b73d893779b3886b577accfabe9f83a16
commit caf6912f3f4af7232340d500a4a2008f81b93f14 upstream.
We're not factoring in the start of the file for where to write and
read the swapfile, which leads to very unfortunate side effects of
writing where we should not be...
Fixes: dd6bd0d9c7 ("swap: use bdev_read_page() / bdev_write_page()")
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Cc: Anthony Iliopoulos <ailiop@suse.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Changes in 5.10.12
gpio: mvebu: fix pwm .get_state period calculation
Revert "mm/slub: fix a memory leak in sysfs_slab_add()"
futex: Ensure the correct return value from futex_lock_pi()
futex: Replace pointless printk in fixup_owner()
futex: Provide and use pi_state_update_owner()
rtmutex: Remove unused argument from rt_mutex_proxy_unlock()
futex: Use pi_state_update_owner() in put_pi_state()
futex: Simplify fixup_pi_state_owner()
futex: Handle faults correctly for PI futexes
HID: wacom: Correct NULL dereference on AES pen proximity
HID: multitouch: Apply MT_QUIRK_CONFIDENCE quirk for multi-input devices
media: Revert "media: videobuf2: Fix length check for single plane dmabuf queueing"
media: v4l2-subdev.h: BIT() is not available in userspace
RDMA/vmw_pvrdma: Fix network_hdr_type reported in WC
iwlwifi: dbg: Don't touch the tlv data
kernel/io_uring: cancel io_uring before task works
io_uring: inline io_uring_attempt_task_drop()
io_uring: add warn_once for io_uring_flush()
io_uring: stop SQPOLL submit on creator's death
io_uring: fix null-deref in io_disable_sqo_submit
io_uring: do sqo disable on install_fd error
io_uring: fix false positive sqo warning on flush
io_uring: fix uring_flush in exit_files() warning
io_uring: fix skipping disabling sqo on exec
io_uring: dont kill fasync under completion_lock
io_uring: fix sleeping under spin in __io_clean_op
objtool: Don't fail on missing symbol table
mm/page_alloc: add a missing mm_page_alloc_zone_locked() tracepoint
mm: fix a race on nr_swap_pages
tools: Factor HOSTCC, HOSTLD, HOSTAR definitions
printk: fix buffer overflow potential for print_text()
printk: fix string termination for record_print_text()
Linux 5.10.12
Signed-off-by: Greg Kroah-Hartman <gregkh@google.com>
Change-Id: I6d96ec78494ebbc0daf4fdecfc13e522c6bd6b42
commit b50da6e9f42ade19141f6cf8870bb2312b055aa3 upstream.
The scenario on which "Free swap = -4kB" happens in my system, which is caused
by several get_swap_pages racing with each other and show_swap_cache_info
happens simutaniously. No need to add a lock on get_swap_page_of_type as we
remove "Presub/PosAdd" here.
ProcessA ProcessB ProcessC
ngoals = 1 ngoals = 1
avail = nr_swap_pages(1) avail = nr_swap_pages(1)
nr_swap_pages(1) -= ngoals
nr_swap_pages(0) -= ngoals
nr_swap_pages = -1
Link: https://lkml.kernel.org/r/1607050340-4535-1-git-send-email-zhaoyang.huang@unisoc.com
Signed-off-by: Zhaoyang Huang <zhaoyang.huang@unisoc.com>
Acked-by: Vlastimil Babka <vbabka@suse.cz>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Hugh Dickins <hughd@google.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Export si_swapinfo symbol which is used as part
of meminfo collection from minidump module.
Bug: 176277894
Change-Id: I5dc1672ce649c22dc33d4a544ee5a38f8376becf
Signed-off-by: Vijayanand Jitta <vjitta@codeaurora.org>
Pull block updates from Jens Axboe:
- Series of merge handling cleanups (Baolin, Christoph)
- Series of blk-throttle fixes and cleanups (Baolin)
- Series cleaning up BDI, seperating the block device from the
backing_dev_info (Christoph)
- Removal of bdget() as a generic API (Christoph)
- Removal of blkdev_get() as a generic API (Christoph)
- Cleanup of is-partition checks (Christoph)
- Series reworking disk revalidation (Christoph)
- Series cleaning up bio flags (Christoph)
- bio crypt fixes (Eric)
- IO stats inflight tweak (Gabriel)
- blk-mq tags fixes (Hannes)
- Buffer invalidation fixes (Jan)
- Allow soft limits for zone append (Johannes)
- Shared tag set improvements (John, Kashyap)
- Allow IOPRIO_CLASS_RT for CAP_SYS_NICE (Khazhismel)
- DM no-wait support (Mike, Konstantin)
- Request allocation improvements (Ming)
- Allow md/dm/bcache to use IO stat helpers (Song)
- Series improving blk-iocost (Tejun)
- Various cleanups (Geert, Damien, Danny, Julia, Tetsuo, Tian, Wang,
Xianting, Yang, Yufen, yangerkun)
* tag 'block-5.10-2020-10-12' of git://git.kernel.dk/linux-block: (191 commits)
block: fix uapi blkzoned.h comments
blk-mq: move cancel of hctx->run_work to the front of blk_exit_queue
blk-mq: get rid of the dead flush handle code path
block: get rid of unnecessary local variable
block: fix comment and add lockdep assert
blk-mq: use helper function to test hw stopped
block: use helper function to test queue register
block: remove redundant mq check
block: invoke blk_mq_exit_sched no matter whether have .exit_sched
percpu_ref: don't refer to ref->data if it isn't allocated
block: ratelimit handle_bad_sector() message
blk-throttle: Re-use the throtl_set_slice_end()
blk-throttle: Open code __throtl_de/enqueue_tg()
blk-throttle: Move service tree validation out of the throtl_rb_first()
blk-throttle: Move the list operation after list validation
blk-throttle: Fix IO hang for a corner case
blk-throttle: Avoid tracking latency if low limit is invalid
blk-throttle: Avoid getting the current time if tg->last_finish_time is 0
blk-throttle: Remove a meaningless parameter for throtl_downgrade_state()
block: Remove redundant 'return' statement
...
Pull arm64 updates from Will Deacon:
"There's quite a lot of code here, but much of it is due to the
addition of a new PMU driver as well as some arm64-specific selftests
which is an area where we've traditionally been lagging a bit.
In terms of exciting features, this includes support for the Memory
Tagging Extension which narrowly missed 5.9, hopefully allowing
userspace to run with use-after-free detection in production on CPUs
that support it. Work is ongoing to integrate the feature with KASAN
for 5.11.
Another change that I'm excited about (assuming they get the hardware
right) is preparing the ASID allocator for sharing the CPU page-table
with the SMMU. Those changes will also come in via Joerg with the
IOMMU pull.
We do stray outside of our usual directories in a few places, mostly
due to core changes required by MTE. Although much of this has been
Acked, there were a couple of places where we unfortunately didn't get
any review feedback.
Other than that, we ran into a handful of minor conflicts in -next,
but nothing that should post any issues.
Summary:
- Userspace support for the Memory Tagging Extension introduced by
Armv8.5. Kernel support (via KASAN) is likely to follow in 5.11.
- Selftests for MTE, Pointer Authentication and FPSIMD/SVE context
switching.
- Fix and subsequent rewrite of our Spectre mitigations, including
the addition of support for PR_SPEC_DISABLE_NOEXEC.
- Support for the Armv8.3 Pointer Authentication enhancements.
- Support for ASID pinning, which is required when sharing
page-tables with the SMMU.
- MM updates, including treating flush_tlb_fix_spurious_fault() as a
no-op.
- Perf/PMU driver updates, including addition of the ARM CMN PMU
driver and also support to handle CPU PMU IRQs as NMIs.
- Allow prefetchable PCI BARs to be exposed to userspace using normal
non-cacheable mappings.
- Implementation of ARCH_STACKWALK for unwinding.
- Improve reporting of unexpected kernel traps due to BPF JIT
failure.
- Improve robustness of user-visible HWCAP strings and their
corresponding numerical constants.
- Removal of TEXT_OFFSET.
- Removal of some unused functions, parameters and prototypes.
- Removal of MPIDR-based topology detection in favour of firmware
description.
- Cleanups to handling of SVE and FPSIMD register state in
preparation for potential future optimisation of handling across
syscalls.
- Cleanups to the SDEI driver in preparation for support in KVM.
- Miscellaneous cleanups and refactoring work"
* tag 'arm64-upstream' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux: (148 commits)
Revert "arm64: initialize per-cpu offsets earlier"
arm64: random: Remove no longer needed prototypes
arm64: initialize per-cpu offsets earlier
kselftest/arm64: Check mte tagged user address in kernel
kselftest/arm64: Verify KSM page merge for MTE pages
kselftest/arm64: Verify all different mmap MTE options
kselftest/arm64: Check forked child mte memory accessibility
kselftest/arm64: Verify mte tag inclusion via prctl
kselftest/arm64: Add utilities and a test to validate mte memory
perf: arm-cmn: Fix conversion specifiers for node type
perf: arm-cmn: Fix unsigned comparison to less than zero
arm64: dbm: Invalidate local TLB when setting TCR_EL1.HD
arm64: mm: Make flush_tlb_fix_spurious_fault() a no-op
arm64: Add support for PR_SPEC_DISABLE_NOEXEC prctl() option
arm64: Pull in task_stack_page() to Spectre-v4 mitigation code
KVM: arm64: Allow patching EL2 vectors even with KASLR is not enabled
arm64: Get rid of arm64_ssbd_state
KVM: arm64: Convert ARCH_WORKAROUND_2 to arm64_get_spectre_v4_state()
KVM: arm64: Get rid of kvm_arm_have_ssbd()
KVM: arm64: Simplify handling of ARCH_WORKAROUND_2
...
SWP_FS is used to make swap_{read,write}page() go through the
filesystem, and it's only used for swap files over NFS. So, !SWP_FS
means non NFS for now, it could be either file backed or device backed.
Something similar goes with legacy SWP_FILE.
So in order to achieve the goal of the original patch, SWP_BLKDEV should
be used instead.
FS corruption can be observed with SSD device + XFS + fragmented
swapfile due to CONFIG_THP_SWAP=y.
I reproduced the issue with the following details:
Environment:
QEMU + upstream kernel + buildroot + NVMe (2 GB)
Kernel config:
CONFIG_BLK_DEV_NVME=y
CONFIG_THP_SWAP=y
Some reproducible steps:
mkfs.xfs -f /dev/nvme0n1
mkdir /tmp/mnt
mount /dev/nvme0n1 /tmp/mnt
bs="32k"
sz="1024m" # doesn't matter too much, I also tried 16m
xfs_io -f -c "pwrite -R -b $bs 0 $sz" -c "fdatasync" /tmp/mnt/sw
xfs_io -f -c "pwrite -R -b $bs 0 $sz" -c "fdatasync" /tmp/mnt/sw
xfs_io -f -c "pwrite -R -b $bs 0 $sz" -c "fdatasync" /tmp/mnt/sw
xfs_io -f -c "pwrite -F -S 0 -b $bs 0 $sz" -c "fdatasync" /tmp/mnt/sw
xfs_io -f -c "pwrite -R -b $bs 0 $sz" -c "fsync" /tmp/mnt/sw
mkswap /tmp/mnt/sw
swapon /tmp/mnt/sw
stress --vm 2 --vm-bytes 600M # doesn't matter too much as well
Symptoms:
- FS corruption (e.g. checksum failure)
- memory corruption at: 0xd2808010
- segfault
Fixes: f0eea189e8 ("mm, THP, swap: Don't allocate huge cluster for file backed swap device")
Fixes: 38d8b4e6bd ("mm, THP, swap: delay splitting THP during swap out")
Signed-off-by: Gao Xiang <hsiangkao@redhat.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: "Huang, Ying" <ying.huang@intel.com>
Reviewed-by: Yang Shi <shy828301@gmail.com>
Acked-by: Rafael Aquini <aquini@redhat.com>
Cc: Matthew Wilcox <willy@infradead.org>
Cc: Carlos Maiolino <cmaiolino@redhat.com>
Cc: Eric Sandeen <esandeen@redhat.com>
Cc: Dave Chinner <david@fromorbit.com>
Cc: <stable@vger.kernel.org>
Link: https://lkml.kernel.org/r/20200820045323.7809-1-hsiangkao@redhat.com
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
The BDI_CAP_STABLE_WRITES is one of the few bits of information in the
backing_dev_info shared between the block drivers and the writeback code.
To help untangling the dependency replace it with a queue flag and a
superblock flag derived from it. This also helps with the case of e.g.
a file system requiring stable writes due to its own checksumming, but
not forcing it on other users of the block device like the swap code.
One downside is that we an't support the stable_pages_required bdi
attribute in sysfs anymore. It is replaced with a queue attribute which
also is writable for easier testing.
Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Jan Kara <jack@suse.cz>
Reviewed-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
BDI_CAP_SYNCHRONOUS_IO is only checked in the swap code, and used to
decided if ->rw_page can be used on a block device. Just check up for
the method instead. The only complication is that zram needs a second
set of block_device_operations as it can switch between modes that
actually support ->rw_page and those who don't.
Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Jan Kara <jack@suse.cz>
Reviewed-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
swap_type_of is used for two entirely different purposes:
(1) check what swap type a given device/offset corresponds to
(2) find the first available swap device that can be written to
Mixing both in a single function creates an unreadable mess. Create two
separate functions instead, and switch both to pass a dev_t instead of
a struct block_device to further simplify the code.
Signed-off-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Arm's Memory Tagging Extension (MTE) adds some metadata (tags) to
every physical page, when swapping pages out to disk it is necessary to
save these tags, and later restore them when reading the pages back.
Add some hooks along with dummy implementations to enable the
arch code to handle this.
Three new hooks are added to the swap code:
* arch_prepare_to_swap() and
* arch_swap_invalidate_page() / arch_swap_invalidate_area().
One new hook is added to shmem:
* arch_swap_restore()
Signed-off-by: Steven Price <steven.price@arm.com>
[catalin.marinas@arm.com: add unlock_page() on the error path]
[catalin.marinas@arm.com: dropped the _tags suffix]
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
Acked-by: Andrew Morton <akpm@linux-foundation.org>
swap_info_struct si.highest_bit, si.swap_map[offset] and si.flags could
be accessed concurrently separately as noticed by KCSAN,
=== si.highest_bit ===
write to 0xffff8d5abccdc4d4 of 4 bytes by task 5353 on cpu 24:
swap_range_alloc+0x81/0x130
swap_range_alloc at mm/swapfile.c:681
scan_swap_map_slots+0x371/0xb90
get_swap_pages+0x39d/0x5c0
get_swap_page+0xf2/0x524
add_to_swap+0xe4/0x1c0
shrink_page_list+0x1795/0x2870
shrink_inactive_list+0x316/0x880
shrink_lruvec+0x8dc/0x1380
shrink_node+0x317/0xd80
do_try_to_free_pages+0x1f7/0xa10
try_to_free_pages+0x26c/0x5e0
__alloc_pages_slowpath+0x458/0x1290
read to 0xffff8d5abccdc4d4 of 4 bytes by task 6672 on cpu 70:
scan_swap_map_slots+0x4a6/0xb90
scan_swap_map_slots at mm/swapfile.c:892
get_swap_pages+0x39d/0x5c0
get_swap_page+0xf2/0x524
add_to_swap+0xe4/0x1c0
shrink_page_list+0x1795/0x2870
shrink_inactive_list+0x316/0x880
shrink_lruvec+0x8dc/0x1380
shrink_node+0x317/0xd80
do_try_to_free_pages+0x1f7/0xa10
try_to_free_pages+0x26c/0x5e0
__alloc_pages_slowpath+0x458/0x1290
Reported by Kernel Concurrency Sanitizer on:
CPU: 70 PID: 6672 Comm: oom01 Tainted: G W L 5.5.0-next-20200205+ #3
Hardware name: HPE ProLiant DL385 Gen10/ProLiant DL385 Gen10, BIOS A40 07/10/2019
=== si.swap_map[offset] ===
write to 0xffffbc370c29a64c of 1 bytes by task 6856 on cpu 86:
__swap_entry_free_locked+0x8c/0x100
__swap_entry_free_locked at mm/swapfile.c:1209 (discriminator 4)
__swap_entry_free.constprop.20+0x69/0xb0
free_swap_and_cache+0x53/0xa0
unmap_page_range+0x7f8/0x1d70
unmap_single_vma+0xcd/0x170
unmap_vmas+0x18b/0x220
exit_mmap+0xee/0x220
mmput+0x10e/0x270
do_exit+0x59b/0xf40
do_group_exit+0x8b/0x180
read to 0xffffbc370c29a64c of 1 bytes by task 6855 on cpu 20:
_swap_info_get+0x81/0xa0
_swap_info_get at mm/swapfile.c:1140
free_swap_and_cache+0x40/0xa0
unmap_page_range+0x7f8/0x1d70
unmap_single_vma+0xcd/0x170
unmap_vmas+0x18b/0x220
exit_mmap+0xee/0x220
mmput+0x10e/0x270
do_exit+0x59b/0xf40
do_group_exit+0x8b/0x180
=== si.flags ===
write to 0xffff956c8fc6c400 of 8 bytes by task 6087 on cpu 23:
scan_swap_map_slots+0x6fe/0xb50
scan_swap_map_slots at mm/swapfile.c:887
get_swap_pages+0x39d/0x5c0
get_swap_page+0x377/0x524
add_to_swap+0xe4/0x1c0
shrink_page_list+0x1795/0x2870
shrink_inactive_list+0x316/0x880
shrink_lruvec+0x8dc/0x1380
shrink_node+0x317/0xd80
do_try_to_free_pages+0x1f7/0xa10
try_to_free_pages+0x26c/0x5e0
__alloc_pages_slowpath+0x458/0x1290
read to 0xffff956c8fc6c400 of 8 bytes by task 6207 on cpu 63:
_swap_info_get+0x41/0xa0
__swap_info_get at mm/swapfile.c:1114
put_swap_page+0x84/0x490
__remove_mapping+0x384/0x5f0
shrink_page_list+0xff1/0x2870
shrink_inactive_list+0x316/0x880
shrink_lruvec+0x8dc/0x1380
shrink_node+0x317/0xd80
do_try_to_free_pages+0x1f7/0xa10
try_to_free_pages+0x26c/0x5e0
__alloc_pages_slowpath+0x458/0x1290
The writes are under si->lock but the reads are not. For si.highest_bit
and si.swap_map[offset], data race could trigger logic bugs, so fix them
by having WRITE_ONCE() for the writes and READ_ONCE() for the reads
except those isolated reads where they compare against zero which a data
race would cause no harm. Thus, annotate them as intentional data races
using the data_race() macro.
For si.flags, the readers are only interested in a single bit where a
data race there would cause no issue there.
[cai@lca.pw: add a missing annotation for si->flags in memory.c]
Link: http://lkml.kernel.org/r/1581612647-5958-1-git-send-email-cai@lca.pw
Signed-off-by: Qian Cai <cai@lca.pw>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Cc: Marco Elver <elver@google.com>
Cc: Hugh Dickins <hughd@google.com>
Link: http://lkml.kernel.org/r/1581095163-12198-1-git-send-email-cai@lca.pw
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
The thp prefix is more frequently used than hpage and we should be
consistent between the various functions.
[akpm@linux-foundation.org: fix mm/migrate.c]
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: William Kucharski <william.kucharski@oracle.com>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Cc: Mike Kravetz <mike.kravetz@oracle.com>
Cc: David Hildenbrand <david@redhat.com>
Cc: "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>
Link: http://lkml.kernel.org/r/20200629151959.15779-6-willy@infradead.org
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
In current implementation, newly created or swap-in anonymous page is
started on active list. Growing active list results in rebalancing
active/inactive list so old pages on active list are demoted to inactive
list. Hence, the page on active list isn't protected at all.
Following is an example of this situation.
Assume that 50 hot pages on active list. Numbers denote the number of
pages on active/inactive list (active | inactive).
1. 50 hot pages on active list
50(h) | 0
2. workload: 50 newly created (used-once) pages
50(uo) | 50(h)
3. workload: another 50 newly created (used-once) pages
50(uo) | 50(uo), swap-out 50(h)
This patch tries to fix this issue. Like as file LRU, newly created or
swap-in anonymous pages will be inserted to the inactive list. They are
promoted to active list if enough reference happens. This simple
modification changes the above example as following.
1. 50 hot pages on active list
50(h) | 0
2. workload: 50 newly created (used-once) pages
50(h) | 50(uo)
3. workload: another 50 newly created (used-once) pages
50(h) | 50(uo), swap-out 50(uo)
As you can see, hot pages on active list would be protected.
Note that, this implementation has a drawback that the page cannot be
promoted and will be swapped-out if re-access interval is greater than the
size of inactive list but less than the size of total(active+inactive).
To solve this potential issue, following patch will apply workingset
detection similar to the one that's already applied to file LRU.
Signed-off-by: Joonsoo Kim <iamjoonsoo.kim@lge.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Acked-by: Vlastimil Babka <vbabka@suse.cz>
Cc: Hugh Dickins <hughd@google.com>
Cc: Matthew Wilcox <willy@infradead.org>
Cc: Mel Gorman <mgorman@techsingularity.net>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Minchan Kim <minchan@kernel.org>
Link: http://lkml.kernel.org/r/1595490560-15117-3-git-send-email-iamjoonsoo.kim@lge.com
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Right now, users that are otherwise memory controlled can easily escape
their containment and allocate significant amounts of memory that they're
not being charged for. That's because swap readahead pages are not being
charged until somebody actually faults them into their page table. This
can be exploited with MADV_WILLNEED, which triggers arbitrary readahead
allocations without charging the pages.
There are additional problems with the delayed charging of swap pages:
1. To implement refault/workingset detection for anonymous pages, we
need to have a target LRU available at swapin time, but the LRU is not
determinable until the page has been charged.
2. To implement per-cgroup LRU locking, we need page->mem_cgroup to be
stable when the page is isolated from the LRU; otherwise, the locks
change under us. But swapcache gets charged after it's already on the
LRU, and even if we cannot isolate it ourselves (since charging is not
exactly optional).
The previous patch ensured we always maintain cgroup ownership records for
swap pages. This patch moves the swapcache charging point from the fault
handler to swapin time to fix all of the above problems.
v2: simplify swapin error checking (Joonsoo)
[hughd@google.com: fix livelock in __read_swap_cache_async()]
Link: http://lkml.kernel.org/r/alpine.LSU.2.11.2005212246080.8458@eggly.anvils
Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
Signed-off-by: Hugh Dickins <hughd@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Alex Shi <alex.shi@linux.alibaba.com>
Cc: Hugh Dickins <hughd@google.com>
Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
Cc: "Kirill A. Shutemov" <kirill@shutemov.name>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Roman Gushchin <guro@fb.com>
Cc: Shakeel Butt <shakeelb@google.com>
Cc: Balbir Singh <bsingharora@gmail.com>
Cc: Rafael Aquini <aquini@redhat.com>
Cc: Alex Shi <alex.shi@linux.alibaba.com>
Link: http://lkml.kernel.org/r/20200508183105.225460-17-hannes@cmpxchg.org
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Memcg maintains a private MEMCG_RSS counter. This divergence from the
generic VM accounting means unnecessary code overhead, and creates a
dependency for memcg that page->mapping is set up at the time of charging,
so that page types can be told apart.
Convert the generic accounting sites to mod_lruvec_page_state and friends
to maintain the per-cgroup vmstat counter of NR_ANON_MAPPED. We use
lock_page_memcg() to stabilize page->mem_cgroup during rmap changes, the
same way we do for NR_FILE_MAPPED.
With the previous patch removing MEMCG_CACHE and the private NR_SHMEM
counter, this patch finally eliminates the need to have page->mapping set
up at charge time. However, we need to have page->mem_cgroup set up by
the time rmap runs and does the accounting, so switch the commit and the
rmap callbacks around.
v2: fix temporary accounting bug by switching rmap<->commit (Joonsoo)
Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Cc: Alex Shi <alex.shi@linux.alibaba.com>
Cc: Hugh Dickins <hughd@google.com>
Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
Cc: "Kirill A. Shutemov" <kirill@shutemov.name>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Roman Gushchin <guro@fb.com>
Cc: Shakeel Butt <shakeelb@google.com>
Cc: Balbir Singh <bsingharora@gmail.com>
Link: http://lkml.kernel.org/r/20200508183105.225460-11-hannes@cmpxchg.org
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
The cgroup swaprate throttling is about matching new anon allocations to
the rate of available IO when that is being throttled. It's the io
controller hooking into the VM, rather than a memory controller thing.
Rename mem_cgroup_throttle_swaprate() to cgroup_throttle_swaprate(), and
drop the @memcg argument which is only used to check whether the preceding
page charge has succeeded and the fault is proceeding.
We could decouple the call from mem_cgroup_try_charge() here as well, but
that would cause unnecessary churn: the following patches convert all
callsites to a new charge API and we'll decouple as we go along.
Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Alex Shi <alex.shi@linux.alibaba.com>
Reviewed-by: Joonsoo Kim <iamjoonsoo.kim@lge.com>
Reviewed-by: Shakeel Butt <shakeelb@google.com>
Cc: Hugh Dickins <hughd@google.com>
Cc: "Kirill A. Shutemov" <kirill@shutemov.name>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Roman Gushchin <guro@fb.com>
Cc: Balbir Singh <bsingharora@gmail.com>
Link: http://lkml.kernel.org/r/20200508183105.225460-5-hannes@cmpxchg.org
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
The memcg charging API carries a boolean @compound parameter that tells
whether the page we're dealing with is a hugepage.
mem_cgroup_commit_charge() has another boolean @lrucare that indicates
whether the page needs LRU locking or not while charging. The majority of
callsites know those parameters at compile time, which results in a lot of
naked "false, false" argument lists. This makes for cryptic code and is a
breeding ground for subtle mistakes.
Thankfully, the huge page state can be inferred from the page itself and
doesn't need to be passed along. This is safe because charging completes
before the page is published and somebody may split it.
Simplify the callsites by removing @compound, and let memcg infer the
state by using hpage_nr_pages() unconditionally. That function does
PageTransHuge() to identify huge pages, which also helpfully asserts that
nobody passes in tail pages by accident.
The following patches will introduce a new charging API, best not to carry
over unnecessary weight.
Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Alex Shi <alex.shi@linux.alibaba.com>
Reviewed-by: Joonsoo Kim <iamjoonsoo.kim@lge.com>
Reviewed-by: Shakeel Butt <shakeelb@google.com>
Cc: Hugh Dickins <hughd@google.com>
Cc: "Kirill A. Shutemov" <kirill@shutemov.name>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Roman Gushchin <guro@fb.com>
Cc: Balbir Singh <bsingharora@gmail.com>
Link: http://lkml.kernel.org/r/20200508183105.225460-4-hannes@cmpxchg.org
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Fix the heading and Size/Used/Priority field alignments in /proc/swaps.
If the Size and/or Used value is >= 10000000 (8 bytes), then the
alignment by using tab characters is broken.
This patch maintains the use of tabs for alignment. If spaces are
preferred, we can just use a Field Width specifier for the bytes and
inuse fields. That way those fields don't have to be a multiple of 8
bytes in width. E.g., with a field width of 12, both Size and Used
would always fit on the first line of an 80-column wide terminal (only
Priority would be on the second line).
There are actually 2 problems: heading alignment and field width. On an
xterm, if Used is 7 bytes in length, the tab does nothing, and the
display is like this, with no space/tab between the Used and Priority
fields. (ugh)
Filename Type Size Used Priority
/dev/sda8 partition 16779260 2023012-1
To be clear, if one does 'cat /proc/swaps >/tmp/proc.swaps', it does look
different, like so:
Filename Type Size Used Priority
/dev/sda8 partition 16779260 2086988 -1
Signed-off-by: Randy Dunlap <rdunlap@infradead.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Andrew Morton <akpm@linux-foundation.org>
Cc: Hugh Dickins <hughd@google.com>
Cc: Alexander Viro <viro@zeniv.linux.org.uk>
Link: http://lkml.kernel.org/r/c0ffb41a-81ac-ddfa-d452-a9229ecc0387@infradead.org
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
In some swap scalability test, it is found that there are heavy lock
contention on swap cache even if we have split one swap cache radix tree
per swap device to one swap cache radix tree every 64 MB trunk in commit
4b3ef9daa4 ("mm/swap: split swap cache into 64MB trunks").
The reason is as follow. After the swap device becomes fragmented so
that there's no free swap cluster, the swap device will be scanned
linearly to find the free swap slots. swap_info_struct->cluster_next is
the next scanning base that is shared by all CPUs. So nearby free swap
slots will be allocated for different CPUs. The probability for
multiple CPUs to operate on the same 64 MB trunk is high. This causes
the lock contention on the swap cache.
To solve the issue, in this patch, for SSD swap device, a percpu version
next scanning base (cluster_next_cpu) is added. Every CPU will use its
own per-cpu next scanning base. And after finishing scanning a 64MB
trunk, the per-cpu scanning base will be changed to the beginning of
another randomly selected 64MB trunk. In this way, the probability for
multiple CPUs to operate on the same 64 MB trunk is reduced greatly.
Thus the lock contention is reduced too. For HDD, because sequential
access is more important for IO performance, the original shared next
scanning base is used.
To test the patch, we have run 16-process pmbench memory benchmark on a
2-socket server machine with 48 cores. One ram disk is configured as the
swap device per socket. The pmbench working-set size is much larger than
the available memory so that swapping is triggered. The memory read/write
ratio is 80/20 and the accessing pattern is random. In the original
implementation, the lock contention on the swap cache is heavy. The perf
profiling data of the lock contention code path is as following,
_raw_spin_lock_irq.add_to_swap_cache.add_to_swap.shrink_page_list: 7.91
_raw_spin_lock_irqsave.__remove_mapping.shrink_page_list: 7.11
_raw_spin_lock.swapcache_free_entries.free_swap_slot.__swap_entry_free: 2.51
_raw_spin_lock_irqsave.swap_cgroup_record.mem_cgroup_uncharge_swap: 1.66
_raw_spin_lock_irq.shrink_inactive_list.shrink_lruvec.shrink_node: 1.29
_raw_spin_lock.free_pcppages_bulk.drain_pages_zone.drain_pages: 1.03
_raw_spin_lock_irq.shrink_active_list.shrink_lruvec.shrink_node: 0.93
After applying this patch, it becomes,
_raw_spin_lock.swapcache_free_entries.free_swap_slot.__swap_entry_free: 3.58
_raw_spin_lock_irq.shrink_inactive_list.shrink_lruvec.shrink_node: 2.3
_raw_spin_lock_irqsave.swap_cgroup_record.mem_cgroup_uncharge_swap: 2.26
_raw_spin_lock_irq.shrink_active_list.shrink_lruvec.shrink_node: 1.8
_raw_spin_lock.free_pcppages_bulk.drain_pages_zone.drain_pages: 1.19
The lock contention on the swap cache is almost eliminated.
And the pmbench score increases 18.5%. The swapin throughput increases
18.7% from 2.96 GB/s to 3.51 GB/s. While the swapout throughput increases
18.5% from 2.99 GB/s to 3.54 GB/s.
We need really fast disk to show the benefit. I have tried this on 2
Intel P3600 NVMe disks. The performance improvement is only about 1%.
The improvement should be better on the faster disks, such as Intel Optane
disk.
[ying.huang@intel.com: fix cluster_next_cpu allocation and freeing, per Daniel]
Link: http://lkml.kernel.org/r/20200525002648.336325-1-ying.huang@intel.com
[ying.huang@intel.com: v4]
Link: http://lkml.kernel.org/r/20200529010840.928819-1-ying.huang@intel.com
Signed-off-by: "Huang, Ying" <ying.huang@intel.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Reviewed-by: Daniel Jordan <daniel.m.jordan@oracle.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Minchan Kim <minchan@kernel.org>
Cc: Tim Chen <tim.c.chen@linux.intel.com>
Cc: Hugh Dickins <hughd@google.com>
Link: http://lkml.kernel.org/r/20200520031502.175659-1-ying.huang@intel.com
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Now, the scalability of swap code will drop much when the swap device
becomes fragmented, because the swap slots allocation batching stops
working. To solve the problem, in this patch, we will try to scan a
little more swap slots with restricted effort to batch the swap slots
allocation even if the swap device is fragmented. Test shows that the
benchmark score can increase up to 37.1% with the patch. Details are as
follows.
The swap code has a per-cpu cache of swap slots. These batch swap space
allocations to improve swap subsystem scaling. In the following code
path,
add_to_swap()
get_swap_page()
refill_swap_slots_cache()
get_swap_pages()
scan_swap_map_slots()
scan_swap_map_slots() and get_swap_pages() can return multiple swap
slots for each call. These slots will be cached in the per-CPU swap
slots cache, so that several following swap slot requests will be
fulfilled there to avoid the lock contention in the lower level swap
space allocation/freeing code path.
But this only works when there are free swap clusters. If a swap device
becomes so fragmented that there's no free swap clusters,
scan_swap_map_slots() and get_swap_pages() will return only one swap
slot for each call in the above code path. Effectively, this falls back
to the situation before the swap slots cache was introduced, the heavy
lock contention on the swap related locks kills the scalability.
Why does it work in this way? Because the swap device could be large,
and the free swap slot scanning could be quite time consuming, to avoid
taking too much time to scanning free swap slots, the conservative
method was used.
In fact, this can be improved via scanning a little more free slots with
strictly restricted effort. Which is implemented in this patch. In
scan_swap_map_slots(), after the first free swap slot is gotten, we will
try to scan a little more, but only if we haven't scanned too many slots
(< LATENCY_LIMIT). That is, the added scanning latency is strictly
restricted.
To test the patch, we have run 16-process pmbench memory benchmark on a
2-socket server machine with 48 cores. Multiple ram disks are
configured as the swap devices. The pmbench working-set size is much
larger than the available memory so that swapping is triggered. The
memory read/write ratio is 80/20 and the accessing pattern is random, so
the swap space becomes highly fragmented during the test. In the
original implementation, the lock contention on swap related locks is
very heavy. The perf profiling data of the lock contention code path is
as following,
_raw_spin_lock.get_swap_pages.get_swap_page.add_to_swap: 21.03
_raw_spin_lock_irq.shrink_inactive_list.shrink_lruvec.shrink_node: 1.92
_raw_spin_lock_irq.shrink_active_list.shrink_lruvec.shrink_node: 1.72
_raw_spin_lock.free_pcppages_bulk.drain_pages_zone.drain_pages: 0.69
While after applying this patch, it becomes,
_raw_spin_lock_irq.shrink_inactive_list.shrink_lruvec.shrink_node: 4.89
_raw_spin_lock_irq.shrink_active_list.shrink_lruvec.shrink_node: 3.85
_raw_spin_lock.free_pcppages_bulk.drain_pages_zone.drain_pages: 1.1
_raw_spin_lock_irqsave.pagevec_lru_move_fn.__lru_cache_add.do_swap_page: 0.88
That is, the lock contention on the swap locks is eliminated.
And the pmbench score increases 37.1%. The swapin throughput increases
45.7% from 2.02 GB/s to 2.94 GB/s. While the swapout throughput increases
45.3% from 2.04 GB/s to 2.97 GB/s.
Signed-off-by: "Huang, Ying" <ying.huang@intel.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Acked-by: Tim Chen <tim.c.chen@linux.intel.com>
Cc: Dave Hansen <dave.hansen@intel.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Minchan Kim <minchan@kernel.org>
Cc: Hugh Dickins <hughd@google.com>
Link: http://lkml.kernel.org/r/20200427030023.264780-1-ying.huang@intel.com
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
