These hooks are designed to set or clear OEM reserved pageflags when the
memory state may change.
Bug: 336964184
Change-Id: I9cb288ef6eef7a719d4f4748d6b71010645b7d50
Signed-off-by: Xiaofeng Yuan <yuanxiaofeng@vivo.com>
with this vendor_hook, oem can dynamically adjust fault_around_bytes to
balance memory usage and performance
Bug: 340744332
Change-Id: I24414c7ba7e77ae06ce5e8cf52377c3485185cfe
Signed-off-by: Dezhi Huang <huangdezhi@hihonor.com>
Since SPF is an out-of-tree feature, the risks of changing its behavior
are higher. Add a vendor hook to enable speculative swap pagefaults. By
default it's disabled and should not cause troubles for current users.
Bug: 322762567
Change-Id: I3df7c545aa27d2707ee51ea42368f785c5faa735
Signed-off-by: Suren Baghdasaryan <surenb@google.com>
We introduce an interrupt hook in Android to manage memory compression
using madvise, improving user experience.
Currently, when a user returns to the home screen, memory compression is
triggered using madvise. The vma and PAGEOUT flag are sent to
process_madvise, initiating page reclaim.
However, if an app is re-opened soon after starting compression, the
reclaim process can cause read delays, leading to potential lag.
To resolve this, we propose to skip pte range traversal. By comparing
the vma's task uid with the current app's uid, we can identify and
interrupt the madvise operation for that vma.
Implementing this requires a vendor hook for should_end_madvise. This
allows us to skip traversal, enhancing user experience.
Bug: 307846869
Change-Id: If2bdbc200b7305e92f836353b7356aa115e00705
Signed-off-by: zihan ju <zihan.ju@transsion.com>
We want to skip swapcache in some scenarios to reduce
swap cache
Bug: 300857016
Change-Id: I8cbc45733d861be6f56628b40a1ef936cae3b45d
Signed-off-by: Xiaojun Ji <jixj@motorola.com>
Our Android phones occur Panic as follows:
[77522.303024][ T9734] Call trace:
[77522.303039][ T9734] dump_backtrace.cfi_jt+0x0/0x8
[77522.303052][ T9734] dump_stack_lvl+0xc4/0x140
[77522.303061][ T9734] dump_stack+0x1c/0x2c
[77522.303123][ T9734] mrdump_common_die+0x3a8/0x544 [mrdump]
[77522.303177][ T9734] ipanic_die+0x24/0x38 [mrdump]
[77522.303189][ T9734] die+0x340/0x698
[77522.303199][ T9734] bug_handler+0x48/0x108
[77522.303210][ T9734] brk_handler+0xac/0x1a8
[77522.303221][ T9734] do_debug_exception+0xe0/0x1e0
[77522.303233][ T9734] el1_dbg+0x38/0x54
[77522.303242][ T9734] el1_sync_handler+0x40/0x88
[77522.303255][ T9734] el1_sync+0x8c/0x140
[77522.303264][ T9734] plist_requeue+0xd4/0x110
[77522.303297][ T9734] tran_get_swap_pages+0xc8/0x364 [memfusion]
[77522.303329][ T9734] probe_android_vh_get_swap_page+0x1b4/0x220 [memfusion]
[77522.303342][ T9734] get_swap_page+0x258/0x304
[77522.303352][ T9734] shrink_page_list+0xe00/0x1e0c
[77522.303361][ T9734] shrink_inactive_list+0x2f4/0xac8
[77522.303373][ T9734] shrink_lruvec+0x1a4/0x34c
[77522.303383][ T9734] shrink_node_memcgs+0x84/0x3b0
[77522.303391][ T9734] shrink_node+0x2c4/0x6e4
[77522.303400][ T9734] shrink_zones+0x16c/0x29c
[77522.303410][ T9734] do_try_to_free_pages+0xe4/0x2bc
[77522.303418][ T9734] try_to_free_pages+0x388/0x7b4
[77522.303429][ T9734] __alloc_pages_direct_reclaim+0x88/0x278
[77522.303438][ T9734] __alloc_pages_slowpath+0x464/0xb24
[77522.303447][ T9734] __alloc_pages_nodemask+0x1f4/0x3dc
[77522.303458][ T9734] do_anonymous_page+0x164/0x914
[77522.303466][ T9734] handle_pte_fault+0x15c/0x9f8
[77522.303476][ T9734] ___handle_speculative_fault+0x234/0xe18
[77522.303485][ T9734] __handle_speculative_fault+0x78/0x21c
[77522.303497][ T9734] do_page_fault+0x36c/0x754
[77522.303506][ T9734] do_translation_fault+0x48/0x64
[77522.303514][ T9734] do_mem_abort+0x6c/0x164
[77522.303522][ T9734] el0_da+0x24/0x34
[77522.303531][ T9734] el0_sync_handler+0xc8/0xf0
[77522.303539][ T9734] el0_sync+0x1b4/0x1c0
The analysis shows that when we iterate the swap_avail_heads list, we get
node A, but before we access node A, node A is maybe deleted, and by the time
we actually access node A, it no longer exists, as follows:
CPU1 thread1 CPU2 thread2
plist_for_each_entry_safe()
get si->avail_lists[node] from swap_avail_heads
remove si->avail_lists[node] from swap_avail_heads
plist_requeue(&si->avail_lists[node])
BUG_ON(plist_node_empty(node)); // trigger
Due to when we use vendor hook of get_swap_page, the get_swap_pages() function
is overridden, use our own spin_lock to protect when iterate swap_avail_heads
list, but now use native swap_avail_lock spin_lock protect when the
swap_avail_heads list to add and delete nodes, so there will be concurrent
access.
So add vendor hook of add/delete/iterate node for avail_list, in this way, we
can use our own spin_lock to protect the swap_avail_heads list to add, delete
and iterate node.
Due to enable_swap_info function to call vendor hook of add_to_avail_list,
need first init swap_avail_heads, so also add vendor hook of
swap_avail_heads_init.
Due to the vendor hook of __cgroup_throttle_swaprate need to call
blkcg_schedule_throttle function, so export it also.
Bug: 225795494
Change-Id: I03107cbda6310fa7ae85e41b8cf1fa8225cafe78
Signed-off-by: Lincheng Yang <lincheng.yang@transsion.com>
Suggested-by: Bing Han <bing.han@transsion.com>
We need to abort the reclaim/compaction by sending
signal(such as SIGUSR2) to the reclaim thread, or
just abort when cpu-usage is too-high or free-mem is enough.
Bug: 289987875
Change-Id: I4b637cbd2b37235eec27a985a9b5b95598247c59
Signed-off-by: shenjiangjiang <shenjiangjiang@oppo.com>
(cherry picked from commit 024628cc9203cbd4f8471d98435b3a3d6f85764d)
In some situations, we want to decrease readaround size for better
performance. So we add this hook.
Bug: 288216516
Change-Id: If2f5f75976c99ff1f82ce29d370f9216926055ab
Signed-off-by: Oven <liyangouwen1@oppo.com>
memory allocations
We add these hooks to avoid key threads blocked in memory allocation
path.
-android_vh_free_unref_page_bypass ----We create a memory pool for the key threads. This hook determines whether a page should be free to the pool or to buddy freelist. It works with a existing hook `android_vh_alloc_pages_reclaim_bypass`, which takes pages out of the pool.
-android_vh_kvmalloc_node_use_vmalloc ----For key threads, we perfer not to run into direct reclaim. So we clear __GFP_DIRECT_RECLAIM flag. For threads which are not that important, we perfer use vmalloc.
-android_vh_should_alloc_pages_retry ----Before key threads run into direct reclaim, we want to retry with a lower watermark.
-android_vh_unreserve_highatomic_bypass ----We want to keep more highatomic pages when unreserve them to avoid highatomic allocation failures.
-android_vh_pageset_update ----We found the default per-cpu pageset is quite few in smartphones with large ram size. This hook is used to increase it to reduce zone->lock contentions.
-android_vh_rmqueue_bulk_bypass ----We found sometimes when key threads run into rmqueue_bulk, it took several milliseconds spinning at zone->lock or filling per-cpu pages. We use this hook to take pages from the mempool mentioned above, rather than grab zone->lock and fill a batch of pages to per-cpu.
Bug: 288216516
Change-Id: I1656032d6819ca627723341987b6094775bc345f
Signed-off-by: Oven <liyangouwen1@oppo.com>
1. android_vh_dm_bufio_shrink_scan_bypass
To adjust dm_bufio-buffer shrinker's policy in some cases.
2. cleanup_old_buffers_bypass
To adjust the policy of dm_bufio-buffer periodic eviction
in some cases.
Bug: 281467813
Signed-off-by: Peifeng Li <lipeifeng@oppo.com>
Change-Id: I29a9d91d18d2e279170533db83b59cfc3b17ebe2
This reverts commit cc51dcbc60.
Reason for revert: The vendor hooks were reverted but they are needed.
Bug: 243629905
Signed-off-by: xiaofeng <xiaofeng5@xiaomi.com>
Signed-off-by: Suren Baghdasaryan <surenb@google.com>
Change-Id: I4b2eab1a9bf3bbbb200f9d09f2c57fb4d9f2c143
Fix Bug: scheduling while atomic
In these vendor hooks, we will perform schedule due to competion. This will
lead to kernel exception.
To solve this problem, we need to add these restrcted hooks to replace the
former regular vendor hooks.
Bug: 234214858
Signed-off-by: Bing Han <bing.han@transsion.com>
Change-Id: I151125a7119a91d1339d4790a68a6a4796d673e3
Add a hook in madvise_cold_or_pageout_pte_range() to allow
vendor modules to influence the shared pages reclaim.
Bug: 242678506
Change-Id: I269a385b59f7291c2e96478674bb3d05f94584cb
Signed-off-by: Pavankumar Kondeti <quic_pkondeti@quicinc.com>
This reverts commit dec2f52d08.
The hooks android_vh_alloc_pages_reclaim_bypass and
android_vh_alloc_pages_failure_bypass are not used by any vendor, so
remove it to help with merge issues with future LTS releases.
If this is needed by any real user, it can easily be reverted to add it
back and then the symbol should be added to the abi list at the same
time to prevent it from being removed again later.
Bug: 203756332
Bug: 243629905
Cc: xiaofeng <xiaofeng5@xiaomi.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@google.com>
Change-Id: Id313f6971e0b5437fcfc1ed3f8d4c56706217133
This reverts commit f06daa5a0b.
The hook android_vh_ra_tuning_max_page is not used by any vendor, so
remove it to help with merge issues with future LTS releases.
If this is needed by any real user, it can easily be reverted to add it
back and then the symbol should be added to the abi list at the same
time to prevent it from being removed again later.
Bug: 203756332
Bug: 229839032
Cc: liang zhang <liang.zhang@transsion.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@google.com>
Change-Id: Id9211dfd9e1fa19d2ccb14302c60f0d55579f59d
This reverts commit eb99e6d80e
The hook android_vh_free_pages is deleted, due to the symbol is
not added to the abi list. The symbol is added to the abi list in
patch:2183484. This patch is to add the hook android_vh_free_pages
again.
Bug: 234214858
Bug: 203756332
Cc: Greg Kroah-Hartman <gregkh@google.com>
Signed-off-by: Bing Han <bing.han@transsion.com>
Change-Id: I2c97ea4d310e2004b94d891678127c17f7b07c93
This reverts commit: d0590b99c9
The hooks android_vh_init_swap_info_struct and android_vh_alloc_si
are deleted, due to the symbols are not added to the abi list. The
symbols are added to the abi list in patch:2183484. This patch is to
add the hooks android_vh_init_swap_info_struct and android_vh_alloc_si
again.
Bug: 234214858
Bug: 203756332
Cc: Greg Kroah-Hartman <gregkh@google.com>
Signed-off-by: Bing Han <bing.han@transsion.com>
Change-Id: Id5524a726d213c5eab55570fd28d28da978974e7
This reverts commit 86be1a3d9f
The hook android_vh_si_swapinfo is deleted, due to the symbol
is not added to the abi list. The symbol is added to the abi
list in patch:2183484. This patch is to add the hook
android_vh_si_swapinfo again.
Bug: 234214858
Bug: 203756332
Cc: Greg Kroah-Hartman <gregkh@google.com>
Signed-off-by: Bing Han <bing.han@transsion.com>
Change-Id: Ifd1e05f44ac04b67816618139badd5c2ee786b50
This reverts commit ed2b11d639.
The hook android_vh_si_swapinfo is not used by any vendor, so remove it
to help with merge issues with future LTS releases.
If this is needed by any real user, it can easily be reverted to add it
back and then the symbol should be added to the abi list at the same
time to prevent it from being removed again later.
Bug: 203756332
Bug: 234214858
Cc: Bing Han <bing.han@transsion.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@google.com>
Change-Id: Ib474a0911dd97d54d2f086258e9d53ddd3451967
This reverts commit 667f0d71dc.
The hooks android_vh_init_swap_info_struct and android_vh_alloc_si are
not used by any vendor, so remove it to help with merge issues with
future LTS releases.
If this is needed by any real user, it can easily be reverted to add it
back and then the symbol should be added to the abi list at the same
time to prevent it from being removed again later.
Bug: 203756332
Bug: 234214858
Cc: Bing Han <bing.han@transsion.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@google.com>
Change-Id: If01c284ebf15e804e7cf34e854b7db0d5b68ae1b
This reverts commit 01680ae117.
The hook android_vh_free_pages is not used by any vendor, so remove it
to help with merge issues with future LTS releases.
If this is needed by any real user, it can easily be reverted to add it
back and then the symbol should be added to the abi list at the same
time to prevent it from being removed again later.
Bug: 203756332
Bug: 234214858
Cc: Bing Han <bing.han@transsion.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@google.com>
Change-Id: I36d4bfb83e1605c6fd1f9ff8dcd39cdbcdef8760
Add hooks to support trylock in rmaplock when reclaiming in kswapd or
direct_reclaim, in order to avoid wait lock for a long time.
- android_vh_handle_failed_page_trylock
- android_vh_page_trylock_set
- android_vh_page_trylock_clear
- android_vh_page_trylock_get_result
- android_vh_do_page_trylock
Bug: 240003372
Signed-off-by: Peifeng Li <lipeifeng@oppo.com>
Change-Id: I0f605b35ae41f15b3ca7bc72cd5f003175c318a5
Support two hooks as follows to protect multi-mapcount pages in kernel:
- trace_android_vh_page_should_be_protect
- trace_android_vh_mapped_page_try_sorthead
Bug: 236578020
Signed-off-by: Peifeng Li <lipeifeng@oppo.com>
Change-Id: I688aceabf17d9de2feac7c3ad7144d307de6ef29
Support five hooks as follows to account
the amount of multi-mapped pages in kernel:
- android_vh_show_mapcount_pages
- android_vh_do_traversal_lruvec
- android_vh_update_page_mapcount
- android_vh_add_page_to_lrulist
- android_vh_del_page_from_lrulist
Bug: 236578020
Signed-off-by: Peifeng Li <lipeifeng@oppo.com>
Change-Id: Ia2c7015aab442be7dbb496b8b630b9dff59ab935
Provide a vendor hook to remove additional fields when remove_vm_area
for slab/vmalloc memory leak debugging.
Bug: 240869642
Signed-off-by: Peifeng Li <lipeifeng@oppo.com>
Change-Id: Iafecd7c6e75cdc2df0e77ae105283590d8852f74
Introduce PROTECT_TRACE_INCLUDE_PATH to allow trace/hooks/mm.h to be
included from a header file. When trace/hooks/mm.h is included from a
header file PROTECT_TRACE_INCLUDE_PATH should be defined before the
inclusion so that the existing TRACE_INCLUDE_PATH gets preserved.
Bug: 236578020
Signed-off-by: Suren Baghdasaryan <surenb@google.com>
Change-Id: Ia5890ba51675de41ff3ea4ab38bcb458202df5fb
Since android_vh_alloc_pages_slowpath is revert by
commit e09000ee19 ("Revert half of "ANDROID: vendor_hooks: Add hooks
for memory when debug""). re-add hooks here to measure the duration
Bug: 182443489
Signed-off-by: liuhailong <liuhailong@oppo.com>
Change-Id: Ie4534047105d8409623692cc3811b55d9ddbd17d
Add vendor hook android_vh_set_shmem_page_flag in shmem_writepage to
set a flag in page_ext, which indicates that this page is a shmem page,
to be used in android_vh_get_swap_page. The shared page should not be
reclaimed to the extended memory, ie, the specified swap location.
Bug: 234214858
Signed-off-by: Bing Han <bing.han@transsion.com>
Change-Id: I33a9007c88b4d8aab3da044c8a05eb45d7e74f3a
Provide a vendor hook android_vh_free_pages to clear the
information in struct page_ext, when the page is freed.
Bug: 234214858
Signed-off-by: Bing Han <bing.han@transsion.com>
Change-Id: Iac8e3a72f59f8d3ae16dbc93d94034fe4b627d61
Provide a vendor hook android_vh_swap_slot_cache_active to
pass the active status of swap_slots_cache. This status
will be used in the process of reclaiming the pages that
is required to be reclaimed to a specified swap location.
Bug: 234214858
Signed-off-by: Bing Han <bing.han@transsion.com>
Change-Id: I8211760e0f37fe4a514f6ceaae9993925da8cd6d
Three vendor hooks are provided to extend struct swap_slots_cache.
The extended data are used to record the information of the
specified reclaimed swap location:
1) android_vh_alloc_swap_slot_cache, replace the function
alloc_swap_slot_cache adding allocation of the extension
of struct swap_slots_cache;
2) android_vh_drain_slots_cache_cpu, replace the function
drain_slots_cache_cpu adding the initialization of the
extension of struct swap_slots_cache;
3) android_vh_get_swap_page, replace the function get_swap_page,
according to the reclaimed location information of the page,
get the the swap page from the specified swap location;
Bug: 234214858
Signed-off-by: Bing Han <bing.han@transsion.com>
Change-Id: I3bce6e8cf255df1d879b7c4022d54981cce7c273
Provide a vendor hook android_vh_si_swapinf to replace the
process of updating nr_to_be_unused. When the page is swapped
to a specified swap location, nr_to_be_unused should not be
updated. Because the specified swap is regarded as a reserved
extended memory.
Bug: 234214858
Signed-off-by: Bing Han <bing.han@transsion.com>
Change-Id: Ie41caec345658589bf908fb0f96d038d1fba21f3
Two vendor hooks are added to extend the struct swap_info_struct:
android_vh_alloc_si, extend the allocation of struct swap_info_struct,
adding data to record the information of specified reclaimed location;
android_vh_init_swap_info_struct, adding initializing the extension of
struct swap_info_struct;
Bug: 234214858
Signed-off-by: Bing Han <bing.han@transsion.com>
Change-Id: I0e1d8e38ba7dfd52b609b1c14eb78f8b0ef0f9e6
When the page is unused, a vendor hook android_vh_unuse_swap_page
should be called to specify that the page should not be swapped
to the specified swap location any more.
Bug: 234214858
Signed-off-by: Bing Han <bing.han@transsion.com>
Change-Id: I3fc3675020517f7cc69c76a06150dfb2380dae21
Provide a vendor hook to replace the function free_swap_slot,
adding the free_swap_slot process of pages swapped to the
specified swap location(i.e., the reserved expended memory)
Bug: 234214858
Signed-off-by: Bing Han <bing.han@transsion.com>
Change-Id: Idd6d0007e64d56d556d1234a8b931fce06031809
The specified swap is regarded as reserved extended memory.
So nr_swap_pages and total_swap_pages should not be affected
by the specified swap.
Provide a vendor hook android_vh_account_swap_pages to replace
the updating process of nr_swap_pages and total_swap_pages.
When the page is swapped to the specified swap location,
nr_swap_pages and total_swap_pages should not be updated.
Bug: 234214858
Signed-off-by: Bing Han <bing.han@transsion.com>
Change-Id: Ib8dfb355d190399a037b9d9eda478a81c436e224
Add android_vh_page_referenced_one_end at the end of function
page_referenced_one to update the status that whether the page
need to be reclaimed to a specified swap location.
Bug: 234214858
Signed-off-by: Bing Han <bing.han@transsion.com>
Change-Id: Ia06a229956328ef776da5d163708dcb011a327fb
android_vh_count_pswpin, Update the write I/O statistics of the swap;
android_vh_count_pswpout, Update the read I/O statistics of the swap;
android_vh_count_swpout_vm_event, Replace count_swpout_vm_event with
adding updating the I/O statistics of the swap;
Bug: 234214858
Signed-off-by: Bing Han <bing.han@transsion.com>
Change-Id: I4eb69db59fe2d822555a508c2f0c6cd5ca9083d1
Provide a vendor hook to copy the status whether the page need to be
reclaimed to a specified swap location.
Bug: 234214858
Signed-off-by: Bing Han <bing.han@transsion.com>
Change-Id: I1a451b40407718900b56de6ed17b7fd5ef56da01
android_vh_waiting_for_page_migration: provide a vendor hook
to force not to reclaim the page under migration to a specified
swap location, until the migration is finished.
Bug: 234214858
Signed-off-by: Bing Han <bing.han@transsion.com>
Change-Id: Iceeae91cbd912d9c44d7eac25f1299bbff547388
android_vh_handle_pte_fault_end: after handle_pte_fault, update
the information that whether this page need to be reclaimed to
a swap location.
Bug: 234214858
Signed-off-by: Bing Han <bing.han@transsion.com>
Change-Id: I0ceb02422fc858ed96fbb47e220bf96bdc8fa68c
android_vh_swapin_add_anon_rmap: after add pte mapping to an anonymous
page durning do_swap_page, update the status that whether this page
need to be reclaimed to a swap location, according to the information
of vm_fault.
Bug: 234214858
Signed-off-by: Bing Han <bing.han@transsion.com>
Change-Id: I8a2d603102c315323817e6c9366db9b0da878344
android_vh_cow_user_page: when copy a page to a new page, set the
status that whether the new page should be reclaimed to a specified
swap location, according to the information of vm_fault.
Bug: 234214858
Signed-off-by: Bing Han <bing.han@transsion.com>
Change-Id: Ie445c7b034ca176ec1e8fd1cd67c88581bf9ddf4
Provide a vendor hook android_vh_page_isolated_for_reclaim to
process whether the page should be reclaimed to a specified
swap(i.e., the expanded memory).
This strategy will take into account the state of the current
process/application, resource usage, and other information.
Bug: 234214858
Signed-off-by: Bing Han <bing.han@transsion.com>
Change-Id: Id80a377c87bea13922e7b23963b050ab37ba0cb0
Tune ReadAhead size for better memory usage and performance.
accordding to Read-Ahead Efficiency on Mobile Devices: Observation,
Characterization, and Optimization form IEEE
Bug: 229839032
Change-Id: I91656bde5e616e181fd7557554d55e7ce1858136
Signed-off-by: liang zhang <liang.zhang@transsion.com>
Provide a vendor hook to skip cma-pages to add in pcplist when
free_unref_page_commit.
The patch is revelant to skip drain_all_pages in alloc_contig_range,
the revelant hooks is android_vh_cma_drain_all_pages_bypass
which is to avoid to delay in drain pcppages when drain_all_pages.
In most case, pcp->high is small so that free-pages with other mt_types
can also fill with pcplist full.
Bug: 224732340
Signed-off-by: Peifeng Li <lipeifeng@oppo.com>
Change-Id: Ifdeeed9f8934d87671ec3fa6787a02675b993082
