Greg Thelen 0610c25daa
memcg: fix dirty page migration ...
The problem starts with a file backed dirty page which is charged to a
memcg. Then page migration is used to move oldpage to newpage.
Migration:
- copies the oldpage's data to newpage
- clears oldpage.PG_dirty
- sets newpage.PG_dirty
- uncharges oldpage from memcg
- charges newpage to memcg
Clearing oldpage.PG_dirty decrements the charged memcg's dirty page
count.
However, because newpage is not yet charged, setting newpage.PG_dirty
does not increment the memcg's dirty page count. After migration
completes newpage.PG_dirty is eventually cleared, often in
account_page_cleaned(). At this time newpage is charged to a memcg so
the memcg's dirty page count is decremented which causes underflow
because the count was not previously incremented by migration. This
underflow causes balance_dirty_pages() to see a very large unsigned
number of dirty memcg pages which leads to aggressive throttling of
buffered writes by processes in non root memcg.
This issue:
- can harm performance of non root memcg buffered writes.
- can report too small (even negative) values in
memory.stat[(total_)dirty] counters of all memcg, including the root.
To avoid polluting migrate.c with #ifdef CONFIG_MEMCG checks, introduce
page_memcg() and set_page_memcg() helpers.
Test:
0) setup and enter limited memcg
mkdir /sys/fs/cgroup/test
echo 1G > /sys/fs/cgroup/test/memory.limit_in_bytes
echo $$ > /sys/fs/cgroup/test/cgroup.procs
1) buffered writes baseline
dd if=/dev/zero of=/data/tmp/foo bs=1M count=1k
sync
grep ^dirty /sys/fs/cgroup/test/memory.stat
2) buffered writes with compaction antagonist to induce migration
yes 1 > /proc/sys/vm/compact_memory &
rm -rf /data/tmp/foo
dd if=/dev/zero of=/data/tmp/foo bs=1M count=1k
kill %
sync
grep ^dirty /sys/fs/cgroup/test/memory.stat
3) buffered writes without antagonist, should match baseline
rm -rf /data/tmp/foo
dd if=/dev/zero of=/data/tmp/foo bs=1M count=1k
sync
grep ^dirty /sys/fs/cgroup/test/memory.stat
(speed, dirty residue)
unpatched patched
1) 841 MB/s 0 dirty pages 886 MB/s 0 dirty pages
2) 611 MB/s -33427456 dirty pages 793 MB/s 0 dirty pages
3) 114 MB/s -33427456 dirty pages 891 MB/s 0 dirty pages
Notice that unpatched baseline performance (1) fell after
migration (3): 841 -> 114 MB/s. In the patched kernel, post
migration performance matches baseline.
Fixes: c4843a7593gthelen@google.com >
Reported-by: Dave Hansen <dave.hansen@intel.com >
Acked-by: Michal Hocko <mhocko@suse.com >
Acked-by: Johannes Weiner <hannes@cmpxchg.org >
Cc: <stable@vger.kernel.org > [4.2+]
Signed-off-by: Andrew Morton <akpm@linux-foundation.org >
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org >
2015-10-01 21:42:35 -04:00
2015-09-04 09:14:20 -07:00
2015-08-18 09:08:47 +03:00
2015-07-12 21:13:22 +02:00
2015-09-17 20:14:27 +03:00
2015-09-01 13:00:04 -07:00
2015-06-25 11:49:31 +03:00
2015-05-26 09:18:29 +05:30
2015-05-22 18:58:44 +09:00
2015-09-04 15:49:32 -07:00
2015-08-04 17:03:52 -07:00
2015-08-08 12:50:40 +01:00
2015-07-11 17:27:36 -07:00
2015-09-10 16:42:49 -07:00
2015-09-04 11:46:02 -07:00
2015-09-09 08:33:31 -07:00
2015-09-09 08:33:31 -07:00
2015-08-27 14:50:52 +02:00
2015-08-10 23:07:05 -04:00
2015-09-02 16:32:56 -07:00
2015-04-04 13:26:52 -04:00
2015-06-23 14:08:54 -07:00
2015-04-04 12:47:04 -04:00
2015-04-04 12:52:06 -04:00
2015-07-31 15:01:14 +01:00
2015-06-03 14:13:41 +09:00
2015-06-01 15:48:12 +02:00
2015-09-08 16:33:16 -07:00
2015-05-23 22:43:28 +02:00
2015-04-22 08:00:41 +10:00
2015-08-30 14:40:11 +01:00
2015-05-20 15:05:39 -07:00
2015-06-10 15:10:26 +08:00
2015-06-19 15:18:28 +02:00
2015-09-01 13:00:04 -07:00
2015-09-16 20:44:47 +01:00
2015-06-21 21:52:24 +02:00
2015-09-17 18:01:28 -04:00
2015-05-13 12:04:55 -05:00
2015-08-18 09:58:45 -07:00
2015-03-18 17:27:03 +01:00
2015-04-15 16:35:22 -07:00
2015-03-26 15:13:07 +00:00
2015-09-01 03:41:19 +02:00
2015-03-25 20:28:11 -04:00
2015-04-22 17:06:52 +02:00
2015-03-27 13:44:43 +00:00
2015-08-05 13:38:07 +01:00
2015-04-22 08:00:42 +10:00
2015-08-03 12:01:54 -04:00
2015-07-23 18:00:42 -07:00
2015-08-12 11:58:59 +02:00
2015-08-06 16:17:25 -04:00
2015-08-20 14:10:23 -07:00
2015-07-02 08:46:00 -06:00
2015-09-24 16:48:52 -04:00
2015-06-23 15:47:35 +01:00
2015-07-27 15:01:05 +02:00
2015-06-21 21:55:33 +02:00
2015-08-19 14:26:02 -07:00
2015-08-05 09:38:08 +02:00
2015-08-05 09:38:08 +02:00
2015-09-03 12:28:30 -07:00
2015-08-18 15:49:36 -07:00
2015-06-01 14:35:56 -06:00
2015-09-19 18:57:09 -07:00
2015-06-24 17:49:41 -07:00
2015-05-19 08:39:11 +02:00
2015-08-09 22:50:06 -07:00
2015-06-10 23:33:58 -07:00
2015-07-02 01:32:44 -04:00
2015-05-24 12:31:33 -07:00
2015-04-15 16:35:22 -07:00
2015-09-10 18:56:14 -07:00
2015-09-16 11:51:12 -04:00
2015-08-25 14:19:29 -04:00
2015-04-14 16:49:03 -07:00
2015-08-24 16:49:11 -07:00
2015-05-06 11:58:51 +01:00
2015-07-17 16:39:53 -07:00
2015-09-14 11:00:55 +02:00
2015-05-19 16:12:32 +02:00
2015-04-14 16:49:04 -07:00
2015-04-15 16:35:17 -07:00
2015-07-17 16:39:53 -07:00
2015-06-25 17:00:38 -07:00
2015-06-25 17:00:38 -07:00
2015-08-12 11:58:58 +02:00
2015-07-17 16:39:53 -07:00
2015-05-10 19:15:52 +02:00
2015-06-25 17:00:38 -07:00
2015-07-07 10:59:04 +02:00
2015-07-07 10:59:04 +02:00
2015-08-05 13:30:16 -07:00
2015-07-15 13:30:38 +01:00
2015-05-04 21:27:52 -07:00
2015-07-17 16:39:53 -07:00
2015-07-22 09:58:02 +02:00
2015-09-16 02:17:49 +02:00
2015-08-28 15:14:54 +02:00
2015-05-28 11:05:20 +09:30
2015-05-26 15:26:43 +02:00
2015-05-30 22:42:24 -07:00
2015-09-04 16:54:41 -07:00
2015-08-17 16:53:53 +08:00
2015-09-08 15:35:28 -07:00
2015-07-17 16:39:53 -07:00
2015-03-23 16:52:26 -04:00
2015-07-20 18:44:50 +01:00
2015-03-30 17:09:41 +09:00
2015-09-11 14:23:29 +09:00
2015-08-13 12:31:57 -06:00
2015-09-01 19:45:46 -07:00
2015-05-13 14:35:57 +05:30
2015-03-09 13:05:47 +01:00
2015-08-20 12:01:03 +05:30
2015-09-08 15:35:28 -07:00
2015-06-12 11:33:52 +02:00
2015-06-25 09:06:57 +02:00
2015-06-29 10:34:42 -07:00
2015-06-10 08:01:20 -06:00
2015-04-14 16:49:05 -07:00
2015-08-17 12:14:53 -07:00
2015-08-10 11:48:55 +09:00
2015-08-21 22:45:13 -07:00
2015-03-25 15:07:05 +11:00
2015-08-20 10:20:11 +03:00
2015-07-22 15:27:32 -07:00
2015-07-30 11:13:21 -07:00
2015-06-24 17:49:45 -07:00
2015-04-09 11:39:55 -05:00
2015-09-08 15:35:28 -07:00
2015-04-02 14:28:53 +01:00
2015-08-14 16:50:36 -07:00
2015-08-07 22:59:34 -05:00
2015-09-05 20:34:28 -07:00
2015-07-24 13:58:14 -04:00
2015-04-03 23:23:51 +02:00
2015-09-04 16:54:41 -07:00
2015-07-17 08:41:53 -06:00
2015-09-08 15:35:28 -07:00
2015-05-31 11:40:14 +09:00
2015-05-12 10:46:53 +02:00
2015-05-19 08:39:11 +02:00
2015-07-13 14:18:07 -07:00
2015-06-01 14:33:35 +02:00
2015-05-19 08:39:14 +02:00
2015-04-02 18:46:20 +02:00
2015-06-19 15:18:28 +02:00
2015-04-24 15:36:50 +02:00
2015-09-08 15:35:28 -07:00
2015-09-08 15:35:28 -07:00
2015-03-16 21:45:54 +11:00
2015-05-02 09:54:30 +03:00
2015-08-05 11:44:28 -07:00
2015-08-24 14:05:19 +02:00
2015-05-05 13:40:42 -06:00
2015-08-14 17:49:53 +02:00
2015-05-31 13:40:53 +02:00
2015-06-15 17:23:03 -07:00
2015-06-23 06:14:04 -07:00
2015-05-11 10:50:17 -04:00
2015-06-01 16:50:52 -07:00
2015-08-28 13:28:47 -07:00
2015-06-15 19:49:22 -07:00
2015-06-24 02:15:54 -07:00
2015-09-16 11:51:12 -04:00
2015-07-05 23:59:14 -04:00
2015-08-12 16:23:33 +02:00
2015-04-22 17:06:52 +02:00
2015-08-10 23:07:05 -04:00
2015-08-27 19:40:58 -04:00
2015-04-18 12:32:59 -07:00
2015-07-17 16:39:53 -07:00
2015-04-15 16:35:22 -07:00
2015-09-03 15:02:28 -05:00
2015-08-13 21:27:19 -07:00
2015-03-23 10:55:21 +01:00
2015-09-16 15:47:51 +02:00
2015-07-02 22:34:38 +02:00
2015-09-16 15:47:51 +02:00
2015-07-30 00:14:38 +02:00
2015-09-16 15:47:51 +02:00
2015-06-25 12:06:45 +02:00
2015-09-03 12:52:19 -07:00
2015-04-03 12:52:29 -04:00
2015-09-01 14:04:50 -07:00
2015-09-15 07:12:06 +02:00
2015-03-26 23:51:36 +01:00
2015-08-22 14:54:55 +02:00
2015-04-21 09:44:55 -03:00
2015-07-17 16:39:53 -07:00
2015-08-18 15:49:15 -07:00
2015-09-10 13:29:01 -07:00
2015-07-28 08:50:42 +01:00
2015-06-24 17:49:46 -07:00
2015-09-10 13:29:01 -07:00
2015-07-17 16:39:53 -07:00
2015-08-04 10:16:54 +02:00
2015-04-15 16:35:17 -07:00
2015-09-04 16:54:41 -07:00
2015-05-13 10:19:35 +02:00
2015-09-06 16:27:10 +02:00
2015-05-26 12:40:17 +02:00
2015-03-31 09:45:50 -06:00
2015-03-30 11:55:48 -07:00
2015-05-04 11:05:55 -07:00
2015-06-19 10:03:12 +02:00
2015-03-24 11:52:08 +10:30
2015-07-15 11:47:24 -04:00
2015-04-29 17:17:17 -05:00
2015-08-28 23:40:05 -04:00
2015-08-17 18:39:45 -04:00
2015-05-19 23:58:43 +02:00
2015-08-12 11:59:08 +02:00
2015-06-24 15:09:40 -07:00
2015-07-13 13:31:58 -04:00
2015-07-28 13:18:19 +10:00
2015-06-11 22:19:45 +05:30
2015-08-10 14:29:27 +05:30
2015-05-28 12:21:08 +02:00
2015-05-14 22:35:13 -04:00
2015-08-03 17:30:00 -07:00
2015-09-08 15:35:28 -07:00
2015-09-10 13:29:01 -07:00
2015-08-27 19:40:58 -04:00
2015-04-15 16:35:18 -07:00
2015-09-09 17:19:14 -07:00
2015-04-14 16:49:05 -07:00
2015-08-05 10:35:49 -07:00
2015-09-08 15:35:28 -07:00
2015-06-24 17:49:41 -07:00
2015-10-01 21:42:35 -04:00
2015-07-17 16:39:53 -07:00
2015-09-10 13:29:01 -07:00
2015-09-08 14:35:59 -07:00
2015-08-07 15:03:42 +02:00
2015-07-05 23:59:14 -04:00
2015-03-12 18:46:08 -07:00
2015-07-01 10:49:25 -07:00
2015-04-18 11:20:31 -04:00
2015-06-16 14:35:06 +08:00
2015-07-21 10:39:05 -07:00
2015-07-30 00:14:39 +02:00
2015-05-15 01:10:45 -04:00
2015-06-25 04:20:04 -04:00
2015-08-06 23:51:30 -07:00
2015-05-12 18:43:55 -04:00
2015-09-23 14:32:50 -07:00
2015-07-30 13:37:42 +02:00
2015-06-18 21:14:31 +02:00
2015-05-14 01:10:05 -04:00
2015-08-27 16:35:51 -07:00
2015-09-02 16:32:56 -07:00
2015-09-09 21:43:22 -07:00
2015-08-17 13:22:49 -05:00
2015-08-27 19:45:27 -04:00
2015-08-17 13:32:56 -05:00
2015-06-10 18:32:40 -04:00
2015-09-07 12:36:13 -04:00
2015-05-05 17:10:11 -07:00
2015-09-08 12:28:10 -07:00
2015-09-07 15:17:08 -04:00
2015-09-07 15:17:09 -04:00
2015-08-18 11:56:13 -06:00
2015-08-05 13:43:44 -07:00
2015-08-05 13:43:12 -07:00
2015-07-27 08:23:27 -05:00
2015-05-09 17:11:25 +05:30
2015-07-01 19:40:18 -07:00
2015-07-28 13:55:36 +02:00
2015-06-24 11:18:48 +10:00
2015-07-30 00:14:37 +02:00
2015-03-11 17:56:28 -04:00
2015-03-09 23:50:20 -04:00
2015-08-25 11:29:55 -05:00
2015-08-26 01:46:39 +02:00
2015-09-01 09:59:20 +10:00
2015-03-06 12:39:48 +02:00
2015-09-08 15:35:28 -07:00
2015-05-08 12:27:59 +02:00
2015-09-10 13:29:01 -07:00
2015-09-10 13:29:01 -07:00
2015-07-17 16:39:54 -07:00
2015-09-10 13:29:01 -07:00
2015-09-08 15:35:28 -07:00
2015-07-04 19:36:06 -07:00
2015-06-01 07:08:18 +09:00
2015-07-17 16:39:53 -07:00
2015-08-16 13:26:01 -03:00
2015-04-08 16:25:25 -05:00
2015-04-09 14:20:11 -05:00
2015-08-13 15:59:58 -05:00
2015-09-08 15:35:28 -07:00
2015-05-29 07:39:34 +10:00
2015-07-14 17:43:56 -04:00
2015-08-15 13:52:10 +02:00
2015-08-09 22:50:05 -07:00
2015-04-12 21:03:31 +02:00
2015-08-31 14:48:02 -07:00
2015-09-24 23:04:53 -07:00
2015-06-16 14:12:37 -04:00
2015-05-12 23:55:37 +02:00
2015-08-29 01:54:43 +02:00
2015-09-09 02:20:39 +02:00
2015-07-28 08:50:41 +01:00
2015-07-21 23:14:22 +02:00
2015-05-30 02:17:15 +02:00
2015-05-20 01:56:31 +02:00
2015-03-18 15:54:27 +01:00
2015-06-19 01:18:14 +02:00
2015-08-27 19:40:59 -04:00
2015-03-18 22:39:16 +01:00
2015-09-10 13:29:01 -07:00
2015-06-10 16:15:54 +02:00
2015-08-03 12:21:24 +02:00
2015-09-10 13:29:01 -07:00
2015-08-13 16:58:29 -07:00
2015-08-07 14:37:04 +02:00
2015-08-03 12:33:39 +01:00
2015-03-23 14:06:10 +11:00
2015-03-31 12:01:19 -04:00
2015-07-15 11:52:52 -07:00
2015-08-17 15:40:20 +02:00
2015-07-31 19:13:33 +01:00
2015-05-28 10:47:45 -05:00
2015-03-18 21:55:08 +01:00
2015-07-23 20:59:10 +02:00
2015-06-10 19:14:04 +08:00
2015-05-28 11:32:04 +09:30
2015-05-28 11:32:06 +09:30
2015-05-28 11:32:04 +09:30
2015-05-27 12:58:04 -07:00
2015-09-20 20:50:06 -07:00
2015-07-22 15:26:58 -07:00
2015-07-22 15:26:58 -07:00
2015-04-15 16:35:23 -07:00
2015-09-04 17:22:10 +01:00
2015-06-17 09:57:12 +03:00
2015-09-01 09:56:58 +02:00
2015-05-16 18:08:26 -04:00
2015-05-26 15:23:23 +02:00
2015-09-04 16:54:41 -07:00
2015-06-25 01:13:43 +02:00
2015-06-23 06:56:18 -07:00
2015-08-24 14:28:01 -06:00
2015-09-16 11:51:12 -04:00
2015-05-24 12:13:36 -07:00
2015-07-15 11:52:54 -07:00
2015-09-20 17:00:26 -07:00
2015-09-11 15:21:34 -07:00
2015-07-01 10:49:25 -07:00
2015-07-23 17:43:30 -07:00
2015-05-06 22:27:00 +02:00
2015-05-10 19:06:38 +02:00
2015-07-16 10:30:55 -07:00
2015-08-05 08:48:00 +05:30
2015-09-24 14:09:13 -07:00
2015-09-04 16:54:41 -07:00
2015-04-20 09:08:49 -07:00
2015-09-04 16:54:41 -07:00
2015-06-15 19:49:22 -07:00
2015-04-11 15:53:35 -04:00
2015-09-01 08:40:25 -07:00
2015-04-14 09:50:27 -07:00
2015-06-25 17:00:38 -07:00
2015-07-29 00:13:25 -07:00
2015-08-03 12:21:27 +02:00
2015-09-10 13:29:01 -07:00
2015-06-25 17:00:40 -07:00
2015-05-11 15:06:43 +08:00
2015-09-08 15:35:28 -07:00
2015-09-08 15:35:28 -07:00
2015-09-11 15:21:34 -07:00
2015-07-01 10:36:39 -05:00
2015-07-01 10:36:45 -05:00
2015-06-25 17:00:39 -07:00
2015-05-23 01:22:35 -04:00
2015-09-14 07:39:46 -07:00
2015-08-05 13:24:12 -07:00
2015-09-02 10:33:22 +02:00
2015-08-17 11:25:28 -07:00
2015-06-12 11:15:49 +02:00
2015-08-20 21:13:22 +02:00
2015-06-19 15:18:28 +02:00
2015-04-22 17:06:49 +02:00
2015-05-27 15:22:15 +02:00
2015-08-06 15:29:14 -03:00
2015-04-08 09:39:56 -04:00
2015-08-03 15:24:43 -07:00
2015-07-23 18:37:31 -07:00
2015-07-22 15:27:29 -07:00
2015-06-15 16:02:52 -07:00
2015-08-28 16:27:27 -07:00
2015-03-18 22:00:34 -04:00
2015-05-14 17:55:51 -07:00
2015-04-11 22:29:44 -04:00
2015-07-31 10:38:06 +02:00
2015-04-28 12:48:57 +02:00
2015-08-18 09:59:12 -07:00
2015-09-04 16:54:41 -07:00
2015-05-05 17:10:11 -07:00
2015-08-12 17:01:01 +01:00
2015-04-14 18:06:47 -07:00
2015-04-07 11:14:38 -06:00
2015-06-01 15:48:54 +02:00
2015-06-01 15:48:54 +02:00
2015-04-13 21:04:16 +09:30
2015-04-15 12:41:14 +09:30
2015-03-12 18:46:08 -07:00
2015-06-12 17:26:57 -07:00
2015-06-01 15:48:54 +02:00
2015-09-22 15:09:53 -07:00
2015-09-04 16:54:41 -07:00
2015-03-24 09:48:14 -07:00
2015-08-17 15:48:24 -04:00
2015-06-02 08:40:20 -06:00
2015-09-08 15:35:28 -07:00
2015-09-10 13:29:01 -07:00
2015-09-08 15:35:28 -07:00
0610c25daa