hwpoison.rst 5.9 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184
  1. .. hwpoison:
  2. ========
  3. hwpoison
  4. ========
  5. What is hwpoison?
  6. =================
  7. Upcoming Intel CPUs have support for recovering from some memory errors
  8. (``MCA recovery``). This requires the OS to declare a page "poisoned",
  9. kill the processes associated with it and avoid using it in the future.
  10. This patchkit implements the necessary infrastructure in the VM.
  11. To quote the overview comment::
  12. High level machine check handler. Handles pages reported by the
  13. hardware as being corrupted usually due to a 2bit ECC memory or cache
  14. failure.
  15. This focusses on pages detected as corrupted in the background.
  16. When the current CPU tries to consume corruption the currently
  17. running process can just be killed directly instead. This implies
  18. that if the error cannot be handled for some reason it's safe to
  19. just ignore it because no corruption has been consumed yet. Instead
  20. when that happens another machine check will happen.
  21. Handles page cache pages in various states. The tricky part
  22. here is that we can access any page asynchronous to other VM
  23. users, because memory failures could happen anytime and anywhere,
  24. possibly violating some of their assumptions. This is why this code
  25. has to be extremely careful. Generally it tries to use normal locking
  26. rules, as in get the standard locks, even if that means the
  27. error handling takes potentially a long time.
  28. Some of the operations here are somewhat inefficient and have non
  29. linear algorithmic complexity, because the data structures have not
  30. been optimized for this case. This is in particular the case
  31. for the mapping from a vma to a process. Since this case is expected
  32. to be rare we hope we can get away with this.
  33. The code consists of a the high level handler in mm/memory-failure.c,
  34. a new page poison bit and various checks in the VM to handle poisoned
  35. pages.
  36. The main target right now is KVM guests, but it works for all kinds
  37. of applications. KVM support requires a recent qemu-kvm release.
  38. For the KVM use there was need for a new signal type so that
  39. KVM can inject the machine check into the guest with the proper
  40. address. This in theory allows other applications to handle
  41. memory failures too. The expection is that near all applications
  42. won't do that, but some very specialized ones might.
  43. Failure recovery modes
  44. ======================
  45. There are two (actually three) modes memory failure recovery can be in:
  46. vm.memory_failure_recovery sysctl set to zero:
  47. All memory failures cause a panic. Do not attempt recovery.
  48. early kill
  49. (can be controlled globally and per process)
  50. Send SIGBUS to the application as soon as the error is detected
  51. This allows applications who can process memory errors in a gentle
  52. way (e.g. drop affected object)
  53. This is the mode used by KVM qemu.
  54. late kill
  55. Send SIGBUS when the application runs into the corrupted page.
  56. This is best for memory error unaware applications and default
  57. Note some pages are always handled as late kill.
  58. User control
  59. ============
  60. vm.memory_failure_recovery
  61. See sysctl.txt
  62. vm.memory_failure_early_kill
  63. Enable early kill mode globally
  64. PR_MCE_KILL
  65. Set early/late kill mode/revert to system default
  66. arg1: PR_MCE_KILL_CLEAR:
  67. Revert to system default
  68. arg1: PR_MCE_KILL_SET:
  69. arg2 defines thread specific mode
  70. PR_MCE_KILL_EARLY:
  71. Early kill
  72. PR_MCE_KILL_LATE:
  73. Late kill
  74. PR_MCE_KILL_DEFAULT
  75. Use system global default
  76. Note that if you want to have a dedicated thread which handles
  77. the SIGBUS(BUS_MCEERR_AO) on behalf of the process, you should
  78. call prctl(PR_MCE_KILL_EARLY) on the designated thread. Otherwise,
  79. the SIGBUS is sent to the main thread.
  80. PR_MCE_KILL_GET
  81. return current mode
  82. Testing
  83. =======
  84. * madvise(MADV_HWPOISON, ....) (as root) - Poison a page in the
  85. process for testing
  86. * hwpoison-inject module through debugfs ``/sys/kernel/debug/hwpoison/``
  87. corrupt-pfn
  88. Inject hwpoison fault at PFN echoed into this file. This does
  89. some early filtering to avoid corrupted unintended pages in test suites.
  90. unpoison-pfn
  91. Software-unpoison page at PFN echoed into this file. This way
  92. a page can be reused again. This only works for Linux
  93. injected failures, not for real memory failures. Once any hardware
  94. memory failure happens, this feature is disabled.
  95. Note these injection interfaces are not stable and might change between
  96. kernel versions
  97. corrupt-filter-dev-major, corrupt-filter-dev-minor
  98. Only handle memory failures to pages associated with the file
  99. system defined by block device major/minor. -1U is the
  100. wildcard value. This should be only used for testing with
  101. artificial injection.
  102. corrupt-filter-memcg
  103. Limit injection to pages owned by memgroup. Specified by inode
  104. number of the memcg.
  105. Example::
  106. mkdir /sys/fs/cgroup/mem/hwpoison
  107. usemem -m 100 -s 1000 &
  108. echo `jobs -p` > /sys/fs/cgroup/mem/hwpoison/tasks
  109. memcg_ino=$(ls -id /sys/fs/cgroup/mem/hwpoison | cut -f1 -d' ')
  110. echo $memcg_ino > /debug/hwpoison/corrupt-filter-memcg
  111. page-types -p `pidof init` --hwpoison # shall do nothing
  112. page-types -p `pidof usemem` --hwpoison # poison its pages
  113. corrupt-filter-flags-mask, corrupt-filter-flags-value
  114. When specified, only poison pages if ((page_flags & mask) ==
  115. value). This allows stress testing of many kinds of
  116. pages. The page_flags are the same as in /proc/kpageflags. The
  117. flag bits are defined in include/linux/kernel-page-flags.h and
  118. documented in Documentation/admin-guide/mm/pagemap.rst
  119. * Architecture specific MCE injector
  120. x86 has mce-inject, mce-test
  121. Some portable hwpoison test programs in mce-test, see below.
  122. References
  123. ==========
  124. http://halobates.de/mce-lc09-2.pdf
  125. Overview presentation from LinuxCon 09
  126. git://git.kernel.org/pub/scm/utils/cpu/mce/mce-test.git
  127. Test suite (hwpoison specific portable tests in tsrc)
  128. git://git.kernel.org/pub/scm/utils/cpu/mce/mce-inject.git
  129. x86 specific injector
  130. Limitations
  131. ===========
  132. - Not all page types are supported and never will. Most kernel internal
  133. objects cannot be recovered, only LRU pages for now.
  134. ---
  135. Andi Kleen, Oct 2009