On a related note and FWIW OOM kills can be reduced or entirely mitigated by a combination of some kernel settings and confining applications to CGroups. Some of the generalized and basic settings are:
- vm.overcommit_ratio should be set to 0 on non development machines. Some applications are greedy and do not play well when memory is constrained so that requires working with the application developers to improve their memory management.
- vm.min_free_kbytes which should be set based on a formula. Redhat had a decent formula but DBA's don't like when it is used because they want every last bit of ram so that battle is left to the sysadmins. It becomes a circular argument and ever-increasing memory purchases.
- vm.admin_reserve_kbytes and vm.user_reserve_kbytes also based on a formula but each company will have to come up with their own based on how fast memory is allocated by their in-house applications.
- vm.swappiness and vm.vfs_cache_pressure will also vary by the role of the server and intended uses but having higher cache pressure can reduce the risk of a race condition that leads to OOM on systems with low free memory.
- vm.compaction_proactiveness settings can cause lag and race conditions on systems under high memory pressure. This can be exacerbated by /sys/kernel/mm/transparent_hugepage/enabled and /sys/kernel/mm/transparent_hugepage/defrag settings among other related settings but that's a long story with a lot of variable conditions especially on systems with TB's of ram and long uptimes.
There is no magic bullet or "correct" setting. All of these will vary by roles, usage, etc... I will leave it to the reader to research each setting. This will require some load testing that puts each system under real world high memory pressure. Oh and if someone starts trying to solve this with swap, just slowly back away and don't make any sudden movements. Distract them and run. I managed over 50K servers with anywhere from 96GB to 3TB of ram and the only OOM's were caused by operations teams that had full control over memory allocation of java and wanted to use every last KB of physical ram installed not factoring in memory outside of the heap. None of these servers had any swap as it would have to be encrypted and nobody wanted to deal with that. OOM's are almost always PEBKAC.
On a related note and FWIW OOM kills can be reduced or entirely mitigated by a combination of some kernel settings and confining applications to CGroups. Some of the generalized and basic settings are:
- vm.overcommit_ratio should be set to 0 on non development machines. Some applications are greedy and do not play well when memory is constrained so that requires working with the application developers to improve their memory management.
- vm.min_free_kbytes which should be set based on a formula. Redhat had a decent formula but DBA's don't like when it is used because they want every last bit of ram so that battle is left to the sysadmins. It becomes a circular argument and ever-increasing memory purchases.
- vm.admin_reserve_kbytes and vm.user_reserve_kbytes also based on a formula but each company will have to come up with their own based on how fast memory is allocated by their in-house applications.
- vm.swappiness and vm.vfs_cache_pressure will also vary by the role of the server and intended uses but having higher cache pressure can reduce the risk of a race condition that leads to OOM on systems with low free memory.
- vm.compaction_proactiveness settings can cause lag and race conditions on systems under high memory pressure. This can be exacerbated by /sys/kernel/mm/transparent_hugepage/enabled and /sys/kernel/mm/transparent_hugepage/defrag settings among other related settings but that's a long story with a lot of variable conditions especially on systems with TB's of ram and long uptimes.
There is no magic bullet or "correct" setting. All of these will vary by roles, usage, etc... I will leave it to the reader to research each setting. This will require some load testing that puts each system under real world high memory pressure. Oh and if someone starts trying to solve this with swap, just slowly back away and don't make any sudden movements. Distract them and run. I managed over 50K servers with anywhere from 96GB to 3TB of ram and the only OOM's were caused by operations teams that had full control over memory allocation of java and wanted to use every last KB of physical ram installed not factoring in memory outside of the heap. None of these servers had any swap as it would have to be encrypted and nobody wanted to deal with that. OOM's are almost always PEBKAC.