Revision 3de7d4f25a7438f09fef4e71ef111f1805cd8e7c authored by Roman Gushchin on 24 January 2021, 05:01:07 UTC, committed by Linus Torvalds on 24 January 2021, 17:20:52 UTC
Imran Khan reported a 16% regression in hackbench results caused by the
commit f2fe7b09a52b ("mm: memcg/slab: charge individual slab objects
instead of pages"). The regression is noticeable in the case of a
consequent allocation of several relatively large slab objects, e.g.
skb's. As soon as the amount of stocked bytes exceeds PAGE_SIZE,
drain_obj_stock() and __memcg_kmem_uncharge() are called, and it leads
to a number of atomic operations in page_counter_uncharge().
The corresponding call graph is below (provided by Imran Khan):
|__alloc_skb
| |
| |__kmalloc_reserve.isra.61
| | |
| | |__kmalloc_node_track_caller
| | | |
| | | |slab_pre_alloc_hook.constprop.88
| | | obj_cgroup_charge
| | | | |
| | | | |__memcg_kmem_charge
| | | | | |
| | | | | |page_counter_try_charge
| | | | |
| | | | |refill_obj_stock
| | | | | |
| | | | | |drain_obj_stock.isra.68
| | | | | | |
| | | | | | |__memcg_kmem_uncharge
| | | | | | | |
| | | | | | | |page_counter_uncharge
| | | | | | | | |
| | | | | | | | |page_counter_cancel
| | | |
| | | |
| | | |__slab_alloc
| | | | |
| | | | |___slab_alloc
| | | | |
| | | |slab_post_alloc_hook
Instead of directly uncharging the accounted kernel memory, it's
possible to refill the generic page-sized per-cpu stock instead. It's a
much faster operation, especially on a default hierarchy. As a bonus,
__memcg_kmem_uncharge_page() will also get faster, so the freeing of
page-sized kernel allocations (e.g. large kmallocs) will become faster.
A similar change has been done earlier for the socket memory by the
commit 475d0487a2ad ("mm: memcontrol: use per-cpu stocks for socket
memory uncharging").
Link: https://lkml.kernel.org/r/20210106042239.2860107-1-guro@fb.com
Fixes: f2fe7b09a52b ("mm: memcg/slab: charge individual slab objects instead of pages")
Signed-off-by: Roman Gushchin <guro@fb.com>
Reported-by: Imran Khan <imran.f.khan@oracle.com>
Tested-by: Imran Khan <imran.f.khan@oracle.com>
Reviewed-by: Shakeel Butt <shakeelb@google.com>
Reviewed-by: Michal Koutn <mkoutny@suse.com>
Cc: Michal Koutný <mkoutny@suse.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
1 parent d3921cb
.gitignore
# SPDX-License-Identifier: GPL-2.0-only
#
# NOTE! Don't add files that are generated in specific
# subdirectories here. Add them in the ".gitignore" file
# in that subdirectory instead.
#
# NOTE! Please use 'git ls-files -i --exclude-standard'
# command after changing this file, to see if there are
# any tracked files which get ignored after the change.
#
# Normal rules (sorted alphabetically)
#
.*
*.a
*.asn1.[ch]
*.bin
*.bz2
*.c.[012]*.*
*.dt.yaml
*.dtb
*.dtb.S
*.dwo
*.elf
*.gcno
*.gz
*.i
*.ko
*.lex.c
*.ll
*.lst
*.lz4
*.lzma
*.lzo
*.mod
*.mod.c
*.o
*.o.*
*.patch
*.s
*.so
*.so.dbg
*.su
*.symtypes
*.tab.[ch]
*.tar
*.xz
*.zst
Module.symvers
modules.builtin
modules.order
#
# Top-level generic files
#
/tags
/TAGS
/linux
/vmlinux
/vmlinux.32
/vmlinux.symvers
/vmlinux-gdb.py
/vmlinuz
/System.map
/Module.markers
/modules.builtin.modinfo
/modules.nsdeps
#
# RPM spec file (make rpm-pkg)
#
/*.spec
#
# Debian directory (make deb-pkg)
#
/debian/
#
# Snap directory (make snap-pkg)
#
/snap/
#
# tar directory (make tar*-pkg)
#
/tar-install/
#
# We don't want to ignore the following even if they are dot-files
#
!.clang-format
!.cocciconfig
!.get_maintainer.ignore
!.gitattributes
!.gitignore
!.mailmap
#
# Generated include files
#
/include/config/
/include/generated/
/include/ksym/
/arch/*/include/generated/
# stgit generated dirs
patches-*
# quilt's files
patches
series
# cscope files
cscope.*
ncscope.*
# gnu global files
GPATH
GRTAGS
GSYMS
GTAGS
# id-utils files
ID
*.orig
*~
\#*#
#
# Leavings from module signing
#
extra_certificates
signing_key.pem
signing_key.priv
signing_key.x509
x509.genkey
# Kconfig presets
/all.config
/alldef.config
/allmod.config
/allno.config
/allrandom.config
/allyes.config
# Kconfig savedefconfig output
/defconfig
# Kdevelop4
*.kdev4
# Clang's compilation database file
/compile_commands.json
# Documentation toolchain
sphinx_*/
Computing file changes ...
