From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from fout-b8-smtp.messagingengine.com (fout-b8-smtp.messagingengine.com [202.12.124.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9BC1E4A64DB for ; Wed, 16 Sep 2026 09:32:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=202.12.124.151 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789551136; cv=none; b=mw7vHQW9PXiyFyOScrLDF4wO9tgUe9q7SymvWXLksypOc55VQbqWy8NjQHr0aZjaUC1Q2ZgjXCze5D1G8hbqb/TQg4NOUslj+h8ZHiQ652Zmvl9P07Bj2I7nhXJDpD1BBAh8HWjJhHRM708XenTpjQAG9xLDWphYOpHhPGHytu0= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789551136; c=relaxed/simple; bh=TKB6T8VtIMFCYymtjKb5LK/4Whb6YCWyLf1ODb2k4yM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=sfz3B8cdarBm/fjyF0hiIv+9FwiF8NB72YWxidki7pdhYp1gGZjjSuABv/hqU7K0V6/HZcQwlB68gvb+c+7Y6tTcl5/0g8nN0zRS5sM0wQbdPq5h6TyWodGOzFjoEKMxFhijcl5tVekKlJ5sTeITr4sTexE+ec11j+N8yd4N6GM= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name; spf=pass smtp.mailfrom=shutemov.name; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b=FZrwW60a; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=djeJHwrz; arc=none smtp.client-ip=202.12.124.151 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=shutemov.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b="FZrwW60a"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="djeJHwrz" Received: from phl-compute-05.internal (phl-compute-05.internal [10.202.2.45]) by mailfout.stl.internal (Postfix) with ESMTP id A1EE61D00084; Wed, 16 Sep 2026 05:31:59 -0400 (EDT) Received: from phl-frontend-03 ([10.202.2.162]) by phl-compute-05.internal (MEProxy); Wed, 16 Sep 2026 05:32:00 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov.name; h=cc:cc:content-transfer-encoding:content-type:date:date:from :from:in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm2; t=1789551119; x= 1789637519; bh=avMkfLVeU1+RrFVq7dqGuSLkfIFTUHtIkOYpYaIIbGQ=; b=F ZrwW60a1m5sQGSEnlfYHRzgoLkG3D5P3kbyYZypS4G3D1Y/sjXhpr+7XMk6fPFFc CB4jWSRtzSCkTUPHawA5jvvTbt131TRr573B2NaRjDQxcbLXdfsIAhB3E1DefPle bngTFW6EQS2zIxkMX6HgVaKLMT6uF4JOlDePTF/tK+W5IjXtNjjAG5FHX/c/Y+MP +qKbycwVvjWyhQ3Rw9SR4JwG78QD9Uqa0blpocyFxp6zXe8cx5vHRR/kGKaGYoPf ZwjzbG/oFXjNTSDozQ3OpU5DU0dAYIwpaJbr5RGlXlOZ+lj0lRDYz+T0LDczyUP3 h29BdUPt4biFOn7V9GDtA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; t=1789551119; x=1789637519; bh=a vMkfLVeU1+RrFVq7dqGuSLkfIFTUHtIkOYpYaIIbGQ=; b=djeJHwrzEVStD9qyQ ZxZOoZpNzwDjC1N9/9C41R68YCIusyuxXLL62GTE3BfGRGQr+3uCX6Z4STy9KQgZ v1lnlGMWKyIOq5DNRt0o6NO7AJos5JR6akblkt63RY0m2/jk1fOxdl4uYMapOiMS EWpgiMkf5L5wOTAot3xc8PABlHDOn+HNeBCV9fwtC2h+az2Shs6HJTRjkLyWZwtg LNuouYC245nIGuua80XJZ+2L+YWJ82WtIWuT25sxY4bAnKLevX+sQOW+0P6TZbWT e6bnxQlO0rCrOLdht18tOgpWLzFGgINRMWKJzIwDgzVt6cZmEN190Ope9RS+iBBR JTLyg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTEOpyDQEfQMjLyldXBPAs1OiAaYjuycA/zprc2XA0sZE/gLY53LzRl6wisjmJsNcj shXh7L3amtO3W7Dj4gJ1brhV8efHSvbbvvisq8qNCVrU/hVVqWW2fJ8EDoeL9EfJt8J4fa r63UA+S0A+UN3c16swmwWrTj5sK0gOy7RYH1CN+xe4D6TTQ1ncSXBFkweWqSBfk0S52Say Dntsl8DI3TlZwS0igZd1iO+ltDo9n3qHIbwcYcvcbVUlu6Rxag/XK6s48+7ZTTYlI2aJe4 hkL/jN/KxntOAh39VLdXeraBW39brB+PkStvfnz0wMlwqVlAhF3nsCCzTv21LQWeSF/iTb MYrnHQOmaofpdATEhW+iKqNMndVUsZkNXro4u1ND3HQ+RLuEHTdkLWeBbFKVfegYNYqNlt X0JHLDrQLD/yJxqXyO2trj9CcN/tXL6neVs2190GRVKkh/LvWIaouJinauIO1qOlOd9xe3 jTDXmMnXkyZwXmnXqPrLOnKd/ybmml26i+ErYaHUo20TmQcmlcVzeB+TjmfO9B95GMfhsm duaCze2HijSX0CfsyWMswWKk99NTMjR5xQp6haMLPl2xKWlu2BgRSz4hIqt2zAWsJuTo3Q asDhRbeixb8zNswBS2kN4z6ROulafxWeWuICeF4opQx4KFDkTKvNGnlsyXYQ X-ME-Proxy: Feedback-ID: ie3994620:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 16 Sep 2026 05:31:58 -0400 (EDT) From: Kiryl Shutsemau To: Andrew Morton , David Hildenbrand , Lorenzo Stoakes , Zi Yan , Baolin Wang Cc: "Kiryl Shutsemau (Meta)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, "Liam R. Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Vlastimil Babka , Jann Horn Subject: [PATCH v3 04/12] mm/collapse: add collapse.h for the collapse interface Date: Wed, 16 Sep 2026 10:31:31 +0100 Message-ID: <20260916093145.4022188-5-kirill@shutemov.name> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260916093145.4022188-1-kirill@shutemov.name> References: <20260916093145.4022188-1-kirill@shutemov.name> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: "Kiryl Shutsemau (Meta)" khugepaged.c holds both the users of collapse and the machinery that performs it. The daemon's scan loop, the sysfs tunables, MADV_COLLAPSE's entry point and the collapse itself all sit in one file and reach into each other freely. Nothing marks where a user ends and the engine begins. Start drawing that line. Add mm/collapse.h for what the two sides have to agree on: - enum scan_result - what the engine hands back; - struct collapse_control - the state a request carries. And two constants move with them: - KHUGEPAGED_MAX_PTES_LIMIT -> COLLAPSE_MAX_PTES_LIMIT; - KHUGEPAGED_MIN_MTHP_ORDER -> COLLAPSE_MIN_MTHP_ORDER. Neither is a fact about the daemon, so both lose the KHUGEPAGED_ prefix. No functional change. Assisted-by: LLM Reviewed-by: Zi Yan Reviewed-by: Baolin Wang Signed-off-by: Kiryl Shutsemau (Meta) --- MAINTAINERS | 1 + mm/collapse.h | 64 +++++++++++++++++++++++++++++++++++++++ mm/khugepaged.c | 79 ++++++++----------------------------------------- 3 files changed, 78 insertions(+), 66 deletions(-) create mode 100644 mm/collapse.h diff --git a/MAINTAINERS b/MAINTAINERS index 35262aa3e67e..36315972156b 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -17434,6 +17434,7 @@ F: Documentation/admin-guide/mm/transhuge.rst F: include/linux/huge_mm.h F: include/linux/khugepaged.h F: include/trace/events/huge_memory.h +F: mm/collapse.h F: mm/huge_memory.c F: mm/khugepaged.c F: mm/mm_slot.h diff --git a/mm/collapse.h b/mm/collapse.h new file mode 100644 index 000000000000..b115034d9018 --- /dev/null +++ b/mm/collapse.h @@ -0,0 +1,64 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __MM_COLLAPSE_H +#define __MM_COLLAPSE_H + +#include +#include +#include +#include + +#define COLLAPSE_MAX_PTES_LIMIT (HPAGE_PMD_NR - 1) +#define COLLAPSE_MIN_MTHP_ORDER 2 + +enum scan_result { + SCAN_FAIL, + SCAN_SUCCEED, + SCAN_NO_PTE_TABLE, + SCAN_PMD_MAPPED, + SCAN_EXCEED_NONE_PTE, + SCAN_EXCEED_SWAP_PTE, + SCAN_EXCEED_SHARED_PTE, + SCAN_PTE_NON_PRESENT, + SCAN_PTE_UFFD, + SCAN_PTE_MAPPED_HUGEPAGE, + SCAN_LACK_REFERENCED_PAGE, + SCAN_PAGE_NULL, + SCAN_SCAN_ABORT, + SCAN_PAGE_COUNT, + SCAN_PAGE_LRU, + SCAN_PAGE_LOCK, + SCAN_PAGE_ANON, + SCAN_PAGE_LAZYFREE, + SCAN_PAGE_COMPOUND, + SCAN_ANY_PROCESS, + SCAN_VMA_NULL, + SCAN_VMA_CHECK, + SCAN_ADDRESS_RANGE, + SCAN_DEL_PAGE_LRU, + SCAN_ALLOC_HUGE_PAGE_FAIL, + SCAN_CGROUP_CHARGE_FAIL, + SCAN_TRUNCATED, + SCAN_PAGE_HAS_PRIVATE, + SCAN_STORE_FAILED, + SCAN_COPY_MC, + SCAN_PAGE_FILLED, + SCAN_PAGE_DIRTY_OR_WRITEBACK, +}; + +struct collapse_control { + bool is_khugepaged; + + /* Num pages scanned per node */ + u32 node_load[MAX_NUMNODES]; + + /* Num pages scanned (see khugepaged_pages_to_scan) */ + unsigned int progress; + + /* nodemask for allocation fallback */ + nodemask_t alloc_nmask; + + /* Each bit marks a PTE the scan accepted as a collapse source */ + DECLARE_BITMAP(eligible_ptes, MAX_PTRS_PER_PTE); +}; + +#endif /* __MM_COLLAPSE_H */ diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 081f705cfca2..8889f75cf45f 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -26,44 +26,10 @@ #include #include +#include "collapse.h" #include "internal.h" -#include "page_alloc.h" #include "mm_slot.h" - -enum scan_result { - SCAN_FAIL, - SCAN_SUCCEED, - SCAN_NO_PTE_TABLE, - SCAN_PMD_MAPPED, - SCAN_EXCEED_NONE_PTE, - SCAN_EXCEED_SWAP_PTE, - SCAN_EXCEED_SHARED_PTE, - SCAN_PTE_NON_PRESENT, - SCAN_PTE_UFFD, - SCAN_PTE_MAPPED_HUGEPAGE, - SCAN_LACK_REFERENCED_PAGE, - SCAN_PAGE_NULL, - SCAN_SCAN_ABORT, - SCAN_PAGE_COUNT, - SCAN_PAGE_LRU, - SCAN_PAGE_LOCK, - SCAN_PAGE_ANON, - SCAN_PAGE_LAZYFREE, - SCAN_PAGE_COMPOUND, - SCAN_ANY_PROCESS, - SCAN_VMA_NULL, - SCAN_VMA_CHECK, - SCAN_ADDRESS_RANGE, - SCAN_DEL_PAGE_LRU, - SCAN_ALLOC_HUGE_PAGE_FAIL, - SCAN_CGROUP_CHARGE_FAIL, - SCAN_TRUNCATED, - SCAN_PAGE_HAS_PRIVATE, - SCAN_STORE_FAILED, - SCAN_COPY_MC, - SCAN_PAGE_FILLED, - SCAN_PAGE_DIRTY_OR_WRITEBACK, -}; +#include "page_alloc.h" #define CREATE_TRACE_POINTS #include @@ -91,7 +57,6 @@ static DECLARE_WAIT_QUEUE_HEAD(khugepaged_wait); * * Note that these are only respected if collapse was initiated by khugepaged. */ -#define KHUGEPAGED_MAX_PTES_LIMIT (HPAGE_PMD_NR - 1) unsigned int khugepaged_max_ptes_none __read_mostly; static unsigned int khugepaged_max_ptes_swap __read_mostly; static unsigned int khugepaged_max_ptes_shared __read_mostly; @@ -101,24 +66,6 @@ static DEFINE_READ_MOSTLY_HASHTABLE(mm_slots_hash, MM_SLOTS_HASH_BITS); static struct kmem_cache *mm_slot_cache __ro_after_init; -#define KHUGEPAGED_MIN_MTHP_ORDER 2 - -struct collapse_control { - bool is_khugepaged; - - /* Num pages scanned per node */ - u32 node_load[MAX_NUMNODES]; - - /* Num pages scanned (see khugepaged_pages_to_scan) */ - unsigned int progress; - - /* nodemask for allocation fallback */ - nodemask_t alloc_nmask; - - /* Each bit marks a PTE the scan accepted as a collapse source */ - DECLARE_BITMAP(eligible_ptes, MAX_PTRS_PER_PTE); -}; - /** * struct khugepaged_scan - cursor for scanning * @mm_head: the head of the mm list to scan @@ -267,7 +214,7 @@ static ssize_t max_ptes_none_store(struct kobject *kobj, unsigned long max_ptes_none; err = kstrtoul(buf, 10, &max_ptes_none); - if (err || max_ptes_none > KHUGEPAGED_MAX_PTES_LIMIT) + if (err || max_ptes_none > COLLAPSE_MAX_PTES_LIMIT) return -EINVAL; khugepaged_max_ptes_none = max_ptes_none; @@ -292,7 +239,7 @@ static ssize_t max_ptes_swap_store(struct kobject *kobj, unsigned long max_ptes_swap; err = kstrtoul(buf, 10, &max_ptes_swap); - if (err || max_ptes_swap > KHUGEPAGED_MAX_PTES_LIMIT) + if (err || max_ptes_swap > COLLAPSE_MAX_PTES_LIMIT) return -EINVAL; khugepaged_max_ptes_swap = max_ptes_swap; @@ -318,7 +265,7 @@ static ssize_t max_ptes_shared_store(struct kobject *kobj, unsigned long max_ptes_shared; err = kstrtoul(buf, 10, &max_ptes_shared); - if (err || max_ptes_shared > KHUGEPAGED_MAX_PTES_LIMIT) + if (err || max_ptes_shared > COLLAPSE_MAX_PTES_LIMIT) return -EINVAL; khugepaged_max_ptes_shared = max_ptes_shared; @@ -378,19 +325,19 @@ static unsigned int collapse_max_ptes_none(struct collapse_control *cc, if (is_pmd_order(order)) return max_ptes_none; /* - * for mTHP collapse with the sysctl value set to KHUGEPAGED_MAX_PTES_LIMIT, + * for mTHP collapse with the sysctl value set to COLLAPSE_MAX_PTES_LIMIT, * scale the maximum number of PTEs to the order of the collapse. */ - if (max_ptes_none == KHUGEPAGED_MAX_PTES_LIMIT) + if (max_ptes_none == COLLAPSE_MAX_PTES_LIMIT) return (1 << order) - 1; /* - * For mTHP collapse of values other than 0 or KHUGEPAGED_MAX_PTES_LIMIT, + * For mTHP collapse of values other than 0 or COLLAPSE_MAX_PTES_LIMIT, * emit a warning and return 0. */ if (max_ptes_none) pr_warn_once("mTHP collapse does not support max_ptes_none" " values other than 0 or %u, defaulting to 0.\n", - KHUGEPAGED_MAX_PTES_LIMIT); + COLLAPSE_MAX_PTES_LIMIT); return 0; } @@ -476,7 +423,7 @@ int __init khugepaged_init(void) return -ENOMEM; khugepaged_pages_to_scan = HPAGE_PMD_NR * 8; - khugepaged_max_ptes_none = KHUGEPAGED_MAX_PTES_LIMIT; + khugepaged_max_ptes_none = COLLAPSE_MAX_PTES_LIMIT; khugepaged_max_ptes_swap = HPAGE_PMD_NR / 8; khugepaged_max_ptes_shared = HPAGE_PMD_NR / 2; @@ -1571,8 +1518,8 @@ static enum scan_result mthp_collapse(struct mm_struct *mm, * any smaller order enabled. When at the smallest order * we must always move to the next offset. */ - if (order > KHUGEPAGED_MIN_MTHP_ORDER && - (enabled_orders & GENMASK(order - 1, 0))) { + if (order > COLLAPSE_MIN_MTHP_ORDER && + (enabled_orders & GENMASK(order - 1, 0))) { order--; continue; } @@ -1636,7 +1583,7 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm, * is then checked again in mthp_collapse() for each attempted order. */ if (enabled_orders != BIT(HPAGE_PMD_ORDER)) - max_ptes_none = KHUGEPAGED_MAX_PTES_LIMIT; + max_ptes_none = COLLAPSE_MAX_PTES_LIMIT; pte = pte_offset_map_lock(mm, pmd, start_addr, &ptl); if (!pte) { -- 2.54.0