From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from fhigh-b4-smtp.messagingengine.com (fhigh-b4-smtp.messagingengine.com [202.12.124.155]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 67D594A92C8 for ; Wed, 16 Sep 2026 09:32:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=202.12.124.155 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789551162; cv=none; b=CjfTVxFrqW4N+H2BNBnsnXck4BF7jUGrlXK1dwHmjQQnzQwIIukfydQ19dMsWL8P06xB4U75v98mZcejku1NwiaecwdM5J/sTI4Wj2AzsDtdF7D2MhXzAdtwJ8YCrK0x614wqANJ+poIOq2vlTbFz6gLXr+5zWa1pvh9myX0odM= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789551162; c=relaxed/simple; bh=/WjhvGACiwNe9/LGX3y5/h7X87Y+zDJYwy+7DsnslnQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=BJim/YYCTiqx4T89R6ZWjUwtFGBXcUC9FZfYw667y2GWOTjxIbLMSXMp9lqxPs8LJimEv5pEtzp+YuXqUy7QuyXhfDzQLpDDwAuceQ76/ftRAcN9BvFYJrf/SM3zvIEkd94Af6PJ6bphixb/Zyn0EiGsbaolq3Oylo/F/aWPy9I= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name; spf=pass smtp.mailfrom=shutemov.name; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b=h+nDuZ4L; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=DamH363K; arc=none smtp.client-ip=202.12.124.155 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=shutemov.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b="h+nDuZ4L"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="DamH363K" Received: from phl-compute-03.internal (phl-compute-03.internal [10.202.2.43]) by mailfhigh.stl.internal (Postfix) with ESMTP id C033B7A007B; Wed, 16 Sep 2026 05:32:13 -0400 (EDT) Received: from phl-frontend-03 ([10.202.2.162]) by phl-compute-03.internal (MEProxy); Wed, 16 Sep 2026 05:32:14 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov.name; h=cc:cc:content-transfer-encoding:content-type:date:date:from :from:in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm2; t=1789551133; x= 1789637533; bh=PRWFDwvM0fZr06tYfSZoOVoDNXjF/NbOi9PwgZXG13s=; b=h +nDuZ4L3d7Exo1VnWVHkFTiWmBJ3LgIXQqeNIIwptbCr3rocTTogmeSBRfhparYF 8kgDFCvJL7mBpQZbetk0u9hmVR6g5yZn50P8MISvcSghAJqOQA0voEnJT4/QgyhA GDj831STE2yzmfNUzOohrZzAZZyQzVOEwC7USAAY8Lr6KwgPrJ13zFKFFdQLm8p8 Yb259Q3GPN8AWvUYzvymY2q1ClRPBjA0sILyAiu0+O8zvqqbwC//b17J8h3m4kIC Cr4wjVwJM2U8G//TZoNzgQkHENcWcrEUOxLwrC52SlRwgWsDXJAAn0g8zgyw+dVX QMbQYjvg3wJAXw+3Q2M0Q== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; t=1789551133; x=1789637533; bh=P RWFDwvM0fZr06tYfSZoOVoDNXjF/NbOi9PwgZXG13s=; b=DamH363KmwhSZ5zJu fhU6H/Ey+B7XjK9/lw6VNU7q7qLYVoT4I4fUzQBv/orFWgEdqDdXd+KZurLR/3JV NxsGojIyz8eGDOxEtArmG85Ut+JFBZPugFv1xJSQGTZmWZVQMyhiUldxb+Hh22pt IopaLgTUWX+llHnaqMo1+S7+DAo8YYDHltvyrUOpNQZdViGlTms4zQ5KhalJvaYm YKUzYulz+bw+eZ/ovtFDvl5VjCg5/f2j7NM3Pm0eiJ0wwFgkjhCHKoFh1ml8qd5/ isye1zUR9hdNyHV6WLUSWj2EjEd508yrriVlWflkVEEc06+bZWEhCzx7LmZ43AyL PmG7Q== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTEOpyDQEfQMjLyldXBPAs1OiAaYjuycA/zprc2XA0sZE/gLY53LzRl6wisjmJsNcj shXh7L3amtO3W7Dj4gJ1brhV8efHSvbbvvisq8qNCVrU/hVVqWW2fJ8EDoeL9EfJt8J4fa r63UA+S0A+UN3c16swmwWrTj5sK0gOy7RYH1CN+xe4D6TTQ1ncSXBFkweWqSBfk0S52Say Dntsl8DI3TlZwS0igZd1iO+ltDo9n3qHIbwcYcvcbVUlu6Rxag/XK6s48+7ZTTYlI2aJe4 hkL/jN/KxntOAh39VLdXeraBW39brB+PkStvfnz0wMlwqVlAhF3nsCCzTv21LQWeSF/iHT pyMAuIQkGFQ08rzLATOXG16m6WQQDf0qrhK3S8NG5J8zh2mNM5qcdfG+xe89hRG/m+7INR ADrck8zKABvA0aAWvVdCuEeBi2oCpNFHiAk2Khx6KyEamwPIIWWx1aCyjtcXhsnR1VCXvK jOfLqdyuZxn83ODbS+0qUPAqOyBYa449EcVKzMv4hGVxgKwrNVR4sBWB7R/NCH/wZHyXB7 AWwvIeSxHVuN/dy3GL/fBK1LiTnhq1VJmRsV1IMAH0TLses6J0QUD6xT6cFIZlQl8915KE pMjHkmd7973FH6bjWFDUCbqDKgjeWhNiAp/maWL8JO1SbJWp0+skjlFQ/FUA X-ME-Proxy: Feedback-ID: ie3994620:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 16 Sep 2026 05:32:12 -0400 (EDT) From: Kiryl Shutsemau To: Andrew Morton , David Hildenbrand , Lorenzo Stoakes , Zi Yan , Baolin Wang Cc: "Kiryl Shutsemau (Meta)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, "Liam R. Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Vlastimil Babka , Jann Horn Subject: [PATCH v3 10/12] mm/collapse: work out the orders a VMA allows once per VMA Date: Wed, 16 Sep 2026 10:31:37 +0100 Message-ID: <20260916093145.4022188-11-kirill@shutemov.name> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260916093145.4022188-1-kirill@shutemov.name> References: <20260916093145.4022188-1-kirill@shutemov.name> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: "Kiryl Shutsemau (Meta)" The scan asked collapse_possible_orders() for every PTE table, for an answer that is a property of the VMA. Both callers walk a VMA a table at a time, so let them work it out once and pass the mask in. It is only good while the lock that produced it is held, so madvise_collapse() takes it again after every collapse. The mask is then sampled once per VMA rather than once per table. A thp enabled knob written during a walk takes effect one VMA later, and cannot widen a collapse: hugepage_vma_revalidate() tests the order again under the lock the collapse retakes. Assisted-by: LLM Reviewed-by: Zi Yan Reviewed-by: Baolin Wang Signed-off-by: Kiryl Shutsemau (Meta) --- mm/khugepaged.c | 32 ++++++++++++++++++-------------- 1 file changed, 18 insertions(+), 14 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 9e6b2af6519e..12cb67d8df32 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1549,12 +1549,12 @@ static enum scan_result mthp_collapse(struct mm_struct *mm, unsigned long addres } static enum scan_result collapse_scan_anon_pmd(struct vm_area_struct *vma, - unsigned long start_addr, struct collapse_control *cc) + unsigned long start_addr, struct collapse_control *cc, + unsigned long enabled_orders) { const unsigned int max_ptes_shared = collapse_max_ptes_shared(cc, HPAGE_PMD_ORDER); const unsigned int max_ptes_swap = collapse_max_ptes_swap(cc, HPAGE_PMD_ORDER); unsigned int max_ptes_none = collapse_max_ptes_none(cc, vma, HPAGE_PMD_ORDER); - enum tva_type tva_flags = cc->policy.tva_type; struct mm_struct *mm = vma->vm_mm; pmd_t *pmd; pte_t *pte, *_pte, pteval; @@ -1565,7 +1565,6 @@ static enum scan_result collapse_scan_anon_pmd(struct vm_area_struct *vma, struct folio *folio = NULL; unsigned long failed_pfn = -1; unsigned long addr; - unsigned long enabled_orders; spinlock_t *ptl; int node = NUMA_NO_NODE, unmapped = 0; @@ -1579,8 +1578,6 @@ static enum scan_result collapse_scan_anon_pmd(struct vm_area_struct *vma, collapse_scan_reset(cc); - enabled_orders = collapse_possible_orders(vma, vma->vm_flags, tva_flags); - /* * If PMD is the only enabled order, enforce max_ptes_none, otherwise * scan all pages to populate the bitmap for mTHP collapse. The bitmap @@ -2765,7 +2762,8 @@ static void collapse_control_release(struct collapse_control *cc) } static enum scan_result collapse_scan_pmd(struct vm_area_struct *vma, - unsigned long addr, struct collapse_control *cc) + unsigned long addr, struct collapse_control *cc, + unsigned long orders) { enum scan_result result; pgoff_t pgoff; @@ -2775,7 +2773,7 @@ static enum scan_result collapse_scan_pmd(struct vm_area_struct *vma, collapse_put_scan_file(cc); if (vma_is_anonymous(vma)) - return collapse_scan_anon_pmd(vma, addr, cc); + return collapse_scan_anon_pmd(vma, addr, cc, orders); pgoff = linear_page_index(vma, addr); result = collapse_scan_file(vma->vm_mm, addr, vma->vm_file, pgoff, cc); @@ -2885,15 +2883,17 @@ static void collapse_scan_mm_slot(unsigned int progress_max, vma_iter_init(&vmi, mm, khugepaged_scan.address); for_each_vma(vmi, vma) { - unsigned long hstart, hend; + unsigned long hstart, hend, orders; cond_resched(); if (unlikely(collapse_test_exit_or_disable(mm))) { cc->progress++; break; } - if (!collapse_possible_orders(vma, vma->vm_flags, - TVA_KHUGEPAGED)) { + /* One mask for the whole VMA */ + orders = collapse_possible_orders(vma, vma->vm_flags, + cc->policy.tva_type); + if (!orders) { cc->progress++; continue; } @@ -2922,7 +2922,7 @@ static void collapse_scan_mm_slot(unsigned int progress_max, /* move to next address */ khugepaged_scan.address += HPAGE_PMD_SIZE; - *result = collapse_scan_pmd(vma, addr, cc); + *result = collapse_scan_pmd(vma, addr, cc, orders); /* Nothing to do here, and the lock is still ours */ if (*result != SCAN_SUCCEED && *result != SCAN_PTE_MAPPED_HUGEPAGE) { @@ -3207,14 +3207,16 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start, { struct collapse_control *cc; struct mm_struct *mm = vma->vm_mm; - unsigned long hstart, hend, addr; + unsigned long hstart, hend, addr, orders; enum scan_result last_fail = SCAN_FAIL; int thps = 0; BUG_ON(vma->vm_start > start); BUG_ON(vma->vm_end < end); - if (!collapse_possible_orders(vma, vma->vm_flags, TVA_FORCED_COLLAPSE)) + orders = collapse_possible_orders(vma, vma->vm_flags, + TVA_FORCED_COLLAPSE); + if (!orders) return -EINVAL; hstart = ALIGN(start, HPAGE_PMD_SIZE); @@ -3252,9 +3254,11 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start, } vma = found; hend = min(hend, vma->vm_end & HPAGE_PMD_MASK); + orders = collapse_possible_orders(vma, vma->vm_flags, + cc->policy.tva_type); } - result = collapse_scan_pmd(vma, addr, cc); + result = collapse_scan_pmd(vma, addr, cc, orders); /* Nothing to do here, and the lock is still ours */ if (result != SCAN_SUCCEED && result != SCAN_PTE_MAPPED_HUGEPAGE) goto tally; -- 2.54.0