From: Mike Galbraith <umgwanakikbuti@gmail.com>
To: Peter Zijlstra <peterz@infradead.org>, Ingo Molnar <mingo@elte.hu>
Cc: LKML <linux-kernel@vger.kernel.org>,
Linus Torvalds <torvalds@linux-foundation.org>
Subject: [PATCH] sched: update_rq_clock() must skip ONE update
Date: Sun, 30 Mar 2014 09:24:04 +0200 [thread overview]
Message-ID: <1396164244.28950.15.camel@marge.simpson.net> (raw)
NEWSFLASH: Wimpy latent scheduler buglet turns axe-murderer...
Root cause being us staying in kernel for weeks at a time during boot:
Subverting ftrace, we see below that updates _can_ be skipped for AGES,
which can make very bad things happen. It's supposed to skip ONE, as an
optimization. not lord knows how many. Here we see we've been skipping
since 4915630446, resulting in the watchdog being credited with a 137ms
delta_exec. Bogus, and..
modprobe-134 [000] 5.027903: update_rq_clock <-scheduler_tick
modprobe-134 [000] 5.037857: update_rq_clock <-scheduler_tick
modprobe-134 [000] 5.037864: update_rq_clock <-enqueue_task
modprobe-134 [000] 5.037866: enqueue_task_watchdog: WATCHDOG/0 enqueue 4915630446
modprobe-134 [000] 5.044965: update_rq_clock <-scheduler_tick
modprobe-134 [000] 5.046275: update_rq_clock <-scheduler_tick
modprobe-134 [000] 5.050274: update_rq_clock <-scheduler_tick
modprobe-134 [000] 5.052862: update_rq_clock <-dequeue_task
modprobe-134 [000] 5.052866: pick_next_task_watchdog: WATCHDOG/0 select 4915630446
watchdog/0-7 [000] 5.052868: finish_task_switch: WATCHDOG/0 runs 5052867644
watchdog/0-7 [000] 5.052869: watchdog: WATCHDOG/0 DELAYED 137238048 max: 137238048
watchdog/0-7 [000] 5.052872: update_rq_clock <-dequeue_task
watchdog/0-7 [000] 5.052873: dequeue_task_watchdog: WATCHDOG/0 dequeue 5052872018
watchdog/0-7 [000] 5.052874: put_prev_task_watchdog: WATCHDOG/0 STOP trace : delta_exec 137241572 sum_exec 137254287
..delta_exec was sufficient in a NOPREEMPT kernel on zillion disk box
that disk discovery combined with swsusp wanting to find out if it needs
to resume, that the featherweight watchdog was credited with delta_exec
HUGE enough to be incapacitated by the rt throttle long enough for the
hard lockup detector to trigger.
Throttled watchdog is bad juju, led to -ENOBOOT for IO beast.
Cc: <stable@vger.kernel.org>
Signed-off-by: Mike Galbraith <umgwanakikbuti@gmail.com>
---
kernel/sched/core.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -118,7 +118,7 @@ void update_rq_clock(struct rq *rq)
{
s64 delta;
- if (rq->skip_clock_update > 0)
+ if (rq->skip_clock_update-- > 0)
return;
delta = sched_clock_cpu(cpu_of(rq)) - rq->clock;
next reply other threads:[~2014-03-30 7:24 UTC|newest]
Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-03-30 7:24 Mike Galbraith [this message]
2014-03-31 0:12 ` Linus Torvalds
2014-03-31 4:20 ` Mike Galbraith
2014-03-31 16:00 ` Steven Rostedt
2014-04-01 3:10 ` Mike Galbraith
2014-04-01 3:26 ` Steven Rostedt
2014-03-31 16:13 ` Linus Torvalds
2014-03-31 18:27 ` Mike Galbraith
2014-03-31 18:37 ` Linus Torvalds
2014-04-01 3:28 ` Mike Galbraith
2014-04-01 9:55 ` Ingo Molnar
2014-04-03 8:02 ` Mike Galbraith
2014-04-08 15:53 ` Peter Zijlstra
2014-04-08 16:56 ` Mike Galbraith
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1396164244.28950.15.camel@marge.simpson.net \
--to=umgwanakikbuti@gmail.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@elte.hu \
--cc=peterz@infradead.org \
--cc=torvalds@linux-foundation.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®