"hung task" when background; expected behavior?

[bukzor]

When running a bleeding edge nvim (which has a libuv using io_uring) I was getting "hung task" kernel panics. After much anguish, we find this was due to the related kworker processes entering D whenever nvim was background. And so whenever nvim was background for more than two minutes, my kennel panicked.

The response of chromium upstream maintainers is to start disabling "hung task" panics in various places. Is this the correct course of action?

A quick survey of Google for D-state kworkers seems to indicate that (apart from my problem) this is almost always indicative of a major bug that requires a reboot. I wonder if a better approach would be to mark such kworkers (stuck due to sigstop) as T rather than D?

---

Full context:
https://issuetracker.google.com/issues/284728129

[ammarfaizi2]

[...] mark suck kworkers (stuck due to sigstop) as T rather than D?

This SIGSTOP issue has been closed a year ago. It's marked as false positive and not worth fixing.
See the discussion here: #448 (comment)

[bukzor]

I did see that. It may be worth revisiting that decision, depending on the answers to my questions (above).

The fallout is more than a failed test, as described there. It's inappropriate kernel panics.

If your thinking is that everyone should disable the hung task panic, that may be completely reasonable. But it's also the question I was trying to ask.

[axboe]

I only see a backtrace, not a kernel panic. Those are two very different things.

[bukzor]

I only see a backtrace, not a kernel panic. Those are two very different things.

I agree, except this is a panic:

https://issuetracker.google.com/issues/284728129

[  +0.0]  kthread+0x13a/0x152
[  +0.0]  ? process_one_work+0x482/0x482
[  +0.0]  ? kthread_blkcg+0x31/0x31
[  +0.0]  ret_from_fork+0x1f/0x30
[  +0.0]  </TASK>
[  +0.0] Kernel panic - not syncing: hung_task: blocked tasks
ERROR vsh: [utils.cc(50)] Failed to read message size from socket: Connection reset by peer (104)
                                                                                                 ERROR vsh: [vsh_client.cc(266)] Failed to receive message from server: Connection reset by peer (104)
crosh>

I'd be delighted if someone would address the OP questions.

[axboe]

It's not, but of course if you have the setting on that causes a panic on (what appears to be) a hung task, then it can turn into a problem. All that trace is telling you that here's the stack trace for this process, and it's been stuck uninterruptibly for X amount of time. Obviously it'd be better if we didn't have this oddity to begin with. It's really because the kworker is waiting on requests to get finished, but someone sent a SIGSTOP to the tasks that are supposed to finish them. This is what is causing the kworker to appear stalled.

As such, there's obviously nothing wrong by disabling panic on a hung task. The problem is that the system may have real hung tasks and want to panic on that condition, and potentially reboot.

Let me ponder this for a bit. Since the kworker exit part doesn't take signals to begin with, a quick work-around would be to have the ring exit work just sleep interruptibly instead. We can still warn if that takes longer than we think, but at least we would not have D state workers which is what is triggering this to begin with.

[axboe]

Something like this should do it:

diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index a467064da1af..f181876e415b 100644
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -3121,7 +3121,18 @@ static __cold void io_ring_exit_work(struct work_struct *work)
 			/* there is little hope left, don't run it too often */
 			interval = HZ * 60;
 		}
-	} while (!wait_for_completion_timeout(&ctx->ref_comp, interval));
+		/*
+		 * This is really an uninterruptible wait, as it has to be
+		 * complete. But it's also run from a kworker, which doesn't
+		 * take signals, so it's fine to make it interruptible. This
+		 * avoids scenarios where we knowingly can wait much longer
+		 * on completions, for example if someone does a SIGSTOP on
+		 * a task that needs to finish task_work to make this loop
+		 * complete. That's a synthetic situation that should not
+		 * cause a stuck task backtrace, and hence a potential panic
+		 * on stuck tasks if that is enabled.
+		 */
+	} while (!wait_for_completion_interruptible_timeout(&ctx->ref_comp, interval));
 
 	init_completion(&exit.completion);
 	init_task_work(&exit.task_work, io_tctx_exit_cb);
@@ -3145,7 +3156,12 @@ static __cold void io_ring_exit_work(struct work_struct *work)
 			continue;
 
 		mutex_unlock(&ctx->uring_lock);
-		wait_for_completion(&exit.completion);
+		/*
+		 * See comment above for
+		 * wait_for_completion_interruptible_timeout() on why this
+		 * wait is marked as interruptible.
+		 */
+		wait_for_completion_interruptible(&exit.completion);
 		mutex_lock(&ctx->uring_lock);
 	}
 	mutex_unlock(&ctx->uring_lock);

[axboe]

And here's a basic reproducer for this condition, in case anyone is curious:

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <liburing.h>

int main(int argc, char *argv[])
{
	struct io_uring_sqe *sqe;
	struct io_uring ring;
	int fds[2];
	char r1;

	if (pipe(fds) < 0) {
		perror("pipe");
		return 1;
	}

	io_uring_queue_init(8, &ring, 0);

	sqe = io_uring_get_sqe(&ring);
	io_uring_prep_read(sqe, fds[0], &r1, sizeof(r1), 0);
	sqe->user_data = 1;

	io_uring_submit(&ring);
	io_uring_queue_exit(&ring);
	kill(getpid(), SIGSTOP);

	return 0;
}

[bukzor]

Thank you for considering

On Sun, Jun 11, 2023, 7:27 PM Jens Axboe ***@***.***> wrote: It's not, but of course if you have the setting on that causes a panic on (what appears to be) a hung task, then it can turn into a problem. All that trace is telling you that here's the stack trace for this process, and it's been stuck uninterruptibly for X amount of time. Obviously it'd be better if we didn't have this oddity to begin with. It's really because the kworker is waiting on requests to get finished, but someone sent a SIGSTOP to the tasks that are supposed to finish them. This is what is causing the kworker to appear stalled.

As such, there's obviously nothing wrong by disabling panic on a hung task.

The problem is that the system may have real hung tasks and want to panic on that condition, and potentially reboot.

That's what I was trying to express in my original message. Thank you for rephrasing :)

Let me ponder this for a bit. Since the kworker exit part doesn't take signals to begin with, a quick work-around would be to have the ring exit work just sleep interruptibly instead. We can still warn if that takes longer than we think, but at least we would not have D state workers which is what is triggering this to begin with.

Thanks for your consideration! I saw your patch and it's sounds like it's just what I needed. I wonder whether anyone in the kernel would like to consider this case. The same line of reasoning would apply, I'd think. I think it's work asking but I don't actually know where I should ask. —

…

Reply to this email directly, view it on GitHub <#878 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAE4KSGLL2DK7XJOCTSWF7DXKZ5BRANCNFSM6AAAAAAZCPOYJ4> . You are receiving this because you authored the thread.Message ID: ***@***.***>