Stupid RCU Tricks: Waiting for Grace Periods From NMI Handlers

Suppose that you had a state machine implemented by NMI handlers, and that some of the transitions in this state machine need to wait for an RCU grace period to elapse. How could these state transitions be implemented?

Before we start, let's dispense with a couple of silly options. First, we clearly are not going to invoke synchronize_rcu() from within an NMI handler. This function can sleep, and we cannot sleep even within non-threaded interrupt handlers, let alone within NMI handlers. Second, we are not going to invoke call_rcu() from within an NMI handler. This function disables interrupts to exclude concurrent invocations on a single CPU, and disabling interrupts does nothing to keep NMI handlers from running. Worse yet, when running on rcu_nocbs CPUs (which offload callback invocations to rcuo kthreads), call_rcu() acquires a lock. Therefore, invoking call_rcu() from within an NMI handler would at best result in deadlock, and might also result in corruption of RCU's lists of callbacks.

So what can we do?

One approach would be to use a self-spawning RCU callback, that is, a callback that invokes call_rcu() on itself. (Yes, this is perfectly legal, just as it is with timer handlers. See the function rcu_torture_fwd_prog_cb() in kernel/rcu/rcutorture.c of v5.14 of the Linux kernel for an example.) This callback could also increment a counter that could be checked by the NMI handler. When the NMI handler wished to defer a state transition until the end of a future RCU grace period, it could transition to an additional “waiting for RCU” state while recording the value of that counter. Later, when the counter had advanced sufficiently, a subsequent NMI handler could complete the transition to the desired state.

Of course, it is not sufficient to wait for the counter to advance just once. The reason is that the initial NMI might have occurred just before the RCU callback executed, and the next NMI might happen just afterwards. Yes, the counter has advanced, but almost no time has elapsed, much less a full RCU grace period. It is instead necessary to wait for the counter to advance by two, and also to have the needed memory barriers in place.

But there is a better way that makes use of RCU's existing counters and memory barriers. RCU provides these four functions for this purpose, two of which are usable from NMI handlers:

1. get_state_synchronize_rcu(), which was first used in v4.10 in 2015, returns a “cookie” that can be checked later. SRCU provides a similar get_state_synchronize_srcu() API.
   
2. start_poll_synchronize_rcu() returns a cookie as well, but also ensures that the needed RCU grace period gets scheduled. Unfortunately, this last requires locks to be acquired, which precludes its use in an NMI handler. SRCU provides a similar start_poll_synchronize_srcu() API, which was first used in v5.14 in 2021.
   
3. poll_state_synchronize_rcu() takes a cookie from the first two functions and returns true if the corresponding RCU grace period has elapsed. SRCU provides a similar poll_state_synchronize_srcu() API, which was first used in v5.14 in 2021.
   
4. cond_synchronize_rcu(), which was first used in v4.10 in 2015, also takes a cookie from the first two functions, but waits (if needed) until the corresponding RCU grace period has elapsed. Unfortunately, the possibility of waiting precludes this function's use in an NMI handler.
   

So the first NMI handler can invoke get_state_synchronize_rcu() to obtain a cookie, then transition to the additional state. Later NMI handlers residing in this additional state could pass that cookie to poll_state_synchronize_rcu(), completing the transition if this function returns true. On busy systems, RCU grace periods are being initiated by many other things, so that there is almost always a grace period in progress, but if this must work on quiet systems, the aforementioned self-spawning callback could be used to force the issue when needed.

Of course, RCU has made internal use of grace-period polling for a very long time, starting prior to the beginning of the Linux-kernel git repository in 2005.

In short, NMI handlers can now work with both RCU and SRCU grace periods without the need to invent counters or to worry about memory-ordering issues!