CVE-2021-46983 affected the Linux kernel’s NVMe target over RDMA (Remote Direct Memory Access). When a connection dropped and an error occurred, the kernel could try to access memory that wasn’t valid anymore, causing a kernel crash (NULL pointer dereference). This is not a direct “remote code execution” bug, but could be a Denial-of-Service (DoS) vulnerability in storage setups using NVMe/RDMA. In this post, you’ll see what caused the bug, real error logs, code snippets, exploit scenario, and references.

What is nvmet-rdma?

nvmet_rdma lets a Linux server expose its NVMe storage devices over an RDMA network. RDMA is used for very high performance, low-latency storage networking.

Whenever there’s an error with a network operation (like SEND), the kernel driver needs to do cleanup using the right context objects (like queues and connections).

Description

After a switch to a "shared CQ" (Completion Queue) mechanism, the code mishandled error completion events. When RDMA communication failed (for example, if the link was dropped), the error handler (nvmet_rdma_error_comp) tried to access context data that was no longer reachable. This caused the kernel to crash while trying to handle a network error on the NVMe connection.

Key point: It used a pointer (cq_context) that _wasn’t valid anymore,_ instead of getting information from the work completion event.

Here's a real-life excerpt of such a crash (aka "Oops")

[ 905.786331] nvmet_rdma: SEND for CQE x00000000e3337f90 failed with status transport retry counter exceeded (12).
[ 905.832048] BUG: unable to handle kernel NULL pointer dereference at 0000000000000048
[ 905.839919] ...
[ 905.846144] CPU: 13 PID: 1557 Comm: kworker/13:1H
[ 905.872135] RIP: 001:nvmet_rdma_error_comp+x5/x1b [nvmet_rdma]
...
[ 906.010315] Call Trace:
[ 906.012778] __ib_process_cq+x89/x170 [ib_core]
[ 906.017509] ib_cq_poll_work+x26/x80 [ib_core]
[ 906.022152] process_one_work+x1a7/x360
[ 906.026182] ? create_worker+x1a/x1a
...

The important part is

BUG: unable to handle kernel NULL pointer dereference at 0000000000000048
RIP: 001:nvmet_rdma_error_comp+x5/x1b [nvmet_rdma]

What does this mean? The error handler tried to dereference a pointer at address x48, but that pointer was NULL—causing an instant kernel panic (DoS).

Let's look at a simplified code snippet close to what was in the affected kernels

static void nvmet_rdma_error_comp(struct ib_cq *cq, struct ib_wc *wc)
{
    struct nvmet_rdma_queue *queue = cq->cq_context; // <--- BAD!
    // ... do something with the queue pointer ...
}

With new shared CQs, cq->cq_context is NULL at this point, so queue becomes NULL, and the next access to queue->something crashes.

The correct way was to get the queue from the work completion, like this

struct nvmet_rdma_queue *queue = wc->qp->qp_context; // <--- CORRECT!

See the patch that fixed it.

Exploiting the Vulnerability

### How An Attacker/Tester Could Crash the System

No userland code needed! Just disconnect the network (or force a bad link state) while workloads are running on NVMe/RDMA.

Set up Linux server with NVMe target over RDMA.

2. Set up an NVMe initiator (such as another Linux or Windows box) and start active workloads or traffic.

4. As errors occur, nvmet_rdma_error_comp is called and tries to access NULL. The server panics and reboots or drops to emergency console.

Denial-Of-Service: A user with access to your NVMe network could reliably crash your storage server with this method.

Here's a conceptual PoC you can try in a safe test lab with the buggy kernel

# On the initiator machine:
nvme connect -t rdma -n testnqn -a <target-ip> -s 442

# On the initiator, start a workload:
fio --filename=/dev/nvmen1 --rw=read --bs=4k --size=1G --numjobs=4 --iodepth=16 --time_based --runtime=60 --name=nvme

# On the network, disrupt the connection:
ip link set dev <rdma-interface> down
# OR unplug cable, or shut down switch port.

# Watch the target machine logs (dmesg or journalctl -k)

You should see the NULL pointer dereference in logs, and likely a kernel panic.

Defense & Patch Status

- Affected: Linux kernels with nvmet-rdma shared CQ feature (mainline, some distros, up to early 2021).
- Fixed in: commit ee81e6bcc7f2 (May 2021)
- Recommendation: Upgrade to a kernel that has the fix, especially if running NVMe over RDMA in production.

References

- Kernel.org Patch Commit
- Red Hat Bugzilla
- CVE Details Page

Conclusion

CVE-2021-46983 is a classic kernel DoS bug: due to a pointer use-after-free (or rather, failing to update how to get the “queue” object after CQs changed), any network disruption could cause the NVMe target server to crash. The bug doesn't let an attacker get code execution, but could knock out important infrastructure from the network.

If you run NVMe over RDMA, double-check that you’ve got this fix in all production servers!

*Post written exclusively for you—feel free to use and share. Questions or corrections? Drop a comment below!*

Timeline

Published on: 02/28/2024 09:15:37 UTC
Last modified on: 12/06/2024 16:02:03 UTC