backend/drm: introduce wlr_drm_bo_handle_table

Using GBM to import DRM dumb buffers tends to not work well. By
using GBM we're calling some driver-specific functions in Mesa.
These functions check whether Mesa can work with the buffer.
Sometimes Mesa has requirements which differ from DRM dumb buffers
and the GBM import will fail (e.g. on amdgpu).

Instead, drop GBM and use drmPrimeFDToHandle directly. But there's
a twist: BO handles are not ref'counted by the kernel and need to
be ref'counted in user-space [1]. libdrm usually performs this
bookkeeping and is used under-the-hood by Mesa.

We can't re-use libdrm for this task without using driver-specific
APIs. So let's just re-implement the ref'counting logic in wlroots.
The wlroots implementation is inspired from amdgpu's in libdrm [2].

Closes: https://github.com/swaywm/wlroots/issues/2916

[1]: https://gitlab.freedesktop.org/mesa/drm/-/merge_requests/110
[2]: https://gitlab.freedesktop.org/mesa/drm/-/blob/1a4c0ec9aea13211997f982715fe5ffcf19dd067/amdgpu/handle_table.c

1 files changed, 43 insertions, 0 deletions

diff --git a/backend/drm/bo_handle_table.c b/backend/drm/bo_handle_table.c
new file mode 100644
index 00000000..026194ce
--- /dev/null
+++ b/backend/drm/bo_handle_table.c
@@ -0,0 +1,43 @@
+#include <assert.h>
+#include <stdint.h>
+#include <stdlib.h>
+#include <wlr/util/log.h>
+#include "backend/drm/bo_handle_table.h"
+
+static size_t align(size_t val, size_t align) {
+	size_t mask = align - 1;
+	return (val + mask) & ~mask;
+}
+
+void drm_bo_handle_table_finish(struct wlr_drm_bo_handle_table *table) {
+	free(table->nrefs);
+}
+
+bool drm_bo_handle_table_ref(struct wlr_drm_bo_handle_table *table,
+		uint32_t handle) {
+	assert(handle != 0);
+
+	if (handle > table->len) {
+		// Grow linearily, because we don't expect the number of BOs to explode
+		size_t len = align(handle + 1, 512);
+		size_t *nrefs = realloc(table->nrefs, len * sizeof(size_t));
+		if (nrefs == NULL) {
+			wlr_log_errno(WLR_ERROR, "realloc failed");
+			return false;
+		}
+		memset(&nrefs[table->len], 0, (len - table->len) * sizeof(size_t));
+		table->len = len;
+		table->nrefs = nrefs;
+	}
+
+	table->nrefs[handle]++;
+	return true;
+}
+
+size_t drm_bo_handle_table_unref(struct wlr_drm_bo_handle_table *table,
+		uint32_t handle) {
+	assert(handle < table->len);
+	assert(table->nrefs[handle] > 0);
+	table->nrefs[handle]--;
+	return table->nrefs[handle];
+}