Need some advice about creating game overlay in Linux

There's two main options to approach this: either you inject into the game like how MangoHUD does it, or you make an overlay window that your window manager/compositor helpfully places on top for you.

The second option is pretty simple as any GUI library will give you the window, you just need some help from the compositor side to do the rest. That part can also be pretty easy thanks to Wayland nesting: you can use whatever compositor you want for this, not just the one from your DE. Gamescope for example might work, if not wlroots+layer-shell.

You can also be your own nested compositor by using Smithay, which is in Rust and was made by System76 to be the library behind COSMIC's compositor. Looking at the examples, it looks not too crazy to use. Then you just run the game under yourself and you can do whatever you want.

Injecting into the game isn't too crazy either, you compile to a library and force load it with LD_PRELOAD and override some OpenGL/Vulkan functions that lets you add your own draw commands on top before the frame is finished.

I think avoiding injecting into the game is cleaner and easier as you're completely independent from the game client, so you won't crash the game and you can also just test out your code as a regular window, and restart it mid game. Injecting would also trigger anticheat if present. Which, what you're doing kind of is, even though anyone could also just pen and paper it.

I'll preface this by saying that I'm not familiar with Rust nor Hearthstone at all, but I do deal with D3D9 and D3D11 on Windows to do similar things. Hopefully this will give you insights how you could approach this. (Closest I've done was code injection on Android)

The most common and robust approach to this is to hook/detour the API functions that the game imports from the renderer backend.

One way you usually do this is by creating a dummy library which overrides/intercepts the system library and passes through every function call to the API, except for the ones you need, you'd put your code before/after the passthrough. This usually requires you to gather all exported symbols and re-create them, which is a very tedious but rewarding task, as it usually is very stable and can work around things such as DRM.

Usually, since that sits quite low on the application's code stack, it is most efficient for it to be a more general-purpose hook which can load other libraries. Examples would be things like the ASI loader or Reshade on Windows.

Another way would be to do code injection via library side-loading. Essentially, you can simply load a library that performs the code hooks and does necessary renderer API hooking. This is usually done in combination with the previous method (it being a "plugin" loader), however, it is also possible to modify game binaries to call dlopen to load your library and its exported function as an entrypoint (in which case you need to do platform's CPU assembly code for a bit).

Those are the entrypoints for your code. After that, it is all about necessary render backend code that you need to do in order to draw graphics/text/etc.

In C/C++ land I'd just tell you to use Dear ImGui, but seeing as that doesn't exist for Rust, you're kinda on your own.

Same with the API detouring. Ideally, you'd make a plugin loader that does the job for you. Not sure if that exists in Rust yet.

For references, Vulkan overlays such as MangoHUD or ReShade could be useful to help you figure out how to draw stuff on screen.

As for the rest of your code - it can run in a separate thread that does the job for you as the game runs. Or, make a client-server relationship and make the game hook be the server for your info that you need.

Maybe someone can correct me if I'm wrong, but doesn't Wayland's security model intentionally prevent this type of "tampering" with another application's display?