You need to install from backports to get the latest version: https://cockpit-project.org/running.html#ubuntu

Yes, I want to keep everything optional. Caddy being setup like that is temporary.
I've built a new image without the Caddy UI. You can remove Caddy from the compose file and use the image:
image: lllllllillllllillll/dweebui:v0.06-dev-nocad