You could setup WireGuard on your UDM-SE and install the app on your phone. You can tell the app what wifi networks to not establish the vpn when connected to. This works for iPhone, not sure about android.