How to set LightDM and XFCE to only start when I want them and not at boot

sudo systemctl disable lightdm

You'll boot into a text tty login prompt. Log in, and run startx to start your X session. You don't need lightdm at all.

Or, if you do want to run the session manager, log in and run sudo systemctl start lightdm. You'll still be logged into tty1, so you might want to switch back to that and log out if you're concerned about physical security at your device, but it doesn't otherwise hurt anything.

Use systemctl:

• to turn off lightdm, sudo systemctl stop lightdm. This will also shut down xfce if it was started with lightdm.
• to permamently not start lightdm at boot, sudo systemctl disable lightdm.
• to start xfce whenever you want it just run start-xfce.

Note: this will mean that you computer will boot directly to tty instead of any graphical display, which I assume is what you wanted.

To stop the lightdm from starting, you should be able to disable it by this command:

sudo systemctl disable lightdm

If XFCE still starts automatically, then check your .bash_profile. If there is a line like:

exec startxfce4

then put a # in front of it. Once that is done, you can start XFCE any time by typing exec startxfce4 at the command line. When in XFCE, if you log out it should put you back to your command prompt.