What my perfect use-case would be:

• Private instance (@mydomain.org)
• Allow other instance USERS to be able to DM me (user@matrix.org --> me@mydomain.org and vice versa)
• NOT allow my instance users to join rooms on other instances (child@mydomain cannot join room@matrix.org)

But given how DMs are essentially rooms...I'm not sure that level of fidelity is possible

I'm on the same path trying to find a solution. Is it possible to just whitelist certain accounts without all the public rooms. E.g., I have my own instance, but want to allow a DM from friend@matrix.org without allowing public rooms or other random DMs?

As far as I can tell...you can't. Debating just using XMPP instead.

For a whole house filter, what's the best kind? Is reverse osmosis system worth it?

Don't go so niche so quickly. I've enjoyed the broader groups here.