You only need to track the stuff with a GP cost; everything else your can use a focus for or assume it's in a component pouch of you don't use a focus. The GP cost items, yeah, I definitely track. Also the costs in time and ink (ie gold) for copying spells into your spellbook. My wizard of constantly broke, and the rest of the party have thousands of gold each.
I've done this as well. With most casters starting with a spellcasting focus though, we usually ignore the pouches too. I'm too nice of a DM to take the focus away.
Sometimes I've used the material components of my spells as an RP restriction for my druid. Like, I can only prepare spells for which I could reasonably find materials in the environment I'm currently in. I still don't track anything without gold cost, but I do like the material components for rp reasons. also makes me play a bit more creatively because I can't just pick the best spells every time
I think it would be awesome to have something where, for whatever reason, the caster loses their component pouch and has to make do with what they can find. Ideally with substitutions, like in Monkey Island.
I do use the materials when describing my spell casting, which I only describe if it feels pivotal.
Like insect plague needs fat, sugar and grains, so my cleric hocks a big loogie of powdered whale blubber sashimi he's been chewing at the center of the radius of the spell.