Cat has a point. When I had cats, they ALWAYS got weirded out everytime I cleaned their box. Karmalee especially would run back into her litter box, dig around in the fresh litter, then look at me like "WHAT HAVE YOU DONE??? WHERE'S MY TREASURE???"
She wouldn't poop. Just dig around trying to find the missing poop. Then she'd look at me like a weirdo. She was very judgemental everytime I cleaned her box.
Speed on the other hand would look at me in confusion. Like ".......but why? Why do you collect the poop? What am I not getting here? Help me understand...."
One of mine legitimately gets angry when I scoop his poop, and even more upset when I do a full litter change. I have to wait until he's napping so that I can get at least part way through the process before he comes to yell at me. If I'm not fast enough, he'll even nip at my feet and/or (soft paw) swat at me.
It's like how my one cat always comes into the bathroom and looks at me like I'm crazy whenever I'm taking a bath. "Why would you do that to yourself?"
My cat used to wait for me to clean the litter box. Sometimes she'd look in it and walk away if it had too many clumps, but then use it immediately after I clean it.
The leading theory is that scooping the litter box throws the smell of feces into the air, which might attract predators (which is the reason they bury their poop, to hide the smell)
Since they think you're doing something dangerous, they want to keep watch for predators