Tbh, it used to be justified, and in some circumstances it still is justified.
Jews used to be the biggest "live-in" minority in Europe. So, a minority that lives in the same country and doesn't have a separate homeland. Kinda similar to black people in the USA. Contrary to regular migration, groups like that can't return home and they can't just assimilate.
If you look at regular migration, if someone's family has lived in the country for 4 generations, they aren't foreigners any more. Someone might say they have a Polish grandma, or they might say they are of Italian ancestry, but they aren't Polish or Italian themselves any more.
Jews, especially before WW2, couldn't do that. They were a discriminated against minority that was kept separate of the rest of society. A bit like black people in the USA after slavery was abolished.
So racism against them was a whole other order than the regular racism/xenophobia faced by other immigrants.
WW2 showed how bad that special kind of racism turned out to be, but WW2 wasn't an isolated event in time or location. Anti-Semitism had been rampant for centuries if not millennia before that, and it wasn't just localized to Germany. Just read up on e.g. Henry Ford, just to pick a random name from the bucket.
That said, things have shifted after WW2. Specifically antisemitic laws are pretty much gone in the western world, religion in general is not nearly as critically important as it was and there are now more than enough people of Jewish descent in the western world who don't identify as Jews or who aren't noticeable as Jews. And for a large part, society has accepted a special "protection" status for Jews to prevent a second Holocaust.
In consequence, the hate against Jews has mostly shifted into hate against Muslims, and many far-right/right-extremist people are now arguing that they can't be Nazis because they now don't hate Jews but instead hate Muslims. As if the problem with the Nazis wasn't genocide and suppression of minorities, but instead genocide and suppression of Jews specifically.
But yeah, Nazis will be Nazis, and they will argue in bad faith to justify themselves. Nothing new there.