I don't do small talk very well and I very quickly run put of things to say to someone I don't know so I don't like to just talk rubbish with someone, I prefer to remain quiet and get on with what I am doing.
I don't mean that the person isn't worth talking to or I don't like them, if they need something from me or have a question then I'll galdly answer or help them, but almost everyone takes it as a slight against them when i dont want to engage in idle chit chat and assume I'm an arsehole when I'm really not trying to be.
Being bluntly honest. People who are neurodivergent can struggle with being "politely dishonest" and can tell you what they think in a very blunt manner without meaning to offend.
Not engaging in small talk. Again, people who are neurodivergent tend to prefer talking about things that fascinate them and can have a hard time understanding the point of talking about just whatever.
Struggling with being on time, struggling to focus on someone or something, struggling with eye contact. In general, neurodivergent traits tend to be seen as "asshole behavior" because they are abnormal and don't conform to society. People who aren't normal tend to be viewed as assholes because how dare they inconvenience me by being different.
Source: personal experience as well as listening to the experiences of others. I've been hit with all these things at least once and accused of being an asshole, aloof, and/or self-centered.
Not agreeing to false logic (say, out of pressure to be polite or non-confrontational), especially when the next step would be doing something based on that logic. People sincerely don't understand why deceiving you once like this won't work another time and think it makes you an asshole.
I turn the question around... people who are clearly liars, deceivers... politicians and businessmen that people line up to vote for with their money or public votes. You really wonder what people think an "asshole" is when you see the kind of politicians that get massive support in a population - to a point people have their photograph on the wall of their workplace or home, put stickers on their cars, etc. to support people that are clearly monstrous. A lot of people do not seem to like to study the crowds of Europe 1930's terrible leaders and just how many lined up to cheer on such persons.
The scientists a person believes also is a huge indicator of who they consider to be an 'asshole'. Just passively listening to people who support denial of climate change, denial of microscopic germs and virus, etc. The enthusiasm that followers to non-factual science seem to be very high, and they draw crowds in ways that fact-based science does not seem to do.
I am often thought of as an asshole because I am not much of a smiler and much of my politeness is perfunctory. I am somewhat reclusive and a loner by nature. I find my time at work having to mask exhausting and overstimulating. That much said, once people get to know me they generally discover that I am passionate and care deeply for people who are suffering or experience discrimination and will fight for them.