A Broader Look at Literalism
By this description, I only show literalism on rare occasions. I have no trouble with metaphors or figures of speech. I only misunderstand the occasional context-dependent phrase, such as a nurse asking 'where did you come from?' and expecting to hear a hospital ward instead of a hometown. In the vast majority of cases, when someone says something they didn't mean literally, I know it wasn't literal. Or at least strongly suspect it.
But there is more to literalism than that. When I hear a non-literal statement, even if I know what it means, I always first interpret it literally, and only then do I take the non-literal meaning. (And this isn't a conscious process, by the way. My conscious mind receives both the literal and non-literal meaning from my sentence parser.)
Most people, when they hear something with multiple meanings, they will activate all meanings they know for the thing (this has been shown by semantic priming). But about 300 miliseconds later, they have suppressed all the meanings except the correct one. Although I haven't been through any priming experiments, my guess is that I take much longer, if at all, to suppress the wrong meaning, and so it enters conscious awareness.
This means that, even when I know what you really meant, part of me is thinking as if you actually meant to convey the literal meaning.
This can be funny - most figures of speech make me smile or even chuckle when I hear them, because they're so silly when taken literally. But other times it's upsetting. I've seen NTs playfully insult each other, both knowing that the other person intends no harm. For me, this kind of interaction is extremely upsetting even to observe, because part of me reacts as if they were serious about every insult they say to each other.
My Dad likes to jokingly pretend to be a cruel tyrant, for example if I ask if I can have some food, he'll act like he's considering refusing. I recently told him to stop, explaining what I've said in this post. When he fakes being a tyrant, I feel real panic, even though I know he's faking.