I'd bet my money on no. To my understanding, those hallucinations are basically brain-level disturbances in perception, where the brain does its best to fill in ambiguous activity with known objects. So I guess if you've never seen hats (or men), your brain would just interpret the signal as something different.
Extreme tangent but interesting (to me): people in different cultures experience voice hallucinations differently. In the west, "hearing voices" is often frightening because the voices are hostile. In other cultures, the voices are friendly!
That's an interesting question, but the answer probably isn't worth the trauma you'd inflict on a child by intentionally raising them to be completely unaware of the existence of hats.
