Great question. Thanks for the ask!
Fear is such an interesting emotion. Oddly enough, its closest relative is humor. Both rely on subverting expectations to be fully effective, and both work best with 'punch lines' you don't see coming.
In that sense, I think the best way to explore fear is to consider what your audience is most unnerved by followed by what they'll least expect. Mash these together, and you have an effective scare.
For example, if you're writing sci-fi, your audience might be spooked by the concept of alien abductions. The typical set-up involves a flying saucer descending and little green men zapping up pedestrians and then probing them in... Well, places.
But what if, instead of a flying saucer, it was a solar eclipse?
And what if everybody who looked directly at the eclipse had aliens beamed into their minds?
What if the aliens didn't abduct people, but people inadvertently abducted aliens, and now both parties suffered as a result?
You see what I'm getting at.
These psychic entities could torment a person in a language they can't comprehend (maybe shrill tones, high frequencies, insanity, etc.), and all of it kicked off because they broke a social taboo by looking at an eclipse.
This is a rough example, but it shows how you can flip concepts on their heads using familiar concerns. Throw your audience off their game.
Nobody's scared of vampires or werewolves because we know too much about them—they're a solved equation. To really terrify readers, you need to unbalance them. Present them with a problem without an answer. Force them to discover the answer for themselves, and in the process, they'll explore all the horrible ways it might go wrong.
The real trick giving people the proper puzzle pieces. Once they have them, they'll build their own boogeyman, one more terrifying than words could ever conjure. Stephen King might have said that... or maybe it was somebody else.
Point is, you'll never beat out a person's imagination when channeling their greatest fears. Your best bet is to give their psyche a bit of ammunition, then sit back and let human nature do the rest.