So, I’m trying to write a horror/physiological book but the main character is the antagonist. I’ve seen many people saying the main character has to be relatable and I’m not sure what to do about it?
So, there’s a catch here, I need to point out. Having your main character as an antagonist, in the strictest sense, isn’t necessarily that out there, but it’s also not what you’re thinking of.
So the antagonist is the character who works against your heroes. It doesn’t matter who they are, and there are entire genres built around stories where the main character is also their own worst enemy.
Technically, you can break this apart. Usually it’s aspects of the character working against themselves. For example: and alcoholic character’s antagonist could easily be their own alcoholism. In that sense, your main character would be both the protagonist and the antagonist.
A specific example would be The Gambler, with Mark Wahlberg. The main character is a compulsive gambler. It’s part of who he is, and that aspect is the film’s main antagonist. So, the main character is the antagonist.
Now, there is another side to this, and I suspect this what you were thinking of to begin with. You can tell stories where the protagonist is the villain. The immediate example that comes to mind there are the Ripley novels by Patricia Highsmith. These can be, reductively, described as a series of mystery novels where the killer gets away. Your protagonist can be the villain.
Making a character relatable helps, but what you need is a protagonist that’s compelling. One that grabs the audience and holds their attention. Being someone the audience identifies with can help getting there, but that’s not the real goal. It’s mistaking the shortcut for the destination, understandable, but potentially deceptive.
Also remember, most villains should have compelling, plausible, motivations driving them. The trick to getting an audience to side with your villain is digging into that motivation. There are many villainous or borderline villainous characters that audiences are quite happy to excuse their behavior, because they look at their behavior makes sense.
So, you can a villain who’s getting revenge for whatever, or working to defend their home. You can present a scenario like this, where your protagonist is the bad guy. Not even, the least bad, but straight up the villain.
There’s also plenty of room for protagonists who are evil, but principled. Such as an inquisitor for an evil empire. Even a corrupt cop can make for a compelling protagonist.
Ironically, a couple Nicholas Cage films come to mind immediately, including Lord of War (where he plays an arms dealer) and Bad Lieutenant: Port of Call New Orleans (which is both a very long title, and an equally bizarre film where Cage plays a corrupt cop.) Seriously, Bad Lieutenant is a really strange trip of a film; if you haven’t seen it, it’s worth a watch. (Also, a film where the protagonist pulls double duty as the primary antagonist.)
In some ways, all you really need for a villainous protagonist is a character who doesn’t care about social norms, ethics, morality, or any other pesky distractions on their path towards getting what they want. This can persist even if their goals are laudable.
Some long-form examples of heroic villains include, Michael Chiklis’s Vic Mackey in The Shield, or Kiefer Sutherland’s Jack Bauer in 24. In both cases, we have characters who are presented as heroes, but are willing to “do whatever it takes” to achieve their goals, justifying it to themselves that they only people they really harm are deserving of their fate. This can be a seductive mindset, but it’s also worth remembering these characters are doing some pretty horrific things. That said, The Shield put this conflict at the front of the series as a major theme, while 24 had a hard time admitting Jack wasn’t a good person.
Your villain needs to have a coherent plan. Then they just need to look for the most efficient path to their goals. Note: this does not mean they need to create an unnecessary bodycount to get their message across. There’s no value in killing everyone in your path if you only need to kill one person. Of course, if other people try to get in your way, that’s their funerals. I’m not saying your character can’t be a sadist, just that they don’t need to be one.