Does roast garlic affect them?
With handguns at close range, it shouldn’t. Though, once you start getting past about 50ft, I’d worry. I’d be more worried about the garlic slipping out and jamming the mechanisms, though. If it was ground into paste, and then capped with something, it should be fine.
That’s certainly not the only creative ammunition option though. High explosive rounds come to mind. There are a lot explosives that will detonate on contact, and can be fired from a gun… mostly, safely. mercury fulminate is the first one that comes to mind, thanks to an old Law and Order episode. I’m not sure if picric acid would detonate when the weapon was fired, or only on impact, but it would also deliver a devastating wound from what you could pack into a hollowpoint.
If mass tissue disruption is enough to stop them, Glaser safety slugs might actually be a legitimate choice. These things are designed to shatter on contact spraying birdshot everywhere. I could easily see someone taking the basic design and loading it with a far more disruptive payload, like holy water, or maybe even the garlic paste above. This might be a better delivery method for an explosive round too.
With fey, if any iron would do, steel core AP rounds might actually be a better option. The softer metals should slough off on contact, and the resulting iron would do… whatever it was supposed to in the first place.
In theory you could make the entire bullet out of iron, but, with anything other than a very soft metal, you’ll irreparably damage the barrel’s rifling after the first or second shot. That said, you can stick a soft jacket over it, lead or copper are common choices. This protects the barrel from damage, but allows for much harder bullets to be fired. If you’re curious, that’s what the term Full Metal Jacket refers to.
Copper is a good option, even for lead rounds, because, unlike the lead, the copper isn’t toxic. So you can handle the rounds without having to be as paranoid about lead exposure.
You can use iron shot in a normal shotgun load, so that might be an easier option. I think you can actually buy up to 6mm steel shot commercially.
For iron bullets, there are apparently issues with them losing momentum faster than with normal rounds. I don’t know if this is relevant at handgun ranges or if it’s a rifle issue. That is the case with silver rounds, as I found out a couple years ago. (They’re fine for pistols, but rifles lose range and accuracy.) This has something to do with the density of silver, but explaining it requires a slightly better grasp on ballistics than mine.
If you’re wanting to take a more high tech look at vampire hunting, my recommendation will always be the British TV series Ultraviolet. Not to be confused with the 2006 American film. It takes a very non-mystical approach to tracking and eliminating vampires, with characters using graphite fragmentation rounds, and re-purposed gas grenades that disperse the active ingredient from garlic that affects them. Also, it’s got Jack Davenport and Idris Elba as the leads with some very sharp writing. This really is worth watching if you want to do vampire hunters in urban fantasy.