Clarification need on Hawking Radiation

First, so you know what level of technicality to answer with, or point me to, I'm late in my mathematics undergrad so I have quite a few tools under my belt at least at an introductory level. I haven't studied much physics in academia, but I make an effort to teach myself what I can.

To my understanding, one way that Hawking radiation can be explained is by particle\anti-particle pairs forming at the event horizon, the particle flys off, and the anti-particle falls into the black hole and annihilates with mass inside the black hole. But if this is the case, why wouldn't the 'regular' particle fall in just as often as the anti-particle? If so, then how could the black hole 'evaporate'? Furthermore, why doesn't the energy from the particle/anti-particle collision just remain inside the event horizon and the energy contribute to the overall mass/energy of the black hole?

Thanks!