Cant join friend's lobby HD but can join public lobby?

Hi, So I decided to play with my friend oversea through Steam but I cant seems to join his lobby. He and his friend could join each other's lobby, and all of us could also join public lobby, but when we tried to join each other lobby we couldnt, since it stucked at joining multiplayer game... We also have the same version as well as all of our expansions and mods turned off but we still couldnt join. Is there anything wrong with that?