cannot connect to players who join after i do

this doesnt happen all the time but i have noticed at a certain point if 2 people had joined after me they will show up in the lobby and have 0 ping and we wont be connected, if i rejoin i wont be connected to some one else i have to log out and log back in then rejoin and viola it works fine. ports are forwarded etc as i can play fine most the time. would there be some way to implement a "retry" if a person has already joined the lobby but is not connected to some one?