I think it's a mutual agreement. I don't get diplomatic penalties for entering their territory after accepting it.
If you want to test it move a ship into the territory of the race you made the agreement with, end the turn, and then check the diplomacy screen to see if there is a penalty for having a ship in their territory.