I mean the culture lines on one side were friendly, the other side hostile.
For a planet to be considered under the effect of your culture, you have to overpower enemy culture. Your culture must not only dominate the entire phase line leading to the planet, it must completely overwhelm the planet and continue to move across phase lines in the other directions. Any enemy planet with enough culture to give you a military bonus is losing allegiance. In general, this makes it very difficult to fight in enemy gravity wells with your own cultural advantage.
In general, culture is something more defensive in nature, and difficult to use offensively.