You'll usually have time to take them out. Build your planetary invasion techs, or use a lot of transports if you don't have the techs.
That's actually my concern - I'd like to see them actually do something once. They've never lived long enough on DA to make any real impact - once they were wiped out literally the same turn they appeared. And by someone else, too, which was the really annoying part - their conquerors apparently stole Massive Scale Building in the invasion.
TOA beefs the event up some, though - they build up much faster. Still, maybe they do show up only in the late game.
