Not at all. The ships that are left over aren't able to be attacked, same for the startbases and shipyards, but you will still win the game without problem.
OK. That's good to know; however, it isn't clear how that can be accomplished. If no other race in the game has colonies, and no other race in the game has ships etc. except for the white, immobile, invulnerable ones, then what deed will end the game with a victory?