this might be due to the building being destroyed in the invasion.
That would be consistent. I suppose this is a consequence of the constraint that unique buildings can only be built once on a world. You can't, for example, raze your capital and rebuild it somewhere else because a better location opened up. I can't say I hate this mechanic. Actions do have consequences.
My own frustration is that you can't even look at the world before deciding whether to put a governor there or not. It is common for me to have two reasonable choices but one has far better terrain for what I want it to do. In this case I might want to leave this one as a colony because it is so unspectacular and I can't even build a capital.