if it hasn't been completed, right click on the icon in the planet management menu.
if it has been completed, select the building and hit the scuttle icon (bottom centre of the screen, looks like construction tape), or press alt-h.
beware that you do not get the full cost of the building back if you do this.