Ok, basic working as it should behavior seems to happen with level 1 towns. You build a building (eg, a momument) and your ZoC increases by 1. You then demolish the monument (because why not) and it shrinks by 1 (incorrectly I might add, but building the monument again fixes things). Also when demolishing the monument the fence around the village retracts, no longer covering the area where the monument once was.

Then you upgrade to a level 2 settlement (tested on both town and conclave) and things change. First, ZoC doesn't shrink when you destroy the monument. Second, it grows again when you build it again (repeated building and destruction of monuments leads to very large ZoC). Third, destruction of the monument doesn't shrink the fence, the fence continues to encircle the area that held the monument, and the town's ZoC (and snaking effects) act as if there is still a building there (and you can build in slots only reachable because of it's presence, as well as rebuilding on it again).