Towers have 0 armor, and suffer armor debuffs from Spike Wave / Ground Spikes / whatever it is called. This is why towers go down so fast later levels: they are taking the full dps from the demigods.
First off only one armor debuff works on them and it's on an uncommon DG. Ground Spikes is an excellent counter for tower pushes, but QoT isn't that hot for much of anything else.
Second I think you're looking at towers the wrong way. They have 2800 HP and they hit for over 200. Even if the enemy is up to 50% mitigation if they walk into half a dozen towers they're still taking 600 damage a pop, that's easily doubling your autoattack damage, and nearly as much damage as landing a 1700 slam every single time it's off cooldown.
That's ignoring that if the enemy puts 2800 autoattack damage into your tower that's at least 1400 damage you've saved yourself. Killing towers, no matter how quickly you do it, is not really a viable option against Rook in most situations unless Rook has died or abandoned them.
Towers are something to do. A single tower late game isn't really a big deal, but when you're holding a portal, waiting for the enemy to come try to push you out, you can be building towers. No DG other than reg has anything remotely similar that they can do between battles to prepare themselves for the upcoming battle.
The only time that tower damage becomes truly shitty is when everything but autoattack becomes shitty, at extreme artifact levels.