yeah it does... did you think it was 2 fame per turn?  NOT

I wouldn't be surprised, you can do that with a lot of other buildings, even like monuments in some situations to get crazy high influence areas.

Some of you people are really devious. 

OK, potentially a simple fix for this problem.  Make fame a non stored resource such that it only grows when you have additional +Fame buildings quests and treasure rewards would grant it  to the base of the faction.  Henchmen could still be used with this system as they are as they would lower the max fame by the same amount thereby limiting how many henchmen you could get within the game.  The only problem with this method comes from the improvements that produce fame per turn currently such as the adventure's guild which would have to have something done about it.  Anyhow that is my thought on fixing it.

Why is this even necessary?

If you want a one-time fame boost, simply make the fame be lost when the building is destroyed! Just like grain improvements.