Well I was completely wrong
I made a copy of my mod, removed all carriers; hanger defense structures; tradeports; refineries; and the tradeport/hanger defense upgrades for the starbases, fired up my favorite map (a 3 star 140 planet galaxy), filled it up with 9 unfair AIs, and it still slowed down around the same point. I did notice though that it's not time-linked but planets-owned linked. I was buttery smooth until I approached 20 planets, then somewhere around there BOOM. Complete crawl! It took me a lot longer to get those planets cuz I'm not used to all my capital ships being fighterless (I always just build a ton of capitals since I got tired of having to replace destroyed frigates/cruisers, I had become used to my bombers clearing out structures on a newly captured grav well for me
) Anyway, at least we can narrow the list of potential culprits down by these 3 things:
Not directly linked to the number of fighters/bombers (my game had 0 and still started crawling)
Not directly linked to the number of tradeships (same as above) and
Not directly linked to the number of refineries (same as above)
Also, I didn't mention this earlier, but I absolutely HATE mines (I've never lost a ship to them yet, but I hate having to either A. Ignore them on a freshly captured planet and keep playing as if they weren't there, or B. Spend time working on cleaning up each and every one on every new grav well I take), so my mod already had those removed. Thus mines aren't the cause of this crazy lag either.
Any other Ideas?