Consider threading, please!

It really won't be that difficult

from Sins Forums

The problem is, while the game engine was written to utilise a single thread, it would be fairly easy to add in thread support, and the game needs it for the late game battles. Even just running the game without battling when 100~ planets have been colonised can be terrible laggy on high-end computers, just from all the ships flying around.

It is not a huge overhaul that people seem to be expecting; it can be as simple as having trade and refinery ships running their path-finding on a separate thread. Or as complex as having each AI running their own thread, while running audio and interface on another. Any of these, or somewhere in between, would drastically increase performance for any processor set-up.

"But multi-threading is a huge change, and takes a long time, and wouldn't work with an old engine."

No not really. The two most complex aspects would be keeping the threads synced together (for example, the AI doesn't run faster than it should), and making the code threadsafe (making the threads play nicely with each other, and not attempting to change the same variable without appropriate locking). Adding thread capability is more tiresome than it is difficult, and can be done at any stage of a game engine life cycle.


Seeing as the game is in BETA, this is the PERFECT time to go ahead and implement threading. Start small, for example, just the interface on a separate thread. Work your way up, threading more and more, to properly utilise the processing power of the modern computers.

19,800 views 22 replies
Reply #1 from Sins Forums Top

That's a really good suggestion.  I'm thinking it won't happen though.

Reply #2 from Sins Forums Top

ive played games where once you get into tripple digits gameplay (over 100 hours) it slows to a crawl but just watch moves and check the game once an hour or so .

 

The one thing that i just do not understand and it oculd be this patching stuff but if a game is slow when there are 9 players left allive - cut to later in the game and there is now just 3 or 4 players - regardless of the fact we still occuppy the game amount of space the game does not speed up - id expect it to since its no longer devoting processing power to determining the dead AI's actions

ObjectDock - Access shortcuts and frequently used applications from an animated dock.
Reply #3 from Sins Forums Top

Quoting User45701, reply 2
id expect it to since its no longer devoting processing power to determining the dead AI's actions
End of User45701's quote

I doubt that the AI "thinking" takes up a lot of processing power....the biggest issue for the game is pathfinding and handling for all the objects...the shear amount of mesh points, path finding, particle effects, etc. that go with each individual trade ship or strikecraft or anything is immense, and whether I have one player with 100 planets or 10 players with 10 planets, there's still 100 planets worth of stuff....

Reply #4 from Sins Forums Top

Quoting User45701, reply 2
ive played games where once you get into tripple digits gameplay (over 100 hours) it slows to a crawl but just watch moves and check the game once an hour or so .

 

The one thing that i just do not understand and it oculd be this patching stuff but if a game is slow when there are 9 players left allive - cut to later in the game and there is now just 3 or 4 players - regardless of the fact we still occuppy the game amount of space the game does not speed up - id expect it to since its no longer devoting processing power to determining the dead AI's actions
End of User45701's quote

 

I havent actually ran the game while debugging it to see if there are memory leaks or where the inefficiencies are. But as you say, the game gets slower the longer it has been played.

Saving at a slow point, and then reopening the entire game to load the save, still leads to the game being slowed down. That almost rules out a memory leak, as those are normally gradual over the time an application is running.

My theory is the micromanagement the AI does with its ships, and the hundreds of trade and refinery ships calculating paths, is the main cause. Simply because as the game progresses, more and more ships are generally around (even with less AIs, planets still end up with trade ports etc). Even your own ships that you control, still require processor power.

Reply #5 from Sins Forums Top

You have to consider EVERYTHING that is on the map. Plus all the movement, weapon calculations, buffs, debuffs, particles, etc. etc. To me its easy to see how we get late game lag.

However. The lets make Sins LAA multicore/64 bit issue is a dead horse that has been beaten for years. Its not gonna happen. So says the Dev's.

Reply #6 from Sins Forums Top

I wonder, whether they alongside rebellion, sins of dark age, etc... work on iron engine 2.0

Reply #7 from Sins Forums Top

Why do people always think they can do a better job than others even if they don't know anything about it?

Reply #8 from Sins Forums Top

Quoting tlesta, reply 7
Why do people always think they can do a better job than others even if they don't know anything about it?
End of tlesta's quote

Byproduct of a democracy...even the dumbest dumb-dumb has a say in who the next President will be....

Reply #9 from Sins Forums Top

As good as this is an idea threading is actually very complicated and can cause even bigger memory management issues if done incorrectly.  Threads have to be managed and properly cleared when no longer in use.  Take a look at the Elemental FE Beta which has implemented threading and when it first was added they had lots of problems.  Also, keep in mind that FE is a turned based game, so it's easier to determine when to close a thread to return memory.

Real time is a whole other ball game.  I've always thought that CPU sand GPU manufactures should have some level of intellgent threading backed in and built in load management to assist devs, but hey that just my opinion.  That way the devs can focus on game play and bug fixes and not have to dedicate so much time to performance management,

 

 

 

 

Reply #10 from Sins Forums Top

Quoting tlesta, reply 7
Why do people always think they can do a better job than others even if they don't know anything about it?
End of tlesta's quote

Why do people always attempt to judge other people's qualifications and experience online, even when they don't have any knowledge about them?

Because they are narrow-minded fools.

I happen to work in software design, and have done for a decade now. I am more than qualified to discuss such a generic addition to an engine that drastically needs an overhaul. If you don't have anything useful to add to the debate, then don't click the reply button.

Reply #11 from Sins Forums Top

Quoting DarkSide73, reply 9
As good as this is an idea threading is actually very complicated and can cause even bigger memory management issues if done incorrectly.  Threads have to be managed and properly cleared when no longer in use.  Take a look at the Elemental FE Beta which has implemented threading and when it first was added they had lots of problems.  Also, keep in mind that FE is a turned based game, so it's easier to determine when to close a thread to return memory.

Real time is a whole other ball game.  I've always thought that CPU sand GPU manufactures should have some level of intellgent threading backed in and built in load management to assist devs, but hey that just my opinion.  That way the devs can focus on game play and bug fixes and not have to dedicate so much time to performance management,

 

 

 

 
End of DarkSide73's quote

I agree with you in part; for most software this would be the case, due to the original design not conforming to a thread-able build. But in the case of this game, most of the actions can easily be segregated to their own threads, simply because of the way that different objects interact with one another.

Take for example the AIs, they could easily be situated on an individual thread each, and run concurrently. Only when interacting with one another would thread-safety need to be adhered to, and this could be easily managed by enforcing global level variable locking for all the object attributes. Of course that would not be the most efficient method, but it would certainly get the job done, and any minor bugs would be picked up in the BETA.

If Ironclad did not wish to undertake this, due to inexperience with threading, lack of time, or one of a number of equally valid reasons. Then outsourcing the game engine for an overhaul would be a relatively safe bet, considering the game is still in BETA. As long as all of the constraints are adhered to, and the overlaying game isn't hard-coded into the engine, then an overhaul should have no effect other than performance increase.

Reply #12 from Sins Forums Top

This is also a very good idea, though I do wonder if the problem is not more related to paging as opposed to threading.  I've been on this game since it's 1st beta in 2007 and this has been an ongoing problem.  At that time I was running it on a hyper threaded dual core box I think.

Initially it was issues with ram limitations, but that crashing at 2gb of usage issue was fixed a few years back.

reducing the graphics options does help some what and adding a larger and vastly more powerful gpu also helped.  I specifically tend to see it happen towards the late game but not nearly as bad as it was before Rebellion.    Maybe IC will comment on what the issue is to help clarify things.

Reply #13 from Sins Forums Top

Quoting BoneyBoy, reply 11


 If Ironclad did not wish to undertake this, due to inexperience with threading, lack of time, or one of a number of equally valid reasons.  Then outsourcing the game engine for an overhaul would be a relatively safe bet....
End of BoneyBoy's quote

 

This statement is exactly what i'm talking about.

Reply #14 from Sins Forums Top

Quoting BoneyBoy, reply 0
It is not a huge overhaul that people seem to be expecting; it can be as simple as having trade and refinery ships running their path-finding on a separate thread.
End of BoneyBoy's quote

I don't know what type of organization you do software for (more importantly, how large the group you work with is), but Stardock has continually had a manpower issue for as long as I can remember...as an example, one of the most recent balance updates for GalCiv II: ToA had to be done by an Stardock employee during his free time on the weekends, and that game still is far from being balanced despite it's place as Stardock's crown jewel...

Stardock is struggling to get all the bugs stamped out while simultaneously balancing/polishing the new content, and now you want them to rewrite the parts of their game that do work?  Hell, Stardock didn't even have the time or resources to properly polish sins diplomacy...furthermore, the people who made the game engine (Ironclad) are off doing other things...you are asking Stardock, who didn't even design the game engine in the first place, to now rewrite it...

This is not a software decision, it is a business decision...this game is old and will be facing serious competition from a host of other 4x space games that are slated for the coming year (Endless Space, Legends of Pegasus, Star Drive, Gemini Wars)...furthermore, the game works just fine if you have even a half decent processor...I have a meager 2.4 GHz Dual Core processor and I can run large 10 player games with all graphics on highest, no lag at all....the only times I get lag are 1) MP or 2) running the game on faster than 2x speed...

Anyway, adding threading is not going to resolve the majority of lag problems which are related to MP....essentially you have a solution that is costly for SD and provides little benefit to the community...

Quoting BoneyBoy, reply 10
I happen to work in software design, and have done for a decade now. I am more than qualified to discuss such a generic addition to an engine that drastically needs an overhaul. If you don't have anything useful to add to the debate, then don't click the reply button.
End of BoneyBoy's quote

90% of the people who ask for multi-threading don't understand the amount of software work that would have to be done...the 10% that do understand the software work don't seem to get the business perspective of the issue...I'm trying not to be mean, I really am, but this issue has been beaten to death like none other...Ironclad is done with the game, they have passed the torch to SD, and SD isn't going invest the time and expense into something with so little return value...

 

 

Reply #15 from Sins Forums Top

Perhaps a simple solution to eek more performance out would be to mod trade ports to produce less trade ships.  In the late-game when there are hundreds of ports up, it wouldn't be surprising to me if you could actually improve performance noticeably by cutting the number produced.

I'll have to try that some time...

Reply #16 from Sins Forums Top

I have played with a mod that did this (as well as slightly reduce the number of SC involved)...hard to say if it helped since it was for a friends computer, not mine, but if you get results let me know...

 

Reply #17 from Stardock Forums Top

Quoting BoneyBoy, reply 10
I happen to work in software design
End of BoneyBoy's quote

Then you should know you can't make detailed assumptions about the internal architecture of a system you've only seen from a user perspective. Late-life overhauls are expensive, can potentially create many new issues, and consequently aren't to be undertaken without very strong justification.

As others have stated, the bottom line is that the bulk of the engine was written before multi-core processors became commonplace on consumer desktops, and at this point in the product's life, the limited project resources available are better spent elsewhere.

Reply #18 from Sins Forums Top

Quoting kryo, reply 17

Quoting BoneyBoy, reply 10I happen to work in software design

Then you should know you can't make detailed assumptions about the internal architecture of a system you've only seen from a user perspective. Late-life overhauls are expensive, can potentially create many new issues, and consequently aren't to be undertaken without very strong justification.

As others have stated, the bottom line is that the bulk of the engine was written before multi-core processors became commonplace on consumer desktops, and at this point in the product's life, the limited project resources available are better spent elsewhere.
End of kryo's quote
Ouch...

Reply #19 from Sins Forums Top

If they had more money? Sure. But they're not like EA or Valve with tonnes of cash to splash around. Even late-game games on huge multi-star maps work fine for me, on a 4.8Ghz FX-4170. Not 100+ hours (yet) but still, you get the picture. Sins 2 should definitely be made multi-threaded and at least have 64bit as an option.

 

Still, this is why I like open source; some random could end up making a version that uses multiple cores on their own. 

Reply #20 from Sins Forums Top

Quoting Volt_Cruelerz, reply 15
Perhaps a simple solution to eek more performance out would be to mod trade ports to produce less trade ships.  In the late-game when there are hundreds of ports up, it wouldn't be surprising to me if you could actually improve performance noticeably by cutting the number produced.

I'll have to try that some time...
End of Volt_Cruelerz's quote

 

It helps. How much *shrugs*

 

My biggest boost in performance late game is simply un pinning everything in the Empire Tree. 

And I started this habit back in Entrenchment. Since then have suggested as often as I can. 

My thought on this is a main cause of later game lag. But I cant confirm it to suggest changes for the beta.

Reply #21 from Sins Forums Top

Quoting kryo, reply 17

Quoting BoneyBoy, reply 10I happen to work in software design

Then you should know you can't make detailed assumptions about the internal architecture of a system you've only seen from a user perspective. Late-life overhauls are expensive, can potentially create many new issues, and consequently aren't to be undertaken without very strong justification.

As others have stated, the bottom line is that the bulk of the engine was written before multi-core processors became commonplace on consumer desktops, and at this point in the product's life, the limited project resources available are better spent elsewhere.
End of kryo's quote

I hate to take a superiority complex here, but black box testing and analysis is exactly that.

Regardless, I am sure you have done your own internal cost benefit analysis, and the little to no increased sales from increasing the performance, doesn't bode well for the stakeholders.

Reply #22 from Sins Forums Top

Quoting BoneyBoy, reply 21
Quoting kryo, reply 17
Quoting BoneyBoy, reply 10I happen to work in software design

Then you should know you can't make detailed assumptions about the internal architecture of a system you've only seen from a user perspective. Late-life overhauls are expensive, can potentially create many new issues, and consequently aren't to be undertaken without very strong justification.

As others have stated, the bottom line is that the bulk of the engine was written before multi-core processors became commonplace on consumer desktops, and at this point in the product's life, the limited project resources available are better spent elsewhere.

I hate to take a superiority complex here, but black box testing and analysis is exactly that.

Regardless, I am sure you have done your own internal cost benefit analysis, and the little to no increased sales from increasing the performance, doesn't bode well for the stakeholders.
End of BoneyBoy's quote

 

I assume you mean shareholders; and thats between them and SD.