Connection Issue - Need advice on possible solution

i cant help Zehdon. But do keep updated of your problem solving. As i think my router may have a similar problem.

I have 2 people trying to play online, one computer (generally with the lower ports) connects fine, whereas the other takes longer and generally cant connect to one person. We may be purchasing a knew router. I thought we managed at least one game with you? Although i do remember my housemate always had problems connecting to you ><

Will do. I can't find anything to help me, so I'm thinking maybe I'm going to have to learn how to program my modem. I really don't want to, it's very bloody complicated.

Anyway, you might have the same problem as me if the first person you connect to, the host, is speedy and then the next person takes longer to connect, the one after that takes even longer, etc. I've also noticed that if I try to connect to someone with the same issue, it's a stalemate - I left it for nearly a full 10 minutes. Then I got kicked from the lobby. 3v3 is do-able, but not easy. Anything above that becomes impossible, as 6 connections seems to really slow things down. I'm guessing this might be the 'critical mass' point, so to speak.

can you tell us what router and modem you have?

I think you have discoverd an issue many people have.

As some hosts open one port at a time, allowing ppl to join in an orderly fashion. I cant say much more than that except email sales support.

I included the Modem details in the OP. I have no router, as my Net PC contains nothing worth protecting and my Work PC is standalone and they are not networked.


They are made by Thompson, and are often referred to as ST or Thompson 536. However, they aren't the only modem with this 'feature' and I believe attempting to elliminate this on our end might result in a substantial improvement in our connectivity. Well, at least in my case.

Head over to Glasnost and use their 'Broandband Characteristics Test' and see if it reports a simillar problem; higher upstream queue. There are many other sites that can do this kind of test, however this is apparently the most used and trusted. Also, if you experience problems with Bit-torrent, namely slower download speeds despite forwarding ports and reducing upload speed via the client, you might have a simillar issue to myself.

check if this is your modem: http://bc.whirlpool.net.au/bc/hardware/?action=h_view&model_id=704

Rate limiting of IP traffic  

Rate limiting of LAN ports  

sorry, doesnt look like you can change the upload speed.

i dont know if you can do it over the CLI.... im only a tafe student

Hmm, I just checked out that test your broadband site, and apparently my 'Upstream Queue' is 1478ms.

"Your upstream queue is so large that the packets can get delayed for more than a second. This can be a problem for real-time traffic such as VoIP or online games!"

 

I'm not 100% sure on what this means, or any idea on how to fix it.  Other than what you've posted that is ?

I was advised that I can; it's certainly not easy, but it's do-able. However, it's not a GUI available option so it has to be coded - which is a pain in the ass. I'm going to look into software options first though, no point building the great wall of china if a wire fence will do

bummer

It's not all bad - if I force my game to use the Proxy Servers when they go live, it'll create a situation simillar to client/server for me - all my PC has to talk to is the Proxy Server, which frees up my upload queue of ACK Packages because they're only coming from a single source instead of 6 seperate sources. Adding around 50ms of latency to my in-game ping to be able to play with 100% of players is fine by me.

The reason I'm interested in finding an alternative is because if the Proxy Servers, for whatever reason, don't fix my problem then Demigod remains unplayable for me, and I've yet to be defeated by a game. Hell, if I finished Ultima IX un-patched I can handle this

I'll do my best to explain. This is my understanding, so take it with a grain of salt:
When your computer receives information it sends out a little piece of information, called a package, that basically says 'Ok, got that information you sent' to the sender. These are called ACK Packages, or Acknowledgement Packages. From a single source, these little packages have zero impact on your modems ability to process information.
The problem we both seem to have is that because of the way Demigod's multiplayer works, P2P or Peer to Peer, we're receiving information from lots of different people at once, meaning we have to send lots of these little packets out at the same time telling everyone we got their information. Because of our Upstream queue, these little packets are backed up - it takes too long to send them and because the other machines aren't receiving the 'Ok, got that information you sent' message, they don't send further messages; this slows down the game and creates lag.
The more people we connect to, the slower things go. Connecting to one or two people is pain free, even five isn't horrible. Six seems to be the straw on the cammels back, and it clogs up our end of things with these ACK Packages.
Lowering the Upload speed reduces the amount of ACK Packages that are produced by our computer - the modem is able to handle them all instead of creating more than it can handle and slowing us down. Usually, you also lower your download speed slightly as well to give yourself breathing room, however this is reported - according to several Bit-Torrent forums - to have a minimal impact.

This having any impact on Demigod is all theory at this point, however, it's merely one situation which supports what I've been able to observe. I'm attempting to find ways to lower my Upload speed to observe what, if any, effects this has on my ability to connect to other players. However, as I stated above, the Proxy servers might fix the problem anyway.

this confuses me because what i think you are refering to is SYN-ACK, which is TCP only!

Connection establishment

To establish a connection, TCP uses a three-way handshake. Before a client attempts to connect with a server, the server must first bind to a port to open it up for connections: this is called a passive open. Once the passive open is established, a client may initiate an active open. To establish a connection, the three-way (or 3-step) handshake occurs:

1. The active open is performed by the client sending a SYN to the server.
2. In response, the server replies with a SYN-ACK.
3. Finally the client sends an ACK back to the server.

At this point, both the client and server have received an acknowledgment of the connection.

Here is some information on UDP, which is what the game uses:

UDP uses a simple transmission model without implicit hand-shaking dialogues for guaranteeing reliability, ordering, or data integrity. Thus, UDP provides an unreliable service and datagrams may arrive out of order, appear duplicated, or go missing without notice. UDP assumes that error checking and correction is either not necessary or performed in the application, avoiding the overhead of such processing at the network interface level.

UDP applications use "datagram sockets" to establish host-to-host communications. Sockets bind the application to ports, that function as the endpoints of data transmission.

i also found this: http://en.wikipedia.org/wiki/Maximum_transmission_unit,

(MTU) A higher MTU value brings greater efficiency because each packet carries more user data while protocol overheads, such as headers or underlying per-packet delays remain fixed, and higher efficiency means a slight improvement in bulk protocol throughput.

hopefully that helps.

 

ACKs are used not only in the initial handshake but during the entire connection, although indeed only for TCP not UDP. Demigod uses TCP only for logging into Impulse, the rest is all UDP.

Setting MTU above what your connection can handle will lead to IP fragmentation which only adds more overhead and potential trouble. I think Demigod mostly sends small packets anyway so hitting MTU won't happen very often.

Interesting information guys, thank you very much - I think may have figured out another part of this puzzle.

Once in a game with Demigod, I don't have any real problems remaining connected - at least no more than anyone else. This is because, as Aprogas pointed out, it swaps over to UDP once the game is launched. However while in the lobby - and still using TCP - I would still be receiving the 'handshake' messages, no? This would explain why connecting to other players is a problem in the lobby - once I manage to connect, I can play without further issues or four figure latency. It seems the messages themselves are small however it's the number of messages that is the problem for my particular modem.

Based on this information, limiting my upload would only work in a lobby - once in game it would probably effect my latency, however wouldn't provide additional stability in connecting? I can't pretend to understand much about networks, I'm educating myself as I go, however this makes sense in my understanding? Please correct me if I'm wrong.

this could possibly be the reason you find it hard to connect to players in the lobby. it basically means the ports may not be connecting you to another player properly. it puzzles me why they use UDP for this since it is a crucial part of the game.

anyhow this is only a suggestion.

it always begs the question.. if i can play other games online with many people with the same network setup, why wont this game work?

 

Hmm, I agree.

I can only hope GPG are working on a solution in the actual net-code for Demigod, rather than the behind the scenes stuff Stardock have been doing, as it seems this will be the only chance for a full-proof solution.

Limiting your upload speed from a program itself might reduce the amount of ACK sent, but limiting the total upload speed of your connection only makes the ACK problem worse. A better solution would be to give the ACKs priority so that if the upload is clogged, the ACKs are sent first and the remaining available speed is used for other stuff; however only few routers support this and it requires technical knowledge to setup. Neither of these things will help you with Demigod connectivity though because while playing you shouldn't be running any network intensive background processes anyway. You could use a tool like NetLimiter 2 Monitor to check if there are any other processes eating a lot of network.

wtb your networking knowledge.

 

I'm glad there are intelligent people who know more about networking than I ever will trying to figure out this problem .  I did the test you linked ZehDon and I got similar results as Grieve.  All these suggestions put a lot of light on the situation.  I'm not as frustrated at not being able to connect when I have some insight on why I'm not able to connect.  Just wanted to say thanks and to keep up the good work! You all get cookies! 

	Downstream Bandwidth:	>=10 Mbps
	Upstream Bandwidth:	2592 Kbps			Upstream Queue:	1046 ms

I used the trial version of Net Limiter 2 to check my theory, with interesting results.
I did as you suggested and limited the upload of Demigod itself rather than my entire connection, and I found I was still able to connect - however my connection time to each person was consistant rather than increasing per person I connected to. Basically, connecting to the host took significantly longer than normal however instead of the connection time increasing as I connected to more people, it appeared that it remained the same for everyone. The problem however was that doing this increased my ping significantly (from 100ms for Australian connections to around 550ms) and I was kicked from the game; at which point Demigod crashed.
I then connected to a game with Net Limiter disabled, and was able to connect to all but one person - interestingly enough the connections to all the other people in the game were very very quick, however the last person and I were simply unable to connect to one another. Once he left, another person joined who I was also unable to connect to. They left, and we launched a 3v3 instead of a 4v4. One person did disconnect, at which point the lag spikes seemed to disappear, however the match was fairly solid despite a large pause in the mid-game for unknown reasons (has happened before).
I think the improvements I've seen are from the most recent update rather than anything I've done, so I believe Stardock might fix my issue from their end before I will fix it from mine.

this actually happens to me now ZZZ, as soon as the 5th player joins BOOM my ms to every other player goes up to 1500ms

you should also try configuring QoS on your router. unfortunately mine doesnt support QoS , but yours does

more info on it here: http://en.wikipedia.org/wiki/Quality_of_service

Thanks man, I'll look into it tomorrow - though reading that Wiki page, seems like it's certainly a step in the right direction. I cannot wait to have all this sorted out!

Another thing that I worry about, which may or may not be an issue, is that they have debug logging activated in the lobby.  I have a 10k RPM harddrive thats obnoxioulsy loud and it goes berserk during the lobby connection phase.  Once everyone is done, it calms down.  But hopefully thats not impacting any performance.

But anyway, interesting read.  I was also noticing I was having a problem connecting to the 'last' person in the lobby everytime.  Didn't make a whole lot of sense though.  Yes - the number of connections goes up in a n^2 fashion as you add people (since everyone has to connect to everyone else).  But, if I currently have 8 connections and have to add a 9th, thats not increasing the bandwidth that my computer needs send out dramatically.  Its only one additional connect.

I just find it odd that connecting to the last person is being troublesome.  Makes me wonder if something more than hardware buffer size limitations going on.

Anyway, totally different problem but wanted to add to the thread, since they are within the same part of the code

 

 

 

 

 

 

 

I've spent the last few days testing out the new patch and proxies, seeing what kind of difference it has made to my problem. Firstly, I no longer have any troubles connecting to people - the Proxy system kicks in on those people I wasn't able to connect to before, which is fantastic. However, since the Proxy servers are located in America and I'm an Australian, my Ping goes from the standard 80-120ms range for local connections to around 550-1250ms. I understand at current there is no location logic in the system, however it's now left me unable to play when the system uses the Proxy servers to connect me to anyone else which kind of annoyed me. I was under the impression the Proxy patch was held back so that location logic could be in the patch. Until that logic is implemented, I don't think anyone apart from North American players will receive much benefit. And those people from Sweeden with their impossibly fantastic internet connections.

Secondly, I've managed to get a few games where I didn't use the Proxy servers to connect to anyone. A new problem has arisen where the game is rather 'jumpy' and all the players feel rather frequent lag spikes. Turns out, it's me causing this problem. After doing some math based on Demigods requirements, it seems my 1500kb download/256kb upload connection (considered middle of the range in Australia) isn't sufficent enough to handle Demigod's unforgivingly high bandwidth requirements. This, it seems, is the central cause of my problem; not the ACK Packages as I believed above. After playing around with other games, it seems Demigod's bandwidth requirements per connection are way above other games - even other RTS games such as Starcraft and Supreme Commander (figure that one out and you're better than me) which is odd because the latter throws around thousands of units with minimal network slowdown. This has apparently been identified already and is being incorporated into patch 1.01 which provides a, roughly, 35% decrease in the bandwidth requirements. It doesn't explain why the hell Demigod uses so much upload. I'm thinking poor optimisation and lack of forethought. Steam identifies that a 2048 Kbps downstream; 256 Kbps upload is the most commonly used network connection among gamers on it's Network, so I can only speculate on why Demigod wasn't optimised for this speed of connection.

So, it seems I'm back to the waiting game while they bring out 1.01 which may or may not fix my problem. Anyone have better luck?