Hawken
Free Mech Test Drive: Sharpshooter, Grenadier and Raider
HAWKEN "War is a machine" coin
Crush the opposition event!
Weekly Update - 2015-08-28
1. Technical Artist interviews
2. Gameplay programmer interviews
3. Private servers should now automatically recover from hard locks in most all circumstances
4. Added another gameserver host in Singapore
5. We got game servers working in the new Staging environment
- Now we can really hammer away at configurations
Upcoming over the next week
1. We need to decide on whether or not to keep server hosts in Australia
- We could shift the hosting fees toward other hosts in the Asia region
- We may find a data center that has better peering with a wider range of locations
- Let us know via a support ticket what you think
2. Working with concept artists
3. More interviews
4. More build and deployment work
Priority Target | Bronze match & Finals (Sat 29/08)
This Saturday, 29th, will be the Finals of the Priority Target Tournament. First tough, we're going to do a bronze match for the 3rd and 4th place, at 15:00 UTC. Then, at 16:30 UTC, will be The Casuals vs Pizzabangstormweaselparty64. Both will be streamed through Twitch.
15:00 UTC / 8:00 PDT : Crushing Kill vs Mystery Magic Rozer
16:30 UTC / 9:30 PDT : The Casuals vs Pizzabangstormweaselparty64
The streamers will be Erzunterweltler: http://www.twitch.tv/erzunterweltler and Freeze: http://secure.twitch.tv/dr_freeze001. Also, the Bronze match will be streamed and commentated by members of the teams who are not playing, so keep an eye out for links in announcements for each stream, when the matches begin.
For more information: https://community.playhawken.com/topic/3183-priority-target-bronze-match-finals-sat-2908/
Double HC Weekend and Free Mech Test Drive: Berserker, Brawler and Raider
Weekly Update - 2015-08-20
1. Set up new gameservers hosts in Phoenix and Atlanta
- After looking at more ping data, we're going to try some more data center options
- These, along with Chicago, are looking pretty good so far, a tangible improvement
2. The new Storm hosts are performing well; login times are a little better
- there are still reports of "very long sync" times, so submit a support ticket, especially if you'd be willing to run a wireshark profile while you log in and experience the problem
- We set up some telemetry to better monitor how long the client waits to complete account syncing
- The vast majority of clients experience relatively quick account syncing, so it's not a systemic, everyone-experiences-this issue
- We think we can find more ways to improve the situation though
3. We got a new Storm environment set up that is already being used for new development work
- Now it's much better documented :)
4. Double HC weekend starts today until Monday, Noon PDT.
What's up in the next week:
1. More staging environment work - gameservers will be required soon
2. We've got a bead on a possible lead Gameplay/Generalist programmer, so I'm excited, but availability may not be immediate
3. Sr. Technical Artist/programmer - still looking
4. UI Programmer or a gameplay person with UI experience
5. The Priority Target semi-finals are happening this Saturday
- We want feedback on how you think we can better support tournaments like this
- And, yes, we want to help your next tournament be a success
- Submit a support ticket and we'll get things rolling
I'll try to post up some of that nifty ping data we've got going. It's pretty interesting.
-capnjosh
Weekly Update - 2015-08-13 - Server Performance
I figured I'd describe the hosting situation and what we've seen in regards to reported "server performance problems":
1. I have personally obtained dedicated physical servers in all locations. They are generally running Xeon E3-1270 processors. We cannot really get much faster CPUs.
2. All gameserver hosts are on at least 1Gb internet uplinks.
3. We have had gameserver hosts in 13 different datacenters and we've used no fewer than 6 independent hosting companies. We have tried both C-States enabled and disabled, various power configurations, hyperthreading enabled and disabled, turbo enabled and disabled, spread spectrum enabled and disabled.
4. Several times now when we have received Wireshark capture files, we have seen very high levels of "Duplicate ACKs" and "retransmission retries", both of which most often are indicative of packet loss.
5. CPU utilization on the gameserver hosts is relatively low
6. We have built out reporting that shows all client-reported ping times by server. The ping times are as expected, given geographical proximity. There are exceptional cases too, where a single user is showing either high pings or wild ping ranges even when they are geographically right next to many users who report low, steady pings.
7. The survey results came back showing that only 4% of respondents indicated they had very bad network problems, with another 11% or so indicating slightly higher levels of performance issues. The remaining 89% indicated no performance issues or they "sometimes" experienced them.
8. Gameserver telemetry has not been indicating high tick processing times (indicating server CPU overload)
9. Gameserver management systems are exactly as they were before we ever worked on Hawken
Now, there have been cases where gameserver hosts have been overloaded, or we have received splash damage from DDoS activity, or some routing peer has had network problems, or I have screwed up some server configuration, or the back-end data layer has had problems.
However, from all I have seen, the problem is very likely *not* the hosting, and that's based on the above data.
Remaining things to pursue may be:
1. optimizations to replication data size
2. adding further network monitoring features to the client and server (to comprehensively monitor all routes for all players, e.g. WinMTR and UDP delivery rate software that can run in the background on all clients all the time or as desired)
3. adding client-side network configuration reporting features (to be able to see what configs may or may not predict network performance results)
4. investigate what else could have changed over the past year
5. More WinMTR routes from client machines while they are experiencing what looks like server performance issues (these will show if it's due to packet loss and at which router packets are being lost)
Regarding the high number of gameserver instances, we will be reducing those, but we want to gather data from all the gameserver hosts during the same period of time so we can compare "apples to apples".
The dev team hiring process is still ongoing. I know for a fact there is frustration we have not staffed up and already released the first patches - hey, I wish we could have already reached that point, so I feel your pain. It's moving and it's definitely happening though. Once again (as we all feel like is stated too often), it's a matter of "when" not "if".
-capnjosh
50th Ultra-Violent Wednesday!
We've come a long way with UVW, developed many game-modes and met many people. From Adhesive, through the Great Silence and now with Reloaded, we've been going for a while. I'd like to thank everyone who has made this possible, both current and past admins, people helping out with organizing and entertaining the people, the behind the scenes folk at Reloaded bringing the servers back when they go down and, of course, everyone who's joined in the past and, hopefully, will join again.
So I invite everyone to join in tonight's event. Get on the Officially Unofficial Hawken TS (ts5.gameservers.com:9152) and enjoy the carnage!
https://community.playhawken.com/topic/3053-50th-ultra-violent-wednesday/
Double HC Weekend and Free Mech Test Drive: Infiltrator, Vanguard, and Raider
Also, for a limited time, you can test drive the Infiltrator, Vanguard, Raider Mechs for free. Stop by your garage to test drive these Mechs until Friday, August 14th.