Shadowrun Chronicles: Boston Lockdown
New update out
More Karma for side missions and final mission!
Challenge missions coming up next!
Updates and an outlook
The first update will contain:
- New clothing and pets...yes, pets. Though only for cosmetic purposes and only in the hub
- New clothing
- A fix for the infamous "cannot invite bug"
- Summons will die with their controlling character
- A number of fixes to skill effects and descriptions
- All Side mission grant 3 Karma in co-op and campaign
- Final Mission grants 6 karma to co-op and 3 more for campaign
---
The next one will have:
The first challenge mission(s) featuring one or two multi-stage missions and new weapons, gear and more pets to get as loot.
Challenge missions are non-story related special challenges, which pit players against three waves of enemies with unusual combinations or new abilities. After each stage players can opt out an go for a consolation price or continue onwards to get the ultimate reward. The setting is prepared by your favorite party dragon Damon, who has taken it on himself to provide the entrapped citizens of Boston with fresh trideos and an ongoing show. Following his great success with The Hunt series in the Rox, he has devised a new show where runners are pitted against a combination of enemies. Already betting is getting frantic around these events!
---
Following this we will focus on the asymmetrical PvP, meaning player(s) against a game master. This will take a bit, so in the meantime we may release more challenges or a mini story connecting the campaign to the Salem add-on.
Frag the lag!
We have finally fragged the lag issue it seems when our servers will be patched one last time at 15:00 alongside some other fixes.
We don’t want to jinx it, there may still be gremlins lurking around, but we seem to be back to normal. Today, with our hopefully last lag related server patch, you will find your well-deserved apology gifts on your inventory:
- 3 Flamethrowers (so you can flame enemies and think of us)
- 1 additional character slot
- 2 new Bioware augmentations (Zero Essence, usable for awakened characters as well)
Many people have given us good advice on how to handle the issues and told us what they suspected the culprits were. As it turns out, we had three main issues and tragically two of them have very little to do with our team’s ability to code and code well. It is both more tragic and more comical tale (that is if it hadn’t affected our rating s and sales so massively. Maybe one day, looking back, we can laugh without bitterness about it).
The first issue was not enough optimization in certain areas – as described previously, the game had been working so brilliantly from a technical and performance standpoint, that we did not dedicate too much resources to it. We knew we needed to do more optimization, but it did not seem to be a big concern. If it hadn’t been for the other two things, we might have been right.
Issue two was one of our middleware licenses actually putting the break on our servers once a certain number of concurrent users were active (which we were not aware of). Above that number it would slow down throughput, so all our other services (which kept listening to certain commands or pushing things to the client) slowed down with it. As some of them were serialized, meaning they were acting like a chain, they were actually really waiting for something to happen from the previous link of the chain, multiplying the effect of the license slowing down the speed at which commands were coming and going and putting other services like load balancing and thread management into overdrive, crashing CPU. Our servers overheated with waiting and managing that slowness, absurdly enough. This was fixed easily enough once discovered but it took us a few days to identify this weird behavior and the culprit.
The third and most tragic point has to do with the letter D. As in “enableD=garbage collector”. While we optimized the garbage collector (which cleans up “leftovers” on the server, so they don’t clog memory) and following some of the suggestions we got from our code-slinging community members, it did not seem to do the trick. We dug deeper and deeper into garbage collector settings but to no avail – each time the game would just use up the memory available and despite further optimization efforts this did not change considerably. The reason was finally found to be a missing “D” in the config file, which read “enable=garbage collector” instead of enabled. Of course we had looked at every line of this configuration file what must have been a hundred times, but you know how the brain tends to just create the word it expects to see, especially when you did not have more than two hours of sleep for days fighting a lag that was killing your game? This was one of the cases (and enable and enabled are not that far apart). Once we found it, we hit ourselves over the head repeatedly with blunt objects.
All three issues combined killed us in a most unfortunate way. Now with those resolved we actually have 10.000 concurrent players in our mass test and the servers don’t even get warmed up about that. We are cautiously optimistic we are ready to declare the lag fragged for good!
Now we can get back to actually improving gameplay and adding end-game missions and rewards! Please spread the good news and let your friends and Steam or Metacritic ratings know, so we can try and get the game off the ground again.
Thank you all for your patience and support in this really rough patch of our life as a developer.
Legend Run with Jordan Wiseman today!
Once more unto the breach dear friends, once more
Many of you have been first hand witness to the frantic post-launch struggle of Shadowrun Chronicles as we patched time and time again only to still end up with a lag when playing. Now to the players this looks like we are constantly running against the same wall until our heads bleed and have only managed to make small dents in said wall. From the inside however, it is a completely different story and we wanted to give you an inside perspective on what is going on.
In the last 6 days we have applied an increasingly large number of fixes and changes, from optimizing garbage collection, sequential threading to deep-copy of data to wrenching open our middleware, which actually contributed to our issue in slightly invisible ways (think of middleware as an engine under a hood, so you don’t always see what is going on exactly but you can hear the engine stuttering)
Each of the changes and fixes was profiled and supposed to bring a good amount of improvements – and it did – we are winning the battle against lags millisecond by millisecond, but it is a war of attrition. This is exacerbated by the amount of time we spend in lag and new players getting bad impressions while our Steam review ratings plummet. Now we have a simple but cruel choice: Do it properly and test a solution for days while our game rots away in lag-land or pushing a fix online with a minimal period of testing, which however results in a bit of profiling and a sanity check if the game explodes immediately or shortly after starting the servers - tempered by a judgement call by our coders of “how dangerous it is versus how likely to fix our problem”. So we send out an annoyingly high amount of patches and updates with annoyingly small effect and are basically really field testing things on the live servers, which we all know we should not do, but then we are between a rock and a hard place. This is why online games always launch as beta…
All this time, our brave coders fight invisible foes such as a maximum user cap in a license that then puts the brakes on certain parts of the performance and carry the burden of working above and beyond the call of duty while being rained with derision by angry customers. The rest of the team watched in agony and frustration as we could not do anything to help and all other things took a back seat to solving the problem while our sales and our users patience dwindles away. Imagine the morale in the team! And still we worked at night, with the community team taking the brunt of the damage while taking turns to be there 24/7. I want to personally thank them and the remaining team for enduring and persevering. You guys and gals are the finest team I have had the privilege to work with. Even though we fragged this up you have been exemplary in every aspect.
So what now? Well, yesterday’s update resolved a lot of issues (reducing frequency and of lags), but foreseeably opened others exploding our servers after around 3 hours of operations. Today we rolled out two server updates – the second one of them dealing with the infamous garbage collector and settings. As this pushed CPU load to a baseline of around 30% and lag spikes appearing again, we reverted the changes…only to see lag creep in 2 hours into operations. So now we relaunch servers every two hours…because this is the only option we have.
Tonight we have finally accepted that “hoping this will be the fix that really does it” is simply wishful thinking. We have run out of silver bullets and we will focus on making the frequent server restarts automated and manageable, creating schedules and having automated in game announcements preferably everyone can see, so everyone can live more comfortably in the emergency state. This should create some air for us to breathe and look at our data and find new options to improve performance.
As a personal note, many of you – even those who have been critical of the game – have been incredibly supportive and understanding in the past days. Helping us, explaining things to others and cheering us on, honoring the work and dedication we have put in. This literally is what has got us through this week (other than coffee and sheer stubbornness) and we are deeply grateful for that.
So let’s see what the next days may bring and keep our fingers crossed.
Oh and Legend Run Event No.3 with none other but Jordan Wiseman himself happens on Friday 3 pm PST or Midnight CEST! Lag or no lag!
See you in the shadows
Primetide
The best laid plans...
We know the risk associated with online games, especially if it is structurally and technically an MMO (in the sense of most of the AI, Logic, Persistence and all the other stuff that makes the game operate being done via server). This is why we have been extra careful before we released Shadowrun Chronicles – the game had been running for months before launch without any issues (as our Early Access and Kickstarter players can attest to), we did a series of Mass Tests (simulating thousands of concurrent user sessions) and a day prior to launch we invited our Early Access community to come in droves by allowing them a day head-start for their characters (whom the previously had to endure to see wiped time and time again as we balanced and reworked the game). We upgraded our hardware to be extra safe, even though our servers rarely went above 5% load.
Come launch day and things work without a glitch…until they suddenly don’t. Shortly after our studio fell victim to a district wide blackout at the most inopportune of times, our external server inexplicably slows down. The source for this was quickly identified (once we were back on line) – a specific task called “garbage collector”, which cleans up leftover allocated things in the memory and with the amount of players suddenly has too much to do, which results in things cueing up and eating away the hardware resources for every other process (like walking from a to b). So we reduced the amount of garbage produced, patched, and off we go…
Only to see lag creeping into the game again. Looking at the server actions and our logs, it turns out our servers clog up with threads they cannot compute fast enough, leaving ever more threads hanging, which results in the game becoming ever more unresponsive. So, what do players do, if the game does not react when they click on something? Right, they click on it again and again and again…creating more commandos for the server, who is struggling as it is. Vicious circle anyone?
Now these issues are something that accumulates over time…which is almost impossible to test without actual people doing a thousand actual things which differ from our mass tests. It also exposes problems by adding a magnitude. Things that worked badly but did not register when we had a little over hundred players concurrently suddenly explode with 500 players. Take our hub: To be able to see other players running around, the server needs to collect the “commands” from every player and update every other client about the movements of each character in the hub. Previously that was never an issue. But with dozens of hubs open the small things stack up. And if they do, the server gets clogged up…and if that happens long enough it simply cannot work through commands fast enough…resulting in lag. One of the patches has reduced the size of each command (so the amount of data the server has to compute) to 20% of what it was and optimized the associated logic at the server, so it is 40 times as fast as it has been before.
So why didn’t we do this before? Simple: It wasn’t a problem, so we weren’t fixing it – every change of a running system has the risk of creating new bugs and if it ain’t broken…
The example above is but one part of the complex and interwoven web of logic and relations and we wish we could enter that code Shadowrun style and hack our way through things with ease, but unfortunately that is not the case. Instead our 6 coders and IT as well as QA have worked day and night for three days to correct problem after problem, but as with any intricate webs, when you start pulling one strings, the whole thing changes and transforms. So every patch improved parts of the problem…only to see another issue creep up. And we may not have seen the end of this, as we continue to fix things. And it does not help if people don’t sleep for over 40 hours…
TL;DR: This is fragged and we are desperately trying to fix it, but we are stretched to the limits.
As a way of saying sorry, we want to give everyone who endured this week of constant patching and those who just gave up a little gift and ask them to give the game one more shot.
Mid next week, every user will receive three things:
- A free additional character slot (so if all that patching messed up your builds, you can start a clean one)
- Two pieces of cultured bioware – these cost zero Essence so you can use them with any character even mages or make sure you cybered up street sams get that extra boost they have been looking for
- A flamethrower, so whenever you get really angry at us for that drek, you can just incinerate a few enemies and think of us.
We are still working on creating a PvP and Endgame mode – most of this is design and thus not overly affected by the massive amount of work our coders had to do, but then our coders will need some sleep after this marathon.
Again, we are sorry for this and after putting years of work into this, it is unbelievably disappointing for us to see this happen and leave a bad impression.
The best laid plans...
We know the risk associated with online games, especially if it is structurally and technically an MMO (in the sense of most of the AI, Logic, Persistence and all the other stuff that makes the game operate being done via server). This is why we have been extra careful before we released Shadowrun Chronicles – the game had been running for months before launch without any issues (as our Early Access and Kickstarter players can attest to), we did a series of Mass Tests (simulating thousands of concurrent user sessions) and a day prior to launch we invited our Early Access community to come in droves by allowing them a day head-start for their characters (whom the previously had to endure to see wiped time and time again as we balanced and reworked the game). We upgraded our hardware to be extra safe, even though our servers rarely went above 5% load.
Come launch day and things work without a glitch…until they suddenly don’t. Shortly after our studio fell victim to a district wide blackout at the most inopportune of times, our external server inexplicably slows down. The source for this was quickly identified (once we were back on line) – a specific task called “garbage collector”, which cleans up leftover allocated things in the memory and with the amount of players suddenly has too much to do, which results in things cueing up and eating away the hardware resources for every other process (like walking from a to b). So we reduced the amount of garbage produced, patched, and off we go…
Only to see lag creeping into the game again. Looking at the server actions and our logs, it turns out our servers clog up with threads they cannot compute fast enough, leaving ever more threads hanging, which results in the game becoming ever more unresponsive. So, what do players do, if the game does not react when they click on something? Right, they click on it again and again and again…creating more commandos for the server, who is struggling as it is. Vicious circle anyone?
Now these issues are something that accumulates over time…which is almost impossible to test without actual people doing a thousand actual things which differ from our mass tests. It also exposes problems by adding a magnitude. Things that worked badly but did not register when we had a little over hundred players concurrently suddenly explode with 500 players. Take our hub: To be able to see other players running around, the server needs to collect the “commands” from every player and update every other client about the movements of each character in the hub. Previously that was never an issue. But with dozens of hubs open the small things stack up. And if they do, the server gets clogged up…and if that happens long enough it simply cannot work through commands fast enough…resulting in lag. One of the patches has reduced the size of each command (so the amount of data the server has to compute) to 20% of what it was and optimized the associated logic at the server, so it is 40 times as fast as it has been before.
So why didn’t we do this before? Simple: It wasn’t a problem, so we weren’t fixing it – every change of a running system has the risk of creating new bugs and if it ain’t broken…
The example above is but one part of the complex and interwoven web of logic and relations and we wish we could enter that code Shadowrun style and hack our way through things with ease, but unfortunately that is not the case. Instead our 6 coders and IT as well as QA have worked day and night for three days to correct problem after problem, but as with any intricate webs, when you start pulling one strings, the whole thing changes and transforms. So every patch improved parts of the problem…only to see another issue creep up. And we may not have seen the end of this, as we continue to fix things. And it does not help if people don’t sleep for over 40 hours…
TL;DR: This is fragged and we are desperately trying to fix it, but we are stretched to the limits.
As a way of saying sorry, we want to give everyone who endured this week of constant patching and those who just gave up a little gift and ask them to give the game one more shot.
Mid next week, every user will receive three things:
- A free additional character slot (so if all that patching messed up your builds, you can start a clean one)
- Two pieces of cultured bioware – these cost zero Essence so you can use them with any character even mages or make sure you cybered up street sams get that extra boost they have been looking for
- A flamethrower, so whenever you get really angry at us for that drek, you can just incinerate a few enemies and think of us.
We are still working on creating a PvP and Endgame mode – most of this is design and thus not overly affected by the massive amount of work our coders had to do, but then our coders will need some sleep after this marathon.
Again, we are sorry for this and after putting years of work into this, it is unbelievably disappointing for us to see this happen and leave a bad impression.
The new Shadowrun Chronicles Update is out!
New Features
- Team up! Coop has been extensively reworked and now allows up to four players to go on a run together. Please take note that now everyone on your team can take action at the same time, so no more waiting for your turn!
- Meet up! Other players will join you now in the hub, even if they are not in your group. Meet up, swap tales of your deeds or tactical advice and roleplay to your heart's content.
- Cyber up! Improve your body with Cyberware. Keep in mind though that augmenting your meatbody reduces your essence and thus affects your spellcasting!
- Going deeper: A lot of new missions take you deeper into the story of the Boston Lockdown than ever before! The 3rd Act of our game and several side missions have been added for you to explore.
- Talk to me! Partial Voice overs have been added to the game, bringing our main characters to life! Hear Smedley, Jane, Payday, Sarah and others talk!
- New Map: The Map of Boston has been extensively reworked and now shows districts and sights, but also signals the state the districts are in.
- Skin Color: It is now possible to change the skin tone of your character during Character Creation.
- Paint and Needles: Facepaint and Tattoos are now available in stores to customize your character further.
- Checking in: The Matrix Hub now lets you check back with Johnsons and other characters in situations when you cannot return to the normal hub.
- Starting up: Shadowrun Chronicles now features a proper launcher that allows you to set the language, screen resolutions and to redeem coupons. Please note: While it is already possible to redeem coupons (i.e. weapon or promotion codes), you will NOT receive these items in game till launch.
- Sprechen Sie Deutsch? German text is now available in the game.
Fixes, Changes and Improvements
- The ragdoll animations have been adjusted, so graphical issues when a character dies should now occur less often.
- When starting a new career you will now start the first mission automatically before you get to enter the hub. Please note that it is now also impossible to return to character creation once you’ve made your choices there and entered the game!
- Dialogues in the hub will now be presented with a new camera angle, giving you a proper view of who you are talking to.
- Items in the gear and shop screens can now be filtered by category.
- Female characters can now have beards.
- Some parts of the levels are now destructible (i.e. glass panes).
- New Outfits have been added.
- From now on only one item will be taken into the run per consumable slot, even if you have a whole stack available.
- Beards and hairstyles cannot be sold anymore.
- It is now possible to log out of your current career and return to the selection screen from the options menu.
- In the friends list you will now also see the account name of your friends, not just the name of their current character.
- Characters in the hub now have idle animations.
- The loot notification has been improved.
- The player character should not become available as a henchman anymore.
- Autopicker, Remote Det, Portable Bodyscan and Hacking Autosoft have been removed from the game.
- Some issues with the chat have been fixed.
- Backgrounds, effects, missions. Skills and skilltrees have been iterated, improved and polished.
- Lots of other small fixes, corrections, clarifications and improvements
Shadowrun Chronicles gets new update and Q&A session on March 17th!
On the same day, we will also host the 9th LIVE Q&A with the Devs. Join us at 7pm CET (2pm EDT / 11am PDT) on http://www.twitch.tv/cliffhangerproductions as we play the new version and answer YOUR questions LIVE!
See you there, Chummers!
The Shadowrun Online February Update is OUT NOW!
As you will be able to play a slice of our second act, every character gets Karma points, Nuyen and weapons to start out with. Be sure to learn a few skills and equip the appropriate weapons BEFORE going into a mission!
New Features:
- Create your Runner! The Character Creation Screen is now available in game, including metahuman traits, lots of different backgrounds and many new portraits! In later versions this will only be available when you create a character, for this version however you can come back and play around with the settings!
- All new hub! An extended real-time hub has been added to the game, featuring not only quest givers, but also shopkeepers and NPCs to talk to! Later your team will be able to play in the same hub as a group!
- Improved grouping: You will now be able to form groups with people who aren’t on your friend list – just click on them in the chat and invite them to join you. Groups will also not disbanded anymore once you finished a run, so you can keep playing together i to the next mission!
- Ready for Combat! The gear selection has been added, allowing you to select weapons, armor and consumables for your run.
- Wanting to buy? Shopkeepers will now sell items to better prepare you for the upcoming challenges!
- Dressed Up: The new clothing screen allows you to change your outfits and see which new pieces are available in your considerable wardrobe.
- Improved UI: The in-mission UI has been updated again and now shows more information about your characters and enemies. Hover over any character or enemy to find out more details.
- Rewards: Johnsons will now reward your efforts with Nuyen (and sometimes also items).
- Protection, please! Armor is now in the game, protecting you from all sorts of damage.
- Use this! Grenades, Medkits and more consumable goodness!
- More Skills: All skill trees have been extended to rank 10 and skill values have been rebalanced.
- Drones Incoming! The Rigger skill tree is now in the game.
- Specialist: Henchmen are now archetypes with a primary skill group.
- Check Me Out: In the team selection screen hovering over a henchman will show you vital information about the character.
- New enemies: Fight the raging hordes of the Headcases faction!
- More and prettier! More Missions, more effects, more weapons, new customization items, more animations!