AI War 2
Beta 3.705 Blazing Collections And The New Spire
(Note: all savegames won't work when loaded into this version, because of the major under the hood changes. There will be another release or two like that coming up in the near future, then I'll be done with that.)
Okay, this is getting a bit ridiculous. This is five days of really intensive work (like 12-14 hour days for all five). Initially I started with a few mild bugfixes (fun!), and then I started moving in to splitting up the fallen spire faction and necromancer so that they have the new relationship that they need (cities and their mobile fleets separated out (exciting!), and I got partway through that. I also started working on modules/loadouts (even more exciting!).
Then something else grabbed my attention again: the performance of our collections (lists, dictionary, queues, stacks, etc) left a lot to be desired. The ones that were most problematic were ones from Microsoft, and the source is open. The others that were problematic tended to wrapper these problematic ones.
So at first this led into a big investigation with myself and NR SirLimbo profiling the heck out of the microsoft classes and then my adapted version that stripped out some suboptimal things in their code. In general, depending on what we were testing -- and SirLimbo had to create a new in-engine testing framework for us to be able to measure this with a sensitivity in the nanosecond range -- down to a millionth of a second or so, for our practcal measurement purposes).
The general results of that were that, on average, the worst result from my new versions on windows or OSX was about a 0% to a 2% improvement for certain activities, and then closer to a 10% improvement for others (Sort in particular). Then for certain things that have to do with initializing data to the proper size, the improvement was about 300%.
I then started porting over more data structures, and finding all sorts of really unfortunate things in there. Doing things like looping over dictionary.Keys instead of just dictionary actually is hugely slower, did you know that? It's not documented anywhere, you have to just check the code. There were a bunch of other strange gotchas.
And our own ArcenSparseLookup collection, which Keith implemented 5 years ago with notes in there saying "this is TEMPORARY, and has bad performance, will need to replace this later" has been all through our codebase. I spent the last 9 hours or so (split between last night and today) replacing that thing. It was used all over the place, and had at best half the performance of the newer version of the dictionary that I now use, before you even get into further improvements. And I did get into further improvements.
So how much faster are things? I don't rightly know! These were a thousand little paper cuts, all over the place, slowing things down. We no longer are covered in said paper cuts. Things feel snappier, but I haven't run any real stressful load cases just yet. I'll be very interested to hear what you experience.
I'm going to be out for the next day or two, but then I'm going to be circling back around to bugfixes, the fallen spire, and so on. The fallen spire _might_ be playable right now in this build, but I'm not certain. The ablity to switch which mobile fleet each city is bolstering isn't there right now, and that's the main thing I just outright didn't finish.
Anyhow, lots of exciting progress on a lot of things. I didn't really expect another digression into performance improvements, but these ones are extremely nice and are things I can carry with me into future projects, too.
More to come soon.
Enjoy!
Beta 3.704 Unexpected Turbo Sim Speed
Weeell... this is cool and unexpected. I sped up the sim last release, right? Um... well, I did it again, except even moreso this time. I am able to run 111 factions at almost full sim speed now, which is just plain insane. My framerate stinks when I do it, but that's a separate matter. We also can run far more planets than anytime in the past, so we've increased the max planet counts to 300 on all maps. It does take a more powerful computer to run these, which the game warns you about, but there's no reason for it not to be an option now.
This isn't just about crazy high planet counts or crazy high faction counts, though. This will have tangible benefits to lots of players, whether they are on weaker hardware or are just getting into giant late game battles that previously slowed down a lot, etc. It's possible that battles of sufficient size will still cause slowdown, but the effect won't be compounded by this general load, so there's a lot more margin for giant battles to take up room if they need to.
Quite aside from the above, this release contains just an absolute ton of bugfixes, almost all of which are things that resulted from the great refactor, but a few of which were much longer-standing. It's getting cleaner and cleaner, which is thanks to the folks who keep testing things out and reporting what they find. Thank you in particular to Vinco, BoBy, Arctic, whimsee, nas1m, and HarryT -- you all rock!
Let's see... I also got started on the very earliest part of modular loadouts for ships, although there's not a lot to see yet. But it's nice to be getting started on that. A new planet naming style by Angel of death is now in the game, as well.
If anyone is wondering how NR SirLimbo is coming with his work on his mods (AMU and the like), it's an impressive and intensive task that he's going through. He's documenting it on discord, and it's really awesome to see how he's not just porting things over, but rather is actively making his modding framework and tools even better than it already was.
More to come soon.
Enjoy!
Beta 3.703 Smooth Sim
This one does two really cool things that I didn't have on my radar, but now I'm super excited about:
First, the performance of the entire simulations should now be better. You heard me! Giant planet counts, giant battles, whatever. I'm not saying it will be buttery smooth on every machine, but it should be somewhere between moderately to drastically improved, depending on your machine. A lot of the reasons why I was able to do this have to do with the timings of triggering sim steps, and less to do with the actual sim steps themselves. Essentially we had some dead air time (less than fully utilized CPU) between certain types of actions. Those gaps are now closed, so your CPU will run flat-out when it has to. Also along with this, you can now see (by right-clicking the clock in the bottom left) how close your simulation is to being at the point where it would start slowing down. Nifty!
Secondly, the speed of the simulation is measured more accurately AND it does a much better job of sticking closer to 100% sim speed when things are going well. It averages between 99-103% now, whereas previously it was almost always 95%. That 5% drop was causing visual micro-stutters in the movement of ships, so now things move much more smoothly around. I've tested this at 30, 60, and 100 fps.
There are a number of notable bugfixes in this one, including the fix to the "AIP goes up by 10 every time I load a save" bug, and errors with wave and spire debris notifications.
I also made two new youtube videos today to help modders and faction designers understand the intricacies of multiplayer and threading in the game as much as I can:
https://youtu.be/EV2WTB5l0vA
https://youtu.be/7JxIbYIy_2I
And for those who would rather skip that degree of info dump, there are diagrams: https://wiki.arcengames.com/index.php?title=AI_War_2:The_Great_Refactor#New_Diagram_For_Modders.2FProgrammers:_AI_War_2_Threading_Model
I am now done with all of the videos and similar supplemental materials I had planned for modders, which is excellent. Now I can really continue to focus just on bugfixes and other improvements.
At the moment I know there must be plenty of bugs in here, but there have been very few reports. If you have some spare time and would be interested in testing, please do give the latest betas a play!
More to come soon.
Enjoy!
Beta 3.702 Bugfixes And Mod Conversion
This one is mostly bugfixes, but also converts the Civilian Industries mod to work in the new framework. There's a four and a half hour video where you can watch me doing that, if you're inclined.
This also specifically disables all of the other code mods that are known to not work. They are still distributed to you, but if you go in the file system and get them to be enabled again, all you'll get is errors. So... uh... maybe wait for the mod authors to get those updated first. Non-code mods largely were not affected at all, but now there's no longer a case of you having to wonder "should this mod work, or should it error?" If a mod that you see in the distributed list from us is broken now, we want to hear about it.
On that subject NR SirLimbo is working like crazy to get his selection of popular mods ported over, but it's going to be a TON of work. He's been building his own modder framework on top of our codebase (that's what "AMU" is for), and so naturally that's a lot more difficult to port over than a "traditional native code mod" like Civilian Industries. You can keep up with his progress and send him encouragement on the modding section of our discord server, but he's estimating it may take him a couple of months of very intensive work to get it all going again (after all, unlike me, this isn't his fulltime job, and that's still with him really working like crazy on his projects).
This build fixes most of the really major bugs that I am aware of, although I am aware of several that I didn't have time to fix today. I'll get those tomorrow, along with continuing on with whatever else is new. Thank you to everyone doing the testing on this new code framework!
Once the errors start quieting down, then some of the other exciting things I have planned are going to get some attention, like loadouts for fallen spire ships, better fallen spire fleet organization/setup, and... well, there's a whole list of things prior to DLC3, and then there's DLC3 stuff as well. Expert mode will be getting some more attention soon, too, but it's all mostly based around how long I am responding to bugfixes. I'm definitely pleased with where things are, overall, at the moment. The game is going through those first bumpy bits after a major transition, but it's not quite as bad as I had feared it might be (so far -- it's early yet).
Enjoy!
Beta 3.701 Natural Object Order
This one is just bugfixes, as you might expect. I also made something like 4 hours of video tutorials on how to mod the game and how to use a variety of the code, which you can check out if you're into that sort of thing. The goal of that was to help ease the transition for existing modders into the new framework.
The number of bugs that people found were not overwhelming in number, which is nice, but I think at least part of that was because the ones that were found were severe enough to prevent too much testing. Ironically, most of those really severe bugs were just a few small things that were pretty easy to fix, so... well, that's a good thing.
The really big culprit was that I had allowed NaturalObject (aka no ownership) units to be in any order, rather than always being the 0th faction in the list, and that caused all sorts of crazy bugs if you happened to have it out of position 0 in your new save. If it was in position 0, then a lot of those same things were fine.
Other bugs included a few xml typos, a couple of years-old bugs that just became relevant but were not new, and some places where I had switched something to a new more efficient format, but the old format would throw an error if it was still called, and there were still some lingering calls to them.
All in all, so far none of the bugs were really very time consuming to fix, but as noted they were severe enough to really prevent deep testing by very many people. Hopefully this build is a lot more straightforward for people to test further with.
Thank you for the help with testing!
Enjoy!
Beta 3.700 Bulk Of The Great Refactor
Wow, it's been 26 days since I last pushed a public release. You know, the last release was called "Code Will Change," and that was SUPPOSED to just be a reference to a Persona 5 Song (Life Will Change), and a kind of play on words about how much code had changed in the prior build. Instead, uh... it was prophetic.
The last 26 days have been grueling and intense, but at the same time really satisfying. The release notes JUST FOR THIS RELEASE are 44,000 words. That is novella-length, and it is also similar in length to entire sections of release notes from AIW2 and AIWC in the past.
So... what the heck is going on? Essentially, I've reconstructed most of how the game talks about "external" data, and made it a lot more controlled, better-designed, and multiplayer-friendly. It's a lot harder for modders (or faction programmers, or me) to make errors now. RAM usage is down, CPU usage is improved, bandwidth requirements should be way down, and many other benefits.
On the front end, in-game, not a whole lot is different. In this specific build, there are a few things that don't work or are temporarily disabled. Tutorials, random factions, beacons, multiplayer, and most mods are out of commission temporarily. The savegame format has also been completely reworked, so all savegames prior to this version are now unusable in this version.
Why was this worth it? Well, I had kind of hit a wall with multiplayer where it was just going to be a mess in an ongoing fashion, and this was the only way forward. But even for single player, this brings clarity to the code, makes modding easier, and reduces the chances for errors. It also should help performance, particularly in large and complex games.
We're working towards completing all aspects of this game and putting it into "long term support" by January 2022, and I wanted to really make things shine here. On the surface, not a whole lot is different in this new version, but under the hood this is just a whole new world, and I'm really proud of how it's turning out.
Expect some bugs in the short term, but please do help us test, if you don't mind. You do need to use the beta branch to see this build, of course, for now. Also, I will be breaking savegames more in the near future, just as a fair warning. There are some notable improvements I want to make to various factions (like fallen spire, etc), that should make the user experience better. I also am planning on getting loadouts in place, and adding more features to DLC1 to finish up the last kickstarter features.
Anyhow, we have a lot of things going on at the moment: kickstarter last features, DLC1 additions, loadouts, expert mode, ship encyclopedia, all the stuff for DLC3, general polish, and finishing up the last of multiplayer. This giant refactor is a strange thing to do late in the life cycle of a game, but it's hopefully a solid demonstration of our commitment to wanting you all to be able to enjoy this game for years and years. My time working on this in a fulltime capacity is winding down, and we plan on releasing the Complete Edition in January, but this game has a long and healthy life ahead of it, I predict.
In a month or so we'll also be adding a Super Supporter optional DLC that lets you help offset the costs of development, which really have been way higher than earnings. At the moment, the game needs to gross another $800,000 or so before I'll see my first dollar of actual profit from this five year project. I don't know that that will ever happen, but at the moment I'm cool with that. I'm just really proud of how this is turning out, and I'd like to see this end on a really positive note.
More to come soon. Expect rapid releases on the beta branch again, now that the bulk of The Great Refactor is done.
Enjoy!
Beta 3.605 Code Will Change
This one's title is a reference to the Persona 5 song that I'm rocking out to right now (Life Will Change), because I'm feeling pretty darn good after today's work.
We now have some networked personal settings that can be handled really easily with xml and which transmit themselves to the host ONLY if they differ from the defaults, and which allow us to make intelligent automation decisions on the host based on client data that was previously inaccessible.
We also now have a whole new framework for checking achievement statuses, tips to give you, automation to run, and many other things that were previously very expensive (in terms of CPU time) when you add it all up cumulatively. We used to run that stuff (until today) on the main simulation thread in what we call the "stage 3 faction code."
It turns out that had a number of problems with various things like the host being a spectator in MP, or the host not being a human empire (maybe they're a necromancer), or even in solo games if there is no human empire at all (maybe you're playing solo ark plus vassals, or maybe ark empire). All those cases now work, whereas before they would not have.
More than just solving problems, though, this provides a really high-performance and easy-to-read model for modders to follow with their own factions. It is quite "puppy proof" (as described in the prior release notes), without restricting what coders in that part of the codebase can do. So, yeah. I'm feeling good.
There's still more to do for multiplayer (waves, some notifications, and AIP are not visible on clients right now, and I'm not sure MP clients even connect properly because of general bugs that I just haven't bothered to fix yet during this rip and tear phase). And there are a couple of other little odds and ends there. But overall these are great strides forward, they benefit both solo and MP, and we're coming closer and closer to being back off the beta branch for solo, and hopefully then shortly after out of beta for MP.
Happy weekend!
Beta 3.604 Puppy Proofing, Part 1
Not a lot is different in this one, from an end-user player perspective, but... uh... a LOT is different, at the same time? I fixed a couple of minor bugs, or made them report themselves better so I can see them and smash them when they surface again. Also, chain lightning looks suitably awesome again. But that's about it...
...Until you see this giant section called "Rip and Tear for Multiplayer." I am essentially ripping the guts out of many parts of the game, and putting it back together differently. At the moment, on the beta branch, it means multiplayer is completely trashed, by the way. Why do all this -- is this just shuffling the furniture around?
No, it's... super important, but it's really also very hard to explain. But I'll give it a shot. See... coding multiplayer games is hard. And no two are exactly alike. Trying to code multiplayer is like trying to... um... analogies are also hard. It's a bit like trying to teach in a foreign language that's just similar to one you actually know? The assumption is that you're going to make many mistakes, some of them hilarious, and people will correct you and it will be fine.
The problem is, that's assuming that you have someone to TALK to, in that analogy. Let's instead imagine you are writing a paper in this foreign language you sort of know. It's going to be a long paper, maybe a dissertation. Maybe a whole series of novels! This is a big important project to you, and you keep it mostly under wraps until it is done. You test it all in single player and it's great, and then you show it to people and... well, native speakers are going to give you an absolutely apocalyptic amount of feedback, and require you to pretty much redo the whole thing.
Right, so who am I talking about, anyway? Me? Badger? Modders who code to the game?
...Yeah, pretty much all of the above. Every multiplayer game is like a new foreign language, and as I've been developing this one, I've been putting in safeguards against even myself, and learning what doesn't work. But what gets really discouraging is when there's a seemingly infinite surface area for cross-threading issues or memory leaks to pop up in, in particular. So, with my code architect hat on, I have to think about that less from a "just fix what's in front of you" sort of manner, and more from a "make it so that we can't break it so easily" standpoint.
The engine of your car is really dangerous. It's hot, stuff moves around really fast, there are gears, etc. But it's also REALLY hard to get your hand caught in there. A lot of thought goes into making it so that the appropriate service technicians can get in and do what needs doing for maintenance or for more drastic repairs, but so that they don't cause collateral damage and nobody gets hurt. This is a lot like that.
I'm less worried about the bugs of today, as far as multiplayer goes, than I am the bugs of next year, or the year after. If AI War 2 has a rich modding scene, I want it to thrive for years after my main time with it is through, and to do that I need to code to protect against errors in code that hasn't even been written yet, potentially by people who have never even heard of the game yet. As a side bonus, lest that seem needlessly ambitious, it also solves some bugs that have very much been present in the last year in actual code, and which I have struggled repeatedly to eradicate.
This last month has been a lot of stepping back and trying to make the whole thing "safer" (harder to create bugs via mistake-because-did-not-know-secret-word-conjugate or whatever), as well as lowering the amount of data being sent. In essence, pulling it a bit more toward the single player experience.
I'm getting close, but new things pop up all the time still. When I'm done and it's tested, we should be out of MP beta for good.
Beta 3.603 Last of the Recent Known Regressions
This one fixes all of the recent regressions of which I am aware. I wish chain lightning was a bit prettier than it is at the moment, but I'll worry with that later on if I need to. This still hasn't really been stress tested with a bajillion tractor beams using the new beams pipeline, and I have not tested to see if the hacking beam works, but all of the engineering beams are working great, and chain lightning works great (if slightly less fancy than I want).
All of the other strange stuff of late with pause causing some oddities, or starting new maps being not centered properly on your planet, or saves not seeming to remember your last planet you were at, are all fixed. So is the Exit to Desktop just taking you back to the main menu!
There are certainly other bugs that I'm aware of, but all of the really annoying or serious ones seem to be dealt with now. Actually in the most recent beta you couldn't change lobby settings! So that's also fixed.
Zweihand added in a new starting fleet option, for Zenith Onslaught, which has a bunch of ships from that DLC.
There's still lots more to come in terms of bugfixes and QoL and such, but depending on the urgency of those things I may just focus on multiplayer for a bit. There's still a pair of remaining issues where I really need to rip and tear some code before things get better (notifications at the top bar, and the "description appenders" for certain units), so I had probably better start there. Those might accidentally break some things for single-player (again, sigh), but those are the last really main such things involved for multiplayer.
After that I need to load up multiplayer and start fixing whatever I find until it's fully functional again in the new reduced-scope-on-client paradigm. There are some optional things that I can do to help with RNG determinism past client-or-host-gates, but honestly when it comes to those the game self-repairs so fast in MP that I'm not overly worried about the problems those cause. I will get to that, but it's probably not urgent. We shall see.
Anyhow, MP is almost certainly not going to be ready for play this weekend, because this stuff with the ship to ship lines took VASTLY longer than I expected, but I will be supremely frustrated if we aren't in a MP-ready state for the weekend after. My hope is that maybe sometime next week, or the week after at the latest, that we can merge the beta branch back into stable. I won't be ready to declare MP out of beta until we have more testing on it, but all these code upheavals of the last month should mean that the surface area where things can go wrong in MP is way reduced, as well as bandwidth itself also being reduced.
Now that we're taking more time on DLC3, I can also try for the original vision for Expert mode, so that will be sometime soon, too. But not prior to merging back to the stable branch, unless I'm for some reason just waiting on tester feedback.
More to come soon.
Enjoy!
Beta 3.602 Linesanity
Please bear in mind that this one has some known issues. If you have a bunch of problems, please use the most_recent_stable_beta to get the prior build.
If you have big cases of tons of beam weapons or chain lightning or tractor beams or tachyon beams (if you have those turned visible) in a savegame, then seeing how that runs for you in the prior version versus the new version would also be really interesting to me on a variety of hardware. Some folks were struggling before with ultra-many-tractors being shown, but that should be a thing of the past (knock on wood).
Overall this new build has a couple of small QoL improvements, mainly better hover and tooltips over wormholes, and then a tiny bit of balance (one unit -- electric bombers), and then everything else is bugfixing work / refactoring.
The way that we draw "ship to ship lines" like repair lines, claim lines, chain lightning, beam weapons, and so forth has been re-coded from absolute scratch. It was having performance problems before, as well as three interlocking pools that were really complicated for me to manage, as well as various bugs like some beam weapons or hacking lines not showing up in the past.
There are NEW bugs now, but these ones are at least in a simpler (from the outside, to manage) framework that doesn't require pools. It did require quite a lot of computer science / mental acuity testing on my part, which I may or may not have passed with various levels of success. ;) There's a list of known issues, most of which are cosmetic or minor QoL right now. I don't think any of the new issues are actually about correctness or performance, but fingers crossed. This is hot off the presses, and I've only been able to test it on so many scenarios.
I'm releasing this one now, despite knowing there are still some things wrong with it, because (as noted) those are mostly cosmetic, and I could really use the testing help from folks who are interested. I'll be banging on this more tomorrow to get the rest of this out. This is one of those things that affects both single player and multiplayer, but one thing that was part of my plan this week was to get a lot of multiplayer stuff shored up and to hopefully get us closer to exiting the beta branch for both SP and MP sooner than later.
Also, our timeframe for releasing DLC3 (aka NA) has been pushed back to January. There's an actual much more accurate description of what is in the new DLC there now, though: https://store.steampowered.com/app/1466780/AI_War_2_The_Neinzul_Abyss/
This is going to be the final DLC, and it's a really big one. Some of the things that you may notice that have vacated from the old page description are roguelike stuff (that is out NOW, for free, in the base game instead!), and the Obscura (those got cut and replaced with a ton of other stuff).
My goal is to finish up all the kickstarter last items by sometime in November still, and so when the AI War 2 Complete Edition comes out in January, that should really be the complete thing. We will of course still have some bugfix releases, and the modding scene will hopefully continue to grow post-complete-collection, but I'll be working on some other projects. At the moment a smallish hex-based turn-based strategy game is sitting at the top of my interest pile, but I keep switching concepts, so we'll see what sticks. Part of my time will start going to that starting in November.
Anyway! In the short term, it's all AI War 2, all the time. Lots more to come soon.
Enjoy!