Dev-ops for Reborn, Rejuvenation and Desolation

In my first post I mentioned that I’m the dev-ops guy. But what does that even mean? Most of the playerbase likely won’t be familiar with it or only have a vague idea. So in this post I’d like to give you some insight into what I’m doing for the team.

Part of it is, of course, Eevee which I introduced last time. Another part of my work is about automating boring repetitive stuff like releases which need to be done very carefully and are very easy to mess up. Aside from actual game scripts, maps and graphics there are a lot of other parts that need to work correctly. Typically, compilation for all four platforms including all the libraries and binaries the game is using internally. The requirements have increased dramatically because of some of our new features and it would be next to impossible to maintain all of that manually across three games and four platforms.

Disclaimer: Some of this will be very technical but if you’re curious what developing these games involves behind the scenes it can be interesting nonetheless. It could also be good for other game authors and modders to see how automation can help them.

Patches

After the initial version of eevee, the next thing on my hit list was the release process. I wanted to automate the process of detecting all files that were changed, zip it and upload the zip somewhere for the players to download. The first part was this Git command:

git diff --name-only --diff-filter=ACMRTUX <commit>..HEAD

Git knows the entire history of the codebase so you can get some interesting data from it. In this case the command above gives you a list of all the files that were changed since the given commit. Normally I’d feed the result to another command that would zip all those files– however, with eevee involved it’s not as easy because Git would give you the yaml / ruby files instead of the rxdata files that the players need. So, I wrote the logic to generate a patch into eevee directly. It uses this command internally but then detects which files need to be replaced by rxdata files in the patch.

GitHub Actions

Even when you have the patch zip you still need to upload it somewhere. And while we’re at it why would you even make the zip yourself even when it’s faster using the command? Software developers automate all of it these days. GitHub Actions can do exactly that so I wrote a simple workflow that would trigger any time Ame tagged a new release in Git, use eevee to generate a patch and immediately upload it to her server where the players can get it. With that making a new patch was just a few clicks, Ame could focus more on e19 development and easily release several patches per hour if she felt like it. I think we first used this for Reborn’s e19 public release. Since then the workflow has evolved a lot. Desolation e6 and Rejuvenation v13.5 were also using some newer versions of this and I made even more changes for the upcoming version of Reborn. Of course, now it’s mostly me and the new team actually triggering the workflow.
 

mkxp-z

With eevee and my release workflow being adopted by Rejuv and Deso I suddenly had Git access to not just Reborn but all three games. So… time for more fun, right?

The workflow I mentioned above could only handle the small incremental patches; the main releases of Reborn e19 and Desolation e6 were still handled manually. Changing that would be a much bigger fish to catch– more like a whale, in fact. The main complication is that the games are not Windows-only but also have macOS and Linux releases. It wasn’t enough to just remove the dev-specific files and zip the rest because the repository didn’t contain the runtimes for the other platforms.

The engine the games are using for runtime is called mkxp-z. It can be built for all three platforms and compiling it is not an easy process. Particularly because to compile it for a particular platform you need to compile it on that platform. And I don’t have a mac. There are pre-compiled versions of it available but to complicate things further the games are using a modified version of mkxp-z so that they could have a custom game icon and increased frame rate cap. If only there was a way to compile this automatically with our modifications, right? Well, guess what, there is. The maintainers of mkxp-z have their own GitHub Actions workflows to compile that thing for all platforms, so I didn’t need my own mac. I just forked it, applied the tweaks we needed and soon I was able to get a compiled version for Rejuv. It took several tries and each compilation takes over an hour because mkxp-z is a beast, but it worked. Afterwards I could set up another workflow to combine the compiled mkxp-z with the game files. A couple test versions later and we could fully automate Rejuvenation’s main release 13.5.0 for all platforms; no need for Jan, Cass or anyone else to painstakingly prepare it on their own.

I won’t share the workflows here as I’m still working on some details for the community release. But there is no harm in sharing these so in case you want to set up some automation like this for your own mods or games and you feel getting your hand on our workflows would help, hit me on Discord.

Engine tweaks

This ended up opening several more possibilities in the future. For instance, I was able to adjust the default controls of the game which are baked into mkxp-z’s source code itself. Next I was able to make it so that the engine contained some extra libraries which we needed for some other new features– more on those in the next post!

Then there was the very controversial topic of input repeat– how often an action is repeated while holding down a key. This has a big impact on how fast you can scroll through Bag and debug menus. The mkxp-z defaults (which were used in Rejuvenation 13.5) are too slow, while Reborn e19 had it adjusted for debug convenience which was slightly too fast. In our mkxp-z fork I was able to change these settings to be configurable in the mkxp.json file. So, while the default input repeat is ultimately set somewhere in between the normal defaults and Reborn e19 speed, you can now adjust it to your liking.

And finally it means we can fix some bugs in the engine. It’s written in C++ which I’m not familiar with so my options are limited but I can tweak a line or two when we find a problem.
 

Segmentation Fault hunt

First I should clarify what a segmentation fault even is. I’m no expert on this topic, but from my understanding a segfault happens when a program tries to access an area of memory it doesn't have access to, causing the operating system to kill the program. In general, there is next to no way to recover from these errors because the system just kills the program with no chance to react. In general, these happen only when incorrectly using some low-level things such as pointers in C. Higher level programming languages such as Ruby, Javascript, PHP, Go and so on generally won’t let you mess up like that, and if it does happen anyway it usually means there is a bug in the language itself.

Well guess what? We had a segmentation fault in Reborn during 19.5 alpha. You know how the game sometimes shows a popup with an error message? Those are our errors in the Ruby code and generally not a big issue. Segfaults instead just end the game: no popup, no message in the log, nothing. This signifies a bug in the game engine itself - mkxp-z. To make things worse, we didn’t have a reliable way to reproduce the error, it just sometimes happened during or after a battle with no consistency. Sometimes you could play for a week without getting a crash, other times you could get it three times in a day. (Like I did! -editor Orsan)

There are, of course, some ways to debug these cases. I consulted the mkxp-z devs about this issue and was able to make a mkxp-z build with debug symbols. Debug symbols are basically a mapping between the compiled program and the original source code, so if a segfault happens the GNU Debugger (gdb) can be used to process the core dump and find the lines in the original code where the problem occurred. It took a few more hours but I did manage to get a gdb backtrace of the issue.
 

With that additional info we eventually discovered that someone else from the mkxp-z community already had a fix for the problem (which had to do with certain move animations accessing invalid files). Without the backtrace, there was no way to know what to look for, so we wouldn’t have found the connection between the issue and the available patch. The game has been stable ever since, so it was well worth it. And, because I already had all the infrastructure around our own mkxp-z fork already developed, I was able to simply merge the patch into the fork and use the fixed mkxp-z in the 19.5 beta release for Ame’s patrons.

That’s all for today! Sorry if my posts are more technically oriented. Next time I’ll finally get into some improvements for the players! If you have some questions feel free to ask in the comments or Discord DMs.