Accessibility

The idea of adding some accessibility features for players with sight handicap has been briefly discussed a few times but there was always one main blocker. There is not much point in that unless there is a TTS (text-to-speech) functionality to let blind players hear the dialogues and menus. Ame linked a library that could be used to send a text to a screen reader called NVDA to read the text out loud. But since it wasn’t a Ruby library, just a DLL, we had no way to actually use it.

Encoding the Endians!

However during my previous tweaks of mkxp-z I randomly stumbled upon FFI bindings. I had no clue mkxp-z even had this feature but I immediately knew this might become useful. FFI means Foreign Function Interface. It lets you call functions from shared libraries, even if they were written in a completely different programming language than what you’re working with. So I made a note that FFI exists in mkxp-z but didn’t investigate further at the time as I had other priorities like actually fixing the game or onboarding the new team members. Also I just roughly knew what FFI was but never actually used it myself so I didn’t have practical experience with it.

Few months later a blind player joined Ame’s Patreon so the topic of accessibility was discussed once more. But once again I thought there is no point without TTS but since I had that hunch that it might be possible with FFI, I went ahead and tried it. And it didn’t work. So I did more research and tried various tweaks but still nothing. Only suddenly, after an hour or so, the library started talking. Except not really. Instead of the text I was feeding it it only read the last letter. Ehm what ???

Of course I had no clue what was going on but thought that the mkxp-z community might have some idea so I asked them. We were brainstorming the issue for a while before the mkxp-z maintainer Splendide Imaginarius had the correct idea that it’s an encoding issue. Here I should make a short note of what an encoding is. Basically an encoding determines how letters and other text symbols are represented in memory and are often language-specific. Ruby comes from linux so it is generally using UTF-8. He had a theory that the library was instead expecting UTF-16. It didn’t work either but after some more trial and error I finally stumbled on a trick that actually fixed it. The library didn’t want just UTF-16 but specifically UTF-16LE where LE stands for Little Endian. See the issue was not just encoding but also endianness. Endianness affects the internal order of bytes in memory. This is an oversimplification but basically Windows is using the reverse order (Little Endian) than most other systems (Big Endian). And since Ruby originates from the Linux world, it is using Big Endian internally. So with explicit UTF-16LE encoding TTS using NVDA finally worked.

[image description for blind players: It's a Discord screenshot with AiedailEclipsed saying: "enumag once again destroying everything in his path. great job!"]

Of course all of this is windows-only since we didn’t find any equivalents for other platforms but Windows is the most important in this case anyway.

The roadmap

With that proof of concept, accessibility suddenly became a realistic topic so we started working on it. Accessibility ofc isn’t just about TTS. We also integrated an existing mod from Torre which adds some ambient sounds to help the blind players know the distance from the closest obstacle in each direction.

That was still just the first step though. I might have kickstarted it but from here on it was a team work where everyone contributed some parts. For TTS I pretty much just figured out the proof of concept. It was Orsan who did the heavy lifting to add the TTS calls throughout most of the codebase.

And there was a lot more than that. Some of the features we implemented because of accessibility made sense for all players - for instance puzzle skipping because even with TTS and other improvements there is no way blind players will be able to solve the puzzles.
 

Text-to-speech

Hi! Orsan here. I don’t have a whole lot to say– a lot of what’s happened so far has been fairly simple. Did you know there’s a lot of text in this game, displayed in a lot of different ways? There is! Messages, choices, battle, shop screen, town map, dex, field notes, gear apps, jukebox, tile puzzles… Most of what I’ve done for accessibility has been TTS calls on various pieces of this text– but there’s still a lot to be done. I also implemented a few other accessibility features for blind folks, such as Itemfinder now giving a message, but there’s still a lot to be done in that regard as well.

enu’s note: Orsan really spent a lot of time on adding TTS calls to all the core parts of the game like dialogue, choices, battle, bag, move learning and so on. Later on both Lucent Flash and Stardust also made significant contributions for adding TTS support to various menus. If I remember correctly I think Stardust handled the Time & Weather app + Jukebox while Lucent took the most complicated part which was the field notes.

Puzzle skips

This is a project I (hi, stardust here!) have been working on for quite a while, but became more relevant with the push towards making the game more accessible.

If you’re not aware, Reborn has a new game plus (NG+) feature that keeps track of whether you’ve done some specific puzzles/sidequests on previous saves so that you can skip them on future playthroughs. A couple months after Reborn e19 came out, I started work on an NG++ mod that added more puzzle skips, and this mod will be integrated into the base game in 19.5 Reborn. 

One of the big limitations of this feature is that it only applies to repeat playthroughs: first-timers are out of luck. And while I may consider Reborn’s puzzles a core part of the experience, I (and the rest of the dev team) am aware that not everyone shares that opinion. Especially blind players, for whom many of the puzzles are much more difficult or near-impossible without someone else helping them through.
 

[image description for blind players: Just showing off the "Do you want to skip this tile puzzle?" option you get in Agate Circus with blindstep password]

Recently, I went through (hopefully) all of Reborn’s various puzzles, looking at which could and should be skippable to help improve player experiences. The result: as of v19.5, there will be a new password to add to your games: “nopuzzles”, which activates nearly all the NG+ skips (old and new) without needing save data, in addition to a couple minor skips not covered by the NG+ feature. This feature is also activated with the blindstep accessibility password mentioned later.

[image description for blind players: Showing passwords menu with active nopuzzles password]

There’s also a few additional NG+ puzzle skips that you’ll be able to find scattered around the game. In total, between the existing, new, and password-specific ones, there’s over 40 such skips in the new version. 

blindstep password

Next we needed a way for the blind players to activate the improvements we added for them specifically such as the mod from Torre. This is what the blindstep password is for. It also enables several other things we implemented such as custom item finder by Orsan, separate fly menu because the map was a bit difficult for them and some other things. Then there is the F9 key to get the current map and coordinates read to them for navigation… And of course the password also activates stardust’s puzzle skips. 
 

[image description for blind players: Showing the custom fly menu for blindstep password]

TTS actually isn’t locked behind this password. It’s enabled by default but for most players it has no effect because they don’t have a screen reader such as NVDA running on their PC.

Contextual sounds

Enu again although this isn’t just my work. Next we needed to improve the sound effects. For example the bump sounds needed to be different based on what you bump into. So stardust implemented a feature where if you bump into an obstacle the game can detect what you bump into and use a different sound based on if it was an NPC, a strength boulder, an item or like 10 other different things we accounted for. This is an improvement for all players so it’s not restricted to the blindstep password.

Next we also needed a way for blind players to know when they’re on a diving spot so the game is now able to play a background sound while you’re in a place where you can dive or resurface. Again no need to restrict this for blindstep password.

Then there are sound effects when you jump over the jumping stones - along with a dust animation for sighted players. There is also new debris animation for rock climbs. Both of these improvements were only possible thanks to eevee because doing it all manually would be madness.

Also fishing has some sound effects now so that blind players can actually catch their Lanturns.

In all cases Amethyst is providing the actual sound effects!
 

Tolk

During testing one of the blind players also recommended a library to support more screen readers than just NVDA. There are actually two libraries - Tolk and Universal Speech. I went with Tolk for now but there doesn’t seem to be much difference between them. What it does is that it detects the screen reader you’re using and sends the line to it for TTS.

There are still a lot more improvements we can do. Certain menus still don’t have full TTS support, minigames such as game corner or mining need a skip, we should have a way to detect doors, itemfinder should detect non-hidden items, maybe a key to replay the last line, sound effect when activating or deactivating turbo mode… the list goes on. Let us know if you’d like to help with this effort. We’re slowly working on these tasks but we have other features to work on as well so this isn’t our only focus. Any such improvement means a lot to these players.