throwback wednesday 2: thursday

hi kids!

so after talking about things last night, i realized that there were some changes that i could make to the anim file easily that will make things slightly better for everyone while we work on the Way Better stuff. it cuts half a second off the load time! so that's pretty neat.

but also

yo what the FUCK was that post yesterday????

who reads this shit???

there's a distinct lack of quality here and someone's got to do something about it.

so without further ado, welcome to today's episode of 

how not to write a devblog post

now, out of respect for their privacy, i will not refer to the offending poster by name. they might not even be real. you never know.

exhibit A:

so right off the bat, i don't know who the fuck this person thinks they are, but what kind of asshole just refuses to use capital letters???

do you have an allergy????

are so you full of shit that it's interfering with your optical nerves and you are unable to see the shift key?

while we're here, this paragraph (if you can call it that) is entirely incoherent. "blowing up bullshit"? the fuck does that even mean?

if you have a backlog of bullshit, find a bathroom.

exhibit B:

this is actually entirely accurate. no problems here. post was shit.

exhibit C:

let's talk about that first section.

there's a severely unhealthy mixture of technical explanation mixed in with meaningless bullshit. how is this supposed to mean anything for anyone?

now, to be fair, the subject matter itself is about borderline meaningful bullshit. perhaps the post composition is actually an artistic reference to this, in which: kudos.

but i don't think so.

Quote

it is organized like actual ass garbage.

what the fuck does that even mean???

exhibit D:

bro what

sometimes people are just pointless assholes in the internet! get over yourself! in what world does this wind up in a devblog post?

exhibit E:

this paragraph false starts twice and fails to even explain anything when it finally settles on a subject. what's a symbol? what's a hash? how does this even pertain to the screenshot you just posted? these are basic questions!

you're not going to be making shit easier for anyone if you explain it like this.

now the next bit of this post is actually fine. it's probably the most coherent part of the whole thing. this bit feels like i'm not actively losing brain cells while reading it.

but, uh oh!

exhibit F:

grammar fail!!!!! can i get an XD in the comments???? SO embarrassing. frankly i should probably dm the author to inform them that they don't know what words are.

the rest of the post is honestly a lot cleaner. it's practically like it was written by an entirely different person!

it's just too bad that person didn't write the whole thing to begin with.

so today we're going to have a REAL devblog post. a good one. strong. professional. informative.

today i'm going to tell you about data structures

this is a tale of Ruby and Marshal.

obviously ruby is the programming language. marshal is the method of loading/saving ruby objects. 

the animations file is 14.1 11.3MB. it consists of 1400 PBAnimation objects (as of right now). there are a lot of attributes that belong to each object, but if we're talking about data size, the attributes account for ~750KB of the file size.

the true villains of this story are the cell arrays.

so, it's hard to precisely gauge exactly how "large" a specific value is in ruby. data types have specific sizes: in, say, c++ a number between 0-255 is a byte, 256-65535 is two bytes, and those sizes are specifically defined.
with ruby it's harder to know exactly what data type a number is, or how large an object is. so you just kind of have to guess.

which brings me to bullshit.

let's talk hawt features. hitting f6 lets us run arbitrary code. it's great for running little snippets of code like this. we can print out data to the console, rescue people out of emergencies, run single functions from the scripts...

and, in this case, dump bullshit.

this line of code has Ruby tell Marshal to make a file that just has a zero in it.

so if i want to see how large something is, i can just dump it and take a look!

that 0 is four bytes.

neat!

if you're a compy nerd, you can literally just use the f6 menu to see how big...anything is.

how big's a pokemon? just save it and find out. how big's a battle? you can just save that too! the possibilities are endless.

so i did some experimenting. the bulk of the animation file is nested arrays filled with nils and numbers, with some other objects strewn throughout. 

one such object is the animation timing:

so a raw timing is 257 bytes.

i mentioned earlier that you don't necessarily know how large an object really is on observation in ruby. the 0 that i dumped earlier isn't actually four bytes itself- the file that gets dumped includes some overhead that ruby uses to process the object. since the animation timing is a non-standard object, the information about what that object is has to be marshaled out with the object itself.

so let's say i marshal out five of them.

the know-it-alls in the peanut gallery are screaming out to their monitors that this is NOT 5*257. and that's true. ruby lied to us :(

so in reality a raw timing object is 100 bytes.

now, 100 bytes is nothing. it takes 100 bytes to display a line of text. it probably takes 100 bytes to eat a pizza. 

100 bytes matters here because we have 1400 animations with (a rough estimate of), say, 5 timings each.

that's 750KB.

(i'm actually researching this stuff as i write this post- i hadn't tried dumping multiple objects to find the size of a single one until just now.

but it turns out this accounts for almost all of the overhead i mentioned a little bit ago!

so that's kind of weird.)

now, the main reason i care about this is because of how it affects performance. (i'm sure the people with shit internet connections wouldn't mind some size reductions either.) i have to open and close this game a lot and it has to reload everything in the cache each time you do that. that 14MB animation file takes 3 seconds to load on my computer.

that's a really long time. ...granted, i'm horribly impatient- the last thing that i want to do while i'm testing something is sit and wait.

but i also have a beefy compy.

my processor, according to an arbitrary benchmark, gets a score of about 3200.

let's take a shit computer.

like this guy:

this is probably the minimum spec computer you'd need in order to run the game. coincidentally, it's also one of the most common desktop computers, so it's more likely that at some point you've seen or used one of these.

this compy, according to that same arbitrary benchmark, gets a score of about 700.

i'm ~4.5x faster than it is.

so if it takes me 3 seconds to load the animation file, it takes this computer almost fourteen to load it.

it'd be worse if you're still using the rxmp engine! 

(again, dear god, if you are a fan game and you're using our animations via the old method, my sense of justice compels me to fix that. god, imagine spending one minute each turn of a battle loading the animation file.)

coincidentally the math on this works out great: it takes that poor crap computer about 1 second to marshal 1MB of data. cutting the size of a timing in half would mean cutting load times by half a second. that's not bad! 

but the timings aren't the biggest contributor to the filesize.
that honor goes to the cel arrays.

every cel, every frame, has an array of data associated with it. that array has 27 elements, each containing a number between 0 and 255. i'm pretty sure that, if you count every cel-frame, you'll have a total number of arrays around 240,000.

so let's call up marshal.

marshal is screaming.

yikes!

so the current system, by necessity, has this ridiculously large file size associated with it.

what can we do about that?

1. get rid of the garbage values

someone (not me, ofc) mentioned last post that 7 of the values in each of the 240,000 arrays are completely unused. that knocks us down from 6.5 million array entries to 5 million. that shaves almost 25% of the filesize right off the bat. it's also practically free- and by practically free, i mean that this change will be out next patch and it cuts 2.8MB off the file. that's three seconds of life that the person with the shitty shitty computer gets back!

2. maybe don't use numbers for everything

every parameter in the array is defined as a number, even if it's just to indicate whether a value is true or false. sure, a number is only a few bytes, but a few bytes millions of times matters!

so let's try this out.

that's half the size! crazy.

...now obviously i can't just do this. most of the parameters need to be more than just a boolean. 

but the parameters that this can be done for get half their filesize shaved off! it's like shaving another 1.5 values from that array.

3. symbols are literally magic...

now, ordinarily, i would try and explain symbols to you right now.

but that's actually really difficult. integers are easy to understand. strings are concrete.

symbols are abstract. they're borderline magic. when you use a symbol, you are willing something into existence. want something to be a :THING? just call it a :THING! that :THING will always be the exact same :THING across anything that you do in ruby. let's say that later you need some :STUFF. just call it some :STUFF! again, anything you describe as :STUFF will always be the same :STUFF no matter where in the scripts you talk about :STUFF.

so let's slap some in that array.

for the size cost of a number, you can save a :THING!

and unlike strings, the length of the :THING doesn't matter.

*googles for good copypasta

why not?

same size.

so symbols are great for defining :THINGs.

hashes are where they really shine.

you can think of a hash like an array that can use anything as an index. like, say, a symbol!

take, for example, this bit of code:

celhash is my hash, :FOCUS/:PRIORITY is my index (the technical term is "key"), and then :BACK/:FRONT/:BEHIND/etc.... are my symbols. instead of using a number to define a state, i just.... define the state. (this is done a LOT in the PBS rewrites. it's basically the foundational idea behind them. the rewrite just has a ton of symbols that point at each other.)

now, turning everything into a hash isn't just what makes things smaller.
hashing an array will have the opposite effect:

(yikes!)

the advantage of a hash is that i get to leave a ton of things blank.

3.5 ...and hashes let you do sorcery

you can search for anything in a hash.

so what happens if you search for something that isn't there?

the hash will return a default value.

default values are just
*chef's kiss

most of those array elements in each cel-frame are unused. we're not using every parameter at the same time, so those unused parameters just have useless garbage data sitting in them.

hashes let me ignore those.

so let's say that i set a default value of 100 for my zoom. any time that my zoom value on a cel-frame is 100, i don't have to record it.

and that means that i don't have to record... well, most of the array values.

and that's the secret of how i pulled off doing this:

*chef's kiss

delicioso.

4. what does the future hold?

i've mentioned that every cel-frame has those same 27 parameters regardless of what the cel is actually doing, but one other major disadvantage of the current system is it has the same parameters regardless of what the cell is doing.

if a cell is sitting still and doing nothing for 60 frames, that nothing will be copied over to each one of the 60 frames.

also, the user and target both count as cels.

i had originally made an edit to the file that would use the previous cel as a reference under certain circumstances, but we ran into some problems with partner animations not working and it had to be scrapped. since we're making larger backend changes to the system, i'm inclined to revisit it- it cut a solid third off the animation file when i had tried it before.

if we managed to squish this file down to 5MB, i could sleep well at night knowing that shitty compy guy gets a whole 9 seconds of his life back each time he opens the game.

5. hi cass i'm one of the people who uses your animations. is this going to break everything for me?

wow look at me breaking structural rules to turn this into a Q&A! so meta.

so, yes.

right now we're only planning to release bullet point 1. that'll cause some crashes if you don't do anything else and just drop the animation file in.

there'll be some code that you can copy in to fix everything, though. replace class AnimFrame with this:

class AnimFrame
    X          = 0
    Y          = 1
    ZOOMX      = 2
    ANGLE      = 3
    MIRROR     = 4 
    BLENDTYPE  = 5
    PATTERN    = 6
    OPACITY    = 7
    ZOOMY      = 8
    COLORRED   = 9
    COLORGREEN = 10
    COLORBLUE  = 11
    COLORALPHA = 12
    TONERED    = 13
    TONEGREEN  = 14
    TONEBLUE   = 15
    TONEGRAY   = 16
    LOCKED     = 17
    PRIORITY   = 18
    FOCUS      = 19
end

and then delete these lines:

bam! your animation loading will suck slightly less ass.

once we keep going, though, i have no idea what will happen. i'm hoping it'll still be pretty easy to change things- the animation code is almost entirely unchanged between essentials versions, so you might be able to drag and drop our code into your game and it'll just work.

hard to know for sure.

see, now this was a post. that fucker from yesterday is so getting fired.