Asciibrain - Memory Issues
How a simple feature, developed in one afternoon, turned into a memory corruption bug hunt that lasted almost a month…
I’ve been so proud of how simple, and easy to use, my custom memory allocator has worked. Then suddenly the memory issues started happening and I had to deep dive into all kind of memory related debugging. And what started as one bug, soon turned out to be three different ones.
A quick note regarding C++ and memory management bugs:
- Memory bugs ARE hard to track and debug, but this is true of other type of bugs as well! Tricky bugs are part of our jobs as programmers!
- “No pain, no gain” they say. If you haven’t experienced it, then it’s hard to build strategies to avoid it in the future.
- There are strategies and tools that can help!
So I’m going to go through here what happened in my case…
Memory bug 1 - Corrupted Stack
Yikes, this one felt scary! BUT it turns out that Visual Studio has some truly remarkable tools!
First: Visual Studio actually detects where the suspected stack corruption happened! Around the variable wRngTable. This is of tremenous help (and quite frankly feels almost magical).
So with a bit of logical thinking - “What is a possible cause for a stack corruption?” I quickly realized that it might be me writing outside the bounds of a locally created array. And guess what: wRngTable is a locally created array.
And here’s the answer! For some reason I have declared MAX_ABILITIES twice in different places, and used the wrong one. My array is of size 5 but the data can be up to 10 entries long.
With some logical thinking I solved this one in less than 15 minutes, on my laptop while my kids were watching TV next to me.
Memory bug 2 - Writing outside text buffer
Second bug was trickier. Game crashed with a sudden null pointer exception in random situations, that did not seem to have any obvious logic to it… at first.
Here, an old trick helped me tremendously! Fixed Random Seed.
When you have a game where everything is randomly generated - MAKE SURE YOU CAN START IT WITH A FIXED SEED!!!! I can’t even begin to tell you how often this has helped me track down very rare and crazy bugs.
By being able to start the exact game over and over again, I could narrow down exactly what was going on. At a certain point, something changed the pointer address to the text buffer of my fake console, so next time it pointed to invalid memory and crashed. BUT WHO CHANGED THAT VARIABLE???
Again: Visual Studio to the rescue! You can tell the debugger to break when a value changes! Incredible!!
Now I could easily see that one text buffer did an overflow and wrote garbage into the other (it’s a double buffer, so I have two), when I did a FillRect() where the rect extended outside of the buffer width/height. All writes into the buffer had bound-checks except FillRect which I had forgot.
Memory Bug 3 - Linked Lists and Memory Arenas
Last one took most time to solve. In the end it was a badly designed/named api that led to it. Even though I wrote the api myself, it made me make the wrong assumptions and use it wrong. But I learned a lot in the process.
The bug was this: I have a linked list that is allocated in chunks, using my own allocator. I use it to store all entities in the game (that means: players, enemies, doors, chests… everything). If it runs out of space, I’ll allocate a second chunk and keep filling it up. The linked list is meant to stay around for the entire game (moving back one level and enemies should still be there), but every time I extended the chunk I used the memory arena that would be cleared after each level, so trying to iterate over it after a level changed would get you into de-allocated memory.
To try and diagnose this I used the Memory Viewer in Visual Studio. It is still hard to use because it’s hard to navigate, and hard to know what’s what. My memory allocator does pre-allocate a chunk and lay everything out in sequence, so you won’t have to jump all over, but still very hard to know where things start and end.
That’s where my brother had a brilliant suggestion. Let’s tag each allocation!
So every time we allocate, we allocate one extra 32 bit integer at the beginning. We then write 4 chars to it (4 chars = 4 bytes = 1 32bit integer) in the format of 'abcd'. This will display in the memory viewer as 'dcba' (because of endianness). So my allocator now has the ability to allocate with tags!
And in the memory viewer, we now see where the allocations for spat and eView starts. We have not yet filled it with data, so they’re all zeroed out.
(Note that the allocations swizzle the 4 chars so I don’t have to write them backwards)
Conclusions
These were some tricky and time consuming bugs. It does not mean “stay away from manual memory management”. As a programmer, solving tricky bugs is one of the tasks you must eventually face. No matter how careful you are or what language you use.
Some things will always help you in these situations.
Tools
Tools, such as debuggers, are absolutely invaluable. Without them you are in the dark.
But there are other tools that debuggers. You can write your own. Like being able to start the game from a fixed seed. To quickly be able to run your algorithms through a pre-defined data set. In some cases, unit tests will help a lot.
Try to structure your code with as little dependencies to the rest of your environment is the main thing to help you test it in isolation!
Experience
Experience will help you with reasoning around what went wrong.
It will also help you write the code in a way that is easy to debug and test.
Don’t be afraid
Try new things! That’s how you get experience. And experience is extremely valuable!
That’s all for this time. See you soon with a more game-y status update.