Design Patterns for Game Menus

About what I assume to be a year and a half ago, I made and posted this picture on r/justgamedevthings on Reddit

In the comments however, I wrote a fantastic explanation to the best approach I found when it comes to programming a menu system. From the comments on my Reddit post, struggling with game menus and UIs seems like a very common headache. Some games avoid it by making the menu an actual in-game mechanic, like the menu from the first-person puzzle game Antichamber. and sometimes just parts of it, like a character editor.

Below is my unedited explanation from a year and a half ago (unedited since then).

The Best Design Pattern for Menu Systems I Found So far

A note: Somebody else surely has come up with this before me, I doubt I’ve invented anything new but I came to this solution on my own.

The approach I used before (I use GameMaker Studio 2 so I have to basically code it from scratch) was using a state machine, and coding the buttons and the response to those buttons being selected and clicked on.

Something like a switch statement with different states, and states having things like:

if (buttonPressed(“Play Game”)) startThatLevel();

It worked OK but it was hard to manage. It drew the button and also checked if it was being pressed.

…

Second decent menu system I tried to make, I did it a bit better but it’s still complex enough to the point where I can’t import in my projects, it’s not flexible. (and I had a very fun time trying to make the mouse and the keyboard work to select different menu buttons).

…

The best I got so far….

Now this time, this time I think I’ve got it. Menus aren’t states, they’ll probably be made of two states however: normal state, and “waiting for input” (this will allow for yes/no confirmation boxes). Other than that they’ll basically be data structures only.

To create a menu, I give it a list of button ids. Button ids are unique, basically an enumerator, and all menus can share them. If I want X menu to have 10 times the same button it can, but they still all refer to the same button. (technically the buttons could be singleton objects).

When a button that brings you to a different menu is activated, the current menu becomes inactive, a new menu instance with the correct buttons gets created. That new menu knows the id of the menu that created it, so if the player presses the back button, that menu instance gets removed from memory, and the active menu instance becomes the one from before.

If the player exits the whole menu, I can destroy a menu’s parent’s parent’s parent’s parent’s…. and then go back down until they’re all destroyed.

Because each menu instance is different, the current selected button will always be valid; if you were at the third button on the previous menu, you won’t initially be at the third button on this new menu.

Annnnnnnnnd this is actually pretty similar to an entity component system. Menus don’t actually exists, they’re just generic objects with lists of button ids. The buttons are similar to components.

I might make an article out of this to be honest, because holy cow is it hard to find a good explanation about coding flexible, modular game UIs and game menus.

Edit: the solution for the mouse + keyboard/gamepad compatibility is vey simple. You have a system that checks only for the keyboard, and one that checks only for the mouse. Have some effect when the mouse hovers over a button, and when clicked once, have that button be the one “buttonSelectedByMouse”. Then if it gets clicked again, do what you gotta do.

For keyboard, same thing. pressing Enter and left clicking shouldn’t be considered the same thing when it comes to menus, have two different things handling both types of inputs.

The structure of things and data, menu objects and button IDs

Here’s a comment from myself clarifying how I go about structuring things:

There are no menu ids, there are instances of the menu object.

There are button ids, and that’s where the functionality for each button is described

When creating an instance of a menu, I pass in the button ids that I want said menu to be made of. I set that new menu instance’s ‘parent/owner’ to be the menu from which that button was pressed.

When the player wants to press back, that menu gets deleted, and the active menu becomes the previous one that was stored as a parent/owner of this one.

I hope you found this useful. Here is the link to the Reddit thread, here’s my itch.io page (for when I someday finally release something decent…) and here is my Youtube channel.