I quite enjoyed working with and developing tools during our different courses and game projects. In particular, I liked the iterative focus on user experience and usability. But despite this, the time sensitive nature of the projects didn't allow for much time to be spent on a specific tool, since the effort was needed elsewhere.
Therefore, I saw this project as an opportunity for me to spend a longer amount of time developing a tool where I could focus a significant amount of effort simply on how the tool feels to use. In order to make it slightly easier for myself, I decided to try and replicate an already existing tool from a different program, so I had a clear goal of what the end product should look like, as well as what could be improved. FMOD Studios' Sandbox was my tool of choice, as I had prior experience with FMOD Studio and its C++ API.
Table of Contents
My goal for this project was to recreate an existing tool and get it to production level quality when regarding the user experience.
Analysis of FMOD Sandbox
Screenshot taken from FMOD Sandbox
In order to realistically have enough time to focus on the user experience, some sacrifices needed to be made regarding the functionality. Because of this, I analysed the tool and decided to split up the features into three categories, Core, Non-Core and UX. Core features are features which I considered as required so the tool can achieve its goal as stated by the user manual:
"FMOD Studio's sandbox window lets you audition events in complex 3D environments. This means you can audition what your events would sound like playing in a game and use that audition to mix your project, even if your game is not yet ready to play FMOD Studio events."
Non-Core features, are therefore features which I didn't consider to be neccessary. The only feature which could potentially be considered a Core feature was Snapshots, but I felt that implementing them was going to take too much of my limited time. UX features are somewhat similiar to Non-Core features, but are in my opinion what makes the tool less of a hassle to work with.
- Scene overview
- List of addable events
- Event playback
- Parameter alteration
- Grid and icons to visually represent the world
- Drag & Drop events
- Command history (adding, moving)
- Saving and loading scene
- Keyboard shortcuts (ctrl-z, ctrl-y)
- Multiple active scenes
- Multiple active listeners
- Tags & Notes
- Folders for scenes
I started by deciding how I would create this tool. I decided to use my current game projects game engine, known as the LLL Engine. I chose this engine because since it had support for ImGui (which is the graphical interface I was going to use) as well as a window panel system and an audio engine which was already using the FMOD API. This would allow me to spend as little time as possible on setting everything up, and thus give me more time to spend on the tool itself.
During the first part of the project I made a list of the window panels that were going to be required and what functionality they each should have. These panels were:
- FMOD Events panel which shows the currently imported audio events that could be added to the scene
- Playback panel which allows the user to play, stop and pause all the events in the scene simultaneously
- Overview panel which lists all events in the scene, as well as the audio listener
- Event Inspector panel which allows the user to view and change properties of the currently selected event
As the last part of the setup period, I created empty versions of all the panels as ImGui windows and moved them around to match the layout of FMOD Sandbox. After the empty panels has been set up, I started to implement their functionality one by one.
Image of empty panel layout
Core Feature Implementation
A problem encountered early during this implementation period was that the current audio engine didn't return an identifier when an audio event had been created. This prevented me from selecting and editing the properties of an event instance post-creation, such as position, volume or custom parameter values.
In order to solve this issue I changed the code so whenever an event is created, it is added to an std::unordered_map which links each event instance to a unique identifier. The identifier is then returned as a variable. Each type of event stores their own instances in one of these maps and allows for quick access as long as the identifier is known. Now each event instance could be properly stored by the audio scene and edited using the Event Inspector panel.
At the end of this part of the project, I had a minimum viable product version of the audio sandbox, which included all of the Core Features previously mentioned during analysis. This was honestly all the features a user needed to test audio, however the tool would be absolutely dreadful to work with. Therefore, the next obvious step was to improve the user experience.
A video showing the minimum viable product version of the tool
At this time, the events were represented as events in a list of names under the Overview panel. A user could select an event instance from this list and change their properties under another panel. Despite having the properties of a three dimensional world, it was difficult for a user to ascertain where in an audio scene each event instance existed. In order to fix this problem, I started to implement a grid canvas to visually represent the world.
In order to create the grid canvas, I decided to re-use a class currently being used by my groups script editor. That class was called imgui_canvas and was a part of the Imgui Node Editor library. This class would allow for transformation of other Imgui elements into the local space of a canvas.
A piece of ImGui text moving around in the local space of a canvas
In addition to this, I used an ImGui::DrawList to draw custom primitives inside the canvas. I added a filled rectangle as well as intersecting lines, as it would create the image of a grid. After that, I made it so that each event instance and the active listener (where the sound was heard) had their own icon which was rendered out unto the grid.
A simple canvas grid with audio scene elements shown as circles
Now that the audio scene was visually represented, it was time to make the grid more interactable. In FMOD Sandbox, you can move around elements in the world by clicking and dragging around their icons. Compared to editing the positional values manually, this way of moving elements was much faster and easier to use (you could for example move things in more than one axis at once!). During the implementation of this element drag-to-move, I also overhauled the visuals of the grid to more closely match FMOD Sandbox.
First version of the icon drag-to-move
As you can probably notice from the video above, the icon drag-to-move, while functional, was clunky since it stopped dragging an icon around when the mouse cursor was no longer over it. This happened despite me still clicking and dragging with my mouse. In order to avoid frustrations that would occur from this, I decided to spend some time improving the drag-to-move. While improving the drag-to-move, I also added a visual representation of the attenuation range of audio event instances (which is how far away a sound can be and still be heard). The movement turned out quite nicely and the result can be viewed in the video below.
Second and improved version of icon drag-to-move
Additional UX Features
With visual representation of the audio scene completed, it was time to implement some additional experience to improve the user experience.
At this point, in order to add an event to the audio scene, the user was required to press a small "+" button next to its name in the FMOD Events panel. This simple approach wasn't very intuative because it required a lot of mouse movement in order to interact with the recently added event. In order to improve it for users, and to more closely emulate the FMOD Sandbox tool, I decided to implement drag-and-drop functionality.
ImGui already had built-in support for drag-and-drop and it required only an hour or two of work in order to fully implement it. A user would be able to drag a selectable object from the FMOD Events panel and drop in inside the grid in order to add it to the audio scene. During this drag motion, the cursor changes depending on whether the user is hovering over a legal (grid) or illegal space. The user can also see the name of the event they are going to drop, so it is easier to know what they just grabbed. Lastly, the event is placed in the same position as the mouse after drop, in order to allow faster correct placement in the scene.
Showcase of adding an audio event through drag-and-drop
After that, I found it quite frustrating to recreate the same audio scene over and over in order to test the same basic things. What this tool should do was to allow me to save or load my work. Because of this, my next implemented UX feature was to save and load files containing information about an audio scene. This part will be more code-heavy than the rest, since most of the work was turning the scene into data, with not much UX involvement.
In order to do this I used the json file format alongside the nlohmann library, since it already had support in the engine I was using. When I implemented the saving part of this feature used a pre-existing function to open the windows file menu and get back a location to output a file. Afterwards I serialized all the data from the entire audio scene into a json object. Then I dumped all of the data into a single string, which was later put into a file using a std::ostream. When this was finished, I implemented all the code in reverse in order to be able to load the file. A sample of this code can be found below:
Function to serialize audio scene into a string object
After the functionality of this feature had been implemented, I also added a notification using imgui notify that informed the user that the scene had been successfully saved or loaded. I felt that it was important for the user to gain some sort of feedback from their actions, since any confusion on if the scene was saved or not could cost the user valuble time and effort.
Showcase of audio scene saving and loading
For the last UX feature implemented, it would be a so called "command history". A vast amount of tools have this feature in some form or another, as it is important in preventing mistakes. If you write a large amount of text, but accidentaly delete it through the stroke of a button, it would be awful if there was no way to get it back, right?
A command history is simply a list where some of the users actions are registered as commands (moving, adding or deleting anything) and added to that list. These actions could then undone or redone with the press of a button by the user, upon which the next action could be undone or redone. The command history is only as complex as the commands, and the ones that I used made it quite simple to implement. In order to more easily use the command history, as well as some other functions such as saving and loading, I additionally spent some time implementing keyboard shortcuts.
Showcase of command history with keyboard commands
A video showing the final version of the audio sandbox tool
Reflections & Improvements
Would I say that I managed to achieve my goal of having a production level tool? Kind-of, but I think that I didn't get as far as I could have with the five weeks we had. I learned a lot about the different ways a professional tool makes itself easier to use. Still, there are some features which could have been implemented which, while relatively simple, I feel would have improved user experience further.
Whenever an event has been added to my audio sandbox tool, there is no visual feedback to the user what kind of state the event is in. Is it currently playing? Is it paused? Has it been stopped altogether? The only way a user can know these things if they hear it, which can be problematic if a specific audio event contains large pauses. FMOD Sandbox solves this problem through a couple different means and I would have tried to emulate them if I had more time.
- First method is by highlighting which state the event is in by changing the color of the playback buttons.
- Second method by changing the the event icon itself based on state, where it both changes the
transparency of the icon and the icon's texture.
- Third method is by having a small info-bubble next to the playback buttons, which outright tells the user which state the event is in, as well as some other information
How FMOD visually represents playback state for events
How FMOD visually represents playback state for listeners
In my tool, audio events aren't required to be three-dimensional, since the actual 3D effect is created by each event using a spatializer component in FMOD. Without this component, the audio will simply be percieved as if it is playing right in front of you, like listening to music with headphones. In order to prevent confusion, I feel that some distinction should be made between events with and without any three-dimensional attributes.
Perhaps the icon itself should be made a different color? Maybe it shouldn't show up on the grid as an icon at all? Maybe something as simple as an additional piece of text telling the user if the event is 3D?
Any of these solutions could be used, but ultimately it would be up to the user to decide what works well and what doesn't. Thank you for reading through my specialization text! if you want to know more about it, my contact information can be found in the page footer below.