This series of game audio relevant links may serve as sort of an origin story for me, as I am floored on a monthly basis by the amount of related output that goes by on a monthly basis. What makes this the genesis of the 'lab you might ask? Well as few as 5 short years ago, you'd be remiss to find much out there regarding game audio on the net, at least information about the technical side was significantly lacking. I often found myself buried in SDK docs for HL2 or reading through OpenAL programmer API's in order to understand some of the fundamentals of how audio get's integrated into games. To say that we are in an information age is understating the obvious, but there really is a ton of stuff out there flowing down the stream of interactive audio if you've got your ears on.
Hopefully there's something of interest in this latest batch of madness, keep em' peeled!
Now that 2010 is closed out, I just wanted to take a second and round up a batch of links to various projects that I was involved with and get them all lumped into a 'splosion of game audio. Every single one of them was made better by the involvement of others, and for that I am thankful.
It is truly a beautiful thing to work along side such creative individuals and to be able to share the common goal of engaging people in the art of sound. I hope that the new year comes with new prospect, and that you will all continue to shape and inform the future of interactive sound to the fullest.
This bit hit too close to home when I read it the first time. I though "In a few short paragraphs someone at a production level has boiled down the plight of game audio across the industry." I say that because it has been my experience that most of the time what he describes is not the exception but the rule in today's development cycle. I can say no more, but feel inclined to excerpt "What Went Wrong #5" from the recent Game Developer postmortem for Spider-Man Shattered Dimensions.
"Along the chain of production, many design choices occur, and just as many changes are made, if not more. When delays happen at any stage in the process, the last team in the line of production is the one trapped against the deadline wall. This was the audio team on Shattered Dimensions.
As the entire development team iterated on every aspect of the game simultaneously throughout development, nothing was ever really final until the very end. Hence it was impossible for the audio team to start working on anything with final quality in mind, and it was very hard to determine what portions of the game could be worked on at what time, with minimum risk. Eventually, when everything in the game reached final quality simultaneously, the amount of accumulated audio work was just too much for the team to handle."
You can read a preview of the article page which includes the full exceprt here:
It's great to see honest and clear insight into the production process as it relates to the sound team.As someone who has personally seen the ripple effect of other disciplines missed deadlines with little understanding of how that effects other stakeholders standing down stream, it's refreshing (if not a little disheartening) to see it in black and white. It's clear that there are many ways to approach a solution to this problem, not all involve jockeying the schedule, but many involve increasing the the awareness of where audio falls in the pipeline and how each discipline affects the eventual workflow.
I'll be looking forward to seeing some of these practices pave the way for a tighter integration of game audio scheduling in the future!
Hot on the heels of my AES 2010 Wrap Up I'm corraling the last months Game Audio related news into a impossibly verbose and undigestible chunk of internet link goodness.
"BioShock 1's audio system was... umm... "sub-optimal". This time around our sound team demanded a new audio pipeline based on AudioKinetic's WWise technology that supported 5.1 with adjustable dynamic range and a fully dynamic mixing system. Not only did our engineers rise to that task, but they subsequently took it up a notch and implemented both a custom sound propagation system (so voices properly echo down corridors and around buildings) and a dynamic wind audio system that reinforces the dynamic weather in the world."
It is a special privilege to be able to connect to other like minded individuals out there in the world today. Whether trolling the internets, tweeting the hottness, or conversing in person; it's a beautiful thing to share your geek with peeps that are into it. It was with this appreciation that I descended upon San Francisco this past weekend to take part in and contribute to the 2011 Audio Engineering Society Game Audio Track.
Having been to a handful of Game Developer Conferences in the past 5 years, I had never made the pilgrimage to AES. Either due to circumstances, or what seemed like a lack of technically inspired game audio programming, the annual GDC has always matched or exceeded what was offered at the emerging AES Game Audio Track. Couple that with the high volume of game geeks present at GDC and the scales have always tipped in favor of connecting with the development tribe of professionals I'm so used to working with.
After having spent the weekend soaking in the current state of affairs I feel like I'm quickly coming to terms with exactly what Game Audio is and could be to the AES. As an example of where my line of thinking is leading, the mission of Siggraph is to "promote the generation and dissemination of information on computer graphics and interactive techniques." What comes of that statement is a successful cross pollination of very technical white papers focused on furthering the state of the art and the adoption and absorption of such idea's by today's computer graphics creative leaders.
I think there is a growing correlation between what is going on with crossover at Siggraph in the potentials for crossover at AES with regard to game audio. In addition to several presentations targeted at the general audio professionals in attendance, there was a slice of technical and future focused talks that helped cement game audio as an industry that belongs as part of the AES whose acknowledged role is as "an international organization that unites audio engineers, creative artists, scientists and students worldwide by promoting advances in audio and disseminating new knowledge and research."
Here's a brief rundown of the things I found most exciting.
Friday saw Kristoffer Larson waxing philosophical on the use of high level programming language Lua in games and went on to illustrate it's accessibility and use in an open source 2D game engine called "LÖVE" which utilizes OpenAL to play back audio.
Later that day Michael Kelly followed up with an overview of XML for use in games.
Highlight Up My Life
Richard Dekkard and Tim Gedemer dug into 5.1 Orchestral recording and ended up unloading a pile of technical information about how sound is handled in games. Their stories from the trenches and wisdom regarding process and motivation made for a great introduction for audio professionals coming from outside the game industry.
David Mollerstedts presentation on 'Mixing the DICE Way' was a well placed and laid back presentation disguised as a treasure trove of technical implementation that illustrated (warts and all) the HDR approach to Battlefield Muti-Player and the need for additional creative control moving forward with their single player campaign. Needless to say the topic of interactive and dynamic mixing is a hot topic these days, and the solution that DICE has implemented seems to be serving their aesthetic design quite well. If that wasn't enough David is also one of the great minds behind the Teenage Engineering OP-1, the only piece of hardware to have me salivating in recent years. (or at least since the Tenori-On)
Adam Levenson gave a lunchtime keynote on the "Trappings of Hollywood" during which he crystallized my favorite theme of the conference when he suggested that the future of interactive audio involves synthesis and procedural techniques to a great degree. This was echoed by a whitepaper written and presented on by Simon Hendry entitled "Physical Modeling and Synthesis of Motor Noise for Replication of a Sound Effects Library" in which Max/MSP was used to simulate the sound of DC motors for interactive applications. Moving every step a little bit closer to a believable model of reality.
Closing things out on Sunday afternoon, the 'Physics Psychosis' panel I had the pleasure of kicking off was one that (obviously) had a special place in my heart. After having presented several time on the technical side of physics in games, I chose to pull back and instead focus on the artistic considerations of approaching a dynamic physics system, while laying down some fundamentals and illustrating with examples from The Force Unleashed. I joined the audience in what was set to be a ramp up in technical detail as Stephen Hodde from Volition proceeded to blow minds with footage and information from the forthcoming Red faction installment and iteration of their GeoMod technology. Of special interest was a super-sexy debug fly-through that enabled visualization of the different sounds emanating from a structure during it's destruction.A beautiful sight to behold. Next up in the continuation of rapid technical acceleration was Jay Weinland from Bungie showing off the Bonobo Toolset that they've been building behind closed doors for the past 10 years. What looked like a powerhouse of properties and potential for the customization of sound playback in games resulted in very articulate and accurate playback of the 100's of surface types and potential object interactions. The cleanup crew was brought in shortly after to clean up grey matter from the room dividers after the spontaneous explosion of several attendee's.
Wrap it Up
Which brings me to an interesting intersection that occured en-route to the convention while reading the latest copy of Computer Graphics World. In an article by Noriko Kurachi presenting an overview of technologies from this years Siggraph she outlines one of the idea's presented that was based on "sound rendering" and Harmonic Fluids. Essentially the simulation of fluid dynamics tied inherently to the synthesis and reproduction of sound, based on the simulation in realtime. This convergence of graphics and sound at an unabashedly graphics focused conference makes me long for the same symbiosis between the acedemic and practical in game audio.
My head is still spinning from the overspilling information that I was witness to during the conference, but my key take-away is that feeling of belonging to a group of people who is as passionate about audio as I am. It's during ties like these that I feel like it's important to note that in alot of ways we are all on our own solving the same problems; whether it's Physics, Mixing, Synthesis, or Surround. Industry events where people can come together and share their experiences and dream a brighter future are the petri dishes where the state of the art get's challenged and eventually pushed forward.
10AES-G01
Code Monkey Part 1: What Game Audio Content Providers Need to Know About C++ Programming 10AES-G04
Code Monkey Part 2: LUA is not a Hawaiian Picnic - The Basics of Scripting for Dynamic Audio Implementation 10AES-G08
Code Monkey Part3: XML
The Tweak:
10AES-G03
The Wide Wonderful World of 5.1 Orchestral Recordings 10AES-G12
Mixing the DICE Way - Battlefield, HDR Audio, and Instantiated Mixing 10AES-G14
Physics Psychosis
The Freak:
10AES-G10
Audio Cage Match!
The Biz:
10AES-G13
Takin' Care of Business
The Buzz:
10AES-SE07
Lunchtime Keynote: Adam Levenson
There's other good ones in there, for a summary of these see above!
Procedural Sound is a topic that I feel has a great potential to solve some of the challenges facing game audio. Whether it's the dynamic composition of music in real-time, geometric sound propagation modeling , or harnessing already available simulations to create approximations of sound using synthesis, there exists a tremendous amount of solutions to the problems of how to make our virtual worlds more directly reactive and immersive. It's with optimism that I set upon trying to raise awareness of some of these techniques and technologies available outside the confines of console specific runtimes. It's my hope that by looking outside of our industry we can catch a glimpse of the future for game audio, and set our sights on some new ways to provide engaging, dynamically reactive, and emotionally resonating interactive sound.
In a companion article entitled "Procedural Sound Now!" at DesigningSound.org as part of the Audio Implementation Greats series, I've made a case for a return to the formative years of game audio with an eye towards new developments in synthesis and procedural. The article wraps with a QA with three audio professionals working with procedural sound in some capacity.
This post is an attempt to round up the cadre of links I was able to unearth in relation to the article. You'll get a bit of sound modeling, some procedural, a bit of music, white papers, and research materials. Keep an eye on this space over time, as I hope to keep adding to this list as a resource for people who are interested in new developments and articles. Please send along any further suggestions or post below in the comments section.
It's taken the help of many dedicated sound professionals to bring to light and assemble the different pieces of the puzzle. My deepest thanks and gratitude go to: Anton Woldhek, Karen Collins, Andy Farnell, David Thall, Mads Lykke, Francois Thibault, and all of the inspirational audio futurists pushing the envelope.
I hope that get's you started on the wide world of Procedural Sound and Synthesis.
If you have any additional suggestions for links to interesting materials please drop a line or comment.
Inspired by Tim Prebble and his Detritus series over at the incredible Music of Sound Blog I've decided to round up a host of links floating around online that have specific relevance to Game Audio. Maybe not the first time i've attempted such a round up, but hopefully I can get onboard with keeping these digestible from now on, this one is rather lengthy...we'll see how it goes.
The wealth of knowledge on the subject of game audio is somewhat staggering these days. It wasn't so long ago that I can't remember reading between the lines in articles trying to figure out how they did this and that in whatever game or interactive project was being discussed.
Hopefully the details of different techniques being exposed will lead to a better understanding of what goes on behind the scenes, and continue us down the path of standardization and best practices as an industry.
After the recent renaming of the Austin Game Developers Conference to Game Developers Conference Online. With the change in name has also come news that the Audio Track has been removed from the proceedings, and GDC San Francisco will carry the torch.
"That is correct. We have combined the program with the Audio Pass at GDC in San Francisco. Come join us in San Francisco in March 2011!" - Think Services
With otherconferences focusing on Game Audio it's not the worst thing to lose it, but I'm keenly interested in growing the next generation of technically minded Sound Designers, and I felt that the Austin GDC Audio Track had become a reliable destination for deeper thinking on the subject - as opposed to the audio boot camp at GDC SF, who's open door policy is a magnificent gateway into the throng of like-minded noise makers.
Thankfully there are still a fewgoodreasons to head down to Austin.
Y'all come back now, y'hear!
THE STOCK MARKET
Lately, I've been taking stock. Not the usual “What have I done with my life?” or “Where is everything headed?” (although those questions perpetually tumble around my brain stem on a regular basis); I somehow found myself obsessed with the minute details of movement sound and system design. If you're working in games today, chances are good that you've recorded, implemented, or designed systems for the playback of character footsteps and Foley at some point during the course of your career. It's even more likely that you've played a game where, at some point during your experience, footstep sound wrestled your focus away from the task at hand and demanded your listening attention.
Yet, let it be said, all footsteps are not created equal – which seems obvious given that no two games are exactly the same, neither should their footsteps or the way in which they are implemented be (necessarily) the same. At the end of the day, as content creators, we should be slaves to the games we are helping to make and not showboating unnecessarily in our own art by accentuating or spending time of things that have little consequence outside of our own satisfaction; however, for a sound type that may be heard for countless hours across every level in a game, surely they deserve more than a passing thought. (or maybe I'm trying to justify my current obsession!)
Now that that's been said I want to take a moment to recognize the ubiquity of footsteps across almost every genre – they're everywhere! It's with the utmost care and delicacy that this simple aspect of sound be handled so as to lend itself to player immersion, lest the veil of realism drop and expose the pixels on the screen as what they truly are: groups of flashing lights to stimulate the visual cortex. Oops...SPOILER!
*A Squid Eating Dough in a Polyethylene Bag Is Fast and Bulbous, Got Me?
So it is with this soft eared attention to detail that we will delve into the details of movement design from a systems perspective, and try to carve out some common practices. When jumping (hah) into the throng of movement oriented sounds the core components boil down to: Footsteps and Foley.
THE FAIRY TALE
Once upon a time it was enough to provide a small number of step-only audio files that could be sequenced or randomized in time with the foot falls of the player on screen. In the not so distant past, it was not uncommon to deliver these same steps with a portion of Foley movement embedded into each to help add character to the movement. As available memory and increased polyphony became available, the ability to split these elements into separate assets that could be layered and recombined at runtime by the audio engine became a reality that many have been quick to adopt.
Now that we have these elements separated and playing back independently, we are free to swap out material and clothing types without redelivering our step-only content for each outfit and surface material type. We can keep our content footprint (lol) down by not having to exponentially create combinations, and the flexibility we gain opens up a greater diversity when these elements are shuffled on top of each other during character movement. Coupled with the ability to randomize pitch, volume, and low pass filter between a set of values this content now becomes a seemingly endless array of diversity for any given action.
Developing in parallel to these audio aspects were the refining of animations, increased subtlety of character movement, and increased depth of player control: all of which helped unlock a level of detail necessary to bring Footsteps and Foley towards a model approaching reality. Games that are now reaching for a level of realism have been set free to further granularize their sound content sets to include: additional step types beyond walk or run, to allow for the changing of clothing types, to increase the number and types of movements available to the player at a given time, and expand the list of surface materials a character may be able to move across.
Where once a game may have been able to get away with only walk, run, jump and land it's not uncommon to see scuff, stealth, crawl, or crouch – not to mention the breakdown of heel/ toe, in addition to the Foley layering. Once you factor in the recent spate of free running/ Parkour flavored games which integrate various hand grabs, slides, and other acrobatic feats it's easy to see that the fidelity we're dealing with demands careful thought at every step (heh) of the audio pipeline. From the Audio Director defining the style, the Technical Sound Designers defining the systems, Foley actors portraying these movements, Sound Editors granularizing the files, and Audio Implementers wiring up animations for sound – the potential to infuse the sound with variation and detail that will lend itself to a greater belief in the actions portrayed on screen is immense.
THE SET UP
Sometimes writing about sound can be like dancing about architecture, and so I've assembled a series of game video's that focus specifically on footsteps and Foley within the current generation of games in an attempt to provide perspective. While some might cry foul at the removal of music in the examples presented, the goal was not to assess footstep mix related issues (which could fill up a whole other study) but to allow for a distanced abstracted critical listening experience. In games where as the player you find yourself running for extended periods with only ambient sound and your perpetual footsteps to keep you company it makes solid sense to focus on the persistent ticking of heels and toes whilst traversing the landscape. If, as the player character, you would never find yourself walking or creeping stealthily between places, then the complete lack of subtlety for this aspect of footstep sound can usually be forgiven.
A comparison of the 3 titles whose core gameplay is centered around the players ability to deftly traverse the environment turned out to present a stunning example of aesthetic choices in sound design, as well as showcasing a high level of detail inherent within each of the system designs. While they all focus on player movement and Foley in a similar way, each communicates a style that I feel services the visual side of the equation in a complimentary way. Where Assassin's Creed 2 is the most fluid and “real” sounding, the added iconic/ earconic elements in the Prince of Persia Foley (recognizable – for example – in the identifiable hand grabbing the ring sound) really lent a sense of satisfaction to a successful execution of this action.
While I didn't grab any footage from the original Assassin's Creed for comparison, this article with lead audio designer Mathieu Jeanson of Ubisoft Montréal from Mix Magazine exposes the specifics utilized in the original:
“the footstep system uses more than 1,500 original recorded samples. We managed 22 surface materials, with 14 different step intentions — sneak, walk, run, jump, land, pivot, et cetera — including three to eight variations for each intention per surface.”
In contrast, Mirror's Edge's movement sound was delivered stylistically in compliment to it's grand visual design – stark, clean, and clearly bristling with crispy audio detail. From an interview at Gamasutra with Owen O'Brien, senior producer on DICE's Mirror's Edge:
“In most games, footsteps are a pretty simple thing to add, but running and moving was so integral to Mirror's Edge that we had to create a huge library of footsteps and a system to manage them. We had them for different speeds, different surfaces, different landings; the list goes on and on.”
The squad or group based gameplay of Dragon Age and Mass Effect introduces it's own set of challenges when it comes to movement sounds. In support of the player character, you may have have additional party members following your every move. This means as they trail behind you in close proximity, their footsteps can be heard in a chorus of movement mirroring your progression through the level. This can be good and bad, depending. It's likely that, in order to combat the repetition of playing the same sound files twice, the content requirements demand an increased number of variations to keep things from sounding too similar. Also, it becomes critical that the AI for your group members be intelligent enough to keep pace smoothly without the walk, stop, walk stutter which presents itself when the player is walking slowly. Hearing the AI react poorly in this situation is just as bad as watching an AI walk endlessly into a wall – both expose the underlying shortcomings of the systems behind the action and contribute to a lost sense of immersion.
In a list of suggestions for making Mass Effect 2 a better experience, one player cited “less generic footsteps” as a possible solution:
“I never realized how important it is to hear the footsteps of your character while you are stomping around in a game. To be more precise: how important it is to hear those footsteps reflect the surface that you are walking on....With the amount of walking that you do in Mass Effect, the lack of variety can really get on your nerves. Even changing the pitch of the sound somewhat for different surfaces would improve things dramatically.”
The same could be said about many games where a small number of surface types are repeated throughout. Aside from the battle against the grey-ification of our game worlds, as sound professionals we may be able to help by suggesting variances to surface types that would be reflected in the sound of player movement.
One area that Dragon Age succeeds greatly, is in changing out Foley movement sounds based on the type of armor equipped on both the player and any following party members. By enabling the layering technique mentioned above, Bioware was able to communicate the change of armor through sound, and helped add a level of audio feedback to their system. This feedback, while seemingly inconsequential, enables the player to feel connected to their decisions when reflected back at them not just visually in the armor displayed on-screen, but by the resulting sound when moving. The small touches of detail help to sell the player's role in defining the soundscape in their game, and gives a greater sense of involvement in character progression.
The first person perspective brings with it a certain level of disconnect. The oft-cited “floatiness” inherent with the inability to actually see you feet as the player can be difficult to anchor – with any sense of physicality – to the environment. Optimistically, this is a perfect opportunity for sound to swoop in and save the day by providing an audible feedback representation of the action on screen. While the immersive aspect of the first person viewpoint in games is widely debated, I don't think anyone would argue with the the addition of footstep sounds as providing a necessary connection to the game world.
When I first assessed the footsteps in Crackdown I commented in this thread at the Game Audio Forum that “while I think (the footsteps) sounded ok when running/ running fast...the slow walk did have too much thunk to sell the finesse of the movement.” Thanks to a revelatory response by Tom Todia at Engine Audio, I was able to peel back the academic wax that had accumulated in my ears and remind myself that the game is all about grand, larger-than-life gestures. I can imagine spending almost no time in consideration of actually walking during normal gameplay in Crackdown, which I think highlights what was likely a creative decision that the developer needed to make regarding what was (more) important: the addition of walk steps or increased variations of giant explosions!
In a blog post by Raymond Usher at Gamasutra, he discusses footsteps and “the expectation (from the development team) that we 'need' footsteps” and how that might take focus away from other area's that may be more important.
In a discussion of advanced Modern Warfare 2 perks, the subject of changing the footstep sounds as a part of gameplay is overviewed:
“These advanced perks provide a small secondary benefit, usually of little consequence, to the perk’s main function. For instance, the first game had a perk called “Dead Silence” that muffled the sound of that player’s footsteps. While theoretically useful for stealthy players, in practice it was easily outclassed by every other perk in that tier. In the sequel, silent footsteps became a secondary effect of the pro version of “Ninja” (invisibility to heartbeat sensors).”
Another interesting example I stumbled across in this group comes in the form of Fallout 3 Audio Director Mark Lampert's decision to include a slider in the menu options specifically for adjusting the footstep volume. Where it has been common to see volumes for Music, Sound, and Voice, this ability to adjust the footstep sound in relation to the rest of the mix to suit user preference is an interesting addition.
In games that focus on melee combat, or are otherwise consumed with bringing across multiple systems while catering significantly towards cutting a path through a sometimes endless hoard of baddies, the role of footsteps is often downplayed by design with wall to wall carnage. Which is to say that detailed player movement sound is certainly not the focus for any sustained length of time. For each character in Star Wars: The Force Unleashed we had about 10-16 file variations per step type randomly shuffled across 20 materials for walk, run, and land. In Conan we had 4 file variations per foot across 9 materials and 2 step types. There is no magic number when it comes to deciding at what point you have enough variation, often it becomes a complicated equation of space vs. perceived need for diversity.
In the current generation we have 10-20 times the memory of last gen, our tools are more intuitive, and our understanding of the process is greater. What I've come to find in my assessment and experience is that despite this ability, there seems to be two schools of thought regarding footsteps in the current generation, of which Lost Odyssey is currently my poster-child.
1. Maximum variation: Both at the file level (lots of steps) and through randomized pitch and volume.
2. Iconic (Earconic?): The “right” footsteps all of the time with minimal variation.
I think the idea of maximum variation is covered pretty well above regarding our current capabilities, but the idea of Iconic footsteps might still be a bit vague. Essentially the idea of is that, as an aesthetic choice, a minimal amount of footstep variations are chosen because they exemplify the particular character and step type without the need for randomness or variation. The thinking seems to be that if exactly the right (for example) two footsteps for a given step type are chosen, then the action can succeed with minimal variation and the sound of the character can be defined iconically by the sound instead of bowing to a perceived reality.
While the choice to go Iconic for footsteps could be due to limitations of RAM, I'm finding it difficult to believe that this is the case, and so I've started to think that it is a conscious design choice. I'm also trying hard not to label it a "last gen" technique and chalk it up as a hangover from a time when you were lucky to get 2MB of RAM for the entire level of a game. I would like to believe that this is a choice that some designers are making, instead of a response to limitations – especially because of the pervasive presence of footsteps throughout the entirety of a game makes it hard to ignore and certainly is a decision that must be consciously made.
I have no info about the technical choices and tradeoff's that were made in the creation of the footsteps for Lost Odyssey, but it's easy to hear that they have implemented a smaller more iconic set of footsteps for each step and material type with little to no pitch or volume randomization. This is especially apparent when climbing up and down ladders. That said, they sound exceedingly appropriate and are designed to a high level of detail.
(The interactive bell ringing playground equipment is also a must see/hear!)
THE CONCLUSION
At the end of the day, it should be about building appropriate systems that support well designed content in order to best portray the action on screen as a way to sell the players role in the environment. Whether this comes with or without fast bulbous squids, it all boils down to the same thing: we're all slaves to the game.
Until next time!
*That's right, The Mascara Snake. Fast and bulbous! Also, a tin teardrop!
Special thanks to contributers to the contributers to the Game Audio Forumpost that fueled my fire and our special guests on Episode 2 of the Game Audio Podcast where we discussed some of these things.
**EDIT**
I ended up spinning this into a talk given at GDC 2011 and further reprised it at my local IGDATC meeting.
