In the video, I directly manipulate the RAM of Nintendo’s Super Mario Brothers to transform it from a game into a strange instrument. I use two hardware controllers: illucia (a patchbay instrument I designed that lets me connect computer software with cables), and the Soundplane (an amazing multitouch surface by Madrona Labs).
I begin by playing the game as one normally would, just using buttons on illucia.. but I also have access to the game’s memory, so I use the Y axis on the Soundplane to alter the value in the memory address that determines Mario’s Y position onscreen. This is how I make Mario fly and hover during the playthrough.
Also, before I start playing, notice that I flip a switch on illucia. This triggers recording — not video, but actually recording the entire memory state of the NES for each game frame. Because I’m saving the game ~30 times a second (and keeping log of all saves) I’m able to go back to any moment in Mario’s life. Sort of like a Super Mario time machine.
So then I use the X-axis of the Soundplane to sweep through the timeline of Mario’s universe. Not only that, but the Soundplane is multitouch, so I use a second finger to specify start and endpoints in a playback loop. This is similar to the way samplers and granular synths work, but for recordings of the entire memory state of the NES rather than audio data. Conceptually, it is like Super Mario meets Groundhog Day. Mario’s universe computer / time machine gets caught in hellish loops.
Then I use illucia to send alien data into Mario’s universe, which makes for all sorts of audiovisual insanity amidst the spacetime loops. I found some memory addresses that produce interesting results, so I use illucia to pump them with unexpected data. This is sort of like circuit bending, but in a protected sandbox – at any point I can revert back to the clean recording of RAM states (aka moments in Mario’s universe).
At that point I try to go back to „playing“ the game, watching Mario navigate a melting world of glitched-out ephemera. Toward the end of the video I use a pair of rubberband mallets on the Soundplane to jump around in Mario’s timeline, all while illucia is left pouring a heavy stream of alien data into Mario’s RAM state. I eventually (accidentally/luckily) land at a place that triggers the game over music, and decide to end the take.