A behind-the-scenes look at CAPSULE's VR concert "CAPSULE Live in VRChat 'Metro Pulse'" (Sound & Recording Magazine, 23.08.17)
On August 5th, 2023, electro duo CAPSULE held their concert for the album Metro Pulse. The location was a metaverse space known as VRChat. An audience of around 1,200 people gathered, many from overseas, in spite of the high technological requirements for entry. People shared their excitement for the event via social networking websites (SNS), but as artists have increasingly returned to in-person concerts following the coronavirus pandemic, how did Nakata Yasutaka, the brains behind CAPSULE, decide to go with a virtual performance instead? We'll take a look at the story behind the production with Nakata and the show's director, computer graphics artist ReeeznD.
Translation by ystk-archive ・ 3,712 words ・ Original interview ・ Concert
【Start of VR live project】 The virtualization that stemmed from VR meetings
I'm going to start off with some basic information about VRChat. Also known as "VR SNS," VRChat is a platform where users create avatars and communicate with one another. Areas in VRChat called "worlds" serve various purposes that range from conversation hubs, to places where one can enjoy beautiful virtual scenery, to hosting fashion shows or, in this case, live concert events. Putting it simply, VRChat is being improved by the day through social input, and its cutting-edge atmosphere is part of its allure.
─ When did the plan for this VR concert start?
Nakata: The music videos for our latest album Metro Pulse already featured virtual worlds, but the MV creator Saisho Kentarou-san suggested that if we could use a videogame engine, it'd be possible to move around those worlds in real-time. The music videos had to be done using Cinema 4D, but I kept thinking about getting more usage out of my virtual avatar. So it didn't start with the idea of doing a concert in VR, but rather it came about from wanting to make stuff that could go into a CAPSULE music video.
─ In addition to the VR concert hall area, you also created a space called the "CAPSULE HOUSE" for users to explore.
Nakata: It's a building that looks exactly as it did in the "Virtual Freedom" music video. I designed it alongside the director [Saisho] with a "resort studio" concept, and I proposed that we use it as a sort of lobby area for the concert hall since we already had the data.
─ At what point did you earnestly begin working on this VR concert?
ReeeznD: I was asked to work on it sometime in March this year. Since the most important thing was to make sure the concert would be a success, I started by considering how much could be put into it, all while being conscientious of how much time I had until the concert date. Because of that, I suggested a realistic duration of four songs/fifteen minutes, but even with it being that short it was still worked on up to the last second (laughs).
─ What sorts of back-and-forth conversations did you two have?
Nakata: Those took place in VR without us realizing we were in VR.
─ What do you mean?
Nakata: Everyone wore headsets and entered the VR space to work on it, just like if it was a venue in real life. We had discussions like "it'd be better if the height was like that" and "the view from the audience seating should be more like this" and whatnot. Instead of everyone discussing while having one screen to look at, each of us could move around in the 3D environment and gesture towards things. It was pretty different from how we made the music videos.
─ So you held your meetings as avatars in the VR world.
Nakata: Yep, that's right. It was an interesting hands-on experience, but it's difficult to explain how fun VR is to people who haven't tried it. I think it's hard to convey the coolness of it to people reading this article and looking at the screenshots on a flatscreen monitor, so I hope they'll get ahold of a headset and jump in, no matter what it takes (laughs).
─ Lots of users on SNS who had never been in VR before were posting about their excitement for the show. The experience has an impact that's indescribable.
Nakata: That's true. I think it's important for a VR concert to be seen in VR in order to be fully experienced, just like live shows in real life. If you don't see it in person, you can't appreciate it properly. And it's amazing that VR concerts have gotten to the point where they can give attendees the same sort of feeling they'd get at concerts in the real world.
─ Being there as it's happening is now the same experience whether it's VR or in real life.
Nakata: It is different from just watching something on a screen by yourself; users can chit-chat together before the concert starts. In the early stages of development when ReeeznD-san and I were going over what they'd made so far, I altered the sound accordingly. You could say the audio initially sounded sort of dry, like it was left as-is the way it sounds on the CD release, but I wanted the audience to get the sense that there was more space for the music to permeate. VR differs from in-person concerts in that it's up to the user how they experience the music, but we still consulted about sound effects, like "how would it sound if someone moved over here?" and "we should alter it depending on how wide or small that part of the soundstage is." So from there ReeeznD-san would show me his progress and I'd add sound effects as I saw fit.
ReeeznD: Here's an example for you: during "Give me a ride," Nakata-san added an explosion sound when the logo appears.
─ So not only does Nakata-san handle the music for the performance itself, he also does sound effects. That's very like him, considering how much he loves film.
Nakata: That logo scene was originally supposed to happen at a key moment (laughs). But since it shows up after the song ends, I could put in a sound effect without messing up the music.
ReeeznD: Sound effects were supervised by Kinu-san. He works on his own VR concert projects, doing everything from sound to stage direction, and if the sound effects aren't good enough he'll add them in himself. But other artists' music is untouchable in a sense, people on the production side can't do whatever they want to it without getting permission. The two of us talked about how sound effects were going to be difficult to pull off for this particular show, so we added the logo and explosion visual when there was a lull in the action. I was really impressed when Nakata-san put his own sound effect in that part because we didn't ask him to — Kinu-san was excited too (laughs).
─ That's some valuable insight into what it's like for people working on a creative team together.
Nakata: Also while we're talking about it, I added a 40-50Hz, rumbling sort of low sound effect between each song. Just for the sake of giving it some ambience. I don't even know how many users could hear it, but I think if you wear headphones it's audible. I'd like for people to listen to the show using their best possible audio set-up.
【Building VR sounds】 Sounds that can "only be heard in that spot" while accounting for attenuation and venue dimensions
Although you could watch this concert with only a PC, it's infinitely more enjoyable to connect via VR headset and get immersed in the environment. Compared to VR's first year on the market (in 2016; so nostalgic...), with offerings like the Meta (Oculus) Quest 2 and HTC Vive et cetera, the technology is now widely available: Apple just announced their Vision Pro, and the Meta Quest 3 goes on sale this year. The only major disappointment is that VRChat is currently only available on Windows, with no Mac support. Regarding audio, you can listen via the headset's internal mic or, as Nakata suggested, connect your preferred DA converter and use your headphones.
─ How did you tackle the issue of sound quality?
ReeeznD: In VRChat, worlds are created using an engine called Unity, and many creators leave the audio up to Unity as well. However, since this was a concert, we wanted to retain as much quality as possible. What's challenging about doing a concert is the amount of people in the audience can cause the frame rate to lag, which slows down the visuals of the show, so because of that we were searching for a way to maintain sound quality without weighing down the processing. We compressed the audio while checking the quality and processing load in Unity.
─ You set it up with speakers on the left and right sides of the stage.
Nakata: We sourced it via stereo, but ReeeznD-san adjusted the sound within the world to give it a stereophonic, 3D effect. I added ambience to the stereo source, and ReeeznD-san set up attenuation processing on his side of things.
─ Of course the way head tracking works causes the left and right sides where the sound comes in to change, but height and distance also impact how things are heard. How did you go about managing that?
Nakata: Because the space itself is interactive by nature, users aren't, for example, looking at pre-rendered visuals, but are seeing things moving in real time. The music side of things also involves real-time positional processing. When we first did a VR soundcheck, it sounded like the music was coming from a tape deck that was left onstage. I wanted it to feel encompassing, so ReeeznD-san focused more intently on sound sources, and from there adjustments were made to things like from how many meters away sound is heard and so on.
─ In regards to a sense of distance, was there a particular tool you used to tweak the sound?
ReeeznD: Using VRC Spatial Audio Source, I was able to simulate the music in a 3D environment and make it stereophonic. The attenuation is essentially a graph that simulates the actual real sound, but for our purposes I rewrote the whole thing to create a live listening environment that's pleasant on the ears. However, only using stereophonic sound would cause the audio quality to drop, so I mixed in an unnoticeable amount of non-3D sound to balance it out.
Nakata: The concert audio is different depending on where you're listening to it from. I wanted to get that aspect of it right. There may be people who want to hear it the exact same way from no matter where they're standing in the venue, but I want it to be a listening experience you can only have in VR.
ReeeznD: Initially the floor of the concert hall was completely level instead of being an amphitheater. Partway through development, Nakata-san suggested changing it, and this altered the acoustics as well. Since my background is in music video production, I was thinking at first that people would be happy with a final stereo mix like you hear in a video, but Nakata-san was very considerate of the fact that this was a VR concert and made suggestions to capitalize on that, so we kept coming up with ways to match his expectations.
Nakata: There a lot of things you could call VR, like fixed images that have a 360° camera but won't allow the viewer to move around in the environment, or even non-3D objects, but in that sense of the term I think this was the most "VR" live show I've seen so far. Users could have conversations with each other and them moving around would change what they could see and hear. I'd like as many people as possible to know about the world of VR.
【VR and DTM* production environment】 An amazing era where you can do anything with a modern PC
*DTM = desktop music, or music that's created using MIDI/computers
With the mention of the game engine Unity, a DTMer reading this may think that VR is a whole separate world from music creation. On the contrary, Nakata sees similarities between the two fields. ReeeznD agrees: "Actually, the difference isn't that great."
─ From the sound production side of things, producing VR content seems extremely challenging. Being able to do both would make one a force to be reckoned with...
Nakata: Basically, people who do music and people who create VR content are not entirely dissimilar. Both can be done in real time without the need for rendering. Just a CPU is enough for today's DAWs, and you can work on projects in real time without having to write things out in advance; there's a lot you can achieve. I think we live in a time where we have an incredible amount of power to take advantage of that. It's huge that someone could do this in their own individual environment.
─ It's a time where anyone with motivation could do anything.
ReeeznD: Yes. The path is open, and there are outlets to showcase it.
Nakata: Of course having skills and good taste are important, but the means to create are widely available now. That's amazing, to me. ReeeznD-san, how did you get to the point where you can create content like this using game engines? From studying Unity, right?
ReeeznD: I worked as a director at a game company for about ten years. Afterwards, I went freelance, doing CG and music video production. I first started using Unity as part of motion capture work with lower-priced consumer goods; I'd bought an HTC Vive for somewhere around seventy- to eighty-thousand yen to do mo-cap with. After that, I started seeing screenshots on SNS from VRChat servers like Ghost Club and thought it looked really fun, so that's how I got into the world of VRChat.
Nakata: It's interesting to me how in VRChat, you have hobbyists' work intermingling with that of professionals. It'd be fun if I could get to the point where I can make my own world.
ReeeznD: I was influenced the most by hobbyist worlds while making this VR concert, rather than pros. There are a massive amount of music worlds in VRChat and I've been to a lot of them. Kinu-san, production team member tanitta-san, and cap.-san, they've all got more experience in VR than I have, and they've made worlds as well as avatars.
Nakata: You see the same sort of thing in music. When you're trying to create something, it's not helpful to take cues from what the people at the top are doing. I think the more interesting creations come from regular people.
─ That seems to be how it goes with new cultures, they spread out from a grassroots origin but eventually become commercialized.
Nakata: I think a whole lot of people would agree with that (laughs). VR as a platform is relatively young, isn't it? I was really lucky to be able to work with ReeeznD-san and Kinu-san along with numerous other people active in the field. If I only concerned myself with music production and nothing else, I don't think I would've done this, but I'm interested in the tech part of it and I think the culture around VR and the user experience are important. VRChat wasn't designed for music performances in the first place. If we'd just thrown the songs into a VR concert without any care, we wouldn't have been able to accomplish so many things. I learned — and goofed off — with others throughout the whole process and it was a very meaningful experience. I really want to learn how to make my own world. Even though I know how hard it is to start from zero when learning something like DTM (laughs), it's easy to blindly dive into a new thing.
─ For someone starting out with no knowledge of it, how long would it take to learn Unity well enough to create a VRChat world?
ReeeznD: Assuming it's their first time ever using Unity... I think it's doable in about a month.
─ In that short amount of time?
ReeeznD: Making your own VRChat world is a daunting task, but resources for them are plentiful, so I recommend starting there by changing things to suit your preferences. That way you'll learn the basics, like adding music and placing a video player, and it'll most likely be about a month until you feel ready to publish the world. Once you've done all that, you can gradually make it more original by replacing purchased assets with your own modeled ones.
Nakata: Similarly in DTM there's something called a construction kit that has pre-made loops in it that you can use as a basis when making your own songs.
【Possibilities of VR live】 A future where avatars can switch the presets on a synthesizer in real time
I experienced firsthand through attending this concert that it's a different experience from simply watching something on a screen. Terms such as "virtual" and "immersive" are overused, and it's understandable how someone only familiar with VR in its current early stage of development would write it off or just imagine what it's like before they've seen it. To be entirely honest, I was one of those people. However, Nakata points out the tremendous entertainment potential of VR live shows, which brings to mind certain prejudices myself and others have about the superiority of "real concerts." Maybe a lot of creators who read this interview will come away from it wanting to learn more about this new field immediately.
─ Before, you mentioned that the production during the concert was done in real time. In which specific parts is this noticeable?
ReeeznD: An easy example is during the chorus parts of "Hikari no Disco," you can see streaks of light going every which way, and since the only instruction given was for light to scatter within a certain area, there's an element of randomness. But above all, the true random element is how users can view the performance from any vantage point they like. Even the sound changes depending on where in the 3D environment they're standing. It's difficult to notice because this kind of thing is obvious in real life concerts, but it's only possible here because a computer is undergoing real-time processing to achieve it.
Nakata: It's important not only to create the environment in real time, but to be actively working on behalf of the user's experience too. If I learn a bit more about the sound aspect of it, I think I'll be able to amp up the interactive elements. For this concert I prepared the music ahead of time, but if we could do it in real time I could make it more interactive and it would better convey the advantages of the VR format. If we really wanted to, we should be able to have the avatar up on stage push a preset button on the synth and change the music.
ReeeznD: Actually, there is a world called Fractone where you can play the synthesizer and program a sequencer.
Nakata: But if we try to implement things like that, your workload is gradually gonna increase (laughs). Honestly, out of all my performances, this show was the one that utilized the most of my ideas. When you try to do it in real life, it's difficult to find a venue to begin with, or what you want to do is flat-out physically impossible, or you just can't do it regardless of budget. However, in VR there aren't limitations like these. Rather, you can add aspects that give it a more realistic feel.
ReeeznD: When I talk to people familiar with the VR experience, nowadays they aren't sure if what they're seeing is fiction or reality. Another way you could look at it is Disneyland was made from fiction, but going to Disneyland is a real-life experience. I think that's the kind of sensation happening in VR. It may be a world created with CG, but it's still a place you can experience with your friends that isn't any different from reality.
Nakata: For example, when an online meeting concludes, I just think "it's done." But when I've been in a VR world, I take off my headset and think "I'm heading back." Like I'm going home (laughs). I can really tell the difference. It's as if I left the house for a bit.
─ Between ReeeznD-san and Kinu-san's talents and Nakata-san connecting with them, this turned out to be a great collaboration between fellow creators.
Nakata: Oh, no, not at all, it's more like I was lucky to get mixed up with some VRChat pioneers.
─ How are you planning to make use of VR in your future activities?
Nakata: Places where music is played bring with them a set of experiences, I think. Festivals are definitely like this, but there's a lot of people who just enjoy the process of going to them in itself. So I consider music to be a part of the production of an event like that, and I feel it's best to create things that suit each respective platform. In VR's case — as we were talking about sound effects a little while ago — sound doesn't have to be created under the assumption that it'll only be coming out of two speakers, so it's necessary to create a different mix specifically for a VR environment. It's been a rule-of-thumb to do things like increase the low end for songs meant to be played in clubs, or to make a song slightly louder for karaoke purposes; this is just the VR equivalent of that. It's just like how a film's multi-channel audio is mixed differently from its soundtrack. Since I mastered a stereo file this time, I want to try parallel processing for the next one.
─ With Nakata-san having cut the ribbon, if other artists get involved in the VR concert arena, I get the feeling we'll be able to see a new sort of future.
Nakata: I think so too. Then again, I'm worried about ReeeznD's schedule if everyone starts doing this (laughs).
11 notes
·
View notes