One look at their intense, hypnotic imagery and they’ll be etched in your memory. It’s the sound, as well as the all-out, unique visual perspective of Australian electronic duo Empire Of The Sun, consisting of Luke Steele and Nick Littlemore, that is mesmerizing music fans around the world.
Empire Of The Sun is “a different dimension created after the psychedelic experience of embracing the end of the world (EOTS Japanese website).”
From their hit debut album, which sold over 1 million copies worldwide, to the present day, they have created a cohesive story through their works and performances that is best described as a grand epic. On November 10, 2017 they released the music video for “Way to Go” from their recording On Our Way Home EP, with direction led by WOW, Inc. We sat down for an interview with Creative Director Nobumichi Asai and Director Ryo Kitabatake.
- How did you get started with this project?
This March I got an offer to direct the music video for EOTS. They wanted to make a video like “INORI” or “Kacho Fugetsu” which featured face mapping, but at the time I was interested in Digital Humans (CG technology making photo-realistic human expressions). I proposed the music video which uses “Face” as expression, and quickly got the okay, so I started production.
- Why did you decide to use Digital Humans instead of face mapping?
There were two main reasons. The first was that it became clear in the initial stages that it would be very difficult to fulfill their request using face mapping. Secondly, the two members of EOTS agreed with Nobumichi and I in that we wanted to try and make something interesting using new technology. We had been focused on using facial capture CG expressions for a while, and it fit with what they were trying to convey, so we went in that direction.
In the world of visual effect s, the current trend is trying to find out “how human-like can photo-realism get?” We understand the principles, but in practice, because many schools of thought overlap, it’s the know-how that’s necessary to produce something. If WOW isn’t able to try new things, we’ll get left behind, so there was a motivation to drive forward. Ryo felt the same way, so we thought it was the perfect chance, and we decided to face the challenge.
- Where did you get the concept for the video? Also, how were you able to create the tones, textures, colors, etc. for the unique world in the video?
We took the concept from the theme of “face,” as well as the lyrics and themes from the song. Also, we got some input from meeting with the band. As for the tones and motifs, we incorporated the psychedelic art of Keiichi Tanaami as well as the visuals from EOTS’s past concerts. Although, their visual style isn’t really something you come up with in a normal state of mind. It’s more like something you come up with at a weird time, like right before crashing after being up all night drinking (laughs).
- Ryo, it’s a bit of a departure from the style of your previous work, wouldn’t you say?
I was hoping that I don’t have a distinct style, per se…I guess I have some habits. The members of EOTS said, “We want something psychedelic and trippy,” so through some trial and error we combined their visual concept with our styles and tastes.
- Can you tell us about the how the collaboration with Zukun Lab came about, and why you chose to work together? How do you feel about how this collaborative project turned out?
We tried to do it on our own, in our own style, but we found out how truly difficult it was working with Digital Humans technology.
To complete the facial capture in the short timeframe we were given for the project was incredibly difficult. Most of the people we consulted with declined, saying it was too difficult in that short of a time period. Only Zukun Lab was willing to say, “Let’s think of a way to make it happen.” Zukun Lab was our savior.
Zukun Lab is a Japanese lab, but it’s a prominent lab in using and developing Digital Humans technology. Their video “3D CG High School Girl Saya,” which uses similar technology to our project, created a stir online and on social media.
Well, there was a lot of technical tinkering at WOW and Zukun Lab from the beginning of the project, and we were barely able to finish coordinating everything. Especially when it came to the transfer of data, we met whenever we had the time to transfer data while trying to lose as little of it as possible, and so the verification of all of this added up. Under these circumstances, each section had experience dealing with the wealth of information being transferred through Zukun Lab’s flexible pipeline. With them, I don’t think we would have been able to succeed in getting this project done in that short timeframe.
- I’m sure there were some technical challenges during the process, how did you try to solve them?
This project combined the multiple elements of photo scanning, facial capture, motion capture, and filming on a blue screen. In order to create the video, it was imperative that each process was processed accurately. In addition, EOTS was on tour at the time, so our schedule was pretty tight. Accurately processing each step quickly to complete the project on time was the biggest challenge this time around. For the facial and motion capture processes in particular, Zukun Lab set up a customized rig specifically for this project. We were able to seamlessly integrate it at our building to help complete the process. Because we were able to forge the partnership with Zukun Lab, we were able to work both quickly and accurately throughout the entire production to realize our goal.
For me, there were two main challenges. The first was syncing the 3D scanning of facial expressions, the tracking of the facial expressions, and the composite from the actual photographic images. Secondly, getting a composite of the entire body by setting the camera position and tracking to get both the head and body was a challenge. Since we were on a tight budget and timeline, we initially were just going to animate the face, without composite images. However, we felt in order to fulfill our vision and create something truly fascinating, we felt strongly about animating the entire body, so decided to do so. With the support and understanding of Zukun Lab, of course, jitto, who was in charge of the composite photography, and others, we were able to succeed in making the music video without sacrificing quality.
- Please tell us about the production process.
“Way to Go” is the product of a number of processes. One particularly irregular process was that to start with, we took the facial photo-scan shooting at a studio in L.A. We didn’t go ourselves, but with Zukun Lab we remotely elicited the correct poses as we watched them via computer.
After analyzing images from dozens of optical cameras, we scanned them to make a 3D model. Through this process, we can reproduce facial pores and even minute unevenness of the skin. A great thing about this is that at the same time we can capture the texture without losing anything between the model data and the texture.
Incidentally in the face mapping process, the photographer continuously takes photo after photo for the texture, then manually pieces the images together, so there are small inconsistencies. But, if you think about facial expressions, both processes have their merits. A photographer’s sense, combined with things like lighting and timing, can raise the quality of the facial expression capture process. Face mapping has become a really important creative element.
Zukun took the data from L.A. and used Retopology and displacement mapping to create the textures, which they then entered into the rig for us. They then delivered it to WOW, and with the developed visuals from the WOW team, we ran an animation test. From there, in order to film, the 2 EOTS members came to Japan. At Zukun Lab’s studio, there was facial capture in the morning, followed by a photo shoot, including motion capture, in the afternoon.
It’ll show up in the “Making of” video, but we did the facial expression tracking using a number of sensors from a company called “VICON.” In all, there were about 400 markers attached to the face in order to capture the tiniest of facial movements. Those are inserted into the 3D model of the face. By doing this, we can create vivid, lifelike CG facial animations. By the way, the members of EOTS lay down for us so that a number of staff could stick those 400 markers on them. It took a long time and was quite the trial (laughs).
After finishing shooting, Zukun entered the capture data in the rig, and we used FBX skeletal animation for the head motion we captured and used Alembic to produce the facial expression animation. After that, the WOW team merged the two sets of data with the rig we received beforehand, and we completed the effects and animation to be used in the music video.
Next, for the CG head and facial expression work, there is a composition process for the images taken of the two members. In order to make it work, the “CG head and facial expressions” and the “true image of the head” must be combined so that the size, position, and angle is perfect. In order to achieve this, motion capture clusterization markers are placed on the artists’ heads and the camera, and then the motion capture process begins. In theory, if these sensors capture the position and angle from both the camera and the head object, the same state can be reproduced. In reality, the data captured by the sensor can contain measurement errors, so after the motion capture, there is a data management process in which we must delete any noise, correct any inconsistencies…an enormous undertaking awaits.
Actually, facial expression tracking is quite a difficult problem. No one facial expression is the same from one moment to the next, so even precise scanning and tracking can’t produce the exact same images as a photograph of a human face. There are inaccuracies and gaps due to camera positioning and distance, as well as the difficulties in putting together the composite image.
- Sticking the markers on the face during the making of the video was incredible, what did the EOTS members think?
They said, “Cool,” but I got some goose bumps just looking at the bumpy-looking things all over their faces. As for the staff, they were freaking out (laughs).
- We talked about how the project developed and changed already, but could you tell us more in detail?
We had staff in different places around the world: L.A. (Luke, manager), Sydney (Nick, label director), Tokyo (WOW, Zukun Lab, etc.). Because of this we used appear.in for communication for the first contact through the video going live online. We only met EOTS in person over a two day period, when they came to Japan. Other than the image capturing in Japan, all of the communication, including data and spoken exchanges, were completed over the internet.
This time, our project was a virtualization of data from the artists’ physical self, so we used the internet to collaborate and co-create the work. Through the creation process, we could bridge the gap between countries in real time to communicate with each other and keep the project moving forward. Every bit of it felt like we’re moving into the Virtual Age.
- Any reflections on the production?
As much as Zukun Lab, we owe a debt of gratitude to jitto as well. They oversaw everything from the photography planning to the composite imagery, and their advice at every step helped us get through the complicated process smoothly. Taking over the online and color grading for us helped raise the quality of the entire project.
For this project, I don’t feel like we re-established how to use existing technologies and methods, but more that we avoided falling behind. Hollywood visual effects are more amazing and more common, too. I’d say, knowing Japan is lagging, we took that to heart and tried our best. From here on out, I want to continue to progress and evolve.
Empire Of The Sun MV 「Way To Go」 Credit
Creative Director：Nobumichi Asai
Director：Ryo Kitabatake (WOW inc.)
CG Designer：Tsutomu Miyajima, Daisuke Moriwaki, Shota Oga (WOW inc.)
Technical Director：Nobumichi Asai
Assistant Producer：Ayaka Motoyoshi
Lighting Director：Ryu Shima (NAGAIHOSHI Co.,Ltd.)
Compositor&Colorist：Akio Sakamaki (jitto inc.)
【Digital Humans & Performance Capture】
Motion Capture Supervisor：Kenya Miki
Facial Capture Supervisor：Hiromu Kinoshita
Digital Artist：Tomoya Tanaka, Deng Xiaohui, Cheng Ze, Satoko Itakura
Motion Capture Specialist：Yuuka Tomita, Kouta Isomura, Kazuya Takahashi, Riku Okada, Rina Sawada
Facial Capture Specialist：Taisei Kurita, Manami Kira, Diogo Ribeiro
CG Project Manager：Yuko Sadaki
CG Producer：Sawami Takahashi
Production Supervisor：Ayumu Kasai
Translator：Mika Iwasaka, Michiru Sasaki (WOW inc.), Sumiko Shiraishi (F.O.A. Inc.)