Late Peking Opera artist comes alive through VR

Amid the melody of Jinghu, a musical instrument used in Peking Opera, an AI-powered virtual replica of Peking Opera master Mei Lanfang walked towards the center of the stage at a gentle pace holding a foldable fan.

The vivid scenes played out at the virtual avatar laboratory of the Beijing Institute of Technology (BIT). In 2020, the Central Academy of Drama (CAD) in Beijing and the BIT jointly launched this public welfare research project aiming to create an interactive "digital Peking Opera artist."

Mei Lanfang (1894-1961) was a world-renowned Chinese artist of Peking Opera who made prominent contributions to the improvement and popularization of the art form.

"Information technology and artificial intelligence were applied to create the digital version of Mei," said Weng Dongdong, a researcher at BIT.

From face, posture, tone to clothing, props, and so on, all details revolving around a person in real life need to be digitized and visualized before replicating the subject into the virtual world.

However, it's never easy to digitize the deceased as it is no longer possible to collect their three-dimensional data.

"We gathered a lot of relevant historical photos of Mei and turned to professionals from the Central Academy of Fine Arts (in Beijing) for a portrait sculpture based on the pictures. Thereafter, we scanned it with a high-precision laser scanner to secure Mei's digital facial structure," Weng said.

He added that the sculpture itself is short on microscopic features of skin and hair, so the team found an impersonator of Mei, and collected his facial data, captured some basic expressions, and "transplanted" them to the digital image.

"Plastic surgery experts were invited for advice so as to help obtain precision, as far as possible, in the facial replica," Weng added.

Song Zhen, director of the Center for Advanced Research on Digitalization of Traditional Drama, CAD, said the team reviewed scores of relevant documents and visited many tailoring shops in Beijing in a bid to dress the virtual image.

"We discovered that Mei's costumes were sewn with gold thread and the technique is no longer in practice. But we managed to find cloth samples passed down from the past," Song said, adding that with the help of experts in the Peking Opera field, the team pinpointed the most suitable clothing and scanned it for data.

"Before undertaking the project, we had no idea it would involve so much knowledge from various professional fields as well as such extensive research," Weng said, adding that many details are yet to be figured out, including the costumes and the helmets Mei used to don for his performance.

Weng said they will invite Peking Opera artists to imitate Mei performing for motion captures in the future. As for the digital Mei's voice, the research team had to recreate it through emulation as most of the existing audio-visual records of Mei have poor sound quality.

"In the future, we hope to create an immersive and interactive character of Mei, which will enable the audience to appreciate his Peking Opera performance and virtually interact with him in real-time by wearing a VR headset," Weng said, expressing hopes that the technology can also be applied for the popularization of science and educational purposes.

"My grandpa passed away in 1961. But with the assistance of cutting-edge technology, young people can watch his performance and know him. This is very meaningful," said Fan Meiqiang, Mei's grandson.

"The combination of technology and China's excellent traditional culture can intrigue the audience of Peking Opera, and help them better understand the quintessence of Chinese culture," Weng said.