Student research bridges gap between art and technology

Sumin Jung, a professional dancer living in the Blacksburg area, performed her choreography during the ICAT Creativity + Innovation Day on April 29 at the Moss Arts Center. Throughout the performance, the display projected onto the walls of the studio changed, and at times, the sound did as well. Petals fell from a tree, ocean waves crashed, stars twinkled, and the pitch of the sound changed.

These visual and auditory transitions that accompanied Jung’s choreography were not, however, directly controlled by a technician.

Jung affected them with her breath and movement.

People may not instinctually associate computer science with visual art, but Jacqueline Bruen, a computer science doctoral candidate in the College of Engineering, has collaborated with Jung to help bridge the gap between technology and artistic expression by creating a visual and auditory representation of embodied knowledge.

“With artistic practices like dance, painting, crochet, and all activities that require broader gestures in a particular environment, sensors and various technologies can collect that information, classify it, and understand it,” Bruen said. “Then, the computer can interact with it or elicit some reaction from the computer based on the information we collect from the embodied knowledge of the artist."

In traditional dance performances, movement is based on musical selection and then lighting and scenery are added. Bruen and Jung’s approach flips that model.

During the performance at the Moss Arts Center, Jung wore respiratory sensors and reflective markers on her body. Multiple infrared-based cameras tracked the markers as Jung danced to calculate her location and movements. The respiratory sensors tracked her breathing patterns. That information, carefully and thoughtfully mapped by Bruen, altered the scene and sounds.

“This is kind of reverse mapping,” said Myounghoon “Philart” Jeon, associate professor in the Grado Department of Industrial and Systems Engineering with a courtesy affiliation in computer science and an ICAT Senior Fellow. “Dancers make choreography first, and then that impacts visuals and sound.”

Jeon, Bruen’s faculty advisor for this project, has worked with many dance projects at Virginia Tech and is a friend of Jung’s. Two years ago, Jung told Jeon about her interest in Korean traditional dance and the breathing techniques that are an essential component for choreography. She presented on this topic at the Future Bodies Symposium at Virginia Tech in 2022, informing the audience she would “like to collaborate with technology which will reinterpret respiration and heart rate by visualizing,” a concept that expands the idea of artistic expression.

“She [Jung] wanted to practice and express those breathing techniques, so we started to discuss ideas about how we can utilize technology to make this breathing even more salient so the audience can see this breathing while a dancer is making choreography. So that’s how we started to approach this,” said Jeon.

As their idea grew, Jeon saw a connection with Bruen’s research focus. Bruen took the lead on this first experiment of the idea, an experiment that will serve as part of Bruen’s dissertation.

Jung had a vision for the visuals based on a picture book she encountered while researching Korean comfort women — women forced into sexual slavery by the Japanese military during World War II. In the future, Jung will create a larger dance composition that explores this part of her cultural heritage. Jung said more research is needed to convey that history through dance. This performance was an experiment that explored how technology and dance can work together to help Jung accomplish her artistic goals moving forward.

“Technicians are typically very cautious to come over to my boundary as a maker, but this time, we’re more open to share our thoughts and have discussions,” said Jung. “There is no boundary.”

“We want to make a playground where technology or new technologies can help out artists and dancers so they can express new things,” said Jeon. “The ultimate goal for all of us — artist, audience, and technicians — is about expanding the perceptual dimensions.”

Collaborating across disciplines also speaks to Bruen’s research goals and beliefs that these experiences enhance creative problem-solving. As someone who loves both art and technology, Bruen hopes her research will demonstrate that it is possible to work in both fields.

“I want to bridge that divide and facilitate greater collaboration between these fields and also encourage people who think that technology isn’t for them that, actually, it can be,” said Bruen.