Holocaust Survivor Holograms Give History New Depth
When you meet Pinchas Gutter, he's ready to talk. Seated comfortably in an armchair, the Holocaust survivor makes eye contact and lets you have the first move.
"Hello!"
"Hello, how are you?" he asks.
"Good. How are you doing?"
"Not too bad."
While this may sound like an average interaction with someone you've never met, it is far from it. Gutter isn't physically present. In fact, he's probably at home in Toronto. This conversation is with a 3-D recording of Gutter recounting his experiences in the Holocaust: surviving the Warsaw Ghetto, a death march and six Nazi concentration camps.
As the remaining Holocaust survivors like Gutter reach their twilight years, so do the last living testimonies to the genocide. New Dimensions in Testimony, a new project from the University of Southern California's Shoah Foundation and Institute for Creative Technologies answers the question, "What if you could ask a Holocaust survivor anything, even after they've died?"
The Shoah Foundation already had thousands of recorded Holocaust survivor testimonies in its Visual History Archive when it was approached by concept developer Heather Maio of Conscience Display with a fresh approach: What if we made the testimonies interactive?
"Being able to talk to a Holocaust survivor is something that stays with you forever," NDT Project Manager Kia Hays said. "We're getting to a place where that's not going to be a reality anymore."
Currently on exhibition at Illinois Holocaust Museum and Education Center near Chicago, NDT allows visitors to have a virtual conversation with Gutter, who will appear through a faux 3-D presentation technique called a Pepper's ghost, an approach used in Disney's Haunted Mansion ride.
To start a conversation with Gutter, museum visitors hold a mouse's button down and speak into a nearby microphone. Gutter has prepared responses to thousands of questions about his life before, during and after the Holocaust. He even will tell a joke and sing songs, if you ask.
When you ask, "What do you remember from the Holocaust?" Gutter responds with a memory about a police officer who saved his life.
After being assigned to the policeman's work group, the official realized he knew Gutter's family. He was in need of a nurse for his wife, who was very sick, and Gutter fit the bill. Later, on the day many of the people in the camp were shot, the policeman took Gutter into the barracks and cleaned him up, adding his then-deceased wife's lipstick to Gutter's cheeks for a sense of life, and he survived.
Gutter is able to answer questions through ICT's natural language technology that interprets questions and finds the most relevant response from hours of recordings of possible answers. Playback technology enables him to seamlessly answer the question he was asked.
Conversing with Gutter is similar to using Siri on an Apple device. When the system doesn't recognize a question, or if you've not spoken clearly, a polite Gutter says, "Just repeat that." But instead of interacting with a computer, the experience is more human. Gutter's responses are authentic. They're just recorded with future dialogue in mind.
His account feels natural, like an in-person conversation. It isn't edited down and there are 15-20 second pauses where Gutter collects his thoughts. While his response to the question above is moving, it goes in a slightly different direction than anticipated, much like many in-person conversations do.
"If your question is not something that the system has seen before, but uses some of the same words, then you are likely to get a related response," ICT Director for Natural Language Research David Traum, said. "Very often these answers are very interesting in their own right, and move the conversation forward, suggesting new things to delve deeper into."
For the best results, it is important to ask questions that are as specific as possible.
The process of transforming Gutter's testimony from the interviews to its current state was lengthy. His testimony is the furthest along of a dozen survivors initially selected to participate in NDT from the Shoah Foundation's Visual History Archive. To narrow down the number of participants, factors like survivors' health, age and level of comfort sharing their stories were considered — aspects that would aid in the rigorous interview cycle.
Interviews were shot for five hours daily over the course of a week on ICT’s Light Stage, near USC's campus in South Los Angeles. Gutter answered thousands of questions from students, staff members and the public from the center of the stage's half-dome that's lined with a green-screen backdrop.
Answering hundreds of questions about what happened during the dark time of the Holocaust is exhausting, but, if given the chance, Hays thinks most participants would participate in the NDT interview process again.
"I think it's a very intense process, but [survivors] really see it as a part of their legacy that they're leaving behind," Hays said.
To ensure the recordings could be used with evolving technology in the future, the project was shot with around 100 cameras and aided with lights and microphones. The survivor's 3-D presence will be able to answer thousands of questions people will most likely ask.
It is tempting to refer to the finished product as a hologram or an avatar, but NDT is more of a time-offset interaction. Unlike Jon Hamm's role as a holographic version of Walter, who provides companionship and memory support to his widow in her last years of life with dementia in "Marjorie Prime," interactions with Gutter are with a previously recorded statement from a real person, not a computer, and there are no computer-generated elements in his responses.
While the data to create a hologram is there from the recorded interviews, for Traum, the best projection technique for NDT will be one that is possible to maintain and ensures survivors will appear naturally.
"We're not tied to one particular 3-D solution," he said. "The idea behind all the different camera recordings and different angles and different types of quality was so that when people figure out the best way to do it, the data's ready and they can apply that data either directly or through image processing techniques to get something reusable for whatever the 3-D of the future turns out to be."
After a pilot presentation of Gutter's testimony in Illinois, more than 95 percent of the visitors surveyed agreed that the technology enhanced their ability to connect with Gutter's story.
"It's not just some new cool gadget. It's something that has some value," Hays said. "All those people are really connecting and engaging with [the survivors]…. There's a sort of relationship that's being built between people when they interact, and I think that that's a very rewarding thing."
Gutter's great eye contact comes from the camera positioning during his interview. Because it feels lifelike, almost like a Skype call, kids often wave at him and wonder why he doesn't wave back. For the older crowd, Gutter's tales of the horrors he experienced often bring tears.
"This project is really preserving more than just the testimonies themselves," Traum said. "It's preserving a lot of the interactive experience that is just asking a survivor your own questions, and being able to take a little bit of control in terms of what you're learning rather than just passively watching."
The Illinois Holocaust museum exhibition marks the first time NDT has come out of the beta testing phase. In October, it will become a part of a pilot program in a New York museum. Before the end of 2017, there will be 16 testimonies total, including a testimony in Mandarin from a survivor of the 1937 Nanjing Massacre.
Eventually, the program hopes to place more testimonies in museums and classrooms to preserve survivors' legacies. Far after Gutter is gone, his 3-D image will be sitting in the armchair, waiting for the next question.
Top Image: Matus Stolov, a Holocaust survivor being filmed | Courtesy of USC Shoah Foundation