Siggraph 2001: Conference Abstracts and Applications (Sketches & Applications p.178)

Elmo's World: Digital Puppetry on Sesame Street

Emre Yilmaz
www.digitalpuppetry.com
PO Box 460022, San Francisco, CA 94146-0022
emre at digitalpuppetry dot com

Collaborators:
Sesame Workshop
Dave Satin, Les Rudner
Eric Gregory (Protozoa)




Recently, we received an unusual but exciting request: to make a drawer bark like a dog, to make a TV set cry like a baby, and to do it all live on set with Elmo.  Elmo's World is a daily 15-minute show-within-a-show on "Sesame Street." In his imaginary world, the popular red Muppet monster learns about a new concept each day (for instance, dogs, babies, or bananas). Among Elmo's friends in this world are four pieces of furniture. We bring these furniture creatures to life using real-time computer animation, or "digital puppetry." Specifically, we use an experimental motion-capture method.

The Need

As we worked live on set, our furniture characters were improvising their performances alongside the real puppets. Sesame Street Executive Producer Arlene Sherman further requested that the furniture not only be able to do basic functions, like opening and closing a drawer, but also act like living, thinking beings with personalities and quirks.  The idea is that Elmo has drawn these creatures, but they have come to life, and they are not always cooperative. The window shade may be more interested in tickling Elmo than in opening up for him. The TV set may try to come to Elmo, but waddle and fall over like a baby unsuccessfully taking its first steps.

Digital Puppetry

The models were relatively easy to make, but finding a suitable performance method was difficult. We decided to try an experimental method for puppeteering the characters, using foam rubber blocks outfitted with magnetic motion- capture sensors. If a puppeteer twisted one foam rubber block, the table would twist. If they hopped the foam up and down across the floor, the door character would hop up and down across Elmo's floor. We can't use the captured motion quite straight; we do a lot of real-time modifications on the data to fit the character’s proportions and position in the room. In addition to the foam rubber blocks and motion sensors, we provided foot pedals, joysticks, and sliders so the puppeteers could augment the performances. For instance, one foot pedal opened and closed the drawer.  That way, one puppeteer could both make the drawer walk around and make its "mouth" open and close.

We felt this would be a very puppeteer- friendly approach, because it allows intuitive control and a great range of motion. The puppeteers could actually grab and manipulate the drawer, the TV set, and the door directly, just as if they were real puppets, watching the live video feedback to see how their characters were moving.

Production

During taping, the digital puppeteers watch their characters interacting live onscreen with Elmo. The camera sees Elmo in front of the real set, and a quick live composite allows the puppeteers and the director to see how the furniture will look in this shot. What they are seeing is very close to the final result. The puppeteering performance data are captured from the foam rubber blocks and other devices for later playback. Keeping the motion data, and waiting till
post-production to commit it to videotape, allowed fixes as necessary.

Three of Sesame Street's regular Muppet performers enact the puppet furniture. The real Muppet performers are highly accomplished at bringing character to inanimate objects. They are also very experienced at interacting with Elmo! Many of the best interactions are improvised live on set. We wanted to enable them to bring all their experience and talent, including improvisation, to the digital characters as seamlessly as possible.


Note: We originated this project at Protozoa (r.i.p.)  Protozoa also created the ALIVE custom software we used on the project.

Elmo and other characters are Copyright © Henson Associates and CTW (Childrens' Television Workshop).