F E A T U R E S    Issue 2.04 - April 1996

No Strings Attached

By Matt Bacon

The boy's head looks up at me from the bench-top, wide eyes open, lips slightly pouting. His skin has the grain of finely carved wood. Beside his grapefruit-sized head lies a disembodied wooden arm, roughly jointed at wrist and elbow, its articulated hand attached to a rod draped in blue fabric. This is Pinocchio, the puppet who cast off his strings and came alive.

The most famous puppet in fiction is now starring in a new film, being shot at Bray. Except that, today, it isn't. The puppet has a problem, and so it has been sent home to have its head examined. Home is Jim Henson's Creature Shop in Camden - an organisation that sits in more or less the same relation to the late Jim Henson of Muppet fame as Industrial Light and Magic does to George Lucas. The Creature Shop uses animatronic technology to provide puppetry for films that need it, films like the new Pinocchio.

"Right, let's see what's the matter," says the pony-tailed technician poring over the "wooden" head. He loosens a cunningly hidden screw buried in the fibreglass quiff curling above the puppet's forehead, and prises the sides of its face apart. A quick wrench twists off the top of its head to reveal a mass of electronics, tiny motors and gears, all held in place by Velcro straps: "There's 24 movements in here," he says, rolling the boy's eyes up and down with a twirl of a screwdriver, "and we didn't have room to build a skull, so everything's attached to everything else."

It doesn't take long to diagnose the problem: the fan which blows cooling air across the tightly-packed mechanisms has failed. The new fan takes only a few minutes to fit, and the head is ready to return to the set at 6am the next morning. As the engineer delves inside the lack of skull, I sneak a look at the innards. Behind the latex face are tiny solenoids - tiny magnetic rams which move the skin from grin to grimace - and a few of the micro-motors you might find inside a radio-controlled model aircraft, which are there to curl the lips and roll the eyes.

But there's one capability the electro-mechanical endoskeleton beneath the flexible face lacks, and it's Pinocchio's most important feature. This puppet's nose is just a snub of silicone made to look like wood. "So how's his nose going to grow?" I ask. "Oh, that's just computer graphics," he answers, sniffily. The nose just isn't possible in silicone and animatronics - but it's quite possible in the computer. After all, what isn't? Toy Story has shown that whole films can be shot in the computer. And that, you would think, means the beginning of the end for "traditional" puppetry. Computers painting pixels faster and faster must inevitably replace these enormously complex confections of electronics, gears and engineering, musn't they? Puppetry - even high-tech animatronic puppetry - must be a dying art. It can't be long before silicone is superseded by silicon, and the puppeteers' creatures shut up shop.

But at the Creature Shop they don't see it like that. Because, for all its gadgets and gear, the Creature Shop is not really about puppets. It's about the performance that turns them into living characters. That's why the Shop's work on Pinocchio - the story of a mannequin coming to life - is so resonant. It doesn't matter whether the mannequin is wooden, as it is in the story, or silicone and metal, as it is in the shop, or a computer generated illusion, as it may be in the future. There is a wonder in bringing the inanimate to life that the Creature Shop understands full well - and whether the inanimate object is atoms or bits, the wonder remains.

The meanings of muppetry

A glance around the workshop shows the Creature Shop's record of success. On a nearby bench, nestled among boxes of servomechanisms and pots of latex, sits a gallant pig. Beside Babe, strangely still and silent, sits his friend Ferdinand the Duck. A Ninja-Turtle-like head is being covered in yellow fur to take on the role of the Honey Monster, and a handful of Dalmatian puppies - 4% of the total required in the next film's title - are being furnished with their distinctive spots.

At the moment, the Creature shop is doing a lot of business. Babe's Oscar nomination shows that it is achieving ever greater success. And, far from coincidentally, the place is on a technological roll. In the R&D lab, director Dave Housman is perfecting a new control system which will bring even greater subtlety to the shop's characters; upstairs the computer-generated imagery (CGI) team under Hal Bertram is putting the finishing touches to the monster from Loch Ness, a movie which, he hopes, will prove the value of putting virtual creatures into the delicate hands of highly-trained puppeteers.

Next to the workshop where Pinocchio sits in pieces, Housman's high-ceilinged ex-classroom is an engineer's dream. Racks full of every conceivable mechanism line the walls; Powerbooks rub shoulders with an Acorn Archimedes and a brand-new top of the line CAD workstation; drawers full of carefully sorted electronic components spill open; everywhere you look there is some sort of gizmo-in-progress. As I arrive, Housman and a colleague are discussing intently how they might get the eyes of one puppet to follow the movements of another: the conversation ranges over micro-mechanical iris diaphragms, infrared lasers, film sensitivity, ultrasonic trackers, Bang & Olufsen speakers, servoing and feedback loops. So I'm a little taken aback when Housman tells me, "You've got to remember, we don't care about the technology." When he continues, though, I begin to get the point. "It can be an Onyx or fishing line. What matters is to get the performance on screen."

Looking more like a suburban bank-manager - albeit one in a black T-shirt and jeans - than one of the most broadminded engineers in the country, Housman is quietly spoken. But his eyes take on an evangelist's gleam when he talks about per- formance. He's been the driving force behind the Creature Shop's electronics since Labyrinth, the 1986 fantasy epic that saw its establishment as a permanent creative force. But the new techno- logical relationship between puppet and perform- er started a long time before that. It began in the mid-'50s, when Jim Henson did his first TV work with the Muppets.

"In Punch and Judy shows or at theatres, puppeteers perform to an audience," Housman tells me. "The puppeteer is above, or to one side, or underneath. But on TV, puppeteers were watching the performance on a monitor; for the first time, they could see their performance exactly as the audience was seeing it." So a muppeting puppeteer is conscious of a whole performance, rather than figuring out which string to pull.

According to Mak Wilson, the Geordie behind Babe and many of the Creature Shop's other stars, "I'm watching it as though I'm not the operator, but someone watching the film."

The demands of TV - the fact that the shape of the screen more or less dictated that most shots were "head-and-shoulders" - shaped the classic Henson Muppet: one hand in the head to control the face and expression, the other moving the puppet's hands and arms with thin wire rods. Even now, with a five-metre Loch Ness monster head or a dragon that exists only inside a computer, the same rule applies. No matter how much more technology is involved, how many motors and microprocessors, data buses and digital artwork, that bodily metaphor remains in place: the head is one hand, the body's movements are the other.

Thinking this way, in terms of metaphors and meaning, you start to understand what a puppet isn't. Housman is clear on the matter: "We don't do robots," he says, leaning forward with zealous intensity to make sure I get this straight. "Robots are about increased productivity - and performance isn't that. Our puppets are deeply not robots. What we do is all about communication."

The difference, he says, is one of meaning. In a robot, you move everything around bit by bit with lots of controls; in a puppet you manipulate not machinery, but feelings. "You want to say 'smile' and have all the elements move to a smiling position. That gives you one control signal with a high semantic content." One thing you notice looking at Babe's animals is just how thin the control cables are. Although there are over a hundred movements under Babe's latex skin, a single umbilical with a five- pin connector carries all the control signals. The amount of information compression that concentrating on semantic control allows is extraordinary - if you have skilled designers and skilled puppeteers.

"In a face," Housman explains, "there are 100 muscles, each of which could have 100 positions. That's an astronomical number of possible combinations - but most of them are meaningless. The answer is to define an expression space and make sure the controls stay within that space. Most people have only half a dozen or so favourite expressions - surprise, pleasure, disgust, for example - and the stronger the character you're dealing with, the fewer and more extreme those expressions are. So if you define those expressions as the boundaries of your expression space, and drive the character within the boundaries, you should have access to all the expressions you want, and lock out all the meaningless combinations."

And we know it works: most of us manage to control our own faces - give or take the occasional involuntary giggle. "Obviously it's possible to hand control of a face over to one person, no matter how many move-ments they have to master. You do it all the time. The question is how to make such complex controls intuitive. What we do is take the natural mappings between body movements, communication and emotion, and put a control there."

"What I'm here to do is build a natural user interface for performers. I try to get the technology out of the way. The problem is that electronics engineers and computer scientists have a devoutly reductionist view, which is deeply inimical to performance, which spills out of people spontaneously. Spontaneous people are terribly vulnerable to reductionists: the easiest way to destroy a performance instantly is to ask the performer to explain how they are doing it. A pianist doesn't think about what notes they're pushing, just about the performance. That's all I want our puppeteers to have to think about."

Essentially, what puppeteers are good at is mapping emotions to their hands. "When I'm puppeteering," says Mak, "I'm acting. I talk with my eyebrows, my blinks and so on. And my hands are subconsciously transferring the movements of my face to the puppet's face." Housman's achievement has been to build a standard toolkit - a control rig which still shows its roots in Jim Henson's original TV Muppets - which makes that transfer easy. The right hand fits into a skeletal mechanical mitten on a two-axis turntable, which takes the place of the puppet's head. The effect is identical to having your hand in a moulded creature's cranium. The left hand grasps the "big stick", a joystick that gives each finger a trigger button and offers the thumb a miniature two-axis joystick. The big stick is the guide through which puppeteers navigate their creature's expression space. Some use the main joystick axis for physical movements and the thumb joystick for emotions, others the reverse. The trigger buttons can act as subtly variable potentiometers of emotion.

The standard rig allows the performers to concentrate on performance and ignore technology. Inside Babe there are approximately 25 motors, or, more accurately, electric muscles: 16 in its nose and head, 3 in its neck and 6 in its body, which allow the puppet to move its tail, wiggle its bum, shift its shoulders, cock its head, and look up, down and around. But the performers do not need to worry about the increasing complexity of the media they perform through; they just need to get the message across.

The puppeteer doesn't feel the technology - but the illusion depends on it, and never more so than in Babe. While no one has seen the Loch Ness monster, almost everyone has seen a pig. So Babe had to be seriously realistic, all the more so because the film was relying on a combination of trained pigs, animatronics and computer graphics, often in the same shot. That meant the emotional space of the puppeteer's hand had to be almost exactly the same shape as a piglet's head.

"Most of our creatures are somewhat anthropomorphic," comments Dave Housman. "They conform to human rules of expression. Eyes pose a really tricky problem - to you and me, eyes on the front of the head equals person; eyes on the side equals lunch. Characters have to look at you. With Babe, we tried bringing the eyes forward and round, but then of course you couldn't cut to the real creature. In the end, we had to rely on the fact that pigs' eyes are almost completely black, so you can't see where they're actually looking." The Shop's model-makers also burned the midnight oil re-inventing the silicone rubber technology that creates realistic animal skins. Every one of Babe's 15,000 bristles had to be inserted by hand. The result is an utterly convincing pig. This pig, though, also has to talk. And talking is a complex activity. You and I move 40 or more muscles in the process of shaping letters. Even an animatronic pig has to have at least 16 electric muscles - 10 to move its mouth and lips and 6 to move its snout. The Creature Shop's puppeteers have highly-honed lip-synching skills, but the com-plexity of the lip movements needed to make the very realistic Babe speak convincingly would have soaked up all of the puppeteers' concentration, leaving the pig's body lifeless and still.

So Housman added a new wrinkle to the technology which seems, at first sight, to run directly counter to the Creature Shop's belief in spontaneity: a way of recording the performance. Along with Hal Bertram, the computer specialist, Housman developed the performance editor - essentially a multi-track digital recorder with a suite of editing tools. The editor records the semantic signals sent by a puppeteer, charting the character's course through expression space.

The idea was to let the puppeteer separate out two aspects of the performance. "What we did was to record the word sequences before the performance. Then, when the puppet is triggered to speak a line, the controls switch. Instead of moving the puppet directly, the control rig modulates the parameters of the recorded sequence, changing the timing and delivery, the depth of the movement an so on. When the line is finished, the controls switch smoothly back to direct action."

The performance editor adds another degree of freedom to the interaction between puppeteer and puppet. On screen, it looks like a conventional digital sound-editing program, with all the usual functions of cutting and pasting and layering. But it can mix and match meanings. In a 3D physical space, the performance editor would be limited - a lot of left layered with a bit of right would just result in a bit less left. But in multi-dimensional expression space, a lot of disgust layered with a bit of interest produces something quite unlike either.

Back in the box

When I go upstairs to Hal Bertram's CGI studio, the first thing I notice is a control rig just like those in Dave Housman's workshop. The second thing I notice is a slim, middle-aged woman contorted on the floor holding a familiar plush and velvet cartoon cat high above her head in front of the eye of a video camera. She watches the cat on a large monitor intensely. So do I. Because it's the cat with the hat on the screen, not the woman on the floor, that you look at when she speaks. The cat is basically a sock - a beautifully made sock, with some nice foam add-ons and a bit of fur, but a sock nevertheless. But with a twist of her hand, raising an eyebrow and curling the cat's mouth into a sneer, she brings it to life.

Later, I watch a recording of Jim Henson and Kermit on The Tonight Show with the same feeling of wonder. Henson talks, Kermit an inanimate green glove on his hand: and then the frog interrupts, his only action to turn a quizzical head and move one arm. I can see the wire attached to his hand; Henson makes no attempt to stop his lips moving, but all of a sudden it's the puppet I'm looking at. It doesn't take much: just as Picasso could convey the substance of a whole body with a single line, Kermit is brought to life by a few subtle movements.

Back on the TV screen in the CGI studio, though, something new is being added - a surreal painted backdrop being created by another artist gazing at a computer monitor as we watch. This is where the origins and the possible future of the Creature Shop collide.

It's a very practical, even-handed collision. There is no digital euphoria, just a recognition that the new technology offers new possibilities. Despite his vested interest as head of computer graphics, Hal Bertram points to the advantages of older technology as often as he sings the praises of the new. "You get a lot of things for free with an animatronic on set. The lighting will always match perfectly and" - Bertram points at the screen showing footage from Loch Ness - "you have things like reflections from water ripples. We wouldn't begin to try to do that in CGI. But CGI is cheaper, and of course if you have several creatures, you can duplicate a computer model with one mouse click for free, instead of building another animatronic which costs about as much as the first."

And with CGI you can do things that puppets just can't: fly, run long distances, enlarge noses in a quite unreasonable manner. You don't have to worry about the puppet's size; it can be as big as you like, or as small. The Creature Shop's first CGI movie - Muppet Treasure Island - stars talking mosquitoes. Bertram's job is to harness those abilities, but in a form that works for the Creature Shop, with its emphasis on performance. Just as the puppeteers communicate with their animatronic alter egos using signals with the maximum sem-antic content, so they do with the products of the CGI studio. "What we're doing is storing and applying performances to our models, rather than just capturing motion," he explains.

A computer creature begins life just like an animatronic: a series of sketches by a character designer. The sketches then form the basis of a sculpted model; Bertram and his team have yet to find a 3D modelling program that allows the freedom of expression and creation that modelling clay gives a sculptor. The physical model, usually crafted in plaster and then cast in resin, is digitised in three dimensions using a laser scanner.

Bertram is cagey about exactly how the animation is done; "We get the model into the computer, teach it how to move, puppeteer it and then render it," is all he'll really say on the subject. But the process involves creating a skeleton inside the 3D surface, putting constraints on the movements of that skeleton's joints and elements, and linking movements of the skeleton to changes in the 3D shape of the skin - and in this it's not unlike designing one of the real world animatronics. The computerised joints have to allow the CGI model to articulate itself in the same way as the mechanical joints of the hardware version do - even if one skeleton consists of strings of numbers in a Silicon Graphics workstation while the other is made out of universal joints "liberated" from a ten-year-old Land Rover.

By incorporating a degree of intelligence and some constraints into the computer model, Bertram helps the puppeteer concentrate on the performance. "An animatronic has to obey physical laws, because it is physical," he says, "but a computer model doesn't have to. You have to build in some of those constraints so the puppeteers know what they're doing, and feel comfortable with the creature." Mak agrees. "I love working with animatronics. But when I first worked with CGI, I felt liberated. You can do anything you want. In fact, half the problem is stopping yourself."

In general, the puppeteers make the transition easily - because the technologists have set it up that way. Watching the CGI puppet on a monitor is little different from watching an animatronic. The controls are exactly the same as those for animatronic creatures.

"Years of speaking with a hand produces a strong mapping," explains Bertram, "and you can't re-map a puppeteer's brain every time a new creature comes along." Besides, in an increasing number of cases, a movie character will have both a CGI avatar and an animatronic, which will be used at different points in the film. The puppeteers move back and forth happily, their creations transcending the changes in the toolkit they are using. It's the creation of the creature that counts.

In most films, animatronics would be handled by one specialist house, models by another and CGI by a third. But this is as alien to the Creature Shop philosophy as the laborious frame-by-frame approach of traditional animation would be. "The nature of these characters is not immanent in their shape, but in their perform- ance," says Dave Housman.

The practical effect of this concern is that the character of both the computer graphic and the animatronic has to be created by the same person: the puppeteer. And that character is, as much as possible, created on the fly, right here, right now. Which brings its own advantages.

Traditionally (inasmuch as an industry less than 15 years old can be said to have traditions) CGI is a post-production business. The talent rushes around on set, acting terrified of a man standing on a ladder, and two weeks later some rough cuts appear with the ladder replaced by a digital dragon. But the Creature Shop believes that its puppeteers need to perform to the actors, and the actors need to see the characters to create a believable performance. Until recently, that was not possible in CGI. Now, though, by using a speedy Silicon Graphics box and keeping the resolution reasonably low, the computer models of characters can interpret the semantic signals from the puppeteers fast enough for them to appear live on screen. Playback on an off-stage monitor allows actors, puppeteers and director to see the scene playing pretty much as it will on the final print. Combine that with the performance editor's tricks and you've got computer-generated characters as believable and biddable as most human actors (and a lot more than some). Puppetry is not replaced by computer animation; it enhances computer animation.

When I arrived at the Creature Shop, I thought that the obvious question was: "Will computer graphics replace animatronics?" Now, having spent time with Dave Housman and Hal Bertram, I realise that the Creature Shop just doesn't care. Its products are not animatronics, or computer graphics, or hand puppets: they are characters. Whether it's a puppeteer picking up a limp green lump of felt and foam to create an exasperated, wisecracking frog, or a team using the whole panoply of mechanical and digital techniques now available to get a pig to talk, it's about bringing the inanimate to life. That process is what the Shop sells.

Lying on the bench-top, a confection of silicone, resin, electronics and actuators, there's no doubt that this evening, Pinocchio is a puppet. But tomorrow morning, when Mak the puppeteer puts his hand in the mitten, twirls the joystick and triggers a blink, a yawn and a cheery whistle, that wooden boy will be alive and awake. It's that simple. No strings attached.

"Eyes are really tricky - to you and me, eyes on the front of the head equals person; eyes on the side equals lunch."

Sitting on an engineer's bench, this metallic skeleton doesn't look much like a window onto Babe's soul, but build it into a hyper-realistic face, add the skilled fingers of a puppeteer, and those big black eyes will bring a lump to even the most hardened carnivore's throat.

Pinocchio may have cast off his strings, but behind the wood-grain effect are electronics and controls that create expression out of bits streaming through the multiplexed cables driving his head. His nose, thanks to silicone's limited stretchiness, stays stubbornly short until the CGI studio gets hold of it.

Matt Bacon (mattb@london.virgin.net) is executive producer for news and information at Virgin.Net. He tries hard to forget an unfortunate incident with a zippo lighter, a can of hairspray and his cuddly Kermit that took place ten years ago.