They are well-known in Holland, the old-fashioned picture frames with holes you can stick your head in.

For the Dutch Film Festival in September 2015, Hoog+Diep, commissioned by Tree-ON, developed the installation “Jij bent de regisseur” (“You are the director”) where you can place yourself in a movie and control the movie’s story. A centuries-old concept, in a high-tech remix.

Passers-by can sit in a directors chair across a large screen and use an iPad to control what will happen after every scene. Better even, the spectators and the director are composited into the movie in real-time, together with the pre-recorded actors.

As the director and the audience perform their personal movie, everything is recorded. When the movie is done, the director can enter an e-mail address. A few minutes later a YouTube link of the personal movie on YouTube is delivered to his or her inbox.

This installation ran for two weeks in the busiest walkway of shopping mall Hoog Catharijne in Utrecht. This made for a large group of newly aspiring actors who stole the show together with well-known Dutch actors Gijs Naber and Sigrid ten Napel:

Technology
The hardware setup consists of an iPad, a professional SDI camera, a BlackMagic Intensity Extreme capture device, an Apple Mac Pro and a 90 inch LCD screen.

The camera input is being captured by the Mac Pro in HD and streamed to a hardware-accelerated texture on the graphics card. There, HD footage with alpha channel (in ProRes 4444 codec) is blended over the live camera image.

To produce this footage, actors have been recorded in a green-screen studio, animated computer graphics were added but without completely filling the background. This way, much of the original green remains open, where the live camera image can be seen through in the final installation.

The Mac Pro runs a local web server that serves a dynamic web page to the iPad, which translates the state of the system to a fullscreen touch interface and influences that state with the choices made by the director. We developed this interface and the interaction with standard web technology, but with custom server-side links to the video engine.

As soon as the video engine sees that a scene has finished, the last choice that was received via the webserver is being checked to determine which video will be played next. Since all previous choices have an effect on the next shots, we need many separate movie files in which all options are present. For 5 choices, we need 2 to the power of 5 = 32 separate scene options.

When the interactive movie is final and the e-mail address is entered (optionally of course), the recording is added to a queue, which compresses all recordings serially to a h264 format with limited bandwidth suitable for a relatively quick upload (between 100 and 150 MB for 2-3 minutes of video). Another queue uploads the compressed videos to YouTube and sends the link to an online server which generates a graphically designed email and sends it. All this happens in the background, while the new recordings are already underway!

The biggest challenge with all this was to make everything happen simultaneously on 1 computer. That is:

• Real-time blending of camera image and movie at HD resolution
• Recording the blended result to disk
• Compress the recordings
• Upload the compressed recordings to YouTube
• Email links
• Communicate and dynamically control the user interface on the iPad

Obviously this should all happen without ever stalling any of the processes and making sure the video always plays without hiccups. The only way to make this happen on a single computer was by spreading the calculations over separate cores and the graphics card of a Mac Pro.

We believe the whole system worked like a charm!