Q&A with Softkinetic bigs Michel Tombroff and Mike Nichols

Softkinetic
, which has recently plucked Mike Nichols from his perch as Project Natal Executive Producer for Microsoft, is at the forefront of 3D gesture recognition.

‘What the heck is that’ you ask? Well, it’s potentially the biggest thing to come to home entertainment since DVD.

Imagine never needing the remote again. Imagine actually interacting with your home electronics — your TV, your video game systems — rather than pushing buttons all over the place. Imagine you’re Tom Cruise in Minority Report.

OK, well, don’t really do that. But the idea is similar and certainly a Science-Fiction dream that Softkinetic is making a reality. Even the Adult Entertainment industry is curious about how ‘immersive’ home entertainment can become using Softkinetic’s technology.

I got the chance to speak with Michel Tombroff (CEO) and Mike Nichols (Studio Head, Softkinetic Studios) about this far out idea that may soon be a part of your home (and won’t break your checking account either).

Vitka: Describe Softkinetic’s work to me.

[Michel] Softkinetic develops gesture recognition software for 3D cameras. Our software uses the input of a single, low cost 3D camera to identify the user(s) and recognize their body parts and gestures, in real-time. Using our software solutions, companies can bring natural, gesture-based interfaces to transform the way people interact with electronic devices – video games, television, personal computers, digital signage systems, fitness and rehabilitation solutions, etc. Our solution supports all 3D cameras available on the market today, and can be used on any platform (PC, set top box, console, embedded system, etc.)

Vitka: Gesture recognition technology – the ability to control your TV or other electronic devices with certain hand movements – at first sounded like something ridiculous out of a Phil Dick novel. But, it’has become the bleeding edge of home tech. Everyone’s talking about it. What do you credit with bringing this sci-fi idea to the consumer?

[Michel] It took indeed a little while for people to accept that gesture-based interfaces are a good candidate for controlling and navigating electronic devices. There are two reasons for that. First, the solution has to be robust, reliable, and perform well – three parameters that have now been achieved thanks to the progress made by the various 3D camera manufacturers and Softkinetic. Second, because the technology is now being adopted by all ecosystem players – semiconductor companies (see our work with Texas Instruments), set top box manufacturers, PC manufacturers (see Quanta’s recent investment in Canesta), telecommunications services providers (see our work with Orange and Belgacom), etc. In other words, a critical mass of industrial players have now decided to bring this technology to market, which drives overall growth in said market.

[Mike] – The games industry has always worked on the bleeding edge of technology. Let’s face it: You don’t need a faster computer to run a spreadsheet program. We have always pushed technology to enhance the user experience. In developing Natural Gesture Input experiences, that is, controlling devices using only your body without the need of a physical remote device, you have to give careful thought about the user experience.

The sci-fi aspect of this new technology comes up a lot. We grew up with a very complicated vision of technology through sci-fi, and there are issues there we’re trying to avoid. Many times in sci-fi novels, and certainly in Dick’s own writing, technology is a barrier to humanity. We grew up with the idea that technology needs to be needlessly over engineered. But in practice we strive for the exact opposite. That is to make technology accessible, helpful, and even friendly. For technology to really be adopted by popular culture, experiences need to be designed this way.

The technology behind 3D cameras has been in development for some time, but it wasn’t until we defined what the user experience should be like that it really took off. For me the key was to put the user first and not to create technology in a bubble. Design experiences you want users to have, and shape the technology around that. You could have the best, most advanced technology in the world, but if the experience doesn’t allow you to connect emotionally with the content, it will fail.

Vitka: Can you describe an average setup using Softkinetic technology in the home?

[Michel] A RGB-Z (3D + 2D) camera is attached to your TV set. When it detects your presence, it starts by presenting you the information relevant to your profile. You can browse the system using hand gestures, zoom on specific information, click on items of interest, move images on the screen, etc. With hand gestures, you can navigate and control the whole TV experience, selecting movies and programs. The set-top box also allows you to launch gesture-based video games – sports, dance, family and casual games. You can then also launch a video conference with your friends that are currently connected on the Internet, and, using the 3D capability of the camera, remove the background from the image and replace it with an image of your choice.

In another room, a 3D camera could be used to help you control your home automation system – adjusting temperature, lighting, security systems, etc.

[Mike] The key is that there is no complicated setup from a user perspective. Softkinetic has developed its gesture recognition software to feel natural and intuitive. From the very first time you use it, the navigation just feels right. For me, it’s hard to use natural gesture input in the office, then have to go home and pick up a remote; they feel so archaic now.

Vitka: And the dreaded question: how much do you anticipate that the setup would cost the consumer?

[Michel] It is expected that cameras will cost less than $100 from 2011 onwards. The cost of the other equipment will would vary by manufacturer, and we cannot anticipate that cost.

Vitka: How does your 3D camera work, exactly?

[Michel] Our 3D camera, “Optricam,” works using the “time-of-flight” mechanism. The camera emits infrared light to the scene, and then collects the light that comes back after having been reflected by the objects in the scene (people, furniture, etc.) For each pixel, the camera calculates the “time-of-flight” of the light wave and, from that, the distance of the point from the camera. This technique is very robust, can deal with many ambient light conditions, and provides a very accurate measurement of each pixel. The camera also includes an RGB sensor, so that it produces both a “depth” image and a color image (e.g. for video conferencing), and a set of microphones, for audio capture and control.

[Mike] Think of it like how a bat uses its sonar to navigate. Even in total darkness the Optricam is able to see all the same details as if it were daylight. The data you get from a time of flight 3D camera is incredibly detailed. It allows us to precisely calculate movement and features within its view. What that means to the user is a more robust and fulfilling experience.

Vitka: Now, how can I, as the consumer, watch my favorite movie on my television using Softkinetic’s technology? Walk me through the motions of getting Kurt Russell and crazy space monsters on the screen without hitting buttons on a remote control or other device.

[Michel] Most of the time, the user is simply watching a program – and the camera is therefore in stand-by mode. By performing a specific gesture (“raising your hand” for instance), the user then takes control of TV’s user interface, and can then navigate and control the TV’s experience (selecting channels, pausing movie, raising volume, etc.) The TV UI presents a rich mosaic of menus that can be navigated through hand gestures, or through voice activated control. An intuitive, easy to use point-and-click and keyboard environment allows you to make selection based on keywords, or to type specific keywords.

[Mike] From a users perspective, navigating using natural gesture is so much more intuitive than using a remote. If I want to change the channel, I just swipe my hand through the air. I don’t have to look for the remote, position my fingers on the correct button, point the device in the direction of the TV, then select the button. And that’s after you’ve already spent hours programming your remote to talk to all your devices; the TV, cable box, receiver, etc. It’s so much easier to simply point at what you want.

Vitka: Mike, your hiring away from Project Natal definitely brings gaming into focus for Softkinetic. Are games going to become a major part of Softkinetic’s plan?

Will they be more ‘casual’ or ‘hardcore’? Are there any specific games projects you’re working on now that you can tell me about? Most importantly: is Microsoft annoyed with you now?

[Mike] – Softkinetic Studios was formed to create all types of entertainment and UI for natural gesture input. We don’t think of ourselves as traditional video game developers. This technology is enabling us to create new categories of interactive entertainment. You can certainly categorize the experiences as both “hardcore” and “casual”, but our focus is to bring ALL types of entertainment to a broad audience. Traditional controllers have been a barrier of entry for most people. Many new to gaming feel alienated because controller input schemes have become too complicated. It’s hard to enjoy a game when you’re struggling to learn how to control it. This technology enables us to have control over our interactive entertainment in ways that are natural and intuitive. The experiences we’re able to create with this technology makes you feel truly connected and the enjoyment is instantaneous.

The great thing about Softkinetic Studios is that we are not confined to one platform. Using Softkinetic’s iisu we’re able to bring Natural Gesture Input experiences to a variety of platforms including arcades, to set top boxes, PCs, consoles, and more. This really gives us the ability to branch out, to cater the types of experiences based on the platform. We’re focused and passionate about developing for natural gesture input. The team has been busy creating a new game, Silhouette, which is first game of its kind that combines music and rhythm. Players’ own body movements control their onscreen avatars, and they must get into specific physical poses to match ever-changing onscreen shapes. It’s a total riot! Silhouette, by the way, has been licensed for use in arcades; you should be seeing the game in the middle of this summer, and we’re still working with publishers to have the game available on other platforms. The team has also been developing new concepts and prototypes, but we’re not at liberty to announce any new projects just yet (but check with us closer to E3).

My former colleagues at Microsoft couldn’t be more supportive of me really. I have the upmost respect for the teams, and what they’ve been able to accomplish. We really all share a common bond and are passionate about this technology and the experiences being developed for it. We are all in the business because we love it. I’ve had the great privilege to work alongside some of the most brilliant and inspired people I’ve ever met.

Vitka: One of the most interesting uses of Softkinetic’s technology is for rehabilitation. (I myself work with a charity called Blue Redefined [blueredefined.org] that aims to help the disabled by way of video games). Did your 3D motion tech just cry out for use in rehabilitation, or was there another driving force? How did you get involved?

[Michel] Fitness and rehabilitation have always been markets where we knew this technology would have a great impact. The opportunity presented itself when Silverfit, a leading rehabilitation specialist company based in The Netherlands, decided to build a rehabilitation solution using our technology. At the same time, iTech Fitness, an innovating fitness group, also decided to develop prototypes for the fitness market. As the market for 3D cameras is now starting to dramatically accelerate, we expect many such solutions to appear on the market, and we are currently in discussion with several top-tier medical systems and exercise companies.

[Mike] One of the great benefits of this technology is that it allows us to create programs that can really improve your real-world skills. In rehabilitation it’s important that your movements be precisely monitored so you don’t relapse. With this technology we’re able to see the position of your entire body, and adjust a regiment based on your unique ability. It’s like having a personal trainer in the comfort of your home anytime you want.

Vitka: What devices can one currently use Softkinetic tech with?

[Michel] Our software supports personal computers, set-top-boxes based on Intel’s Atom processor family, and any devices that embed the Texas Instruments (TI) OMAP processor. Our camera supports the same platforms, as well as the TI DM365 processor.

Vitka: What devices will be compatible in the future (actual and anticipated)?

[Michel] We plan to adapt our software to a wide range of processing platforms, including chipsets used in embedded systems (industrial, medical, handheld, etc.).

Vitka: And finally, I feel strangely obliged to ask if there’s anyone in the adult entertainment industry who’s approached you on the possibility of working together, as the adult entertainment industry has often been on the leading edge of adopting new tech.

[Michel] Yes, we have been approached by adult entertainment companies to discuss about how to use 3D camera technology for offering customers “immersive, natural, gesture-based” adult experience. As the market of 3D cameras matures and becomes widely available, this is certainly an industry that will leverage this technology. We are content-agnostic, and our tracking solution would certainly be very useful for such types of content!