The development of sensitive surfaces was motivated by a search for alternatives to the screen and the promotion of joint content exploration. The capacity sensor technology used in the multi-touch tables, and later, in amorphous sensitive surfaces, is an in-house development that has been employed in many installations.
In the late 90s, when most interactive systems were operated manually by hand via a classic single-user interface such as a keyboard, mouse or joystick, ART+COM was already developing multiple-user interfaces that enabled direct physical interaction with virtual objects.
Sensitive floors
The project Bodymover (2000) marks the beginning of ART+COM’s exploration of sensitive floors. It enabled users to move around the space freely to interact with not just the system, but also with each other through the system. The physicality in the use of space that visitors were encouraged to engage in stood in stark contrast to the more sedentary modes of interaction of the time and anticipated the motion-intensive console games of the present. The entire body became an interface: visitors generated light silhouettes around themselves on a projection floor. With fast movements of their arms and legs users were able to cast light beams from their ‘auras’. When the light beams met objects positioned around the space, tones sounded. The result of this playful process was a permanent audio-visual happening shaped by visitors and their movements. The installation had a floor that seemed to visitors to be sensitive and interactive; technically, however, that was not the case. Visitors were recorded by a camera mounted on the ceiling, and their outlines, calculated by an image detection algorithm, were projected around them in real time as moving ‘auras’.
The first ‘true’ sensitive ground with integrated sensors was realised with Duality (2006). The input and output interfaces in Duality were the same: one four by four metre area that produced light waves when trodden on by passers-by. The system was based on load cells, four of which were attached to the corners of each floor panel, which could sense feet positions and step energy. There was a grid of white LEDs under the sandblasted glass panels that produced virtual waves based on the values measured.
Between the temporary installation Bodymover, shown at Expo 2000 in Hannover, and the permanent installation Duality, ART+COM created Grouse Experience (2002), the first permanent sensitive floor in the world, as a part of an interactive room. Grouse Experience is based on the same technology — camera tracking — as Bodymover, but with the addition of microphones in the four corners of the floor to measure the loudness of footsteps. Interactive graphics on the floor reacted to these two data streams — the tracking of image and sound. For example, to break a virtual sheet of ice, visitors had to stomp noisily. Light, soft steps had no effect — only once a certain volume was reached could the ice be broken and the virtual liquid underneath released.
Sensitive tables, objects and installations
In the early 2000s, to escape the limitations of the screen, ART+COM researched technologies that made it possible for sensitive surfaces of any size and shape to be produced, and developed the first touch sensitive tables. Behind the Lines (2003) was a single user/single touch media table that worked by means of pressure sensitive lip switches hidden beneath the surface. All media tables that followed used capacity sensors, which can precisely capture multiple, simultaneous surface touches. Users can interact with dynamic image and text information projected onto the surface from above by touch. As a cultural-historical place of communication and exchange, the table intuitively prompts communal interaction.
The first multi-touch table of this type was floating.numbers (2004) with an eighteen square metre surface, which could be explored by users in tandem with one-another. From a stream of virtual numbers flowing across the table, individual numbers continuously emerged and changed colour temporarily. This visual signal communicated that users could, at that moment, interact with that particular number. The points that were to be sensitive were set in the table during the design process so that the content could be integrated with them, and capacity sensors, invisible to the user, were installed there only. From a design perspective, it was perfectly adequate to work with sensitive areas distributed across the table. This way, user interaction could be focused on several areas instead of extending over the entire table, in which case the size of the table would have made joint exploration and interaction more difficult to achieve. And this solution had the economic advantage of not needing the entire surface to be equipped with sensors.
The second version of the installation, which was designed in 2005 for an exhibition in Taiwan, ‘folded’ up on the horizontal. Capacity sensors were installed on the horizontal plane only, and figures flowed from the top down where they could be interactively explored. There were significantly fewer sensitive areas in comparison with the first version, partly due to cost considerations. In both installations, horizontal lines were used in the design of the stream of numbers in order to avoid edge blending issues, the solving of which would have exceeded budgetary limits.
The capacity sensor technology used in the multi-touch tables, and later, in amorphous sensitive surfaces, is an in-house development. Over the course of years it has been continuously refined and has been used in many installations, in documenta mobil (2005) for example, and in the O2 Sculpture (2005), a combination of sensitive floor and table, and the media table Salt Worldwide (2010).
Two kinds of capacity sensors were developed at ART+COM. In addition to plate-based capacity sensors, we worked with raster-scanning: a grid of wires under the surface detects touch at the intersection points. The advantage of this system was being able to cost-effectively sensitise large surfaces and, in tandem with Multi-Touch, enabled hand-gestures such as swiping and dragging to be used for moving and zooming. However only flat surfaces could be so equipped.
An installation that combines a sensitive object and a multi-touch table is The Player (2007): visitors can touch a life-sized, stylized figure of a football player at different points to learn what violations are typical for footballers at those spots. The image and text information that then appears on the table surface can be engaged with further.
In contrast to raster-scanning, plate-based capacity sensors offer significantly more design freedom, enabling even amorphous objects to be made sensitive. This creative freedom resulted in installations such as Die Quelle, which inspires associative play on the theme ‘sustainability’ in the exhibition Level Green (2009). Virtual water flows down a sculptural object; phrases associated with the concept of sustainability emerge from the water. The surface, milled from a single piece, was designed as a relief inspired by the rocky ground of a mountain spring. The exact form of the relief was derived from the same water simulation that was programmed for the projected virtual water. Under the milled surface were capacity sensors that enabled visitor interaction with the water and the phrases.
In recent years flat, touch-sensitive multi-touch surfaces have become part of everyday life: in consumer electronics, tablets and mobiles, as well as in museums in the form of interactive exhibits. What is standard now was still uniquely fascinating in the mid-2000s, and even needed instructions. Exhibition visitors had to be motivated to engage with the virtual content. This was often achieved by museum staff taking the lead, so that visitors would immediately understand the basic interaction principle and could then virtually ‘pass on’ their interactions to those that followed. In the current era of ubiquitous touch-sensitive surfaces, the opposite is required, hence the ‘Please do not touch’ signs increasingly encountered in exhibitions.