a multisensory instrument for design optimization

Joe Juhász & Bob Flanagan

12/15/2003

This research is intended to improve the design interface available to tomorrow's decision maker.The weakness today in the technology of digital design is in the interface of the hand and the eye, and the media they employ.  The present interface is hopeless dated into the 1950 's:  it is little removed from its origins as a typewriter, a TV, and a remote control. 

The technology that is currently under development at the leading edge of today 's science and technology is a multidimensional interface that increases environmental awareness through its multisensory capabilities.  The companies that we have contacted that are developing the interface technology are BARCO, DISCREET and Digital ArtForms.  The fundamental technology is under development by HP SONY and CANON. 

Our objective is not to write any software other than simple scripts that allow otherwise unrelated programs (written for different purposes) to work together.  The hardware that we will be using is also off-the-shelf cutting-edge technology.

The goal of this research, then, is twofold: First, to overcome inherent dimensional limitations presented by the predominant technologies employed in today 's computer-user interface: Second, to develop a dimensionally expanded design interface, an optimized decision making environment of integrated multi-sensory and time-based capabilities.

In short, we are pursuing the development of an expanded time-based design interface (x,y,z+t) capable of extending the present visual display into a multimodal and multi-sensory dimension.  We see an amazing opportunity in combining such an expanded display with 3-D visualization hardware and software that already exists.  We are convinced from our research that several other groups are working on a similar expansion of the interface between user and computer.  We are definitely at a point in time where someone is going to get something like this going.

At present, for example there is in existence a 3-D simulation environment, run by a supercomputer that places an interactive viewer with a set of joystick controls into a '"roomÓ having three walls and a floor in which a database is represented as a 3-D "object."  This is coupled with 4-channel sound. The basic technique involves polarized glasses, four projectors, and mirrors. More advanced versions bypass the need for mirrors, polarized glasses, multiple projectors, helmets and other distracting stimuli and make use of a unobtrusive, immersive environment.  Bob and I can probably gain access to an existing setup like this at CU Boulder; I think that Ray McCall is also familiar with this setup (The BP Center for Visualization).  There is, no doubt, a similar environment at Harvard, MIT, Stanford and CALTECH.

We believe that this existing 3-D imaging software and hardware for time-space representation is at a threshold akin to the threshold that existed between the most advanced silent films and Vitaphone.  In this sense The Matrix is to two-dimensional sound film what the most advanced silents were before the onset of talkies.  The next-generation interface that we envision is an audio-visual display reminiscent of the Theremin for the manipulation of objects and sound.  In this space the conductor conducts not merely the soundtrack but the visual space itself.

If we now shift the time-space scenario to a geological-geographic (GIS) data bank, and the "music" to a sound archive (including possible voiceover commentary) the methodology lends itself to the analysis and management of forest fire damage and other disturbed sites (including the impact of urbanization or suburbanization). Your existing data from fire-sites could in principle be combined with real-time monitoring[1] , and with future plans.  In time all of the individual sites could ultimately be optimized into a single composite for a decision-making model like the one envisioned by Ray Studer.  Here it is crucial to understand that ultimately we deal with the symbolic nature of the data matrix rather than a brute, literal visualization:  thus a composite as a multidimensional data matrix is very plausibly a pragmatic possibility.

Specifically we can use this scheme to create a super-library, a resource much like current battlefield visualizations, where each object (i.e. each fire-site) is dynamically linked to cost and specifically to cost trade-offs represented in text or audio or both.[2]   The library would be used to coordinate at the State and National level the most efficient application of resources in an integrated audio-visual database.  We would be designing an instrument of ecological monitoring, management and restoration.

Here the musical theater of the demonstration yields a landscape-architectural application to the psychology of the interface. 

We want to design the tools of the interface.  We want to design the psychology of the space; and the architecture of the visual interface (audio- and haptic-visual).  We have time-site, time-finance, time-visual, and time-space.   Nothing static.  Nothing object-like. A dynamic representation.

We believe this is an ideal format for you to find the research funding for your current research agenda from the US Forest Service, NSF or EPA and other environmental management and cleanup instrumentalities as well other major funding sources.