Research on Future Communications Based on Collaboration
between Artists and Engineers
NAKATSU Ryohei
Our ultimate goal for art and technology is to create a new communication environment
that strongly appeals to human kansei (sensitivity) by integrating different aspects
of art and technology. In particular, interactive art is a new area that will
enable viewers to interact with the art and achieve two-way communication. Initially,
we began with the idea that collaboration between artists specializing in this
new art and engineers would have great potential for this new form of communication.
Art and technology is very significant in the overall picture of our project,
so each department of our lab has kept this concept in mind. We have established
an independent Art & Technology Department to focus on some research themes that
mainly artists will pursue. Toward this goal, we established the following three
projects and engaged in relevant research:
- Interactive Characters
The generation of computer characters that can communicate with people emotionally,
and the development of various systems using the created characters
- Interactive Environments
Research on environments that can be changed according to the circumstances
of communication
- Active Direction Engineering
Systemizing artistic techniques to achieve active direction technology, including
the generation of content.
Our research was divided into three periods: a two-year first term, a two-year
middle term, and a three-year final term. During the first term, we sought
to establish the concept of a new form of communication. During the middle
term, we focused on the development of various systems based on that concept.
In the final term, we sought to refine the developed systems and make them
more compact and economical for commercial development.
Interactive Characters
During the first term, we proceeded with the basic concept of emotional communication.
We built one system called "MIC" capable of recognizing human emotion as well
as another system called "Interactive Poem1
" through which humans and computers recite poems to each other.
During the middle term, we tried to achieve deeper immersion by interacting with
the systems. Specifically, we built an "Interactive Theater2
" (Fig. 1) capable of changing stories
through dialogues with the characters. As an improvement, we enabled the participation
of multiple individuals and added a function enabling reaction to spontaneous
interaction.
In the final term of the project, we sought to create characters that can visualize
the underlying meaning of superficial communication. The embodiment system, called
"Unconscious Flow," is based on the concept of visualizing subconscious communication.
"Neuro-Baby Internet-Mail" is another system in which characters react according
to the emotional content of sentences in e-mail, and the animated characters move
around. In addition, in preparation for commercial development, we succeeded in
porting this new system to a PC.
Interactive Environments
During the first term of the project, we developed "MIC Space3
," a system in which virtual plants surrounding people grow in response to the
communication between the people. This is a basic system for creating an interactive
environment. We also developed "Gulliver's Travels," a system that gives users
the sensation of becoming a pygmy or giant.
In the middle term, our target was to enable people to feel deep immersion by
interacting with the systems. We developed systems that visualize the relationship
between verbal communication and non-verbal communication between people. For
example, "Life Spacies4
" is an application system that converts input human messages into artificial
life forms. In addition, we developed the "PICO_SCAN" system, which transforms
an image of a person retrieved from a camera into an abstract computer-generated
graphic. This graphic depicts one's inner feelings toward another based on the
distance between them.
During the final term, we wound up our research by developing "Riding the Net"
(Fig. 2), a visual communication system.
Once some keywords have been uttered and the Internet has been accessed, the system
retrieves images related to the keywords. Finally, the images flow onto the display.
As well, we succeeded in porting this new system to a PC in preparation for commercial
development.
Active Direction Engineering
We developed the basic technology for a system that elicits a sensation of immersion.
During the first term, we developed a fractal structure generation algorithm that
uses fractals to create high-definition images. This system effectively appeals
to human kansei and comforts viewers. We also developed a technique incorporating
a neural network that recognizes emotions in the voice5
. With this system, people can interact nonverbally with animated characters.
In the middle term, we focused on advancing what we had developed in the first
term. Through evaluations, we confirmed that the fractal structure generation
algorithm activates human brains and enhances the viewer's comfort. We also enhanced
the emotion recognition method by enlarging the recognition database. Additionally,
we developed an authoring tool that facilitates the generation of interactive
scenes and the control of interaction.
In the final term, we sought to port the new system to a PC to make the system
easier to use. We also succeeded in developing "CosTune" (Fig.
3), a wearable musical instrument as an application system.
As stated, Art & Technology is a new type of research that aims to develop innovative
communication through the collaboration between artists and engineers. As a result,
various types of technologies and systems have been developed. This project is
now well known worldwide and has earned an excellent reputation.
Reference
