Creating Media for Co-Experience via the Internet




1. Background
　　The internet has become widely available and popular. A huge amount of information and verbal knowledge is presented, shared, and communicated on a worldwide basis. The Japanese government plans to construct a super-high-speed network infrastructure enabling users to smoothly create and receive high-quality video, voice and graphics by 2005 for over 10 million homes[1]. The next-generation internet will provide ubiquitous information sharing, allowing information to be communicated anywhere, anytime, and with anybody.
　　E-mail and the internet have clearly accelerated the trend towards globalization in politics, economics, and culture since the 1990's, while also exacerbating communication gaps between different cultures and generations, and between power users and non-power users in the "digital divide". In order to cope with these communication gaps due to globalization, media must be developed to accommodate the needs of different generations, regions, occupations, cultures, and societies. We call this "diversity communication media".
　　Looking back over the history of media usage, in the mass media era, before the 1950s and 1960s, experts disseminated messages to the general public based on text (e.g., newspapers), audio (e.g., radio), and visual (e.g., television) content. This was followed by the person-to-person communication era of the 1980s, when personal communication was facilitated by e-mail, mobile devices, and cellular phones. In the 1990s, groupware and the World Wide Web (WWW) expanded person-to-community and community-to-community communication. Now the era of diversity communication media mentioned above is approaching (Fig. 1). A variety of media corresponding to this communication era has also been developed. The history of media usage might imply that people have an inherent, never-ending quest to disseminate their feelings and experiences to others and to share them using various media. That is, media are deeply related to the formation of new cultures.
　　From this viewpoint, the current WWW based on hypertext, such as text, sound, and video, is insufficient for sharing specific feelings and intentions over the internet because it merely transmits hypertext from one person to others in an asynchronous, one-way system of knowledge communication. In reality, non-verbal information, such as haptic, smell and augmented reality, may be more effective for sharing feelings and experiences. Tele-existence techniques, which enable an operator to remotely control tasks, are also very promising in bi-directional communication. Our research, therefore, focuses on media based on these non-verbal forms of information in asynchronous bi-directional communication.
　　Verbal machine translation is a core technology for allowing diversity communication media to cope with the communication gaps between people using different languages. Much research on corpus-based machine translation has been done. The corpora consist of bilingual parallel texts. In the case of non-verbal communication, "machine translation" corresponds to "media translation", a method of converting media information between two users, because it deals with different media between two users instead of different languages. In the case of media translation, the corpora contain information about the five senses rather than text. With this motivation, we are conducting research on communication sharing with digital feeling and digital experience information, mainly non-verbal information. We intend this research to lead to a "co-experience web" that allows users to share human experiences and creative activities.

2. Research target
　　"Experience" means (1) a form of practical knowledge, skill, or practice derived from the direct observation of or participation in events or in a particular activity[2], or (2) that which one sees, hears, or feels as a personal meaning because it is not yet generalized in knowledge and cannot be transmitted to another person[3]. However, we assume that people may partly share information concerning their experiences by representing them in a real or virtual sense via external information from their behavior, facial expressions, gestures, gait, height, weight, etc., and internal information such as their heart rate, blood pressure, and so on
　　We aim to create media for sharing one's experiences using easy-to-use tools such as email or current web browsers. Such a co-experience might create different experiences between communicating persons because one person's experience is fed back to the other and vice versa. For example, it may give us a chance to learn about the distinctive non-verbal knowledge and creativity of such people as craftspersons and artists, which cannot be learned through the current web or from textbooks.
　　Fig. 2 shows an example of a new lifestyle provided by the co-experience web. A living room could be transformed into a co-experience communication space by providing a family with haptic devices for feeling the tactile sensations of objects as well as with a robot and a doll that have the communication functions of seeing, hearing, talking, and moving their arms and heads as if they were human. The boy excitedly learns about dinosaurs in school one day and transmits his diary via the co-experience web. Back home, he talks about dinosaurs while using his web information. His father can understand what his son is talking about by virtually touching the skin surface of a dinosaur represented with the web information. The robot and doll guide the family so as to successfully represent the experience of the boy and then give additional information on the dinosaur to the family during their co-experience. As a result, the family can create a new co-experience by virtually representing the experience of the boy's class. 　　This research aims to create core technologies for sharing information about one's experiences via the web. We will construct a prototype of the co-experience web as shown in Fig. 2.

3. Requirements of diversity communication media
　　Diversity communication media are a kind of interaction media that interact with humans and/or other media. They require functionality, user-friendliness, popularity, and social aspects for diverse communication, as described below.

Functionality
　　Diversity communication media must observe the experiences and/or feelings of persons, represent them, and share them among users. Since users generally utilize different media, the observed data of one's experiences are insufficient for representing them in another user's media. One possible method for representing them is to label observed data by a specified symbolic set by using recognition and understanding algorithms for the data. Then, the symbolic information, as well as the observed data, helps to interpret the differences among media and allows the experience to be represented in the other user's media. This automatic indexing research is based on techniques in pattern recognition and media understanding as well as content-based retrieval. Tele-existence and virtual reality may also suggest methods of representing experiences via the web. Thus, diversity communication media related to sensory information are referred to as "sensory interaction media."

User-friendliness
　　Diversity communication media should also allow user-friendly communication. These media communicate with humans as co-creative partners, in the form of robots, dolls, wearable interfaces, agents, and furniture. Co-creative partners direct the mechanical operation of observing a user's experience and representing it as another user's experience. The direction will improve the user-friendliness, precision, and reproduction of the experience, since users do not need to carefully follow complicated procedures for observation and reproduction. Thus, diversity communication media for communicating among users are referred to as "cooperative interaction media."

Popularization
　　The popularization of the co-experience web will depend on how many interesting forms of experience content are generated worldwide. Content for the creative processes of skilled craftspersons and distinguished artists may become popular, because they represent a kind of high-quality experience, and many people will want to co-experience them. Therefore, we focus on media for making a dictionary that includes non-verbal information, which is built by observing creative processes. The media will become more popular if they enable users to learn many kinds of techniques using the dictionary. We call these "co-operative learning media."

Social aspects
　　The co-experience web should be available to everybody, regardless of age, health, and physical condition. Therefore, we will analyze the mechanism of co-experience communication between humans with several diversity communication media. We will also simulate models of co-experience communication for personal and community levels from the viewpoint of cognitive science in order to predict changes in society.

4. Basic operation of the co-experience web and research topics
　　The basic operation of the co-experience web is shown in Fig. 3. Users may attach several wearable interfaces to themselves to observe their own physiological or cognitive information, such as heart rate, blood pressure, respiration, and gaze. Cooperative interaction media will control and integrate information obtained not only from these wearable interfaces but also from several sensory interaction media. This includes the user's behavior, interaction with co-creative partners, and the 3D spatio-temporal information around the user. The experience of user A is observed by using several co-creative partners and ubiquitous interfaces as in Fig. 3. The observed data are recognized and labelled by specified symbols in the sensory interaction media. The co-creative partners can try to observe the experience again if the observed data is insufficient for producing a corpus. When user B wishes to share the experience of user A on the web, media translation is performed by comparing the corpora of users A and B. The interaction corpus for user A is then converted to the media of user B and the co-creative partner directs the representation presented to B.
　　Cooperative learning media, another form of diversity communication media, are fundamentally similar to cooperative interaction media except that user A is an artist or craftsperson. Therefore, the resulting corpus in the observation of user A corresponds to a kind of "kansei" (creativity) dictionary related to such experts.

5. Conclusion
　　This report outlined our proposed diversity communication media and related research topics. First, we presented "the co-experience web" through which people can share feelings and experiences. Second, our research topics and challenges were introduced. Our aim is to create the core technologies of the co-experience web.



Reference