Aiming for Robust and Flexible Speech Translation Technology
for Real-World Applications
Seiichi Yamamoto, President
ATR Spoken Language Translation Research Laboratories
Globalization has been accelerated by information technologies where information
exchange between different languages is indispensable. Consequently, speech translation
technology has been emphasized more strongly than it used to be. ATR Interpreting
Telecommunications Research Laboratories (ATR-ITL), which was established in 1986
to research basic technologies for speech translation for natural spoken dialogue,
has developed a Japanese and English bilateral speech translation system that
was evaluated to be available as communication tools between speakers of different
languages. However, it is still limited to spoken dialogue in well-specified tasks
under moderate conditions, such as in a quiet office.
ATR Spoken Language Translation Research Laboratories (ATR-SLT), which was established
to follow up on the above research results, Iis aiming for further advancement
of current speech translation technologies to cope with problems Iin real-world
applications, Our research topics range from robust acoustic front-end processing
to flexible language processing. These research efforts would not only widen the
use of speech translation technology for spoken dialogues but also provide a chance
for more challenging applications such as simultaneous translation of monologues.
1. Extending the Use of Speech Translation Technology for Spoken Dialogues
Current speech recognition, which is based on stochastic methods, generally uses
a language model that expresses the transition probability between words, and
it has been necessary to create a new language model for each individual task.
The creation of a stochastic model requires the construction of a large-scale
corpus that gathers huge volumes of spoken dialogues. In the case of speech recognition
for spoken dialogues, which cannot use existing large-scale corpora (such as newspapers),
an enormous amount of time and effort is required for the construction of new
corpora. For this reason, it is difficult to transfer speech translation technologies
for use in other tasks.
In the case of language translation technologies this issue takes on an even greater
level of difficulty. In corpus-based language translation, large-scale parallel
corpora are created, and sentences with identical expressions are retrieved from
among the parallel corpora; these sentences are then used for the language transfer.
It is very rare for an entire sentence to match up exactly, although this occurs
occasionally in the case of idiomatic expressions. The most commonly applied method
is thus to search for examples of semantically similar parts of input expressions,
which are then combined to form full sentences. That is so say, the input expression
is analyzed into some structures (or patterns) to match the examples of each structure,
and if there is not an exactly-matched expression in the translation examples,
a nearest-matching method is used to find the closest-matching translation example
by measuring the intuitive semantic conceptual distance of the linguistic expression.
Exactly-matched examples and examples selected with the nearest-matching method
are then used to construct a full translated sentence.
Currently functions such as the selection of the patterns used for these sentence
segments rely entirely on human effort, while the selection of patterns and examples
for each pattern are dependent on the task. Task portability in this situation
is therefore even more difficult than in the case of speech recognition systems,
which are basically capable of machine learning as long as a corpus exists.
Resolving the problem of task portability is a technical issue that is essential
for the broad application of speech translation technologies, that is, for the
efficient development of speech translation systems that can be used in a variety
of fields. We are thus conducting research and development regarding technologies
that will allow us to overcome the above technical issues and facilitate efficient
expansion of the fields of application for spoken dialog translation. This is
our first research goal. To broaden the fields of application for spoken dialogue
translation, we must accommodate a variety of application environments as well
as the differences in the speaking styles of users.
2. Speech Translation for Monologues
Until now, the target of research in the field of speech translation technologies
has been spoken dialogue translation. This is in part due to the great importance
of spoken dialogue translation as one field of application. Another reason, however,
is that features such as spoken dialogues can contain more ellipses and fragmental
expressions than can written language, attracting much interest from researchers
of natural language processing. Nevertheless, in the case of spoken dialogue translation
a speaker exists on both sides of the system, so one can expect that questions
and answers will develop based on the speakers' understanding of the dialogue
content. For example, the following scenario would be possible: if one segment
of an utterance is unclear － for example, due to an error in speech recognition
－ then the utterance would be translated with that segment omitted, and, if necessary,
the speakers could clarify using additional questions and answers. Furthermore,
even when dealing with phrase utterances, it is usually possible to understand
the overall meaning of such an utterance simply by translating each phrase separately,
without special consideration for the relationship with content that precedes
or follows the phrase in question.
In contrast, most technologies like those discussed above cannot be used in the
case of monologues such as lectures. Utterances in such monologues are longer
than in regular daily conversations, which would increase the technical difficulties
involved for both speech recognition and language translation. In addition, it
would be impossible to adopt methods such as those used in spoken dialogue translation,
in which speech translation is carried out after each utterance is complete. In
effect, what is required is the execution of speech translation that "tracks"
the monologue, much as in the case of simultaneous interpretation.
Although we refer here to simultaneous interpretation, we do not believe that
it is necessary to use methods identical to those that might be used by a professional
simultaneous interpreter of Japanese/English, such as, for example, the frequent
use of phrase inversion to account for differences in sentence structure. This
is because, unlike a human interpreter, a speech translation system uses an almost
unlimited amount of short-term memory to output a translated sentence that does
not have inversions after a certain amount of processing time.
Speech translation of monologues is as extremely challenging research theme －
one in which full-scale research has yet to be conducted. Nevertheless, we feel
that once this research comes to fruition, we will be able to apply the results
in the form of interpretation technologies with a form far removed from what has
been seen in the past.
Below, we offer a brief explanation of the research theme being conducted to achieve
our research goals.
(1) Robust Speech recognition
The performance of existing speech recognition technologies varies greatly depending
on differences in input devices, surrounding noise, and speaking styles. For example,
unlike close-talking microphones, desktop microphones pick up environmental noise
and echoes reflected from nearby walls and other objects along with the voice
being input directly, resulting in a deterioration of the speech recognition performance.
One can assume that speech translation systems will be implemented as PDAs and
other portable devices in the near future, and it is also likely that these types
of devices will also have built-in microphones. It will thus be necessary to develop
technologies that demonstrate a high level of speech recognition performance even
when using such microphones. Furthermore, as illustrated in Figure
1, spoken dialogue translation presumes at least two speakers, so it is also
necessary to have a technology that can differentiate between two (or more) voices.
To respond to these demands, we are promoting research into technologies that
can differentiate between the voices of both speakers by using multiple microphones,
and technologies that incorporate sensors used to increase the performance of
speech recognition by using inputs of visual images along with the sound.
We are currently studying separate acoustic models required for speech recognition
in specific languages such as Japanese, English, and Chinese, but are also planning
research into technologies that can execute speech recognition using a single
acoustic model integrating these separate models.
(2) Online universal speech parsing of spontaneous speech for various topics
Currently, the method most commonly used for speech recognition is one in which
the transition probability of subsequent words － known as "N grams" － is calculated
beforehand from a large-scale language database. The N gram is then used to predict
the next word in a sequence. A method in which N=2, that is, when a subsequent
word is predicted based only on the word immediately preceding it, is called a
"bi-gram"; when N=3, the method is called a "tri-gram". The larger N becomes,
the more the prediction performance increases, but considering the volume of data
that can be collected, N will be no more than 3 even under the best of conditions.
For this reason, it is not possible to incorporate into the model any dependency
relationships that extend across more than three words. As a result, the system
occasionally outputs recognition results in the form of word strings that are
incomprehensible to human beings, leading to a significant loss in the reliability
of speech recognition technologies. This situation arises due to an insufficient
use of both linguistic knowledge and knowledge of the targeted topic of conversation,
so the creation of a more efficient model is essential.
Current speech translation technologies involve the execution of cascade processing
－ Illustrated in Figure 2 － in which
speech recognition results derived from waiting for a single utterance to complete
are then passed on to the translation module. This is not a problem in situations
such as dialogues, where each utterance is comparatively short and the utterances
alternate between two (or more) speakers. In the case of monologues, however,
where are utterance continues for some time, a problem arises from the substantial
delays that occur. A technology is thus needed which can sequentially output speech
recognition results in the middle of an utterance, without waiting for it to be
complete. Conversely, in cases such as phrase-by-phrase utterances, we must execute
speech recognition while keeping in mind the relationships between such phrases.
(3) Corpus-based translation
Speech translation uses a method that applies parallel corpora but in terms of
using translation examples for the sentences themselves, there are so many kinds
of sentences that no matter how many sentence examples were contained in each
corpus, it would be impossible to achieve exactly-matched examples. For this reason,
patterns that are abstractions of such sentence segments are used, but there is
a reliance on human labor for choices such as what types of patterns should be
extracted from the large-scale database, or what specific examples should be selected
for each pattern. Such selection of patterns and examples of each pattern is dependent
on the targeted topic, and criteria for such selections are slightly different
depending on the human involved. As a result, each time the targeted topic changes,
it is necessary for a human to become involved in the operation, and the portability
of translation systems thus becomes extremely difficult.
Thus, we will develop a technology to efficiently support people to select patterns
and examples by calculating pattern matching frequencies from a morphological
analysis of the large-scale database. We will also study automatic learning methods
to select patterns and examples through the probabilistic model.
Two critical issues that arise when collecting data for the large-scale database
are what volume of sentence examples should be gathered, and how they should be
gathered. These are extremely difficult issues, but we will continue to search
out ways of resolving these issues through the process of collecting data for
the large-scale database and through research relating to example-based translations
using this database.
Pattern selection is dependent upon the structural proximity between the two languages
targeted for translation. For example, we can assume that the patterns required
to translate between Japanese and English － which are structurally very different
－ are entirely unlike the patterns required to translate between Japanese and
Korean, which have similar structures. Translating between Japanese and English,
then, may be technically difficult. Thus, for translating spoken dialogues, we
are aiming at researching translation technologies between different structural
languages, such as one for translations between Japanese and English, and one
for translations between Japanese and Chinese.
(4) Incremental spoken language generation
To execute simultaneous speech translation of monologues, it is necessary to divide
those utterances into appropriate units before translating. Generally, we can
assume that the complexity of the utterances within the units can be reduced by
setting shorter lengths for those divisions, and, as a result, we can improve
the performance of the resulting speech recognition and language translation.
If the divisions are too short, however, the contextual relationship is lost and
the translation results become difficult to understand. What is required first,
then, is research to determine the optimum unit size for these divisions from
the perspective of understanding the translation results. We believe that one
way to approach this is to examine the intervals that a professional simultaneous
interpreter uses to begin a segment of a translation; therefore, we plan to construct
a database made from the speech of a simultaneous interpreter translating Japanese
monologues into English. However, since the speech translation system can principally
use an unlimited short-term memory, it is not necessary to translate speech with
the same method used by professional simultaneous interpreters.
Furthermore, in the context of generating speech output that follows along with
the original speech, there are cases where, because of time limitations, it is
impossible to generate speech output at a fixed speed. It is thus necessary to
have "summarizing" technologies that can omit unimportant segments, and to generate
speech synthesis for unimportant phrases at a higher rate of speed than for important
phrases. Current speech synthesis technologies have reached a point where it is
possible to synthesize high-quality voices in certain limited topics using corpus-based
speech synthesis technology, one of which was speech synthesis technology, one
of which was originated at ATR. This performance, however, can only be obtained
at a fixed speaking speed. There has been absolutely no research, from the standpoint
of application for traditional speech synthesis, into controlling prosody such
as speech rate to correspond to situational requirements. Technology that allows
appropriate control of prosody, including speech rate that corresponds to the
importance of the speech content, is important for synthesizing speech with expressive,
attractive speech quality, which will be differentiated from conventional speech
synthesis. We believe that this technology, as with speech translation, will first
be applied in fields where the sentence structure of the speech output is clarified.
Speech translation of monologues is an unexplored field － one in which no one
up to now has undertaken research. For this reason, unlike the case of spoken
dialogue translation. Where we have already gained some experience in the past,
we plan to proceed with research targets in the field of speech translation of
monologues to translate Japanese utterances of limited topics into English.
