chapter 12

Question 1

neural machine translation

Answer 1

Google Translate employed statistical machine- translation methods until 2016

Google researchers had developed what they claimed was a superior translation method based on deep learning, called neural machine translation.

Question 2

encoder network

Answer 2

 During the encoding stage, at each time step one word of the sentence is input to the network in the form of a word vector

 from one time step to the next are recurrent connections in the hidden layer.

 One word at a time, the network builds up a representation of the English sentence, encoded in the activations of its hidden units.

 At the final time step, the encoder network is given a special END symbol, and the activations of the hidden units are now an encoding of the sentence.

Question 3

decoder network

Answer 3

the final hidden-unit activations from the encoder are then given as input to a second network

 The decoder network is simply another recurrent network, but one in which the outputs are numbers representing the words that form the translated sentence—each of which is also fed back to the network at the next time step

Note that the French sentence has seven words, whereas the English sentence has six. This encoder-decoder system can in principle translate a sentence of any length into a sentence of any other length.

Question 4

downside of encoder/decoder

Answer 4

when sentences get too long, an encoder network eventually loses useful information; that is, at later time steps it “forgets” important earlier parts of the sentence.

 Humans are pretty good at processing this kind of convoluted sentence, but recurrent neural networks can easily lose the thread.

Question 5

long short- term memory” (LSTM) units

Answer 5

specialized weights that determine what information gets sent on at the next time step and what information can be “forgotten.”

o the idea is that these units allow for more “short-term” memory that can last throughout the processing of the sentence.
o The specialized weights are learned via back- propagation just like the regular weights in a traditional neural network.

> helps with the forgetting problem

 Deep recurrent neural networks made up of LSTM units and trained on large data collections have become the bread and butter of modern natural-language processing systems

Question 6

two methods of evaluating translation results

Answer 6

1. an automated method—a computer program —that compares a machine’s translation with those of humans and spits out a score.
   > the program used in virtually all evaluations of machine translation is called bilingual evaluation understudy, or BLEU.
2. The second method employs bilingual humans to manually evaluate translations.

Question 7

BLEU

Answer 7

counts the number of matches—between words and phrases of varying lengths—in a machine-translated sentence and one or more human-created “reference” (that is, “correct”) translations.

 While the ratings produced by BLEU often correlate with human judgments of translation quality, BLEU tends to overrate bad translations.

Question 8

Given the drawbacks of BLEU, the “gold standard” for evaluating a machine-translation system is

Answer 8

for bilingual humans to manually rate the translations produced by the system.

Question 9

drawbacks to the gold-standard approach

Answer 9

 hiring humans costs money, of course, and unlike computers humans get tired after rating more than a few dozen sentences.
 Thus, unless you can hire an army of bilingual human raters who have a lot of time on their hands, your evaluation process will be limited.

Question 10

can we interpret these results to justify a claim that machine translation is now close to “human level”?

this claim is unjustified for several reasons

Answer 10

1. averaging over ratings can be misleading.
   a. while most sentence translations are rated “terrific,” there are many that are rated “horrible.” The average would be “pretty good.” However, you’d probably prefer a more reliable translation system that was always “pretty good” and never “horrible.”
2. the claims that these translation systems are close to “human level” or at “human parity” are based entirely on evaluating translations of single, isolated sentences rather than longer passages.
3. the sentences in these evaluations are all drawn from news stories and Wikipedia pages, which are typically written with care to avoid ambiguous or idiomatic language; such language can cause serious problems for machine-translation systems.

Question 11

Lost in translation

Answer 11

ambiguous words

idioms

correct grammar is occasionally missing in action

The main obstacle is this: like speech-recognition systems, machine- translation systems perform their task without actually understanding the text they are processing.

Question 12

Translating images to sentences
(Automated image captioning )

Answer 12

 The image is fed to a deep convolutional neural network instead of an encoder network.

 The ConvNet here is similar, except that this ConvNet doesn’t output object classifications; instead, the activations of its final layer are given as input to the decoder network.

 The decoder network “decodes” these activations to output a sentence.

 To encode the image, the authors used a ConvNet that had been trained for image classification on ImageNet

 The task here is to train the decoder network to generate an appropriate caption for an input image.

 In the case of image captioning, each training example consists of an image paired with a caption.
o captions for these images were produced by Amazon Mechanical Turk workers. Because captions can be so variable, each image was given a caption by five different people.