The problem
A caption has to name the objects, the scene and the action in a picture, and get the words in the right order. Occlusion, lighting and position make even the first part hard.
How I built it
See.
A pre-trained Xception CNN turns each image into a feature vector.
Describe.
An LSTM takes those features and writes the caption one word at a time, between start and end tokens.
Clean data.
Scalable preprocessing pipelines for the Flickr8k and Flickr30k datasets.
Score.
Generated captions are scored against the human ones with corpus BLEU.
Results
0.535
ROUGE, ahead of Google NIC by 0.5% and LRCN by 0.7%.
0.265
METEOR, ahead of Google NIC by 1.1%.
+15%
model accuracy; BLEU-1 to BLEU-4 of 0.623, 0.427, 0.282 and 0.181.
