Visual feature extraction

A video is nothing but a sequence of frames. Each frame is a 2D image with n channels. In sequential architectures, either the frames are directly passed into ConvLSTM ( Xingjian et al., 2015 ) layer(s) or the frames are individually passed through a convolutional block and then are passed into LSTM ( Gers, Schmidhuber & Cummins, 2000 ) layer(s). For our computational limitations, our model uses the latter option. Like “Sequence to Sequence?Video to Text” ( Venugopalan et al., 2015 ), our model uses a pre-trained VGG16 model ( Simonyan & Zisserman, 2014 ) and extracts the fc7 layer’s output. This CNN layer converts each (256?×?256?×?3) shaped frame into (1?×?4096) shaped vectors. These vectors are primary inputs of our model.

Textual feature representation

Each video sequence has multiple corresponding captions and each caption has a variable number of words. In our model, to create captions of equal length all the captions are padded with “pad” markers. The “pad” markers help create a uniformity in the data structure. The inclusion of “pad” markers do not create any change in the output as they are omitted during the conversion of tokenized words to complete sentences. A start marker, and an end marker, marks the start and end of each caption. The entire text data is tokenized, and each word is represented by a one-hot vector of shape (1?×? uniquewordcount ). So, a caption with m words is represented with a matrix of shape ( m ?×?1?×? uniquewordcount ). Instead of using these one-hot vectors directly, our model embeds each word into vectors of shape (1?×? embeddingdimension ) with a pre-trained embedding layer. The embedded vectors are semantically different and linearly distant in vector space from others on the basis of relationship of the corresponding words.

Base architecture

Like most sequence-to-sequence models, our base architecture consists of a sequential encoder and a sequential decoder. The encoder converts the sequential input vectors into contexts and the decoder converts those contexts into captions. This work proposes an encoder with double LSTM layers with stacked attention. The introduction of a mechanism that stacks attention and the mechanism of pulling spatial information from input vectors are the two novel concepts in this paper and are discussed in detail in later sections. The purpose of using the hard-pull layer is to bring superior extraction capabilities to the model. Since the rest of the model relies on time-series information, the hard-pull layer is necessary for combining information from separate frames and extract general information. The purpose of stacking attention layers is to attain a higher quality temporal information retrieval capability.

Multi-layered sequential encoder

The proposed method uses a time-distributed fully connected layer followed by two consecutive bi-directional LSTM layers. The fully connected layer works on each frame separately and then their output moves to the LSTM layers. In sequence-to-sequence literature, it is common to use stacked LSTM for encoder. For it, our intuition is, the two layers capture separate information from the video sequence. Figure 4 shows having two layers ensures optimum performance. The output of the encoder is converted into a context. In relevant literature, this context is mostly generated using a single attention layer. This is where this paper proposes a novel concept. With the mechanism mentioned in later sections, our model generates a spatio-temporal context.

Single-layered sequential decoder

The proposed decoder uses a single layer LSTM followed by a fully connected layer to generate a word from a given context. In relevant literature, many models have used stacked decoder. Most of these papers suggest, each layer of decoder handles separate information, while our model uses a single layer. Our experimental results show that having stacked decoder does not improve the result much for our architecture. Therefore, instead of stacking decoder layers, we increased the number of decoder cells. Specifically, we have used twice as many cells in decoder than in encoder and it has shown the optimum output during experimentation.

Training and inference behaviour

To mark the start of a caption and to distinguish the mark from the real caption, a “start” token is used at the beginning. The decoder uses this token as a reference to generate the first true word of the caption. Figure 2 represents this as “first word”. During inference, each subsequent word is generated with the previously generated word as reference. The sequentially generated words together form the desired caption. The loop terminates upon receiving the “end” marker.

During training, if each iteration in the generation loop uses previously generated word, then one wrong generation can derail the entire remaining caption. Thus error calculation process becomes vulnerable. To overcome this, like most seq-to-seq papers, we use the teacher forcing mechanism ( Lamb et al., 2016 ). The method uses words from the original caption as reference for generating the next words during the training loop. Therefore, the generation of each word is independent of previously generated words. Figure 2 illustrates this difference in training and testing time behaviour. During training, “Teacher Forced Word” is the word from the reference caption for that iteration.

Stacked attention

Attention creates an importance map for individual vectors from a sequence of vectors. In text-to-text, i.e. , translation models, this mapping creates a valuable information that suggests which word or phrase in the input side has higher correlation to which words and phrases in the output. However, in video captioning, attention plays a different role. For a particular word, instead of determining which frame (from original video) or frames to put more emphasis on, the stacked attention emphasizes on objects. This paper uses a stacked LSTM. Like other relevant literature ( Venugopalan et al., 2015 ; Song et al., 2017 ), this paper reports separate layers to carry separate information. So, if each layer has separate information, it is only intuitive to generate separate attention for each layer. Our architecture stacks the separately generated attentions and connects them with a fully connected layer with tanh activation. The output of this layer determines whether to put more emphasis on the object or the action. (1) ${\displaystyle f_{attn}\left(\left[h,ss\right]\right)=a_s\left(W_2?a_{tanh}\left(W_1\left[h,ss\right]+b_1\right)+b_2\right)}$ (2) ${\displaystyle c_{attn}=dot\left(h,f_{attn}\left(\left[h,ss\right]\right)\right)}$ (3) ${\displaystyle c_{st}=a_{relu}\left(W_{st}\left[c_{att{n}_1},c_{att{n}_2},\dots ,c_{att{n}_n}\right]+b_{st}\right)}$ where,

• h ?= encoder output for 1 layer

• ss ?= decoder state is repeated to match h ’s dimension

• $a_s\left(x\right)=\frac{exp\left(x?max\left(x\right)\right)}{\sum exp\left(x?max\left(x\right)\right)atten}$

• n ?= number of attention layers to be stacked

• c _attn ?= context for single attention

• dot () function represents scalar dot-product

• c _st ?= stacked context for n encoder layers.

Equation (1) is the attention function. Equation (2) uses the output of this function to generate the attention context for one layer. Equation (3) combines the attention context of several layers to generate the desirable spatio-temporal context. The paper also refers to this context as “stacked context”. Figure 3 corresponds with these equation. In SSVC, we have particularly used n ?=?2, where n is the number of attention layers in the stacked attention.

The stacked attention mechanism generates the spatio-temporal context for the input video sequence. All types of low-level context required to generate the next word is available in this novel context generation mechanism.

Spatial Hard Pull

Amaresh & Chitrakala (2019) mentions that most successful image and video captioning models mainly learn to map low-level visual features to sentences. They do not focus on the high-level semantic video concepts - like actions and objects. By low-level features, they meant object shapes and their existence in the video. High-level features refer to proper object classification with position in the video and the context in which the object appears in the video. On the other hand, our analysis of previous architectures shows that almost identical information is often found in nearby frames of a video. However, passing the frames through LSTM layer does not help to extract any valuable information from this almost identical information. So, we have devised a method to hard-pull the output of the time-distributed layer and use it to add high-level visual information to the context. This method enables us to extract meaningful high-level features, like objects and their relative position in the individual frames.

This method extracts information from all frames simultaneously and does not consider sequential information. As the layer pulls spatial information from sparsely located frames, this paper names it “Spatial Hard Pull” layer. It can be compared to a skip connection. But unlike other skip connections, it skips a recurrent layer, and directly contributes to the context. The output units of the fully connected (FC) layer of this spatial-hard-pull layers determines how much effect will the sparse layer have on the context. Figure 5 indicates the performance improvement in the early stages due to SHP layer and the fall of scores in the later stages due to high variance.

SS score calculation

Combining equations Eqs. (4) , (5) and (6) , the equation for the SS Score can be obtained. (7) ${\displaystyle SSScore=\frac{1}{N}\underset{n=1}{\overset{N}{\sum }}\left(S_{grammar}?\frac{S_{element}+S_{action}}{2}\right)}$

During this research work, the SS Score was calculated by 4 final-year undergraduate students studying at the Department of Computer Science and Engineering at the Islamic University of Technology. They are all Machine Learning researchers and are fluent English speakers. Each caption was separately scored by at least two annotators to make the scoring consistent and accurate.