Lately I’ve been working on enabling Pylearn2 to iterate over variable-length sequences. In this post, I’ll discuss my progress so far.

The problem

Some types of models (such as convolutional or recurrent neural nets) naturally deal with variable-length inputs. Unfortunately, for the moment, this type of input is not well supported in Pylearn2: all Space subclasses expect the data to be a tensor whose first dimension is the batch axis and whose other dimensions are of fixed size. This means a sequence of fixed-sized elements cannot be stored in those spaces, because all time steps of the sequence would be considered as separate examples.

Even more fundamentally, there is no straightforward way to represent data structures containing variable-length elements in Theano. This means even if we solve the Space problem in Pylearn2, we’re limited to batches of size 1 unless some TypedList data structure is implemented in Theano.

New spaces

I wrote two new Space subclasses (VectorSequenceSpace and IndexSequenceSpace) to deal with variable-length sequences. They’re very similar to the corresponding VectorSpace and IndexSpace, with few key differences:

• Because of Theano restrictions, an object in living in a *SequenceSpace is considered to represent a single example, unlike e.g. VectorSpace, which considers objects as batches of examples.
• A *SequenceSpace expects objects living in its space to be matrices whose first dimension is time and whose second dimension represent a fixed-sized state, e.g. a features vector.
• In order to enforce the fact that we’re dealing with a single example, it is impossible to convert a *SequenceSpace into a *Space. Doing otherwise would give rise to confusing behaviour: by going from a VectorSequenceSpace to a VectorSpace, suddenly every time step of the sequence is considered as a separate example. The only conversion allowed is from an IndexSequenceSpace to a VectorSequenceSpace.
• Some methods such as get_total_dimension() don’t make sense when dealing with variable-length sequences and are not implemented.

New TIMIT wrapper

I also wrote a new TIMIT wrapper called TIMITSequences, which uses VectorSequenceSpace and IndexSequenceSpace to represent its data. Iterating over this dataset returns whole sequences. These sequences are segmented in frames of frame_length and form matrices whose first dimension is time and whose second dimension is what a sliding window of this length sees as it’s passing through the sequence.

As a proof-of-concept, I also wrote a toy RNN model (which you can find here) to train on this dataset. I haven’t had time to play with it a lot, but I hope to find time to do so this week and next week and present some results in another blog post.