Predicting where you'll look in a 360° video

I did this as a researcher at UMass Amherst, from May to August 2018. A 360° video needs 4 to 6 times the bandwidth of a normal one, but you only ever see a slice of it, about 120° across and 55° tall. If the player could guess where you'll turn, it could stream that slice in full quality and the rest at lower quality.

I built a model to make that guess, a sequence-to-sequence LSTM in PyTorch. That's a kind of network that reads one sequence and writes another. I first tried two simpler networks, a plain LSTM and a GRU, as a classification problem, then switched to the encoder and decoder design, since it fits a sequence better.

It takes the last 120 frames of where a viewer's head pointed, plus a precomputed saliency map for each frame, which is a heat map of what in the picture draws the eye. Then it guesses the next 120 frames. Each guess names one of 32 tiles in a 4 by 8 grid laid over the video.

The data came from a public head-movement study with 57 participants, and I trained on its rollercoaster video. Head rotations came as quaternions, four numbers that describe a turn in 3D, and I wrote the math that maps them onto a flat video frame by hand. Now and then a point came out as NaN, short for "not a number," which my README from that summer calls "just a math issue."

The first item on the README's to-do list, a core part of the idea that I hadn't built yet, was to give the decoder the saliency maps for the frames it's predicting.

Stack

Languages

Python

Models and agents

PyTorch

Data

HDF5OpenCV