Image processing (Video tracking)

by Jesper Ellerbæk Nielsen and David Getreuer Jensen

This html is intended as a help for loading videos into matlab and as a help to analyse the indvidual frames of the movie. The video you should work with can be downloaded here.

Contents

Always start matlab coding with the following line which is necessary to clear the memory (both command window and workspace) and close all previosly opened figures.

clc; clear; close all;

Step 1: Load the video frames into matlab

The first thing you need to do is to load the frames from the "apple" video into usable matlab data (preferably doubles). Matlab, as you probably already know, has a variaty of inbuilt functions. In this case the VideoReader function can be used (the homepage of MathWorks.com is very useable) or type doc VideoReader in the command window.

Use of the VideoReader function is easy just type in the path and the returning variable saves the output. The output variable is in this case called an object:

Video = VideoReader('120905-135843.wmv');

An object means that it contains underlaying varibles. The underlaying variables can be seen using the function get:

get(Video)

obj = 

  VideoReader with properties:

   General Properties:
            Name: '120905-135843.wmv'
            Path: 'D:\Dokumenter\Undervisning\Measurement Technology and D...'
        Duration: 98
     CurrentTime: 0
             Tag: ''
        UserData: []

   Video Properties:
           Width: 640
          Height: 360
       FrameRate: 1
    BitsPerPixel: 24
     VideoFormat: 'RGB24'

The function returns information about number of frames, the resolution of the video, and video format amoung other things. This information can be accessed and saved as shown below:

nFrames = Video.NumberOfFrames;
vidHeight = Video.Height;
vidWidth = Video.Width;

Preallocate or build an empty video structure. Said in another way build an empty variable or you could say a empty car park for the frames of the video.

mov(1:nFrames) = struct('cdata', zeros(vidHeight, vidWidth, 3, 'uint8'),'colormap', []);

In the following "for" loop each of the frames from the "apple" video are parked in the car park or more technically correct saved within the prior preallocated variable. This is done by reading one frame at a time and save it into the array: (access data as with the object Video)

for k = 1 : nFrames
    mov(k).cdata = read(Video, k);
end

The frames can be seen with either the image or imshow function as seen below (frame 24):

figure; imshow(mov(24).cdata);

You can also view the loaded movie using the movie player in Matlab

implay(mov,5);

The variable mov(x).cdata contains three color codes; Red, Green, and Blue (therefore the color code are "RGB"). These colors can be used to easily identify an object of a certain color in the picture (for instance a red apple). The data contained in the mov(:).cdata are in unsigned 8 bit integers (uint8) but to make things less complicated we make a convertion from uint8 to doubles. The function im2double does exactly that. In the following are each of the three colors, from frame 24, saved in their own variable:

imR = im2double(mov(24).cdata(:,:,1));
imG = im2double(mov(24).cdata(:,:,2));
imB = im2double(mov(24).cdata(:,:,3));

The values of each color can be seen in the following figure. Don't be confussed that for an example the red color isn't red. The colors in the figure is just how much of the specific color is present in the picture at the given location of the frame. In other words it is the intensity of that color.

figure('position',[68 169 479 787]);
subplot(3,1,1); imagesc(imR); colorbar; title('Red'); axis image;
subplot(3,1,2); imagesc(imG); colorbar; title('Green'); axis image;
subplot(3,1,3); imagesc(imB); colorbar; title('Blue'); axis image;

Now we are able to load the frames into double precision that matlab can process. The next logical step is to compute the position of the apples within the frames.

Step 2: Finding the position of the apples

Several methods for finding the position of the apples are available. The easiest way is to threshold the intensity of the color red if we want to find red apples in the frames. Otherwise it is also possible to use another form of blob analysis or edgde tracking algorithms. For this exercise it is recommended to use the simplest way for apple segmentation using colors and use 2D cross correlation to find the movement of the apples from one frame to the next.

Using colors to segment (or identify) the red apple in the specific video frame is the most simple. By using the information of each colors intensity can the "red" apple be identified from the background by simple thresholding. In order to use the red color for segmentation should the red time mark in the lower right corner be cropped. It is always a good idea to crop the image so that only the interesting area is left. This way alot of computational time can be saved. Use the function imcrop for this purpose. When the position of the apples is known in each frame can the movement of the apples easily be computed.

As a part of the assignment should 2D cross correlation be used to compute the movement of the apples from frame to frame. The 2D cross correlation function can be used to describe the movement of objects from one frame to the next. The inbuild matlab function is called xcorr2. The function finds the highest correlation between two matrices (frames). For speed can the normalized 2D cross correlation also be applied. Try the difference between the two methods yourself.

The following line shows how easy it is to use 2D cross correlation. In the following example the 2d cross correlation are performed on itself so the maximum correlation should be when the two frames are exactly on top of each other (the middel coordinate in the 2d cross correlation matrix "X") which also can be seen from the figure:

X = xcorr2(imR,imR);
figure; imagesc(X); colorbar; title('2D cross correlation')

Example

In the following picture has the color intenstity of red been used to computed the position of the apples and 2D cross correlation to compute the movement.

The above plot shows the position of the apples and the number of apples along with the velocity vector. The velocity can be computed from the knowledge of the frame rate and distance travelled (number of pixels).

However estimating the pixel size in meters aren't easy due to the perspective of the video. Figure out a simple way to give a fair estimate of the pixel size and consecutively calculate the velocity of the apples as they passes by the webcam.

A function that works well when you want to illustrate movement vectors is the function quiver.

If you have time, you could make a comprehensive animated visualisation of your tracking of apples.

Summary of assignment

Published with MATLAB® R2015b