Creating a Synthetic Video Dataset for the VAST 2009 Challenge

Mark A. Whiting, Carrie Varley, Jereme HaackPresented by Jean Scholtz

BELIV 201003-28-2010

IEEE Visual Analytics Science and Technology (VAST) Annual Challenge – if you don’t know about it, you should review it

Why look at theVASTChallenge? There are scenarios, tasks, and data available for use and modification; plus award-winning write-ups and software solutions. When known ground truth is available, a new world opens up for evaluation of analytic software.

Accuracy measures now become possible.

See http://hcil.cs.umd.edu/localphp/hcil/vast/index.php to start with. Catherine Plaisant and her assistants have done a great job in capturing the 2006 – 2009 challenge information

VAST 2009 Features the First Video Challenge

Multimedia and video analysis are becoming increasingly important in information analysis, as people do more with these media

First video analytics workshop at VAST 2009; special edition of IEEE CG&A on multimedia analytics soon

TRECVID looks at event detection; we wanted to look at events within scenarios

Questions can be asked: Why are those people there, and what does it have to do with the story being investigated?

Was this a “synthetic” video?

Of course. See Herb Simon’s discussion of artificial and synthetic in The Sciences of the Artificial. We needed to engineer known ground truth into the scenes. What did we do? We sent our actors in front of an existing webcam to act out a couple of scenes.

What was the scenario?

A US embassy employee in “Flovania” was meeting their “handler,” a

person who was acting friendly to the employee while really working

for a criminal organization. The task was to find instances of possible

meetings between the two as well as other events that might indicate

espionage within a reasonably long segment of recorded video.

Evidence planted in the video was the illegal transfer of information

from the employee to the handler as well as other meetings.

What webcam did we pick? Walla-Walla Washington.

Southwest Camera Angle: toward Starbucks

West Camera Angle: looking west on Main

Northwest Camera Angle: toward the canal

East Camera Angle: looking east on Main

Analysis IssuesHad multiple (4) views to contend with

Only needed to analyze small segments of the video field of view

What did we plant?

A scene where the Embassy employee “dupe” was meeting the handler outside the coffee shop and handing off information

A scene where the handler was meeting another member of her criminal organization. They did the old briefcase “switcheroo”. See the light and dark cases.

How did we coordinate all the activities?

One team was dispatched to Walla Walla; another was in the lab recording the web cam video off the web.

Teams were connected via cell phone.

The WW team has a timed script. When we needed to pass a message to the actors, we had a “non-actor” walk by and whisper instructions.

The lab team recorded several hours of web cam video overall, to ensure enough “noise” video to make the search challenging.

Video duration was 8 hoursEach “scene” was viewed for several seconds before the camera switched to another view.Three segments that had to be recognized for accuracy (7 seconds, 5 seconds and 7 seconds long)

Contestant Results: one example

University of Stuttgart – using video perpetuograms to track people from between scenes and views to enhance continuity

Lessons Learned & Next Steps

Participants liked the challenge but few teams were prepared to do video.

We would like to coordinate more closely with groups like the TRECVID organizers for future video analytics challenges.

And encourage teams entering VAST Challenge to seek out groups with expertise.

This was a very new kind of challenge for both our visualization and information analysts – there wasn’t much of a baseline for either group to base assessments on.

Little software available.Question by analysts was what might be missed due to automatic recognition.

As always it is a challenge to fit the mini challenges into the overall Grand Challenge.