Upload
vicky
View
29
Download
2
Tags:
Embed Size (px)
DESCRIPTION
EV-Surv: Electronic and Visual Signals Integration for Efficient Surveillance. Dong Xuan, Ph.D. The Department of Computer Science and Engineering The Ohio-State University http://www.cse.ohio-state.edu/~xuan - PowerPoint PPT Presentation
Citation preview
EV-Surv: Electronic and Visual Signals Integration for Efficient Surveillance
Dong Xuan, Ph.D.The Department of Computer Science and Engineering
The Ohio-State University
http://www.cse.ohio-state.edu/~xuan
Key Collaborators: Yuan F. Zheng, Jin Teng, Junda Zhu, Xinfeng Li, Boying Zhang and Qiang Zhai
Outline
Deficiency of Visual Surveillance Systems A Brief of Our EV-Surv System Case Studies A Broader View of Our EV-Surv System Final Remarks
2/55
Visual Surveillance Important for protecting public and personal security
3/55
Visual Surveillance Massive deployment in urban areas
Over 500 surveillance cameras in a Philadelphia neighborhood (below ) New York has 4176 video cameras in lower Manhattan area [1].
4/55
Surveillance in Action
Anomaly?Suspicious
Action?
Finding all white males in red, medium
stature, from Mon through Fri last week
Online monitoring
Offline retrieval
5/55
Failure Examples
Chicago police installed 10,000 surveillance cameras in the city, only 1 of 200 crimes is captured by the visual surveillance [2]!
In San Francisco, in the first three years after the city installed cameras, they helped police charge suspects in a grand total of six cases [2]!
One of the bombers in London bombing (July, 2005) is not identified by the surveillance system and escaped [3]!
6/55
Why fail?
Large volume of video dataTemporal: 2.07*106 frames per camera per daySpatial: tons of surveillance cameras in a city
Monitored objects may be visually occluded or
have multiple inconsistent appearance
Visual technologies are not efficient and accurate enough to do automatic localization and tracking, and a lot of human power is needed!
7/55
Outline
Deficiency of Visual Surveillance Systems A Brief of Our EV-Surv System Case Studies A Broader View of Our EV-Surv System Final Remarks
8/55
Our Methodology: E-V Integration
Combining electronic and visual signals, such as GSM, 3G, WiFi, Bluetooth and NFC signals together for efficient surveillance
E-V Integration makes it possible to efficiently and accurately localize and identify objects in large volume of video data
Indexing & Sorting Localization Accuracy
E-Signal Easy Low
V-Signal Hard High
9/55
Visual Signal-based Surveillance
Can accurately localize and continuously track a person But who is he or she?
Difficult to recognize people, e.g., through face recognition
Who are they?
10/55
Electronic Signal-based Surveillance Electronic Signals
Name Distance Frequency Data Rate (down)
GSM 35 km850, 900,
1800, 1900 MHz80 kb/s (GPRS), 236
kb/s (EDGE)LTE 30 km–100 km 700 MHz–2.6 GHz >100 Mb/s
WiFi 100 m2.4 GHz (802.11b/g), 5
GHz (802.11a)54 Mb/s
2.4 GHz, 5 GHz 450 Mb/s
Bluetooth 10 m2.4 GHz,
Frequency Hopping2.1 Mb/s
(up to 24 Mb/s)
NFC < 4 cm 13.56 MHz 106 kb/s–424 kb/s
Besides data exchanged, these communication channels also contain unique and identifiable electronic identities: IMEI (International Mobile Equipment Identity), IMSI (International Mobile
Subscriber Identity), WiFi, Bluetooth MAC addresses, RFID Number.11/55
Electronic signals are emitted by many mobile device Mobile device’s popularity is increasing
Smartphone as an example: 450 million shipped in 2011
Pervasiveness of Electronic Signals
Source: Technology Review, Sept/Oct 2011
Number Units Sold(millions)
12/55
Electronic Signal-based Surveillance
Cannot accurately localize a person with a mobile device Large error in localization due to interference
But can identify the device through electronic identifiers
Interferences, e.g., vehicles, building, humans etc.
-The with SIM card number 358985010745743 is here, -But the error can be as large as 100 meters
13/55
EV-Surv: A Bird’s Eye View
14/55
User Description:-Features, Clothing-Electronic Identity-Time Range- Area
User Description:-Features, Clothing-Electronic Identity-Time Range- Area
Visual signal
Visual signal
Visual signal
Electronic signal
Electronic signal
Electronic signal
E-V Integration
E-V Integration
E-V RetrievalE-V Retrieval
Backend Database
Gateway
Other Information Databases
All frames relevant to user inquiry
EV-Surv: the Workflow
15/55
Outline
Deficiency of Visual Surveillance Systems A Brief of Our EV-Surv System Case Studies
EV-Retrieval EV-Tracking
A Broader View of Our EV-Surv System Final Remarks
16/55
Case Study I: EV-Retrieval
Introducing electronic signals to help sort out videos for accurate and efficient person identification
Jin Teng, Junda Zhu, Boying Zhang, Dong Xuan and Yuan F. Zheng, “E-V: Efficient Visual Surveillance with Electronic Footprints”, IEEE INFOCOM 2012
17/55
Person Identification How a person of interest looks like at the time of surveillance
May be very different from any image or video of him in record A type of retrieval
Reference Photo
Appearance in video
A missing child A crime suspect
Problem:Given the identity of a person,
find his appearance
18/55
Traditional ways
Let people search through a huge pile of videos If done automatically, computers need to extract all
human figures in each frame and compare.
Too Costly
19/55
E Signal-assisted Retrieval With complementary electronic information, we can
find out the electronic identity of the person, and use that information to guide our visual search
In critical surveillance context, it is possible to acquire this information
E.g., the government can request service providers to disclose the user information in anti-terrorism operations
Or, sometimes, it may be the case that we only have an electronic identity
FBI gets a suspicious conversion through phone tapping
Much smaller search space after screening with E signals!
Less processing burden on the V side
20/55
Problem Formulation: Notations V-sensing: V-ID and V Frame
V-ID: Visual identity, such as human figure VID*: Our target V-ID V Frame: a set of V-IDs with some background captured
by visual sensors (cameras) in certain area and time
E-sensing: E-ID and E Frame E-ID: Electronic identity such as MAC address etc. EID*: Our target E-ID E Frame: a set of E-IDs captured by electronic sensors in
certain area and time
21/55
Problem Formulation Input: EID*, and a set of E frames and corresponding
V frames Output: VID* in video frames
We consider a baseline case with perfect E-IDs and V-IDs All E-IDs and V-IDs are clear and distinguishable. No
ambiguity. No false positives or negatives in the detection and
extraction of E-IDs and V-IDs We will discuss more practical cases later
22/55
A Basic Solution
Three steps:Step 1: Find out all E frames which include EID*Step 2: Find a subset of E frames, whose intersection
is EID*Step 3: Identify VID* in their corresponding V frames
Comments: Few V frames to process because V frames without VID* are filtered out, but there may be still many V frames
23/55
E frame 1
EID*
EID2
EID3
E frame 1
EID*
EID12
EID3
Example
E frame 1
EID*
EID2
EID3
E frame 1
EID*
EID2
EID7
E frame 1
EID*
EID2
EID3E frame 1
EID*
EID2EID3
E frame N+1
EID*
EID91
EID13
E frame 1
EID*
EID2
EID4
EID4E frame 1
EID*
EID2
EID3……
E frame 2
EID*EID2
EID4
E frame 1
EID* EID1
Millions of E Frames
E frame 3
EID* EID2
E frame 4
EID* EID5……
Step 1:Extract all E frames with EID*
24/55
E frame 1
EID* EID1
Example (cont’d)
E frame 2
EID* EID2
EID3
E frame 3
EID* EID2
E frame 4
EID* EID5
E frame 1EID1
E frame 2
EID2
EID3E frame 3
EID*EID2
V frame 1VID1
V frame 2
VID2
VID3V frame 3
VID*VID2
Step 1
Step 2
Step 3
25/55
Find the minimum number of E Frames, whose intersection is the given E-ID, i.e. EID*
Further less frames for V side processing
A Better Solution
EID*
E frame 1
EID* EID1
E frame 3
EID* EID2
E frame 2
EID* EID3
EID2Two E Frames are enough identify EID* through intersection.
E frame 1
EID1
E frame 2
EID3EID2
26/55
Nature of E-Filtering Finding the minimum number of frames, whose
intersection is EID* NP-complete: equivalent to the set cover problem
Whether each E-ID appears in each E frame is summarized in a matrix, with 1 meaning ‘appear’ and 0 ‘not appear’.
At least one 0 in each non-EID* column Use these 0s to ‘cover’ all non-EID* column (next page)
EID* EID1 EID2 EID3
e1 1 1 0 0
e2 1 0 1 1
e3 1 0 1 0
At least one 0 in each non-EID* column
27/55
Reduction to Set Cover ProblemEID* EID1 EID2 EID3
e1 1 1 0 0
e2 1 0 1 1
e3 1 0 1 0
E frame 1
EID3EID2
E frame 2
EID2
E frame 3
EID1 EID3
EID1
EID2
EID2
Set to be covered
Sets to cover
Set Cover Problem: Find the fewest sets from a pool of sets (left hand side), whose union includes the set to be covered (right hand side)
Cover
28/55
Solution: EDP Algorithm Element Distinguishing Problem (EDP)
The element to be distinguished is EID*
Greedily select E Frames in which the most number of E-IDs can be told apart from EID* In the example, the greedy algorithm will select e1 or e3
first, because we can tell two E-IDs are not EID* Repeat the greedy selection until EID* is distinguishable
EID* EID1 EID2 EID3
e1 1 1 0 0
e2 1 0 1 1
e3 1 0 1 0
29/55
EDP(cont’d) Approximation results can be achieved with the greedy
heuristic algorithm for the set cover problem
30/55
VID* Retrieval
Find the corresponding VID* from the frames selected by E-Filtering.
VID* is the only one that should appear in all the frames after E filtering.
Find all V-IDs in the selected frames, then an intersection operation can give VID*.
31/55
More Cases Vagueness and completeness of V-ID/E-ID
Vagueness: reflect how clearly a V-ID/E-ID can be identified Completeness: reflect if V-IDs/E-IDs are complete in a V/E frame
(false positive/negative)
√
√ The baseline case we have studied
□ practical case of our focus addressed
Input Target Input Frames
EID* VID* EIDs VIDs
Vagueness Clear □ □
Vague □ □
Completeness Complete
Incomplete □ □
√
√
√
√
√√
32/55
Practical Case I: Handling Vague V-IDs
Vague V-IDs Do not know for sure which person is which in different
frames
Difficulty in the intersection of V frames to find VID* Solution
nBM algorithm: find the VID with the largest probability of appearing in all V frames.
Same?
33/55
The nBM Algorithm n-partite Best Match Problem (nBM)
Put all VIDs in different frames in n different circles
n-partite graph (right)
Similarity matrix for all V-IDs which have appeared
1VID
1v
2v
1VID
2VID 4VID
3VID 5VID
3v
1VID
6VID
7VID 9VID
8VID
VID1 VID2 VID3 VID4 VID5 VID6
VID1 N/A 0 0.9 0.34 0.1 0.76
VID2 0 N/A 0.12 0.51 0.72 0.23
VID3 0.9 0.12 N/A 0 0.85 0.12
VID4 0.34 0.51 0 N/A 0.23 0.35
…
VID(m-1)
0.1 0.72 0.85 0.23 N/A 0
VIDm 0.76 0.23 0.12 0.35 0 N/A
v1 v2 v3
34/55
nBM (cont’d) Maximum Likelihood matching
Given the observed VID1 … VIDm Which VID is the best candidate
Calculate the probability of all VIDi across all V frames Select the VID with the largest probability
1VID
1v
2v
1VID
2VID 4VID
3VID 5VID
VID1 is not in v2
VID1 is in v2, and appears as VID2
35/55
Solutions to Other Practical Cases Careful Deployment
Make sure that the coverage of the camera and the wireless detectors are roughly the same
nBM is probability based, so it is naturally resistant to noises Select appropriate threshold in nBM for better tradeoff between noise
resistance and performance
Generalized EDP Handle missing/ghost E-ID Introduction of fuzzy logic to improve the robustness of EDP Use RSSI for estimation and smoothing
EID* EID1 EID2 EID3 EID4
e1' 0.98 0.95 0.1 0.01 0.06
e2' 0.9 0.01 1 0.94 0.04
e3' 0.88 0.99 0.03 0.1 0.12
e4' 0.99 0.02 0.89 0.27 0.23
EIDi 10
EIDi 1010
10
smoothing
smoothing
Time
EIDi
EIDi
36/55
Implementation
Real world implementationOne camera viewing from above to collect V frames1-3 laptops around sniffing the WiFi traffic to
collect E frames Tested on campus
GymnasiumLibrary
37/55
Combine E and V signals for more accurate localization and tracking
Preliminary work: EV-Loc
Case Study II: EV-Tracking
Boying Zhang, Jin Teng, Junda Zhu, Xinfeng Li, Dong Xuan and Yuan F. Zheng. “EV-Loc: Integrating Electronic and Visual Signals for Accurate Localization”, in ACM MobiHoc12.
38/55
EV-Loc for Localization Basic idea
Simultaneous E and V localization Same localization result E-IDi matches V-IDj Can collect localization results over time and perform
statistical matching
Visual Localization
Electronic Localization
V-IDj
E-IDi
39/55
Multiple Objects Localization
Minimize sum of localization differences
Localized E object
Localized V objectx1
x2
x3
y3
y1
y1
xi, yi: Localization results (coordinates)
x1 x2
y1
x3
y3y2
π=(3, 1, 2, 4)
x4
y4
x4
y4
40/55
Nature of The Problem Linear assignment problem (bipartite matching)
Minimum matching cost:argmin πi Σ||xi - yπi|| = ||x1 - y4|| + ||x2 - y3|| + ||x3 - y2|| + ||x4 – y1||
We can use the Hungarian algorithm to solve this optimization problemObjects’ EIDs Objects’ VIDs
e1
e2
e3
e4
v1
v2
v3
v4
41/55
In EV-Retrieval, we only use 0/1 to indicate E-IDs’ existence in E frames
However, we can improve it with EV-LocE-ID’s existence in a much smaller regionEasier for intersection
EV-Loc for EV-Retrieval
42/55
EV-Loc for EV-Retrieval (cont’d)
E frame 1
EID* EID1
E frame 2
EID*EID1
Original Scheme Now with Localization
EID1
EID*E frame 3
EID* EID1
EID1
EID0
Possible location
E frame 1
EID* EID1
E frame 2
EID* EID1
E frame 3
EID* EID1
Suppose all basestations can hear all mobile devices
43/55
Outline
Deficiency of Visual Surveillance Systems A Brief of Our EV-Surv System Case Studies A Broader View of Our EV-Surv System Final Remarks
44/55
EV Surveillance: Problem Space
TrackingOnsite Offline
Cooperative
Uncooperative
45/55
Problem Space (Cont’d) X: Tracking: offline or onsite Y: Object of monitoring:
Individual : non-coordinative Group: coordinative, tied by relation in terms of vicinity, social relationship,
appearance, action/behavior
Z: Object friendliness: Cooperative: E/V signals on purposely for easy tracking Uncooperative: neutral (E/V signal on/off follows its own cause), and even
misleading
Other possible dimensions: objects can be human and vehicles etc.
46/55
Typical Cases Case 1: <X: offline tracking, Y: individual object, Z: cooperative), i.e.
offline tracking of individual and cooperative object E.g. missing elders searching
Case 2: <X: offline tracking, Y: group objects, Z: cooperative) , i.e. offline tracking of group and cooperative objects E.g. public health monitoring
Case 3: <X: onsite tracking, Y: individual/group objects, Z: cooperative), i.e. onsite tracking of individual/group and cooperative objects E.g. sports training/traffic monitoring
Case 4: <X: offline/onsite tracking, Y: individual object/group objects, Z: uncooperative, i.e. tracking of uncooperative objects E.g. criminal tracking
47/55
Open Issues
E-V sensor deployment Electronic signal capturing E-V data analysis Privacy Real-world implementation and evaluation
48/55
Electronic and Visual Sensor Deployment
Electronic sensor deployment problem
How to do E/V sensors joint optimal deployment?
Visual sensor deployment problem
49/55
Mobile Devices emit electronic signals at different time
Electronic interference is always there Non-cooperative targets exist Different electronic signal capturing times vary
Electronic Signal Capturing
Wi-Fi Bluetooth RFID
Capturing Time 1~2 seconds 10.24 seconds 0.1 seconds
50/55
E-V Data Analysis
E-V integration discussed so far is simple E-Filtering aiming to minimize the number of V
frames not necessarily results in best performance E-Filtering an V-Retrieval should be integrated
together to get the “best” number of V frames for “best” VID* identification
Data mining on a huge, distributed but incomplete E-V data sets
51/55
Visual surveillance has been widely accepted (or tolerated)
How about electronic surveillance?E-IDs can be inferred to expose user’s real identity
Privacy
52/55
The key to successful E-V integration technologies
Small scale, medium scale and then large scale real-world implementation and evaluation
Real-World Implementation and Evaluation
53/55
Final Remarks Existing visual surveillance system is not efficient Our EV-Surv system
Integrates the E signals and V signals for efficient visual surveillance
Implemented in real world
Many open issues left, still a long way to go
54/55
References
[1] Big Apple is Watching You: http://www.slate.com/articles/news_and_politics/explainer/2010/05/big_apple_is_watching_you.html
[2] http://articles.chicagotribune.com/2010-05-06/news/ct-oped-0506-chapman-20100506_1_surveillance- cameras-vandalism-effect-on-violent-crime
[3] http://news.bbc.co.uk/2/hi/4659093.stm
[4] D. Smith, et.al, “Approaches to Multisensor Data Fusion in TargetTracking: A Survey”, Knowledge and Data Engineering, IEEE Transactionson, 2006.
[5] S. Cho, et.al, “Association and Identification in HeterogeneousSensors Environment with Coverage Uncertainty”, IEEE AdvancedVideo and Signal Based Surveillance, 2009.
55