Robotic Cameras and Sensor Networks for High Resolution Environment Monitoring
Ken Goldberg and Dezhen Song
Alpha Lab, IEOR and EECS
University of California, Berkeley
NetworkedRobots
internettele-robot:
RoboMotes: Gaurav S. Sukhatme, USC
Smart Dust: Kris Pister, UCB (Image: Kenn Brown)
NetworkedCameras
Security Applications
Banks, Airports, Freeways, Sports Events, Concerts, Hospitals, Schools, Warehouses, Stores, Playgrounds, Casinos, Prisons, etc.
Conventional Security Cameras • Immobile or Repetitive Sweep• Low resolution
New Video Cameras:Omnidirectional vs. Robotic
• Fixed lens with mirror• 6M Pixel CCD• $ 20.0 K• 1M Pixel / Steradian
• Pan, Tilt, Zoom (21x)• 0.37M Pixel CCD• $ 1.2 K• 500M Pixel / Steradian
Where to look?
Sensornet detects activity
• “Motecams”
• Other sensors:audio, pressure switches,
light beams, IR, etc
• Generate bounding boxes
and motion vectors
• Transmit to PZT camera
Activity localization
Viewpoint Selection Problem
Given n bounding boxes, find optimal frame
Related Work• Facilities Location
– Megiddo and Supowit [84]– Eppstein [97]– Halperin et al. [02]
• Rectangle Fitting – Grossi and Italiano [99,00]– Agarwal and Erickson [99]– Mount et al [96]
• Similarity Measures – Kavraki [98]– Broder et al [98, 00]– Veltkamp and Hagedoorn [00]
Problem Definition
Requested frames: i=[xi, yi, zi], i=1,…,n
Problem Definition• Assumptions
– Camera has fixed aspect ratio: 4 x 3– Candidate frame = [x, y, z] t
– (x, y) R2 (continuous set)– z Z (discrete set)
(x, y)3z
4z
Problem Definition• “Satisfaction” for user i: 0 Si 1
Si = 0 Si = 1
= i = i
• Symmetric Difference
• Intersection-Over-Union
SDArea
AreaIOU
i
i
1)(
)(
)(
)()(
i
ii
Area
AreaAreaSD
Similarity Metrics
Nonlinear functions of (x,y)
• Intersection over Maximum:
),(
)(
),max(
)1,)/min(()/(),(
i
i
i
i
biiii
Max
Area
aa
p
zzaps
Requested frame i , Area= ai
Candidate frame
Area = a
Satisfaction Metrics
pi
Intersection over Maximum: si( ,i)
si = 0.20 0.21 0.53
Requested frame i
Candidate frame
),(),( yxpyxs iii
),( yxpi
Requested frame i
Candidate frame (x,y)
)1,)/min(()/(),( biiii zzaps
(for fixed z)
Satisfaction Function
– si(x,y) is a plateau
• One top plane• Four side planes• Quadratic surfaces at corners• Critical boundaries: 4 horizontal, 4 vertical
Objective Function• Global Satisfaction:
n
iii
n
i
biii
yxpyxS
zzapS
1
1
),(),(
)1,)/min(()/()(
for fixed z
Find * = arg max S()
S(x,y) is non-differentiable, non-convex, butpiecewise linear along axis-parallel lines.
Properties of Global Satisfaction
Approximation Algorithm
spacing zoom :
spacing lattice :
zd
dx
y
d
Compute S(x,y) at lattice of sample points:
Approximation Algorithm
– Run Time: – O(w h m n / d2)
* : Optimal frame
: Optimal at lattice ~
: Smallest frame at lattice that encloses *
)ˆ()~
()( * sss
)(
)ˆ(
)(
)~
(1
**
s
s
s
s
ddz
z
z
min
min...
Exact Algorithm• Virtual corner: Intersection between boundaries
– Self intersection:– Frame intersection:
y
x
Exact Algorithm
• Claim: An optimal point occurs at a virtual corner. Proof:– Along vertical boundary, S(y) is a 1D piecewise
linear function: extrema must occur at boundaries
Exact Algorithm
Exact Algorithm:
Check all virtual corners(mn2) virtual corners(n) time to evaluate S for each (mn3) total runtime
Improved Exact Algorithm
• Sweep horizontally: solve at each vertical – Sort critical points along y axis: O(n log n)– 1D problem at each vertical boundary O(nm) – O(n) 1D problems– O(n2m) total runtime
O(n) 1D problems
Examples
Summary• Networked robots
• High res. security cameras
• Omnidirectional vs. PTZ
• Viewpoint Selection Problem
• O(n2m) algorithm
Future Work
• Continuous zoom (m=)• Multiple outputs:
– p cameras – p views from one camera
• “Temporal” version: fairness– Integrate si over time: minimize accumulated
dissatisfaction for any user
• Network / Client Variability: load balancing• Obstacle Avoidance
Related Work• Facility Location Problems
– Megiddo and Supowit [84]– Eppstein [97]– Halperin et al. [02]
• Rectangle Fitting, Range Search, Range Sum, and Dominance Sum– Friesen and Chan [93] – Kapelio et al [95]– Mount et al [96]– Grossi and Italiano [99,00]– Agarwal and Erickson [99]– Zhang [02]
Related Work
• Similarity Measures – Kavraki [98]– Broder et al [98, 00]– Veltkamp and Hagedoorn [00]
• Frame selection algorithms – Song, Goldberg et al [02, 03, 04], – Har-peled et al. [03]
Problem Definition• Assumptions
– Camera has fixed aspect ratio: 4 x 3– Candidate frame c = [x, y, z] t
– (x, y) R2 (continuous set)– Resolution z Z
• Z = 10 means a pixel in the image = 10×10m2 area • Bigger z = larger frame = lower resolution
(x, y)3z
4z
Problem Definition
Requests: ri=[xli, yt
i, xri, yb
i, zi], i=1,…,n
(xli, yt
i) (xri, yb
i)
Optimization Problem
n
iii
zyxcrcsS
1],,[
),(max
User i’s satisfaction
Total satisfaction
Problem Definition• “Satisfaction” for user i: 0 Si 1
Si = 0 Si = 1
= c ri c = ri
• Measure user i’s satisfaction:
)1),/min(()/(
1,)(
)(min
)(
)(),(
zzap
cResolution
rResolution
rArea
rcAreacrs
iii
i
i
ii
Coverage-Resolution Ratio Metrics
Requested frame ri
Area= ai
Candidate frame c
Area = a
pi
Comparison with Similarity Metrics
• Symmetric Difference
• Intersection-Over-Union
SDcrArea
crAreaIOU
i
i
1)(
)(
)(
)()(
crArea
crAreacrAreaSD
i
ii
Nonlinear functions of (x,y), Does not measure resolution difference
Optimization Problem
n
iii
zyxcrcs
1],,[
),(max
),(),( yxpyxs iii
),( yxpi
Requested Frame ri Candidate
Frame c
)1,/min()/(),( zzapcrs iiii
(for fixed z)
Objective Function Properties
• si(x,y) is a plateau
• One top plane• Four side planes• Quadratic surfaces at corners• Critical boundaries: 4 horizontal, 4 vertical
Objective Function for Fixed Resolution
4z x
y
3z
4(zi-z)
Objective Function• Total satisfaction:
n
iii
n
iiii
yxpyxS
zzapcS
1
1
),(),(
)1),/min(()/()(
for fixed z
Frame selection problem: Find c* = arg max S(c)
S(x,y) is non-differentiable, non-convex, non-concave, but piecewise linear along axis-parallel lines.
Objective Function Properties
4z x
y
3z
4(zi-z)
3z y
si
3z
(z/zi)2
3(zi-z)
x
si
4z 4z
(z/zi)2
4(zi-z)
Plateau Vertex Definition• Intersection between boundaries
– Self intersection:– Plateau intersection:
y
x
Plateau Vertex Optimality Condition
• Claim 1: An optimal point occurs at a plateau vertex in the objective space for a fixed Resolution. Proof:– Along vertical boundary, S(y) is a 1D piecewise
linear function: extrema must occur at x boundaries
y
S(y)
Fixed Resolution Exact Algorithm
Brute force Exact Algorithm:
Check all plateau vertices (n2) plateau vertices(n) time to evaluate S for each (n3) total runtime
Improved Fixed Resolution Algorithm
• Sweep horizontally: solve at each vertical – Sort critical points along y axis: O(n log n)– 1D problem at each vertical boundary O(n) – O(n) 1D problems– O(n2m) total runtime
for m zoom levels
O(n) 1D problems
y
S(y)
x
y
A New Architecture
Activity and Video databaseActivity localization
Activities
Frame selection
Active surveillance
Control commands
Videos
Software diagram
TCP/IP
TCP/IP
Activity & video database
Core (with shared memory segments)
RPC module
RPC module
RPC module
Communication
Console/Log
Activity server
Activity generation
Motescam
Wireless Camera control
Calibration
Panoramic image generation
Video server
Panasonic HCM 280 Camera
Visual C++
NesC + Tiny OS
Gnu C++MySQL
Database: indexing video data
• Activity Index– Timestamp
– Speed (Or other sensor data)
– Range
• Query video data using activity – Show video clips of moving objects with speed faster than
1 meter per second in zone 1 in last 10 days
– Show video clips of zone 1 when CO2 concentration exceeded the threshold in Jan. 2004 (Assuming CO2 sensor is used in detecting activity)
Activity and Video database
Camera control
Recommended
