Frame Selection Algorithms for Collaboratively Tele-Operated Robotic Cameras

1

Frame Selection Algorithms for Collaboratively Tele-Operated

Robotic Cameras

2

Conventional robotic webcamera

Collaboratively controlled robotic webcamera

3

4

5

Frame Selection Problem: Given n requests, find optimal frame

One Optimal Frame

6

Requested ViewingZones

Optimal Satellite Frame

Oct. 27, 2003

7

Satellite Imaging

• 2.44 Billion Market in 2001

• Increasing 14% per year since 1999

• Major clients– Government / Military– Oil Exploration– Weather Prediction– Agriculture

Ikonos, 1999

8

Related Work• Facility Location Problems

– Megiddo and Supowit [84]– Eppstein [97]– Halperin et al. [02]

• Rectangle Fitting, Range Search, Range Sum, and Dominance Sum– Friesen and Chan [93] – Kapelio et al [95]– Mount et al [96]– Grossi and Italiano [99,00]– Agarwal and Erickson [99]– Zhang [02]

9

Related Work

• Similarity Measures – Kavraki [98]

– Broder et al [98, 00]

– Veltkamp and Hagedoorn [00]

• CSCW, Multimedia – Baecker [92], Meyers [96]

– Kuzuoka et al [00]

– Gasser [00], Hayes et al [01]

– Shipman [99], Kerne [03], Li [01]

10

Problem Definition• Assumptions

– Camera has fixed aspect ratio: 4 x 3– Candidate frame c = [x, y, z] t

– (x, y) R2 (continuous set)– Resolution z Z

• Z = 10 means a pixel in the image = 10×10m2 area • Bigger z = larger frame = lower resolution

(x, y)3z

4z

11

Problem Definition

Requests: ri=[xli, yt

i, xri, yb

i, zi], i=1,…,n

(xli, yt

i) (xri, yb

i)

12

Optimization Problem

n

iii

zyxcrcsS

1],,[

),(max

User i’s satisfaction

Total satisfaction

13

Problem Definition• “Satisfaction” for user i: 0 Si 1

Si = 0 Si = 1

= c ri c = ri

14

• Measure user i’s satisfaction:

)1),/min(()/(

1,)(

)(min

)(

)(),(

zzap

cResolution

rResolution

rArea

rcAreacrs

iii

i

i

ii

Coverage-Resolution Ratio Metrics

Requested frame ri

Area= ai

Candidate frame c

Area = a

pi

15

Comparison with Similarity Metrics

• Symmetric Difference

• Intersection-Over-Union

SDcrArea

crAreaIOU

i

i

1)(

)(

)(

)()(

crArea

crAreacrAreaSD

i

ii

Nonlinear functions of (x,y), Does not measure resolution difference

16

Optimization Problem

n

iii

zyxcrcs

1],,[

),(max

17

Co-Opticon Problem Versions

• Fixed Resolution Exact Algorithm

• Variable Resolution Exact Algorithm

• Approximate Algorithm for Arbitrarily-Shaped Requested Frame

• Distributed Algorithms

D. Song, A.F. van der Stappen, and K. Goldberg, Exact and Distributed Algorithms for Collaborative Camera Control, the Fifth International Workshop on Algorithmic Foundations of Robotics. Nice, France, Dec 15~17, 2002.

18

),(),( yxpyxs iii

),( yxpi

Requested Frame ri Candidate

Frame c

)1,/min()/(),( zzapcrs iiii

(for fixed z)

Objective Function Properties

19

• si(x,y) is a plateau

• One top plane• Four side planes• Quadratic surfaces at corners• Critical boundaries: 4 horizontal, 4 vertical

Objective Function for Fixed Resolution

4z x

y

3z

4(zi-z)

20

Objective Function• Total satisfaction:

n

iii

n

iiii

yxpyxS

zzapcS

1

1

),(),(

)1),/min(()/()(

for fixed z

Frame selection problem: Find c* = arg max S(c)

21

S(x,y) is non-differentiable, non-convex, non-concave, but piecewise linear along axis-parallel lines.

Objective Function Properties

4z x

y

3z

4(zi-z)

3z y

si

3z

(z/zi)2

3(zi-z)

x

si

4z 4z

(z/zi)2

4(zi-z)

22

Plateau Vertex Definition• Intersection between boundaries

– Self intersection:– Plateau intersection:

y

x

23

Plateau Vertex Optimality Condition

• Claim 1: An optimal point occurs at a plateau vertex in the objective space for a fixed Resolution. Proof:– Along vertical boundary, S(y) is a 1D piecewise

linear function: extrema must occur at x boundaries

y

S(y)

24

Fixed Resolution Exact Algorithm

Brute force Exact Algorithm:

Check all plateau vertices (n2) plateau vertices(n) time to evaluate S for each (n3) total runtime

25

Improved Fixed Resolution Algorithm

• Sweep horizontally: solve at each vertical – Sort critical points along y axis: O(n log n)– 1D problem at each vertical boundary O(n) – O(n) 1D problems– O(n2) total runtime

O(n) 1D problems

y

S(y)

x

y

26

Speed comparison

Random inputs

Curve B:

Brute force approach

Curve V:

using line sweeping

27

More Improvements for Fixed Resolution

• Har-Peled, Koltun, Song, and Goldberg. [03] – Exact algorithm O(n3/2 log3 n)

– Near Linear Approximation Algorithm O(NlogN)• N = O(nE)

• E = (log(1/ε)/ε)2, where ε is the approximation bound

S. Har-Peled, V. Koltun, D. Song, and K. Goldberg, Efficient Algorithms for Shared Camera Control, In Proceedings of the 19th ACM Symposium on Computational Geometry, 2003.

28

Co-Opticon Problem Versions

• Fixed Resolution Exact Algorithm

• Variable Resolution Exact Algorithm

• Approximate Algorithm for Arbitrarily-Shaped Requested Frame

• Distributed Algorithms

Dezhen Song, A. Frank van der Stappen, and Ken Goldberg, An Exact Algorithm Optimizing Coverage-Resolution for Automated Satellite Frame Selection, (To appear) IEEE International Conference on Robotics and Automation (ICRA) 2004

29

Virtual Corner• A two-requested frame case

– Requested frame:

y

x

30

Virtual Corner • Virtual corner definition

– Real corner:– Extended edge intersections:

y

x

31

Recall: Plateau Vertex Definition• Intersection between boundaries

– Self intersection:– Plateau intersection:

y

x

32

Virtual Corner and Plateau Vertex• Intersection between boundaries

– Candidate frame:– Frame intersection:

y

x

33

Virtual Corner and Plateau Vertex• Intersection between boundaries

– Candidate frame:– Virtual corner:

y

x

34

Variable Resolution Exact Algorithm

• Lemma: At least one optimal frame has its corner overlapped with virtual corner.– O(n2) Virtual corners– One 3D problem→ O(n2) 1D sub problems

r6

r2

r5

r3

x

y

r4

r1

O0.00

0.40

0.80

1.20

1.60

0 20 40 60 80 100 120 140 160z

S(z)

Candidate frame

35

• Overall complexity– O(n2) 1D problems– O(n) sub 1D problems– O(n) to compute

polynomial coefficient for each sub 1D problem

s(z) = g0z-1+g1+g2z +g3z2

– O(1) to compute the max s(z) for each polynomial

– O(n4) in total 0.00

0.40

0.80

1.20

1.60

0 20 40 60 80 100 120 140 160

S(z)

z

36

0.00

0.40

0.80

1.20

1.60

0 20 40 60 80 100 120 140 160

Improved Variable Resolution Exact Algorithm

• Incremental computing– Computing polynomial

coefficients• O(n) for first smooth

segment, • O(1) for additional• Introduce sorting cost

– O(n log n) for each virtual corner

– O(n3logn) total

S(z)

z

37

Improved Variable Resolution Exact Algorithm

• Diagonal Sweeping– No need to do

sorting for each virtual corner

– O(n) to get new sorted sequence

– Total complexity O(n3)

x

y

O(a)

O

y

r1 r2

x(b)

x(c)

y

x(d)

O O

r1 r2

r1 r2

Order of VCs

y

r1 r2

38

Speed comparison

Random inputs

0

10

20

30

40

50

60

20 30 40 50 60 70 80

Seco

nds

O (n 4 )

O (n 3 logn )

O (n 3 )

n

Brute force approach

Using Incremental computing

Using incremental computing and

diagonal sweeping

39

Co-Opticon Problem Versions

• Fixed Resolution Exact Algorithm

• Variable Resolution Exact Algorithm

• Approximate Algorithm for Arbitrarily-Shaped Requests

• Distributed Algorithms

D. Song, K. Goldberg, and A. Pashkevich, ShareCam Part II: Approximate and Distributed Algorithms for a Collaboratively Controlled Robotic Webcam, IEEE/RSJ International Conference on Intelligent Robots and Systems, 2003.

40

Arbitrarily-Shaped Requested Frame

Requested frames

41

Approximation Algorithm

n)dd

whgO(

spacing resolution :

spacing lattice :

z2

zd

d x

y

d

Compute S(x,y) at lattice of sample points:

w, h : width and height,g: size range

42

Approximation Bound Definitionc* : Optimal frame

: Optimal at lattice (Algorithm output)

c~

)~()( * cscs

?)(

)~(1

*

cs

cs

43

Derive Approximation Boundc* : Optimal frame

: Optimal at lattice (Algorithm output)

c~

: Smallest frame at lattice that encloses c*

c

)ˆ()~()( * cscscs

?)(

)ˆ(

)(

)~(1

**

cs

cs

cs

cs

44

Derive Approximation Boundc* : Optimal frame

: Smallest frame at lattice that encloses c*

c

?)(

)ˆ(*

cs

cs

• fully enclose c*

What is the ratio between their objective functions if one candidate frame is enclosed by the other?

c

45

Approximation Bound

Requested frames

46

Approximation Bound

c

Requested frames

Candidate frame

z

zcrscs

zz

zzap

crs

n

iin

iii

i

iii

ii

1

1

),()(

/

)/)(/(

),(

47

Approximation Bound

ca

cb

Requested frames

Candidate frames

b

n

iib

a

n

iia

zzcs

zzcs

/)(

/)(

1

1

b

a

a

b

z

z

cs

cs

)(

)(

48

Approximation Bound

b

a

a

b

z

z

cs

cs

)(

)(

ca

cb

Requested frames

Candidate frames

49

Derive Approximation Boundc* : Optimal frame

: Smallest frame at lattice that encloses c*

c

z

z

cs

csˆ)(

)ˆ( *

*

What is the resolution ratio between a candidate frame and the smallest frame on the lattice that encloses it?

50

Approximation Algorithm

ca

cb

za

a

b

a

z

dz

z

z

z

dd

2

3set

dz: Lattice spacing in z axis

d

d

51

Approximation Algorithm

– Run Time: – O(n / 3)

c* : Optimal frame

: Optimal at lattice (Algorithm output)

c~

: Smallest frame at lattice that encloses c*

c

)ˆ()~()( * cscscs

)(

)ˆ(

)(

)~(1

** cs

cs

cs

cs

zdz

z

2...

min

min

1

52

Speed Comparison

Random inputs

0

10

20

30

40

50

60

20 30 40 50 60 70

ε=0.2

ε=0.1

ε=0.05

ε=0.035

Exact algorithm

ε=0.2

Time

(Sec.)

#Requests

53

Enclosing the Optimal

c* : Optimal frame

: Frame at lattice that encloses c*

c

z

z

z

z

cs

csˆˆ)(

)( min*

*

54

Cutting in Feasible Set

xy

z)',','(' zyxc

x

y

c’

Screen Space Solution Space

Φ

Φc’

55

Branch and Bound on Lattice

y

z

xkdz

dz

kdz

Layer 1

Layer 2

Layer 3

Survived nodes

Deleted nodes

56

Speed Improvements

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0 0.02 0.04 0.06 0.08 0.1

n=5

n=20

n=40

ε

Grid Basicusing time nComputatio

BnBusing time nComputatio

57

Improvement: Functional Box Sums

• Efficient reporting of

n

i i

ii

n

ii rArea

crAreacscS

11 )(

)()()(

ir

c

jr

kr

[Zhang et al 2002]

58

Origin-Involved Functional Box Sums

=

_ _ +

59

Dominance Sums

13iω

ir(14,8)

(1,3) (9,3)

c)3()1( yxωi

)3()9( yxωi

)(cS )38()114(13

)38()914(13

520

60

Dominance Sum Queries• Data structure:

– ECDF-tree • Guttman (84)

–

–

• ‘Simple’ updates when increasing zoom level

)log( onconstructi 2 nnO)(logquery 2 nO

)log)/1(( 23 nnO

61

Co-Opticon Problem Versions

• Fixed Resolution Exact Algorithm

• Variable Resolution Exact Algorithm

• Approximate Algorithm for Arbitrarily-Shaped Requested Frame

• Distributed Algorithms

62

Distributed Algorithms

63

Distributed Algorithms

• Fixed Resolution Algorithms O(n2)– Server O(nlogn)– Client O(n)

• Approximate Algorithm O(n/3)– Server O(n+1/3)– Client O(1/3)– Robustness to dropouts…