1

CLUE Framework Issues

CLUE virtual interim meetingJan 27, 2014

Mark Duckworthdraft-ietf-clue-framework-13

2

Spatial info within composed MCC

• The MCC addition to the framework added some text about describing layout within a composition.

• Ticket #5 already closed, saying we don’t want to address describing layout within a composition.

• Design team discussion on Jan 14 agreed.• Is there consensus that the framework should say

the consumer should not assume anything about video layout within a composed MCC, and an advertisement does not provide this information?

3

MCC Referencing Other Captures

• Framework already says MCC can refer to other captures to be included in the MCC, for both advertisement and configure messages

• Proposal – update framework to clarify that framework examples are conceptual, and not necessarily using the syntax from the data model

• Continue the discussion about “MCC grouping labels” and regular expressions for captureIDs in the context of the data model

• Does the group agree?

4

MCC Example clean up

• discussed on list, example section 12.3.3• part of broader multipoint switching

discussion• Example needs some cleanup – best way could

depend on results of broader multipoint switching discussion and issues.

5

Switched Capture Example

• Topo-Mixer – media switching variety– see draft-ietf-clue-rtp-mapping

and draft-ietf-avtcore-rtp-topologies-update• Mixer (middle box) provides conceptual

sources (Media Captures), selecting one source at a time from the original sources

6

Example Video Layout• This endpoint is

receiving 9 video captures.

• MCC1, MCC2, MCC3 has the current talker

• MCC4 – MCC9 has other recent talkers

MCC1 MCC2 MCC3

MCC4 MCC5 MCC6 MCC7 MCC8 MCC9

Blue person starts talking, mixer switches video

London ParisMCC1 MCC2 MCC3

MCC4 MCC5 MCC6 MCC7 MCC8 MCC9

7

Switching Mixer Diagram• Mixer sends 9 Video

Captures to A• Mixer selects which

sources to send– based on its own

policy– or based on explicit

requests from the consumer

Mixer

A

B

D

C

E

RTP Rewrite

9

31

3

2

FG

11

9

8

Different Approaches

1. Provider (mixer) makes switching decisions, mixer's advertisement has one big capture scene with many MCCs, plus other information about the original source scenes and captures.

2. Same as above, but Consumer (terminal) makes switching decisions (group seems not interested in this, so won’t discuss today).

3. Provider (mixer) makes switching decisions, mixer's advertisement has several smaller scenes with spatial information.

9

Approach 1 – Advertisement to A• Scene1:

– CSE1(video): MCC1(VC1,VC4,VC6,VC9,VC10,VC11)– MCC2(VC2,VC5,VC7,VC9,VC10,VC11)– MCC3(VC3,VC8,VC9,VC10,VC11)– MCC4, … MCC9– CSE2(audio): MCC20(), MCC21(), MCC22() (three dominant talkers, each spatially

covering whole scene?) – if Adv has these VCs for each MCC, it might as well use Approach 3 with separate scenes– MCCs can have spatial attributes – Can this really work?

• Scene2 (B) CSE1: VC1, VC2, VC3• Scene3 (C) CSE1: VC4, VC5• Scene4 (D) CSE1: VC6, VC7, VC8• Scene5 (E) CSE1: VC9• Scene6 (F) CSE1: VC10• Scene7 (G) CSE1: VC11• Consumer picks number of captures it wants to receive

10

Approach 2 – Advertisement to A

• Advertisement could be the same as Approach 1, or it could be more flexible and allow any VC in each MCC

• Scene1:– CSE1: MCC1(<all individual captures>)– MCC2(<all individual captures>)– MCC3(<all individual captures>)– MCC4,MCC5,MCC6,MCC7,MCC8,MCC9– MCCs don’t have spatial attributes

• Consumer sends Configure messages to select which VC it wants in each MCC, and sends a new message every time it wants a change

11

Approach 3 – Advertisement to A• Each scene represents another endpoint, or small group of endpoints• Capture Scene 1 (current talker, higher priority):

– CSE1: MCC1(VC1,VC4,VC6,VC9,VC10,VC11) – left, soundlevel:0– MCC2(VC2,VC5,VC7,VC9,VC10,VC11) – center, soundlevel:0– MCC3(VC3,VC8,VC9,VC10,VC11) right, soundlevel:0– CSE2(audio): MCC20(), MCC21(), MCC22() (three dominant, spatially each covers whole

scene)• Capture Scene 2: (lower priority, less recent dominant speakers)

– CSE1: MCC4(VC1,VC4,VC6,VC9,VC10,VC11) – left, soundlevel:1– MCC5(VC2,VC5,VC7,VC9,VC10,VC11) – center, soundlevel:1– MCC6(VC3,VC8,VC9,VC10,VC11) – right, soundlevel:1

• MCCs do have spatial attributes• More scenes like the first two, with lower priority, higher soundlevel#• Could also be done without advertising VC1 – VC11 at all. In that case, MCCs

could just be normal VCs I think.

12

1 Provider choose, one large scene

2 Consumer choose

3 Provider choose, multiple scenes

Need to use MCC and Adv all VC

Yes Yes No, but might want to for other reasons (spatial audio/video relations?)

Need to relate incoming packets to original source MC

Yes, to get original spatial info

Yes, if using audio streams as basis for switching

No, but might want to for other reasons

Consumer needs recent speaker info

No Yes No

Disadvantages Encodings can remap to different rendering area with each switch.

Provider and Consumer make more assumptions about what the other is doing

Advantages Consumer knows best how to choose, based on it’s desired UX

Simpler in terms of protocol messages

13

Associate Rx streams with captures

• Need ability for consumer to associate received encodings with CLUE captures

• Has support on the email list• Issue for signaling and RTP mapping, but should

be summarized in framework• Associate encoding in RTP with provider’s

encoded capture (MCC or individual capture)• Associate encoding in RTP with the original

capture, which is a constituent of provider’s MCC

14

Consumer needs “recent talker” info(not discussing today)

• Related to “approach 2” of switched capture discussion, for case when consumer chooses.

• Can the consumer get the info it needs from just the audio encodings it receives?

• Or should there be a more explicit signaling mechanism for the provider to give this to consumer?

• Note this general mechanism is already deployed in Microsoft Lync, and seems to work well

15

Need “active capture” info• Signaling the “active video capture”, not using just audio energy in the

audio stream. Old issue discussed a couple years ago in the working group

• Example: endpoint provider sends 3 video captures, and 1 mono audio– Consumer (MCU topo-mixer) wants to know which video has the loudest

talker, so it can send that one to a simple endpoint that just receives one video stream

– How does this MCU consumer know which one it is?• Provider can signal this somehow, if it knows locally which video is

associated with the talker– in RTP?– in CLUE channel?

• Relevant for the MCC example in framework 12.3.3

16

“overloading” media channels

• Editor note in section 5, and section 10• Propose clarifying this, to be consistent with

signaling document• m-line is either non-CLUE or CLUE, not both• CLUE can cause changes in encoded streams,

but must be compliant with most recent SDP O/A

17

RTP multiplexing

• Editor note in section 5, about multiplexing and RTP mapping.

• Propose remove the sentences about multiplexing, as the note suggests

18

Open Tickets

• #10 sufficient info for receiver – ready to close this?• #19, #26 site switching – close after MCC example

cleanup?• #32 Roles – Christian proposed new terms, will

propose text for framework based on design team Dec 17.

• #33 Security – Mary has proposal, need to add it to framework

• #35 consistency with data model - ongoing