1 MMI Man Machine Interface Ohad Stein Sefi Epel Technion - IsraelSummer 2002

1

MMIMan Machine Interface

Ohad Stein

Sefi Epel

Technion - Israel Summer 2002

2

Introduction


•It is known for years that stress, press, fear, happiness and many other feelings, can be measured by several techniques .

•Based on this knowledge, we would like to build an interface, that will enable people to communicate without using any muscle, and without making any physical effort.

•Our interface, will enable communication by being a “mind-reader”. Detection of voltage deflections measured from subjects arms, will enable us to estimate the subject’s will.

3

How can we do it?

•One way, is to measure the resistance of the skin, to the passage of a very small electric current.

•There are several ways to “speak” with the machine:

•Our way was to measure the change in DC potential across neurones of the autonomic nervous system connected to the sensori-motor strip of the cortex.


•The main disadvantage of this technique is that it is hard to calibrate the system. The measured values changes between different subjects, and between different times with the same subject.

•In this way – we will detect a signal only after causing it.

4

Part 1: Experimental backgroundSummer 2002

•On the following slides, 3 basic experiments are shown.

•In order to gain some basic knowledge about the MMI abilities we have done some basic experiments, in several different ways.

Technion - Israel

•The first and second experiments have a strong emotional effect. We did them in order to have a basic indication about the ability of building a MMI.

•The third experiment do not have any emotional aspect. The subject should try to control the voltage deflections measured from his arm.

•The third experiment is actually the prototype for our project.

5

•Exp #1: “Pictures Experiment” – several pictures where shown to the subject, changes on the skin voltage ratio where measured according to the pictures emotional effect.

•Be aware that the fall of the voltage ratio is a short time after the “strange/unusual” picture is shown.

Pictures experiments

6

5 10 15 20 25 30Seconds


7

5 10 15 20 25 30Seconds


8

•Exp #2: “Voice Experiment” – several words were announced to the subject, different voltage ratio was measured according to the words emotional effect.

•The words that did not affect the subject were words like “silence”, ”beautiful”, ”sunrise”, and the words that did affect the subject were “war”, “accident” and etc.

•Note that in this experiment the result can be affected by screaming, changing tone, speaking quietly and etc.

Voice experiment


9

5 10 15 20 25 30Seconds

GOOD GOOD GOOD GOODBAD!!!

The person heard 5 different good/bad words. Here are the results:


10

5 10 15 20 25 30Seconds

GOOD GOOD GOOD GOODBAD!!!

The person heard 5 different good/bad words. Here are the results:


11

•Exp #3: “Letters Experiment” – several letters were shown to the subject, one by one . Different voltage ratio was measured according to the picture emotional effect.

•Exp #3 is a much more complicated experiment. The emotional effect is dependent on the assistant decision and concentration.

Letters experiment


12

•Based on Exp# 3 – we will try to prove the ability of the man to write down a word without using a pen, pencil, keyboard or anything else. Just by measuring the voltage ratio on his arm ! ! !

•The red letter shown in the following graphs, is the letter that the subject was waiting to be shown.


13

0

6 11 17 22 27 33

M Q K F T C

Second

6 11 17 22 27 33

M Q K F T C

Second


14

0 Second6 11 16 22 27 33

B O L S W U

0 Second6 11 16 22 27 33

B O L S W U


15

The experiments success

•As expected, not all the experiments succeeded.

•There were some reasons that caused it:

1. The training session was not long enough.

2. The session training was not led by Bio-feedback experts.

3. Not always the system was calibrated.

4. The low-frequency signal is not reduced by the system.

•We had about 70% successful experiments.An experiment is characterized “successful”, if a simple algorithm can detect which simbol the subject chose.

16

0 Second5 11 16 22 28 33

T Q K M C F


In this slide, you can see the importance of reducing the low-frequency signal.

17

0 Second6 11 16 22 27 33

B O L S W U


Here the calibration system is missing, you see the “initializing” effect.

18

0 6 11 17 22 27 33

M Q K F T C

Second

In this slide it is hard to detect the effect.


19

5 10 15 20 25 30Seconds


In this slide there isn’t any effect.

20

Part 1 Summary

•Basic experiments indicate that there is a linkage between the emotions and the voltage deflections measured from our skin.


•Emotions can be evoked by pictures and sounds.

•Emotions can be controlled by the subject decision.

21

Part 2 – Bio-Feedback

•Every hospital uses, nowadays, biofeedback experts in the pain clinics, in order to help the people help themselves.

•The technique is based on a long training session, led by experts, in order to learn the “patient emotionalism”, and the special biofeedback way that can be useful for him.

•The meaning of “Bio-Feedback” is that man can “learn” his reflexes, can control them, and can use them as a way of therapy.


22

• There could be some differences between patients:

4. In the time of training session which is needed.

3. In the psychological ways of the training.

2. In the detection way.

1. In the recording site.


23

•In order to improve our biofeedback abilities, and after consulting with some experts from the academy and from the medicine area, we have built 3 different train tools for the training.

1. A balloon. The balloon radius is affected by the measured voltage ratio, of the subject skin. The subject should try to increase the balloon radius.


24

The balloon radius changes , due to voltage ratio change

25


26


27

2,3. A beeper and a colored column. The column height and the beeper tone are the feedback in this tool.


28

29

30

31

32

Part 3 – Visual BasicReal-Time implementation

•In this part of the project we have build an real-time user-interface, for our implementation.

•Stage #1 – a matrix is shown to the subject. 5 lines of 5 letters each.

•Each line become bold for 5 seconds.

•The subject choose one of the 25 letters, therefore he will “like” one line more than the others (the one that includes the chosen letter…)

•The following slides will show the interface and the effect of the “chosen line” to our mind…


33

The user interface


34

3 6 9 12 15 18Seconds

Line 1 Line 4 Line 5 Line 3 Line 2


35

3 6 9 12 15 18Seconds



36

3 6 9 12 15 18Secon

ds



37

How does it work?

• We can see, that the voltage ratio measured from the subject skin, is affected from the bold line.

• There are some stages on the interface process:

1. Calibration. On the first 15 seconds the system calibrates.


38

2. After calibrating, the system starts with bolding each line in a random order for 5 seconds.

The screen appearance is shown in the following slide.


39

We can see here the bolded line.

Line #3


40

• 3. After choosing a line, by the changes of the voltage ratio on the subject arm, we stop bolding lines and start bolding letters on the chosen line. Now, the subject will be able to choose the wanted letter and print it in the right place. The screen will look as followed.


41

Bolding letters oneby one, in random

order.


42

The chosen letters are written here.


43


Summary

•The main target was to be able to write a whole word in the real-time implementation. We have not completely reached this target.

•We can point at some areas for improvement:

1. A much longer training session.

2. Try to use other Bio-feedback tools.

3. The algorithm should solve some problems that our algorithm did not solve, like reducing the low-frequency signal.

•In accuracy of 70% we are able to write one chosen letter.

•The real-time implementation functionality is not completed.

Documents

1 MMI Man Machine Interface Ohad Stein Sefi Epel Technion - IsraelSummer 2002