1

Project:Violin Bowing Trainer

Bow Motion Recordingand Presentation (playback)

Rev. 10/14/2016

2

Project Inspiration• A classroom experience (1996)

– Teacher says (about bowing)• “I can tell you what to do, but can’t tell you how it feels” –

Jack Tuttle, “Fiddler Magazine” music editor• That comment was the inspiration for this project

– Bowing customarily learned by• Teacher demonstrates and/or marks sheet music with bow

direction symbols – student records sound• Home practice: lots of trial and error until sound is right

• Project plan and beginning (2012)– Record bow motion data

• Play back data to run “bow robot”– Student places hand on robot to get the feel

Rev. 10/14/2016

3

The Vision• Bow motion data recording

– Teacher plays violin using a bow rigged with motion sensors

– Student records sound and motion data for use at home

• Data analysis software– Extracts bow velocity (speed and direction)

from motion data– Extracts music notes from sound data– Merges velocity and notes for presentation

and the learning experience

Rev. 10/11/2016

4

Violin Bowing Trainer System

Motion & SoundData

Analysis

RobotMicrocontroller

Motor

4. Commands to position robot bow – synchronized with sound

5. Controls motor 6. Motor positions bow

Teacher plays a tune usingan ordinary violin and

an ordinary bow with attachedmotion sensors (accelerometers)

Rev. 10/08/2016

2. Motion sensors on bowprovide acceleration data

in real time

AccelerationRecording

Device

3. Playback at any speed

1. Teacher moves bow

7. Student “rides” robot

SoundRecording

Device

Sound and accelerationdata are recorded on aneasily portable computer

(cell phone or laptop)

Student takes recordings home for playback to facilitate tune learning

and practice

and listens to sound

and violin produces sound

(or, robotrides student)

5

Scope• What kind of bow motion data is needed

– Primarily: Velocity (Speed and direction)– Other: Bow rotation

• Ways to get data– Video recorder

• Extract position from each frame image• Compute velocity by differentiation

– Bow position sensor (linear encoder of some sort)• Compute velocity by differentiation

– Ultrasound: two transmitters, one receiver• Detect phase difference between two signals• Compute position, then velocity by differentiation

– Accelerometer• Compute velocity by integration

• What data rates are needed for each of the above• Ways to send data to a recorder

– Wired– Wireless (WiFi, Bluetooth, Bluetooth Low Energy)

• Ways to record data– Stereo sound recorder (one channel for motion, other for sound)– Laptop - Windows app– Phone or tablet - Android app

Rev. 10/11/2016

6

7

Analysis• Bow travel – 20 inches max (508 mm)

– 0.2778 sec/cycle (down-up) @ 108 BPM– 128 bow cycles (entire tune – 35.56 sec)

• Gravity acceleration (on Earth at sea level)– 9810 mm/sec/sec

• Observed bow acceleration (in data)– 5000 mm/sec/sec

• Processing units– Acceleration – mm/sec/sec– Velocity – mm/sec– Displacement – mm

Rev. 10/12/2016

8

Data Acquisition• 6 axes (2 accelerometers, 3 axes each)

– 3.0 msec (333 packets per second)• Data packet – 20 bytes

– Sequence number – 32 bit– Time stamp – 32 bit

• 1 count = 100usec (.1 msec) (4 + days before wrap)– 6 axes – 16 bits each

• Data volume– Entire tune – 35.56 sec– 333 packets/sec– 11,841 packets (236,820 bytes, 1,894,560 bits)

• Bow down-up cycle resolution– 0.2778 sec/cycle– 92.5 packets/cycle

Rev. 10/09/2016

9

Means• Problem analysis

– Bow activity upper limit– Data sample rate needed

• Motion sensing alternatives considered– Video recorder– Bow position sensor (linear encoder of some sort)– Accelerometer mounted on bow selected

• Data communication to recorder– Wired– Wireless selected Bluetooth Low Energy

• Recording– Windows app and/or Android app selected both

• Data processing (Windows app)– Motion: filters and integrators– Sound: chromatic spectrum analysis

Rev. 10/14/2016

10

Result Presentation• Graphic

– Notes and motion combined– Standard music notation with bow motion

symbols or slurs• Mechanical

– Bow robot for motion• Safety issues

– Speaker for sound– Variable playback speed

Rev. 10/05/2016

11

The Bow• Bow parts

– Stick and horse hair– Frog (gripping end): hair tension adjuster– Tip (other end)– Electronics added: easy to remove items

• Accelerometer orientation– X-axis parallel to hair (+ “up bow” & gravity effect)– Z-axis perpendicular to hair, passing through stick

and hair (+ gravity effect)• Operation

– Bow cycle: down - up

Rev. 10/12/2016

12

Bow Electronics

Facing playerHand just off photo to right

Button and LEDs upper right

Away from playerHand just off photo to left

Button and LEDs opposite battery

Upper - Original EZ-BLE “breadboard”

Lower – PCB replacement mounted on bow

Frog< Tip Tip >

Rev. 10/12/2016

13

Configuration Evolution• Initial

– Two accelerometers (analog) and connector pins on-bow• wired connection to

– STM32F0 microcontroller off-bow (ST Microelectronics)• wired connection to

– Audio recorder (frequency modulation of velocity data)• Several intermediate variations

– Change from analog to I²C for noise immunity• Still wired to STM32F0 off-bow

– Recording on PC (serial port) or Android phone (Bluetooth classic)• STM32F0 serial port sends acceleration data packets• Bluetooth Mate - Silver (Sparkfun – Roving Networks RN42) (Z.)

• Current (photos – EZ-BLE breadboard, EZ-BLE PCB)– Two accelerometers (I²C) on-bow

• wired connection to EZ-BLE– EZ-BLE microcontroller on-bow (Cypress)

• wireless connection to recorder (Bluetooth Low Energy)– Recording on PC or Android phone

• Data rate issues on Android (and iOS as well)– F-RAM buffer and packet density reduction

Rev. 10/09/2016

14

The Adventure• Knowledge acquisition

– Learn microcontroller application development– Study digital signal processing book– Learn Android application development

• Tool building (Windows app – C#)– Data visualization

• Multi-channel graphs• Scrolling graph

– Simple data flow programming system design• Motion data processing

– Simple recursive filter: gravity and noise elimination– Integration: Acceleration Velocity Displacement

Rev. 10/12/2016

15

Acceleration Signal Processing

Rev. 02/19/2016

High PassFilter

acceleration-rawSum

High PassFilter

gravity removal

with gravity + noise

+-

gravity

Higher PassFilter

noise

Sum+-

acceleration-clean

acceleration-nng

High PassFilter

Integrateacceleration-clean

noise, no gravity

acceleration-nng

velocity velocity-clean

Integratedisplacement

velocity-clean

displacement-clean

residual gravity removal

residual gravity removal

Due to poor quality accelerometer data, gravity and high pass filtering, this

cannot deliver long term absolute x-axis

velocity or displacement

However, short term z-axis motion establishes the bow

rotation pivot point and therefore bow displacement for a correction calculation

Gravity provides bow orientation relative to the gravity vector, but

not to compass direction

Clean is a relativeterm in usage here

- far from ideal

Main output is velocity, the indication of bow

direction

16

ProcessAcceleration(Int32 TimeDelta100usec, Int32 AccelerationRWU)

1 represents .1 millisecond 1 represents 1 millimeter/sec/secGravity: 9810 mm/sec/sec

Rev. 10/10/2016

TimeDelta = TimeDelta100usec * 0.0001;

Accumulators:Velocity mm/secDisplacement mm

TrapezoidalIntegration

Method

17

18

19

Rev. 10/08/2016

Data Processing Application

Readsrecorded

accelerationdata packets

WritesDSP resultpackets:gravity,

acceleration,velocity,

displacement,rotation

20

ConfigureDataFlow()

Rev. 10/12/2016

Data Flow: Analyze MotionReads

recordedaccelerationdata packets

Writes resultpackets:gravity,

acceleration,velocity,

displacement,rotation

Parsesdata packets

Handlesacceleration

packets

Data displaywindow

Filter andintegrate

21

Rev. 10/12/2016

22

Rev. 10/12/2016

23

Current State• Recording

– Android data rate problem• On bow buffering to slow down delivery (coded, needs

testing)– Windows laptop recording works fine

• Signal processing (Windows application)– Motion analysis works

• Bow rotation determination needs work– Sound analysis works

• Needs parameter tweaks• Graphic presentation

– Current format needs improvement (automatic sync)– Standard music notation – not started

• Bow robot – false start – needs start over• Usage experience – not very much yet

Rev. 10/12/2016

24

Refinement Possibilities• Not a precision instrument

– Lot of room for improvement– Is this 1st approximation approach good

enough?• Motion processing tweaks

– Careful accelerometer calibration• Axes - alignment with bow, zero, scale

– Filter parameter tweaks• Better rotation analysis

– Combine Y and Z axes– X-axis acceleration effect

Rev. 10/12/2016

25

Misadventures• Rush to try something new

– Erroneous assumptions– Not enough attention to detail

• I2C hang up– Reset on power supply to zero (app note)

• Stepper erratic in a stepping rate range – Overshoot and rebound time vs. rate

Rev. 10/09/2016

26

Accelerometers in Robotics• Balancing (gravity sensor)• Navigation (displacement via integration)

– Not good over long distance• Bumper sensor (contact proximity)

– Accelerometer tilt alters gravity reading– Obstacle avoidance or accommodation

• Slow rotation (multiple accelerometers)• Fast rotation sensor (axial acceleration)• Vibration sensor

Rev. 10/112016

27

Questions - Discussion

Rev. 10/11/2016

28

Development Approach• New microcontroller setup priority

– Get a serial port working• A valuable debugging aid

– Even if the development environment has a good debugger• Add module to format text output to a terminal

– Cursor position, basic numeric conversion• Add module to form and parse simple data packets

– Effectively a multichannel communication system– Get a basic timer working

• e.g. 32 bit, 1 count = 100usec (4+ days before wrap to 0)• Add module with functions for

– Delay– Elapsed time tracking– Time past specified interval

• Explore other hardware features needed for application

• Work unknowns and hard problems first• Leave easy stuff to last or as short rest breaks from the

hard stuff

Rev. 10/11/2016

29

Simple Communication Packet

Rev. 10/13/2016

Outgoing formatting

30

Tools Used• PCB Design -- www.upverter.com• Audio analysis tool -- www.seventhstring.com

– Transcribe!• IDE

– Visual Studio– Android Studio– PSoC Creator– Keil uVision

• Languages:– C#– C++– C– Java

Rev. 10/14/2016

31

Reference• DSP book

– http://www.dspguide.com• Bluetooth Low Energy

– See following slides • Microcontrollers

– STM32 -- http://www.st.com– EZ-BLE -- http://www.cypress.com

• Accelerometer– LIS331DLH -- http://www.st.com

• PCB Design– Upverter -- http://www.upverter.com

• Violin brain surgery aid -- 2014– http://liveforlivemusic.com/news/man-plays-violin-during-brain-

surgery-to-get-rid-of-tremors/#.U_TvKBb0fme

Rev. 10/13/2016

32

Reference – BLE

Rev. 10/13/2016

Short intro to Bluetooth LEhttp://mbientlab.com/blog/bluetooth-low-energy-introduction/

Better discussions of custom UUIDhttp://www.argenox.com/bluetooth-low-energy-ble-v4-0-development/library/a-ble-advertising-primer/

Long intro to Bluetooth LEhttps://www.bluetooth.org/DocMan/handlers/DownloadDoc.ashx?doc_id=227336

http://www.cypress.com/forum/psoc-4-ble/how-can-i-get-uuid-services-and-characteristics

Explains attribute handles and morehttps://epxx.co/artigos/bluetooth_gatt.html

Maximizing BLE Throughput on iOS and Androidhttps://punchthrough.com/blog/posts/maximizing-ble-throughput-on-ios-and-android

33

Reference – Cypress BLE

Rev. 10/14/2016

/documentation/development-kitsboards/cy8ckit-042-ble-bluetooth-low-energy-ble-pioneer-kit

/products/ez-ble-proc-module-bluetooth-smart

/products/bluetooth-low-energy-ble-connectivity-solutions

/user/281781 -- Cypress employee posts – “Rules” on or before 8/24/2015

http://www.cypress.com …