The CT_DATFSM (Creative Technologists Defence against the Flying Spaghetti Monster)

A submission for the Small Object of Delight Project of the MDA4600 Module of the Middlesex University

MSc Creative Technology Course.

By: Ayodele Arigbabu 27th January 2016

Summary The CT_DATFSM is a small object of delight developed as a whimsical protective device against a mythical Flying Spaghetti Monster said to be responsible for spaghetti code and all other such troubling phenomena (including brain fatigue) that tend to trouble the creative technologist.

Image 1: The Flying Spaghetti Monster image sourced from- http://www.venganza.org/

In practical terms, the project was conceived to feature an array of LED lights being triggered via an augmented reality app. For added playfulness, this array of LEDs was meant to be mounted on a remote controlled vehicle so that one user could drive the RC vehicle (with the lights mounted) while another user could chase after it with an augmented reality app on a smartphone with which to trigger the flashing of the lights according to different pre-programmed patterns.

Project Description

a. What is the project?

An interactive toy that fuses physical and digital aspects of gameplay as a ‘small object of delight’.

b. What does it do?

Based on ‘gameplay’ where Player 1 views and aims at a trigger object (mounted on a remote controlled vehicle) via an augmented reality app on a

smart phone, while Player 2 moves the trigger object out of target by controlling the remote control vehicle. Player 1’s task is to score hits on the moving target.

c. Who is it for?

For the curious and playful.

d. How does it come together?

Through gameplay. The parts (physical and digital) don’t come together properly unless when in use. This is deliberate as the interaction between digital and physical was desired. Digital integration is to be achieved through Bluetooth technology for the app to be able to trigger the LED lights, and of course, Unity compiles the AR functionalities into an android app.

Research

a. Tools / skills required to deliver the project

1. 3D modelling and 3D printing for the orb. 2. Unity and Vuforia for the augmented reality app 3. Arduino hardware and programming for the LED bulbs 4. Bluetooth connection between the AR app and the Arduino

b. Content (Graphics? Video?) required to be created for the project.

3D assets will be built in FormZ / Blender

Additional animation / particle effects in Unity Illustration for the image target(s)

c. How similar projects have been handled elsewhere

Most augmented reality apps do not depict the level of interaction between trigger and app as being attempted here, so that would be a challenge to overcome.

d. What would make this one different

The orb should generate curiosity and a desire to play, while the app should sustain this desire. The gameplay should elicit delight, thus the lights, explosions and other effects should reflect this. It is intended that the gameplay would be compelling and intuitive. Please view bibliography for useful links visited in researching how to implement the different technologies considered for the project.

Target audience?

An audience of 1….in other words, me. Following from the brief for the Small Object of Delight project, I chose to pursue a project that I would find interesting and to

which I could apply self-imposed constraints, part of which included the desire to learn something new and do something hands-on on my own. I was able to achieve this at the end of the project period.

Project Implementation: Initial Thoughts

For the physical object,

The target is made of a 3D printed hemispherical orb (original plan, later modified)

mounted on a remote controlled vehicle.

An augmented reality app built for android phones treats this orb as a trigger and issues an explosion when the screen is tapped while the orb is within target.

The orb is fitted with LED bulbs which flash according to 3 different pre-programmed patterns indicating a state of rest, a mobile state and an agitated state coinciding with when the orb is at rest, in motion and when the explosion is triggered on the app, respectively.

Image 2: 3D visualization made at the start of the project to envisage what the final outcome would look like

For the AR experience, we need to be able to:

Scan a 3D object via Vuforia (this would be the orb but a placeholder would serve

for starters) to serve as the target.

Create an explosion / particle effect that would be activated upon an onTrigger

event (double tap on the screen), the OnTrigger event should only work when

the target is within view (lines of code to determine this?) at a certain proximity

level to the camera (lines of code to determine this?).

Send information to the Arduino on the orb (via a Bluetooth module) to tell it

that a hit has occurred, this information would trigger the Arduino to change the

pattern of lights being emitted by the orb’s LEDs to the agitated state.

It would be desirable to stop the remote controlled vehicle bearing the orb from

moving at this point, but that is to be investigated to see how easily achievable it

is (probably by intersecting the RC vehicle’s power supply with a switch

connected / operated through the Arduino.

Actual experience while developing the project

Further research was conducted via the internet to see how different people had handled

similar parts of the whole that would make up the project (comprehensive list of links

below). This helped me in generating a shopping list of components I would require for

implementing the project:

Main shopping list Arduino Leonardo - £25 http://www.amazon.co.uk/Arduino-A000057-Leonardo-with-headers/dp/B008FZJC74/ref=sr_1_2?ie=UTF8&qid=1450733529&sr=8-2&keywords=arduino+leonardo

WS 2801 RGB led pixel string (100pcs) - £43 http://www.ebay.co.uk/itm/25-500x-WS2811-WS2801-Diffused-Digital-RGB-LED-Pixel-character-string-5V-12V-DC-/231614367325?var=&hash=item35ed4ac65d:m:mFAVBOUEDT5JyVTLDWEx9TQ

Arduino bluetooth serial tranceiver (JY-MCU) - £5 http://www.ebay.co.uk/itm/New-Arduino-Wireless-Bluetooth-Serial-RF-5V-Transceiver-Module-JY-MCU-HC-06-UK-/151836195522?hash=item235a247ac2:g:cHwAAOSw0HVWDqM2 Bluetooth Plugin for Unity - £14 https://www.assetstore.unity3d.com/en/#!/content/16467 RC Toyota tundra monster truck from Maplin - £20 http://www.maplin.co.uk/search?text=Toyota+Tundra+1%3A14+Electric+RC+Monster+Truck

The Arduino Leonardo was recommended on one of the sites visited where users were advised against running the led string and the Bluetooth module at the same time on an Arduino Uno

because of the Uno’s limited memory. This proved to be of no consequence eventually as the Bluetooth module failed to work altogether and the project was

concluded using the Arduino Uno without any noticeable consequences. The challenge with building a shopping list was finding the best options viz. price vs. delivery date, given that

the most affordable components had to be shipped in from China. This later proved a challenge when the Bluetooth module procured failed as there wasn’t enough time left to have another one delivered if ordered. I therefore had to rethink my original plans after days lost to the defective Bluetooth module.

Image 5: Marking out the locations for the holes on the bowl following steps suggested here: https://www.youtube.com/watch?v=gHZ8-8fMTJ8&list=WL&index=18

Image 4: The flying spaghetti monster struck hard, disabling the Bluetooth module and upsetting the best laid plans. Image sourced from- http://www.venganza.org/

Image 3: Checking acquired components for size. The original intention was to have a mixed reality game like experience, hence the RC monster truck

Another deviation from the plan was necessary with the original idea to 3D print the orb / hemisphere that would bear the lights. This was purely for practical reasons because even though I absolutely wanted my 3D designed orb with its amorphous voronoi pattern, I opted for a faster prototyping option instead to avoid delays and costs in achieving the 3D printing especially as that phase of the work was being done during the holidays when the workshops were closed. The option I went with entailed finding an appropriately sized transparent plastic bowl and marking it with the aid of a measuring tape with the locations of the LEDs, a total of 48 holes on a grid of 4 x 12. The process was adapted from the steps taken in creating a similar object in this video: (https://www.youtube.com/watch?v=gHZ8-8fMTJ8&list=WL&index=18). For programming the LEDs, my initial assumption from early research was that I would have to adopt the somewhat cumbersome approach to connecting LEDs in parallel, called multiplexing, thankfully, I found a tutorial and Arduino library from Adafruit that demonstrated how their LED strings (with built in chips) achieved the same thing (connecting numerous LEDs to an Arduino using minimal number of pins) with simpler code and less stress (http://learn.adafruit.com/12mm-led-pixels). Rethinking the idea. Not being able to use Bluetooth, I thought of other ways of achieving interaction between an external device and the lights. In seeking alternative ways of adding interactivity to the lights, I initially tried to use an LCD display to programme patterns that could be used as image targets for the augmented reality app in Unity. This did not work as the images captured from the LCD screen proved to be poor image targets. I did some further research

which led me to a tutorial on music reactive multi-colour led lights (http://www.instructables.com/id/Music-Reactive-Multicolor-LED-Lights/?ALLSTEPS

Image 8: WS801 LED string being tested Image 7: Final assembly of LED string on hemisphere

Image 6: Holes made on plastic kitchen bowl using the pillar drill at CreateSpace, a maker space in the Wembley area of London.

). While I found it quite interesting, I opted eventually to investigate the use of infra-red instead so the user would have actual control of the lights rather than leaving the lights to be triggered solely by the randomness of sound.

Image 9: Attempt at using LCD screen to create dynamic image targets resulted in image targets that were too poor to be used.

The choice of IR was informed by the realisation that with an infrared bulb mounted on the Arduino, the Arduino could be made to receive command with which to programme the lights. Likewise, another Arduino could be programmed to send out the required commands and this second Arduino could be controlled via an AR app running on a smart phone. The intention was to build this app and to build the Arduino RC controller add-on for the smart phone, however time constraints made it possible to only build the IR receiving end of the idea. Infrared transmission was achieved using an IR controller of which I accessed and stored the code generated by each button via the serial monitor in the Arduino IDE, using techniques and Arduino IR libraries explained on this blog by Ken Shirriff: http://www.righto.com/2009/08/multi-protocol-infrared-remote-library.html

To add an extra level of interest, I also wired in a buzzer and following tips and sample code from this tutorial, I was able to programme the Arduino to play back a tune: http://www.instructables.com/id/Play-the-French-Can-Can-Using-an-Arduino-and-Buzze/?ALLSTEPS

Using tips offered on merging Arduino codes from different sketches, (here: http://www.thebox.myzen.co.uk/Tutorial/Merging_Code.html) I was able to successfully combine the code for the LED strip patterns and the Buzzer playing back the tune with the IR detection and decoding sketch from this link serving to loop through both sketches based on the IR code released by the button pressed on the remote control: (https://learn.sparkfun.com/tutorials/ir-control-kit-hookup-guide). This was implemented

Image 10: Assembled lights being tested Image 11: Assembled lights being tested

through a switch / case statement that called a function for playing back the tune and a set of alternate functions to launch different light patterns for each case.

At this level of implementation, the code worked well enough, the only drawback being that the Arduino always waited for the light display to run its course before playing back the tune. This caused a noticeable lag / disconnect between visual feedback and sound which was not intended. I was able to research how to correct this by using a concept called state machines (more explanation available through this blog: (http://www.thebox.myzen.co.uk/Tutorial/State_Machine.html), however, I did not have sufficient time to implement this. Moving forward, I shall be exploring the use of state machines and the other half of the project involving the creation of an Arduino based IR RC control ported to a smartphone app as an add-on and combined with augmented reality to achieve the original objectives I set out for myself at the start of this project.

Image 10: Reference for 2013 proof of concept for use of infa-red in augmented reality by Frank Zhao and fellow students. Details available via this link: http://eleccelerator.com/our-fourth-year-design-project-aruci/

What would I have done differently?

Despite the frustrations encountered, I’m largely pleased with the outcome of the project as it helped me achieve my primary objective of learning something different while making something.

The problems associated with having components delivered from China have led me to realize that in practice as a creative technologist, I would have to pre-order lots of components and tools, even if they are not immediately needed, just so I can readily

find replacement components when working on projects or parts to experiment with when prototyping or trying out new ideas.

Given what I now know about how fragile Bluetooth modules could be, if I had to do this again, I would have settled on Infrared at an earlier stage.

To improve the project, I would replace the buzzer with small speakers for better sound output (the buzzer worked remarkably well though for prototyping).

I would also use a stronger IR receiver bulb as the one I used, though it worked well enough, had some moments when it failed to pick up the code from the IR Remote control.

What I have learnt from this project:

Infrared technology is ubiquitous with television remote controls and remote controlled vehicles and therefore relatively stable. With deep understanding of the inner workings of infrared, there is also the potential to hack the technology in many ways.

LED matrixes can be built using a system called Multiplexing or a variant called Charlieplexing through which the LEDs are arranged in parallel, effectively reducing the number of Arduino pins that would be required to connect them.

LED strings or strips can save you the stress of Multiplexing or Charlieplexing as though connected in series, each bulb has its own integrated circuit, this helps reducing the number of pins required for connecting the entire string (of up to 100 LEDs or more) to the Arduino through just four or five pins.

You can map the infrared code being emitted by most remote controls and with those hexadecimal values, you can build those values into the code driving an application running on an Arduino. Through this, most house hold remote control devices can be hacked to control other things apart from what they were originally made for. Similarly, you can build your own remote control systems from scratch.

Infrared can be incorporated into augmented reality solutions.

Image 11: Production notes and sketches

Link to Project video: https://youtu.be/Tl8d3s-cohg

Other Useful links from research done: https://programmingelectronics.com/using-the-same-input-to-trigger-multiple-arduinos/ http://www.instructables.com/id/Arduino-BlinkyBall-Project/ http://radiokot.ru/circuit/light/run/45/ https://charliex2.wordpress.com/2012/02/11/the-blinky-ball-nullspacelabs/ https://www.arduino.cc/en/Tutorial/ShiftOut https://www.youtube.com/watch?v=bqfPZXEuyuc http://electronics.stackexchange.com/questions/4027/is-multiplexing-ever-superior-to-charlieplexing https://www.youtube.com/watch?v=gHZ8-8fMTJ8&list=WL&index=18

https://www.youtube.com/watch?v=MmCGDJ90Qt4 https://www.arduino.cc/en/Tutorial/RowColumnScanning http://www.thebox.myzen.co.uk/Workshop/LED_Matrix.html http://jimmieprodgers.com/kits/lolshield/makelolshield/ https://insidegadgets.wordpress.com/2010/11/07/fun-with-8x8-led-matrix/ https://www.assetstore.unity3d.com/en/#!/content/16467 https://www.youtube.com/watch?v=x3KAXjnP06o https://www.youtube.com/watch?v=W5okUciwFkI https://www.youtube.com/watch?v=W5okUciwFkI https://www.youtube.com/watch?v=fJvtMszk2G4 http://www.4k2.de/ikosaeder/ http://www.instructables.com/id/Play-the-French-Can-Can-Using-an-Arduino-and-Buzze/?ALLSTEPS https://learn.adafruit.com/adafruit-arduino-lesson-10-making-sounds/playing-a-scale http://www.righto.com/2009/08/multi-protocol-infrared-remote-library.html http://www.instructables.com/id/InfraRed-Receiver-1/?ALLSTEPS https://learn.adafruit.com/ir-sensor/reading-ir-commands https://learn.sparkfun.com/tutorials/ir-control-kit-hookup-guide http://arduinostuff.jimdo.com/home/remote/ir-codes/ http://eleccelerator.com/our-fourth-year-design-project-aruci/