SECOND EDITION

Programming Interactivity

Joshua Noble

O'REILLY4

Beijing • Cambridge • Farnham • Köln • Sebastopol • Tokyo

Table of Contents

Preface xiii

1. Introducing Interaction Design 1 What This Book Is For 1 Programming for Interactivity 2

The Nature of Interaction 3 Messages and Interaction 5 Interfaces and Interaction 6 Languages of Interaction 7

Design and Interaction 15 Art and Interaction 16 Data Exchange and Exploration 17 Working Process 21

2. Programming Basics 23 Why You'll Read This Chapter More Than Once 24 The Nature of Code 24 Variables 25

Simple Types 26 Arrays 31 Casting 35 Operators 35

Control Statements 39 if/then 39 for Loop 40 while Loop 41 continue 42 break 42

Functions 43 Defining a Function 43 Passing Parameters to a Function 44

Some Suggestions on Writing Functions 45 Overloading Functions 46

Objects and Properties 48 Scope 51 Review 52

3. Processing 55 Downloading and Installing Processing 56 Exploring the Processing IDE 56 The Basics of a Processing Application 58

The setupO Method 58 The draw() Method 60

The Basics of Drawing with Processing 63 The rect(), ellipse(), and line() Methods 63 RGB Versus Hexadecimal 64 The fillO Method 66 The backgroundO Method 67 The lineO Method 67 The stroke () and stroke Weight () Methods 68 The curve () Method 68 The vertexO and curveVertexO Methods 69

Capturing Simple User Interaction 70 The mouseX and mouseY Variables 71 The mousePressedO Method 71 The mouseReleasedO and mouseDraggedO Methods 72 The keyPressed and key Variables 75

Importing Libraries 80 Downloading Libraries 80

Loading External Data into Processing 81 Loading and Displaying Images 81 Displaying Videos in the Processing Environment 83 Using the Movie Class 83 Reading and Writing Files 85

Running and Debugging Applications 87 Exporting Processing Applications 88 Conclusion 91 Review 91

4. Arduino 93 Starting with Arduino 94

Installing the IDE 95 Configuring the IDE 97

Touring an Arduino Board 99

iv | Table of Contents

The Controller 99 Uno Versus Mini Versus Mega 99

Touring the Arduino IDE 105 The Basics of an Arduino Application 108

The setup Statement 108 The loop Method 108

Features of the Arduino Language 110 Constants 112 Methods' 112 Arrays 114 Strings 114

How to Connect Things to Your Board 118 Resistors 120

Hello World 121 Debugging Your Application 124 Importing Libraries 127 Running Your Code 130

Running Your Board Without a USB Connection 130 Review 130

5. Programming Revisited 133 133 134 135 136 137 139 141 143 144 146 148 150 151 153 154 154 155 156 157 158

Object-Oriented Programming Classes

The Basics of a Class Class Rules

Public and Private Properties Inheritance Processing: Classes and Files C++: Classes and Files

.cpp and .h A Simple C++ Application

Pointers and References Pointer Reference When to Use Pointers When to Use References Some Rules for Pointers Some Rules for References Pointers and Arrays When Are You Going to Use This?

Review

Table of Contents | v

Your IDE and Computer Windows Mac OS X Linux

Taking Another Quick Tour of C++ Basic Variable Types Arrays Methods Classes and Objects in C++

Getting Started with oF Touring an oF Application

Methods Variables

Creating "Hello, World" Drawing in 2-D

Setting Drawing Modes Drawing Polygons

Displaying Video Files and Images Images Video

Compiling an oF Program Compiling in Xcode Compiling in Code::Blocks

Debugging an oF Application Using the cout Statement Using the GNU Debugger Using the Debugger in Xcode Using the Debugger in Code::Blocks

Importing Libraries ofxOpenCv ofxVectorGraphics ofxAssimpModelLoader ofxNetwork ofxOsc

Review

7. Physical Input Interacting with Physical Controls Thinking About Kinetics Getting Gear for This Chapter Controlling Controls

The Button As an Electrical Object

160 160 161 161 162 163 163 164 165 166 172 172 174 174 176 179 179 182 182 184 186 186 188 188 189 190 190 192 193 195 195 195 195 196 196

..' 199 199 200 201 202 202

vi I Table of Contents

The Button As an Interactive Object Reading the Value of a Button

Turning Knobs The Dial As an Interactive Object Potentiometers

Using Lights Wiring an LED

Detecting Touch and Vibration Reading a Piezo Sensor Getting Piezo Sensors

Detecting Proximity Proximity Through Capacitance

Detecting Motion PIR Motion Sensor

Reading Distance Reading Input from an Infrared Sensor

Understanding Binary Numbers Binary Numbers Bits and Bit Operations Why Do You Need to Know Any of This?

Communicating with Other Applications Sending Messages from the Arduino

openFrameworks Detecting Forces and Tilt Introducing I2C

Gyroscopes What's Next Review

8. Programming Graphics The Screen and Graphics Seeing Is Thinking, Looking Is Reading Math, Graphics, and Coordinate Systems Drawing Strategies

Use Loops to Draw Use Vectors to Draw Draw Only What You Need Use Sprites

Transformation Matrices Creating Motion

Shaping the Gaze Setting the Mood Creating Tweens

202 202 203 203 203 206 206 208 209 210 210 210 216 216 218 220 222 222 222 225 225 228 229 232 237 240 243 244

247 248 250 251 254 254 256 262 263 263 267 268 268 270

Table of Contents | vii

Using Vectors 276 Using Graphical Controls 285

ControlP5 Library 286 Event Handling 286

Importing and Exporting Graphics 288 Using PostScript in Processing 289 Using PostScript Files in oF 290

What's Next 294 Review 294

9. Bitmaps and Pixels 297 Using Pixels As Data 298 Using Pixels and Bitmaps As Input 300 Providing Feedback with Bitmaps 301 Looping Through Pixels 302

ofPixels 303 Manipulating Bitmaps 306

Manipulating Color Bytes 309 Using Convolution in Full Color 310

Analyzing Bitmaps in oF 311 Analyzing Color 312 Analyzing Brightness 314 Detecting Motion 315

Using Edge Detection 321 Using Pixel Data 328 Using Textures 331

Textures in oF 332 Textures in Processing 335

Saving a Bitmap 338 What's Next 339 Review 339

10. Sound and Audio 341 Sound As Feedback 342 Sound and Interaction 345 How Sound Works on a Computer 347 Audio in Processing 350

Instantiating the Minim Library 350 Generating Sounds with Minim 352 Filtering Sounds with Minim 356

Sound in openFrameworks 362 openFrameworks and the FMOD Ex Library 364 Maximilian 371

viii | Table of Contents

Physical Manipulation of Sound with Arduino 381 A Quick Note on PWM 382 Creating Interactions with Sound 385 Further Resources 385 Review 386

11. Arduino and Feedback 389 Using Motors 390

DC Motors 391 Stepper Motors 394 Motor Shields 396

Smart Feedback 397 Using Servos 399

Connecting a Servo 400 Communicating with the Servo 400 Wiring a Servo 401

Using Household Currents 405 Working with Appliances 408 Introducing the LilyPad Board 414 Using Vibration 416

Nano, Fio, and Mini 419 Using an LED Matrix 419

Using the LEDControl Library 419 Using the SPI Protocol 422 Serial LED Matrix 423 Using LCDs 425 Using Solenoids for Movement 429 What's Next 432 Review 432

12. Protocols and Communication 435 Communicating over Networks 436 Using XML 438 Understanding Networks and the Internet 441

Network Organization 441 Network Identification 442 Network Data Flow 443

Handling Network Communication in Processing 443 Client Class 444 Server Class 445 Sharing Data Across Applications 448

Understanding Protocols in Networking 453 Using the ofxNetwork Add-on 454

Table of Contents | ix

Creating Networks with the Arduino 469 Initializing the Ethernet Library 470 Creating a Client Connection 470 Creating a Server Connection 472

Wireless Internet on the Arduino 475 Communicating with Bluetooth 479

Using Bluetooth in Processing 479 Using the bluetoothDesktop Library 480

Communicating Using MIDI 482 Review 486

13. Graphicsand OpenGL 489 What Does 3-D Have to Do with Interaction? 489 Understanding 3-D 490 What Is OpenGL? 491 Working with 3-D in Processing 492

OpenGL in Processing 493 Lighting in Processing 494 Controlling the Viewer's Perspective 496

Making Custom Shapes in Processing 500 Using Coordinates and Transforms in Processing 503

Transformations 506 3-D in openFrameworks 506

Drawing in 3-D 508 Transformations in openFrameworks 509

Lighting in OpenGL 509 Blending Modes in OpenGL 511 Creating 3-D Objects in oF 515 Using Textures and Shading in Processing 519 Using Another Way of Shading 520

What Does GLSL Look Like? 520 Vertex Shaders 521 Geometry Shader 522 Fragment Shader 522 Variables Inside Shaders 523

Using ofShader 524 Using Shaders in Processing 530 What to Do Next 531 Review 532

14. Motion and Gestures 535 Computer Vision 536

Interfaces Without Controls 537

x | Table of Contents

Example CV Projects 538 OpenCV 539 Using Blobs and Tracking 539

Starting with ofxOpenCV 540 Detecting Features with oF 545 Using OpenCV in Processing 549 Feature Tracking in Processing 554 Using Blob Tracking with Physics 559 Exploring Further in OpenCV 565 Detecting Gestures 566

Using ezGestures in Processing 567 Using Gestures in oF 570

Capturing iOS gestures with oF 574 Touch with oF 577

Tuio 577 reacTIVision 578 CCV 578

What's Next 578 Using the Microsoft Kinect 579

Processing 579 openFrameworks 580

Review 580

15. Movement and Location 583 Using Movement As and In Interaction 583 Using Software-Based Serial Ports 585 Understanding and Using GPS 587 Storing Data 599 Logging GPS Data to an Arduino 602

Using the breadcrumbs Library 602 Implementing Hardware-Based Logging 603

Sending GPS Data 606 Getting Location on a Mobile Device 608

On the iPhone 608 On an Android Phone 609

What to Do Next 610 Review 610

16. Spaces and Environments 613 Using Architecture and Space 613 Sensing Environmental Data 614 Using an XBee with Arduino 615

Creating a Simple Test 618

Table of Contents | xi

Configuring the XBee Module Addressing in the XBee XBee Library for Processing

Placing Objects in 2-D Using the X10 Protocol Setting Up an RFID Sensor Reading Heat and Humidity Determine Position of an Object What's Next Review

17. Further Resources What's Next?

Software Tools Construction Processes Artificial Intelligence Physics Hardware Platforms

Bibliography Interaction Design Programming Hardware Art

Conclusion 673

Index 675

620 622 623 627 634 637 643 648 655 655

657 657 657 660 661 662 668 670 670 671 672 672

xii | Table of Contents