Rewiring the WebWith Yahoo Pipes

“It’s all about me” edition

Tony Hirst

http://pipes.yahoo.com

How do you merge content from multiple web searches and add it your own personal page?

How do you quickly and easily plot your photos or a list of addresses contained in a spreadsheet

onto an interactive map?How do you automatically find Youtube movies

related to a news story?

Just because you aren’t a programmer doesn’t mean you shouldn’t be able to get computers to do what you want them to do.

Yahoo Pipes helps you rewire the web, moving content from where it currently is to where you need it to be, in the form that you require.

A. Pipes are like construction blocks that can take content from one location on the web, process it in some way, and then republish it to where you need it to be. You don't even need to create your own in order to get started – just use someone else’s…B. To create your own Yahoo Pipes you’ll need to login with a Yahoo ID.C. Once you’re logged in, click Create Pipe to get started.

Drag and drop is the name of the game (with as little typing as possible required).

Wire the building blocks together and see what happens as content flows through the pipe.

Are you ready to give it a go?

D. This is the central canvas, the place where you build your pipe out of simple building blocks or even other pipes.E. The debug area allows you to preview the output of any part of pipe to make sure that each block is doing exactly what you expect. F. Building blocks are listed in this menu area. Drag individual blocks onto the canvas, or click the + sign to add them to the canvas automatically.G. Many great works of art may be untitled, but it can be hard to tell them apart by name alone. Give your pipe a meaningful name by clicking in this tab.

Use the blank pages to write your own notes, ideas for how to use each pipe, or sketches for how to modify or extend each pipe…

The Simplest Pipe

From the very beginning, Yahoo Pipes was designed as a tool to help content flow around the web.

The first step is to get web content into the Pipes environment…

…and the second is to learn how to publish it back to world.

0. Wires are used to take the output of one part of the pipe and feed it to the next. Content flows out of the connector on the bottom of a block and into the connector on the top of another block. Click on a connector and drag the mouse, and the wire will follow. Make a connection by dropping the wire onto a connector on another block.

1. Many websites and search engine publish content via an RSS or Atom feed, often denoted by the symbol shown opposite. The Fetch Feed block can be used to pull that content into a Pipe from the feed’s web address. In this case, we pull in content from an RSS feed from my blog that has been republished via Feedburner.

2. The output block is always the final block in a pipe. Content is pulled into the pipe via an input blocks such as Fetch Feed and published to the outside world via a Pipe Output block.

Every Yahoo Pipe has its own personal web page.

This page previews the output of the pipe and provides a range of ways publishing that content to the web.

But that’s not all – because every pipe can also be used inside other pipes…

3. A preview of the output of the pipe. If the pipe emits valid RSS, you should be able to see highlights of the first few elements of that web feed here.4. Content came into the pipe via an RSS feed, so it’s only natural that we should be able to republish it as RSS too.5. Pipes are also about sharing and reuse. Clone a pipe (either one of yours, or one of someone else’s) in order to grab a copy of it to keep for ever, or that you can edit to create a modified version of the pipe.6. Each pipe has its own web address. You can give your pipe a nice URL if you want to.

Multiple Sources(and that includes other pipes)

My online identity is distributed across multiple websites. Now can I pull all my personal feeds together in one place.

Although quick and easy to build, this sort of pipe provides a powerful tool for merging content from different sources.

So what are you going to draw together?

7. The My Pipes menu option contains a list of pipes I have cloned or made that I can include in a new pipe, just like any other pipes block.8. Content from multiple sources can be pulled together (that is, aggregated) within a single pipe. Clicking the + button in the Fetch Feed block allows you import content from several different web addresses. This technique can be used as is to extend The Simplest Pipe directly.9. The Union block, which can be found in the operators menu, allows you to merge the content from different sources. To join more sources together, just wire the output of one Union block to the input of another.

Probe the internals of a pipe to see what’s going on.

Inspect the first few items that flow out of a block to check their basic, elemental properties.

And then start putting those properties to work…

10. Clicking on a block highlights it and turns it orange. Selecting a block in this way means we can inspect the output of the block in the preview window.11. Yahoo Pipes process lists of items. In a search results feed, each item will contain details for a single search result. In a feed from a news website or blog, each news story or blog post will be a separate item. In this preview, each item corresponds to a single presentation from my Slideshare account. Each item contains a list of elements. The title element provides the text used to identify each item. The link element contains the web address for the original webpage of each presentation item.

Selecting What’s Relevant

Sometimes we want to be selective about what we let flow out of a pipe.

And sometimes we want to stop particular items from making it through.

What would you focus your pipe’s attention on?

12. I created this pipe by Copying the Multiple Sources pipe.13. The * that is added to the end of the Pipe’s name shows that changes have been made to the pipe since it was last saved.14. If you want to rewire part of a pipe, move the cursor over a connector and a pair of scissors will appear: you may now cut the connecting wire and then rewire pipe as required. In this case, I am going to add a Filter block between the Union and Pipe Output blocks.15. The Filter block, which can be found in the Operators menu, allows you to select which items you do – or do not – what to pass through the block based one or more properties of each item. Rules can be added as required by clicking +. Items may be Permitted to flow through the Filter, or Blocked (i.e. prevented) from passing through it, based on whether any single rule is true or whether all the rule conditions are met.

The best thing about good pipework is that you shouldn’t have to see it – it does its job discreetly.

So if you want a pipe to deliver its results somewhere more convenient – like your Google or Yahoo Personal page – it can do.

16. The preview output of the pipe shows that only items that met the Filter criteria are passed through. 17. If you have a Google personal page, you can add a widget to it based on the output of a Pipe. Whenever the the content in the feeds defined as inputs to the pipe change, the pipe’s output will reflect that change (subject in this case to the Filter conditions.) So if I write a post about a new presentation on my blog, a link to it will now appear on my Google personal page, courtesy of this pipe and The Simplest Pipe before it..

Search Me...

Sometimes it’s useful to be able to let the user of a pipe specify some search terms to be used by a pipe.

If search terms or topic terms appear in the web address of an RSS feed, we can create our own search engine.

Remember, a pipe can fetch more than one feed…

18. The URL Builder makes it easy to construct new URLs from old ones. For example, copy the web address for the RSS feed from a Google News search into the Base text area and the individual parts of the URI will be separated out. 19. Turn the pipe’s homepage into a web form by allowing the user to enter a search term of their own using a Text Input block. Set a default value to demonstrate what the pipe does.20. Look for the part of the URL that contains your original query, and wire the output of the Text Input to corresponding connector.21. Having constructed a URL for an RSS feed from Google News that uses a user supplied search term, we can now fetch the feed with a Fetch Feed block.

INDEX

FFetch Feed 1, 8, 21

UUnion 9

MMultiple sources 8, 9