Yahoo xtra Open Technolgies

Yahoo! Open Strategy – focus on YQLNeil Wilkinson (Wilf)Technical Director, Yahoo!7 and Yahoo!Xtra2nd July 2009

© 2007 Yahoo!7 Pty Limited

An intro to Yahoo!’s Open Strategy – focus on YQL

Wilf (Neil Wilkinson), Technical Director, Yahoo!7 and Yahoo!Xtra

Who am I and what am I doing here?

Wilf

You can call me Neil Wilkinson if you have to

Originally a POM, I’ve been living in Sydney for 8 years

Worked at Yahoo! in Australia for 6 years

Currently the Technical Director of Yahoo!7 and Yahoo!Xtra managing “Technical Services”

o A multi-disciplined team including Design, Front-End Engineering, Development, Operations, IT and Project Management – about 40 people.



Who am I and what am I doing here?

What am I doing here?

Here to share the joys of some of Yahoo!’s Open Strategy

- Browser-independent front-end development

- YQL – The Yahoo! Query Language – focus of today’s talk

Why is Yahoo! opening its technology?

Yahoo! only has thousands of developers

Millions of developers world-wide can use Yahoo!’s technology

meaning that Yahoo! improves its technology from the feedback of

millions of developers.



Yahoo! believes in “open”

Yahoo!’s technology uses Open

Yahoo! is powered by Open Source technologies:

- Linux, FreeBSD, Apache, PHP, MySQL, Bugzilla, Hadoop…

Yahoo! embraces open standards:

- OpenID, OpenSocial, Microformats, OAuth

Yahoo! gives back to open source

- YUI, github, YSlow

- Employs Rasmus Leodorf (PHP), Doug Crockford

(Javascript), Mark Nottingham (Squid), etc



So what technology has Yahoo! opened?

YUI – Yahoo! User Interface

Design and develop for the user, not for the browser

Reset the browser CSS styles so all browsers have similar layout behaviour

Change the fonts so that they look the same cross-browser using percentage sizes (no more excuses for hard-coding font sizes!).

Use CSS grids – its just too hard to do it any other way: http://developer.yahoo.com/yui/grids/builder/

YUI Controls/Widgets include Calendar, Module, DataTable, etc: http://developer.yahoo.com/yui/calendar/

http://developer.yahoo.com/yui/grids/builder/

http://developer.yahoo.com/yui/calendar/




BOSS (Build your own search service)

Currently unlimited access to websearch API




Flickr Most used photo API worldwide

http://api.flickr.com/services/rest/?

method=flickr.photos.search&api_key=88dfe8282b22644fe806f

a33018a3f50&user_id=31508066%40N00&text=new+zealand




Maps<script type="text/javascript” src="http://api.maps.yahoo.com/ajaxymap?

v=3.8&appid=hyBP5mDV34EiUpuYzxFAd5s2dTxQ4ioG3SKT2Yl6r1bo91

8ZTfj59aJ3UaRf61gNUvGKzdoeTQ--"></script>

<div id="map"></div>

<script type="text/javascript">

var map = new YMap(document.getElementById('map'));

map.addTypeControl();

map.addZoomLong();

map.setMapType(YAHOO_MAP_REG);

map.drawZoomAndCenter("242, Queen St, Auckland, New Zealand", 3);

</script>



So what technology hasn’t Yahoo! opened?



YQL – the glue that mashes the APIs together YQL

select * from internets

Is a command-line language to get data from APIs

Is a RESTful interface, returning XML or JSON

Can be used with server-side HTTP requests or json function callbacks (for javascript client-side HTTP requests).

select {what} from {service} where {condition}



YQL – simple RSS feed example

Let’s take a normal RSS feed

http://nz.rss.news.yahoo.com/top_stories.xml

And search the descriptions for a keyword

SELECT title, url, description

FROM rss

WHERE url='http://nz.rss.news.yahoo.com/top_stories.xml’

AND description like '%the%’

• Let’s take a look at the web-based console…

http://nz.rss.news.yahoo.com/top_stories.xml



YQL – What’s going on in the world right now? The description of the first item of the top

stories RSS feed should give us some text that describes what is newsworthy right now

select description from rss

where url="http://nz.rss.news.yahoo.com/top_stories.xml"

limit 1



YQL – What’s going on in the world right now? Finding some key terms from the textual

description

We use an IN clause to use the result of 1 API into the input of another API

Best demonstrated with an example:

select content from search.termextract where context in (



limit 1

)



YQL – What’s going on in the world right now? So now we have some keywords about what’s

going on the news right now

What if we used that data to get the other related news going on in the world

select * from search.news where query in (




limit 1

)

)



YQL – What’s going on in the world right now? There’s more we can do with those keywords

Let’s get web search results too:

select * from search.web where query in (




limit 1

)

)



YQL – What’s going on in the world right now? There’s even more we can do with those

keywords

Let’s get flickr results too:

select * from flickr.photos.search where text in (




limit 1

)

)



YQL – so how do we programmatically use this data? Example PHP code – fetching the data

• Using CURL to make a server-side web request

<?php

$ch = curl_init(“http://query.yahooapis.com/v1/public/yql?q=select…&format=json”);

curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);

$jsonContent = curl_exec($ch);

curl_close($ch);

$content = json_decode($jsonContent);

?>



Example PHP code – displaying the data

• Not exactly rocket science:

<?php

echo(‘<ul>’);

foreach ($content[‘results’] as $result) {

$url = $result[‘clickurl’];

$title = $result[‘title’];

echo(“<li><a href=\”$url\”>$title</a></li>”);

}

echo(‘</ul>’);

?>

YQL – so how do we programmatically use this data?



function SimpleSearch(searchResultsDivId) {

this.searchResultsOL = document.getElementById(searchResultsDivId);

this.display = function(o) {

var innerHTML = '';

var results = o.query.results.result;

for (i in results) {

var result = results[i];

innerHTML = innerHTML + '<li><a href="' + result.clickurl + '">' +

result.title + '</a><br />’ + result.abstract + '</li>';

}

this.searchResultsOL.innerHTML = innerHTML;

}

}

YQL – so how do we use this data?



Example Javascript code• We can then define an object of SimpleSearch:

<script>

var webSearch = new SimpleSearch(“html_list_container”);

</script>

• Using Javascript we can get the browser to make the HTTP request:

<script src="http://query.yahooapis.com/v1/public/yql?q=select…&format=json&callback=webSearch.display"></script>

• That results in a call to object instance webSearch, method

display with the json data from YQL.

YQL – so how do we programmatically use the data from YQL?



What’s going on the world right now?

• News, Web and Flickr results

• From 3 YQL queries

• Using JSON callbacks

• All HTTP requests done by the browser

• No server-side processing

• …here we go…

YQL – A real live example – I’m nervous, are you?



YQL – Table List

flickr.photos.exifflickr.photos.infoflickr.photos.interestingnessflickr.photos.recentflickr.photos.searchflickr.photos.sizesflickr.placesflickr.places.infogeo.placesgeo.places.ancestorsgeo.places.belongtosgeo.places.childrengeo.places.neighborsgeo.places.parentgeo.places.siblingsgeo.placetypesgnip.activity

local.searchmusic.artist.idmusic.artist.popularmusic.artist.searchmusic.artist.similarmusic.release.artistmusic.release.idmusic.release.popularmusic.release.searchmusic.track.idmusic.track.popularmusic.track.searchmusic.video.categorymusic.video.idmusic.video.popularmusic.video.searchmusic.video.similar



YQL – Table Listmybloglog.community.findmybloglog.membermybloglog.member.contactsmybloglog.member.newwithcontactsmybloglog.member.newwithmemybloglog.members.findmybloglog.stats.findupcoming.categoryupcoming.countryupcoming.eventsupcoming.events.bestinplaceupcoming.groupsupcoming.metroupcoming.stateupcoming.userupcoming.venue

search.imagessearch.newssearch.siteexplorer.inlinkssearch.siteexplorer.pagessearch.spellingsearch.suggestsearch.termextractsearch.websocial.connectionssocial.contactssocial.presencesocial.profilesocial.updatesweather.forecastyahoo.identityyap.setsmallview



YQL – what else can you do?

Using XPath, you can get content from a HTML page or any published feed:• This gets all main navigation items from the herald:

select * from html

where url='http://www.nzherald.co.nz'

and xpath='//h2/a'



YQL – what else can you do?

Using open tables, you can create your own data

tables to use in your own query (and mash up with

other people’s data).

Programmatically handle select statements on your

open data using server-side javascript code.



YQL – developing with YQL

Good practice

• Your YQL query is cached for you by Yahoo! infrastructure

• However, especially if your servers are in New Zealand, caching local to your server is important for scale and performance

• Caching is also important to avoid hitting Yahoo!’s limit of 100,000 per application ID per day

• Test using the console, and copy the REST query into your scripting language of choice (PHP) or use the javascript JSON callback.



YQL – Summary

• YQL is a very quick and easy way to get data to your fingertips

• Mash up a huge variety of data including APIs, contributed open tables, your own tables and any publically available feed or webpage

• Use Yahoo! data, don’t use Yahoo! data – up to you.

• Just have fun!

THANKS!

© 2007 Yahoo!7 Pty Limited

Leave your details with Alison to be contacted for future events, including being the first to know about our NZ Hackday competition.

Technology

Yahoo xtra Open Technolgies