Introduction to Web APIs Tianjun Fufutj@email.arizona.eduFeb 2011

Google Custom Search, Amazon, eBay, Flickr, YouTube, Second Life

1

2

Web APIs

Application Programming Interface (API) a particular set of rules and specifications that a software

program can follow to access and make use of the services and resources provided by another particular software program that implements that API.

serves as an interface between different software programs and facilitates their interaction

Web API typically a defined set of HTTP request messages expressed

in SOAP or REST along with a definition of the structure of response messages, typically expressed in JSON or XML.

3

SOAP

Simple Object Access Protocol (SOAP)

SOAP Tutorial: http://www.w3schools.com/soap/soap_intro.asp

A SOAP message is an ordinary XML document containing the following elements: An Envelope element that identifies the XML document as a SOAP

message A Header element that contains header information▪ contains application-specific information about the SOAP message▪ optional▪ must be the first child element of the Envelope element

A Body element that contains call and response information A Fault element containing errors and status information

4

REST

Representational State Transfer (REST) Use HTTP method to invoke remote services (not XML)

The response of remote service can be in XML or any textual format

Benefits: Easy to develop Easy to debug (with standard browser) Leverage existing web application infrastructure

We will focus on REST services programming for the rest of the slides.

5

Server Responses

Really Simple Syndication (RSS, Atom) XML-based standard Designed for news-oriented websites to “Push” content to

readers Excellent to monitor new content from websites

JavaScript Object Notation (JSON) Lightweight data-interchange format Human readable and writable and also machine friendly Wide support from most languages (Java, C, C#, PHP,

Ruby, Python…)

JSON Example

http://www.json.org/example.html

7

Google API

Google has many APIs available: http://code.google.com/more/

We will introduce two of them: Google Custom Search API YouTube Data APIs

8

Google Custom Search

What is custom search? to create a search engine tailored to your needs and

interests Your custom search engine (CSE) can prioritize or

restrict search results based on websites you specify

you can present the results in your website

9

Create your CSE

Developer's Guide http://code.google.com/apis/customsearch/docs/dev_guide.html

You can use a provided custom search wizard to create your own CSE: http://www.google.com/coop/manage/cse/create/1

You can use the “control panel” to manage your CSE. You can access the “control panel” from the “My search engines” page. http://code.google.com/apis/customsearch/docs/start.html#managing

10

What’s in a CSE

A custom search engine has two main components housed in XML files: Context: specifies the global settings of the search

engine. Each CSE has its own context file. Annotations: lists the sites (webpages or websites)

you want your search engine to cover. You can view or download these two XML

files in “control panel”->”Advanced”.

11

Context and Annotation Files:

Your CSE ID and your personer ID

12

Access your CSE

Users can access your CSE in four ways: Your custom search engine homepage A search engine gadget in their iGoogle pages A search engine gadget in their webpages or blogs A search box in your website▪ To get your very own search box, copy the generated

code from the “Code” tab of the control panel, and paste it into the HTML of your webpage.

http://code.google.com/apis/customsearch/docs/start.html#release

13

Google Custom Search API

The Custom Search API lets you skip the control panel altogether and use a client application of your choice to : retrieve a list of CSE under an account Create, update and delete custom search engines retrieve code for a search box retrieve current settings of a CSE associate a CSE with an AdSense Account http://code.google.com/apis/customsearch/docs/api.html

14

Request and Response Format

Request Format: standard HTTP requests, such as: GET, POST, and

DELETE. Need to get an authentication token from ClientLogin

using HTTP post Request▪ Generally an authentication token expires in two weeks▪ http://code.google.com/apis/accounts/docs/AuthForInstalledAp

ps.html Response Format:

XML-based Format Details:

http://code.google.com/apis/customsearch/docs/api.html

15

Request Example

A POST request that create a search engine specification (by submitting a context xml).

authentication token

New context file content

Request Endpoint

16

Access CSE Search Results

XML feeds of search results are only available to paid Business account …

Our Method 1: Parse the content of the search result page:▪ http://www.google.com/cse?cx=creatorId:SEid&q=

query&parameter=value▪ creatorID and Seid can be found in context xml file of your CSE▪ Parameter and query details can be found in

http://code.google.com/apis/customsearch/docs/structured_search.html

17

Access CSE Search Results

Method 2: Google Custom Search Element for JavaScript users▪ The generated code from the control panel use this method▪ http://code.google.com/apis/customsearch/docs/js/cselement-devguide.html

Method 3: Use JSON/Atom Custom Search API (RESTful requests)to get

search results in either JSON or Atom format.▪ http://code.google.com/apis/customsearch/v1/overview.html

Method 2 and 3 need to visit the Google APIs console to set up a new project and get an API key. Currently an API key allows 100 queries per day and 100 results at

most for each query. You can request additional quota from its console.

18

Summary for Google CSE API

Step 1: create your own CSE using the wizard (web interface) or the CSE API CSE API needs to apply for authentication token

Step 2: manage your CSE using the wizard or the CSE API if necessary

Step 3: access CSE search results by method 1,2,3. Method 2 & 3 needs to apply for API key

19

Amazon

Amazon Web Services (AWS) http://aws.amazon.com/ “With AWS you can requisition compute power, storage, and other services

…” Amazon SDK for Java

http://aws.amazon.com/sdkforjava/ Amazon Product Advertising API

https://affiliate-program.amazon.com/gp/advertising/api/detail/main.html

Access Amazon Product Selection▪ gives you access to Amazon’s selection of millions of products in categories such as

books, music … Leverage Amazon Product Discovery capabilities▪ lets you leverage Amazon’s customer-centric features such as Product Search,

Customer Reviews, Similar Products... Monetize your website

20

Amazon Product Advertising API

Programming Guide http://docs.amazonwebservices.com/AWSECommerceService/latest/D

G/index.html?ProgrammingGuide.html Check “Visual Introduction to Product Advertising API” to

see what this API can do. E.g.: to find items

21

Amazon Product Advertising API: Request Format

Tutorial: http://docs.amazonwebservices.com/AWSECommerceService/latest/DG/index.

html?CHAP_MakingRequestsandUnderstandingResponses.html Format: REST & SOAP REST Format:

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService&

Operation=ItemSearch& AWSAccessKeyId=[Access Key ID]&AssociateTag=[ID]&

SearchIndex=Apparel& Keywords=Shirt&Timestamp=[YYYY-MM-DDThh:mm:ssZ]

&Signature=[Request Signature]

22

Amazon Product Advertising API: Request Authentication

Tutorial: http://docs.amazonwebservices.com/AWSECommerceServic

e/latest/DG/index.html?RequestAuthenticationArticle.html

To access the Product Advertising API, you must first create an account at http://aws.amazon.com. AWS will assign you a pair of related identifiers: Access Key ID (a 20-character, alphanumeric sequence) ▪ include it in AWS service requests to identify yourself as the sender

of the request Secret Access Key (a 40-character sequence)▪ To calculate the signature used in your request to verify your identity

23

HMAC-SHA Signatures for REST Requests Authentication

Two authentication parameters: Timestamp: ▪ a dateTime object, with the complete date plus hours,

minutes, and seconds▪ Format : YYYY-MM-DDThh:mm:ssZ (where T and Z

are literals) Signature:▪ created by using the request type, domain, the URI, and a

sorted string of every parameter in the request ▪ Once properly formatted, you create a base64-encoded

HMAC_SHA256 signature using your AWS secret key

24

HMAC-SHA Signatures for REST Requests Authentication (cont.)

Tutorial to sign a request, using sample AWS access identifiers (READ IT!). http://docs.amazonwebservices.com/AWSECommerceService/latest/D

G/index.html?rest-signature.html Basic Authentication Process

25

Amazon Product Advertising API: Response

Response is a XML file which conforms to a WSDL and schema.

WSDL location: http://docs.amazonwebservices.com/AWSECommerceService/latest/DG/i

ndex.html?WSDLLocation.html

Paging Through Results http://docs.amazonwebservices.com/AWSECommerceService/latest/DG/i

ndex.html?PagingThroughResults.html E.g., for itemsearch request, use “&ItemPage=4 ” to access fourth page of

the results.

Sorting Results http://docs.amazonwebservices.com/AWSECommerceService/latest/DG/i

ndex.html?SortingResults.html E.g., for itemsearch request, use “&Sort=titlerank” to sort results by title

or description

26

Amazon Product Advertising API:Steps to search Amazon

Steps to search Amazon: Apply for a Access Key ID and Secret Access Key Generate search conditions Generate a parameter string Generate a string used for signature Generate a signature Generate a query URL Send a search request to Amazon server Receive the query result (XML format) Parse the result and display it on your web site

27

eBay API

Main page: http://developer.ebay.com/common/api/

eBay Web Services Overview http://developer.ebay.com/products/overview/ Available APIs include:▪ Finding API, Best Match API, Merchandising API▪ Shopping API, Feedback API, Trading API, Client Alerts API ▪ Platform Notifications API, Research API for eBay, etc.

Which API to use with your application?▪ http://developer.ebay.com/products/overview/api-by-feature/

28

eBay Finding API

eBay Finding API: to search for eBay items: http://developer.ebay.com/products/finding/ Search for items based on a keyword query Search for items associated with a specific product Browse for items in a specific category or in an eBay store Filter search results by item listing details, such as price, condition, etc. Refine searches by the characteristics of an item (i.e., aspects), such as brand, style, etc. Retrieve category and aspect metadata Get keyword recommendations to improve search results

Formats: XML, SOAP, Name Value, JSON

Protocols: HTTP GET and POST

Need to apply for a API key https://developer.ebay.com/join/default.aspx

29

eBay Finding API: Call Structure

Tutorial for making a API call: http://developer.ebay.com/DevZone/finding/Concepts/MakingACall.html

Each Finding API call consists of the following elements: Service Endpoint HTTP Headers or URL Standard Input Fields Call-specific Fields

URL-based call structure http://svcs.ebay.com/services/search/FindingService/v1?

OPERATION-NAME=findItemsByKeywords

&SERVICE-NAME=FindingService

&SERVICE-VERSION=1.0.0

&GLOBAL-ID=EBAY-US &SECURITY-APPNAME=MyAppID

&RESPONSE-DATA-FORMAT=XML

&REST-PAYLOAD

&Standard input fields

&Call-specific input fields

30

eBay Finding API: findItemsByKeywords

findItemsByKeywords lets you to search for items on eBay by a keyword query http://developer.ebay.com/DevZone/finding/CallRef/findItemsByKeywords.html

Controlling Results with Item Filters▪ &itemFilter(0).name=FreeShippingOnly&itemFilter(0).value=true

&itemFilter(1).name=ListingType&itemFilter(1).value=FixedPrice Refining Results with Aspect and Domain Filters▪ &aspectFilter.aspectName=Sub Style&aspectFilter.aspectValueName=Running

Shoes...▪ &domainFilter.domainName=Digital_Cameras

Finding Local Items▪ &buyerPostalCode=95125&sortOrder=Distance

Sorting the Output: ▪ &sortorder= PricePlusShippingAsc

Paginating the Results▪ &paginationInput.pageNumber=2&paginationInput.entriesPerPage=50

31

eBay Finding API: findItemsByKeywords Examples

Call Example: The user is looking for a specific Harry Potter title and has a maximum price ceiling and doesn't want to pay for shipping. http://svcs.ebay.com/services/search/FindingService/v1?

   OPERATION-NAME=findItemsByKeywords&   SERVICE-VERSION=1.9.0&   SECURITY-APPNAME=YourAppId&   RESPONSE-DATA-FORMAT=XML&   REST-PAYLOAD&   keywords=harry%20potter%20phoenix&   itemFilter(0).name=MaxPrice&   itemFilter(0).value=10.00&   itemFilter(0).paramName=Currency&   itemFilter(0).paramValue=USD&   itemFilter(1).name=FreeShippingOnly&   itemFilter(1).value=true&   paginationInput.entriesPerPage=2

PHP coding Example and Tutorial: http://developer.ebay.com/DevZone/finding/HowTo/GettingStarted_PHP_NV_XML/GettingStarted_PHP_NV_XML.html

32

eBay Finding API: Steps to search eBay

Steps to search eBay: Apply for a API key Construct the request Send the request to eBay Server Receive the query result in your preferred format Parse the result

33

Flickr API: General

Flickr API Main Page: http://www.flickr.com/services/api/

API Key Application http://www.flickr.com/services/api/keys/

Request Format Rest, XML_RPC, SOAP

Response Format Rest, XML_RPC, SOAP, JSON, PHP

34

Flickr API: Photo Search

Flickr Photo Search Method Description: http://www.flickr.com/services/api/flickr.photos.search.html

Explorer: http://www.flickr.com/services/api/explore/?method=flickr.photos.search

Steps: Apply for a Flickr API Key Send a search request to Flickr Server in your preferred

format Receive the response in your preferred format Parse the result and get the photo URLs▪ To construct photo URLs:

http://www.flickr.com/services/api/misc.urls.html

35

Flickr API: Photo Search Example – REST format

Request Format: http://api.flickr.com/services/rest/?

method=flickr.photos.search&api_key=yourkey&argument=value Example:

Request (to search photos that have “tucson” in its description): http://api.flickr.com/services/rest/?method=flickr.photos.search&api_key=e7c88ea43041aff2df564188d4609849&text=tucson

Response:

Extracted Image URL and Web Page URL:▪ http://farm6.static.flickr.com/5288/5368668124_d216292c15.jpg▪ http://www.flickr.com/photos/48553200@N04/5368668124

36

YouTube API

YouTube Data API: lets you incorporate YouTube functionality into your own

application or website. You can perform searches, upload videos, create playlists, and more.

Main Page: http://code.google.com/apis/youtube/getting_started.html#data_api

Java Developer Guide: http://code.google.com/apis/youtube/2.0/developers_guide_java.html

YouTube Player API give you control over YouTube video playback on your

website.

37

YouTube Data API

Advanced functions of the data API requires its java client library http://code.google.com/apis/youtube/2.0/developers_guide_java.html#Getting_

Started

You will need to apply for a developer key and client id to execute authenticated functions. http://code.google.com/apis/youtube/2.0/developers_guide_java.html#Authenti

cation

Two important class: VideoFeed: represent lists of videos, such as standard feeds, uploads,

subscriptions, and favorite videos VideoEntry: Each video entry corresponds to exactly one YouTube video and

contains information about that video.

38

YouTube Video Search

Description of the YouTube video search function http://code.google.com/intl/en/apis/youtube/2.0/developers_guide_java.html#Searching_for_Videos

Query parameter definition http://code.google.com/intl/en/apis/youtube/2.0/reference.html#Query_parameter_definitions

A search query example for videos that match the search term “puppy”. http://gdata.youtube.com/feeds/api/videos?q=puppy&orderby=viewCount

The query example can be generated using the following code: YouTubeQuery query =

new YouTubeQuery(new URL("http://gdata.youtube.com/feeds/api/videos")); query.setOrderBy(YouTubeQuery.OrderBy.VIEW_COUNT); query.setFullTextQuery("puppy"); //print resultVideoFeed videoFeed = service.query(query, VideoFeed.class); printVideoFeed (videoFeed, true);

39

YouTube Video Search Steps

Steps to search YouTube Videos: Download client library Apply for Developer key and client id Construct a search request and send to YouTube Receive the query result Parse the result and display videos on your

websites.

40

Second Life API

Second Life API Main Page: http://wiki.secondlife.com/wiki/Web_Services_Portal

There are eight official APIs: Registration API Map API Exchange Risk API Media Plugin API Live Data Feeds Inventory API Snapshot API Search API

41

Second Life API: Search API

The Second Life Search API enables you to search Second Life for: Events, People, Places, Groups, and information on

its Wiki. http://wiki.secondlife.com/wiki/SearchAPI

Linden Lab uses a Google Search Appliance to index web pages. The Search API allows to search the indexed web pages. No API key is needed.

42

Second Life API: Search API Request Format

Request format: http://search.secondlife.com/client_search.php?

q=yourquery&paraemeter=value Two popular parameters:

start▪ Each query will return 20 results at most (wiki says 10)▪ “start=0” shows the first 20 results, “start=20” shows 21-40, etc.

s▪ To limit the results to specific categories▪ This value defaults to "All", but can also be set to one of the

following: All, Events, Groups, People, Places, Wiki.

43

Second Life API: Search API Response Format

The response format is XHTML. Example:

http://search.secondlife.com/client_search.php?q=arizona&s=All&start=20

44

Second Life API: Search API Steps

Steps: Send a search request to Second Life Server Parse the XHTML response page to get each result

page URL. Parse each result page to get information:▪ Pages in each category share similar structure. You can

develop separate web page parsers for group page, event page, place page, etc.

45

XML Processing

A variety of APIs for accessing XML have been developed and used, and some have been standardized. Existing APIs for XML processing tend to fall into these categories: Stream-oriented APIs accessible from a programming language▪ E.g., Simple API for XML (SAX)

Tree-traversal APIs accessible from a programming language.▪ E.g., Document Object Model (DOM)

XML data binding, which provides an automated translation between an XML document and programming-language objects▪ E.g., Java Architecture for XML Binding (JAXB)

Declarative transformation languages▪ E.g., XSLT, XPath and XQuery.▪ XPath Tutorial: http://www.w3schools.com/xpath/default.asp

46

DOM

Document Object Model (DOM) DOM accesses an XML document through a tree

structure, composed of element nodes and text nodes.

47

DOM Example

<a href="http://www.google.com/">Google's<b>fast</b> home page.</a>

48

DOM

Pros of DOM: Easy to manipulate the document Can traverse the document back and forth Good for small XML files

Cons of DOM: Consumes lots of memory

JDK provides two packages related to DOM: org.w3c.dom javax.xml.parsers

Some popular JAVA xml parsers: Xerces JDOM DOM4j

49

JSON Processing

JSON Tutorial: http://www.json.org/

A lot of developed JSON tools and parsers are listed in the website. Listed by different language

For example: http://www.json.org/java/