CMPE 275

Project 2 Report

RESTful social collaboration through Pinterest re-enactment

Team:

Bhushan Deo (009269403)

Deven Pawar (009275994)

Dhruv Gogna (005067284)

Gaurav Bhardwaj (009297431)

Vinay Bhore (009290931)

Introduction The main goal of Project 2 is to explore the design and challenges in developing RESTful web services for

a social media service such as Pinterest. Sharing content is an important pattern found in all social media

services. Providing robust and well documented web services is very important in such products as there

are several external clients which use them to access and manipulate stored user data. In this report, we

provide a detailed explanation of our team’s design and implementation.

High Level Architecture

The team’s configuration consists of the following components:

1. Web Services:

We expose our functionality to the client using RESTful web services written in Python using the

Bottle micro-framework.

2. Database:

CouchDB NoSQL database is used to persist user data. It provides data storage as JSON

documents and thus naturally maps to the requirements. Also, when client hits our web service

and some data is changed, we can verify changes in the database through the web interface

(Futon), that comes with CouchDB.

3. couchdb-python library:

This library provides interface to interact with CouchDB from python code. In our experience,

this was a very well designed library with advanced features like high-level couchDB document

to python object mapping, intuitive interfaces and good documentation.

4. Client:

For testing, we are using curl to hit our web services and see the response. Curl provides many

advanced options to make testing via HTTP requests easy.

The following sections explain the functionalities implemented and the web service and database

component in detail. These include comments about what we learnt while working on each component.

Functionalities This project also required cross team interaction to come up with a stable specification for the web

services. As per the specification agreed to by the class, following functionalities were implemented by

our team.

Here, we observed how the traditional Create, Read, Update and Delete operations in databases, map

exactly to HTTP methods POST, GET, PUT and DELETE respectively.

Functionality Endpoint (fields in angular brackets are generated on the server)

CRUD? HTTP Met-hod

Request Parameters (all are string type unless specified)

Sign up http://localhost:8080/users/signUp Create POST firstName lastName emailId password

Login http://localhost:8080/users/login/ Create POST emailId password

Create a board

http://localhost:8080/users/<userId>/boards/ Create POST boardName boardDesc category isPrivate(Boolean)

Get all boards

http://localhost:8080/users/<userId>/boards/ Read GET None

Get details of a single board

http://localhost:8080/users/<userId>/boards/<boardId>/ Read GET None

Update Board

http://localhost:8080/users/<userId>/boards/<boardId>/ Update PUT boardName boardDesc category isPrivate(Boolean)

Delete Board http://localhost:8080/users/<userId>/boards/<boardId>/ Delete DELETE None

Create Pin http://localhost:8080/users/<userId>/boards/<boardId>/pins/

Create POST pinName image description

Get a single pin from a board

http://localhost:8080/users/<userId>/boards/<boardId>/pins/<pinId>/

Read GET None

Get all pins from a board

http://localhost:8080/users/<userId>/boards/<boardId>/pins/

Read GET None

Update a pin http://localhost:8080/users/<userId>/boards/<boardId>/pins/<pinId>/

Update PUT pinName image description

Delete a pin http://localhost:8080/users/<userId>/boards/<boardId>/pins/<pinId>/

Delete DELETE None

Add a comment to a pin

http://localhost:8080/users/<userId>/boards/<boardId>/pins/<pinId>/comment/

Create POST description

View comments on a pin

http://localhost:8080/users/<userId>/boards/<boardId>/pins/<pinId>/comment/

Read GET None

The responses were returned in JSON format and contained relevant data and endpoint information for

subsequent operations as agreed to by the class.

Web Services Why Web Services fit in this project?

Essentially, web services help de-couple the client and the server implementations. User performs

various CRUD operations on an external system by sending HTTP requests through the client. The web

service layer, intercepts these requests on the server, performs operations on the database and sends

back appropriate response to the client. This fits well with the sharing and collaboration required by

social media products. From real world examples such as Facebook, Twitter and Pinterest, it can be seen

that a robust and well documented RESTful API is a vital contributor to these products going viral.

Bottle:

Web services were written in Python using the Bottle micro-framework. We found Bottle to be very

intuitive for creating our web services. The framework is very lightweight and in order to use it, it only

requires us to include one bottle.py file in the project directory.

Server:

Bottle runs by default on a single threaded WSGI server, but has adapters to various multi-threaded

servers such as paste and Cherrypy which give better performance for applications that need to handle

large number of concurrent requests. For testing this project, we have used the default server.

Mapping requests to appropriate methods:

Various endpoints as mentioned in the functionalities section, map easily to corresponding methods

which process the requests at those endpoint, simply by placing a @route decorator above that method.

Retrieving data from request:

In REST, the URL path determines the data that will be accessed or manipulated and the HTTP method

determines the operation that will be performed on that data.

- To get information about data that will be accessed, we use wildcards in the URL:

@route(‘http://localhost:8080/users/<userId>/boards/<boardId>/’,method=’GET’)

def getBoard(userId, boardId):

# got userId and boardId from the URL, now perform GET(read) on database

- To retrieve data from the HTTP request, bottle provides convenient methods in the request object:

@route(‘http://localhost:8080/users/<userId>/boards/<boardId>/’,method=’PUT’)

def updateBoard(userId, boardId):

# got userId and boardId from the URL, now perform PUT(update)on database

# with the fields sent in the request by the client

boardName = request.forms.get('boardName')

boardDesc = request.forms.get('boardDesc')

Returning JSON response:

JSON was the decided as the format in which response will be sent back to the client. Even in this case,

Bottle made it easy to return back response with “application/json” content type. We simply had to

return a python dictionary object. Bottle returns the response in JSON and the appropriate field is set in

the HTTP header.

@route(‘http://localhost:8080/users/<userId>/boards/<boardId>/’,method=’PUT’)

def updateBoard(userId, boardId):

# perform appropriate operations

return dict(status='OK', message='sample message')

If other response types such as html or xml are expected by the client, we are aware that we would have

to check for the content types which the client accepts from the HTTP request header and accordingly

format the response before sending it back (content negotiation).

Handling trailing ‘/’ slash:

If some clients put a trailing slash at the end of the endpoint URL, then that is treated as a separate

route by Bottle and might not map to the appropriate method in our web service code. Given that we

would be testing across teams, this was highly probable. To handle this, Bottle documentation suggests

to simply map two routes to the same method, one with a trailing slash and one without a slash.

@route(‘http://localhost:8080/users/<userId>/boards/<boardId>/’,method=’PUT’)

#No trailing slash

@route(‘http://localhost:8080/users/<userId>/boards/<boardId>’,method=’PUT’)

def updateBoard(userId, boardId):

# perform appropriate operations

Database In REST web services, we can think of the URL as a path that is traversed to get to the data that is to be

accessed or manipulated. Further, the HTTP method that is specified in the HTTP request gives us the

operation (create, read, update or delete) that is to be performed on the data. Accordingly the actions

that we take in code depend on the combination of the following factors:

1. Response format accepted by the client (json, xml, html, image, etc..)

2. Data that is to be accessed (URL traversal)

3. What kind of operation on the data (HTTP method type)

Thus, for every endpoint there are several combinations that need to be handled so that we process the

user request appropriately and so that we return desired response back to the user. In this context, it is

clear that we have to organize our data based on the endpoints that we expose to the user.

We observed that this is a notable difference compared to Project 1, where there was only one endpoint

for the client, and the server queue delegated the message to the appropriate resource to handle. In

Project 2, there are several endpoints for the client and each endpoint maps to one method on the server

side.

The flat schema that we have used to store data in CouchDB is as follows. The “_id” is the userId that is

generated server side when the user signs up. This is passed around in each URL and is used to identify

user in each subsequent request.

{

"_id": {

"firstName": "",

"lastName": "",

"emailId": "",

"password": "",

"isLoggedIn": false,

"boards": [

{

"boardId": "",

"boardName": "",

"boardDesc": "",

"category": "",

"isPrivate": "",

"pins": [

{

"pinId": "",

"pinName": "",

"image": "",

"description": "",

"comments": []

}

]

}

]

}

}

Drawback:

The limitation of this schema is that for any request, we get the entire complex document associated

with a userId from the database. Then, in case of long URLs we have to employ multiple nested loops to

get to the data that we require. For example, updating information about a pin needs three nested

loops through the document that is returned. This is shown in the figure above. This would be

problematic if a user had thousands of boards and pins.

A solution to this would be to create separate tables (“databases” in CouchDB) for each step in the

endpoint URL tree, i.e. separate tables for user, board and comments, each record containing reference

to the userId. However, the tradeoff here is that this would require more queries to the database per

user request.

Note on Aerospike:

Initially we decided on using Aerospike as our database. However, its Python client seems to be under

development and there was no support for list and map data structures. Further, there was no query

support in the Python client, only full table scan was supported. Hence we decided to go with CouchDB.

Nonetheless, it was interesting to understand Aerospike’s architecture and concepts such as location

awareness of client, peer relationship among nodes, use of binary wire protocol for communication and

cross data center replication.

Conclusion For our team, following are the major learnings from this project:

1. Programming in Python

2. RESTful web services

3. Database design for RESTful web services

4. Micro-frameworks

5. Comparison of technologies used in Project 1 and 2

6. CouchDB and Aerospike

7. Collaboration in Small and Large Software Development Teams