Upload
pascal-alberty
View
99
Download
5
Tags:
Embed Size (px)
DESCRIPTION
Présentation sur l'architecture de TweetWall Pro, des API Twitter, des challenges de l'évolution du produit
Citation preview
Pascal Alberty@zorrobiwan
CTO Managing Partner
History
More than 50 web specialistsAbout 20 companies or independent people
RueDuWeb
Where ?
Main Architecture
● Front (Akimedia)
○ Public Website○ Private Website○ Flash Animations○ Sessions configuration management
● Back (Tesial)
○ Sessions content management
Front - Key components (Akimedia)
● CMS Akiwi (PHP) & MySQL database (MyISAM)○ Pubic Website○ Private Website (users, eCommerce
● Flash animations○ Displaying messages and other features
● “Manager” in PHP○ Communication between Animations/Front/Back
Back - Key Components (Tesial)
● Engine○ Java web application○ API used by Front for data management
■ sessions, tweets, moderation, tweetos,...
● Batch○ Java web application○ Formerly grabbing Tweets○ Grabbing pictures (avatars & medias)
Technical Infrastructure
● Front/Back Server
● DB Servers○ Front / Back○ Replication (fail over)
● Dev Server○ dev (front / back)○ pré-prod (front / back)
SVN
● KISS
● dev = trunk● pre-prod = branch● prod = tag on branch or trunk
● front = release by switch● back = release by packaging
Back Software Architecture
Maven 2Spring 3.1Hibernate 3.2ehcache 1.5Twitter4J 3Maya 0.2...
Back Software Architecture
● DTO & DAO○ Data Structure & persistence
■ session, tweet, tweetos, hashtags, ...● Services
○ Business functionalities○ Transaction management
● Clients○ Engine○ Batches
Engine
● Java Web Application
● API REST○ CRUD operations
● Used by Front for data management
● Call Services layer
Batches
● Java Web Application
● Quartz Jobs○ Formerly Tweets Grabbing ○ Avatars Grabbing○ Pictures Grabbing
● Call Services Layer
Challenges
● Scalability○ SaaS and Agency are growing
● Twitter○ New technical and business constraints
● Product○ Must evolve by integrating other social networks
Twitter API
● Let’s talk about Twitter API
Twitter APIs
● APIs○ Search API○ REST API○ Streaming API○ https://dev.twitter.com/docs/history-rest-search-api
● Versions○ API version 1 (deprecated)○ API version 1.1
Rate limiting
● Rate limit window duration○ 15 minutes
● Requests allotted○ per user○ via application-only auth
● Example○ GET statuses/user_timeline○ per user: 150 requests per 15 minutes○ via app: 300 requests per 15 minutes
https://dev.twitter.com/docs/rate-limiting/1.1/limits
Other technical limits
● Direct messages○ 250 per day
● Tweets○ 1,000 per day○ Retweets are counted as Tweets.
● Changes to account email○ Four per hour
● Following○ 1,000 per day
● Following (account-based)○ up to following 2,000 other users
Platform objects
● Tweets can be found alone, within user objects, but most often within timelines
● Users can be found tweeting, following, and favoriting on Twitter
● Entities are most often found within Tweets● Places can be found throughout the natural universe,
but typically only appear attached to Tweets on Twitter
Objects type
● JSON !
● XML output will be dropped soon○ Only JSON output with API 1.1
● Be careful with ids (64 bits)○ use String version (especially if using Javascript)○ Twitpocalypse
■ https://dev.twitter.com/docs/twitter-ids-json-and-snowflake○ Idpocalypse
■ https://dev.twitter.com/blog/64-bit-twitter-user-idpocalypse
Tweet Example
https://api.twitter.com/1/statuses/show.json?id=341545927401680897&include_entities=true
User example
https://api.twitter.com/1/users/show.json?screen_name=zorrobiwan&include_entities=true
Entities / Entity Example
● Hashtags, Medias, URLs, User mentions, Symbols
● No need to parse● Data used for display also (ex: display_url)
Place example
https://api.twitter.com/1/geo/id/247f43d441defc03.json
Libraries
● PHP○ Twitter-async (https://github.com/jmathai/twitter-
async)○ tmhOAuth (https://github.
com/themattharris/tmhOAuth)○ Zend Framework (http://framework.zend.
com/manual/1.12/en/zend.service.twitter.html)● Java
○ Twitter4J (http://twitter4j.org/)
https://dev.twitter.com/docs/twitter-libraries
Twitter4J
● Unofficial Java library for the Twitter API● 100% Pure Java (5+)● Android platform and Google App Engine
ready● Zero dependency : No additional jars
required● Built-in OAuth support● Out-of-the-box gzip support● 100% Twitter API 1.1 compatible
Tools
● Twurl○ https://github.com/marcel/twurl○ "curl" for Twitter API○ Manage access tokens (authentication)
● Apigee Console○ https://apigee.com/console○ Free console to execute APIs
Search API
● Similar to http://search.twitter.com
● Criteria○ terms○ geocode○ language ("Language detection is best-effort")○ count, until, since-id, max-id○ result type: mixed, recent, popular○ include entities
● Paginated !○ you have to manage this by yourself
REST API
● Timelines● Tweets● Direct Messages● Friends & Followers● Users● Suggested Users● Favorites● Lists● Saved Searches● Places & Geo● Trends● Spam Reporting● OAuth● Help
Streaming API
● Persistent HTTP connection○ until you decide to close it
● Could be reeeeaaaaaaally huge !○ Think about you architecture (hard and soft) !○ Separate storage and consumption
● Only one stream opened at a time
● Could be tricky to managed○ Use a library
Streaming API endpoints groups
● Public Streams○ GET statuses/sample○ POST statuses/filter○ GET statuses/firehose (limited access)
● User Streams○ Data and events for a specific user
● Site Streams (beta and limited access)○ Real time update for large number of users
About the challenges
● Scalability● Twitter constraints● Product evolution
=> Use of Twitter Stream API=> Other Social Networks integration
=> Maya
Maya
Abstraction layer for feed andpost
Used in TWP as the input sourcefor media
Maya - General architecture
Listener
Listener
SessionSession
Maya - concepts
● Post: content emitted on a (social) network by a user
● Feed: a collection of posts matching a set of filters (PostFilter)
● FeedOperator: the service to register, manage stop and restart feeds
Maya twitter plugin
● Backed by Twitter4J● Rely on the live stream twitter API● Challenges
○ Adding new feeds on the fly without losing any history
○ Single channel for all feeds, need to identify to which feed(s) a tweet belongs to
● Demo!
Front Evolutions
● From Pull to Push○ NodeJS
● Manager from PHP to JS○ NodeJS
● HTML5 Animations
Next Improvements
● Queuing traitement with SI○ Message traitement○ Medias grabbing and/or treatment
● Adding other social networks
● Better scalability - reliability - performance○ load balancing○ replication○ nginx○ ...
Q/A