View
357
Download
3
Category
Preview:
Citation preview
Understanding Business APIs through Statistics
Amila De SilvaDinusha Senanayaka
About WSO2
• Providing the only complete open source componentized cloud platform
– Dedicated to removing all the stumbling blocks to enterprise agility– Enabling you to focus on business logic and business value
• Recognized by leading analyst firms as visionaries and leaders– Gartner cites WSO2 as visionaries in all 3 categories of
application infrastructure– Forrester places WSO2 in top 2 for API Management
• Global corporation with offices in USA, UK & Sri Lanka– 200+ employees and growing
• Business model of selling comprehensive support & maintenance for our products
150+ globally positioned support customers
Overview
● Role of Statistics in API Management○ Statistics to understand user communities○ As tool for making strategic decisions.
● Different Options for analysing data○ Phases in Data analysis○ Offline vs Real-time analysis
● BAM as an offline data analytic engine○ How large event streams gets summarized
● CEP for real time processing○ Architecture of CEP○ Stages of an execution plan
● GA as a feedback channel for app developers● Options for visualising data.
○ Generating reports out of BAM summarized data.
APIs - the way to go
● APIs are no longer a luxury - they are becoming a necessity.
● API - A common interface for disparate platforms
● More open you are more users you will attract.
● Build an ecosystem around your APIs.
Knowing your users
Why do you need statistics?
● There are many connected components● One wrong move can make disastrous
consequences.● Feedback channels are important -
○ See how users access them and learn - It explains what they say.
● Discover hidden potentials within your communities○ Going for a new billing model - See how users
access them.● Can you check them all manually?
What we provide
● Two branches in analysis○ Offline Data analysis○ Realtime Analaysis
● Offline Data Analysis○ Gives you a general idea of your APIs○ Who your users are.○ Leads you to make long term decisions.
● Realtime analysis○ Identifies co-occurrence of several useful business
flows.○ Find and act on it.
How we analyse it
● Done in two main phases○ Data Gathering○ Analysis.
● Gather data at product level.○ multiple event streams for request, response and faults○ each invocation as an event
● Delegating analysis to a different product○ Simply any tool capable of processing event streams can
analyse data.○ Currently BAM and CEP are used.
● Extensions to achieve product specific analysis○ In BAM this is done through a hive script○ In CEP this can be done by a transport adaptor + execution
plan.
Data Gathering
● Event = Invocation● In the gateway we collect data related to
each invocation● Three different event streams
○ Request Stream○ Response Stream○ Fault stream
Structure of a StreamRequest Stream
'payloadData':[ {'name':'consumerKey','type':'STRING'}, {'name':'context','type':'STRING'}, {'name':'api_version','type':'STRING'}, {'name':'api','type':'STRING'}, {'name':'resource','type':'STRING'}, {'name':'method','type':'STRING'}, {'name':'version','type':'STRING'}, {'name':'request','type':'INT'}, {'name':'requestTime','type':'LONG'}, {'name':'userId','type':'STRING'}, {'name':'tenantDomain','type':'STRING'}, {'name':'hostName','type':'STRING'}, {'name':'apiPublisher','type':'STRING'},
{'name':'applicationName','type':'STRING'}, {'name':'applicationId','type':'STRING'} ]
Response Stream
'payloadData':[ {'name':'consumerKey','type':'STRING'}, {'name':'context','type':'STRING'}, {'name':'api_version','type':'STRING'}, {'name':'api','type':'STRING'}, {'name':'resource','type':'STRING'}, {'name':'method','type':'STRING'}, {'name':'version','type':'STRING'}, {'name':'response','type':'INT'}, {'name':'responseTime','type':'LONG'}, {'name':'serviceTime','type':'LONG'}, {'name':'userId','type':'STRING'}, {'name':'tenantDomain','type':'STRING'}, {'name':'hostName','type':'STRING'}, {'name':'apiPublisher','type':'STRING'}, {'name':'applicationName','type':'STRING'}, {'name':'applicationId','type':'STRING'} ]
Fault Stream
'payloadData':[ … {'name':errorCode','type': 'STRING '}, {'name':‘errorMessage','type': 'STRING '}, .. ]
Analysing Data with BAM...
Analysing Data with BAM
● Used for offline Analysis.● Aggregates, stores, then analyses data.● Capable of receiving events from multiple data
agents.○ In APIM 1.5.0 we use a load balancing data publisher.
● All the incoming events are first written to a Cassandra DB
● Data analysed using a hive script running on top of a hadoop cluster.
● Summarised data written back to a Relational DB.
Analysing Data with BAM...
Summarized tables
Analysing Data with BAM...
Table Descriptions
Analysing Data with BAM...
Visualizing summarized data
Analysing with BAM...
Summarized data usages for billing
● Decide on a billing model● Integrate with existing billing engines
Real-time Analysis with CEP
● APIM can publish events to multiple data agents.
● Same streams are used to publish events to CEP
● Whenever it receives a new event CEP executes a query and evaluates the event.
● Siddhi queries define the evaluation criteria.
CEP Architecture
CEP Architecture...
● Transport Adaptors to read/write event sources.○ We can create alternate event streams out of the
original streams.● Event builders convert events to a standard
format.● Queries defined as query plans.● Event processor executes a query plan.
○ It might generate additional events.● Event formatters convert back the output.
Example
● Task : Identify client applications sending too many repeated requests○ Too Many : 5 requests ○ repeated : within 1 minute
● How to identify a client○ Consumer key
And the query is:from AMRequest#window.time(1 min)insert into outStreamcount(consumerKey) as myCount,api_version,api,consumerKeygroup by consumerKey,api_version,apihaving (myCount > 5);
Flow in CEP...
● Transport adaptor reads requestStreamDefn● Creates a second input stream AMRequest
○ This only contains subset of data.● Event builder transform the input stream● Query is defined in a query plan.
○ Even processor executes each of the registered query plans.
● Upon meeting the condition an event is generated.● Event formatter converts the event.● Mail is sent through the transport adaptor.
Monitoring Statistics with Google Analytics
• API Manager can be easily configured with Google Analytics
<GoogleAnalyticsTracking><Enabled>false</Enabled>
<TrackingID>UA-XXXXXXXX-X</TrackingID></GoogleAnalyticsTracking>
Monitoring Statistics with Google Analytics
Visualising
● Needed for interpreting statistics.● Gadgets provided in BAM● Plugin any external tool for generating reports● Combining summary tables to create new
views.
Summary
• Statistics as a channel to verify business objectives.
• WSO2 API Manager provides different mechanisms to analyze data.
• WSO2 BAM can be integrated with API Manager process offline data.• Gather -> Cassandra• Analyze -> Relational DB
• WSO2 CEP can be integrated with API Manager and real time data analyzing.
• Google Analytics integration with API Manager.
Engage with WSO2
• Helping you get the most out of your deployments• From project evaluation and inception to development
and going into production, WSO2 is your partner in ensuring 100% project success
Recommended