API DESIGNBEST PRACTICES

FROM A HACKER’S VIEW

• Overview

• Stories

• CrawlallprojectsandbidsfromFreelancer.com

• Crawl6billionflightticketpricefromatravelwebsite

• Summary

MONOLITHICAPP

• Hidesysteminformationinsideapp• Nointernalsyscallisexposedtooutside

MICROSERVICEAPP

• Hackersknow your systembetter

• Servicecallsareexposedtouser

• RESTful APIasstandard, easy to guess

• Needtoconsidersecurity between every

service

TRIDITIONAL MODERN

XPATH

WEB PAGE API

Pure Data

STORY 1CRAWL FREELANCER.COM

FREELANCER.COM

8MProject Information

Bid Information

• Reputation and price, which is the most important factor for a success bid?• How can I get most chance to be awarded when bidding for Australia employer?

• Should Iputalowestpriceorshould Idomoreprojecttoearnreputation

HOW CAN I GET THE INFORMATION AS FAST AS POSSIBLE?

https://www.freelancer.com/projects/Javascript/Web-Page-Scraper/

• NeedaHTMLparserandjavascript executor

• Heavyworkforbothcpu andbandwidth

• Noteasytoiteratethroughalltheprojects

TIP: MOST OF THE TIMEMOBILE SITE IS MUCH EASIER

TOGETINFORMATION

https://m.freelancer.com/projects/Javascript/Web-Page-Scraper/#info

RESTFUL APIS

https://www.freelancer.com/api/projects/0.1/projects/Javascript%2FWeb-Page-Scraper%2F/?compact=true&full_description=true&job_details=true&location_details=true&selected_bids=true&upgrade_details=true&user_avatar=true&user_details=true&user_employer_reputation=true&user_reputation=true&user_status=true

https://www.freelancer.com/api/projects/0.1/projects/9844976/bids/?compact=true&limit=20&offset=0&reputation=true&user_avatar=true&user_details=true

https://www.freelancer.com/api/projects/0.1/projects/9844976/?compact=true&full_description=true&job_details=true&location_details=true&selected_bids=true&upgrade_details=true&user_avatar=true&user_details=true&user_employer_reputation=true&user_reputation=true&user_status=true

https://www.freelancer.com/api/projects/0.1/projects/${id}/?compact=true&full_description=true&job_details=true&location_details=true&selected_bids=true&upgrade_details=true&user_avatar=true&user_details=true&user_employer_reputation=true&user_reputation=true&user_status=true

API Rate Limit1000 / HOUR

8M/1k=8kHOUR=333DAYS

172.246.149.100216.219.130.17195.227.99.19717.124.253.149137.238.189.20734.155.214.35

• Number of threads depends on how many proxies you have• Https proxy is hard to find• Proxiesareunstable• Proxies will beusedoutquickly• High costifyoubuyproxies

WORKAROUND : USE HIGH ANONYMOUS PROXY

160.124.89.7113.193.36.236182.3.152.4485.72.136.122……

• Loads of IPs, canbe changed every 10s• Highquality socks proxiesacrosstheworld• Able to use docker to start 10 tor clients in 1 minute

WORKAROUND: USE TOR NETWORK

USINGTHESEHACKSIMANAGEDTOGETALLTHEPROJECTSANDBIDSIN10DAYS

USEASINGLEDIGITALOCEAN5$SERVER

WHAT DO I LEARN?

APIRateLimitation MobileAPI

Easytoguess filters PredicableURL

https://www.freelancer.com/api/projects/0.1/projects/Javascript%2FWeb-Page-Scraper%2F/?compact=true&full_description=true&job_details=true&location_details=true&selected_bids=true&upgrade_details=true&user_avatar=true&user_details=true&user_employer_reputation=true&user_reputation=true&user_status=true

Information leak

HOW CANWE FIX THEM?

ONLYSUPPLYINFORMATIONCLIENTNEEDS

MAKESUREURLISNOTPREDICTABLE

https://www.freelancer.com/api/projects/0.1/projects/UUID/?compact=true&full_description=true&job_details=true&location_details=true&selected_bids=true&upgrade_details=true&user_avatar=true&user_details=true&user_employer_reputation=true&user_reputation=true&user_status=true

REDUCEANONYMOUSNETWORKATTACK

• IfyourcustomersareinAUonly,restrictaccesswhenIPaddressisoutsideAU

• Setdifferentlimitationbasedonlocation

• 1k/hAPIusage • 100/hAPIusage• Captchatoverifyhuman

STORY 2LEARN FROM CAWLING FLIGHT TICKET PRICE

Howmany days ahead do I need to get a cheapest price?Ineedtocrawlasmanyflightticketpriceandanalysis.

FIND API FROMMOBILE PAGE

data=%7B%22searchType%22%3A%……

useNative=true&ttid=201300@travel_h5_3.1.0&appKey=12574478

t=1426062775998&sign=3feb52aed67967a2c47aa7a2b9f2a417Ifyouaccessthesameurl toreproduceAPIcalls,itwillafter10seconds

ANALYSEAPI

• Parameters inside data parameter:

• Fixed parameter:

• Sign

HOWCANWEGENRATEAVALIDAPICALL?

FIND TRIGGER POINT

Search source code to find API endpoint

REFORMATSOURCECODE

• Reformat code to get readablesource code

• Help to set breakpoint

FINDAPIURLGENERATOR

• Trace down the code to find out howto generate the url

FIND OUT TOKEN GENERATION ALGORITHM

• Set breakpoint andwatch variables to find out the secret

WHAT DO I LEARN?

• Use time token to generate dynamic urls

• Use parameter sign token to verify parameter

• PreventrepeatAPIcalls

• JS obfuscated code is easy to hack

SUMMARY

• Makesureurl isnotpredictable

• Onlysupplyinformationclientneeds

• Reduceanonymousnetworkattack• By different strategy to different location

• Use time token to generate dynamic url

• Use sign to verify request is valid

THANK YOU

Github: derekhe