OData Paging using SSIS – REST API Paging ExampleReach for the Clouds, Inc.Next Generation SSIS Tasks and Connectors Series

AUTHOR: NAYAN PATEL | SR. ETL SSIS ARCHITECTN PAT E L @ R FTC LO U D S . C O M

Content

• Introduction• REST API Paging concept• OData Paging Example using SSIS JSON Source• Twitter REST API Paging Example using SSIS JSON

Source• Conclusion

Introduction

In this article we will learn how to use SSIS JSON Source Component to fetch JSON data from OData service and perform OData Paging of large results. We will see Twitter REST API Paging example too where we extract JSON Data and Loop through multiple requests will be covered

REST API Paging concept

MOST OF REST API SERVICES LIMIT THE SIZE OF THE DATASET RETURNED IN A SINGLE REQUEST. WHEN MORE DATA IS FOUND IT SIMPLY INCLUDES A POINTER TO THE NEXT RESULT SET. IF YOU WISH TO FETCH ALL DATA (E.G. LOOP THROUGH) THEN YOU HAVE TO CHECK NEXT RESULT SET INDICATOR AND IF ITS NOT NULL THEN YOU CAN CONTINUE FETCHING UNTIL ALL DATA IS RETRIEVED.

OData Paging Example using SSIS JSON SourceLETS TALK ABOUT HOW PAGING IS IMPLEMENTED IN ODATA PROTOCOL. CHECK BELOW THIS IS A TYPICAL ODATA SERVICE RESPONSE.REQUEST URL: HTTP:/ /SERVICES.ODATA.ORG/V3/NORTHWIND/NORTHWIND.SVC/ORDER_DETAILS?$FORMAT=JSONRESPONSE:File1=@c:\data\somedata.txtFile2=@c:\data\someimage.pngSomefield1=abcdefSomefield2=123456Extrafield3WithNewLine=This is line1\r\nThis is line2 with tab\t and more \t tabSomefield3StartsWithAt=\@MyTwitterHandle

{ "odata.metadata": "http://ser "value": [ { "OrderID": 10248,

OData Paging Example using SSIS JSON Source

AS YOU CAN SEE WHEN MORE DATA NEEDED TO COMPLETE RESULT SET ODATA RESULT INCLUDES NEXTLINK ATTRIBUTE. IF YOU ARE USING THE SSIS JSON SOURCE COMPONENT THEN SIMPLY SELECT ODATA FORMAT AS BELOW SCREENSHOT.

ODATA PAGING US ING SS IS JSON SOURCE

OData Paging Example using SSIS JSON Source

ONCE YOU EXECUTE PACKAGE SEE HOW IT WILL PULL ENTIRE RESULTSET BY LOOPING THROUGH NEXT URLS (UNTIL ITS NULL)

Twitter REST API Paging Example using SSIS JSON Source

TWITTER REST API IS NOT ODATA FORMAT BUT YOU CAN SEE SIMILAR TYPE OF CONCEPT THERE TOO. IN TWITTER YOU DON’T HAVE PARTIAL URL LIKE ODATA FOR NEXT L INK BUT IT PROVIDES YOU SOME SORT OF RECORD IDENTIF IER. READ HERE FOR MORE INFORMATION ON TWITTER REST API PAGING TECHNIQUE.

IN OUR CASE SSIS JSON SOURCE SUPPORTS PAGING VERY WELL SO WE ARE COVERED. TO LOOP THROUGH MULTIPLE RESULT SETS OF TWITTER DATA SIMPLY CONFIGURE FOLLOWING 3 PROPERTIES. SEE BELOW SCREENSHOT.

Conclusion

BASED ON ABOVE EXAMPLES YOU CAN SEE THAT HANDLING RAGING IN REST API CALL IS NOT STRAIGHT FORWARD BUT USING COMPONENTS LIKE SSIS JSON SOURCE CAN TAKE SOME WORK OUT OF YOUR PLATE.