Michael RysPrincipal Program Manager, Big Data @ Microsoft@MikeDoesBigData, {mrys, usql}@microsoft.com

Using C# with U-SQL

2016/04/04

Extensible From Ground Up• Type system is based on C#• Expression language IS C#• User-defined functions (U-SQL

and C#)• User-defined Aggregators (C#)• User-defined Operators (UDO)

(C#)U-SQL provides the Parallelization and Scale-out Framework for Usercode• EXTRACTOR, OUTPUTTER,

PROCESSOR, REDUCER, COMBINER, APPLIER

REFERENCE MyDB.MyAssembly;

CREATE TABLE T( cid int, first_order DateTime , last_order DateTime, order_count int , order_amount float );

@o = EXTRACT oid int, cid int, odate DateTime, amount float FROM "/input/orders.txt" USING Extractors.Csv();

@c = EXTRACT cid int, name string, city string FROM "/input/customers.txt" USING Extractors.Csv();

@j = SELECT c.cid, MIN(o.odate) AS firstorder , MAX(o.date) AS lastorder, COUNT(o.oid) AS ordercnt , AGG<MyAgg.MySum>(c.amount) AS totalamount FROM @c AS c LEFT OUTER JOIN @o AS o ON c.cid == o.cid WHERE c.city.StartsWith("New") && MyNamespace.MyClass.MyFunction(o.odate) > 10 GROUP BY c.cid;

OUTPUT @j TO "/output/result.txt"USING new MyData.Write();

Extending U-SQL

Refactoring Code with Code-behind

Sharing Code with U-SQL Assemblies

Managing AssembliesCreate assembliesReference assembliesEnumerate assembliesDrop assemblies

• CREATE ASSEMBLY db.assembly FROM @path;• CREATE ASSEMBLY db.assembly FROM byte[];

• Can also include additional resource files

• REFERENCE ASSEMBLY db.assembly;

• Referencing .Net Framework Assemblies• Always accessible system namespaces:

• U-SQL specific (e.g., for SQL.MAP)• All provided by system.dll system.core.dll

system.data.dll, System.Runtime.Serialization.dll, mscorelib.dll (e.g., System.Text, System.Text.RegularExpressions, System.Linq)

• Add all other .Net Framework Assemblies with:REFERENCE SYSTEM ASSEMBLY [System.XML];

• Enumerating Assemblies• Powershell command• U-SQL Studio Server Explorer

• DROP ASSEMBLY db.assembly;

Assembly Dependencies

• Assembly must be registered to be referenced• All Assemblies needed for compilation must be referenced in script• All Assemblies needed at runtime either• Need to be referenced in script, or• Need to be registered with the assembly as additional files

• Metadata Service does NOT enforce dependencies• Visual Studio Extension provides support for dependency management

Additional Resources

MSDN Articlehttps://msdn.microsoft.com/en-us/magazine/mt614251

Sample Datahttps://github.com/Azure/usql/tree/master/Examples/Samples/Data/Tweets

Sample Projecthttps://github.com/Azure/usql/tree/master/Examples/TweetAnalysis

http://aka.ms/AzureDataLake