@ Couchbase Connect16
LINQing to data:Easing the transition
from SQL
Welcome
Brant BurnettSoftware Development Team Lead
Couchbase Community Expert
Welcome
Quick Facts .Net Data Modeling Basics Querying with LINQ Advanced LINQ Features for Couchbase Efficient Indexing for LINQ Queries Change Tracking + Read Your Own Write Q & A
The Agenda
Celebrating 12 Year Anniversary
Team of 50 in Roxboro, NC
Sister company is Palace Pointe, a 100k sq. ft. Entertainment Venue for which we were developed as an in-house system
Over 600 users across the US and abroad
FEC’s, Waterparks, Trampoline Parks, Amusement Parks, Skating Rinks, Bowling Centers, Zoos & Museums
Quick Facts
Point of Sale
Admissions & Ticketing
Party, Group & Event Bookings
Online Sales & Party Reservations
Time Clock & Labor Management
& More!
Quick Facts
.Net since inception in 2004Customers use a local client/server SQL app, but these integrate with our cloud-based productsStarted using Couchbase Server 1.8 for online stores in 2012Cache SQL Server query results, persisted shopping cartsNeeded scalability and stabilityNew products operate using a pure Couchbase Server 4.5 database layer
Quick Facts
New Cloud Platform Data Flow
Data Data Data
IndexQuery
Web Servers
Users
Remote Application Servers
How to build your POCOs for consistenc
y
Nesting documents
Nesting arrays
Including reference primary
keys
.Net Data Modeling Basics
.Net Data Modeling Basics
public class Airline{ public string Callsign { get; set; } public string Country { get; set; } [JsonProperty("iata")] public string IATA { get; set; } [JsonProperty("iaco")] public string IACO { get; set; } public int Id { get; set; } public string Name { get; set; } public string Type { get; set; }}
Key: airline_10{ "callsign": "MILE-AIR", "country": "United States", "iata": "Q5", "icao": "MLA", "id": 10, "name": "40-Mile Air", "type": "airline"}
// JSON decorators not shown to simplify displaypublic class Airport{ public string AirportName { get; set; } public string City { get; set; } public string Country { get; set; } public string FAA { get; set; } public Coordinate Geo { get; set; } public string ICAO { get; set; } public int Id { get; set; } public string Timezone { get; set; } public string Type { get; set; }}
public class Coordinate{ public int Altitude { get; set; } public double Latitude { get; set; } public double Longitude { get; set; }}
Key: airport_1254{ "airportname": "Calais Dunkerque", "city": "Calais", "country": "France", "faa": "CQF", "geo": { "alt": 12, "lat": 50.962097, "lon": 1.954764 }, "icao": "LFAC", "id": 1254, "type": "airport", "tz": "Europe/Paris"}
.Net Data Modeling BasicsNesting Subdocuments as Properties
Note: Subdocuments don’t need “type” attributes, just the root
document
// JSON decorators not shown to simplify displaypublic class Route{ public string Airline { get; set; } public string AirlineId { get; set; } public string DestinationAirport { get; set; } public double Distance { get; set; } public string Equipment { get; set; } public int Id { get; set; } public List<Schedule> Schedule { get; set; } public string SourceAirport { get; set; } public int Stops { get; set; } public string Type { get; set; }}
public class Schedule{ public int Day { get; set; } public string Flight { get; set; } public TimeSpan UTC { get; set; }}
Key: route_10000{ "airline": "AF", "airlineid": "airline_137", "destinationairport": "MRS", "distance": 2881.617376098415, "equipment": "320", "id": 10000, "schedule": [ { "day": 0, "flight": "AF198", "utc": "10:13:00" }, { "day": 0, "flight": "AF547", "utc": "19:14:00" } ], "sourceairport": "TLV", "stops": 0, "type": "route"}
.Net Data Modeling BasicsNesting Arrays as Lists
Note: List<T> or similar list construct
[DocumentTypeFilter(TypeString)]public class Airline{ public const string TypeString = "airline";
public string Callsign { get; set; } public string Country { get; set; } [JsonProperty("iata")] public string IATA { get; set; } [JsonProperty("iaco")] public string IACO { get; set; } public int Id { get; set; } public string Name { get; set; }
// Type is now read only to maintain consistency public string Type => TypeString;}
Key: airline_10{ "callsign": "MILE-AIR", "country": "United States", "iata": "Q5", "icao": "MLA", "id": 10, "name": "40-Mile Air", "type": "airline"}
.Net Data Modeling BasicsGood Practices And LINQ Improvements
Note: DocumentTypeFilter automatically applies a WHERE type = ‘x’ predicate
Making Type read only ensures it’s always saved correctly
Key: route_10000{ "airline": "AF", "airlineid": "airline_137", "destinationairport": "MRS", "distance": 2881.617376098415, "equipment": "320", "id": 10000, "schedule": [ { "day": 0, "flight": "AF198", "utc": "10:13:00" }, { "day": 0, "flight": "AF547", "utc": "19:14:00" } ], "sourceairport": "TLV", "stops": 0, "type": "route"}
Key: airline_137{ "callsign": "AIRFRANS", "country": "France", "iata": "AF", "icao": "AFR", "id": 137, "name": "Air France", "type": "airline"}
.Net Data Modeling Basics
Note: To support joining, include the document key (or have a
way to build it)
Querying with LINQ
Basic and Advanced Queries
Joining Documents by Primary
Key
Unnesting Arrays
Querying with LINQ
public ActionResult BasicQuery(){ var db = new BucketContext(ClusterHelper.GetBucket("travel-sample"));
var query = from p in db.Query<Airline>() orderby p.Name select p;
return View(query);}
SELECT `Extent1`.* FROM `travel-sample` as `Extent1`WHERE (`Extent1`.`type` = 'airline')ORDER BY `Extent1`.`name` ASC
Note: DocumentTypeFilter (slide 12) automatically added the type predicate
Querying with LINQEasily add filters, sorts, projections and pagination to the query
public ActionResult AdvancedQuery(){ var db = new BucketContext(ClusterHelper.GetBucket("travel-sample"));
var query = (from p in db.Query<Airline>() where p.Callsign.StartsWith("A") orderby p.Name select new AirlineModel {Callsign = p.Callsign, Name = p.Name}).Take(10);
return View(query);}
SELECT `Extent1`.`callsign` as `callsign`, `Extent1`.`name` as `name`FROM `travel-sample` as `Extent1`WHERE (`Extent1`.`type` = 'airline') AND (`Extent1`.`callsign` LIKE 'A%')ORDER BY `Extent1`.`name` ASCLIMIT 10
Querying with LINQJoin to other documents using their primary key
public ActionResult Join(){ var db = new BucketContext(ClusterHelper.GetBucket("travel-sample"));
var query = from route in db.Query<Route>() join airline in db.Query<Airline>() on route.AirlineId equals N1QlFunctions.Key(airline) where route.SourceAirport == "ATL" && route.DestinationAirport == "ABE" orderby airline.Name, route.Stops select new RouteModel { AirlineName = airline.Name, Stops = route.Stops, Schedule = route.Schedule };
return View(query);}SELECT `Extent2`.`name` as `airlineName`, `Extent1`.`stops` as `stops`, `Extent1`.`schedule` as `schedule`FROM `travel-sample` as `Extent1`INNER JOIN `travel-sample` as `Extent2` ON KEYS `Extent1`.`airlineid`WHERE (`Extent1`.`type` = 'route') AND (`Extent2`.`type` = 'airline')AND ((`Extent1`.`sourceairport` = 'ATL') AND (`Extent1`.`destinationairport` = 'ABE'))ORDER BY `Extent2`.`name` ASC, `Extent1`.`stops` ASC
Querying with LINQUNNEST in N1QL
{ "airline": "AF", "airlineid": "airline_137", "destinationairport": "MRS", "distance": 2881.617376098415, "equipment": "320", "id": 10000, "schedule": [ { "day": 0, "flight": "AF198", "utc": "10:13:00" }, { "day": 0, "flight": "AF547", "utc": "19:14:00" } ], "sourceairport": "TLV", "stops": 0, "type": "route"}
[ { "day": 0, "destinationairport": "MRS", "flight": "AF198", "sourceairport": "TLV", "utc": "10:13:00" }, { "day": 0, "destinationairport": "MRS", "flight": "AF547", "sourceairport": "TLV", "utc": "19:14:00" }]
SELECT route.sourceairport, route.destinationairport, schedule.day, schedule.flight, schedule.utcFROM `travel-sample` AS routeUNNEST route.schedule AS scheduleWHERE route.type = 'route'
Querying with LINQUsing UNNEST with a second FROM clause
public ActionResult Unnest(){ var db = new BucketContext(ClusterHelper.GetBucket("travel-sample"));
var query = from route in db.Query<Route>() from schedule in route.Schedule where route.SourceAirport == "ATL" && route.DestinationAirport == "ABE" orderby schedule.Day, schedule.UTC select new UnnestedScheduleModel { Airline = route.Airline, Day = schedule.Day, UTC = schedule.UTC };
return View(query);}SELECT `Extent1`.`airline` as `airline`, `Extent2`.`day` as `day`, `Extent2`.`utc` as `utc`FROM `travel-sample` as `Extent1`INNER UNNEST `Extent1`.`schedule` as `Extent2`WHERE (`Extent1`.`type` = 'route')AND ((`Extent1`.`sourceairport` = 'ATL') AND (`Extent1`.`destinationairport` = 'ABE'))ORDER BY `Extent2`.`day` ASC, `Extent2`.`utc` ASC
Advanced LINQ Features for Couchbase
NULL != MISSING
UseKeys – Get documents by their primary
key
UseIndex – Provide index hints to the
query engine
Asynchronous querying
Advanced LINQ Features for CouchbaseHandling undefined JSON attributes – IsMissing, IsNotMissing,
IsValued, IsNotValued
public ActionResult IsMissing(){ var db = new BucketContext(ClusterHelper.GetBucket("travel-sample"));
var query = from p in db.Query<Airline>() where N1QlFunctions.IsMissing(p.IACO) select p;
return View(query);}
SELECT `Extent1`.*FROM `travel-sample` as `Extent1`WHERE (`Extent1`.`type` = 'airline') AND `Extent1`.`iaco` IS MISSING
Advanced LINQ Features for CouchbaseSelect documents directly using their primary key – UseKeys
public ActionResult UseKeys(){ var db = new BucketContext(ClusterHelper.GetBucket("travel-sample"));
var query = from p in db.Query<Airline>() .UseKeys(new[] {"airline_137", "airline_10765", "airline_1316" }) select p;
return View(query);}
SELECT `Extent1`.*FROM `travel-sample` as `Extent1`USE KEYS ['airline_137', 'airline_10765', 'airline_1316']WHERE (`Extent1`.`type` = 'airline')
Advanced LINQ Features for CouchbaseProvide Query Plan hints – UseIndex
public ActionResult UseIndex(){ var db = new BucketContext(ClusterHelper.GetBucket("travel-sample"));
var query = from p in db.Query<Route>().UseIndex("def_sourceairport") where p.SourceAirport == "ATL" orderby p.DestinationAirport select p;
return View(query);}
SELECT `Extent1`.*FROM `travel-sample` as `Extent1`USE INDEX (`def_sourceairport` USING GSI)WHERE (`Extent1`.`type` = 'route') AND (`Extent1`.`sourceairport` = 'ATL')ORDER BY `Extent1`.`destinationairport` ASC
Advanced LINQ Features for CouchbaseRun Queries Asynchronously – ExecuteAsync
public async Task<ActionResult> Async(){ var db = new BucketContext(ClusterHelper.GetBucket("travel-sample"));
var query = from p in db.Query<Airline>() orderby p.Name select p;
return View( await query.ExecuteAsync());}
Advanced LINQ Features for CouchbaseRun Aggregate Queries Asynchronously – ExecuteAsync
public async Task<ActionResult> AsyncAggregate(){ var db = new BucketContext(ClusterHelper.GetBucket("travel-sample"));
var query = from p in db.Query<Route>() select p;
return View( await query.ExecuteAsync( p => p.Average(q => q.Distance)));}
ExecuteAsync() can even run any immediate execution method (aggregates, First, Single, Explain) by passing the method as a
lambda expression
Efficient Indexing For LINQ Queries
Differences between SQL and
N1QL indexes
Indexing on “type”
attribute for speed
Indexing DateTime attributes
Efficient Indexing For LINQ Queries
AirlineSQL Table
AirportSQL Table
travel-sample Bucket
Airline Indexes
Airport Indexes
Bucket Indexes
Note: Remember that GSI indexes are similar to SQL indexes, but not the same
Efficient Indexing For LINQ Queries
/* Use predicate to only index documents of a certain type */CREATE INDEX `airport_sourceairport` ON `travel-sample` (`sourceairport`)WHERE `type` = 'airport'
/* Index the same attribute across multiple document types by including type first */CREATE INDEX `def_type_id` ON `travel-sample` (`type`, `id`)
/* A good practice is to create a fallback in case other indexes aren't used */CREATE INDEX `def_type` ON `travel-sample` (`type`)
Note: Be sure to index on the “type” attribute if you’re using [DocumentTypeFilter(“…”)]
Efficient Indexing For LINQ QueriesIndexing DateTime attributes – STR_TO_MILLIS
var cutoffDateTime = new DateTime(2010, 1, 1, 0, 0, 0, DateTimeKind.Utc);var query = db.Query<Beer>() .Where(e => e.Updated <= cutoffDateTime);
/* LINQ automatically wraps all DateTime constants and properties in STR_TO_MILLIS *//* STR_TO_MILLIS converts an ISO8601 string to a Unix numeric representation *//* It also handles the time zone specifier */SELECT `Extent1`.* FROM `beer-sample` as `Extent1`WHERE (`type` = 'beer')AND (STR_TO_MILLIS(`Extent1`.`updated`) <= STR_TO_MILLIS("2010-01-01T00:00:00Z"))/* So STR_TO_MILLIS must also be used in the index, or the index cannot be used */CREATE INDEX `beer_updated` ON `beer-sample` (STR_TO_MILLIS(`updated`))WHERE `type` = 'beer'
Change Tracking + Read Your Own Write
Document POCO
considerations
Updating, inserting, and
deleting documents
Using MutationState to read your own writes
Note: Linq2Couchbase change tracking is currently in developer preview
Change Tracking + Read Your Own Write[DocumentTypeFilter(TypeString)]public class Route{ public const string TypeString = "route";
// Read only property to return the calculated primary key // This is used when saving the document [Key] public string Key => TypeString + "_" + Id;
// Properties must be virtual for changes to be tracked public virtual string Airline { get; set; } // some properties not shown for clarity...
// Lists now use IList<T> instead of List<T> public virtual IList<Schedule> Schedule { get; set; }
// some properties not shown for clarity...
public string Type => TypeString;}
Update Your Document Models To Support Proxies
Change Tracking + Read Your Own WriteUnit of Work – BeginChangeTracking, SubmitChanges
_db.BeginChangeTracking();
// Query must return the document class, without a select projectionvar query = from p in _db.Query<Airline>() where p.Id == id select p;
// Query must execute after call to BeginChangeTrackingvar airline = query.FirstOrDefault();if (airline == null){ return HttpNotFound();}
airline.Name = model.Name;airline.Callsign = model.Callsign;
// Save the changes, if any_db.SubmitChanges();
Change Tracking + Read Your Own WriteDocument INSERT/DELETE – Save, Remove
_db.BeginChangeTracking();
// To delete a document when SubmitChanges is called_db.Remove(airline);
// To insert a document when SubmitChanges is called_db.Save(new Airline() { Id = 1, Name = model.Name, Callsign = model.Callsign});
// Save the changes_db.SubmitChanges();
Note: Save and Remove execute immediately if called before BeginChangeTracking
Change Tracking + Read Your Own WriteDo I have the latest changes? – MutationState, ConsistentWith
_db.SubmitChanges();
// MutationState represents the changes made by SubmitChanges// Pass into the query to ensure the changes are indexed before the query executes// This will slow the query, but much less than using RequestPlus consistency// Couchbase Server 4.5 is required for this featurevar query = from p in _db.Query<Airline>().ConsistentWith(_db.MutationState) orderby p.Name select p;
Why use Couchbase, N1QL and LINQ?
More scalable and performant than traditional SQL in the cloud
Easy transition for .Net developers trained on SQL and LINQ,and still produces unit testable code
More logical, nested data models rather than dozens of subsidiary tables
Schema-less JSON increases flexibility as your system evolves, leaving schema enforcement in your data access layer
Conclusion
What's next?
Source and Documentation:https://github.com/couchbaselabs/Linq2Couchbase
Example project:https://github.com/brantburnett/Couchbase.Linq.Example
Questions:https://forums.couchbase.com/c/net-sdk - @btburnett3Email - [email protected] - @btburnett3
Conclusion
Q & A
Thank YouVisit us at centeredgesoftware.com