Integrating Bedework, a CalDAV Calendar Server, into OAE

Talk CalDAV To Me: Integrating Bedework into OAE

Chris Tweney <chris@media.berkeley.edu>Senior Software DeveloperEducational Technology ServicesUC BerkeleyJune 14, 2011

myBerkeley

Instance of Sakai OAE with customizations

Esp. those that allow advisors to communicate to students

myBerkeley widgets

myBerkeley widgets

myBerkeley Notifications

Notification Dependencies

Calendaring FundamentalsiCalendar specificationiCal4jCalDAV protocolWebDAV protocol

iCalendarSimple text-based format for

calendar data exchange

iCalendar's fundamental objects are:◦Events (VEVENT)◦To-dos (VTODO)◦Journal entries (VJOURNAL)◦Free-busy info (VFREEBUSY)

iCalendar

BEGIN:VCALENDAR

PRODID:-//Ben Fortuna//iCal4j 1.0//EN

VERSION:2.0

CALSCALE:GREGORIAN

BEGIN:VTODO

DTSTAMP:20110603T222847Z

DTSTART:20110505T151506

DUE:20110505T151506

SUMMARY:Pay your bill

UID:f84347ab-575b-4274-9436-a5ac906381f9

DESCRIPTION:Pay your bill by the deadline of May 5.

END:VTODO

END:VCALENDAR

Usually you encounter iCalendar data in the form of files with a “.ics” extension. Here’s a typical (abbreviated) example:

iCalendar is defined in RFC 5545http://tools.ietf.org/html/rfc5545

iCalendar Parts

ical4jOpen-source Java library to

wrangle messy iCalendar data

iCalendar records with time zone data get very complicated; ical4j makes it pretty simple to work with them

ical4j home page:http://wiki.modularity.net.au/ical4j/index.php?

title=Main_Page

ical4j Java Exampleprotected Calendar buildVTodo(String summary) {

CalendarBuilder builder = new CalendarBuilder();

Calendar calendar = new Calendar();

calendar.getProperties().add(new ProdId("-//Ben Fortuna//iCal4j 1.0//EN"));

calendar.getProperties().add(Version.VERSION_2_0);

calendar.getProperties().add(CalScale.GREGORIAN);

TimeZoneRegistry registry = builder.getRegistry();

VTimeZone tz = registry.getTimeZone("America/Los_Angeles").getVTimeZone();

calendar.getComponents().add(tz);

DateTime due = new DateTime(DateUtils.addDays(new Date(), new Random().nextInt(28)));

VToDo vtodo = new VToDo(due, due, summary);

vtodo.getProperties().add(new Uid(UUID.randomUUID().toString()));

vtodo.getProperties().add(CalDavConnector.MYBERKELEY_REQUIRED);

vtodo.getProperties().add(new Description("this is the description, it is long enough to

wrap at the ical specified standard 75th column"));

vtodo.getProperties().add(Status.VTODO_NEEDS_ACTION);

calendar.getComponents().add(vtodo);

return calendar;

}

ical4j pitfall: Line foldingPer the iCalendar RFC, long lines

of text get newlines inserted at the 75th column

Sadly, ical4j does not handle this by default

Need to specify “ical4j.unfolding.relaxed=true” in an ical4j.properties file

CalDAVDialect of WebDAV that provides

calendar functionality

Much syntax and structure is inherited from its underlying specifications: WebDAV and iCalendar

CalDAV is defined in RFC 4791:http://tools.ietf.org/html/rfc4791

Sakai 2 CalDAV

GA Tech had a CalDAV project in Sakai 2 that was never released due to gaps in Zimbra's API

Zach Thomas left excellent docs that hugely helped our efforts

Sakai 2 CalDAV Doc Page:https://confluence.sakaiproject.org/display/CALDAV/Developer's+Guide

CalDAV, WebDAV, HTTP

Learn by snooping

Speaking CalDAV with curl

curl –u user:pass -X PROPFIND http://localhost:8080/ucaldav/user/300846/calendar/

Doing a PROPFIND is a shortcut way to get a user's whole calendar:

To curl, "-X PROPFIND" means do an HTTP PROPFIND method.

HTTP has several funky methods you've never heard of unless you've worked with WebDAV.

Speaking CalDAV with curl

curl –u user:pass -X PROPFIND http://localhost:8080/ucaldav/user/300846/calendar/

HTTP/1.1 207 Multi-Status<?xml version="1.0" encoding="UTF-8" ?>

<DAV:multistatus xmlns:DAV="DAV:" xmlns="urn:ietf:params:xml:ns:caldav" xmlns:ical="http://www.w3.org/2002/12/cal/ical#"> <DAV:response> <DAV:href>/ucaldav/user/300846/calendar/00137ac8-1638-4193-ac3e-c172dfe3fd35.ics</DAV:href> <DAV:propstat> <DAV:prop> <DAV:getcontenttype>text/calendar; charset=UTF-8</DAV:getcontenttype> <DAV:getcontentlength>339</DAV:getcontentlength> <DAV:getlastmodified>Tue, 31 May 2011 21:23:32 +0000</DAV:getlastmodified> <DAV:displayname>00137ac8-1638-4193-ac3e-c172dfe3fd35.ics</DAV:displayname> <DAV:getetag>"20110531T212332Z-0"</DAV:getetag> <DAV:current-user-principal> <DAV:href>/ucaldav/principals/users/admin/</DAV:href> </DAV:current-user-principal> <DAV:resourcetype/> <DAV:getcontentlanguage>en</DAV:getcontentlanguage> <DAV:creationdate>20110531T212332Z</DAV:creationdate> </DAV:prop> <DAV:status>HTTP/1.1 200 ok</DAV:status> </DAV:propstat> </DAV:response> … more entries snipped …</DAV:multistatus>

CalDAV and WebDAV challenges

Not your everyday HTTP request

Weird methods (PROPFIND, OPTIONS, PUT, etc)

Multi-status responses (HTTP 207)

Fortunately…

Jackrabbit, on which OAE is based, ships with a reasonable WebDAV Java client

We can use this to make our basic CalDAV connections

CalDavConnector.getCalendarUris()

getCalendarUris() is our simplest method

Just return URI wrapper objects for a user's whole calendar

Complexity's a problem

CalDAV and WebDAV are◦Complicated◦Not noob-friendly◦RFCs are almost the only docs

available

Impatience is a virtue

Need lots of trial and errorRedeploying a server takes 60-

90sRunning a test takes 2s

◦2s is too little time to get distracted

Integration-test-driven development

JUnit to the rescue!

"Unit" tests that actually talk to a running Bedework server are the solution

Write the test first…Before doing anything else I write

a test:

public class CalDavConnectorTest() {

@Test() public void getCalendars() throws CalDavException { List<CalendarURI> uris = this.calDavConnector.getCalendarUris(); }

}

…Then make it passNow we'll make it pass in the

hackiest way imaginable

public class CalDavConnector() {

public void getCalendars() throws CalDavException { return new ArrayList<CalendarURI>(); }

}

…Iterate until done

PropFindMethod propFind = executeMethod(new PropFindMethod(this.userHome.toString())); MultiStatusResponse[] responses = propFind.getResponseBodyAsMultiStatus().getResponses(); for (MultiStatusResponse response : responses) { if (response.getHref().endsWith(".ics")) { Status[] status = response.getStatus(); if (status.length == 1 && status[0].getStatusCode() == HttpServletResponse.SC_OK) { DavPropertySet propSet = response.getProperties(HttpServletResponse.SC_OK); DavProperty etag = propSet.get(DavPropertyName.GETETAG); try { CalendarURI calUri = new CalendarURI( new URI(this.serverRoot, response.getHref(), false), etag.getValue().toString()); uris.add(calUri); } catch (ParseException pe) { throw new CalDavException("Invalid etag date", pe); } } } }

Keep adding implementation code until it does what's needed:

Unit testing heresy

These tests talk to and expect a running Bedework server

This makes them heretical according to true unit test dogma

Tests are not supposed to require an external server

Gracefully failing test

This test will succeed even if the Bedework server does not respond (because we catch the IOException):

public class CalDavConnectorTest() {

@Test() public void getCalendars() throws CalDavException { try { List<CalendarURI> uris = this.calDavConnector.getCalendarUris(); } catch (IOException ioe) { LOGGER.error("Trouble contacting server", ioe); } }

}

PROPFIND only gives you URIs…curl –u user:pass -X PROPFIND http://localhost:8080/ucaldav/user/300846/calendar/

HTTP/1.1 207 Multi-Status<?xml version="1.0" encoding="UTF-8" ?>

<DAV:multistatus xmlns:DAV="DAV:" xmlns="urn:ietf:params:xml:ns:caldav" xmlns:ical="http://www.w3.org/2002/12/cal/ical#"> <DAV:response> <DAV:href>/ucaldav/user/300846/calendar/00137ac8-1638-4193-ac3e-c172dfe3fd35.ics</DAV:href> <DAV:propstat> <DAV:prop> <DAV:getcontenttype>text/calendar; charset=UTF-8</DAV:getcontenttype> <DAV:getcontentlength>339</DAV:getcontentlength> <DAV:getlastmodified>Tue, 31 May 2011 21:23:32 +0000</DAV:getlastmodified> <DAV:displayname>00137ac8-1638-4193-ac3e-c172dfe3fd35.ics</DAV:displayname> <DAV:getetag>"20110531T212332Z-0"</DAV:getetag> <DAV:current-user-principal> <DAV:href>/ucaldav/principals/users/admin/</DAV:href> </DAV:current-user-principal> <DAV:resourcetype/> <DAV:getcontentlanguage>en</DAV:getcontentlanguage> <DAV:creationdate>20110531T212332Z</DAV:creationdate> </DAV:prop> <DAV:status>HTTP/1.1 200 ok</DAV:status> </DAV:propstat> </DAV:response> … more entries snipped …</DAV:multistatus>

…which you then must get with a REPORT method

curl –u user:pass -X REPORT http://localhost:8080/ucaldav/user/mtwain/calendar/ -d "<?xml version="1.0" encoding="UTF-8"?>

<C:calendar-multiget xmlns:C="urn:ietf:params:xml:ns:caldav" xmlns:D="DAV:">

<D:prop>

<D:getetag/>

<C:calendar-data/>

</D:prop>

<D:href>http://localhost:8080/ucaldav/user/mtwain/calendar/d12ad881-5769-4d17-85ac-fa0a0196ec04.ics

</D:href>

</C:calendar-multiget>"

(Note: Double quotes not escaped for clarity)

REPORT method's response<DAV:response>

<DAV:href>/ucaldav/user/mtwain/calendar/02bfa2cd-6c04-40f4-93da-896d54fc2987.ics</DAV:href>

<DAV:propstat>

<DAV:prop>

<DAV:getetag>"20110602T211046Z-0"</DAV:getetag>

<calendar-data><![CDATA[BEGIN:VCALENDAR

PRODID://Bedework.org//BedeWork V3.7//EN

VERSION:2.0

BEGIN:VTODO

CATEGORIES:MyBerkeley-Required

CATEGORIES:MyBerkeley-Archived

CREATED:20110601T171120Z

DESCRIPTION:test

… [ snip ]

]]></calendar-data>

</DAV:prop>

<DAV:status>HTTP/1.1 200 ok</DAV:status>

</DAV:propstat>

</DAV:response>

Hairy REPORT SyntaxCalDAV's REPORT syntax is hairy

XML

Have to extend Jackrabbit's ReportInfo classes

This blog entry by Ricardo Martin Camarero has very useful starter code:◦ http://blogs.nologin.es/rickyepoderi/index.php?/archives/15-

Introducing-CalDAV-Part-II.html

REPORT is foundation for search

REPORT methods with parameters let you search by date

Bedework difficulties

Bedework's search implementation spotty

No support for search on X-props

No support for search on CATEGORIES

Filtering on myBerkeley server

Due to Bedework bugs we're forced to do some filtering on the myBerkeley server side

E.g. Required/Not Required fields

Caching on myBerkeley server

CalendarURI wrapper class◦URI: Locates calendar◦Etag: Uniquely identifies its contents

(sort of like a SHA hash)

Caching not implemented yet, but easy enough since all calendars are keyed by CalendarURI

Implementing CalDavProxyServlet

Also done with test-driven development

Write tests against JSON files that contain servlet's expected inputs

When servlet's finished, UI devs use those JSON files as an API reference

Improvements to Nakamura's CalendarService

Future work: Store and search calendars using Nakamura's CalendarService

Add support for VTODO to CalendarService

Beyond CalendarService

Future: Create a full-blown CalDAV Calendar Provider component for Nakamura

Store calendars externally, transparently refer to them within Nakamura code