Streaming XML with Jabber/XMPP

Peter Saint-Andre and Ralph Meijer

May 1, 2005

Keywords

Application architecture, Data inter-change, Distributed Systems, Enterpriseapplications, Finance, Fragment, Health-care, Internet, Interoperability, Middle-ware, SOAP, SVG, Unicode, Web Services,XML, XML-RPC, Atom, Messaging, Pres-ence, Publish-Subscribe, Presence, RSS,Streaming, Syndication

1 Introduction

The phrase ”streaming XML” may strike long-time XML and SGML afficionados as an oxy-moron. Isn’t XML all about stable (if extensible)containers for documents and data?

It was, until Jeremie Miller developed the ideaof XML streams in 1998 and built a working im-plementation in the jabberd instant messagingand presence server that he released in early1999. Since then, many more implementationshave been written, the core XML streamingprotocol has been formalized by the IETF underthe name Extensible Messaging and PresenceProtocol (XMPP), and Jabber/XMPP technolo-gies have been extended far beyond the realmof instant messaging to encompass everythingfrom network management systems to onlinegaming networks to financial trading applica-tions.

This paper provides a brief introduction to theconcepts of streaming XML and XML-basedcommunications as found in Jabber/XMPPtechnologies.

2 Streams and Stanzas

As defined in RFC 3920 [RFC3920] (the coreXMPP specification), an XML stream is a con-tainer for the exchange of XML [XML] elements

between any two entities over a network. Thosewho are accustomed to thinking of XML in adocument-centric manner may wish to viewan XML stream as an open-ended XML doc-ument that is built up over time by sendingXML elements over the stream, with the root<stream/> element serving as the documententity for the stream. However, this perspectiveis a convenience only; XMPP does not deal indocuments but in XML streams and XML stan-zas (the first-level child elements that are sentover those streams).

While peer-to-peer implementations of XMPPexist, most implementations follow a client-server model that is familiar from email. Whena client connects to a server, it opens a streamto the server and the server opens a stream tothe client (resulting in two streams, one in eachdirection). After negotiation of various streamparameters (including channel encryption andappropriate authentication), each entity is freeto send an unbounded number of XML stanzasover the stream. If a client addresses a stanzato a non-local entity, its server will negotiatea server-to-server stream with the foreign do-main, the send the stanza over that server-to-server stream for delivery to the non-local en-tity. As a result, a client can send stanzas to anynetwork-addressible entity.

XMPP defines three core stanza types, eachwith different semantics:

1. The <message/> stanza is a ”push”mechanism whereby one entity pushes in-formation to another entity, similar to thecommunications that occur in a systemsuch as email.

2. The <presence/> stanza is a basic broad-cast or ”publish-subscribe” mechanism,whereby multiple entities receive infor-mation about an entity to which theyhave subscribed (in particular, informationabout an entity’s network availability).

3. The <iq/> (Info/Query) stanza is arequest-response mechanism, similar insome ways to HTTP, that enables an entityto make a request of and receive a responsefrom another entity.

To date, these core stanza types have provensufficient for a wide variety of applications. Thekey is that the stanza types provide the deliv-ery semantics or ”transport layer” for near-real-time communications, whereas the content ofany given stanza is specified by its child el-ements, which may be qualified by any XMLnamespace. Thus the content of an XML stanzais not a MIME type or an attachment but pureXML, and XMPP can be used to exchange anydata that can be represented in XML, enablingdevelopment of a wide variety of applications.

3 Core Services

As can be guessed from the preceding descrip-tion of XML streams and XML stanzas, thecore functions of an XMPP server are to man-age streams and to route stanzas. Since mostXML streams are negotiated over long-livedTCP [RFC793] connections and XML stanzasare simply first-level child elements sent overthe stream, it could be said that a minimalXMPP implementation needs to do only twothings: (1) maintain TCP connections and (2) in-crementally parse XML. In practice, however,while it’s easy to write a basic XMPP imple-mentation, it is much harder to write a reallygood implementation. The reasons will becomeclear as we look at the services expected of anXMPP server or client.

Managing XML streams involves more thanjust maintaining TCP connections. For in-stance, RFC 3920 mandates that a server im-plementation must support Transport LayerSecurity (TLS) [RFC2246] for channel encryp-tion, Simple Authentication and Security Layer(SASL) [RFC2222] for authentication, DNS SRVrecords [RFC2782] for port lookups, variousprofiles of stringprep [RFC3454] for addressesthat can contain any Unicode character, UTF-8 [RFC3629] for fully internationalized streamcontent, and an XMPP-specific protocol forbinding a resource to the stream for networkaddressing purposes. Servers must also stampXML stanzas with a validated ”from” address

to prevent address spoofing (no spam frombill@microsoft.com over XMPP!), enforce var-ious stanza delivery rules, and comply withsome restrictions on XML usage in XMPP (suchas prohibitions on comments, processing in-structions, and DTD subsets).

XMPP-based instant messaging and presenceservers provide even more core services (as de-scribed in RFC 3921 [RFC3921]), including IMsession management, storage of contact lists,implementation of a rather complicated statemachine related to presence subscriptions (i.e.,control over access to a user’s availability infor-mation), and enforcement of user-defined blocklists and allow lists for communication withother entities on the network.

That may sound complex, and it is. Thankfully,a guiding principle of the Jabber philosophysince 1999 has been to keep most of the com-plexity at the server. This has made it much eas-ier to write clients, bots, components, and otherentities that connect to servers. One result hasbeen that there is a veritable plethora of Jab-ber clients. Another has been that libraries forwriting clients and client-like entities (such asbots) exist for just about every major program-ming language (and some minor ones, too).When it comes to experimenting with XMPPand building Jabber applications, developershave a great deal of options.

4 XMPP for Instant Messag-ing and Presence

The first and still dominant application ofXMPP is instant messaging and presence. TheIM and presence implementations, best knownunder the name ”Jabber”, are often called ”theLinux of instant messaging” because they pro-vide a fully open alternative to closed, propri-etary IM services such as ICQ, AIM, MSN Mes-senger, and Yahoo! Instant Messenger. Since thefirst Jabber code was released in January 1999,millions of end users have downloaded one ofthe many Jabber clients, hundreds of thousandsof system administrators have installed Jabberservers, and thousands of developers have con-tributed code or written custom Jabber exten-sions. The result is a thriving open networkwith tens of thousands of servers, and thou-sands more private deployments at companies,

2

universities, non-profit organizations, and gov-ernment agencies worldwide.

Because XMPP is an open wire protocol (re-cently standardized by the IETF) rather than asingle codebase, multiple implementations andlicensing schemes are encouraged. So far thereare half a dozen open-source server implemen-tations in C, Java, Python, and Erlang, as wellas closed-source implementations produced bysoftware vendors such as Jabber Inc., Antepo,Winfessor, and Sun Microsystems. There areopen-source, freeware, shareware, and com-mercial clients for common (Windows, MacOS,Linux) and not-so-common (Amiga) personalcomputing operating systems, handheld de-vices running PalmOS and WinCE, and cell-phone platforms like Symbian and J2ME. Thereare Jabber code libraries for C, C++, C#, COM,Delphi, Erlang, Flash, Java, JavaScript, Mono,Objective-C, Perl, PHP, Python, Ruby, Tcl, andmore. Most server implementations are quitemodular, so it is relatively easy to write server-side components for custom functionality.

The use of XML has helped the Jabber/XMPPcommunity develop a large number of ex-tensions to the core protocol defined in RFC3920. Extensions for basic IM and presence fea-tures such as contact lists are specified in RFC3921. The non-profit Jabber Software Foun-dation (JSF), which contributed XMPP to theIETF, continues to define more advanced XMPPextensions through its series of Jabber En-hancement Proposals (JEPs). Popular IM andpresence extensions include multi-user chat (ala IRC), file transfer, XHTML-formatted mes-sages, and ”extended presence” functionalitysuch as publishing one’s avatar or current song.

Some of these extensions may seem frivolous,and IM has the reputation for being a mediumthat’s appropriate for nothing more weightythan teen chat. Yet it would be a mistake todownplay the impact that presence-enabled,near-real-time messaging can have in an orga-nization. Here are some examples from existingXMPP deployments:

• Many of the world’s leading financial ser-vices firms consider their XMPP-basedmessaging systems to be more mission-critical than their phone systems.

• Jabber technologies provide a crucial pres-ence and communications infrastructure

for over 65 emergency management agen-cies in the Washington, D.C., area.

• XMPP is the basis for presence-driven ser-vices such as push-to-talk and push-to-video at several large mobile telephonyproviders.

• A large healthcare provider uses JabberIM systems to enable rapid communica-tion between patients, nurses, and medicalexperts.

As we can see, IM is not just for teens anymore.

5 XMPP Beyond IM

But XMPP is much more than IM. The seman-tics of the <message/>, <presence/>, and<iq/> stanzas provide a generalized commu-nication layer, making it possible to developand deploy a wide range of presence-enabledapplications. Consider a few examples:

• A partnership among some of the world’slargest foreign exchange trading banksis transitioning its $100-billion-a-day spotdealing system to an XMPP foundationfor improved extensibility and integrationwith back-end systems.

• A content syndication company provides afree service that enables near-real-time de-livery of information from millions of RSSand Atom feeds, using the XMPP publish-subscribe extension to solve the problem ofcontinuous polling for feed updates.

• Using standard XMPP protocols and sev-eral custom extensions, a network man-agement company ships software that en-ables a system administrator to monitorthe availability, uptime, and other vitalstatistics of routers and other deployed in-frastructure.

• A university research project uses XMPPand the Jabber-RPC protocol extension tosimulate catastrophic network failures onthe Internet.

• A telematics company offers an inte-grated hardware/software solution thatuses XMPP presence to track fleet vehicles

3

and thus reduce losses due to theft andvandalism.

• A commercial software company uses theSOAP XMPP binding both to provide aweb services interface to native XMPP ser-vices and to enable interaction betweenXMPP entities and HTTP-accessible webservices.

• An open-source project sponsored by amajor technical institute is building a dis-tributed whiteboarding system that passessnippets of SVG over XMPP in order tosynchronize multiple displays.

6 Key XMPP Extensions

Core XMPP functionality is specified in RFCs3920 and 3921, published by the Internet Engi-neering Task Force. But the Jabber/XMPP com-munity continues to innovate in the realm ofXMPP protocol extensions. Some of the key ex-tensions include the following documents pub-lished in the Jabber Software Foundation’s JEPseries:

• JEP-0030 [JEP0030]: Service Discovery – arobust protocol for determining the fea-tures supported by other entities on anXMPP network.

• JEP-0115 [JEP0115]: Entity Capabilities – areal-time profile of JEP-0030 [JEP0030] foradvertising capability changes via pres-ence.

• JEP-0004 [JEP0004]: Data Forms – a flexi-ble protocol for forms-handling via XMPP,mainly used in workflow applications andfor dynamic configuration.

• JEP-0045 [JEP0045]: Multi-User Chat – a setof protocols for participating in and ad-ministering multi-user chat rooms, similarto Internet Relay Chat but with strongersecurity.

• JEP-0096 [JEP0096]: File Transfer – aprotocol for transferring files from oneXMPP entity to another based on streaminitiation (JEP-0095 [JEP0095]) and sev-eral bytestreaming extensions (JEP-0047[JEP0047], JEP-0065 [JEP0065]).

• JEP-0071 [JEP0071]: XHTML-IM – aW3C-reviewed protocol for exchangingXHTML-formatted messages betweenXMPP entities.

• JEP-0124 [JEP0124]: HTTP Binding – abinding of XMPP to HTTP rather thanTCP, mainly used for devices that cannotmaintain persistent TCP connections to aserver.

• JEP-0060 [JEP0060]: Publish-Subscribe –a generalized framework for publish-subscribe functionality, mainly used to de-ploy content syndication, extended pres-ence, and event notification services.

The JEP series also defines XMPP extensionsfor a wide range of additional features, in-cluding XML-RPC and SOAP bindings (JEP-0009 [JEP0009] and JEP-0072 [JEP0072]), in-band registration (JEP-0077 [JEP0077]), ex-tended presence (JEP-0119 [JEP0119]), geoloca-tion (JEP-0080 [JEP0080]), and reliable messagedelivery (JEP-0079 [JEP0079]).

7 The Future of XMPP

While Jabber/XMPP technologies have beengrowing in usefulness and popularity sincethey were first released in January 1999, IETFapproval of the core protocols in October 2004has led to significant new implementations(Sun, Apple), major new deployments (U.S.Government), and renewed activity by open-source projects and commercial software devel-opers alike.

What does the future hold for XMPP? It is al-ways difficult to know how a particular tech-nology will be applied, but several trends seemclear:

• XMPP technologies (especially the pub-sub extension) will be used in more andmore applications – whenever presence in-formation, event notification, or real-timecontent delivery is needed.

• Increasing adoption will lead to conver-gence on a handful of implementationsfor particular platforms and fewer proto-col extensions over time as more deploy-ments come to depend on XMPP technolo-gies.

4

• Software tools vendors will make it mucheasier for developers to write XMPP frontends (e.g., in J2ME and Flash) and server-side services (e.g., in Python and Java).

• Responding to pressure from financial ser-vices firms and perhaps a major XMPP-based entrant into the market, the tradi-tional consumer IM services will beginto offer federated access to their systemsfrom XMPP deployments.

No one claims that XMPP technologies willachieve world domination next Thursday, orever. Different technologies are appropriate fordifferent requirements, and most technologieseventually become so stale that someone de-cides to design a replacement. But as life be-comes faster and faster, more developers, or-ganizations, and service providers decide todeploy XMPP technologies because they pro-vide near-real-time delivery of structured dataand presence information. And that’s reasonenough to think that streaming XML was notsuch a bad idea after all.

References

[JEP0009] Adams, DJ. JEP-0009: Jabber-RPC.Jabber Software Foundation, De-cember 2002 (Cited in section 6.)

[XML] Bray, Tim, Jean Paoli, C.M.Sperberg-McQueen, Eve Malerand Francois Yergeau. Extensi-ble Markup Language (XML) 1.0(Third Edition). World Wide WebConsortium, February 2004 (Citedin section 2.)

[RFC2246] Dierks, Tim and Christopher Allen.RFC 2246: The TLS Protocol, Ver-sion 1.0. Internet Engineering TaskForce, January 1999 (Cited in sec-tion 3.)

[JEP0004] Eatmon, Ryan, Joe Hildebrand,Jeremie Miller, Thomas Mul-downey and Peter Saint-Andre.JEP-0004: Data Forms. Jabber Soft-ware Foundation, November 2004(Cited in section 6.)

[JEP0072] Forno, Fabio and Peter Saint-Andre. JEP-0072: SOAP OverXMPP. Jabber Software Founda-tion, April 2005 (Cited in section 6.)

[RFC2782] Gulbrandsen, Arnt, Paul Vixie andLevon Esibov. RFC 2782: A DNSRR for specifying the location ofservices (DNS SRV). Internet Engi-neering Task Force, February 2000(Cited in section 3.)

[JEP0080] Hildebrand, Joe and Peter Saint-Andre. JEP-0080: User Geolocation.Jabber Software Foundation, Octo-ber 2004 (Cited in section 6.)

[JEP0115] Hildebrand, Joe and Peter Saint-Andre. JEP-0115: Entity Capabili-ties. Jabber Software Foundation,November 2004 (Cited in section 6.)

[JEP0030] Hildebrand, Joe, Peter Millard,Ryan Eatmon and Peter Saint-Andre. JEP-0030: Service Discov-ery. Jabber Software Foundation,April 2005 (Cited in section 6.)

[RFC3454] Hoffman, Paul and Marc Blanchet.RFC 3454: Preparation of Inter-nationalized Strings (”stringprep”).Internet Engineering Task Force,December 2002 (Cited in section 3.)

[JEP0047] Karneges, Justin. JEP-0047: In-BandBytestreams. Jabber Software Foun-dation, December 2003 (Cited insection 6.)

[JEP0060] Millard, Peter, Peter Saint-Andreand Ralph Meijer. JEP-0060:Publish-Subscribe. Jabber SoftwareFoundation, March 2005 (Cited insection 6.)

[JEP0079] Miller, Matthew and Peter Saint-Andre. JEP-0079: Advanced Mes-sage Processing. Jabber SoftwareFoundation, October 2004 (Cited insection 6.)

[JEP0095] Muldowney, Thomas, MatthewMiller and Ryan Eatmon. JEP-0095:Stream Initiation. Jabber SoftwareFoundation, April 2004 (Cited insection 6.)

5

[JEP0096] Muldowney, Thomas, MatthewMiller and Ryan Eatmon. JEP-0096:File Transfer. Jabber SoftwareFoundation, April 2004 (Cited insection 6.)

[RFC2222] Myers, John. RFC 2222: SimpleAuthentication and Security Layer(SASL). Internet Engineering TaskForce, October 1997 (Cited in sec-tion 3.)

[RFC793] Postel, Jon. RFC 793: TransmissionControl Protocol (TCP). InternetEngineering Task Force, September1981 (Cited in section 3.)

[JEP0077] Saint-Andre, Peter. JEP-0077: In-Band Registration. Jabber SoftwareFoundation, August 2004 (Cited insection 6.)

[JEP0071] Saint-Andre, Peter. JEP-0071:XHTML-IM. Jabber Software Foun-dation, September 2004 (Cited insection 6.)

[RFC3920] Saint-Andre, Peter. RFC 3920: Ex-tensible Messaging and PresenceProtocol (XMPP): Core. Internet En-gineering Task Force, October 2004(Cited in section 2.)

[RFC3921] Saint-Andre, Peter. RFC 3921: Ex-tensible Messaging and PresenceProtocol (XMPP): Instant Messag-ing and Presence. Internet Engi-neering Task Force, October 2004(Cited in section 3.)

[JEP0119] Saint-Andre, Peter. JEP-0119: Ex-tended Presence Protocol Suite. Jab-ber Software Foundation, March2005 (Cited in section 6.)

[JEP0045] Saint-Andre, Peter. JEP-0045: Multi-User Chat. Jabber Software Foun-dation, April 2005 (Cited in sec-tion 6.)

[JEP0065] Smith, Dave, Matthew Millerand Peter Saint-Andre. JEP-0065:SOCKS5 Bytestreams. Jabber Soft-ware Foundation, November 2004(Cited in section 6.)

[JEP0124] Smith, Dave, Peter Saint-Andre andIan Paterson. JEP-0124: HTTP Bind-ing. Jabber Software Foundation,March 2005 (Cited in section 6.)

[RFC3629] Yergeau, Francois. RFC 3629: UTF-8, a transformation format of ISO10646. Internet Engineering TaskForce, November 2003 (Cited insection 3.)

6