EDI to XMLUsing XML to take the headaches out of EDI
processing
Blair Schneider McKayBlair Schneider McKay
Clareos, Inc. Clareos, Inc. http://www.clareos.com
Blair Schneider McKay
Edi To Xml – Slide 2 Oct 21, 2004
Agenda
1.1. Project MotivationProject Motivation
2.2. EDI BasicsEDI Basics
3.3. Syntactic Conversion: EDI to XMLSyntactic Conversion: EDI to XML
4.4. Semantic Conversion: XML to XMLSemantic Conversion: XML to XML
5.5. Validating the Input DataValidating the Input Data
6.6. Storing and Using the ResultsStoring and Using the Results
7.7. System InstrumentationSystem Instrumentation
8.8. Q & A Q & A
Blair Schneider McKay
Edi To Xml – Slide 3 Oct 21, 2004
Project Motivation
AT&T Custom User SolutionsAT&T Custom User Solutions (ACUS) (ACUS) niche telecomm biller for colleges, military bases, niche telecomm biller for colleges, military bases,
state governments, other AT&T units, etc.state governments, other AT&T units, etc.
small development group. XP and agile development.small development group. XP and agile development.
at issue was one of our VAS offerings: reconcile client at issue was one of our VAS offerings: reconcile client facility bills and help them address discrepancies.facility bills and help them address discrepancies.
The Issue: The Issue: Reconciliation was done by manual comparison Reconciliation was done by manual comparison of electronic usage records to paper invoices. This was costly of electronic usage records to paper invoices. This was costly and slow, and it created an extra quality-control burden.and slow, and it created an extra quality-control burden.
Blair Schneider McKay
Edi To Xml – Slide 4 Oct 21, 2004
Agenda
1.1. Project MotivationProject Motivation
2.2. EDI BasicsEDI Basics
3.3. Syntactic Conversion: EDI to XMLSyntactic Conversion: EDI to XML
4.4. Semantic Conversion: XML to XMLSemantic Conversion: XML to XML
5.5. Validating the Input DataValidating the Input Data
6.6. Storing and Using the ResultsStoring and Using the Results
7.7. System InstrumentationSystem Instrumentation
8.8. Q & A Q & A
Blair Schneider McKay
Edi To Xml – Slide 5 Oct 21, 2004
Electronic Data Interchange early alternative to facsimile and postal mailearly alternative to facsimile and postal mail standards-basedstandards-based
- ANSI / ASC X12 / DISA- ANSI / ASC X12 / DISA- UN/EDIFACT- UN/EDIFACT- - http://www.disa.orghttp://www.disa.org
hardware, data, software, and network specshardware, data, software, and network specs ““VANs”, mailboxes, wire-transfer protocols, etc.VANs”, mailboxes, wire-transfer protocols, etc.
widespreadwidespreadDISA estimates over 300,000 companies use it.DISA estimates over 300,000 companies use it.
becoming more integrated with XMLbecoming more integrated with XMLExample: “Electronic Business XML” (ebXML)Example: “Electronic Business XML” (ebXML)
Blair Schneider McKay
Edi To Xml – Slide 6 Oct 21, 2004
Sample EDI DocumentISA~00~ ~00~ ~01~00001123
~ZZ~MICHIGANSTA ~040103~1950~U~00401~000001029~0~T~>GS~CI~006968523CRI~123456XYZ~20040407~1527~128~X~004010ST~811~000000001BIG~20040407~734555121299920040407~~~~~DIREF~EH~773REF~12~7345551212999REF~AP~7345551212999ITD~18~3~~~~20040421...snip...
SE~17921~10290002GE~2~128IEA~1~000001029
Modeled on samples in:Modeled on samples in:
http://www.sbc.com/Common/PDF/811v4010_CRIS-SW_CABS-SW+W_SBCT_IG.pdfhttp://www.sbc.com/Common/PDF/811v4010_CRIS-SW_CABS-SW+W_SBCT_IG.pdf
Blair Schneider McKay
Edi To Xml – Slide 7 Oct 21, 2004
EDI Technical Summary
data element typesdata element typesnumeric, decimal, identifier, string, date, timenumeric, decimal, identifier, string, date, time
delimitersdelimitersdata-element-separatordata-element-separator
first character after “ISA”, often “~”first character after “ISA”, often “~”
sub-element-separatorsub-element-separator
first character of 16first character of 16thth ISA data element ISA data element
segment-terminatorsegment-terminator
character after the sub-element-separatorcharacter after the sub-element-separator
hierarchieshierarchiessegments nest in “loops”segments nest in “loops”
ISA
GS
ST
Details
SE
GE
IEA
Blair Schneider McKay
Edi To Xml – Slide 8 Oct 21, 2004
Agenda
1.1. Project MotivationProject Motivation
2.2. EDI BasicsEDI Basics
3.3. Syntactic Conversion: EDI to XMLSyntactic Conversion: EDI to XML
4.4. Semantic Conversion: XML to XMLSemantic Conversion: XML to XML
5.5. Validating the Input DataValidating the Input Data
6.6. Storing and Using the ResultsStoring and Using the Results
7.7. System InstrumentationSystem Instrumentation
8.8. Q & A Q & A
Blair Schneider McKay
Edi To Xml – Slide 9 Oct 21, 2004
Why XML?
• The 811 document is already hierarchical.The 811 document is already hierarchical.
• It’s It’s easiereasier. Many companies find EDI too . Many companies find EDI too expensive, too complex, and too inflexible.expensive, too complex, and too inflexible.
• Once something is in XML format, you can Once something is in XML format, you can “go anywhere” with it.“go anywhere” with it.
By converting EDI to XML you trade complex, By converting EDI to XML you trade complex, “single-use” tools for generic XML tools like DTD, “single-use” tools for generic XML tools like DTD,
W3C Schema, XSLT, SOAP, DOM, SAX, etc. W3C Schema, XSLT, SOAP, DOM, SAX, etc.
Blair Schneider McKay
Edi To Xml – Slide 10 Oct 21, 2004
More Motivation…
USD SegmentIf USD08 is populated, then the amount in USD08 is the segment amount.If USD08 is not populated, then the following determination is made to determine thesegment amount:If USD05 = “M4”, then the amount in USD06 is the segment amount.If USD05 “M4”, then the product of USD03 and USD06 is the segment amount.If USD01 = “O”, then the segment amount is for information purposes only.The USD segment can be used to provide taper table information for WATS billing. Inthis use, USD01 would equal “O”.Note 1: When both USD03 and USD08 are populated, the amount in USD08 takesprecedence and becomes the segment amount.Note 2: In order to verify amount in USD08, it is necessary to multiply USD03 * USD06* USD07 * Percent of Interstate Usage. The Percent of Interstate Usage canbe found in an SI pair (DE 1000 = ‘PU’) within the SLN loop, when it is present.
From: http://www.sbc.com/Common/PDF/811v4010_CRIS-SW_CABS-SW+W_SBCT_IG.pdf, p6.2.8
Blair Schneider McKay
Edi To Xml – Slide 11 Oct 21, 2004
Sample EDI Document - RevisitedISA~00~ ~00~ ~01~00001123
~ZZ~MICHIGANSTA ~040103~1950~U~00401~000001029~0~T~>GS~CI~006968523CRI~123456XYZ~20040407~1527~128~X~004010ST~811~000000001BIG~20040407~734555121299920040407~~~~~DIREF~EH~773REF~12~7345551212999REF~AP~7345551212999ITD~18~3~~~~20040421...snip...
SE~17921~10290002GE~2~128IEA~1~000001029
Modeled on samples in:Modeled on samples in:
http://www.sbc.com/Common/PDF/811v4010_CRIS-SW_CABS-SW+W_SBCT_IG.pdfhttp://www.sbc.com/Common/PDF/811v4010_CRIS-SW_CABS-SW+W_SBCT_IG.pdf
Blair Schneider McKay
Edi To Xml – Slide 12 Oct 21, 2004
Raw XML<HL> <HL01>4</HL01> <HL02>3</HL02> <HL03>8</HL03> <HL04>1</HL04> <SLN> <SLN01>5.0</SLN01> <SLN02></SLN02> <SLN03>I</SLN03> <SLN04>M4</SLN04> <SLN05>11.85</SLN05> <SLN06></SLN06> <SLN07>I</SLN07> <SLN08>SV</SLN08> <SLN09>0705</SLN09>HL~4~3~8~1
SLN~5.0~~I~1~M4~11.85~~I~SV~0705
Edi2Xml.exe
Blair Schneider McKay
Edi To Xml – Slide 13 Oct 21, 2004
The Edi2Xml Utility
Sample init file:Sample init file:
Nesting ISA 1
Nesting IEA 1
Nesting GS 2
Nesting GE 2
Nesting ST 3
Nesting SE 3
Nesting_FieldComp HL 4 3
Single *
class Segment {class Segment {
public:public:
enum Relationship { child, sibling, uncle };enum Relationship { child, sibling, uncle };
virtual Relationship comp( const Segment& other) const;virtual Relationship comp( const Segment& other) const;
// ...// ...
};};
Blair Schneider McKay
Edi To Xml – Slide 14 Oct 21, 2004
Agenda
1.1. Project MotivationProject Motivation
2.2. EDI BasicsEDI Basics
3.3. Syntactic Conversion: EDI to XMLSyntactic Conversion: EDI to XML
4.4. Semantic Conversion: XML to XMLSemantic Conversion: XML to XML
5.5. Validating the Input DataValidating the Input Data
6.6. Storing and Using the ResultsStoring and Using the Results
7.7. System InstrumentationSystem Instrumentation
8.8. Q & A Q & A
Blair Schneider McKay
Edi To Xml – Slide 15 Oct 21, 2004
XSLT Examples
<xsl:template match="ST[ST01='811']">
<Invoice>
<xsl:apply-templates/>
</Invoice>
</xsl:template>
<xsl:template name="conv_access_type">
<xsl:param name="cd"/>
<xsl:choose>
<xsl:when test="$cd='81'">Local</xsl:when> <xsl:when test="$cd='82'">IntraState/InterLata</xsl:when>
<xsl:when test="$cd='83'">IntraState/IntraLata</xsl:when>
...snip...
Blair Schneider McKay
Edi To Xml – Slide 16 Oct 21, 2004
<HL> <HL01>4</HL01> <HL02>3</HL02> <HL03>8</HL03> <HL04>1</HL04> <SLN> <SLN01>5.0</SLN01> <SLN02></SLN02> <SLN03>I</SLN03> <SLN04>M4</SLN04> <SLN05>11.85</SLN05> <SLN06></SLN06> <SLN07>I</SLN07> <SLN08>SV</SLN08> <SLN09>0705</SLN09>
Translated XML – v.1
HL~4~3~8~1SLN~5.0~~I~1~M4~11.85~~I~SV~0705
Edi2Xml.exe
XSLT
<HL idx='4' parent='3' level='8' haschildren='yes'> <SLN id1='5.0' relcode='I' qty='1' type='money' included='yes' service-rendered='yes' svc_type='0705'> 11.85 </SLN> ...snip...
Blair Schneider McKay
Edi To Xml – Slide 17 Oct 21, 2004
Translated XML – v.2<HL> <HL01>4</HL01> <HL02>3</HL02> <HL03>8</HL03> <HL04>1</HL04> <SLN> <SLN01>5.0</SLN01> <SLN02></SLN02> <SLN03>I</SLN03> <SLN04>M4</SLN04> <SLN05>11.85</SLN05> <SLN06></SLN06> <SLN07>I</SLN07> <SLN08>SV</SLN08> <SLN09>0705</SLN09>HL~4~3~8~1
SLN~5.0~~I~1~M4~11.85~~I~SV~0705
Edi2Xml.exe
XSLT
<GroupLevelCharge> <Charge serviceType='Measured Service‘> $11.85 </Charge> ...snip...
Blair Schneider McKay
Edi To Xml – Slide 18 Oct 21, 2004
Agenda
1.1. Project MotivationProject Motivation
2.2. EDI BasicsEDI Basics
3.3. Syntactic Conversion: EDI to XMLSyntactic Conversion: EDI to XML
4.4. Semantic Conversion: XML to XMLSemantic Conversion: XML to XML
5.5. Validating the Input DataValidating the Input Data
6.6. Storing and Using the ResultsStoring and Using the Results
7.7. System InstrumentationSystem Instrumentation
8.8. Q & A Q & A
Blair Schneider McKay
Edi To Xml – Slide 19 Oct 21, 2004
Validating the Input Data
EDI transaction sets contain checksums.EDI transaction sets contain checksums.• Control codes in matched segments (ISA/IEA, ST/SE, Control codes in matched segments (ISA/IEA, ST/SE,
GS/GE) must line up.GS/GE) must line up.
• Some segment-types (like HL) have global sequence Some segment-types (like HL) have global sequence numbers, parent-references, nesting indicators:numbers, parent-references, nesting indicators:
HLHL~~4545~~3737~~55~~00
HLHL~(~(my idmy id)~()~(my parent’s idmy parent’s id)~()~(my levelmy level)~()~(I have I have childrenchildren))
• Optional transaction totals:Optional transaction totals:
CTTCTT~~33
CTTCTT~(~(# of preceding # of preceding IT1IT1, or line-item, segments, or line-item, segments))
Blair Schneider McKay
Edi To Xml – Slide 20 Oct 21, 2004
Validating the Input Data – With XML!
1. Hook into existing EDI validation rules using time-1. Hook into existing EDI validation rules using time-tested XML methods:tested XML methods:
• DTDDTD• W3C XML Schema LanguageW3C XML Schema Language• SchematronSchematron
2. Create sys-admin dashboards. 2. Create sys-admin dashboards. E.g.., express this in XSLT:E.g.., express this in XSLT:
display in bold-red typedisplay in bold-red type:: any loops whose instance-id is not ( any loops whose instance-id is not (previous-idprevious-id+1)+1)
Blair Schneider McKay
Edi To Xml – Slide 21 Oct 21, 2004
Agenda
1.1. Project MotivationProject Motivation
2.2. EDI BasicsEDI Basics
3.3. Syntactic Conversion: EDI to XMLSyntactic Conversion: EDI to XML
4.4. Semantic Conversion: XML to XMLSemantic Conversion: XML to XML
5.5. Validating the Input DataValidating the Input Data
6.6. Storing and Using the ResultsStoring and Using the Results
7.7. System InstrumentationSystem Instrumentation
8.8. Q & A Q & A
Blair Schneider McKay
Edi To Xml – Slide 22 Oct 21, 2004
Storing and Using the Results
EDI to XML makes great sense.EDI to XML makes great sense.
But EDI to XML to But EDI to XML to RDBMSRDBMS can be tricky. can be tricky.
If the goal is to produce a document (like a detailed If the goal is to produce a document (like a detailed invoice) just use XSLT to produce HTML, PDF, etc.invoice) just use XSLT to produce HTML, PDF, etc.
In many cases, you should just insert into the In many cases, you should just insert into the RDBMS only the values and labels you care about, RDBMS only the values and labels you care about, along with a unique id pointing to the (saved) XML.along with a unique id pointing to the (saved) XML.
Blair Schneider McKay
Edi To Xml – Slide 23 Oct 21, 2004
Agenda
1.1. Project MotivationProject Motivation
2.2. EDI BasicsEDI Basics
3.3. Syntactic Conversion: EDI to XMLSyntactic Conversion: EDI to XML
4.4. Semantic Conversion: XML to XMLSemantic Conversion: XML to XML
5.5. Validating the Input DataValidating the Input Data
6.6. Storing and Using the ResultsStoring and Using the Results
7.7. System InstrumentationSystem Instrumentation (Demo) (Demo)
8.8. Q & A Q & A
Q & A
Blair Schneider McKayBlair Schneider McKay
Clareos, Inc. Clareos, Inc. http://www.clareos.com
Source Code for the Edi2Xml tool, and bothSource Code for the Edi2Xml tool, and bothXSLT and document samples are available at: XSLT and document samples are available at: http://home.comcast.net/~blairschneidermckahttp://home.comcast.net/~blairschneidermckay/y/