The DFDL cartridge opens up Smooks to a wide array of data formats (e.g., SWIFT, ISO8583, HL7). In fact, this cartridge forms the foundation of the EDI and EDIFACT cartridges. The DFDL cartridge deserializes (i.e., parses) non-XML data and serializes (i.e., unparses) XML according to the structure described in a DFDL schema. DFDL (Data Format Description Language) is an open standard modeling language for describing general text and binary data. Take the subsequent DFDL schema as an example:
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:fn="http://www.w3.org/2005/xpath-functions"
xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/" xmlns:ex="http://example.com"
targetNamespace="http://example.com" elementFormDefault="unqualified">
<xs:include schemaLocation="org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd" />
<xs:annotation>
<xs:appinfo source="http://www.ogf.org/dfdl/">
<dfdl:format ref="ex:GeneralFormat" separator="" initiator=""
terminator="" textTrimKind="none" initiatedContent="no" ignoreCase="no"
separatorPosition="infix" occursCountKind="implicit"
emptyValueDelimiterPolicy="both" representation="text" textNumberRep="standard"
lengthKind="delimited" encoding="ASCII" encodingErrorPolicy="error" />
</xs:appinfo>
</xs:annotation>
<xs:element name="file">
<xs:complexType>
<xs:sequence dfdl:separator="%NL;" dfdl:separatorPosition="postfix">
<xs:element name="header" minOccurs="0" maxOccurs="1"
dfdl:occursCountKind="implicit">
<xs:complexType>
<xs:sequence dfdl:separator=",">
<xs:element name="title" type="xs:string" maxOccurs="unbounded" />
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:element name="record" maxOccurs="unbounded">
<xs:complexType>
<xs:sequence dfdl:separator=",">
<xs:element name="item" type="xs:string" maxOccurs="unbounded"/>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:schema>This schema describes the structure of CSV data like the one below:
last,first,middle,DOB
smith,robert,brandon,1988-03-24
johnson,john,henry,1986-01-23
jones,arya,cat,1986-02-19A Smooks config parsing the above CSV using the DFDL cartridge would be written as:
<?xml version="1.0"?>
<smooks-resource-list xmlns="https://www.smooks.org/xsd/smooks-2.0.xsd"
xmlns:dfdl="https://www.smooks.org/xsd/smooks/dfdl-1.0.xsd">
<dfdl:parser schemaUri="/csv.dfdl.xsd"/>
...
</smooks-resource-list>dfdl:parser is a reader and its schemaUri attribute references the DFDL schema driving the parsing behaviour. Assuming input.csv is the source, dfdl:parser will generate the event stream:
<ex:file xmlns:ex="http://example.com">
<header>
<title>last</title>
<title>first</title>
<title>middle</title>
<title>DOB</title>
</header>
<record>
<item>smith</item>
<item>robert</item>
<item>brandon</item>
<item>1988-03-24</item>
</record>
<record>
<item>johnson</item>
<item>john</item>
<item>henry</item>
<item>1986-01-23</item>
</record>
<record>
<item>jones</item>
<item>arya</item>
<item>cat</item>
<item>1986-02-19</item>
</record>
</ex:file>Shown in the next snippet is a pipeline enclosing the dfdl:unparser visitor:
<?xml version="1.0"?>
<smooks-resource-list xmlns="https://www.smooks.org/xsd/smooks-2.0.xsd"
xmlns:core="https://www.smooks.org/xsd/smooks/smooks-core-1.6.xsd"
xmlns:dfdl="https://www.smooks.org/xsd/smooks/dfdl-1.0.xsd">
...
<core:smooks filterSourceOn="#document">
<core:action>
<core:inline>
<core:replace/>
</core:inline>
</core:action>
<core:config>
<smooks-resource-list>
<dfdl:unparser schemaUri="/csv.dfdl.xsd" unparseOnNode="*"/>
</smooks-resource-list>
</core:config>
</core:smooks>
</smooks-resource-list>In contrast to the dfdl:parser schemaUri attribute, the schemaUri schema in dfdl:unparser drives the unparsing behaviour. dfdl:unparser replaces each node in the event stream with its serialized CSV counterpart, essentially implementing a pass-through application.
The DFDL cartridge supports variables, on disk caching, and trace debugging. Consult the XSD documentation for further information.
Many DFDL schemas are freely available from the DFDL Schemas for Commercial and Scientific Data Formats GitHub repository. However, should you decide to author your own schemas, we strongly urge you first to gain a good understanding of DFDL. Resources to get started with DFDL include:
The next sections address common pitfalls to avoid when authoring DFDL schemas.
DFDL v1.0 supports data validation in the form of XSD constraints. Additional validation can be accomplished with the dfdl:assert statement as shown in the DFDL schema snippet below where the failureType attribute is equal to recoverableError so as not to interrupt parsing:
<xs:complexType name="FooType">
<xs:sequence>
<xs:element name="a" type="idl:int32">
<!-- Validate field a; recoverable error if fails -->
<xs:annotation>
<xs:appinfo source="http://www.ogf.org/dfdl/">
<dfdl:assert test="{ . eq 1 }" failureType="recoverableError"/>
</xs:appinfo>
</xs:annotation>
</xs:element>
<xs:element name="b" type="idl:int32"/>
<xs:element name="c" type="idl:int32"/>
</xs:sequence>
</xs:complexType>Apart from XSD constraints and dfdl:assert statements, which are part of the DFDL specification, the DFDL implementation of this cartridge can fire Schematron rules. Despite these validation capabilities, we generally recommend that rich validation of the source is accomplished further downstream, either in Smooks itself or in a different application (e.g., Drools) altogether for the following reasons:
-
The stricter a DFDL schema is, the less portable it becomes across applications that have different definitions of data validity.
-
The validation rules need to be re-implemented if the
dfdl:parseris swapped out with a non-DFDL Smooks reader. -
Pluggable validators such as the Schematron validator load the whole DFDL infoset into memory which means that the Smooks application will not benefit from streaming.
-
Accidental complexity can creep in when business rules are applied to the DFDL infoset. Since a DFDL infoset emphasises the physical format of the data, these rules may become harder to understand compared to when they are applied to a simpler, logical structure. This is of particular relevance when the rules need to be written or tweaked by non-technical users, say, business analysts.
The transformation features of DFDL should not be conflated with mapping. We highly recommend reading section 1.3 (What DFDL is not) of the DFDL specification which expands on this point. The XML schema structure must correspond more or less to the physical data format it is describing. While it is certainly possible to hide non-meaningful data in DFDL using hidden group elements and so on, if the data needs to be viewed in a very distinct way, then the general recommendation is to perform the mapping after parsing. One should consider:
-
Parsing the source first with DFDL,
-
Mapping the streaming infoset (e.g., with XSLT, JavaBean cartridge, FreeMarker, etc…), to then
-
Feed the mapped result to the target consumer.
A possible solution for mapping the DFDL infoset is to leverage the core:rewrite reader in combination with a FreeMarker visitor, within a pipeline, as demonstrated in the pipelines example.
Indent the generated event stream to make it easier to read. Useful for troubleshooting. The default value is false. Usage example:
<smooks-resource-list xmlns="https://www.smooks.org/xsd/smooks-2.0.xsd"
xmlns:dfdl="https://www.smooks.org/xsd/smooks/dfdl-1.0.xsd">
<dfdl:parser schemaUri="/csv.dfdl.xsd" indent="true"/>
</smooks-resource-list>Persist DFDL schema on disk to reduce compilation time in subsequent runs. The default value is false. Usage example:
<smooks-resource-list xmlns="https://www.smooks.org/xsd/smooks-2.0.xsd"
xmlns:dfdl="https://www.smooks.org/xsd/smooks/dfdl-1.0.xsd">
<dfdl:parser schemaUri="/csv.dfdl.xsd" cacheOnDisk="true"/>
</smooks-resource-list>Validation modes for validating the resulting infoset against the DFDL schema. The following values are supported:
| Value | Description |
|---|---|
Off |
Turn off all validation against the DFDL schema. |
Limited |
Perform XSD validation of facets, minLength, maxLength, enumeration, minInclusive, minExclusive, maxInclusive, maxExclusive, and maxOccurs constraints. Validation failures will be printed in the log but will not interrupt parsing or unparsing. Validation failures can be retrieved from the Smooks execution context during or after execution using the |
Full |
Perform full schema validation using Xerces. A validation failure will abort parsing and throw a |
The default value for the validation mode is Off. Usage example:
<smooks-resource-list xmlns="https://www.smooks.org/xsd/smooks-2.0.xsd"
xmlns:dfdl="https://www.smooks.org/xsd/smooks/dfdl-1.0.xsd">
<dfdl:parser schemaUri="/csv.dfdl.xsd" validationMode="Limited"/>
</smooks-resource-list>Validation failures can be retrieved from the Smooks execution context as shown below:
...
org.smooks.Smooks smooks = new org.smooks.Smooks();
org.smooks.api.ExecutionContext executionContext = smooks.createExecutionContext();
smooks.filterSource(executionContext, source, sink);
List<org.apache.daffodil.japi.Diagnostic> diagnostics = executionContext.get(org.smooks.cartridges.dfdl.parser.DfdlParser.DIAGNOSTICS_TYPED_KEY);
...Enable/disable trace debugging. The default value is false. Usage example:
<smooks-resource-list xmlns="https://www.smooks.org/xsd/smooks-2.0.xsd"
xmlns:dfdl="https://www.smooks.org/xsd/smooks/dfdl-1.0.xsd">
<dfdl:parser schemaUri="/csv.dfdl.xsd" debugging="true"/>
</smooks-resource-list>Apply standalone or embedded Schematron rules within the DFDL schema. Note that Schematron validation leads to the input stream being loaded into memory therefore such validation is not recommended for large streams of data.
Standalone Schematron rules are applied like this:
<smooks-resource-list xmlns="https://www.smooks.org/xsd/smooks-2.0.xsd"
xmlns:dfdl="https://www.smooks.org/xsd/smooks/dfdl-1.0.xsd">
<dfdl:parser schemaUri="/csv.dfdl.xsd">
<dfdl:schematron url="rules.sch"/>
</dfdl:parser>
</smooks-resource-list>Embedded rules are applied as follows:
<smooks-resource-list xmlns="https://www.smooks.org/xsd/smooks-2.0.xsd"
xmlns:dfdl="https://www.smooks.org/xsd/smooks/dfdl-1.0.xsd">
<dfdl:parser schemaUri="/csv.dfdl.xsd">
<dfdl:schematron/>
</dfdl:parser>
</smooks-resource-list>Behaves identically to the dfdl:parser cache on disk attribute. Usage example:
<smooks-resource-list xmlns="https://www.smooks.org/xsd/smooks-2.0.xsd"
xmlns:dfdl="https://www.smooks.org/xsd/smooks/dfdl-1.0.xsd">
<dfdl:unparser schemaUri="/csv.dfdl.xsd" unparseOnNode="*" cacheOnDisk="true"/>
</smooks-resource-list>Behaves identically to the dfdl:parser validation attribute. Usage example:
<smooks-resource-list xmlns="https://www.smooks.org/xsd/smooks-2.0.xsd"
xmlns:dfdl="https://www.smooks.org/xsd/smooks/dfdl-1.0.xsd">
<dfdl:unparser schemaUri="/csv.dfdl.xsd" unparseOnNode="*" validationMode="Limited"/>
</smooks-resource-list>Behaves identically to the dfdl:parser debugging attribute. Usage example:
<smooks-resource-list xmlns="https://www.smooks.org/xsd/smooks-2.0.xsd"
xmlns:dfdl="https://www.smooks.org/xsd/smooks/dfdl-1.0.xsd">
<dfdl:unparser schemaUri="/csv.dfdl.xsd" unparseOnNode="*" debugging="true"/>
</smooks-resource-list><dependency>
<groupId>org.smooks.cartridges</groupId>
<artifactId>smooks-dfdl-cartridge</artifactId>
<version>1.0.2</version>
</dependency>Smooks DFDL Cartridge is open source and licensed under the terms of the Apache License Version 2.0, or the GNU Lesser General Public License version 3.0 or later. You may use Smooks DFDL Cartridge according to either of these licenses as is most appropriate for your project.
SPDX-License-Identifier: Apache-2.0 OR LGPL-3.0-or-later
![@dependabot[bot]](https://avatars.githubusercontent.com/in/29110?s=64&v=4){kind=small}
![@dependabot-preview[bot]](https://avatars.githubusercontent.com/in/2141?s=64&v=4){kind=small}
![@github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=64&v=4){kind=small}