Enterprise Integration! … when you work with integration of different systems then sooner or later you have to do the transformation of data. In today’s world very few system’s talk in same data language i.e. for example you may have a HTTP server which understand JSON but it need to talk to some server which only understand SOAP XML or even if it understand JSON than also the chances of having same JSON schema or format is very slim, so in these cases you need to do transformation of data.
Mule provides a very powerful tool to do these kind of transformation known as DataWeave .
DataWeave (DW)
DW is a new data transformation engine of mule. DW is the replacement of DataMapper you may have used DataMapper before it provided a graphical drag and drop approach to do data transformation, Please keep in mind from mule 3.7 mule runtime does’t have DataMapper and if you want to use it you need to separately provide the dependencies and from Mule version 4 there will be no longer support for DataMapper. So, I highly encourage everyone to either move to DW or Convert your DM to other form of transformation.
Canonical Data Format
Even before we embark on journey of understanding DW, we need to understand that DW is designed on canonical data format standards. In simple terms canonical data format is a way by which we represent any kind of data or object into standard format. Any expression we write in DW will result in a canonical format.
There are only 3 data types that can result from any DataWeave expression:
- Simple Types like Strings and numbers,
- Arrays
- Objects
Each of these can be expressed literally as below
| Simple Type Literal | “This is simple type” “This too” 2016 |
| Array Literal | [1,2,3,4] [“One”,”Two”,”Three”] |
| Object Literal | { “company” : “IT Ventures”, “employees”: [ {“firstName”:”John”, “lastName”:”Doe”}, {“firstName”:”Anna”, “lastName”:”Smith”}, {“firstName”:”Peter”, “lastName”:”Jones”} ] } |
DataSense – DataWeave
When you work with DW we get a powerful ability of datasense, by datasense you get the suggestions and view of data with which you are working. You should also be mindful of the mime-types of the Message data you have to transform. When this is not set explicitly at design time and when you use any of our Connectors, this will default to application/java.
For HTTP requests and responses Mule will look at the Content-Type header and set it accordingly. You should set the mime-type explicitly when you are using the set-variable and set-payload processors. Failing to set the mime-type explicitly for xml content will result in it being interpreted as a mere Java string by the engine.
Variable Reference Expression
Understanding DW is all about understanding how DW engine normalize your input data and how it is represented by the engine internally whenever references are made to variables, session variables, inbound properties, outbound properties etc. All these expressions convert the incoming data into the canonical DW format that we discussed above. So thinking in terms of DW simple types, arrays and objects will help to write better transformation.
Below is the sample of xml conversion to DW literals, If we change the output type to application/dw we can see the conversion done by engine of the input xml, Point to note here is how all the elements of xml is normalized as a key:value pair in the object.
In case of complex element types the normalized value is itself an object, repeating values are normalized as repeating key:value pair.
Anypoint Studio
Anypoint Studio supports DW development with a DW editor. In DW editor you will find 3 panes.
On the left DataSense displays the structure of the incoming message along with any example of data you have provided.
On the right pane, the expected outgoing structure is displayed as per DataSense. It will also display design time result of transformation which constantly refreshes itself as the expression of transformation is written.
The middle pane is where you write your transformation logic, It is divided into two sections. The HEADER is where the declaration of mime type of transformation output is done, along with some reusable functions and global variables as also namespaces for xml use cases. The mandatory part for header is declaration of DW version i.e. %dw 1.0 and the output mime-type.
below the dotted line is where you write the transformation expressions.Only one expression is needed for this, there are number of expression types, but majority of the transformations can be done with semi-literal expression of a DW object.
The same DW transformer can produce multiple outputs from the same incoming Message. This can be achieved by clicking the plus circle at the bottom right of the transformation pane, With this you will get another transformation pane in which you can select the output from the drop-down at the top of the transformation pane. Important thing to note is you can use different flow-vars or specify different targets for your transformation.
Output Rendering
The DW engine separates the actual transformation process within the canonical format from the final rendering of the same in the output mime type you defined in the header.
Irrespective of how complex your expression is, If it’s an object with deep nested structure and expressions and operators which combine expressions, all of these must be executed first and return their value before the outer expression returns its value. It is this final value which is rendered in the output mime-type.
Be aware of the choice of mime-type as they will have some limitations associated with them. Arrays will not be rendered in xml. Choose to repeat keys instead. Likewise, repeating keys cannot be rendered in Json. Generate arrays instead.
To render an object to XML, in line with the rule that XML documents may only contain one root element, the object may only contain one key:value pair. Its value can, as we have seen, itself be an object of any complexity.
Future Blogs
This was introduction to DW, I will be creating further video and blogs related to DataWeave to go more in details of various ways to use DW in your transformation.
Do let me know if you find this blog helpful and informative.