Smart way to schedule tasks – Polling -Mule ESB

Polling as the name suggests is a engineering pattern where you write or create a task once and then schedule it to run at some specific frequency, this frequency can be anywhere from milliseconds to days.

Polling is a integral part of enterprise integration, Since Mule ESB is a integration platform it supports Polling and it does so in various ways. Today I will be showing how to poll any of your resources using polling component of mule ESB.

But, before we go ahead below are some use cases where polling can be of use.

• You want to synchronize data between two systems, by polling one and sending it to other.
• Want to create reports on some data periodically
• Eagerly fetch data from your cloud applications like CRM, ERP etc for further processing
• Anything you can poll or schedule.

Today I will be using Salesforce as a poll example and creating a polling batch where i will be continually polling Accounts objects of salesforce and just printing them to logs.

So what happens in this case is as soon as a new account gets Created in salesforce we will get it in our flow and then log it in logger. We can very easily replace this logger with some other system like DB or some kind of BI process where this data could be of use.

I am not going to cover how setup salesforce connector for that please refer to my other blog https://echarish.wordpress.com/2016/10/06/salesforce-to-mule-esb-in-7-simple-steps/ or watch the youtube tutorial for salesforce introduction at my channel https://youtu.be/SrN9OH_-P6c

Flow

As is clear from the image above our flow is very simple we have put a poll component and in the scope of poll is the salesforce connector. So whats going to happen is our polling component will fire repeatedly based the frequency we set and will fire the salesforce connector.

In this polling component we will be setting a watermark which will be persisted by the polling component in our object store and will be updated every time the salesforce connector returns some result.

Polling Component

Above I have set the frequency to be fired in every 30 seconds. In the Watermark section, I am setting the name of watermark, this name will be available to us as flow variable and can be used in every run of flow and will be updated by polling component with the value of CreatedDate at the end of each execution of flow.

default.last.creation.date = 2005-10-08T01:02:03Z

Once we reach the last row in salesforce and their are no more results than the process part of flow will not be executed until unless a new data is created in salesforce and we will get the newly created data in the next run of flow i.e. after 30 sec.

Using Watermark

Here I am using the water mark set in polling component to query the salesforce to get the latest values from the salesforce by using flowVars.lastCreationDate.

Testing

As you can see in the first run we get 19 records and then in second run I got next 6 and in the end once all records are consumed we start getting empty data and no more records and a message is displayed

 Watermark value will not be updated since poll processor returned no results

One more important thing to remember is you can only use watermark if the processing strategy of your flow is synchronous

Video Tutorial

Complete flow XML

<flow name="sfdc-batch-accountFlow" processingStrategy="synchronous">
		<poll doc:name="Poll">
			<fixed-frequency-scheduler frequency="15" 				timeUnit="SECONDS" />
			<watermark variable="lastCreationDate" default-expression="${default.last.creation.date}" 				selector="LAST" selector-expression="#[payload.CreatedDate]" />
			<sfdc:query config-ref="Salesforce__Basic_Authentication" 				query="dsql:SELECT AccountNumber,BillingCity,BillingPostalCode,CreatedById,CreatedDate,LastModifiedById,LastModifiedDate,Name  FROM Account where CreatedDate&gt;#[flowVars.lastCreationDate]" 				fetchSize="10" doc:name="Salesforce" />
		</poll>
		<logger 			message="Payload size is #[payload.size()] and watermark is #[flowVars.lastCreationDate]" 			level="INFO" doc:name="Logger" />
		<json:object-to-json-transformer 			doc:name="Object to JSON" />
		<logger message="#[payload]" level="INFO" doc:name="Logger" />
	</flow>

That is all, If you liked it do let me know.

Follow my blog and youtube channel to keep me encouraged.

Thank you.