DSL using Groovy

Last night, I was playing with dynamic language groovy to write my very own DSL (domain specific language). In this blog I will explain how to write DSL using groovy.

This blog assumes the programming knowledge of groovy and underlying concept where & why DSL should be used.

Lets starts with my aim. My aim is to write dsl like “take 1 pill of disprin after 10 hours”. 

More specifically,

take 1.pill,

    of: disprin,

    after: 10.hours

Write the above script in IDE of your choice. Try to execute the above script.  After execution, following error will result

Caught: groovy.lang.MissingPropertyException: No such property: disprin ..

Quiet logical. Groovy cannot understand “disprin”. Let's add dynamically the property – disprin

this.metaClass.getProperty= {name->

  def metaProperty= this.metaClass.getMetaProperty(name)

    //'getMetaProperty' gets property, which may be an added or an existing one

  metaProperty? metaProperty.getProperty(delegate): "" + delegate

}

Let’s run the script again. After running the script, following exception came

Caught: groovy.lang.MissingPropertyException: No such property: hours for class: java.lang.Integer

The above error tells that, groovy cannot find the property hours in the class Integer. Groovy can magically add the properties to any java class and any jdk class as shown below

Number.metaClass.getHours = { -> new Integer(delegate          )

            println ("getHours delegate >> " + delegate)

}

After running the script again, the following error came

Caught: groovy.lang.MissingPropertyException: No such property: pill for class: java.lang.Integer

Lets add the property – pill to the Integer

Number.metaClass.getPill = { -> new Integer(delegate          )

            println ("getPill delegate >> " + delegate)

}

Let run the exception again; now the following exception came

Caught: groovy.lang.MissingMethodException: take()

Finally, lets add the missing method – take. The missing method can added dynamically at runtime as shown below

metaClass.invokeMethod= {name, args->

            def metaMethod= this.metaClass.getMetaMethod(name, args)

            //'getMetaMethod' gets method, which may be an added or an existing one

            // Call out to missing method can be handled here

            metaMethod? metaMethod.invoke(delegate,args): name

}

Hence the complete script looks like

this.metaClass.getProperty= {name->

  def metaProperty= this.metaClass.getMetaProperty(name)

    //'getMetaProperty' gets property, which may be an added or an existing one

  metaProperty? metaProperty.getProperty(delegate): "" + delegate

}

Number.metaClass.getHours = { -> new Integer(delegate          )

}

Number.metaClass.getPill = { -> new Integer(delegate          )

}

metaClass.invokeMethod= {name, args->

            def metaMethod= this.metaClass.getMetaMethod(name, args)

            //'getMetaMethod' gets method, which may be an added or an existing one

            // Call out to missing method can be handled here

            metaMethod? metaMethod.invoke(delegate,args): name

}

/*

 * Just to test

 * println (take (1.pill,of:disprin,after: 6.hours))

 */

take 1.pill,

     of:disprin,

     after:6.hours

Conclusion

            With groovy it’s very simple to create a DSL. Also the groovy specify code mentioned below

this.metaClass.getProperty= {name->

  def metaProperty= this.metaClass.getMetaProperty(name)

    //'getMetaProperty' gets property, which may be an added or an existing one

  metaProperty? metaProperty.getProperty(delegate): "" + delegate

}

Number.metaClass.getHours = { -> new Integer(delegate          )

}

Number.metaClass.getPill = { -> new Integer(delegate          )

}

metaClass.invokeMethod= {name, args->

            def metaMethod= this.metaClass.getMetaMethod(name, args)

            //'getMetaMethod' gets method, which may be an added or an existing one

            // Call out to missing method can be handled here

            metaMethod? metaMethod.invoke(delegate,args): name

}

can be abstracted out in the domain model and hence DSL will simply looks like

take 1.pill,

     of:disprin,

     after:6.hours

Architectural Patterns That Limit Application Scalability & Performance Optimization

Note - Currently working on this blog

Client & Server Pattern

Master & Worker Pattern

Partition For Parallelism 

Common Approach To Improve Throughput

Performance Optimization Concepts

Latency = time delay between starting an activity and when the results are  available /  detectable

Throughput = ratio of number of tasks completed in a unit of time.

Performance (perceived speed / responsiveness) = number of requests made and acknowledged in a unit of time.

Throughput and Performance are often confused!  (sometimes they are the same)

Example:

Average Throughput = 10 tasks / sec

Average Latency = 1000ms (1sec / 10)

Performance = unknown

NOTE - 

To improve performance = reduce latencies (between request and response)

To improve throughput = increase capacity (or reduce total latency)

Only Execute Mandatory Tasks

Increase CPU speed (scale up)

Optimize Algorithm

Exploit Parallelism (Scale Out)

Optimize Large Latencies

Best of all world - Do them all

Reduce Use Of XML

Use Case - Oracle Coherence In Compute Grid

Flexible Searching Using Compass

            Lucene is a very powerful search engine. Compass is written on top of lucene and provides lucene type search capabilities on top of domain model (jpa entity).In this blog I aim to look into use cases where compass search capabilities can be utilized.

           Compass provides ability to search goggle style on JPA entities. But the question is what are the use cases where somebody would like to have goggle style search on the JPA entities. For example a health insurance claim form contain information like claimant name (first name, last name), address etc. So how to provide search with search information like gaurav malhotra myaddress. The search information can possible comes from the following end points

 UI/Webservice

One can provide UI search screen based on the fixed parameter like

First Name = xxxx

Last Name = yyyyy

-- -- -- -- -- --

and its every easy to construct the SQL/JPQL/JPA2.0 search criteria. But if there is a limitation on the search information and now suppose we want to include the street name also in the search information, this will trigger the change in the UI search screen. Similar challenge exists in case of web services also.

One of nice things about compass is that it fits seamless with JPA (ORM products like EclipseLink). The compass specific search metadata can be directly specified on the JPA entity as shown in the JPA entity Claim.So a typical programming model using compass looks like 

The JPA entity Claim looks like
@Searchable

public class Claim {

    @SearchableId

    private int id;

    @SearchableProperty

    @SearchableMetaDatas({@SearchableMetaData(name = "firstname")})

    private String firstName;

     

    @SearchableProperty

    @SearchableMetaDatas({@SearchableMetaData(name = "lastname")})

    private String lastName;

     

    @SearchableProperty

    @SearchableMetaDatas({@SearchableMetaData(name = "streetname")})

    private String streetName;

     

      -- -- --- ---

  

}

Hence the search code looks like

> Person person = null;

   person = searchService.search(“firstname:gaurav lastname:malhotra streetname=xxx”);

   ** Search Person whose first name is gaurav , last name is malhotra and streetname is xxx

> Person person = null;

   person = searchService.search(“gaurav”);
   ** Search entity which has 'gaurav'

Groovy lovers can also use the search by injecting searchService into the groovy shell and calling search as

Person person = null;

Person = searchService.search(“gaurav malhotra”);  // the search framework automatically finds the Person entity.

Hence some of the import scenarios (EAI) where xml to domain model mapping is provided using groovy logic; adding compass search capabilities to it can be very powerful.

NOTE :- Compass search capability is very vast and support complex querying like inheritance support etc. Refer to Compass Search for more details

Conclusion
          Compass searching provides a very flexible and powerful way of searching capabilities, agnostic of underlying JPA entities, which is sometime referred to google style searching. In my subsequent article I will cover the scabaility of the compass by putting its indexes in the oracle coherence

Predictable Linear Scalability Design Pattern

Aim

            To use a messaging middleware queue to distribute the workload across the jvm is quiet common and helps in linear scalability. This blog explain the design pattern “Polling Consumer” which aids in workload distribution. This blog also explains how the messaging middleware itself become a bottleneck in linear scalability and how to overcome it. Quiet a famous saying I have heard in many of design meetings "Drop a message in a queue, it will be picked up by the consumer to perform the processing. Increase the consumers to increase the performance". Lets see how much truth in it :-)

           

Polling Consumer

In any messaging system (queue), the consumer needs an indication that application is ready so that it can consume the message. The best approach for the consumer is to repeatedly check the channel for message availability. If any message is available, the consumer consumes it (process it). This checking is a continuous process known as polling. The messaging system (queue) can be polled from the multiple jvms. All the consumers/jvms should equally share the workload and should not keep hold of messages on which they are not working.

Solution:

The application should use a polling consumer, one that explicitly makes a call when it wants to receive a message. 

Interactions:

A polling consumer is a message receiver. A polling consumer restricts the number of concurrent messages to be consumed by limiting the number of polling threads. In this way, it prevents the application from being blocked by having to process too many requests, and keeps any extra messages queued up until the receiver can process them.Each consumer/jvm will ask for message whenever it has capacity to perform hence provide perfect load balancing automatically.
Merits

1. Consumer asks for work (as a message) whenever it has capacity to perform, hence provide efficient workload management.
2. No message is lost as all the messages are persisted in the messaging system (queue).
3. Increasing number of consumers (jvms) can increase the scalability/performance to certain level.

Demerits
Predicable scalability/performance cannot be guaranteed, as increasing consumers (jvms) the messaging system can become the bottleneck as shown by the picture below

The messaging middleware (queue) becomes the bottleneck because, as the numbers of consumers are increased to achieve the linear scalability, the messaging middleware fails to handle so many consumers. The messaging middleware  will look like a overloaded horse cart like below

Poor horse chart or should I say messaging middle ware ;-)

The following picture shows the decreasing performance (i.e. no messages processed per seconds) as the numbers of jvms are increased. In below the queue used is a durable queue.

NOTE: - The following tests were performed on windows machine.

Hence with queue in db after the performance reached the optimum level (5 jvms as shown in above picture), adding more jvm results in negative performance.

Predictable Scalability

            One the key feature of Oracle Coherence is to provide “Predictable Scalability” i.e. by increasing a node (a jvm) in a cluster the performance of the system increases. The following picture shows the Queue of the “Polling Consumer” pattern being implemented using Oracle Coherences + AQ that is, AQ being made as the persistent store for the oracle coherence. As a result the consumer always enqueue/dequeue messages from the in-memory queue (hence AQ –DB bottleneck is removed). The enqueue will result a message in the coherence cache and also AQ and similarly dequeue of the message will result in message being removed from the coherence cache and AQ.

In the above design the cache store make sure message are always in sync with persistent storage (db). 

1. enqueue => oracle coherence cache put(message) => AQ enqueue

2. dequeue=> oracle coherence cache remove()   => AQ dequeue

  So the endpoint always perform operation in the cache

Conclusion

      A queue is an efficient way distributing work load. But the linear scalability cannot be always assured.ln functional areas like invoice processing,claim processing etc, using queue for workload distribution may not be always helpful in linear scalability. If the number of the JVMs are less then desired performance can be achieved by fine tunning the consumers.

In my next blog I will share the framework i.e design details to achieve the above. The "Polling Consumer" is an enterprise integration pattern (http://www.eaipatterns.com/PollingConsumer.html). In past, as a part of my job I have developed spring based framework which enable the implementation of enterprise integration design pattern. But spring source team have added a new framework called "Spring Integration", which enable ease of implementing enterprise integration pattern. How to create "Polling Consumer" using spring integration can be found at http://static.springsource.org/spring-integration/reference/htmlsingle/spring-integration-reference.html#endpoint-pollingconsumer

A sneak preview - in order to achieve Polling Consumer backed by queue in a cache (coherence) which also persist messages we need the following

 1) Off course, a queue/channel in oracle coherence backed by persistence storage. Lets call it - CoherenceAwareQueueChannel

2)  A Oracle Coherence cache store which persist the messages

3)  A simple POJO to which poller delivers the message - consumer. In the terminology of enterprise integration its called Service Activator (refer to http://www.eaipatterns.com/MessagingAdapter.html for detail)

Enqueue

Message message = MessageBuilder.withPayload("gaurav in cache").setHeader(JmsHeaders.TYPE, "sometype").build();

coherenceQueue.send(message);

Dequeue

Message msg = (Message) coherenceQueue.receive();

Polling Consumer

<beans:bean id="coherenceQueue" class="com.oracle.aq.cachestore.channel.CoherenceAwareQueueChannel" depends-on="cacheFactoryCreation"/>
<beans:bean id="messageConsumer" class="com.oracle.aq.cachestore.junit.AqCacheStoreTest$ServiceActivator"/>
<service-activator id="coherenceConsumer" input-channel="coherenceQueue"
 ref="messageConsumer" method="onMessage" auto-startup="false">
 <poller>
  <interval-trigger interval="10" />
 </poller>
</service-activator>

As a result of above a message will be passed to the POJO - ServiceActivator method onMessage. The poller will poll the CoherenceAwareQueue every 10 nano seconds. It's also possible to specify the number of polling threads.

The service activator pojo looks like
 public class ServiceActivator {
  
  public void onMessage(Message msg) {
                // Message will be delivered automatically
  }
 }

Simple programming model agnostic of coherence/spring details/jms details so that developer can use it with ease