DSL using Groovy

Last night, I was playing with dynamic language groovy to write my very own DSL (domain specific language). In this blog I will explain how to write DSL using groovy.

This blog assumes the programming knowledge of groovy and underlying concept where & why DSL should be used.

Lets starts with my aim. My aim is to write dsl like “take 1 pill of disprin after 10 hours”. 

More specifically,

take 1.pill,

    of: disprin,

    after: 10.hours

Write the above script in IDE of your choice. Try to execute the above script.  After execution, following error will result

Caught: groovy.lang.MissingPropertyException: No such property: disprin ..

Quiet logical. Groovy cannot understand “disprin”. Let's add dynamically the property – disprin

this.metaClass.getProperty= {name->

  def metaProperty= this.metaClass.getMetaProperty(name)

    //'getMetaProperty' gets property, which may be an added or an existing one

  metaProperty? metaProperty.getProperty(delegate): "" + delegate

}

Let’s run the script again. After running the script, following exception came

Caught: groovy.lang.MissingPropertyException: No such property: hours for class: java.lang.Integer

The above error tells that, groovy cannot find the property hours in the class Integer. Groovy can magically add the properties to any java class and any jdk class as shown below

Number.metaClass.getHours = { -> new Integer(delegate          )

            println ("getHours delegate >> " + delegate)

}

After running the script again, the following error came

Caught: groovy.lang.MissingPropertyException: No such property: pill for class: java.lang.Integer

Lets add the property – pill to the Integer

Number.metaClass.getPill = { -> new Integer(delegate          )

            println ("getPill delegate >> " + delegate)

}

Let run the exception again; now the following exception came

Caught: groovy.lang.MissingMethodException: take()

Finally, lets add the missing method – take. The missing method can added dynamically at runtime as shown below

metaClass.invokeMethod= {name, args->

            def metaMethod= this.metaClass.getMetaMethod(name, args)

            //'getMetaMethod' gets method, which may be an added or an existing one

            // Call out to missing method can be handled here

            metaMethod? metaMethod.invoke(delegate,args): name

}

Hence the complete script looks like

this.metaClass.getProperty= {name->

  def metaProperty= this.metaClass.getMetaProperty(name)

    //'getMetaProperty' gets property, which may be an added or an existing one

  metaProperty? metaProperty.getProperty(delegate): "" + delegate

}

Number.metaClass.getHours = { -> new Integer(delegate          )

}

Number.metaClass.getPill = { -> new Integer(delegate          )

}

metaClass.invokeMethod= {name, args->

            def metaMethod= this.metaClass.getMetaMethod(name, args)

            //'getMetaMethod' gets method, which may be an added or an existing one

            // Call out to missing method can be handled here

            metaMethod? metaMethod.invoke(delegate,args): name

}

/*

 * Just to test

 * println (take (1.pill,of:disprin,after: 6.hours))

 */

take 1.pill,

     of:disprin,

     after:6.hours

Conclusion

            With groovy it’s very simple to create a DSL. Also the groovy specify code mentioned below

this.metaClass.getProperty= {name->

  def metaProperty= this.metaClass.getMetaProperty(name)

    //'getMetaProperty' gets property, which may be an added or an existing one

  metaProperty? metaProperty.getProperty(delegate): "" + delegate

}

Number.metaClass.getHours = { -> new Integer(delegate          )

}

Number.metaClass.getPill = { -> new Integer(delegate          )

}

metaClass.invokeMethod= {name, args->

            def metaMethod= this.metaClass.getMetaMethod(name, args)

            //'getMetaMethod' gets method, which may be an added or an existing one

            // Call out to missing method can be handled here

            metaMethod? metaMethod.invoke(delegate,args): name

}

can be abstracted out in the domain model and hence DSL will simply looks like

take 1.pill,

     of:disprin,

     after:6.hours

Architectural Patterns That Limit Application Scalability & Performance Optimization

Note - Currently working on this blog

Client & Server Pattern

Master & Worker Pattern

Partition For Parallelism 

Common Approach To Improve Throughput

Performance Optimization Concepts

Latency = time delay between starting an activity and when the results are  available /  detectable

Throughput = ratio of number of tasks completed in a unit of time.

Performance (perceived speed / responsiveness) = number of requests made and acknowledged in a unit of time.

Throughput and Performance are often confused!  (sometimes they are the same)

Example:

Average Throughput = 10 tasks / sec

Average Latency = 1000ms (1sec / 10)

Performance = unknown

NOTE - 

To improve performance = reduce latencies (between request and response)

To improve throughput = increase capacity (or reduce total latency)

Only Execute Mandatory Tasks

Increase CPU speed (scale up)

Optimize Algorithm

Exploit Parallelism (Scale Out)

Optimize Large Latencies

Best of all world - Do them all

Reduce Use Of XML

Use Case - Oracle Coherence In Compute Grid

Flexible Searching Using Compass

            Lucene is a very powerful search engine. Compass is written on top of lucene and provides lucene type search capabilities on top of domain model (jpa entity).In this blog I aim to look into use cases where compass search capabilities can be utilized.

           Compass provides ability to search goggle style on JPA entities. But the question is what are the use cases where somebody would like to have goggle style search on the JPA entities. For example a health insurance claim form contain information like claimant name (first name, last name), address etc. So how to provide search with search information like gaurav malhotra myaddress. The search information can possible comes from the following end points

 UI/Webservice

One can provide UI search screen based on the fixed parameter like

First Name = xxxx

Last Name = yyyyy

-- -- -- -- -- --

and its every easy to construct the SQL/JPQL/JPA2.0 search criteria. But if there is a limitation on the search information and now suppose we want to include the street name also in the search information, this will trigger the change in the UI search screen. Similar challenge exists in case of web services also.

One of nice things about compass is that it fits seamless with JPA (ORM products like EclipseLink). The compass specific search metadata can be directly specified on the JPA entity as shown in the JPA entity Claim.So a typical programming model using compass looks like 

The JPA entity Claim looks like
@Searchable

public class Claim {

    @SearchableId

    private int id;

    @SearchableProperty

    @SearchableMetaDatas({@SearchableMetaData(name = "firstname")})

    private String firstName;

     

    @SearchableProperty

    @SearchableMetaDatas({@SearchableMetaData(name = "lastname")})

    private String lastName;

     

    @SearchableProperty

    @SearchableMetaDatas({@SearchableMetaData(name = "streetname")})

    private String streetName;

     

      -- -- --- ---

  

}

Hence the search code looks like

> Person person = null;

   person = searchService.search(“firstname:gaurav lastname:malhotra streetname=xxx”);

   ** Search Person whose first name is gaurav , last name is malhotra and streetname is xxx

> Person person = null;

   person = searchService.search(“gaurav”);
   ** Search entity which has 'gaurav'

Groovy lovers can also use the search by injecting searchService into the groovy shell and calling search as

Person person = null;

Person = searchService.search(“gaurav malhotra”);  // the search framework automatically finds the Person entity.

Hence some of the import scenarios (EAI) where xml to domain model mapping is provided using groovy logic; adding compass search capabilities to it can be very powerful.

NOTE :- Compass search capability is very vast and support complex querying like inheritance support etc. Refer to Compass Search for more details

Conclusion
          Compass searching provides a very flexible and powerful way of searching capabilities, agnostic of underlying JPA entities, which is sometime referred to google style searching. In my subsequent article I will cover the scabaility of the compass by putting its indexes in the oracle coherence

Predictable Linear Scalability Design Pattern

Aim

            To use a messaging middleware queue to distribute the workload across the jvm is quiet common and helps in linear scalability. This blog explain the design pattern “Polling Consumer” which aids in workload distribution. This blog also explains how the messaging middleware itself become a bottleneck in linear scalability and how to overcome it. Quiet a famous saying I have heard in many of design meetings "Drop a message in a queue, it will be picked up by the consumer to perform the processing. Increase the consumers to increase the performance". Lets see how much truth in it :-)

           

Polling Consumer

In any messaging system (queue), the consumer needs an indication that application is ready so that it can consume the message. The best approach for the consumer is to repeatedly check the channel for message availability. If any message is available, the consumer consumes it (process it). This checking is a continuous process known as polling. The messaging system (queue) can be polled from the multiple jvms. All the consumers/jvms should equally share the workload and should not keep hold of messages on which they are not working.

Solution:

The application should use a polling consumer, one that explicitly makes a call when it wants to receive a message. 

Interactions:

A polling consumer is a message receiver. A polling consumer restricts the number of concurrent messages to be consumed by limiting the number of polling threads. In this way, it prevents the application from being blocked by having to process too many requests, and keeps any extra messages queued up until the receiver can process them.Each consumer/jvm will ask for message whenever it has capacity to perform hence provide perfect load balancing automatically.
Merits

1. Consumer asks for work (as a message) whenever it has capacity to perform, hence provide efficient workload management.
2. No message is lost as all the messages are persisted in the messaging system (queue).
3. Increasing number of consumers (jvms) can increase the scalability/performance to certain level.

Demerits
Predicable scalability/performance cannot be guaranteed, as increasing consumers (jvms) the messaging system can become the bottleneck as shown by the picture below

The messaging middleware (queue) becomes the bottleneck because, as the numbers of consumers are increased to achieve the linear scalability, the messaging middleware fails to handle so many consumers. The messaging middleware  will look like a overloaded horse cart like below

Poor horse chart or should I say messaging middle ware ;-)

The following picture shows the decreasing performance (i.e. no messages processed per seconds) as the numbers of jvms are increased. In below the queue used is a durable queue.

NOTE: - The following tests were performed on windows machine.

Hence with queue in db after the performance reached the optimum level (5 jvms as shown in above picture), adding more jvm results in negative performance.

Predictable Scalability

            One the key feature of Oracle Coherence is to provide “Predictable Scalability” i.e. by increasing a node (a jvm) in a cluster the performance of the system increases. The following picture shows the Queue of the “Polling Consumer” pattern being implemented using Oracle Coherences + AQ that is, AQ being made as the persistent store for the oracle coherence. As a result the consumer always enqueue/dequeue messages from the in-memory queue (hence AQ –DB bottleneck is removed). The enqueue will result a message in the coherence cache and also AQ and similarly dequeue of the message will result in message being removed from the coherence cache and AQ.

In the above design the cache store make sure message are always in sync with persistent storage (db). 

1. enqueue => oracle coherence cache put(message) => AQ enqueue

2. dequeue=> oracle coherence cache remove()   => AQ dequeue

  So the endpoint always perform operation in the cache

Conclusion

      A queue is an efficient way distributing work load. But the linear scalability cannot be always assured.ln functional areas like invoice processing,claim processing etc, using queue for workload distribution may not be always helpful in linear scalability. If the number of the JVMs are less then desired performance can be achieved by fine tunning the consumers.

In my next blog I will share the framework i.e design details to achieve the above. The "Polling Consumer" is an enterprise integration pattern (http://www.eaipatterns.com/PollingConsumer.html). In past, as a part of my job I have developed spring based framework which enable the implementation of enterprise integration design pattern. But spring source team have added a new framework called "Spring Integration", which enable ease of implementing enterprise integration pattern. How to create "Polling Consumer" using spring integration can be found at http://static.springsource.org/spring-integration/reference/htmlsingle/spring-integration-reference.html#endpoint-pollingconsumer

A sneak preview - in order to achieve Polling Consumer backed by queue in a cache (coherence) which also persist messages we need the following

 1) Off course, a queue/channel in oracle coherence backed by persistence storage. Lets call it - CoherenceAwareQueueChannel

2)  A Oracle Coherence cache store which persist the messages

3)  A simple POJO to which poller delivers the message - consumer. In the terminology of enterprise integration its called Service Activator (refer to http://www.eaipatterns.com/MessagingAdapter.html for detail)

Enqueue

Message message = MessageBuilder.withPayload("gaurav in cache").setHeader(JmsHeaders.TYPE, "sometype").build();

coherenceQueue.send(message);

Dequeue

Message msg = (Message) coherenceQueue.receive();

Polling Consumer

<beans:bean id="coherenceQueue" class="com.oracle.aq.cachestore.channel.CoherenceAwareQueueChannel" depends-on="cacheFactoryCreation"/>
<beans:bean id="messageConsumer" class="com.oracle.aq.cachestore.junit.AqCacheStoreTest$ServiceActivator"/>
<service-activator id="coherenceConsumer" input-channel="coherenceQueue"
 ref="messageConsumer" method="onMessage" auto-startup="false">
 <poller>
  <interval-trigger interval="10" />
 </poller>
</service-activator>

As a result of above a message will be passed to the POJO - ServiceActivator method onMessage. The poller will poll the CoherenceAwareQueue every 10 nano seconds. It's also possible to specify the number of polling threads.

The service activator pojo looks like
 public class ServiceActivator {
  
  public void onMessage(Message msg) {
                // Message will be delivered automatically
  }
 }

Simple programming model agnostic of coherence/spring details/jms details so that developer can use it with ease

Toplink/Eclipselink Coherence

Integration Using Spring

Introduction

            The aim of this document is to explain how to integrate EclipseLink and Oracle Coherence using Spring Framework.  These white paper target audiences are developers/technical architects/technical managers. This white paper assumes the reader has the knowledge of the following: -

1.       Java EE framework

2.       Spring Framework

Spring framework is collections of smaller frameworks and act as glue, which tie together different layers of enterprise application and provide consistent and simple programming model based on best practices. Some of the key features of the spring framework are: -

·         Provides IOC container (Inversion of control/Dependency Injection)

·         Provides AOP (Aspect Oriented Programming) framework.

·         Provides Data Access framework

·         Provides declarative Transaction management framework.

·         Provides testing framework which enables use to develop code using TDD approach (Test Driven Development)

3.       Eclipse Link

Eclipse Link is a comprehensive open source persistence solution with around 12 years of commercial usage.  Some of the key features of Eclipse Link are: -

·       Fully JPA compliant

·       Enables simplified configuration of the target application server

·       Enable integration with JTA transaction manager

·       Very flexible framework, which allows customizations like SessionCustomizer & DescriptorCustomizer, which allows customizations of eclipse link session.

·       Provide support for concurrency protection – locking

·       EclipseLink cache. It has two types of caches i.e. L1 cache (EntityManager) and L2 cache (EntityManagerFactory)

·       Provides advance query facilitity like object graph loading optimization using Join and Batch Fetch.

 

4.       Oracle Coherence

Oracle Coherence is a JCache-compliant in memory distributed data grid solution for clustered applications and application servers. Some of the key features of Oracle Coherence are: -

·       Container – less clustering of java processing (pojo – pliain old java objects)

·       Real-time event observation (Listener pattern)

·       Parallel queries and aggregation (Purely object based queries)

·       Clustered JMX

·       Pluggable Cache store

 

Overview

Eclipse link and Oracle coherence integration provides highly scalable solution(s) and also provide data grid-computing capabilities. Spring framework allows powerful feature of retrieving objects from the Spring BeanFactory. This feature can be further being extended to retrieve objects configured in a cache scheme from BeanFactory and hence give better control over object instances creation. This is especially true for cache servers configured with CacheStore objects. Typically these CacheStore need to configure with data sources, connection pools, etc. The following document explains how to integrate Eclipse link and Oracle Coherence using Spring Framework and hence provide ability to leverage spring’s framework programming model and ability to provide easy configuration of data sources/pool etc for plain Java object

Eclipse link grid – spring aware cache store/loader

Coherence supports pluggable CacheLoader and CacheStore implementations. As result the entity classes annotated with JPA specifications can directly interact with Coherence API (get, put, etc)

SpringAwareEntityCacheLoader is class which supports reading the JPA entities from the database by implementing load/loadAll methods. SpringAwareEntityCacheStore extendsSpringAwareEntityCacheLoader and also support write, update and delete operations to the database. This class also implements CacheStore and implements the methods store, storeAll, erase, eraseAll.

The figure shows a typical Oracle Coherence and Eclipse Link integration viaSpringAwareEntityCacheStore. 

CACHE FACTORY SPRING INTEGRATION

            The entry point to access caches or any other services is CacheFactory through the use of its static methods (getCache etc). Coherence provides flexibility to override its default behavior via use of cache configuration file (coherence-cache-config.xml). Hence its possible to provide custom implementations of Coherence interfaces CacheStore  & CacheLoader. These interfaces can be configured via a class-scheme. Class schemes provide a mechanism for instantiating an arbitrary Java object. The class-scheme may be configured to either instantiate objects directly via their class-name (using new operator), or indirectly via a class-factory-name and method-name. The class-scheme must be configured with either a class-name or class-factory-name and method-name.

It is possible to tell Coherence to retrieve objects (cache store) configured in a class-scheme from a Spring BeanFactory instead of creating its own instance.

 

SpringAwareCacheStoreFactory is spring aware class, which extends DefaultConfigurableCacheFactory, and provides the ability to delegate class-scheme bean instantiations responsibility to spring’s BeanFactory class.

Key features of SpringAwareCacheStoreFactory

1.       Application Context Aware class: - Since this class implements spring ApplicationContextAware interface, as result at the time bean post processing, this class will be handed over the current ApplicationContext. Hence it will be useful for the cache servers that require beans from the spring container.

2.       Bean Factory Ware class: - This class also implements spring BeanFactoryAware interface, as result at the time of bean post processing current BeanFactory will be handed over to it. As a result a BeanFactory can be provided at the runtime by an application or provided directly by the spring container. This is useful for Coherence applications running in a container that already has an existing BeanFactory.

Note: - Implementation of spring’s InitializingBean interface will make sure BeanFatory is set before making call to DefaultConfigurableFactory class instantiateAny method is called. This is the key method, which creates object (cache store)-using class-scheme. Typically the object will be configured as spring-bean: myCaheStore in the class-scheme tag in the coherence config file.

3.       Ability to set the path of coherence configuration file and hence clean approach to manage coherence configuration files.

 

Conclusion

            Using Spring as a glue framework to integrate Eclipse Link and Oracle Coherence provides following key benefits: -

 

1.       Ability to inject separate data source to be used by SpringAwareCacheStore to perform the database operations. The data source can be setup in spring configuration file

2.       Ability to provide pool of data sources. Again these data sources pools can be configured in spring configuration file.

3.       Provide seprate EntityManager for the SpringAwareEntityCacheStore as result application’s transaction will never be interfered.

4.       Make it possible to use spring’s JpaDaoSupport and hence spring declarative transaction management and also spring programming model (best practices)