Spring with Spark Framework for Microservices

Recently, my team decided to use Spark to build our new REST api.  Spark seems easy with less coding.  We still wanted to use Spring for dependency injection.  We also wanted to use Tomcat as the standalone web container.  After some research, we figured out how to build the REST api with Spring, Spark and Tomcat.

1. Create class SpringSparkFilter which extends SparkFilter. Override method getApplication:
2. Create class MySparkApplication, which implements SparkApplication. Implement init method:
3. Configure web.xml to use SpringSparkFilter:

Then either build a war file, and deploy it to tomcat or use tomcat7-maven-plugin for test run.  I usually use tomcat7-maven-plugin for fast development.  Here is the command:

mvn tomcat7:run

Then access http://localhost:8080/current-date.  The complete example can be found here.

HBase, Thrift2 and Erlang on Mac

There are not to many documents showing how Erlang can connect to HBase using Thrift protocol.  The very useful link which was published more than 5 years ago, and it discussed Thrift1.  Based on Lars George's post, Thrift2 will replace Thrift1 eventually.  The following will show what I have done to get Erlang to work with HBase using Thrift on a Mac machine.  It should be similar to get it work on a Linux machine.  Note that only Java is supported as first class citizen for HBase.  If you have the choice, I suggest using Java directly.

1. Download thrift-0.9.0 and follow this tutorial to install it.
   cp -r thrift-0.9.0/lib/erl/*  /usr/local/opt/erlang/lib/erlang/lib/thrift-0.9.0 
   (homebrew installed erlang to /usr/local/opt/erlang.  Find out your own erlang directory.)
   
2. Download latest stable hbase and decompress it.  I installed it to /usr/local/hbase-0.98.6.1-hadoop2, and will refer it as $HBASE_FOLDER.
3. Download hbase.thrift and generate erlang bindings.
   thrift -gen erl hbase.thrift
   It generates erlang bindings in gen-erl folder.
4. start hbase server:
   before starting hbase server, update conf/hbase-site.xml
   <configuration>
     <property>
       <name>hbase.rootdir</name>
       <value>file:///some-folder-hbase</value>
     </property>
     <property>
       <name>hbase.zookeeper.property.dataDir</name>
       <value>/some-folder-zookeeper</value>
     </property>
   </configuration>
   Then start hbase server:
   $HBASE_FOLDER/bin/start-hbase.sh
   
5. start Thrift2:
   $HBASE_FOLDER/bin/base-daemon.sh start thrift2
6. Create table 'ups' with column family 'ui' using HBase Shell (Thrift2 doesn't support create table yet):
   $HBASE_FOLDER/bin/hbase shell
   hbase(main):001:0>; create 'ups', 'ui'
7. Download erlang client file and save it to gen-erl folder.
8. Go to gen-erl folder, compile erlang files and start erlang shell:
   cd gen-erl
   erlc *.erl
   erl 
   1>hbase_thrift2_client:put("localhost", 9090).
   {{tclient,tHBaseService_thrift,
             {protocol,thrift_binary_protocol,
                       {binary_protocol,{transport,thrift_buffered_transport,
                                                   {buffered_transport,{transport,thrift_socket_transport,
                                                                                  {data,#Port<0 data-blogger-escaped-.716="">,infinity}},
                                                                       []}},
                                        true,true}},
             0},
    {ok,ok}}
   2>hbase_thrift2_client:get("localhost", 9090).
   {{tclient,tHBaseService_thrift,
             {protocol,thrift_binary_protocol,
                       {binary_protocol,{transport,thrift_buffered_transport,
                                                   {buffered_transport,{transport,thrift_socket_transport,
                                                                                  {data,#Port<0 data-blogger-escaped-.717="">,infinity}},
                                                                       []}},
                                        true,true}},
             0},
    {ok,{tResult,<<"zhentao-key1">>,
                 [{tColumnValue,<<"ui">>,<<"test">>,<<"xyz1">>,1411075681134,
                                undefined},
                  {tColumnValue,<<"ui">>,<<"test1">>,<<"abc1">>,1411075681134,
                                undefined}]}}}

The detailed erlang project can be found here

Spring AMQP and RabbitMQ High Availability (HA)

Recently, I finished a project to use RabbitMQ as the message broker.  It is pretty easy to set up a RabbitMQ cluster with High Availability (HA).  The RabbitMQ has the pretty good documentation on how to set up the cluster.  I also document it here on how to set up a 2-node cluster on a single Mac machine.  In this post, I would like to show how we can leverage spring-amqp to connect to RabbitMQ cluster.  I have a rabbit cluster running with 2 nodes: localhost:5673 and localhost:5672.

It is pretty easy to use spring-amqp.  I used maven to manage the dependencies:


jackson-jaxrs-json-provider is used to serialize java object to json, and deserialize json back to java object.

When creating ConnectionFactory, the addresses should be used instead of the host and port:


The addresses are the comma separated host:port pairs which consist of the cluster.

For producer, we use rabbitTemplate to send messages:



For consumer, a MessageListenerContainer is created to consume message asynchronously:



The MessageHandler code is as follows:


This class can be called anything you like, but the method must be called handleMessaege and with the correct signature (here it is Employee to match producer).  If you want to change the method name, you have to call:

          MessageListenerAdapter.setDefaultListenerMethod

The source code can be download from github.

Skip cobertura coverage check when skip unit test

After I introduced embedded mysql for unit testing to my project, the build is taking longer now.  For my local development, sometimes I want to disable the unit test:

      mvn clean package -DskipTests

Note that you can even skip the compiling of test class by using

     mvn clean package -Dmaven.test.skip

However, if you use cobertura for code coverage check, you would get the following error:

[ERROR] Project failed check. Total branch coverage rate of 0.0% is below 85.0%
Project failed check. Total line coverage rate of 0.0% is below 85.0%

So you need to skip cobertura as well.  if you google "skip cobertura", most answers suggested using haltOnFailure. However, cobertura already provides the skip functionality:

     mvn clean package -DskipTests -Dcobertura.skip

With -Dcobertura.skip, cobertura check will be skipped:

[INFO] --- cobertura-maven-plugin:2.5.2:instrument (default) @ tag-service ---
[INFO] Skipping cobertura execution

Using embedded mysql database for unit test with maven and spring

I used H2 in memory database to unit test my DAO.  However, my production database is MySQL.
Generally, H2 is compatible with MySQL.  However, H2 doesn't support ENUM type yet, and my schema uses ENUM for a column definition.  Inspired by this post and another, I successfully used embedded MySQL for unit test.  Here is what I did:

1.  Add mysql-connector-mxj dependency:


2. Create EmbeddedMysqlDatabase.java and EmbeddedMysqlDatabaseBuilder.java

3. Create spring bean datasource:
4. Run unit test with Spring:
A couple things are worth mentioning:

1. The test uses Spring annotation @DirtiesContext(classMode = ClassMode.AFTER_EACH_TEST_METHOD) which reload the context after every test method.  It means every test method starts a new mysql instance.  You may group test methods to 2 different categories, one changes data, and another not.  For those methods don't change data, only need to create one instance of mysql to speed up testing.

2. If you see the following error message on Linux server:
/lib64/libc.so.6: version `GLIBC_2.7' not found (required by /tmp/test_db_2420035739165052/bin/mysqld)

You need to either downgrade mysql-connector-mxj to 5.0.11 or upgrade your linux OS.  I got the above error message on CentOS 5.4 with 5.0.12, but no problem with 5.0.11. However, 5.0.11 isn't in any public maven repo, but you can download it from here and install it to your local repo, or upload to your company's repo. Using the following command to check CentOS version:

cat /etc/redhat-release

3. The code is tested on Mac and CentOS only, and can be downloaded from github

Example for enabling CORS support in Spring Rest Api 3.2

My last year's post "Enable CORS support in REST services with Spring 3.1" seems causing some confusion.  I decided to create an example to show how to enable CORS with Spring rest api.  The CorsFilter is same as before:


Below are 2 endpoints from EmployeeController.java:

The update method adds the header Access-Control-Allow-Origin with "*", but delete method doesn't.  Therefore, the update method is enabled with CORS, but delete isn't.  If delete endpoint is called, the following error will be shown in Chrome:

"cannot load http://localhost:8080/rest/employee/1. Origin http://127.0.0.1:8080 is not allowed by Access-Control-Allow-Origin."

However, the delete method is still invoked on the server side since the pre-flight request (OPTIONS)
allows DELETE method to be called.

The entire project can be downloaded from github.  Following README to test it.

My intention was to disable/enable CORS support in each individual method by setting "Access-Control-Allow-Origin", but it seems not working as expected:  Although the browser returns correct info, the method call is still invoked on the server side even Access-Control-Allow-Origin is not set.  If you are allowed to enable all endpoints with CORS support, the code can be simplified as below:

The only difference is that addHeader("Access-Control-Allow-Origin") is moved out the if check. And then the update method can be simplified as:

The code can be downloaded from github too.

Add an existing project to a newly created git repository

I recently created a project on my local machine using maven archetype.  See my previous post on how to generate a new project from installed maven catalog.  I then got github.com created a repository for this project, for example: git@github.com:zhentao/test1.git

How can I associate my existing project with the newly created git repo?  There are 2 ways to do it:

First option:

1. clone the git repo:
git clone git@github.com:zhentao/test1.git

It will clone the repo to test1 folder.

2. copy your existing project to test1 folder

cp -r /path/to/project /path/to/test1

3.  commit and push all files

git add .
git commit -m "first commit"
git push -u origin master

Second option (more git style)

1. go to your project folder

cd /path/to/project

2. run the following commands:

git init
git add .
git commit -m "first commit"
git remote add origin git@github.com:zhentao/test1.git
  
The above commands associate your project to git remote repo.

3. run the following commands to to merge the conflict:

git fetch
git merge origin/master
 --run it if conflicts exist
git mergetool
git commit

git push -u origin master