Search This Blog

Monday, October 12, 2020

Micrometer with Dynamic Tags

 With Micrometer, it is easy to create a counter with Builder:

Counter okCounter = Counter.builder("http.status.count")
.tag("status", "200")
.description("The number of OK count")
.register(meterRegistry);
Counter notFoundCounter = Counter.builder("http.status.count")
.tag("status", "404")
.description("The number of not found count")
.register(meterRegistry);

However, it is not feasible to create counter for every HTTP status.  We can use Java HashMap to cache the creation dynamically by leveraging the method computeIfAbsent

import io.micrometer.core.instrument.Counter;
import io.micrometer.core.instrument.MeterRegistry;
import java.util.concurrent.ConcurrentHashMap;
import java.util.Map;
import org.springframework.http.HttpStatus;

public class HttpStatusCounter {
private static final String HTTP_STATUS_COUNT = "http.status.count";
private final MeterRegistry meterRegistry;
private Map<Integer, Counter> httpStatusCounterMap;

public HttpStatusCounter(MeterRegistry meterRegistry) {
httpStatusCounterMap = new ConcurrentHashMap<>();
this.meterRegistry = meterRegistry;
}

public void increment(HttpStatus status, int amount) {
Counter statusCounter = httpStatusCounterMap.computeIfAbsent(
status.value(), k -> Counter.builder(HTTP_STATUS_COUNT)
.tag("status", String.valueOf(status.value()))
.register(meterRegistry));
statusCounter.increment(amount);
}
}

Friday, September 4, 2020

Elasticsearch with Geopoint

Install Elasticsearch on Mac

Install the tap

brew tap elastic/tap

Install Elasticsearch

brew install elastic/tap/elasticsearch-full

Start Elasticsearch

elasticsearch

Confirm the installation works

curl http://localhost:9200

Something similar to the following should be shown:

{
  "name" : "MACC02Y753HJGH5",
  "cluster_name" : "elasticsearch_zhentao.li",
  "cluster_uuid" : "mkwsHWW8SoCcPzXg_KOyuQ",
  "version" : {
    "number" : "7.9.1",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "083627f112ba94dffc1232e8b42b73492789ef91",
    "build_date" : "2020-09-01T21:22:21.964974Z",
    "build_snapshot" : false,
    "lucene_version" : "8.6.2",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

Search by Geo Distance

Define the index for Geo Type

put http://localhost:9200/regions
{
"mappings": {
"properties": {
"name": {
type: "text"
},
"location": {
"type": "geo_point"
}
}
}
}

With the following response:

{
"acknowledged": true,
"shards_acknowledged": true,
"index": "regions"
}

Download Geo data file

curl https://restcountries.eu/rest/v1/all | jq -c '.[] | {"index": {"_index": "regions", "_id": .alpha3Code}}, {name: .name, location: [.latlng[1], .latlng[0]] }' > regions

or download it from here.

Import the file

curl -H 'Content-Type: application/json'  -s -XPUT localhost:9200/_bulk --data-binary "@regions"

List all regions

http://localhost:9200/regions/_search?size=100
GET /regions/_search
{
"size" : 100,
"query": {
"match_all": {}
}
}

Filter by Geo distance

{
"size": 50,
"query": {
"bool": {
"must": {
"match_all": {}
},
"filter": {
"geo_distance": {
"distance": "3000km",
"location": {
"lat": 35,
"lon": 105
}
}
}
}
},
"sort": [
{
"_geo_distance": {
"location": {
"lat": 35,
"lon": 105
},
"order": "asc",
"unit": "km",
"distance_type": "arc"
}
}
]
}

The result is like below

{
"took": 15,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 19,
"relation": "eq"
},
"max_score": null,
"hits": [
{
"_index": "regions",
"_type": "_doc",
"_id": "CHN",
"_score": null,
"_source": {
"name": "China",
"location": [
105,
35
]
},
"sort": [
0.0
]
},
{
"_index": "regions",
"_type": "_doc",
"_id": "MNG",
"_score": null,
"_source": {
"name": "Mongolia",
"location": [
105,
46
]
},
"sort": [
1223.1458751104453
]
},
{
"_index": "regions",
"_type": "_doc",
"_id": "MMR",
"_score": null,
"_source": {
"name": "Myanmar",
"location": [
98,
22
]
},
"sort": [
1597.9864780876323
]
},
{
"_index": "regions",
"_type": "_doc",
"_id": "BTN",
"_score": null,
"_source": {
"name": "Bhutan",
"location": [
90.5,
27.5
]
},
"sort": [
1608.420695556179
]
},
{
"_index": "regions",
"_type": "_doc",
"_id": "MAC",
"_score": null,
"_source": {
"name": "Macau",
"location": [
113.55,
22.16666666
]
},
"sort": [
1651.5104987054583
]
},
{
"_index": "regions",
"_type": "_doc",
"_id": "HKG",
"_score": null,
"_source": {
"name": "Hong Kong",
"location": [
114.16666666,
22.25
]
},
"sort": [
1674.4580484734988
]
},
{
"_index": "regions",
"_type": "_doc",
"_id": "LAO",
"_score": null,
"_source": {
"name": "Laos",
"location": [
105,
18
]
},
"sort": [
1890.3163582803475
]
},
{
"_index": "regions",
"_type": "_doc",
"_id": "BGD",
"_score": null,
"_source": {
"name": "Bangladesh",
"location": [
90,
24
]
},
"sort": [
1894.1562153610287
]
},
{
"_index": "regions",
"_type": "_doc",
"_id": "TWN",
"_score": null,
"_source": {
"name": "Taiwan",
"location": [
121,
23.5
]
},
"sort": [
2006.2988706499348
]
},
{
"_index": "regions",
"_type": "_doc",
"_id": "PRK",
"_score": null,
"_source": {
"name": "North Korea",
"location": [
127,
40
]
},
"sort": [
2012.9111514604588
]
},
{
"_index": "regions",
"_type": "_doc",
"_id": "KOR",
"_score": null,
"_source": {
"name": "South Korea",
"location": [
127.5,
37
]
},
"sort": [
2031.4778900496392
]
},
{
"_index": "regions",
"_type": "_doc",
"_id": "VNM",
"_score": null,
"_source": {
"name": "Vietnam",
"location": [
107.83333333,
16.16666666
]
},
"sort": [
2113.0731526216714
]
},
{
"_index": "regions",
"_type": "_doc",
"_id": "NPL",
"_score": null,
"_source": {
"name": "Nepal",
"location": [
84,
28
]
},
"sort": [
2132.4138804176705
]
},
{
"_index": "regions",
"_type": "_doc",
"_id": "THA",
"_score": null,
"_source": {
"name": "Thailand",
"location": [
100,
15
]
},
"sort": [
2279.3258680765903
]
},
{
"_index": "regions",
"_type": "_doc",
"_id": "KHM",
"_score": null,
"_source": {
"name": "Cambodia",
"location": [
105,
13
]
},
"sort": [
2446.291756434398
]
},
{
"_index": "regions",
"_type": "_doc",
"_id": "KGZ",
"_score": null,
"_source": {
"name": "Kyrgyzstan",
"location": [
75,
41
]
},
"sort": [
2697.507030320444
]
},
{
"_index": "regions",
"_type": "_doc",
"_id": "RUS",
"_score": null,
"_source": {
"name": "Russia",
"location": [
100,
60
]
},
"sort": [
2803.2803033336695
]
},
{
"_index": "regions",
"_type": "_doc",
"_id": "JPN",
"_score": null,
"_source": {
"name": "Japan",
"location": [
138,
36
]
},
"sort": [
2975.112941971521
]
},
{
"_index": "regions",
"_type": "_doc",
"_id": "PHL",
"_score": null,
"_source": {
"name": "Philippines",
"location": [
122,
13
]
},
"sort": [
2983.948851295192
]
}
]
}
}

When you search by distance, Elasticsearch won’t return the distance itself as it isn’t a real field. If you need sorting, Elasticsearch advise to use the sort function to return the value.



Sunday, February 21, 2016

Spring with Spark Framework for Microservices

Recently, my team decided to use Spark to build our new REST api.  Spark seems easy with less coding.  We still wanted to use Spring for dependency injection.  We also wanted to use Tomcat as the standalone web container.  After some research, we figured out how to build the REST api with Spring, Spark and Tomcat.
  1. Create class SpringSparkFilter which extends SparkFilter. Override method getApplication:
  2. Create class MySparkApplication, which implements SparkApplication. Implement init method:
  3. Configure web.xml to use SpringSparkFilter:
Then either build a war file, and deploy it to tomcat or use tomcat7-maven-plugin for test run.  I usually use tomcat7-maven-plugin for fast development.  Here is the command:

mvn tomcat7:run

Then access http://localhost:8080/current-date.  The complete example can be found here.

Thursday, September 18, 2014

HBase, Thrift2 and Erlang on Mac

There are not to many documents showing how Erlang can connect to HBase using Thrift protocol.  The very useful link which was published more than 5 years ago, and it discussed Thrift1.  Based on Lars George's post, Thrift2 will replace Thrift1 eventually.  The following will show what I have done to get Erlang to work with HBase using Thrift on a Mac machine.  It should be similar to get it work on a Linux machine.  Note that only Java is supported as first class citizen for HBase.  If you have the choice, I suggest using Java directly.
  1. Download thrift-0.9.0 and follow this tutorial to install it.
    cp -r thrift-0.9.0/lib/erl/*  /usr/local/opt/erlang/lib/erlang/lib/thrift-0.9.0 
    (homebrew installed erlang to /usr/local/opt/erlang.  Find out your own erlang directory.)
  2. Download latest stable hbase and decompress it.  I installed it to /usr/local/hbase-0.98.6.1-hadoop2, and will refer it as $HBASE_FOLDER.
  3. Download hbase.thrift and generate erlang bindings.
    thrift -gen erl hbase.thrift
    It generates erlang bindings in gen-erl folder.
  4. start hbase server:
    before starting hbase server, update conf/hbase-site.xml
    <configuration>
      <property>
        <name>hbase.rootdir</name>
        <value>file:///some-folder-hbase</value>
      </property>
      <property>
        <name>hbase.zookeeper.property.dataDir</name>
        <value>/some-folder-zookeeper</value>
      </property>
    </configuration>

    Then start hbase server:
    $HBASE_FOLDER/bin/start-hbase.sh
  5. start Thrift2:
    $HBASE_FOLDER/bin/base-daemon.sh start thrift2
  6. Create table 'ups' with column family 'ui' using HBase Shell (Thrift2 doesn't support create table yet):
    $HBASE_FOLDER/bin/hbase shell
    hbase(main):001:0>; create 'ups', 'ui'
  7. Download erlang client file and save it to gen-erl folder.
  8. Go to gen-erl folder, compile erlang files and start erlang shell:
    cd gen-erl
    erlc *.erl
    erl 
    1>hbase_thrift2_client:put("localhost", 9090).
    {{tclient,tHBaseService_thrift,
              {protocol,thrift_binary_protocol,
                        {binary_protocol,{transport,thrift_buffered_transport,
                                                    {buffered_transport,{transport,thrift_socket_transport,
                                                                                   {data,#Port<0 data-blogger-escaped-.716="">,infinity}},
                                                                        []}},
                                         true,true}},
              0},
     {ok,ok}}

    2>hbase_thrift2_client:get("localhost", 9090).
    {{tclient,tHBaseService_thrift,
              {protocol,thrift_binary_protocol,
                        {binary_protocol,{transport,thrift_buffered_transport,
                                                    {buffered_transport,{transport,thrift_socket_transport,
                                                                                   {data,#Port<0 data-blogger-escaped-.717="">,infinity}},
                                                                        []}},
                                         true,true}},
              0},
     {ok,{tResult,<<"zhentao-key1">>,
                  [{tColumnValue,<<"ui">>,<<"test">>,<<"xyz1">>,1411075681134,
                                 undefined},
                   {tColumnValue,<<"ui">>,<<"test1">>,<<"abc1">>,1411075681134,
                                 undefined}]}}}
The detailed erlang project can be found here

Friday, August 30, 2013

Spring AMQP and RabbitMQ High Availability (HA)

Recently, I finished a project to use RabbitMQ as the message broker.  It is pretty easy to set up a RabbitMQ cluster with High Availability (HA).  The RabbitMQ has the pretty good documentation on how to set up the cluster.  I also document it here on how to set up a 2-node cluster on a single Mac machine.  In this post, I would like to show how we can leverage spring-amqp to connect to RabbitMQ cluster.  I have a rabbit cluster running with 2 nodes: localhost:5673 and localhost:5672.

It is pretty easy to use spring-amqp.  I used maven to manage the dependencies:


jackson-jaxrs-json-provider is used to serialize java object to json, and deserialize json back to java object.

When creating ConnectionFactory, the addresses should be used instead of the host and port:


The addresses are the comma separated host:port pairs which consist of the cluster.

For producer, we use rabbitTemplate to send messages:



For consumer, a MessageListenerContainer is created to consume message asynchronously:



The MessageHandler code is as follows:


This class can be called anything you like, but the method must be called handleMessaege and with the correct signature (here it is Employee to match producer).  If you want to change the method name, you have to call:

          MessageListenerAdapter.setDefaultListenerMethod

The source code can be download from github.


Tuesday, June 11, 2013

Skip cobertura coverage check when skip unit test

After I introduced embedded mysql for unit testing to my project, the build is taking longer now.  For my local development, sometimes I want to disable the unit test:

      mvn clean package -DskipTests

Note that you can even skip the compiling of test class by using

     mvn clean package -Dmaven.test.skip

However, if you use cobertura for code coverage check, you would get the following error:

[ERROR] Project failed check. Total branch coverage rate of 0.0% is below 85.0%
Project failed check. Total line coverage rate of 0.0% is below 85.0%


So you need to skip cobertura as well.  if you google "skip cobertura", most answers suggested using haltOnFailure. However, cobertura already provides the skip functionality:

     mvn clean package -DskipTests -Dcobertura.skip

With -Dcobertura.skip, cobertura check will be skipped:

[INFO] --- cobertura-maven-plugin:2.5.2:instrument (default) @ tag-service ---
[INFO] Skipping cobertura execution

Monday, June 10, 2013

Using embedded mysql database for unit test with maven and spring

I used H2 in memory database to unit test my DAO.  However, my production database is MySQL.
Generally, H2 is compatible with MySQL.  However, H2 doesn't support ENUM type yet, and my schema uses ENUM for a column definition.  Inspired by this post and another, I successfully used embedded MySQL for unit test.  Here is what I did:

1.  Add mysql-connector-mxj dependency:


2. Create EmbeddedMysqlDatabase.java and EmbeddedMysqlDatabaseBuilder.java

3. Create spring bean datasource:
4. Run unit test with Spring:
A couple things are worth mentioning:

1. The test uses Spring annotation @DirtiesContext(classMode = ClassMode.AFTER_EACH_TEST_METHOD) which reload the context after every test method.  It means every test method starts a new mysql instance.  You may group test methods to 2 different categories, one changes data, and another not.  For those methods don't change data, only need to create one instance of mysql to speed up testing.

2. If you see the following error message on Linux server:
/lib64/libc.so.6: version `GLIBC_2.7' not found (required by /tmp/test_db_2420035739165052/bin/mysqld)

You need to either downgrade mysql-connector-mxj to 5.0.11 or upgrade your linux OS.  I got the above error message on CentOS 5.4 with 5.0.12, but no problem with 5.0.11. However, 5.0.11 isn't in any public maven repo, but you can download it from here and install it to your local repo, or upload to your company's repo. Using the following command to check CentOS version:

cat /etc/redhat-release

3. The code is tested on Mac and CentOS only, and can be downloaded from github

Monday, June 3, 2013

Example for enabling CORS support in Spring Rest Api 3.2

My last year's post "Enable CORS support in REST services with Spring 3.1" seems causing some confusion.  I decided to create an example to show how to enable CORS with Spring rest api.  The CorsFilter is same as before:


Below are 2 endpoints from EmployeeController.java:

The update method adds the header Access-Control-Allow-Origin with "*", but delete method doesn't.  Therefore, the update method is enabled with CORS, but delete isn't.  If delete endpoint is called, the following error will be shown in Chrome:

"cannot load http://localhost:8080/rest/employee/1. Origin http://127.0.0.1:8080 is not allowed by Access-Control-Allow-Origin."

However, the delete method is still invoked on the server side since the pre-flight request (OPTIONS)
allows DELETE method to be called.

The entire project can be downloaded from github.  Following README to test it.

My intention was to disable/enable CORS support in each individual method by setting "Access-Control-Allow-Origin", but it seems not working as expected:  Although the browser returns correct info, the method call is still invoked on the server side even Access-Control-Allow-Origin is not set.  If you are allowed to enable all endpoints with CORS support, the code can be simplified as below:

The only difference is that addHeader("Access-Control-Allow-Origin") is moved out the if check. And then the update method can be simplified as:

The code can be downloaded from github too.

Wednesday, May 15, 2013

Add an existing project to a newly created git repository

I recently created a project on my local machine using maven archetype.  See my previous post on how to generate a new project from installed maven catalog.  I then got github.com created a repository for this project, for example: git@github.com:zhentao/test1.git

How can I associate my existing project with the newly created git repo?  There are 2 ways to do it:

First option:

1. clone the git repo:
git clone git@github.com:zhentao/test1.git

It will clone the repo to test1 folder.

2. copy your existing project to test1 folder

cp -r /path/to/project /path/to/test1

3.  commit and push all files

git add .
git commit -m "first commit"
git push -u origin master

Second option (more git style)

1. go to your project folder

cd /path/to/project

2. run the following commands:

git init
git add .
git commit -m "first commit"
git remote add origin git@github.com:zhentao/test1.git

  
The above commands associate your project to git remote repo.

3. run the following commands to to merge the conflict:

git fetch
git merge origin/master
 --run it if conflicts exist
git mergetool
git commit

git push -u origin master

Tuesday, April 9, 2013

Riak: configure memory_backend

Riak supports several backends:
1. bitcask
2. memory
3. leveldb
4. multi

My recent project required me to use memory backend.  I have a test server with 24GB memory.  I tried to use 4GB as the max_memory.  It seems quite easy at beginning, and I just followed the doc from basho.com.  I just replace riak_kv_bitcask_backend with riak_kv_memory_backend:

{riak_kv, [
  {storage_backend, riak_kv_memory_backend},
 
Then I want to see how much data I can load with default configuration.  I loaded a little over 11 million records, then riak crashed.  Then I set max_memory to 4GB to add the following right under {eleveldb ... configuration:

{memory_backend, [
        ...,
            {max_memory, 4096}, %% 4GB in megabytes
        ...
]}
 
Note that the following is copied from riak's doc.  Then interesting, riak crashed with same number of records (11M+).  I then changed it to 8192, but riak still crashed with the same number of records.  I spent a couple days, and couldn't figure out the reason.  I went to #riak IRC and asked the following question, and those folks were very responsive and solved my problem right away:


[10:30] <zhentao> hi folks, how to configure the max_memory for memory_backend?
[10:30] <@alexmoore> Hi zhentao
[10:30] <zhentao> Hi
[10:30] <zhentao> I added the following to config file:
[10:30] <zhentao> {memory_backend, [              {max_memory, 8192} %% 8GB in megabytes             ]},
[10:31] <zhentao> but it didn't work
[10:31] <@alexmoore> In regards to your earlier question, if you don't specify a max_memory, or a TTL, the memory backend will continue to grow until it runs out of memory.
[10:31] <@alexmoore> Let me look at your config here
[10:31] <zhentao> it seems like that
[10:32] <zhentao> the node crashed after I loaded some data
[10:32] <zhentao> then I specify the max_memory to 8gb, and it still crashed with same amount of data
[10:32] <@alexmoore> How much RAM does the machine have?
[10:33] <sully_> Cluster health seems like it's degrading again.  We're starting to see the same errors as before.
[10:34] <zhentao> 24 GB
[10:34] <sully_> We are planning to add 3 more nodes.
[10:34] <@evanmcc> zhentao: that's 8GB per vnode
[10:34] <@rzezeski> sully_ ok, can you tar.gz the latest log files again, that might allow me to find the cause before it gets rotated out by the logger
[10:34] <@jcaprice> zhentao: did you restart the node after adding the memory constraint?
[10:35] <sully_> Getting you the latest logs.
[10:35] <zhentao> yes, I restarted it
[10:35] <@evanmcc> zhentao: how many nodes, and what is your ring size?
[10:36] <@alexmoore> zhentao, how many physical machines do you have in your cluster, and what is your ring size?
[10:36] <zhentao> it is a test server and just one machine
[10:36] <@jcaprice> ring size?
[10:37] <zhentao> i am new to riak, and where to find th ring size?
[10:37] <@evanmcc> if you didn't set it, it's 64
[10:37] <@alexmoore> In your vm.args file
[10:37] <@evanmcc> so you're limiting memory to 8GB * 64
[10:38] <zhentao> let me check
[10:38] <@evanmcc> alexmoore: it's in app.config under riak_core
[10:38] <@evanmcc> ring_creation_size
[10:38] <@alexmoore> Whoops, make that the app.config
[10:39] <zhentao> it is 64
[10:39] <zhentao> {ring_creation_size, 64},
[10:39] <@evanmcc> so you want to change max memory to 128
[10:39] <@evanmcc> if you want to limit it at 8GB
[10:40] <@jcaprice> zhentao: max_memory limits the amount of memory used per vnode, not for the node itself
[10:40] <zhentao> @jcaprice, so what number I should use for max_memory for my test server?
[10:41] <@evanmcc> 128, like I said above
[10:41] <zhentao> 128 mb?
[10:41] <@evanmcc> yes
[10:41] <@jcaprice> as evanmcc said, you'll want 8192 / 64
[10:41] <zhentao> thx, let me try it
 
So in  summary, {max_memory, 4096} is for each vnode, not for each machine.  Since I wanted to limit the memory usage for one machine to 4GB, I should use the following:
 
{max_memory, 64}
 
The reason is that the ring_creattion_size is 64 which means there are 64 vnodes on my single test server. 
 
64MB * 64 = 4096MB
 
After I changed max_memory to 64, riak is happy, and it didn't crash when I tried to load 20MM records.
 
Some interesting things I noticed:
 
1. since Riak use LRU for memory_backend, the old records are evicted if max_memory can't hold all records.
 
2. With the key as "70f21766-1cde-38d4-5920-a380003723b3", and value as "1365720608095#TQB:1.4:2:1", 
 
4GB memory with 64 vnodes can  hold about 2.5MM records
4GB memory with 4 vnodes can hold about 7MM records
 
Seems the number of vnodes/machine has big overhead


 
 

Thursday, February 14, 2013

eclipse JUNO hangs on start up

I use Eclipse JUNO every day.  However, I couldn't start it today after I restart my Mac.  Mac updated with Microsoft 2011 so I have to reboot Mac machine.  The strange thing happened: Eclipse got stuck.  I manually killed the start up process several times, but it didn't work.  Then I googled it, and found the solution from this post, and here is what I did:

rm $WORKSPACE_DIR/.metadata/.plugins/org.eclipse.e4.workbench/workbench.xmi
 
Then I can start eclipse, and happy coding.  

Friday, December 7, 2012

my shell script cheat sheet

1. find the total line number of a fold for some files:

find test -iname *.java | xargs wc -l | tail -1

2. count # of files in each folder recursively

for t in `find . -type d -ls | awk '{print $11}'`; do echo "$t `find $t -type f  | wc -l`" ; done > ~/count.txt

insert control character with vi

insert a visible ^A, we must use: <ctrl> + v + a.

Thursday, November 1, 2012

Some svn commands

Create tag from trunk or branch:
svn copy http://svn.mydomain.com/repository/myproject/trunk \
           http://svn.mydomain.com/repository/myproject/tags/release-1.0 \
      -m "Tagging the 1.0 release." 
The tag created is the snapshot of the trunk at the time the "svn copy" is executed.  
To be more precisely, the revision # can be passed to "svn copy":
svn copy -r 12345 http://svn.mydomain.com/repository/myproject/trunk \
           http://svn.mydomain.com/repository/myproject/tags/release-1.0 \
      -m "Tagging the 1.0 release."
Merge from Trunk to a branch:

1. check out the branch (assume the path is /path/to/mybranch)
2. go to above folder
3. run the following commmand:
svn merge http://svn.mydomain.com/repository/myproject/trunk .
It will merge all changes from trunk to mybranch since last merge. 
The above command is same as:
svn merge -rLastMergedRevision:HEAD http://svn.mydomain.com/repository/myproject/trunk .
View merge history:
"svn log" doesn't display the merge history.  It only shows the merge commit:
svn log
------------------------------------------------------------------------
r196402 | liz | 2012-11-01 10:39:13 -0700 (Thu, 01 Nov 2012) | 1 line

Merging r196340 through r196401
------------------------------------------------------------------------
r196340 | liz | 2012-10-31 14:52:06 -0700 (Wed, 31 Oct 2012) | 1 line

development branch for new feature
If need to see the merge history, the option --use-merge-history (-g) can be used with svn log:
svn log -g
------------------------------------------------------------------------
r196402 | liz | 2012-11-01 10:39:13 -0700 (Thu, 01 Nov 2012) | 1 line

Merging r196340 through r196401
------------------------------------------------------------------------
r196388 | xyz | 2012-11-01 09:50:28 -0700 (Thu, 01 Nov 2012) | 2 lines
Merged via: r196402

Added new unit tests

------------------------------------------------------------------------
r196340 | liz | 2012-10-31 14:52:06 -0700 (Wed, 31 Oct 2012) | 1 line

development branch for new feature
 
  

Tuesday, July 17, 2012

Run Sqoop on Amazon Elastic MapReduce (EMR) with Amazon RDS

Amazon EMR doesn't have Sqoop installed.  It is possible to run Sqoop with Amazon EMR.  The following blog shows how to install and run Sqoop:

http://blog.kylemulka.com/2012/04/how-to-install-sqoop-on-amazon-elastic-map-reduce-emr/

However, the solution isn't perfect since the input files are usually in S3 and Sqoop doesn't support S3 directly.  Here is my script to install sqoop and export data from S3 to Amazon RDS (mysql):

#!/bin/bash
BUCKET_NAME=zli-emr-test
SQOOP_FOLDER=sqoop-1.4.1-incubating__hadoop-0.20
SQOOP_TAR=$SQOOP_FOLDER.tar.gz

##change to home directory
cd ~

##Install sqoop on emr
hadoop fs -copyToLocal s3n://$BUCKET_NAME/$SQOOP_TAR $SQOOP_TAR
tar -xzf $SQOOP_TAR

##Install jdbc driver (ex mysql-connection-java.jar) to sqoop lib folder
hadoop fs -copyToLocal s3n://$BUCKET_NAME/mysql-connector-java-5.1.19.jar ~/$SQOOP_FOLDER/lib/
##Copy input file from S3 to hdf
HADOOP_INPUT=hdfs:///user/hadoop/myinput
hadoop distcp s3://$BUCKET_NAME/myinput $HADOOP_INPUT
~/$SQOOP_FOLDER/bin/sqoop export --connect jdbc:mysql://RDS-Host-name:3306/DB_NAME --username USERNAME --password PASSWORD --table TABLE_NAME --export-dir $HADOOP_INPUT --input-fields-terminated-by='\t'
The script assumes that sqoop tar ball and mysql-connector-java.jar are in S3 bucket, as well as the input file are in S3 too.

Note that, RDS needs to be configured to allow access to the database with the following 2 EC2 security groups:
ElasticMapReduce-master
ElasticMapReduce-slave

Tuesday, July 10, 2012

Run custom jar with elastic map reduce (EMR) on command line

I followed the instruction to run a custom jar on EMR:

http://aws.amazon.com/articles/3938

I got stuck with step 5:
5. Run the job flow.
 $ ./elasticmapreduce-client.rb RunJobFlow streaming_jobflow.json 
I couldn't find file "elasticmapreduce-client.rb" at all.  After some online searches, I got it work.  The correct command is:
./elastic-mapreduce --create --json path/to/your/flow
Here is my flow file looks like:
   [
      {
         "Name": "Custom Jar Grep Example 1",
         "ActionOnFailure": "CONTINUE",
         "HadoopJarStep":
         {
            "Jar": "s3n://YOUR_BUCKET/hadoop-examples-0.20.2-cdh3u4.jar",
                       ##"MainClass": "fully-qualified-class-name", 
            "Args":
            [
               "grep",
               "s3n://YOUR_BUCKET/input/example",
               "s3n://YOUR_BUCKET/output/example",
               "dfs[a-z.]+"
            ]
         }
      }
   ]
The flow is corresponding to the following hadoop command:
hadoop jar hadoop-examples-0.20.2-cdh3u4.jar grep input output 'dfs[a-z.]+'

Some useful tips:

1. show the log:
 ./elastic-mapreduce --jobflow JOB_ID --logs

Friday, June 29, 2012

mvn test: LocalJobRunner.java with java.lang.OutOfMemoryError

When I run mvn test to unit test some hadoop related program, I got the following:
java.lang.OutOfMemoryError: Java heap space
    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:949)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:428)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212)
I tried to set MAVEN_OPTS=-Xmx2048m, but it didn't work.   After the online research, I fixed this problem by adding the following to pom:

      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <configuration>
          <argLine>-Xms256m -Xmx512m</argLine>
        </configuration>
      </plugin>

Thursday, June 28, 2012

maven shade plugin: Invalid signature file digest for Manifest main attributes

If you get the following error message with maven shade plugin:

Exception in thread "main" java.lang.SecurityException: Invalid signature file digest for Manifest main attributes

You need to add the following to pom.xml:

        <configuration>
          <filters>
            <filter>
              <artifact>*:*</artifact>
              <excludes>
                <exclude>META-INF/*.SF</exclude>
                <exclude>META-INF/*.DSA</exclude>
                <exclude>META-INF/*.RSA</exclude>
              </excludes>
            </filter>
          </filters>
        </configuration>

Explanation:

The above configuration filters all files in META-INF ending with .SF, .DSA, and .RSA for all artifacts (*:*) when creating uber-jar file.

The reason java.lang.SecurityException is raised is because some dependency jar files are signed jar files.  A jar file is signed by using jarsigner, which creates 2 additional files and places them in META-INF:
  • a signature file, with a .SF extension, and
  • a signature block file, with a .DSA, .RSA, or .EC extension.
Since the uber-jar file is created, the signatures and integrity of signed JAR files are no longer valid.  When the uber-jar file is executed, java.lang.SecurityException is thrown.

See jarsigner for detailed explanation of JAR Signing and Verification Tool.

Sunday, June 24, 2012

Eclipse: fix plugin execution not covered by lifecycle configuration

I recently used maven-jaxb2-plugin to generate java classes based on xml schemas (xsd).  I can run "mvn compile" on command line to generate java classes with the following configuration:
            <plugin>
                <groupId>org.jvnet.jaxb2.maven2</groupId>
                <artifactId>maven-jaxb2-plugin</artifactId>
                <executions>
                    <execution>
                        <goals>
                            <goal>generate</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

However, my Eclipse shows the following error:

Plugin execution not covered by lifecycle configuration: org.jvnet.jaxb2.maven2:maven-jaxb2-plugin:0.8.1:generate (execution: default, phase: generate-sources)


This error prevents Eclipse generating java source codes and adding the generated sources to the class path.  In order to fix this problem, I need to install m2e connector for maven-jaxb2-plugin.  I can manually install it to Eclipse with the following update site:

http://bitstrings.github.com/m2e-connectors-p2/milestones/

However, Eclipse provides a better to install it.  Eclipse shows the problem with the following:


Roll the mouse over <execution>, it displays the following:
Click the link "Discover new m2e connector", and eclipse will try to find the correct  connector automatically:
Eclipse found the same m2e connector.  Click finish to install it.  After I installed it successfully, Eclipse is happy with maven-jaxb2-plugin.

This approach can be applied to other plugins too.