tag:blogger.com,1999:blog-68163600652415457542024-03-12T17:04:24.314-07:00Technology Blogthe technology I am interested in and work withzhentaohttp://www.blogger.com/profile/05719795252682912081noreply@blogger.comBlogger47125tag:blogger.com,1999:blog-6816360065241545754.post-47624546385500033282020-10-12T21:04:00.004-07:002020-10-28T12:01:33.434-07:00Micrometer with Dynamic Tags<p> With Micrometer, it is easy to create a counter with Builder:</p><pre style="background-color: white; color: #080808; font-family: "JetBrains Mono", monospace; font-size: 11.3pt;"><span style="color: #871094;">Counter okCounter </span>= <span style="color: black;">Counter</span>.<span style="font-style: italic;">builder</span>(<span style="color: #871094; font-style: italic;"></span><span style="color: #067d17;">"http.status.count"</span>)<br /> .tag(<span style="color: #067d17;">"status"</span>, <span style="color: #067d17;">"200"</span>)<br /> .description(<span style="color: #067d17;">"The number of OK count"</span>)<br /> .register(meterRegistry);</pre><pre style="background-color: white; color: #080808; font-family: "JetBrains Mono", monospace; font-size: 11.3pt;"><span style="color: #871094;">Counter notFoundCounter </span>= <span style="color: black;">Counter</span>.<span style="font-style: italic;">builder</span>(<span style="color: #871094; font-style: italic;"></span><span style="color: #067d17;">"http.status.count"</span>)<br /> .tag(<span style="color: #067d17;">"status"</span>, <span style="color: #067d17;">"404"</span>)<br /> .description(<span style="color: #067d17;">"The number of not found count"</span>)<br /> .register(meterRegistry);</pre><p style="text-align: left;">However, it is not feasible to create counter for every HTTP status. We can use Java HashMap to cache the creation dynamically by leveraging the method <span style="font-family: courier;">computeIfAbsent</span></p><pre style="background-color: white; color: #080808; font-family: "JetBrains Mono", monospace; font-size: 11.3pt;"><span style="color: #0033b3;">import </span><span style="color: black;">io.micrometer.core.instrument.Counter</span>;<br /><span style="color: #0033b3;">import </span><span style="color: black;">io.micrometer.core.instrument.MeterRegistry</span>;<br /><span style="color: #0033b3;">import </span><span style="color: black;">java.util.</span><span style="color: black;"><span style="color: black;">concurrent.</span>ConcurrentHashMap</span>;<br /><span style="color: #0033b3;">import </span><span style="color: black;">java.util.Map</span>;<br /><span style="color: #0033b3;">import </span><span style="color: black;">org.springframework.http.HttpStatus</span>;<br /><br /><span style="color: #0033b3;">public class </span><span style="color: black;">HttpStatusCounter </span>{<br /> <span style="color: #0033b3;">private static final </span><span style="color: black;">String </span><span style="color: #871094; font-style: italic;">HTTP_STATUS_COUNT </span>= <span style="color: #067d17;">"http.status.count"</span>;<br /> <span style="color: #0033b3;">private final </span><span style="color: black;">MeterRegistry </span><span style="color: #871094;">meterRegistry</span>;<br /> <span style="color: #0033b3;">private </span><span style="color: black;">Map</span><<span style="color: black;">Integer</span>, <span style="color: black;">Counter</span>> <span style="color: #871094;">httpStatusCounterMap</span>;<br /><br /> <span style="color: #0033b3;">public </span><span style="color: #00627a;">HttpStatusCounter</span>(<span style="color: black;">MeterRegistry </span>meterRegistry) {<br /> <span style="color: #871094;">httpStatusCounterMap </span>= <span style="color: #0033b3;">new ConcurrentHashMap</span><>();<br /> <span style="color: #0033b3;">this</span>.<span style="color: #871094;">meterRegistry </span>= meterRegistry;<br /> }<br /><br /> <span style="color: #0033b3;">public void </span><span style="color: #00627a;">increment</span>(<span style="color: black;">HttpStatus </span>status, <span style="color: #0033b3;">int </span>amount) {<br /> <span style="color: black;">Counter statusCounter </span>= <span style="color: #871094;">httpStatusCounterMap</span>.computeIfAbsent(<br /> status.value(), k -> <span style="color: black;">Counter</span>.<span style="font-style: italic;">builder</span>(<span style="color: #871094; font-style: italic;">HTTP_STATUS_COUNT</span>)<br /> .tag(<span style="color: #067d17;">"status"</span>, <span style="color: black;">String</span>.<span style="font-style: italic;">valueOf</span>(<span style="color: #851691;">status</span>.value()))<br /> .register(<span style="color: #871094;">meterRegistry</span>));<br /> <span style="color: black;">statusCounter</span>.increment(amount);<br /> }<br />}</pre>zhentaohttp://www.blogger.com/profile/05719795252682912081noreply@blogger.com0tag:blogger.com,1999:blog-6816360065241545754.post-67722040033296613732020-09-04T18:42:00.007-07:002020-09-05T14:34:27.717-07:00Elasticsearch with Geopoint<h1 style="text-align: left;">Install Elasticsearch on Mac</h1><h3 style="text-align: left;">Install the tap<br /></h3><pre class="lang:default decode:true code_snippet2 lang:bash" style="-webkit-text-stroke-width: 0px; background: rgb(245, 242, 240) none repeat scroll 0% 0%; box-sizing: inherit; color: black; display: block; font-family: "consolas", "monaco", "andale mono", "ubuntu mono", monospace; font-size: 14px; font-style: normal; font-variant-caps: normal; font-variant-ligatures: normal; font-weight: 400; hyphens: none; letter-spacing: normal; line-height: 1.5; margin: 0.5em 0px; orphans: 2; overflow-wrap: normal; overflow: auto; padding: 1em; tab-size: 4; text-align: left; text-decoration-color: initial; text-decoration-style: initial; text-indent: 0px; text-shadow: rgb(255, 255, 255) 0px 1px; text-transform: none; white-space: pre; widows: 2; word-break: normal; word-spacing: 0px;">brew tap elastic/tap</pre><h2 style="text-align: left;"></h2><p></p><h3 style="text-align: left;">Install Elasticsearch</h3><pre class="lang:default decode:true code_snippet4 lang:bash" style="-webkit-text-stroke-width: 0px; background: rgb(245, 242, 240) none repeat scroll 0% 0%; box-sizing: inherit; color: black; display: block; font-family: "consolas", "monaco", "andale mono", "ubuntu mono", monospace; font-size: 14px; font-style: normal; font-variant-caps: normal; font-variant-ligatures: normal; font-weight: 400; hyphens: none; letter-spacing: normal; line-height: 1.5; margin: 0.5em 0px; orphans: 2; overflow-wrap: normal; overflow: auto; padding: 1em; tab-size: 4; text-align: left; text-decoration-color: initial; text-decoration-style: initial; text-indent: 0px; text-shadow: rgb(255, 255, 255) 0px 1px; text-transform: none; white-space: pre; widows: 2; word-break: normal; word-spacing: 0px;">brew install elastic/tap/elasticsearch-full</pre><h3 style="text-align: left;">Start Elasticsearch</h3><p></p><pre class="lang:default decode:true code_snippet7 lang:bash" style="-webkit-text-stroke-width: 0px; background: rgb(245, 242, 240) none repeat scroll 0% 0%; box-sizing: inherit; color: black; display: block; font-family: "consolas", "monaco", "andale mono", "ubuntu mono", monospace; font-size: 14px; font-style: normal; font-variant-caps: normal; font-variant-ligatures: normal; font-weight: 400; hyphens: none; letter-spacing: normal; line-height: 1.5; margin: 0.5em 0px; orphans: 2; overflow-wrap: normal; overflow: auto; padding: 1em; tab-size: 4; text-align: left; text-decoration-color: initial; text-decoration-style: initial; text-indent: 0px; text-shadow: rgb(255, 255, 255) 0px 1px; text-transform: none; white-space: pre; widows: 2; word-break: normal; word-spacing: 0px;">elasticsearch</pre><h3 style="text-align: left;">Confirm the installation works</h3><p></p><pre class="lang:default decode:true code_snippet8 lang:bash" style="-webkit-text-stroke-width: 0px; background: rgb(245, 242, 240) none repeat scroll 0% 0%; box-sizing: inherit; color: black; display: block; font-family: "consolas", "monaco", "andale mono", "ubuntu mono", monospace; font-size: 14px; font-style: normal; font-variant-caps: normal; font-variant-ligatures: normal; font-weight: 400; hyphens: none; letter-spacing: normal; line-height: 1.5; margin: 0.5em 0px; orphans: 2; overflow-wrap: normal; overflow: auto; padding: 1em; tab-size: 4; text-align: left; text-decoration-color: initial; text-decoration-style: initial; text-indent: 0px; text-shadow: rgb(255, 255, 255) 0px 1px; text-transform: none; white-space: pre; widows: 2; word-break: normal; word-spacing: 0px;">curl http://localhost:9200</pre><p></p><p></p><p></p><h3 style="text-align: left;">Something similar to the following should be shown:</h3><pre class="lang:default decode:true code_snippet8 lang:bash" style="-webkit-text-stroke-width: 0px; background: rgb(245, 242, 240) none repeat scroll 0% 0%; box-sizing: inherit; color: black; display: block; font-family: "consolas", "monaco", "andale mono", "ubuntu mono", monospace; font-size: 14px; font-style: normal; font-variant-caps: normal; font-variant-ligatures: normal; font-weight: 400; hyphens: none; letter-spacing: normal; line-height: 1.5; margin: 0.5em 0px; orphans: 2; overflow-wrap: normal; overflow: auto; padding: 1em; tab-size: 4; text-align: left; text-decoration-color: initial; text-decoration-style: initial; text-indent: 0px; text-shadow: rgb(255, 255, 255) 0px 1px; text-transform: none; white-space: pre; widows: 2; word-break: normal; word-spacing: 0px;">{<br /> "name" : "MACC02Y753HJGH5",<br /> "cluster_name" : "elasticsearch_zhentao.li",<br /> "cluster_uuid" : "mkwsHWW8SoCcPzXg_KOyuQ",<br /> "version" : {<br /> "number" : "7.9.1",<br /> "build_flavor" : "default",<br /> "build_type" : "tar",<br /> "build_hash" : "083627f112ba94dffc1232e8b42b73492789ef91",<br /> "build_date" : "2020-09-01T21:22:21.964974Z",<br /> "build_snapshot" : false,<br /> "lucene_version" : "8.6.2",<br /> "minimum_wire_compatibility_version" : "6.8.0",<br /> "minimum_index_compatibility_version" : "6.0.0-beta1"<br /> },<br /> "tagline" : "You Know, for Search"<br />}</pre><h1 style="text-align: left;">Search by Geo Distance<br /></h1><h3 style="text-align: left;">Define the index for Geo Type<br /></h3><pre class="lang:default decode:true code_snippet8 lang:bash" style="-webkit-text-stroke-width: 0px; background: rgb(245, 242, 240) none repeat scroll 0% 0%; box-sizing: inherit; color: black; display: block; font-family: "consolas", "monaco", "andale mono", "ubuntu mono", monospace; font-size: 14px; font-style: normal; font-variant-caps: normal; font-variant-ligatures: normal; font-weight: 400; hyphens: none; letter-spacing: normal; line-height: 1.5; margin: 0.5em 0px; orphans: 2; overflow-wrap: normal; overflow: auto; padding: 1em; tab-size: 4; text-align: left; text-decoration-color: initial; text-decoration-style: initial; text-indent: 0px; text-shadow: rgb(255, 255, 255) 0px 1px; text-transform: none; white-space: pre; widows: 2; word-break: normal; word-spacing: 0px;">put http://localhost:9200/regions</pre><pre class="lang:default decode:true code_snippet8 lang:bash" style="-webkit-text-stroke-width: 0px; background: rgb(245, 242, 240) none repeat scroll 0% 0%; box-sizing: inherit; color: black; display: block; font-family: "consolas", "monaco", "andale mono", "ubuntu mono", monospace; font-size: 14px; font-style: normal; font-variant-caps: normal; font-variant-ligatures: normal; font-weight: 400; hyphens: none; letter-spacing: normal; line-height: 1.5; margin: 0.5em 0px; orphans: 2; overflow-wrap: normal; overflow: auto; padding: 1em; tab-size: 4; text-align: left; text-decoration-color: initial; text-decoration-style: initial; text-indent: 0px; text-shadow: rgb(255, 255, 255) 0px 1px; text-transform: none; white-space: pre; widows: 2; word-break: normal; word-spacing: 0px;">{<br /> "mappings": {<br /> "properties": {<br /> "name": {<br /> type: "text"<br /> },<br /> "location": {<br /> "type": "geo_point"<br /> }<br /> }<br /> }<br />}</pre><h3 style="text-align: left;">With the following response:</h3><pre class="lang:default decode:true code_snippet8 lang:bash" style="-webkit-text-stroke-width: 0px; background: rgb(245, 242, 240) none repeat scroll 0% 0%; box-sizing: inherit; color: black; display: block; font-family: "consolas", "monaco", "andale mono", "ubuntu mono", monospace; font-size: 14px; font-style: normal; font-variant-caps: normal; font-variant-ligatures: normal; font-weight: 400; hyphens: none; letter-spacing: normal; line-height: 1.5; margin: 0.5em 0px; orphans: 2; overflow-wrap: normal; overflow: auto; padding: 1em; tab-size: 4; text-align: left; text-decoration-color: initial; text-decoration-style: initial; text-indent: 0px; text-shadow: rgb(255, 255, 255) 0px 1px; text-transform: none; white-space: pre; widows: 2; word-break: normal; word-spacing: 0px;">{<br /> "acknowledged": true,<br /> "shards_acknowledged": true,<br /> "index": "regions"<br />}</pre><h3 style="text-align: left;">Download Geo data file</h3><p></p><pre class="lang:default decode:true code_snippet8 lang:bash" style="-webkit-text-stroke-width: 0px; background: rgb(245, 242, 240) none repeat scroll 0% 0%; box-sizing: inherit; color: black; display: block; font-family: "consolas", "monaco", "andale mono", "ubuntu mono", monospace; font-size: 14px; font-style: normal; font-variant-caps: normal; font-variant-ligatures: normal; font-weight: 400; hyphens: none; letter-spacing: normal; line-height: 1.5; margin: 0.5em 0px; orphans: 2; overflow-wrap: normal; overflow: auto; padding: 1em; tab-size: 4; text-align: left; text-decoration-color: initial; text-decoration-style: initial; text-indent: 0px; text-shadow: rgb(255, 255, 255) 0px 1px; text-transform: none; white-space: pre; widows: 2; word-break: normal; word-spacing: 0px;">curl https://restcountries.eu/rest/v1/all | jq -c '.[] | {"index": {"_index": "regions", "_id": .alpha3Code}}, {name: .name, location: [.latlng[1], .latlng[0]] }' > regions</pre><div style="text-align: left;"><p style="text-align: left;"><b>or download it from <a href="https://github.com/zhentao/geo-regions/blob/master/regions" target="_blank">here</a>.</b></p></div><h3 style="text-align: left;">Import the file</h3><pre class="lang:default decode:true code_snippet8 lang:bash" style="-webkit-text-stroke-width: 0px; background: rgb(245, 242, 240) none repeat scroll 0% 0%; box-sizing: inherit; color: black; display: block; font-family: "consolas", "monaco", "andale mono", "ubuntu mono", monospace; font-size: 14px; font-style: normal; font-variant-caps: normal; font-variant-ligatures: normal; font-weight: 400; hyphens: none; letter-spacing: normal; line-height: 1.5; margin: 0.5em 0px; orphans: 2; overflow-wrap: normal; overflow: auto; padding: 1em; tab-size: 4; text-align: left; text-decoration-color: initial; text-decoration-style: initial; text-indent: 0px; text-shadow: rgb(255, 255, 255) 0px 1px; text-transform: none; white-space: pre; widows: 2; word-break: normal; word-spacing: 0px;">curl -H 'Content-Type: application/json' -s -XPUT localhost:9200/_bulk --data-binary "@regions"</pre><h3 style="text-align: left;">List all regions</h3><p></p><pre class="lang:default decode:true code_snippet8 lang:bash" style="-webkit-text-stroke-width: 0px; background: rgb(245, 242, 240) none repeat scroll 0% 0%; box-sizing: inherit; color: black; display: block; font-family: "consolas", "monaco", "andale mono", "ubuntu mono", monospace; font-size: 14px; font-style: normal; font-variant-caps: normal; font-variant-ligatures: normal; font-weight: 400; hyphens: none; letter-spacing: normal; line-height: 1.5; margin: 0.5em 0px; orphans: 2; overflow-wrap: normal; overflow: auto; padding: 1em; tab-size: 4; text-align: left; text-decoration-color: initial; text-decoration-style: initial; text-indent: 0px; text-shadow: rgb(255, 255, 255) 0px 1px; text-transform: none; white-space: pre; widows: 2; word-break: normal; word-spacing: 0px;">http://localhost:9200/regions/_search?size=100</pre><pre class="lang:default decode:true code_snippet8 lang:bash" style="-webkit-text-stroke-width: 0px; background: rgb(245, 242, 240) none repeat scroll 0% 0%; box-sizing: inherit; color: black; display: block; font-family: "consolas", "monaco", "andale mono", "ubuntu mono", monospace; font-size: 14px; font-style: normal; font-variant-caps: normal; font-variant-ligatures: normal; font-weight: 400; hyphens: none; letter-spacing: normal; line-height: 1.5; margin: 0.5em 0px; orphans: 2; overflow-wrap: normal; overflow: auto; padding: 1em; tab-size: 4; text-align: left; text-decoration-color: initial; text-decoration-style: initial; text-indent: 0px; text-shadow: rgb(255, 255, 255) 0px 1px; text-transform: none; white-space: pre; widows: 2; word-break: normal; word-spacing: 0px;"><span style="-webkit-text-stroke-width: 0px; background-color: #f2f2f2; color: #292929; display: inline; float: none; font-family: Menlo, Monaco, "Courier New", Courier, monospace; font-size: 16px; font-style: normal; font-variant-caps: normal; font-variant-ligatures: normal; font-weight: 400; letter-spacing: -0.352px; orphans: 2; text-align: start; text-decoration-color: initial; text-decoration-style: initial; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px;">GET /regions/_search</span><br style="-webkit-text-stroke-width: 0px; background-color: #f2f2f2; box-sizing: inherit; color: #292929; font-family: "menlo", "monaco", "courier new", "courier", monospace; font-size: 16px; font-style: normal; font-variant-caps: normal; font-variant-ligatures: normal; font-weight: 400; letter-spacing: -0.352px; orphans: 2; text-align: start; text-decoration-color: initial; text-decoration-style: initial; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px;" /><span style="-webkit-text-stroke-width: 0px; background-color: #f2f2f2; color: #292929; display: inline; float: none; font-family: Menlo, Monaco, "Courier New", Courier, monospace; font-size: 16px; font-style: normal; font-variant-caps: normal; font-variant-ligatures: normal; font-weight: 400; letter-spacing: -0.352px; orphans: 2; text-align: start; text-decoration-color: initial; text-decoration-style: initial; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px;">{</span><br style="-webkit-text-stroke-width: 0px; background-color: #f2f2f2; box-sizing: inherit; color: #292929; font-family: "menlo", "monaco", "courier new", "courier", monospace; font-size: 16px; font-style: normal; font-variant-caps: normal; font-variant-ligatures: normal; font-weight: 400; letter-spacing: -0.352px; orphans: 2; text-align: start; text-decoration-color: initial; text-decoration-style: initial; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px;" /><span style="-webkit-text-stroke-width: 0px; background-color: #f2f2f2; color: #292929; display: inline; float: none; font-family: Menlo, Monaco, "Courier New", Courier, monospace; font-size: 16px; font-style: normal; font-variant-caps: normal; font-variant-ligatures: normal; font-weight: 400; letter-spacing: -0.352px; orphans: 2; text-align: start; text-decoration-color: initial; text-decoration-style: initial; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px;"> "size" : 100,</span><br style="-webkit-text-stroke-width: 0px; background-color: #f2f2f2; box-sizing: inherit; color: #292929; font-family: "menlo", "monaco", "courier new", "courier", monospace; font-size: 16px; font-style: normal; font-variant-caps: normal; font-variant-ligatures: normal; font-weight: 400; letter-spacing: -0.352px; orphans: 2; text-align: start; text-decoration-color: initial; text-decoration-style: initial; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px;" /><span style="-webkit-text-stroke-width: 0px; background-color: #f2f2f2; color: #292929; display: inline; float: none; font-family: Menlo, Monaco, "Courier New", Courier, monospace; font-size: 16px; font-style: normal; font-variant-caps: normal; font-variant-ligatures: normal; font-weight: 400; letter-spacing: -0.352px; orphans: 2; text-align: start; text-decoration-color: initial; text-decoration-style: initial; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px;"> "query": {</span><br style="-webkit-text-stroke-width: 0px; background-color: #f2f2f2; box-sizing: inherit; color: #292929; font-family: "menlo", "monaco", "courier new", "courier", monospace; font-size: 16px; font-style: normal; font-variant-caps: normal; font-variant-ligatures: normal; font-weight: 400; letter-spacing: -0.352px; orphans: 2; text-align: start; text-decoration-color: initial; text-decoration-style: initial; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px;" /><span style="-webkit-text-stroke-width: 0px; background-color: #f2f2f2; color: #292929; display: inline; float: none; font-family: Menlo, Monaco, "Courier New", Courier, monospace; font-size: 16px; font-style: normal; font-variant-caps: normal; font-variant-ligatures: normal; font-weight: 400; letter-spacing: -0.352px; orphans: 2; text-align: start; text-decoration-color: initial; text-decoration-style: initial; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px;"> "match_all": {}</span><br style="-webkit-text-stroke-width: 0px; background-color: #f2f2f2; box-sizing: inherit; color: #292929; font-family: "menlo", "monaco", "courier new", "courier", monospace; font-size: 16px; font-style: normal; font-variant-caps: normal; font-variant-ligatures: normal; font-weight: 400; letter-spacing: -0.352px; orphans: 2; text-align: start; text-decoration-color: initial; text-decoration-style: initial; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px;" /><span style="-webkit-text-stroke-width: 0px; background-color: #f2f2f2; color: #292929; display: inline; float: none; font-family: Menlo, Monaco, "Courier New", Courier, monospace; font-size: 16px; font-style: normal; font-variant-caps: normal; font-variant-ligatures: normal; font-weight: 400; letter-spacing: -0.352px; orphans: 2; text-align: start; text-decoration-color: initial; text-decoration-style: initial; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px;"> }</span><br style="-webkit-text-stroke-width: 0px; background-color: #f2f2f2; box-sizing: inherit; color: #292929; font-family: "menlo", "monaco", "courier new", "courier", monospace; font-size: 16px; font-style: normal; font-variant-caps: normal; font-variant-ligatures: normal; font-weight: 400; letter-spacing: -0.352px; orphans: 2; text-align: start; text-decoration-color: initial; text-decoration-style: initial; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px;" /><span style="-webkit-text-stroke-width: 0px; background-color: #f2f2f2; color: #292929; display: inline; float: none; font-family: Menlo, Monaco, "Courier New", Courier, monospace; font-size: 16px; font-style: normal; font-variant-caps: normal; font-variant-ligatures: normal; font-weight: 400; letter-spacing: -0.352px; orphans: 2; text-align: start; text-decoration-color: initial; text-decoration-style: initial; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px;">}</span></pre><h3 style="text-align: left;">Filter by Geo distance</h3><p></p><pre class="lang:default decode:true code_snippet8 lang:bash" style="-webkit-text-stroke-width: 0px; background: rgb(245, 242, 240) none repeat scroll 0% 0%; box-sizing: inherit; color: black; display: block; font-family: "consolas", "monaco", "andale mono", "ubuntu mono", monospace; font-size: 14px; font-style: normal; font-variant-caps: normal; font-variant-ligatures: normal; font-weight: 400; hyphens: none; letter-spacing: normal; line-height: 1.5; margin: 0.5em 0px; orphans: 2; overflow-wrap: normal; overflow: auto; padding: 1em; tab-size: 4; text-align: left; text-decoration-color: initial; text-decoration-style: initial; text-indent: 0px; text-shadow: rgb(255, 255, 255) 0px 1px; text-transform: none; white-space: pre; widows: 2; word-break: normal; word-spacing: 0px;">{<br /> "size": 50,<br /> "query": {<br /> "bool": {<br /> "must": {<br /> "match_all": {}<br /> },<br /> "filter": {<br /> "geo_distance": {<br /> "distance": "3000km",<br /> "location": {<br /> "lat": 35,<br /> "lon": 105<br /> }<br /> }<br /> }<br /> }<br /> },<br /> "sort": [<br /> {<br /> "_geo_distance": {<br /> "location": {<br /> "lat": 35,<br /> "lon": 105<br /> },<br /> "order": "asc",<br /> "unit": "km",<br /> "distance_type": "arc"<br /> }<br /> }<br /> ]<br />}<br /></pre><h3 style="text-align: left;">The result is like below</h3><p></p><pre class="lang:default decode:true code_snippet8 lang:bash" style="-webkit-text-stroke-width: 0px; background: rgb(245, 242, 240) none repeat scroll 0% 0%; box-sizing: inherit; color: black; display: block; font-family: "consolas", "monaco", "andale mono", "ubuntu mono", monospace; font-size: 14px; font-style: normal; font-variant-caps: normal; font-variant-ligatures: normal; font-weight: 400; hyphens: none; letter-spacing: normal; line-height: 1.5; margin: 0.5em 0px; orphans: 2; overflow-wrap: normal; overflow: auto; padding: 1em; tab-size: 4; text-align: left; text-decoration-color: initial; text-decoration-style: initial; text-indent: 0px; text-shadow: rgb(255, 255, 255) 0px 1px; text-transform: none; white-space: pre; widows: 2; word-break: normal; word-spacing: 0px;">{<br /> "took": 15,<br /> "timed_out": false,<br /> "_shards": {<br /> "total": 1,<br /> "successful": 1,<br /> "skipped": 0,<br /> "failed": 0<br /> },<br /> "hits": {<br /> "total": {<br /> "value": 19,<br /> "relation": "eq"<br /> },<br /> "max_score": null,<br /> "hits": [<br /> {<br /> "_index": "regions",<br /> "_type": "_doc",<br /> "_id": "CHN",<br /> "_score": null,<br /> "_source": {<br /> "name": "China",<br /> "location": [<br /> 105,<br /> 35<br /> ]<br /> },<br /> "sort": [<br /> 0.0<br /> ]<br /> },<br /> {<br /> "_index": "regions",<br /> "_type": "_doc",<br /> "_id": "MNG",<br /> "_score": null,<br /> "_source": {<br /> "name": "Mongolia",<br /> "location": [<br /> 105,<br /> 46<br /> ]<br /> },<br /> "sort": [<br /> 1223.1458751104453<br /> ]<br /> },<br /> {<br /> "_index": "regions",<br /> "_type": "_doc",<br /> "_id": "MMR",<br /> "_score": null,<br /> "_source": {<br /> "name": "Myanmar",<br /> "location": [<br /> 98,<br /> 22<br /> ]<br /> },<br /> "sort": [<br /> 1597.9864780876323<br /> ]<br /> },<br /> {<br /> "_index": "regions",<br /> "_type": "_doc",<br /> "_id": "BTN",<br /> "_score": null,<br /> "_source": {<br /> "name": "Bhutan",<br /> "location": [<br /> 90.5,<br /> 27.5<br /> ]<br /> },<br /> "sort": [<br /> 1608.420695556179<br /> ]<br /> },<br /> {<br /> "_index": "regions",<br /> "_type": "_doc",<br /> "_id": "MAC",<br /> "_score": null,<br /> "_source": {<br /> "name": "Macau",<br /> "location": [<br /> 113.55,<br /> 22.16666666<br /> ]<br /> },<br /> "sort": [<br /> 1651.5104987054583<br /> ]<br /> },<br /> {<br /> "_index": "regions",<br /> "_type": "_doc",<br /> "_id": "HKG",<br /> "_score": null,<br /> "_source": {<br /> "name": "Hong Kong",<br /> "location": [<br /> 114.16666666,<br /> 22.25<br /> ]<br /> },<br /> "sort": [<br /> 1674.4580484734988<br /> ]<br /> },<br /> {<br /> "_index": "regions",<br /> "_type": "_doc",<br /> "_id": "LAO",<br /> "_score": null,<br /> "_source": {<br /> "name": "Laos",<br /> "location": [<br /> 105,<br /> 18<br /> ]<br /> },<br /> "sort": [<br /> 1890.3163582803475<br /> ]<br /> },<br /> {<br /> "_index": "regions",<br /> "_type": "_doc",<br /> "_id": "BGD",<br /> "_score": null,<br /> "_source": {<br /> "name": "Bangladesh",<br /> "location": [<br /> 90,<br /> 24<br /> ]<br /> },<br /> "sort": [<br /> 1894.1562153610287<br /> ]<br /> },<br /> {<br /> "_index": "regions",<br /> "_type": "_doc",<br /> "_id": "TWN",<br /> "_score": null,<br /> "_source": {<br /> "name": "Taiwan",<br /> "location": [<br /> 121,<br /> 23.5<br /> ]<br /> },<br /> "sort": [<br /> 2006.2988706499348<br /> ]<br /> },<br /> {<br /> "_index": "regions",<br /> "_type": "_doc",<br /> "_id": "PRK",<br /> "_score": null,<br /> "_source": {<br /> "name": "North Korea",<br /> "location": [<br /> 127,<br /> 40<br /> ]<br /> },<br /> "sort": [<br /> 2012.9111514604588<br /> ]<br /> },<br /> {<br /> "_index": "regions",<br /> "_type": "_doc",<br /> "_id": "KOR",<br /> "_score": null,<br /> "_source": {<br /> "name": "South Korea",<br /> "location": [<br /> 127.5,<br /> 37<br /> ]<br /> },<br /> "sort": [<br /> 2031.4778900496392<br /> ]<br /> },<br /> {<br /> "_index": "regions",<br /> "_type": "_doc",<br /> "_id": "VNM",<br /> "_score": null,<br /> "_source": {<br /> "name": "Vietnam",<br /> "location": [<br /> 107.83333333,<br /> 16.16666666<br /> ]<br /> },<br /> "sort": [<br /> 2113.0731526216714<br /> ]<br /> },<br /> {<br /> "_index": "regions",<br /> "_type": "_doc",<br /> "_id": "NPL",<br /> "_score": null,<br /> "_source": {<br /> "name": "Nepal",<br /> "location": [<br /> 84,<br /> 28<br /> ]<br /> },<br /> "sort": [<br /> 2132.4138804176705<br /> ]<br /> },<br /> {<br /> "_index": "regions",<br /> "_type": "_doc",<br /> "_id": "THA",<br /> "_score": null,<br /> "_source": {<br /> "name": "Thailand",<br /> "location": [<br /> 100,<br /> 15<br /> ]<br /> },<br /> "sort": [<br /> 2279.3258680765903<br /> ]<br /> },<br /> {<br /> "_index": "regions",<br /> "_type": "_doc",<br /> "_id": "KHM",<br /> "_score": null,<br /> "_source": {<br /> "name": "Cambodia",<br /> "location": [<br /> 105,<br /> 13<br /> ]<br /> },<br /> "sort": [<br /> 2446.291756434398<br /> ]<br /> },<br /> {<br /> "_index": "regions",<br /> "_type": "_doc",<br /> "_id": "KGZ",<br /> "_score": null,<br /> "_source": {<br /> "name": "Kyrgyzstan",<br /> "location": [<br /> 75,<br /> 41<br /> ]<br /> },<br /> "sort": [<br /> 2697.507030320444<br /> ]<br /> },<br /> {<br /> "_index": "regions",<br /> "_type": "_doc",<br /> "_id": "RUS",<br /> "_score": null,<br /> "_source": {<br /> "name": "Russia",<br /> "location": [<br /> 100,<br /> 60<br /> ]<br /> },<br /> "sort": [<br /> 2803.2803033336695<br /> ]<br /> },<br /> {<br /> "_index": "regions",<br /> "_type": "_doc",<br /> "_id": "JPN",<br /> "_score": null,<br /> "_source": {<br /> "name": "Japan",<br /> "location": [<br /> 138,<br /> 36<br /> ]<br /> },<br /> "sort": [<br /> 2975.112941971521<br /> ]<br /> },<br /> {<br /> "_index": "regions",<br /> "_type": "_doc",<br /> "_id": "PHL",<br /> "_score": null,<br /> "_source": {<br /> "name": "Philippines",<br /> "location": [<br /> 122,<br /> 13<br /> ]<br /> },<br /> "sort": [<br /> 2983.948851295192<br /> ]<br /> }<br /> ]<br /> }<br />}<br /></pre><p>When you search by distance, Elasticsearch won’t return the distance itself as it isn’t a real field. If you need sorting, Elasticsearch advise to use the <i>sort</i> function to return the value.<br /></p><p><br /></p><p><br /></p>zhentaohttp://www.blogger.com/profile/05719795252682912081noreply@blogger.com0tag:blogger.com,1999:blog-6816360065241545754.post-81686070895029076752016-02-21T23:17:00.000-08:002016-02-21T23:21:06.034-08:00Spring with Spark Framework for Microservices<span id="goog_655204999"></span><a href="https://www.blogger.com/"></a><span id="goog_655205000"></span>Recently, my team decided to use <a href="http://sparkjava.com/" target="_blank">Spark</a> to build our new REST api. Spark seems easy with less coding. We still wanted to use <a href="http://spring.io/" target="_blank">Spring</a><span id="goog_655204997"> for dependency injection. We also wanted to use <a href="http://tomcat.apache.org/" target="_blank">Tomcat</a></span><span id="goog_655204997"> as the standalone web container. After some research, we figured out how to build the REST api with Spring, Spark and Tomcat.</span><br />
<ol>
<li>Create class SpringSparkFilter which extends <a href="https://github.com/perwendel/spark/blob/master/src/main/java/spark/servlet/SparkFilter.java" target="_blank">SparkFilter</a>. Override method getApplication:
<script src="https://gist.github.com/zhentao/20a7738ea6d4659d8cdc.js"></script>
</li>
<li>
Create class MySparkApplication, which implements <a href="https://github.com/perwendel/spark/blob/master/src/main/java/spark/servlet/SparkApplication.java" target="_blank">SparkApplication</a>. Implement init method:
<script src="https://gist.github.com/zhentao/a572d4591804665ffbf6.js"></script>
</li>
<li>
Configure web.xml to use SpringSparkFilter:
<script src="https://gist.github.com/zhentao/25f31c234a195d24a6af.js"></script>
</li>
</ol>
Then either build a war file, and deploy it to tomcat or use <a href="https://tomcat.apache.org/maven-plugin-2.2/tomcat7-maven-plugin/" target="_blank">tomcat7-maven-plugin</a> for test run. I usually use tomcat7-maven-plugin for fast development. Here is the command:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">mvn tomcat7:run</span><br />
<br />
Then access http://localhost:8080/current-date. The complete example can be found <a href="https://github.com/zhentao/spring-sparkjava-example" target="_blank">here</a>.zhentaohttp://www.blogger.com/profile/05719795252682912081noreply@blogger.com0tag:blogger.com,1999:blog-6816360065241545754.post-42617047761932249482014-09-18T14:46:00.001-07:002014-09-24T16:55:18.642-07:00HBase, Thrift2 and Erlang on MacThere are not to many documents showing how Erlang can connect to HBase using Thrift protocol. The very useful <a href="http://kungfooguru.wordpress.com/2009/08/31/erlang-thrift-and-hbase/" target="_blank">link</a> which was published more than 5 years ago, and it discussed Thrift1. Based on Lars George's <a href="http://search-hadoop.com/m/O9OjiuXFJQ1/hbase+thrift2+lars&subj=Thrift+2+Update" target="_blank">post</a>, Thrift2 will replace Thrift1 eventually. The following will show what I have done to get Erlang to work with HBase using Thrift on a Mac machine. It should be similar to get it work on a Linux machine. Note that only Java is supported as first class citizen for HBase. If you have the choice, I suggest using Java directly. <br />
<ol>
<li>Download <a href="http://archive.apache.org/dist/thrift/0.9.0/thrift-0.9.0.tar.gz" target="_blank">thrift-0.9.0 </a>and follow this <a href="http://thrift.apache.org/tutorial/" target="_blank">tutorial</a> to install it. <br />
<span style="font-family: "Courier New",Courier,monospace;">cp -r thrift-0.9.0/lib/erl/* /usr/local/opt/erlang/lib/erlang/lib/thrift-0.9.0 </span><br />
(homebrew installed <span style="font-family: "Courier New",Courier,monospace;">erlang</span> to <span style="font-family: "Courier New",Courier,monospace;">/usr/local/opt/erlang</span>. Find out your own erlang directory.)<br />
</li>
<li>Download latest stable <a href="http://www.apache.org/dyn/closer.cgi/hbase/" target="_blank">hbase </a>and decompress it. I installed it to <span style="font-family: "Courier New",Courier,monospace;">/usr/local/hbase-0.98.6.1-hadoop2</span>, and will refer it as <span style="font-family: "Courier New",Courier,monospace;">$HBASE_FOLDER</span>.</li>
<li>Download <a href="https://git-wip-us.apache.org/repos/asf?p=hbase.git;a=blob;f=hbase-thrift/src/main/resources/org/apache/hadoop/hbase/thrift2/hbase.thrift" target="_blank">hbase.thrift</a> and generate erlang bindings. <br />
<span style="font-family: "Courier New",Courier,monospace;">thrift -gen erl hbase.thrift </span><br />
It generates erlang bindings in <span style="font-family: "Courier New",Courier,monospace;">gen-erl</span> folder.</li>
<li>start hbase server:<br />
before starting hbase server, update conf/hbase-site.xml<br />
<span style="font-family: "Courier New",Courier,monospace;"><configuration><br /> <property><br /> <name>hbase.rootdir</name><br /> <value>file:///some-folder-hbase</value><br /> </property><br /> <property><br /> <name>hbase.zookeeper.property.dataDir</name><br /> <value>/some-folder-zookeeper</value><br /> </property><br /></configuration></span><br />
Then start hbase server:<br />
<span style="font-family: "Courier New",Courier,monospace;">$HBASE_FOLDER/bin/start-hbase.sh </span>
<br />
</li>
<li>start Thrift2:
<br />
<span style="font-family: "Courier New",Courier,monospace;">$HBASE_FOLDER/bin/base-daemon.sh start thrift2</span></li>
<li>Create table 'ups' with column family 'ui' using HBase Shell (Thrift2 doesn't support create table yet):<br />
<span style="font-family: "Courier New",Courier,monospace;">$HBASE_FOLDER/bin/hbase shell</span><br />
<span style="font-family: "Courier New",Courier,monospace;">hbase(main):001:0>; create 'ups', 'ui'</span></li>
<li><span style="font-family: "Courier New",Courier,monospace;"><span style="font-family: inherit;">Download <a href="https://github.com/zhentao/erlang-hbase-thrift2/blob/master/src/hbase_thrift2_client.erl" target="_blank">erlang client file</a> and save it to <span style="font-family: "Courier New",Courier,monospace;">gen-erl</span> folder. </span></span></li>
<li>Go to <span style="font-family: "Courier New",Courier,monospace;">gen-erl</span> folder, compile erlang files and start erlang shell:<br />
<span style="font-family: "Courier New",Courier,monospace;">cd gen-erl</span><br />
<span style="font-family: "Courier New",Courier,monospace;">erlc *.erl</span><br />
<span style="font-family: "Courier New",Courier,monospace;">erl </span><br />
<span style="font-family: "Courier New",Courier,monospace;">1>hbase_thrift2_client:put("localhost", 9090).</span><br />
<span style="font-family: "Courier New",Courier,monospace;">{{tclient,tHBaseService_thrift,<br /> {protocol,thrift_binary_protocol,<br /> {binary_protocol,{transport,thrift_buffered_transport,<br /> {buffered_transport,{transport,thrift_socket_transport,<br /> {data,#Port<0 data-blogger-escaped-.716="">,infinity}},<br /> []}},<br /> true,true}},<br /> 0},<br /> {ok,ok}}<!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--></span><br />
<span style="font-family: "Courier New",Courier,monospace;">2>hbase_thrift2_client:get("localhost", 9090).<br />{{tclient,tHBaseService_thrift,<br /> {protocol,thrift_binary_protocol,<br /> {binary_protocol,{transport,thrift_buffered_transport,<br /> {buffered_transport,{transport,thrift_socket_transport,<br /> {data,#Port<0 data-blogger-escaped-.717="">,infinity}},<br /> []}},<br /> true,true}},<br /> 0},<br /> {ok,{tResult,<<"zhentao-key1">>,<br /> [{tColumnValue,<<"ui">>,<<"test">>,<<"xyz1">>,1411075681134,<br /> undefined},<br /> {tColumnValue,<<"ui">>,<<"test1">>,<<"abc1">>,1411075681134,<br /> undefined}]}}}<!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--><!--0--></span></li>
</ol>
The detailed erlang project can be found <a href="https://github.com/zhentao/erlang-hbase-thrift2" target="_blank">here</a>
zhentaohttp://www.blogger.com/profile/05719795252682912081noreply@blogger.com0tag:blogger.com,1999:blog-6816360065241545754.post-63478960137219080522013-08-30T16:51:00.000-07:002013-08-30T16:52:13.502-07:00Spring AMQP and RabbitMQ High Availability (HA)Recently, I finished a project to use RabbitMQ as the message broker. It is pretty easy to set up a RabbitMQ cluster with High Availability (HA). The RabbitMQ has the pretty good documentation on how to set up the cluster. I also document it <a href="https://github.com/zhentao/rabbit-ha-example" target="_blank">here</a> on how to set up a 2-node cluster on a single Mac machine. In this post, I would like to show how we can leverage spring-amqp to connect to RabbitMQ cluster. I have a rabbit cluster running with 2 nodes: <i>localhost:5673 and localhost:5672</i>.<br />
<br />
It is pretty easy to use spring-amqp. I used maven to manage the dependencies:<br />
<br />
<script src="https://gist.github.com/zhentao/6395202.js"></script><br />
jackson-jaxrs-json-provider is used to serialize java object to json, and deserialize json back to java object.<br />
<br />
When creating ConnectionFactory, the addresses should be used instead of the host and port:<br />
<br />
<script src="https://gist.github.com/zhentao/6395219.js"></script><br />
The addresses are the comma separated host:port pairs which consist of the cluster.<br />
<br />
For producer, we use rabbitTemplate to send messages:<br />
<br />
<script src="https://gist.github.com/zhentao/6395323.js"></script><br />
<br />
For consumer, a MessageListenerContainer is created to consume message asynchronously:<br />
<br />
<script src="https://gist.github.com/zhentao/6395245.js"></script><br />
<br />
The MessageHandler code is as follows:<br />
<br />
<script src="https://gist.github.com/zhentao/6395259.js"></script><br />
This class can be called anything you like, but the method must be called <i>handleMessaege</i> and with the correct signature (here it is Employee to match producer). If you want to change the method name, you have to call:<br />
<br />
<i>MessageListenerAdapter.setDefaultListenerMethod</i><br />
<br />
The source code can be download from <a href="https://github.com/zhentao/rabbit-ha-example" target="_blank">github</a><i>.</i><br />
<br />
<br />zhentaohttp://www.blogger.com/profile/05719795252682912081noreply@blogger.com0tag:blogger.com,1999:blog-6816360065241545754.post-78482107612647796682013-06-11T10:52:00.000-07:002013-06-11T10:52:13.375-07:00Skip cobertura coverage check when skip unit testAfter I introduced <a href="http://zhentao-li.blogspot.com/2013/06/using-embedded-mysql-database-for-unit.html" target="_blank">embedded mysql</a><a href="http://zhentao-li.blogspot.com/2013/06/using-embedded-mysql-database-for-unit.html" target="_blank"> for unit testing </a>to my project, the build is taking longer now. For my local development, sometimes I want to disable the unit test:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"> mvn clean package -DskipTests</span><br />
<br />
Note that you can even skip the compiling of test class by using<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"> mvn clean package -Dmaven.test.skip</span><br />
<br />
However, if you use cobertura for code coverage check, you would get the following error:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"><i>[ERROR] Project failed check. Total branch coverage rate of 0.0% is below 85.0%<br />Project failed check. Total line coverage rate of 0.0% is below 85.0%</i></span><br />
<br />
So you need to skip cobertura as well. if you google "skip cobertura", most answers suggested using <i>haltOnFailure</i>. However, cobertura already provides the skip functionality:<br />
<br />
<i> </i><span style="font-family: "Courier New",Courier,monospace;">mvn clean package -DskipTests -Dcobertura.skip</span><br />
<br />
With <span style="font-family: "Courier New",Courier,monospace;">-Dcobertura.skip</span><i>, </i>cobertura check will be skipped:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">[INFO] --- cobertura-maven-plugin:2.5.2:instrument (default) @ tag-service ---<br />[INFO] Skipping cobertura execution</span>zhentaohttp://www.blogger.com/profile/05719795252682912081noreply@blogger.com3tag:blogger.com,1999:blog-6816360065241545754.post-51686403376205999722013-06-10T10:04:00.000-07:002013-06-10T10:09:33.503-07:00Using embedded mysql database for unit test with maven and springI used H2 in memory database to unit test my DAO. However, my production database is MySQL.<br />
Generally, H2 is compatible with MySQL. However, H2 doesn't support ENUM type yet, and my schema uses ENUM for a column definition. Inspired by this <a href="http://rajandesai.com/blog/2011/10/13/how-do-you-unit-test-your-data-access-layer/" target="_blank">post</a> and <a href="http://blog.palominolabs.com/2011/10/03/embedded-mysql-on-java-with-connectormxj-and-64-bit-linux/" target="_blank">another</a>, I successfully used embedded MySQL for unit test. Here is what I did:<br />
<br />
1. Add mysql-connector-mxj dependency:<br />
<br />
<script src="https://gist.github.com/zhentao/5746725.js"></script><br />
2. Create EmbeddedMysqlDatabase.java and EmbeddedMysqlDatabaseBuilder.java<br />
<script src="https://gist.github.com/zhentao/5750106.js"></script>
<script src="https://gist.github.com/zhentao/5750111.js"></script><br />
3. Create spring bean datasource:
<script src="https://gist.github.com/zhentao/5750132.js"></script><br />
4. Run unit test with Spring:
<script src="https://gist.github.com/zhentao/5750179.js"></script><br />
A couple things are worth mentioning:<br />
<br />
1. The test uses Spring annotation <span style="font-family: "Courier New",Courier,monospace;">@DirtiesContext(classMode = ClassMode.AFTER_EACH_TEST_METHOD) </span>which reload the context after every test method. It means every test method starts a new mysql instance. You may group test methods to 2 different categories, one changes data, and another not. For those methods don't change data, only need to create one instance of mysql to speed up testing.<br />
<br />
2. If you see the following error message on Linux server:<br />
<span style="font-family: "Courier New",Courier,monospace;">/lib64/libc.so.6: version `GLIBC_2.7' not found (required by /tmp/test_db_2420035739165052/bin/mysqld)</span><br />
<br />
You need to either downgrade mysql-connector-mxj to 5.0.11 or upgrade your linux OS. I got the above error message on CentOS 5.4 with 5.0.12, but no problem with 5.0.11. However, 5.0.11 isn't in any public maven repo, but you can download it from <a href="http://downloads.mysql.com/archives/mysql-connector-mxj-5.0/mysql-connector-mxj-gpl-5-0-11.tar.gz" target="_blank">here</a> and install it to your local repo, or upload to your company's repo. Using the following command to check CentOS version:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">cat /etc/redhat-release</span><br />
<br />
3. The code is tested on Mac and CentOS only, and can be downloaded from <a href="https://github.com/zhentao/maven-embedded-mysql" target="_blank">github</a>zhentaohttp://www.blogger.com/profile/05719795252682912081noreply@blogger.com11tag:blogger.com,1999:blog-6816360065241545754.post-29876442327232265632013-06-03T00:22:00.000-07:002013-06-04T09:28:28.225-07:00Example for enabling CORS support in Spring Rest Api 3.2My last year's post "<a href="http://zhentao-li.blogspot.com/2012/06/enable-cors-support-in-rest-services.html">Enable CORS support in REST services with Spring 3.1</a>" seems causing some confusion. I decided to create an example to show how to enable CORS with Spring rest api. The CorsFilter is same as before:<br />
<br />
<script src="https://gist.github.com/zhentao/5699508.js"></script><br />
Below are 2 endpoints from EmployeeController.java:<br />
<script src="https://gist.github.com/zhentao/5699544.js"></script><br />
The update method adds the header <span style="font-family: "Courier New",Courier,monospace;">Access-Control-Allow-Origin</span> with "*", but delete method doesn't. Therefore, the update method is enabled with CORS, but delete isn't. If delete endpoint is called, the following error will be shown in Chrome:<br />
<br />
"<span style="font-family: "Courier New",Courier,monospace;">cannot load http://localhost:8080/rest/employee/1. Origin http://127.0.0.1:8080 is not allowed by Access-Control-Allow-Origin.</span>"<br />
<br />
However, the delete method is still invoked on the server side since the pre-flight request (OPTIONS) <br />
allows DELETE method to be called.<br />
<br />
The entire project can be downloaded from <a href="https://github.com/zhentao/cors-spring-rest">github</a>. Following README to test it.<br />
<br />
My intention was to disable/enable CORS support in each individual method by setting "<span style="font-family: "Courier New",Courier,monospace;">Access-Control-Allow-Origin", </span>but it seems not working as expected: Although the browser returns correct info, the method call is still invoked on the server side even <span style="font-family: "Courier New",Courier,monospace;">Access-Control-Allow-Origin </span>is not set. If you are allowed to enable all endpoints with CORS support, the code can be simplified as below:<br />
<script src="https://gist.github.com/zhentao/5707286.js"></script><br />
The only difference is that addHeader("Access-Control-Allow-Origin") is moved out the if check. And then the update method can be simplified as:<br />
<script src="https://gist.github.com/zhentao/5707319.js"></script><br />
The code can be downloaded from <a href="https://github.com/zhentao/cors-spring-rest-simple" target="_blank">github</a> too.zhentaohttp://www.blogger.com/profile/05719795252682912081noreply@blogger.com5tag:blogger.com,1999:blog-6816360065241545754.post-89234169722772906162013-05-15T16:27:00.000-07:002013-05-24T14:54:09.975-07:00Add an existing project to a newly created git repositoryI recently created a project on my local machine using maven archetype. See <a href="http://zhentao-li.blogspot.com/2011/12/create-maven-archetype-from-existing.html" target="_blank">my previous post</a> on how to generate a new project from installed maven catalog. I then got github.com created a repository for this project, for example: git@github.com:zhentao/test1.git<br />
<br />
How can I associate my existing project with the newly created git repo? There are 2 ways to do it:<br />
<br />
<b>First option:</b><br />
<br />
1. clone the git repo:<br />
<span style="font-family: "Courier New",Courier,monospace;">git clone git@github.com:zhentao/test1.git</span><br />
<br />
It will clone the repo to <span style="font-family: "Courier New",Courier,monospace;">test1</span> folder.<br />
<br />
2. copy your existing project to <span style="font-family: "Courier New",Courier,monospace;">test1</span> folder<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">cp -r /path/to/project /path/to/test1</span><br />
<br />
3. commit and push all files<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">git add .</span><br />
<span style="font-family: "Courier New",Courier,monospace;">git commit -m "first commit"</span><br />
<span style="font-family: "Courier New",Courier,monospace;">git push -u origin master</span><br />
<br />
<b>Second option</b> (more git style)<br />
<br />
1. go to your project folder<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">cd /path/to/project</span><br />
<br />
2. run the following commands:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">git init<br />git add .<br />git commit -m "first commit"<br />git remote add origin git@github.com:zhentao/test1.git</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> <span style="font-family: inherit;"> </span></span><br />
<span style="font-family: inherit;">The above commands associate your project to git remote repo.</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">3. run the following commands to to merge the conflict:</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: "Courier New",Courier,monospace;">git fetch</span><br />
<span style="font-family: "Courier New",Courier,monospace;">git merge origin/master</span><br />
<span style="font-family: inherit;"> --run it if conflicts exist</span><br />
<span style="font-family: "Courier New",Courier,monospace;">git mergetool</span><br />
<span style="font-family: "Courier New",Courier,monospace;">git commit</span><br />
<span style="font-family: "Courier New",Courier,monospace;"><br /></span>
<span style="font-family: inherit;"><span style="font-family: "Courier New",Courier,monospace;">git pus</span>h -u origin master</span>zhentaohttp://www.blogger.com/profile/05719795252682912081noreply@blogger.com0tag:blogger.com,1999:blog-6816360065241545754.post-30861356908076021792013-04-09T16:21:00.000-07:002013-04-09T16:21:21.529-07:00Riak: configure memory_backendRiak supports several backends:<br />
1. bitcask<br />
2. memory<br />
3. leveldb<br />
4. multi<br />
<br />
My recent project required me to use memory backend. I have a test server with 24GB memory. I tried to use 4GB as the max_memory. It seems quite easy at beginning, and I just followed the <a href="http://docs.basho.com/riak/latest/tutorials/choosing-a-backend/Memory/#Enabling-and-Configuring-the-Memory-Backend" target="_blank">doc</a> from basho.com. I just replace <span style="font-family: "Courier New",Courier,monospace;">riak_kv_bitcask_backend</span> with <span style="font-family: "Courier New",Courier,monospace;">riak_kv_memory_backend</span>:<br />
<br />
<pre>{riak_kv, [</pre>
<pre> {storage_backend, riak_kv_memory_backend},</pre>
<pre> </pre>
<span style="font-family: Times,"Times New Roman",serif;">Then I want to see how much data I can load with default configuration. I loaded a little over 11 million records, then riak crashed. Then I set </span>max_memory to 4GB to <span style="font-family: Times,"Times New Roman",serif;">add the following right under</span> {eleveldb ... configuration:<br />
<br />
<pre>{memory_backend, [
...,
{max_memory, <span style="color: #0000dd;">4096</span>}, %% <span style="color: #0000dd;">4</span>GB <span style="color: #008800; font-weight: bold;">in</span> megabytes
...
]}</pre>
<pre> </pre>
<span style="font-family: Times,"Times New Roman",serif;">Note that the following is copied from riak's doc. Then interesting, riak crashed with same number of records (11M+). I then changed it to 8192, but riak still crashed with the same number of records. I spent a couple days, and couldn't figure out the reason. I went to #riak IRC and asked the following question, and those folks were very responsive and solved my problem right away:</span><br />
<span style="font-family: Times,"Times New Roman",serif;"><br /></span>
<br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;"><span class="timestamp">[10:30] </span><span><<span class="hyperlink-whois">zhentao</span>> hi folks, how to configure the max_memory for memory_backend?</span></span></blockquote>
<blockquote>
<div class="linestyle1 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span class="timestamp">[10:30] </span><span><@</span><span class="Xc4"><span class="hyperlink-whois">alexmoore</span></span><span>> Hi zhentao</span></span></div>
<div class="linestyle2 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span class="timestamp">[10:30] </span><span><<span class="hyperlink-whois">zhentao</span>> Hi</span></span></div>
<div class="linestyle1 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span class="timestamp">[10:30] </span><span><<span class="hyperlink-whois">zhentao</span>> I added the following to config file:</span></span></div>
<div class="linestyle2 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span class="timestamp">[10:30] </span><span><<span class="hyperlink-whois">zhentao</span>> {memory_backend, [<span> </span><span> </span><span> </span><span> </span><span> </span><span> </span><span> </span>{max_memory, 8192} %% 8GB in megabytes<span> </span><span> </span><span> </span><span> </span><span> </span><span> </span> ]},</span></span></div>
<div class="linestyle1 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span class="timestamp">[10:31] </span><span><<span class="hyperlink-whois">zhentao</span>> but it didn't work</span></span></div>
<div class="linestyle2 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span class="timestamp">[10:31] </span><span><@<span class="hyperlink-whois">alexmoore</span>>
In regards to your earlier question, if you don't specify a max_memory,
or a TTL, the memory backend will continue to grow until it runs out of
memory.</span></span></div>
<div class="linestyle1 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span class="timestamp">[10:31] </span><span><@<span class="hyperlink-whois">alexmoore</span>> Let me look at your config here</span></span></div>
<div class="linestyle2 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span class="timestamp">[10:31] </span><span><<span class="hyperlink-whois">zhentao</span>> it seems like that</span></span></div>
<div class="linestyle1 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span class="timestamp">[10:32] </span><span><<span class="hyperlink-whois">zhentao</span>> the node crashed after I loaded some data</span></span></div>
<div class="linestyle2 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span class="timestamp">[10:32] </span><span><<span class="hyperlink-whois">zhentao</span>> then I specify the max_memory to 8gb, and it still crashed with same amount of data</span></span></div>
<div class="linestyle1 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span class="timestamp">[10:32] </span><span><@<span class="hyperlink-whois">alexmoore</span>> How much RAM does the machine have?</span></span></div>
<div class="linestyle2 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span class="timestamp">[10:33] </span><span><<span class="hyperlink-whois">sully_</span>> Cluster health seems like it's degrading again.<span> </span>We're starting to see the same errors as before.</span></span></div>
<div class="linestyle1 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span class="timestamp">[10:34] </span><span><<span class="hyperlink-whois">zhentao</span>> 24 GB</span></span></div>
<div class="linestyle2 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span class="timestamp">[10:34] </span><span><<span class="hyperlink-whois">sully_</span>> We are planning to add 3 more nodes.</span></span></div>
<div class="linestyle1 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span class="timestamp">[10:34] </span><span><@</span><span class="Xc4"><span class="hyperlink-whois">evanmcc</span></span><span>> zhentao: that's 8GB per vnode</span></span></div>
<div class="linestyle2 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span class="timestamp">[10:34] </span><span><@<span class="hyperlink-whois">rzezeski</span>>
sully_ ok, can you tar.gz the latest log files again, that might allow
me to find the cause before it gets rotated out by the logger</span></span></div>
<div class="linestyle1 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span class="timestamp">[10:34] </span><span><@</span><span class="Xc4"><span class="hyperlink-whois">jcaprice</span></span><span>> zhentao: did you restart the node after adding the memory constraint?</span></span></div>
<div class="linestyle2 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span class="timestamp">[10:35] </span><span><<span class="hyperlink-whois">sully_</span>> Getting you the latest logs.</span></span></div>
<div class="linestyle1 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span class="timestamp">[10:35] </span><span><<span class="hyperlink-whois">zhentao</span>> yes, I restarted it</span></span></div>
<div class="linestyle2 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span class="timestamp">[10:35] </span><span><@</span><span class="Xc4"><span class="hyperlink-whois">evanmcc</span></span><span>> zhentao: how many nodes, and what is your ring size?</span></span></div>
<div class="linestyle1 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span class="timestamp">[10:36] </span><span><@</span><span class="Xc4"><span class="hyperlink-whois">alexmoore</span></span><span>> zhentao, how many physical machines do you have in your cluster, and what is your ring size?</span></span></div>
<div class="linestyle2 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span class="timestamp">[10:36] </span><span><<span class="hyperlink-whois">zhentao</span>> it is a test server and just one machine</span></span></div>
<div class="linestyle1 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span class="timestamp">[10:36] </span><span><@<span class="hyperlink-whois">jcaprice</span>> ring size?</span></span></div>
<div class="linestyle2 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span class="timestamp">[10:37] </span><span><<span class="hyperlink-whois">zhentao</span>> i am new to riak, and where to find th ring size?</span></span></div>
<div class="linestyle1 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span class="timestamp">[10:37] </span><span><@<span class="hyperlink-whois">evanmcc</span>> if you didn't set it, it's 64</span></span></div>
<div class="linestyle2 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span class="timestamp">[10:37] </span><span><@<span class="hyperlink-whois">alexmoore</span>> In your vm.args file</span></span></div>
<div class="linestyle1 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span class="timestamp">[10:37] </span><span><@<span class="hyperlink-whois">evanmcc</span>> so you're limiting memory to 8GB * 64</span></span></div>
<div class="linestyle2 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span class="timestamp">[10:38] </span><span><<span class="hyperlink-whois">zhentao</span>> let me check </span></span></div>
<div class="linestyle1 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span class="timestamp">[10:38] </span><span><@<span class="hyperlink-whois">evanmcc</span>> alexmoore: it's in app.config under riak_core</span></span></div>
<div class="linestyle2 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span class="timestamp">[10:38] </span><span><@<span class="hyperlink-whois">evanmcc</span>> ring_creation_size</span></span></div>
<div class="linestyle1 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span class="timestamp">[10:38] </span><span><@<span class="hyperlink-whois">alexmoore</span>> Whoops, make that the app.config</span></span></div>
<div class="linestyle2 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span class="timestamp">[10:39] </span><span><<span class="hyperlink-whois">zhentao</span>> it is 64</span></span></div>
<div class="linestyle1 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span class="timestamp">[10:39] </span><span><<span class="hyperlink-whois">zhentao</span>> {ring_creation_size, 64},</span></span></div>
<div class="linestyle2 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span class="timestamp">[10:39] </span><span><@<span class="hyperlink-whois">evanmcc</span>> so you want to change max memory to 128</span></span></div>
<div class="linestyle1 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span class="timestamp">[10:39] </span><span><@<span class="hyperlink-whois">evanmcc</span>> if you want to limit it at 8GB</span></span></div>
<div class="linestyle2 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span class="timestamp">[10:40] </span><span><@</span><span class="Xc4"><span class="hyperlink-whois">jcaprice</span></span><span>> zhentao: max_memory limits the amount of memory used per vnode, not for the node itself</span></span></div>
<div class="linestyle1 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span class="timestamp">[10:40] </span><span><<span class="hyperlink-whois">zhentao</span>> @jcaprice, so what number I should use for max_memory for my test server?</span></span></div>
<div class="linestyle2 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span class="timestamp">[10:41] </span><span><@<span class="hyperlink-whois">evanmcc</span>> 128, like I said above</span></span></div>
<div class="linestyle1 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span class="timestamp">[10:41] </span><span><<span class="hyperlink-whois">zhentao</span>> 128 mb?</span></span></div>
<div class="linestyle2 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span class="timestamp">[10:41] </span><span><@<span class="hyperlink-whois">evanmcc</span>> yes</span></span></div>
<div class="linestyle1 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span class="timestamp">[10:41] </span><span><@<span class="hyperlink-whois">jcaprice</span>> as evanmcc said, you'll want 8192 / 64</span></span></div>
<div class="linestyle2 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span class="timestamp">[10:41] </span><span><<span class="hyperlink-whois">zhentao</span>> thx, let me try it</span></span></div>
</blockquote>
<div class="linestyle2 colourline">
<span> </span></div>
<div class="linestyle2 colourline">
<span>So in summary, </span><span style="font-family: "Courier New",Courier,monospace;">{max_memory, <span style="color: #0000dd;">4096</span>} <span style="font-family: Times,"Times New Roman",serif;">is for each <span style="color: red;">vnode</span>, not for each machine. Since I wanted to limit the memory usage for one machine to 4GB, I should use the following:</span></span></div>
<div class="linestyle2 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span style="font-family: Times,"Times New Roman",serif;"> </span></span></div>
<div class="linestyle2 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span style="font-family: Times,"Times New Roman",serif;"><span style="font-family: "Courier New",Courier,monospace;">{max_memory, <span style="color: #0000dd;">64</span>}</span></span></span></div>
<div class="linestyle2 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span style="font-family: Times,"Times New Roman",serif;"><span style="font-family: "Courier New",Courier,monospace;"> </span></span></span></div>
<div class="linestyle2 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span style="font-family: Times,"Times New Roman",serif;">The reason is that the ring_creattion_size is 64 which means there are 64 vnodes on my single test server. </span></span></div>
<div class="linestyle2 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span style="font-family: Times,"Times New Roman",serif;"> </span></span></div>
<div class="linestyle2 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span style="font-family: Times,"Times New Roman",serif;">64MB * 64 = 4096MB</span></span></div>
<div class="linestyle2 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span style="font-family: Times,"Times New Roman",serif;"> </span></span></div>
<div class="linestyle2 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span style="font-family: Times,"Times New Roman",serif;">After I changed max_memory to 64, riak is happy, and it didn't crash when I tried to load 20MM records.</span></span></div>
<div class="linestyle2 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span style="font-family: Times,"Times New Roman",serif;"> </span></span></div>
<div class="linestyle2 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span style="font-family: Times,"Times New Roman",serif;">Some interesting things I noticed:</span></span></div>
<div class="linestyle2 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span style="font-family: Times,"Times New Roman",serif;"> </span></span></div>
<div class="linestyle2 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span style="font-family: Times,"Times New Roman",serif;">1. since Riak use LRU for memory_backend, the old records are evicted if max_memory can't hold all records.</span></span></div>
<div class="linestyle2 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span style="font-family: Times,"Times New Roman",serif;"> </span></span></div>
<div class="linestyle2 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span style="font-family: Times,"Times New Roman",serif;">2. With the key as "70f21766-1cde-38d4-5920-a380003723b3", and value as "</span></span>1365720608095#TQB:1.4:2:1", </div>
<div class="linestyle2 colourline">
</div>
<div class="linestyle2 colourline">
4GB memory with 64 vnodes can hold about 2.5MM records</div>
<div class="linestyle2 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span style="font-family: Times,"Times New Roman",serif;">4GB memory with 4 vnodes can hold about 7MM records</span></span></div>
<div class="linestyle2 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span style="font-family: Times,"Times New Roman",serif;"> </span></span></div>
<div class="linestyle2 colourline">
<span style="font-family: "Courier New",Courier,monospace;"><span style="font-family: Times,"Times New Roman",serif;">Seems the number of vnodes/machine has big overhead </span></span></div>
<br />
<br />
<pre> </pre>
<pre> </pre>
zhentaohttp://www.blogger.com/profile/05719795252682912081noreply@blogger.com0tag:blogger.com,1999:blog-6816360065241545754.post-53058256129873121612013-02-14T15:30:00.000-08:002013-02-14T15:30:13.613-08:00eclipse JUNO hangs on start upI use Eclipse JUNO every day. However, I couldn't start it today after I restart my Mac. Mac updated with Microsoft 2011 so I have to reboot Mac machine. The strange thing happened: Eclipse got stuck. I manually killed the start up process several times, but it didn't work. Then I googled it, and found the solution from this <a href="http://stackoverflow.com/questions/207843/how-do-i-prevent-eclipse-from-hanging-on-startup" target="_blank">post</a>, and here is what I did:<br />
<br />
<pre class="lang-java prettyprint prettyprinted"><code><span class="pln">rm $WORKSPACE_DIR</span><span class="pun">/.</span><span class="pln">metadata</span><span class="pun">/.</span><span class="pln">plugins</span><span class="pun">/</span><span class="pln">org</span><span class="pun">.</span><span class="pln">eclipse</span><span class="pun">.</span><span class="pln">e4</span><span class="pun">.</span><span class="pln">workbench</span><span class="pun">/</span><span class="pln">workbench</span><span class="pun">.</span><span class="pln">xmi</span></code></pre>
<pre class="lang-java prettyprint prettyprinted"><code><span class="pln"> </span></code></pre>
<pre class="lang-java prettyprint prettyprinted"><code><span class="pln">Then I can start eclipse, and happy coding. </span></code></pre>
zhentaohttp://www.blogger.com/profile/05719795252682912081noreply@blogger.com0tag:blogger.com,1999:blog-6816360065241545754.post-88780526841504265712012-12-07T15:41:00.001-08:002013-11-01T11:50:29.860-07:00my shell script cheat sheet<b>1. find the total line number of a fold for some files:</b><br />
<br />
find test -iname *.java | xargs wc -l | tail -1<br />
<br />
<b>2. count # of files in each folder recursively</b><br /><br />for t in `find . -type d -ls | awk '{print $11}'`; do echo "$t `find $t -type f | wc -l`" ; done > ~/count.txtzhentaohttp://www.blogger.com/profile/05719795252682912081noreply@blogger.com0tag:blogger.com,1999:blog-6816360065241545754.post-88706282647094002952012-12-07T15:38:00.001-08:002012-12-07T15:38:36.621-08:00insert control character with viinsert a
visible <code>^A</code>, we must use: <code><ctrl> + v + a</code><code></code>.zhentaohttp://www.blogger.com/profile/05719795252682912081noreply@blogger.com0tag:blogger.com,1999:blog-6816360065241545754.post-59469765386722402842012-11-01T11:06:00.001-07:002012-11-01T11:06:41.667-07:00Some svn commands<pre class="screen"><b>Create tag from trunk or branch</b>:</pre>
<blockquote class="tr_bq">
<pre class="screen">svn copy http://svn.mydomain.com/repository/myproject/trunk \
http://svn.mydomain.com/repository/myproject/tags/release-1.0 \
-m "Tagging the 1.0 release." </pre>
</blockquote>
<pre class="screen"><span style="font-family: Arial,Helvetica,sans-serif;">The tag created is the snapshot of the trunk at the time the "svn copy" is executed. </span></pre>
<pre class="screen"><span style="font-family: Arial,Helvetica,sans-serif;">To be more precisely, the revision # can be passed to "svn copy":</span></pre>
<blockquote class="tr_bq">
<pre class="screen">svn copy -r 12345 http://svn.mydomain.com/repository/myproject/trunk \
http://svn.mydomain.com/repository/myproject/tags/release-1.0 \
-m "Tagging the 1.0 release."</pre>
</blockquote>
<pre class="screen"><b>Merge from Trunk to a branch</b>:</pre>
<pre class="screen">
</pre>
<pre class="screen"><span style="font-family: Arial,Helvetica,sans-serif;">1. check out the branch (assume the path is <i>/path/to/mybranch</i>)</span></pre>
<pre class="screen"><span style="font-family: Arial,Helvetica,sans-serif;">2. go to above folder</span></pre>
<pre class="screen"><span style="font-family: Arial,Helvetica,sans-serif;">3. run the following commmand</span>:</pre>
<blockquote class="tr_bq">
<pre class="screen">svn merge http://svn.mydomain.com/repository/myproject/trunk .</pre>
</blockquote>
<pre class="screen"><span style="font-family: Arial,Helvetica,sans-serif;">It will merge all changes from trunk to mybranch since last merge. </span></pre>
<pre class="screen"><span style="font-family: Arial,Helvetica,sans-serif;">The above command is same as:</span></pre>
<blockquote class="tr_bq">
<pre class="screen">svn merge -rLastMergedRevision:HEAD http://svn.mydomain.com/repository/myproject/trunk .</pre>
</blockquote>
<pre class="screen"><b>View merge history:</b></pre>
<pre class="screen"><span style="font-family: Arial,Helvetica,sans-serif;">"svn log" doesn't display the merge history. It only shows the merge commit:</span></pre>
<blockquote class="tr_bq">
<pre class="screen">svn log</pre>
<pre class="screen">------------------------------------------------------------------------
r196402 | liz | 2012-11-01 10:39:13 -0700 (Thu, 01 Nov 2012) | 1 line
Merging r196340 through r196401
------------------------------------------------------------------------
r196340 | liz | 2012-10-31 14:52:06 -0700 (Wed, 31 Oct 2012) | 1 line
development branch for new feature</pre>
</blockquote>
<pre class="screen"><span style="font-family: Arial,Helvetica,sans-serif;">If need to see the merge history, the option <code>--use-merge-history (-g) can be used with svn log:</code></span></pre>
<blockquote class="tr_bq">
<pre class="screen"><code>svn log -g</code></pre>
<pre class="screen"><code>------------------------------------------------------------------------
r196402 | liz | 2012-11-01 10:39:13 -0700 (Thu, 01 Nov 2012) | 1 line
Merging r196340 through r196401
------------------------------------------------------------------------
r196388 | xyz | 2012-11-01 09:50:28 -0700 (Thu, 01 Nov 2012) | 2 lines
Merged via: r196402
Added new unit tests
------------------------------------------------------------------------
r196340 | liz | 2012-10-31 14:52:06 -0700 (Wed, 31 Oct 2012) | 1 line
development branch for new feature</code></pre>
</blockquote>
<pre class="screen"> </pre>
<pre class="screen"> <b> </b></pre>
zhentaohttp://www.blogger.com/profile/05719795252682912081noreply@blogger.com0tag:blogger.com,1999:blog-6816360065241545754.post-8040305849233696532012-07-17T15:08:00.000-07:002012-10-15T00:28:58.142-07:00Run Sqoop on Amazon Elastic MapReduce (EMR) with Amazon RDSAmazon EMR doesn't have Sqoop installed. It is possible to run Sqoop with Amazon EMR. The following blog shows how to install and run Sqoop:<br />
<br />
<a href="http://blog.kylemulka.com/2012/04/how-to-install-sqoop-on-amazon-elastic-map-reduce-emr/">http://blog.kylemulka.com/2012/04/how-to-install-sqoop-on-amazon-elastic-map-reduce-emr/</a><br />
<br />
However, the solution isn't perfect since the input files are usually in S3 and Sqoop doesn't support S3 directly. Here is my script to install sqoop and export data from S3 to Amazon RDS (mysql):<br />
<br />
<blockquote class="tr_bq">
<i><span style="font-size: x-small;">#!/bin/bash</span></i><br />
<i><span style="font-size: x-small;">BUCKET_NAME=zli-emr-test</span></i><br />
<i><span style="font-size: x-small;">SQOOP_FOLDER=sqoop-1.4.1-incubating__hadoop-0.20</span></i><br />
<i><span style="font-size: x-small;">SQOOP_TAR=$SQOOP_FOLDER.tar.gz</span></i><br />
<i><span style="font-size: x-small;"><br /></span></i>
<i><span style="font-size: x-small;">##change to home directory</span></i><br />
<i><span style="font-size: x-small;">cd ~</span></i><br />
<i><span style="font-size: x-small;"><br /></span></i>
<i><span style="font-size: x-small;">##Install sqoop on emr</span></i><br />
<i><span style="font-size: x-small;">hadoop fs -copyToLocal s3n://$BUCKET_NAME/$SQOOP_TAR $SQOOP_TAR</span></i><br />
<i><span style="font-size: x-small;">tar -xzf $SQOOP_TAR</span></i><br />
<i><span style="font-size: x-small;"><br /></span></i>
<i><span style="font-size: x-small;">##Install jdbc driver (ex mysql-connection-java.jar) to sqoop lib folder</span></i><br />
<i><span style="font-size: x-small;">hadoop fs -copyToLocal s3n://$BUCKET_NAME/mysql-connector-java-5.1.19.jar ~/$SQOOP_FOLDER/lib/</span></i> </blockquote>
<blockquote class="tr_bq">
<i><span style="font-size: x-small;">##Copy input file from S3 to hdf</span></i> </blockquote>
<blockquote class="tr_bq">
<i><span style="font-size: x-small;">HADOOP_INPUT=hdfs:///user/hadoop/myinput</span></i><br />
<i><span style="font-size: x-small;">hadoop distcp s3://$BUCKET_NAME/myinput $HADOOP_INPUT</span></i> </blockquote>
<blockquote class="tr_bq">
<i><span style="font-size: x-small;"></span></i>
<i><span style="font-size: x-small;">~/$SQOOP_FOLDER/bin/sqoop export --connect jdbc:mysql://RDS-Host-name:3306/DB_NAME --username USERNAME --password PASSWORD --table TABLE_NAME --export-dir $HADOOP_INPUT --input-fields-terminated-by='\t'</span></i></blockquote>
The script assumes that sqoop tar ball and mysql-connector-java.jar are in S3 bucket, as well as the input file are in S3 too. <br />
<br />
Note that, RDS needs to be configured to allow access to the database with the following 2 EC2 security groups:<br />
<blockquote class="tr_bq">
ElasticMapReduce-master<br />
ElasticMapReduce-slave</blockquote>
zhentaohttp://www.blogger.com/profile/05719795252682912081noreply@blogger.com2tag:blogger.com,1999:blog-6816360065241545754.post-50592967758351945042012-07-10T14:37:00.001-07:002012-07-10T15:26:13.774-07:00Run custom jar with elastic map reduce (EMR) on command lineI followed the instruction to run a custom jar on EMR:<br />
<br />
http://aws.amazon.com/articles/3938<br />
<br />
I got stuck with step 5:<br />
<blockquote class="tr_bq">
5. Run the job flow. <br />
<pre class="code"> $ ./elasticmapreduce-client.rb RunJobFlow streaming_jobflow.json </pre>
</blockquote>
I couldn't find file "elasticmapreduce-client.rb" at all. After some online searches, I got it work. The correct command is:<br />
<blockquote class="tr_bq">
./elastic-mapreduce --create --json <i>path/to/your/flow</i></blockquote>
Here is my flow file looks like:<br />
<blockquote class="tr_bq">
[ <br />
{ <br />
"Name": "Custom Jar Grep Example 1", <br />
"ActionOnFailure": "CONTINUE", <br />
"HadoopJarStep": <br />
{ <br />
"Jar": "s3n://YOUR_BUCKET/hadoop-examples-0.20.2-cdh3u4.jar",</blockquote>
##"MainClass": "fully-qualified-class-name", <br />
<blockquote class="tr_bq">
"Args": <br />
[ <br />
"grep", <br />
"s3n://YOUR_BUCKET/input/example", <br />
"s3n://YOUR_BUCKET/output/example",<br />
"dfs[a-z.]+" <br />
] <br />
} <br />
} <br />
]</blockquote>
The flow is corresponding to the following hadoop command:<br />
<blockquote class="tr_bq">
<i>hadoop jar hadoop-examples-0.20.2-cdh3u4.jar grep input output 'dfs[a-z.]+'</i></blockquote>
<br />
Some useful tips:<br />
<br />
1. show the log:<br />
<blockquote class="tr_bq">
./elastic-mapreduce --jobflow JOB_ID --logs </blockquote>zhentaohttp://www.blogger.com/profile/05719795252682912081noreply@blogger.com1tag:blogger.com,1999:blog-6816360065241545754.post-81696225853317017502012-06-29T13:25:00.000-07:002012-06-29T13:25:20.667-07:00mvn test: LocalJobRunner.java with java.lang.OutOfMemoryErrorWhen I run mvn test to unit test some hadoop related program, I got the following:<br />
<blockquote class="tr_bq">
<i>java.lang.OutOfMemoryError: Java heap space<br /> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:949)<br /> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:428)<br /> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)<br /> at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212)</i></blockquote>
I tried to set MAVEN_OPTS=-Xmx2048m, but it didn't work. After the online research, I fixed this problem by adding the following to pom:<br />
<br />
<plugin><br /> <groupId>org.apache.maven.plugins</groupId><br /> <artifactId>maven-surefire-plugin</artifactId><br /> <configuration><br /> <argLine>-Xms256m -Xmx512m</argLine><br /> </configuration><br /> </plugin>zhentaohttp://www.blogger.com/profile/05719795252682912081noreply@blogger.com0tag:blogger.com,1999:blog-6816360065241545754.post-45681754186796485162012-06-28T14:04:00.002-07:002012-11-21T11:33:20.240-08:00maven shade plugin: Invalid signature file digest for Manifest main attributesIf you get the following error message with maven shade plugin:<br />
<br />
<i>Exception in thread "main" java.lang.SecurityException: Invalid signature file digest for Manifest main attributes</i><br />
<br />
You need to add the following to pom.xml:<br />
<br />
<configuration><br />
<filters><br />
<filter><br />
<artifact>*:*</artifact><br />
<excludes><br />
<exclude>META-INF/*.SF</exclude><br />
<exclude>META-INF/*.DSA</exclude><br />
<exclude>META-INF/*.RSA</exclude><br />
</excludes><br />
</filter><br />
</filters><br />
</configuration><br />
<br />
<b>Explanation:</b><br />
<br />
The above configuration filters all files in META-INF ending with .SF, .DSA, and .RSA for all artifacts (*:*) when creating uber-jar file.<br />
<br />
The reason <i>java.lang.SecurityException</i> is raised is because some dependency jar files are signed jar files. A jar file is signed by using <i><a href="http://download.java.net/jdk8/docs/technotes/tools/solaris/jarsigner.html" target="_blank">jarsigner</a></i>, which creates 2 additional files and places them in META-INF:<br />
<ul>
<li>a signature file, with a .SF extension, and</li>
<li>a signature block file, with a .DSA, .RSA, or .EC
extension.</li>
</ul>
Since the uber-jar file is created, the signatures and integrity of signed JAR
files are no longer valid. When the uber-jar file is executed, <i>java.lang.SecurityException</i> is thrown<i>.</i><br />
<i><br /></i>
See <a href="http://download.java.net/jdk8/docs/technotes/tools/solaris/jarsigner.html" target="_blank"><i>jarsigner</i></a> for detailed explanation of <span style="font-size: small;"><span style="font-weight: normal;">JAR Signing and Verification Tool.</span></span>zhentaohttp://www.blogger.com/profile/05719795252682912081noreply@blogger.com17tag:blogger.com,1999:blog-6816360065241545754.post-16097372843572654902012-06-24T17:59:00.000-07:002012-06-24T20:06:31.818-07:00Eclipse: fix plugin execution not covered by lifecycle configurationI recently used maven-jaxb2-plugin to generate java classes based on xml schemas (xsd). I can run "mvn compile" on command line to generate java classes with the following configuration:<br />
<i><plugin><br /> <groupId>org.jvnet.jaxb2.maven2</groupId><br /> <artifactId>maven-jaxb2-plugin</artifactId><br /> <executions><br /> <execution><br /> <goals><br /> <goal>generate</goal><br /> </goals><br /> </execution><br /> </executions><br /> </plugin></i><br />
However, my Eclipse shows the following error:<br />
<i> </i><br />
<i>Plugin execution not covered by lifecycle configuration: org.jvnet.jaxb2.maven2:maven-jaxb2-plugin:0.8.1:generate (execution: default, phase: generate-sources)</i><br />
<i><br /></i><br />
This error prevents Eclipse generating java source codes and adding the generated sources to the class path. In order to fix this problem, I need to install m2e connector for maven-jaxb2-plugin. I can manually install it to Eclipse with the following update site:<br />
<br />
http://bitstrings.github.com/m2e-connectors-p2/milestones/<br />
<br />
However, Eclipse provides a better to install it. Eclipse shows the problem with the following:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-jr7WhFkXPTc/T-e1Sg0ta9I/AAAAAAAANJQ/_EZx0HWtehI/s1600/Screen+Shot+2012-06-24+at+5.46.39+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="114" src="http://2.bp.blogspot.com/-jr7WhFkXPTc/T-e1Sg0ta9I/AAAAAAAANJQ/_EZx0HWtehI/s320/Screen+Shot+2012-06-24+at+5.46.39+PM.png" width="320" /></a></div>
<br />
Roll the mouse over <execution>, it displays the following:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-hpG1RWJ-kBc/T-e1yqISnKI/AAAAAAAANJc/9F2if8PMCTs/s1600/Screen+Shot+2012-06-24+at+5.49.48+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="66" src="http://1.bp.blogspot.com/-hpG1RWJ-kBc/T-e1yqISnKI/AAAAAAAANJc/9F2if8PMCTs/s320/Screen+Shot+2012-06-24+at+5.49.48+PM.png" width="320" /></a></div>
Click the link "Discover new m2e connector", and eclipse will try to find the correct connector automatically:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-x2yEvPOQxQ8/T-e2eGnmgJI/AAAAAAAANJk/CE5BcTBf54k/s1600/Screen+Shot+2012-06-24+at+5.52.53+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="182" src="http://3.bp.blogspot.com/-x2yEvPOQxQ8/T-e2eGnmgJI/AAAAAAAANJk/CE5BcTBf54k/s320/Screen+Shot+2012-06-24+at+5.52.53+PM.png" width="320" /></a></div>
Eclipse found the same m2e connector. Click finish to install it. After I installed it successfully, Eclipse is happy with <i>maven-jaxb2-plugin.</i><br />
<br />
This approach can be applied to other plugins too.<i> </i>zhentaohttp://www.blogger.com/profile/05719795252682912081noreply@blogger.com0tag:blogger.com,1999:blog-6816360065241545754.post-65568294969393802422012-06-08T14:30:00.001-07:002013-06-04T09:52:25.665-07:00Enable CORS support in REST services with Spring 3.1We are working on a web project, which has 2 components, UI and web services. These 2 components are deployed to 2 different servers. UI uses jQuery to make ajax call to back end restful web services. Back end web services use Spring 3.1 MVC framework. We are facing the XSS (cross site scripting) issue since modern browsers don't allow XSS.<br />
<br />
The back end rest web services support HTTP methods GET, POST, PUT and DELETE. The first problem we solved was how to call HTTP method GET from UI using jQuery. Since I used JSONP before, it is a natural solution to me to use JSONP again. My coworker already blog the solution here: http://www.iceycake.com/2012/06/xml-json-jsonp-web-service-endpoints-spring-3-1.<br />
<br />
However, JSONP only supports GET, and you can't set http headers with JSONP request. I also realized that JSONP is kind of out-of-dated too. So how to support other methods like PUT, POST? <br />
<br />
CORS, <a href="http://www.w3.org/TR/cors/">Cross-Origin Resource Sharing</a>, defines a mechanism to enable client-side cross-origin requests. CORS are widely supported by modern browsers like FireFox, Chrome, Safari, and IE.<br />
<br />
The UI code is fairly simple:<br />
<blockquote class="tr_bq">
<html><br />
<head><br />
<script type="text/javascript" src="jquery.js"></script><br />
<script type="application/javascript"> <br />
(function($) {<br />
var url = 'http://localhost:8080/employee/id';<br />
$.ajax({<br />
type: 'put',<br />
url: url,<br />
async: true,<br />
contentType: 'application/json',<br />
data: '{"id": 1, "name": "John Doe"}',<br />
success: function(response) {<br />
alert("success");<br />
},<br />
error: function(xhr) {<br />
alert('Error! Status = ' + xhr.status + " Message = " + xhr.statusText);<br />
}<br />
});<br />
})(jQuery);<br />
</script><br />
</head><br />
<body><br />
<!-- we will add our HTML content here --><br />
</body><br />
</html></blockquote>
If you just run the above javascript and call the backend, you may notice that the PUT request is changed to OPTIONS. The browser sends an OPTIONS request to the server, and the server needs to send response with correct headers. So in order to return the correct response, a filter is created (thanks to the link https://gist.github.com/2232095):<br />
<blockquote class="tr_bq">
package com.zhentao; </blockquote>
<blockquote class="tr_bq">
import java.io.IOException;<br />
import javax.servlet.FilterChain;<br />
import javax.servlet.ServletException;<br />
import javax.servlet.http.HttpServletRequest;<br />
import javax.servlet.http.HttpServletResponse;<br />
import org.springframework.web.filter.OncePerRequestFilter;<br />
public class CorsFilter extends OncePerRequestFilter {<br />
<br />
@Override<br />
protected void doFilterInternal(HttpServletRequest request, HttpServletResponse response, FilterChain filterChain)<br />
throws ServletException, IOException {<br />
if (request.getHeader("Access-Control-Request-Method") != null && "OPTIONS".equals(request.getMethod())) {<br />
// CORS "pre-flight" request<br />
response.addHeader("Access-Control-Allow-Origin", "*");<br />
response.addHeader("Access-Control-Allow-Methods", "GET, POST, PUT, DELETE");<br />
response.addHeader("Access-Control-Allow-Headers", "Content-Type");<br />
response.addHeader("Access-Control-Max-Age", "1800");//30 min<br />
}<br />
filterChain.doFilter(request, response);<br />
}<br />
}</blockquote>
The web.xml needs adding the following too:<br />
<blockquote class="tr_bq">
<i><filter><br /> <filter-name>cors</filter-name><br /> <filter-class>com.zhentao.CorsFilter</filter-class><br /> </filter><br /> <filter-mapping><br /> <filter-name>cors</filter-name><br /> <url-pattern>/*</url-pattern><br /> </filter-mapping></i><br />
</blockquote>
However, this isn't enough yet. The original post didn't address it. You still need to change your rest web services to return Accept-Controll-Allow-Origin header since HTTP is stateless. Here is what I did with Spring Rest api:<br />
<blockquote class="tr_bq">
@RequestMapping(value = "/employee/{id}", method = RequestMethod.PUT, consumes = {"application/json"})<br />
public ResponseEntity<Employee> create(@Valid @RequestBody Employee employee, @PathVariable String id) </blockquote>
<blockquote class="tr_bq">
employee.setId(id);//some logic here<br />
HttpHeaders headers = new HttpHeaders();<br />
headers.add("Access-Control-Allow-Origin", "*");<br />
ResponseEntity<Employee> entity = new ResponseEntity<Employee>(headers, HttpStatus.CREATED);<br />
return entity; </blockquote>
<blockquote class="tr_bq">
} </blockquote>
Update: it seems my post caused some confusion, and I created another post and fully working examples for CORS. See <a href="http://zhentao-li.blogspot.com/2013/06/example-for-enabling-cors-support-in.html">here</a>. zhentaohttp://www.blogger.com/profile/05719795252682912081noreply@blogger.com13tag:blogger.com,1999:blog-6816360065241545754.post-48994784737355680602012-05-30T15:40:00.000-07:002012-05-30T15:40:03.350-07:00log4j filter some text and don't log itUse org.apache.log4j.varia.StringMatchFilter to filter the text not to display. Here is the example for log4j.xml:<br />
<blockquote class="tr_bq">
<?xml version="1.0" encoding="UTF-8" ?><br /><!DOCTYPE log4j:configuration SYSTEM "log4j.dtd"><br /><log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/"><br /> <appender name="console" class="org.apache.log4j.ConsoleAppender"><br /> <layout class="org.apache.log4j.PatternLayout"><br /> <param name="ConversionPattern" value="%d{yyyy-MM-dd HH:mm:ss} [%t] %-5p - %m%n" /><br /> </layout><br /><br /> <filter class="org.apache.log4j.varia.StringMatchFilter"><br /> <param name="StringToMatch" value="the-text-not-to-log" /><br /> <param name="AcceptOnMatch" value="false" /><br /> </filter><br /><br /> <!-- <filter class="org.apache.log4j.varia.DenyAllFilter" /> --><br /> </appender><br /><br /> <root><br /> <priority value="info" /><br /> <appender-ref ref="console" /><br /> </root><br /></log4j:configuration></blockquote>
Then run the following to test:<br />
<br />
LOG.info("This is an info and shouldn't be logged with the-text-not-to-log and blah blah");<br /> LOG.info("This is logged");<br /> LOG.debug("This is a debug and not logged");<br /> LOG.error("This is an error and logged"); <br />
<br />
If you only want to log some text, then change the value for AcceptOnMatch to <i><b>true</b></i> and add <i><b>DenyAllFilter</b></i>.zhentaohttp://www.blogger.com/profile/05719795252682912081noreply@blogger.com1tag:blogger.com,1999:blog-6816360065241545754.post-61588343166665990372012-05-23T16:01:00.001-07:002012-05-23T16:01:09.057-07:00mac: no acceptable C compiler found in $PATHIf you get the following error on mac:<br />
<blockquote class="tr_bq">
<i>configure: error: no acceptable C compiler found in $PATH</i></blockquote>
Here is the step to fix it:<br /><br />1. download Xcode from https://developer.apple.com/downloads/index.action<br />2. install Xcode<br />3. Open Xcode, go to Preferences --> Downloads --> install "Command Line Tools"<br />4. relaunch terminalzhentaohttp://www.blogger.com/profile/05719795252682912081noreply@blogger.com0tag:blogger.com,1999:blog-6816360065241545754.post-16484553998512979102012-05-19T13:27:00.000-07:002012-05-19T13:27:13.248-07:00Enable Spring Jdbc Transaction with AnnotationSpring 3.1 introduced a new annotation <i>@EnableTransactionManagement</i>. With it, all xml configuration can be got rid of now. Here is the example on how to use it:<br />
<blockquote class="tr_bq">
<br />@Configuration<br />@EnableTransactionManagement<br />public class JdbcConfig {<br /><br /> @Bean<br /> public DataSource dataSource() {<br /> //config datasource <br /> }<br /><br /> @Bean<br /> public PlatformTransactionManager txManager() {<br /> return new DataSourceTransactionManager(dataSource());<br /> }<br /><br /> @Bean<br /> public MyDao myDao() {<br /> MyDaoJdbc dao = new MyDaoJdbc();<br /> dao.setDataSource(dataSource());<br /> return dao;<br /> }<br />}</blockquote>
Next you need to annote your dao with @Transactional. The source code can be found at github:<br />
<br />
https://github.com/zhentao/spring-jdbc-transaction-example <br />
<blockquote class="tr_bq">
</blockquote>zhentaohttp://www.blogger.com/profile/05719795252682912081noreply@blogger.com0tag:blogger.com,1999:blog-6816360065241545754.post-81733421233611369722012-05-19T13:03:00.000-07:002012-05-19T13:03:23.959-07:00cobertura-maven-pluginRecently I created a parent pom for my team to use. All my team's projects need to inherit this parent pom. This parent pom is rather simple:<br />
<br />
<blockquote class="tr_bq">
<properties><br /> <default.encoding>UTF-8</default.encoding><br /> <default.jdk>1.6</default.jdk><br /> <line.coverage.target>90</line.coverage.target><br /> <branch.coverage.target>90</branch.coverage.target><br /> </properties><br /><br /> <build><br /> <defaultGoal>install</defaultGoal><br /> <pluginManagement><br /> <plugins><br /> <plugin><br /> <groupId>org.apache.maven.plugins</groupId><br /> <artifactId>maven-compiler-plugin</artifactId><br /> <version>2.3.2</version><br /> <configuration><br /> <source>${default.jdk}</source><br /> <target>${default.jdk}</target><br /> <encoding>${default.encoding}</encoding><br /> </configuration><br /> </plugin><br /> <plugin><br /> <groupId>org.apache.maven.plugins</groupId><br /> <artifactId>maven-source-plugin</artifactId><br /> <version>2.1.2</version><br /> </plugin><br /> <plugin><br /> <artifactId>maven-resources-plugin</artifactId><br /> <version>2.5</version><br /> <configuration><br /> <encoding>${default.encoding}</encoding><br /> </configuration><br /> </plugin><br /> <plugin><br /> <groupId>org.apache.maven.plugins</groupId><br /> <artifactId>maven-javadoc-plugin</artifactId><br /> <version>2.8</version><br /> <configuration><br /> <encoding>${default.encoding}</encoding><br /> </configuration><br /> </plugin><br /> <plugin><br /> <groupId>org.apache.maven.plugins</groupId><br /> <artifactId>maven-release-plugin</artifactId><br /> <version>2.2.2</version><br /> </plugin><br /><br /> <!-- Sonar uses these versions, below --><br /> <plugin><br /> <artifactId>maven-pmd-plugin</artifactId><br /> <version>2.7.1</version><br /> <configuration><br /> <targetJdk>${default.jdk}</targetJdk><br /> <encoding>${default.encoding}</encoding><br /> </configuration><br /> </plugin><br /> <plugin><br /> <artifactId>maven-checkstyle-plugin</artifactId><br /> <version>2.9.1</version><br /> <configuration><br /> <encoding>${default.encoding}</encoding><br /> </configuration><br /> </plugin><br /> <plugin><br /> <groupId>org.codehaus.mojo</groupId><br /> <artifactId>cobertura-maven-plugin</artifactId><br /> <version>2.5.1</version><br /> </plugin><br /> <plugin><br /> <groupId>org.apache.maven.plugins</groupId><br /> <artifactId>maven-surefire-plugin</artifactId><br /> <version>2.12</version><br /> </plugin><br /> </plugins><br /> </pluginManagement><br /><br /> <plugins><br /> <plugin><br /> <groupId>org.codehaus.mojo</groupId><br /> <artifactId>cobertura-maven-plugin</artifactId><br /> <configuration><br /> <check><br /> <!-- Per-class thresholds --><br /> <branchRate>${branch.coverage.target}</branchRate><br /> <lineRate>${line.coverage.target}</lineRate><br /> <!-- Project-wide thresholds --><br /> <totalLineRate>${line.coverage.target}</totalLineRate><br /> <totalBranchRate>${branch.coverage.target}</totalBranchRate><br /> </check><br /> <formats><br /> <format>html</format><br /> </formats><br /> </configuration><br /> <executions><br /> <execution><br /> <phase>package</phase><br /> <goals><br /> <goal>clean</goal><br /> <goal>cobertura</goal><br /> <goal>check</goal><br /> </goals><br /> </execution><br /> </executions><br /> </plugin><br /> <plugin><br /> <groupId>org.apache.maven.plugins</groupId><br /> <artifactId>maven-surefire-plugin</artifactId><br /> </plugin><br /> <plugin><br /> <!-- Ensure that source code is packaged and deployed for inclusion into IDEs --><br /> <groupId>org.apache.maven.plugins</groupId><br /> <artifactId>maven-source-plugin</artifactId><br /> <executions><br /> <execution><br /> <phase>package</phase><br /> <goals><br /> <goal>jar</goal><br /> </goals><br /> </execution><br /> </executions><br /> </plugin><br /><br /> </plugins><br /> </build></blockquote>
We are targeting a pretty high unit test coverage, 90% for both line and branch coverage. However, some classes don't need unit test or don't need high coverage. Then each individual project can include the following to its project pom:<br />
<blockquote class="tr_bq">
<build><br /> <plugins><br /> <plugin><br /> <groupId>org.codehaus.mojo</groupId><br /> <artifactId>cobertura-maven-plugin</artifactId><br /> <configuration><br /> <instrumentation><br /> <ignores><br /> <ignore>org.slf4j.*</ignore><br /> </ignores><br /> <excludes><br /> <exclude>com/zhentao/Some.class</exclude><br /> <exclude>com/zhentao/other.class</exclude><br /> </excludes><br /> </instrumentation><br /> </configuration><br /> </plugin><br /> </plugins><br /> </build></blockquote>
<ignore>org.slf4j.*</ignore> ignores all log statement in the code since we don't need to test logs.<br />
<exclude>com/zhentao/Some.class</exclude> excludes Some.class from the cobertura report.<br />
<br />
However, this flexibility can be abused by some developers. I got an build failed notification today:<br />
<blockquote class="tr_bq">
<pre><i><b>11:52:52</b> </i><i>Archiving artifacts
<b>11:52:52</b> [htmlpublisher] Archiving HTML reports...
<b>11:52:52</b> [htmlpublisher] Archiving at PROJECT level /target/site/cobertura to /htmlreports/Cobertura_Report
<b>11:52:52</b> ERROR: Specified HTML directory '/target/site/cobertura' does not exist.
<b>11:52:52</b> Build step 'Publish HTML reports' changed build result to FAILURE</i></pre>
</blockquote>
One developer excluded all classes from testing. What a developer!zhentaohttp://www.blogger.com/profile/05719795252682912081noreply@blogger.com0tag:blogger.com,1999:blog-6816360065241545754.post-79559002491861965722012-05-04T10:29:00.000-07:002012-05-04T10:29:25.175-07:00Yahoo hadoop tutorialsThis is the link from yahoo for hadoop tutorials:<br />
<br />
http://developer.yahoo.com/hadoop/tutorial/index.htmlzhentaohttp://www.blogger.com/profile/05719795252682912081noreply@blogger.com0