tag:blogger.com,1999:blog-5261056907132640554.comments2015-11-09T12:02:39.938-08:00bizo developer bloglarry ogrodnekhttp://www.blogger.com/profile/01105034385285773975noreply@blogger.comBlogger111125tag:blogger.com,1999:blog-5261056907132640554.post-65889213287967081732013-10-08T08:45:56.751-07:002013-10-08T08:45:56.751-07:00You can access the members of a map using square b...You can access the members of a map using square brackets. eg,<br /><br />select d["timestamp"], d["id"] from sample_data_2011_12 ;<br />Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-5261056907132640554.post-57168206091501497782013-10-08T04:50:09.720-07:002013-10-08T04:50:09.720-07:00In the past, I have explicitly specified column na...In the past, I have explicitly specified column names & data types when creating hive table for importing access logs. If you create a table with the following statement as mentioned in tip 2, how do you query for individual columns?<br /><br />create external table sample_data_2011_12(d map)<br />g3t r00thttps://www.blogger.com/profile/09860128909809442458noreply@blogger.comtag:blogger.com,1999:blog-5261056907132640554.post-51065457497425299322013-09-15T04:09:04.970-07:002013-09-15T04:09:04.970-07:00Hi! I needed to change the bash code a bit to make...Hi! I needed to change the bash code a bit to make the first example work:<br /><br />code=$(cat <<EOF<br />scala.io.Source.stdin.getLines map { $@ } foreach println<br />EOF<br />)<br />scala -e "$code"<br /><br />Great tricks, really liked the post!Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-5261056907132640554.post-5852667825215059822013-09-15T04:08:01.001-07:002013-09-15T04:08:01.001-07:00I needed to change the first example to:
code=$(c...I needed to change the first example to:<br /><br />code=$(cat <<EOF<br />scala.io.Source.stdin.getLines map { $@ } foreach println<br />EOF<br />)<br />scala -e "$code"<br /><br />to make it work. Otherwise, great blog post and great tricks :)<br /><br />GregaAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-5261056907132640554.post-35992798278185748872013-07-18T10:28:59.760-07:002013-07-18T10:28:59.760-07:00great explanation. Simple and to the point. Thanks...great explanation. Simple and to the point. Thanks for writing about map side aggregation techniqueAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-5261056907132640554.post-19914844698178626612013-07-01T23:20:06.117-07:002013-07-01T23:20:06.117-07:00Hi , I need your help in this. Tell me how to set ...Hi , I need your help in this. Tell me how to set the access token in header.<br /><br />I am unable to do that. I have set the network code,application name and passing the company id to get the company.But how to set the access token.<br /><br />Thanks<br />Anonymoushttps://www.blogger.com/profile/00399802759837463904noreply@blogger.comtag:blogger.com,1999:blog-5261056907132640554.post-47082715165908657142013-03-26T07:05:02.008-07:002013-03-26T07:05:02.008-07:00Exactly what I was looking for. My UDTF is already...Exactly what I was looking for. My UDTF is already working. Thanks from Slovakia!Anonymoushttps://www.blogger.com/profile/09049747636654729593noreply@blogger.comtag:blogger.com,1999:blog-5261056907132640554.post-13353700204829987852013-02-11T10:01:12.492-08:002013-02-11T10:01:12.492-08:00Hi pas,
Sorry for the late reply, but currently w...Hi pas,<br /><br />Sorry for the late reply, but currently we're not using the ec2 scripts, and instead running our Spark cluster in an Amazon EMR cluster using bootstrap actions.<br /><br />We're planning on publishing these at some point, either as a blog post or github repo, but haven't yet.<br /><br />They are tied to our particular setup, but would offer others a starting place if they're interested in the same approach.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-5261056907132640554.post-44793295245343898982013-01-30T10:29:35.856-08:002013-01-30T10:29:35.856-08:00Wonderful! I was going mad looking for the source ...Wonderful! I was going mad looking for the source of intermittent failures in Chef's mdadm provider on EC2. You've saved what remains of my hair.Anonymoushttps://www.blogger.com/profile/07005410207387476306noreply@blogger.comtag:blogger.com,1999:blog-5261056907132640554.post-71695133084005129812013-01-23T22:43:00.279-08:002013-01-23T22:43:00.279-08:00I would love to know what (if any) changes you hav...I would love to know what (if any) changes you have made to the spark_ec2 script. As the script is now, it is far from production ready.Peter Sankauskashttps://www.blogger.com/profile/06030981874808382464noreply@blogger.comtag:blogger.com,1999:blog-5261056907132640554.post-13066538898790662072012-12-27T03:52:31.266-08:002012-12-27T03:52:31.266-08:00Thanks for the udevadm tip. I also could understan...Thanks for the udevadm tip. I also could understand why mdadm would report devices as busy, even though nothing was using (no dmraid, not device mapper).Slavik Goltserhttps://www.blogger.com/profile/11327060237995322907noreply@blogger.comtag:blogger.com,1999:blog-5261056907132640554.post-15312976023017222962012-09-14T13:09:05.060-07:002012-09-14T13:09:05.060-07:00Thank you so much! This has been a huge issue for ...Thank you so much! This has been a huge issue for us. Now it's solved.revacuatehttps://www.blogger.com/profile/16711487844239845009noreply@blogger.comtag:blogger.com,1999:blog-5261056907132640554.post-11792406855303810992012-08-10T07:17:50.602-07:002012-08-10T07:17:50.602-07:00Wow, seems like you saved my life here. Thanks a l...Wow, seems like you saved my life here. Thanks a lot !Anonymoushttps://www.blogger.com/profile/16390219938405845416noreply@blogger.comtag:blogger.com,1999:blog-5261056907132640554.post-56945846307615737892012-07-31T15:11:01.708-07:002012-07-31T15:11:01.708-07:00The main reason would be that structs are a better...The main reason would be that structs are a better description of what you're actually returning. In the example, we can access the data inside of the result using "firstName" and "lastName". It might seem pretty intuitive to the developer to simply make these two fields of a string array, but what happens if the return type has a large number of fields or if there is no natural order to the fields?<br /><br />The other reason is that arrays are homogeneous in Hive, so you can't return multiple types of data in a single array. With a struct, for example, you could return a firstName, lastName, age (integer-valued), aliases (array of strings), and known addresses (array of structs).Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-5261056907132640554.post-49212733167397047512012-07-31T14:51:14.463-07:002012-07-31T14:51:14.463-07:00Why not return an array?Why not return an array?Satpreethttps://www.blogger.com/profile/00735298429611768189noreply@blogger.comtag:blogger.com,1999:blog-5261056907132640554.post-84782653567366764512012-07-26T19:30:52.181-07:002012-07-26T19:30:52.181-07:00I was having this same issue! Thank you!I was having this same issue! Thank you!Anonymoushttps://www.blogger.com/profile/03965474314076149566noreply@blogger.comtag:blogger.com,1999:blog-5261056907132640554.post-2054431346764642352012-07-22T12:18:42.571-07:002012-07-22T12:18:42.571-07:00Very useful. I was going mad. Thanks a lot.Very useful. I was going mad. Thanks a lot.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-5261056907132640554.post-56708588443819992392012-07-22T12:14:53.302-07:002012-07-22T12:14:53.302-07:00This comment has been removed by the author.Abhishek https://www.blogger.com/profile/05575241454112520600noreply@blogger.comtag:blogger.com,1999:blog-5261056907132640554.post-23050266919373702902012-07-20T12:26:30.999-07:002012-07-20T12:26:30.999-07:00Thank you so much! It took me a while to find thi...Thank you so much! It took me a while to find this solution via google. I'm surprised it's not more prevalent around the web.Andrewhttps://www.blogger.com/profile/01933762395445066784noreply@blogger.comtag:blogger.com,1999:blog-5261056907132640554.post-47857211057219891962012-07-09T23:14:27.249-07:002012-07-09T23:14:27.249-07:00Thank a lot for the post! I had been pulling my ha...Thank a lot for the post! I had been pulling my hair out trying to solve this issue setting up an NAS.gmanleyhttps://www.blogger.com/profile/06945585592930615452noreply@blogger.comtag:blogger.com,1999:blog-5261056907132640554.post-24942598368751205712012-02-15T08:08:17.223-08:002012-02-15T08:08:17.223-08:00I'd recommend having the query that loads data...I'd recommend having the query that loads data expan the object into its individual fields, as in the "output" table in the post. That way, the only time users need to be aware of the object structure is if they're using the UDF directly.<br /><br />For that case, I'd recommend documenting the object format in a @Description annotation (http://hive.apache.org/docs/r0.7.0/api/org/apache/hadoop/hive/ql/exec/Description.html) on your UDF. The "value" and "extended" fields will then be available inside the Hive console via "describe function foo" and "describe extended function foo".Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-5261056907132640554.post-14624664515893979622012-02-14T13:52:04.489-08:002012-02-14T13:52:04.489-08:00The problem with this though is that the hive-user...The problem with this though is that the hive-users need to have knowledge of the object before hand to properly take advantage of the system. It's not so easy for them to browse each table and find what they want. Do you have any tools to allow people to do that?Matthew Rathbonehttps://www.blogger.com/profile/09876845461210888800noreply@blogger.comtag:blogger.com,1999:blog-5261056907132640554.post-25617453009394456652012-01-13T11:12:28.670-08:002012-01-13T11:12:28.670-08:00Yes, some more information on the scroll-back buff...Yes, some more information on the scroll-back buffer here:<br />http://www.samsarin.com/blog/2007/03/11/gnu-screen-working-with-the-scrollback-buffer/Alex Boisverthttps://www.blogger.com/profile/15673966086206028049noreply@blogger.comtag:blogger.com,1999:blog-5261056907132640554.post-79390124191921462222012-01-13T11:08:56.552-08:002012-01-13T11:08:56.552-08:00Can you retrieve the output that happens in hive w...Can you retrieve the output that happens in hive when the screen is detached? Does it show it all when you reconnect?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-5261056907132640554.post-68208152131064143302012-01-04T09:27:09.266-08:002012-01-04T09:27:09.266-08:00Thanks Larry,
really helpful. I do have 3 questio...Thanks Larry,<br /><br />really helpful. I do have 3 questions and would much appreciate your feedback.<br /><br />1. Why do we need a map side script? If we are just selecting 2 columns from a table and then performing some calculation in the reduce side with a script, can we not just "select" those 2 columns and then use the construct in reduce like:<br />select transform(kv_my_input.col1,kv_my_input.col2) using 'reduce_script' as newcol1,newcol2<br />from kv_my_input cluster by col1;<br />2. Is the map using 'map script' and reduce using 'reduce script' construct equivalent to transform ( list of columns )? If so, how do we know that the transform in question 1 will produce a "reduce" logic from your script and not the "map" logic. Or it performs whatever is inside the script?<br />3. Is clustering mandatory? Is it clustering which actually enables one to perform the reduce side script like you have shown? i.e. without clustering, your rows will be scattered ?<br /><br />thanks again<br /><br />ameetameethttps://www.blogger.com/profile/07926221378130882969noreply@blogger.com