Fatkun's Blog

在reduce InMemoryMapOutput OOM

 Hadoop  102  1 分钟
2017-03-14 15:41:52,724 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#1 at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:376) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: java.lang.OutOfMemoryError: Java heap space at org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:56) at org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:46) at org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.<init>(InMemoryMapOutput.java:63)……
阅读更多 »

hive merge小文件

 Hive  176  1 分钟
2.输出合并 set hive.merge.mapfiles = true #在Map-only的任务结束时合并小文件（默认开启） set hive.merge.mapredfiles = true #……
阅读更多 »

impala断开连接后释放session

 Hadoop  106  1 分钟
代码在这里 https://github.com/cloudera/Impala/blob/cdh5-2.2.0_5.4.0/be/src/service/impala-server.cc 如果不想断开清除session,直接return void ImpalaServer::ConnectionEnd( const ThriftServer::ConnectionContext& connection_context) { return; unique_lock<mutex> l(connection_to_sessions_map_lock_); ConnectionToSessionMap::iterator it =……
阅读更多 »

HIVE-10956 HS2 leaks HMS connections

 Hive  18  1 分钟
https://issues.apache.org/jira/browse/HIVE-10956 HMS链接泄露 https://issues.apache.org/jira/browse/HIVE-13002 HIVE-11934 Transaction lock retry logic results in infinite loop

使用powermock

 J2EE  145  1 分钟
在pom.xml加入 <dependency> <groupId>org.easymock</groupId> <artifactId>easymock</artifactId> <version>3.3.1</version> <scope>test</scope> </dependency> <dependency> <groupId>org.powermock</groupId> <artifactId>powermock-module-junit4</artifactId> <version>1.6.5</version> <scope>test</scope> </dependency> <dependency> <groupId>org.powermock</groupId> <artifactId>powermock-api-easymock</artifactId> <version>1.6.5</version> <scope>test</scope> </dependency> 如果报错，有可能项目中包含有……
阅读更多 »