Fatkun's Blog

hdfs磁盘检查相关文章

 Hadoop  95  1 分钟
DataNode启动优化改进：磁盘检测并行化 https://issues.apache.org/jira/browse/HDFS-8845 检查时不遍历所有子目录 https://issues.apache.org/jira/browse/HDFS-8850 VolumeS……
阅读更多 »

[BUG]HIVE-9613 left join会导致数据错位

 Hive  45  1 分钟
category city rank src_category_en src_city_name_en jinrongfuwu shanghai 1 danbaobaoxiantouzi sh ktvjiuba shanghai 2 zpwentiyingshi sh but int hive0.14,the results in the column **src\_category\_en** is wrong,and is just the **city** contents: category city rank src_category_en src_city_name_en jinrongfuwu shanghai 1 shanghai sh ktvjiuba shanghai……
阅读更多 »

Hive遇到问题汇总

 Hive  176  1 分钟
新增两个字段，但查出来的值为null。 分区有自己的字段信息，检查分区的字段信息是否和……
阅读更多 »

JDK Native Memory Tracking

 J2EE  11  1 分钟
command line option: -XX:NativeMemoryTracking=summary or -XX:NativeMemoryTracking=detail jcmd ${PID} VM.native_memory summary https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr007.html

Hive On Tez cloudera5.4

 Hadoop/Hive  330  1 分钟
配置 tez-site.xml <configuration> <property> <name>tez.lib.uris</name> <value>${fs.defaultFS}/apps/tez-0.8.5/,${fs.defaultFS}/apps/tez-0.8.5/lib/</value> </property> <property> <name>tez.use.cluster.hadoop-libs</name> <value>true</value> </property> <property> <name>tez.runtime.compress</name> <value>true</value> </property> <property> <name>tez.runtime.compress.codec</name> <value>org.apache.hadoop.io.compress.SnappyCodec</value> </property> </configuration> 调优参数 tez.grouping.min-size 分片最小限制 报错处理 找不到lzo……
阅读更多 »

jobhistory关注的issue

 Hadoop  46  1 分钟
JobHistory cache issue https://issues.apache.org/jira/browse/MAPREDUCE-6436 可能导致打印的日志太多影响性能 Job history server scans can become blocked on a single, slow entry https://issues.apache.org/jira/browse/MAPREDUCE-6797 High contention on scanning of user directory under immediate_done in……
阅读更多 »

kafka关注的issue

 Hadoop  25  1 分钟
BrokerChangeListener computes inconsistent live/dead broker list https://issues.apache.org/jira/browse/KAFKA-3085 BrokerChangeListener missed broker id path ephemeral node deletion event. https://issues.apache.org/jira/browse/KAFKA-2448 Controller could miss a broker state change https://issues.apache.org/jira/browse/KAFKA-1120

在reduce InMemoryMapOutput OOM

 Hadoop  102  1 分钟
2017-03-14 15:41:52,724 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#1 at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:376) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: java.lang.OutOfMemoryError: Java heap space at org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:56) at org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:46) at org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.<init>(InMemoryMapOutput.java:63)……
阅读更多 »