Some of the anomalies about kylin

Kylin
  1. 异常信息：可以发现是找不到hive-jdbc驱动包
  2. kylin Container 内存不够
  3. 在kylin中执行SQL语句异常
  4. kylin执行时找不到类

　　在做kylin的时候遇到了的一些问题，如果后面继续发现问题，会持续做更新……

  1. 异常信息：可以发现是找不到hive-jdbc驱动包
  2. kylin运行时Container内存不够
  3. 在kylin中执行SQL语句异常
  4. kylin执行时找不到类

异常信息：可以发现是找不到hive-jdbc驱动包

  • Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.serde2.typeinfo.TypeInfo
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    2017-03-09 15:27:43,438 FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.NoClassDefFoundError: org/apache/hadoop/hive/serde2/typeinfo/TypeInfo
    	at java.lang.Class.forName0(Native Method)
    	at java.lang.Class.forName(Class.java:270)
    	at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2138)
    	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2103)
    	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2197)
    	at org.apache.hadoop.mapreduce.task.JobContextImpl.getInputFormatClass(JobContextImpl.java:184)
    	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:749)
    	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
    	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
    	at java.security.AccessController.doPrivileged(Native Method)
    	at javax.security.auth.Subject.doAs(Subject.java:415)
    	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1796)
    	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
    Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.serde2.typeinfo.TypeInfo
    	at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    	at java.security.AccessController.doPrivileged(Native Method)
    	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    	at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    	at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    	... 13 more

知道问题是啥，但是不知道怎么解决，尴尬！因为把jive-jdbc放入到$KYLIN_HOME/bin没效果，不知道把jdbc驱动放到哪儿，google了一下
hhttp://www.07net01.com/2016/02/1292650.html –>
https://stackoverflow.com/questions/34449561/hadoop-map-reduce-job-class-org-apache-hive-hcatalog-mapreduce-hcatinputformat –>
https://issues.apache.org/jira/browse/KYLIN-1021

  • 解决：
    在$KYLIN_HOME/conf/kylin.properties配置：
    kylin.job.mr.lib.dir=/opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.41/jars

kylin Container 内存不够

  • running beyond physical memory limits

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    Container [pid=22991,containerID=container_1494324779160_0130_01_000002] is running beyond physical memory limits. 
    Current usage: 607.6 MB of 512 MB physical memory used; 1.1 GB of 1.0 GB virtual memory used. Killing container. 
    Dump of the process-tree for container_1494324779160_0130_01_000002 : 
    |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE 
    |- 24045 22991 22991 22991 (java) 1301 77 1076248576 155172 
    /usr/java/jdk1.7.0_67-cloudera/bin/java 
    -Djava.net.preferIPv4Stack=true 
    -Dhadoop.metrics.log.level=WARN 
    -Djava.net.preferIPv4Stack=true -Xmx419430400 
    -Djava.io.tmpdir=/hbase/yarn/nm3/usercache/root/appcache/application_1494324779160_0130/container_1494324779160_0130_01_000002/tmp 
    -Dlog4j.configuration=container-log4j.properties 
    -Dyarn.app.container.log.dir=/hbase/yarn/container-logs/application_1494324779160_0130/container_1494324779160_0130_01_000002 
    -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA 
    -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 10.141.5.25 33530 attempt_1494324779160_0130_m_000000_0 2 |- 22991 22988 22991 22991 (bash) 12 22 110931968 386 /bin/bash -c /usr/java/jdk1.7.0_67-cloudera/bin/java 
    -Djava.net.preferIPv4Stack=true 
    -Dhadoop.metrics.log.level=WARN 
    -Djava.net.preferIPv4Stack=true -Xmx419430400 
    -Djava.io.tmpdir=/hbase/yarn/nm3/usercache/root/appcache/application_1494324779160_0130/container_1494324779160_0130_01_000002/tmp 
    -Dlog4j.configuration=container-log4j.properties 
    -Dyarn.app.container.log.dir=/hbase/yarn/container-logs/application_1494324779160_0130/container_1494324779160_0130_01_000002 
    -Dyarn.app.container.log.filesize=0 
    -Dhadoop.root.logger=INFO,CLA 
    -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 10.141.5.25 33530 attempt_1494324779160_0130_m_000000_0 2 1>/hbase/yarn/container-logs/application_1494324779160_0130/container_1494324779160_0130_01_000002/stdout 2>/hbase/yarn/container-logs/application_1494324779160_0130/container_1494324779160_0130_01_000002/stderr 
    Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143

  • 参考：
    https://stackoverflow.com/questions/21005643/container-is-running-beyond-memory-limits

在kylin中执行SQL语句异常

  • Error while executing SQL

    1
    2
    3
    4
    5
    6
    7
    8
    9
    Error while executing SQL "SELECT part_dt,sum(price) AS total_price,count(DISTINCT seller_id) AS sellers FROM kylin_sales GROUP BY part_dt ORDER BY part_dt LIMIT 50000": 
    org.apache.hadoop.hbase.exceptions.UnknownProtocolException: 
    org.apache.hadoop.hbase.exceptions.UnknownProtocolException: 
    No registered coprocessor service found for name CubeVisitService in region KYLIN_CMQ2D8I0H1,\x00\x04,1495703949441.26edd2674722c672724d25cfa410e061. 
    at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:7773) 
    at org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:1982) 
    at org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:1964) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33652) 
    at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:185) 
    at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:165)

  • 解决：
    http://blog.csdn.net/sikongyuanruo/article/details/53439163 –>
    可能的原因：kylin与当前的hbase匹配有问题，在官网上找到了解决的方法：
    http://kylin.apache.org/docs15/howto/howto_update_coprocessor.html 即，执行下面的shell命令：
    $KYLIN_HOME/bin/kylin.sh org.apache.kylin.storage.hbase.util.DeployCoprocessorCLI $KYLIN_HOME/lib/kylin-coprocessor-*.jar all

kylin执行时找不到类

  • Error: java.lang.ClassNotFoundException

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    Error: java.lang.ClassNotFoundException: org.apache.kylin.engine.mr.KylinMapper at 
        java.net.URLClassLoader$1.run(URLClassLoader.java:366) at 
        java.net.URLClassLoader$1.run(URLClassLoader.java:355) at 
        java.security.AccessController.doPrivileged(Native Method) at 
        java.net.URLClassLoader.findClass(URLClassLoader.java:354) at 
        java.lang.ClassLoader.loadClass(ClassLoader.java:425) at 
        sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at 
        java.lang.ClassLoader.loadClass(ClassLoader.java:358) at 
        java.lang.ClassLoader.defineClass1(Native Method) at 
        java.lang.ClassLoader.defineClass(ClassLoader.java:800) at 
        java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at 
        java.net.URLClassLoader.defineClass(URLClassLoader.java:449) at 
        java.net.URLClassLoader.access$100(URLClassLoader.java:71) at 
        java.net.URLClassLoader$1.run(URLClassLoader.java:361) at 
        java.net.URLClassLoader$1.run(URLClassLoader.java:355) at 
        java.security.AccessController.doPrivileged(Native Method) at 
        java.net.URLClassLoader.findClass(URLClassLoader.java:354) at 
        java.lang.ClassLoader.loadClass(ClassLoader.java:425) at 
        sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at 
        java.lang.ClassLoader.loadClass(ClassLoader.java:358) at 
        java.lang.Class.forName0(Native Method) at 
        java.lang.Class.forName(Class.java:270) at 
        org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2138) at 
        org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2103) at 
        org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2197) at 
        org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196) at 
        org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:745) at 
        org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at 
        org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) at 
        java.security.AccessController.doPrivileged(Native Method) at 
        javax.security.auth.Subject.doAs(Subject.java:415) at 
        org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1796) at 
        org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

  • 解决：
    默认安装kylin后，kylin相关的lib并没有在yarn中，运行任务时job就会抛出异常，java.lang.ClassNotFoundException，解决思路就是添加jar。比如上面就缺少kylin-engine-mr

  1. 上传jar到kylin lib
  2. 并且做软连接(只需要在kylin对应的节点做软连接)
  3. ln -s /hbase/kylin-2.0/lib/kylin-engine-mr /opt/cloudera/parcels/CDH/jars/
  4. 后面还会遇到一些这样的问题，这是我遇到过的，上传的jar
    kylin 上传的jar

当前网速较慢或者你使用的浏览器不支持博客特定功能，请尝试刷新或换用Chrome、Firefox等现代浏览器