Atlas--与Hive集成

  
  本篇博文,讲解与配置Atlas与Hive的集成,以及其中遇到的一些问题,然后把hive的元数据信息导入到atlas里面,我们就可以直观的去看。
这边文章主要在于集成hive,其中的比如一下安装hadoop,hive等,可以查看我之前的博客。该集成不涉及到元数据血缘关系,主要通过atlas自身的导入脚本离线导入元数据,血缘以及实时导入将在后面的博文中介绍

配置Hive

  1. 配置,Hive安装目录/conf/hive-site.xml 添加:

    1
    2
    3
    4
    <property>
    <name>hive.exec.post.hooks</name>
    <value>org.apache.atlas.hive.hook.HiveHook</value>
    </property>
  2. 配置,Hive安装目录/conf/hive-env.sh 添加:

    1
    export HIVE_AUX_JARS_PATH=$atlas编译好的安装目录/hook/hive

例如我这里修改为
export HIVE_AUX_JARS_PATH=/opt/dev/idea/apache-atlas-sources-2.0.0/distro/target/apache-atlas-2.0.0-bin/apache-atlas-2.0.0/hook/hive

异常解决

  • 异常1、在我们配置完上面的内容后,如果在运行hive,可能就会报如下的异常:
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    hive> show tables;
    FAILED: Hive Internal Error: java.lang.ExceptionInInitializerError(null)
    java.lang.ExceptionInInitializerError
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:348)
    at org.apache.atlas.hive.hook.HiveHook.initialize(HiveHook.java:72)
    at org.apache.atlas.hive.hook.HiveHook.<init>(HiveHook.java:41)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at java.lang.Class.newInstance(Class.java:442)
    at org.apache.hadoop.hive.ql.hooks.HookUtils.getHooks(HookUtils.java:61)
    at org.apache.hadoop.hive.ql.Driver.getHooks(Driver.java:1685)
    at org.apache.hadoop.hive.ql.Driver.getHooks(Driver.java:1669)
    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1941)
    at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1526)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227)
    at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
    at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
    at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
    at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821)
    at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
    Caused by: java.lang.NullPointerException
    at org.apache.atlas.hook.AtlasHook.<clinit>(AtlasHook.java:81)
    ... 28 more

    hive>

解决:
原因,hive缺少了相应的脚本atlas-application.properties、atlas-env.sh,需要做一个软连接

1
2
ln -s /opt/dev/idea/apache-atlas-sources-2.0.0/distro/target/apache-atlas-2.0.0-bin/apache-atlas-2.0.0/conf/atlas-application.properties /opt/local/apache-hive-2.3.5-bin/conf/atlas-application.properties
ln -s /opt/dev/idea/apache-atlas-sources-2.0.0/distro/target/apache-atlas-2.0.0-bin/apache-atlas-2.0.0/conf/atlas-env.sh /opt/local/apache-hive-2.3.5-bin/conf/atlas-env.sh

apache atlas
apache atlas

  • 异常2、运行导入hive元数据脚本import-hive.sh,异常
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    2019-05-22T17:55:06,328 INFO [main] org.apache.atlas.AtlasBaseClient - method=GET path=api/atlas/v2/entity/uniqueAttribute/type/ contentType=application/json; charset=UTF-8 accept=application/json status=404
    Exception in thread "main" java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.util.BeanUtil.okNameForSetter(Lcom/fasterxml/jackson/databind/introspect/AnnotatedMethod;)Ljava/lang/String;
    at com.fasterxml.jackson.module.jaxb.JaxbAnnotationIntrospector.findNameForDeserialization(JaxbAnnotationIntrospector.java:1004)
    at com.fasterxml.jackson.databind.introspect.AnnotationIntrospectorPair.findNameForDeserialization(AnnotationIntrospectorPair.java:749)
    at com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector._addSetterMethod(POJOPropertiesCollector.java:620)
    at com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector._addMethods(POJOPropertiesCollector.java:535)
    at com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector.collectAll(POJOPropertiesCollector.java:309)
    at com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector.getJsonValueAccessor(POJOPropertiesCollector.java:196)
    at com.fasterxml.jackson.databind.introspect.BasicBeanDescription.findJsonValueAccessor(BasicBeanDescription.java:252)
    at com.fasterxml.jackson.databind.ser.BasicSerializerFactory.findSerializerByAnnotations(BasicSerializerFactory.java:346)
    at com.fasterxml.jackson.databind.ser.BeanSerializerFactory._createSerializer2(BeanSerializerFactory.java:216)
    at com.fasterxml.jackson.databind.ser.BeanSerializerFactory.createSerializer(BeanSerializerFactory.java:165)
    at com.fasterxml.jackson.databind.SerializerProvider._createUntypedSerializer(SerializerProvider.java:1388)
    at com.fasterxml.jackson.databind.SerializerProvider._createAndCacheUntypedSerializer(SerializerProvider.java:1336)
    at com.fasterxml.jackson.databind.SerializerProvider.findValueSerializer(SerializerProvider.java:510)
    at com.fasterxml.jackson.databind.SerializerProvider.findTypedValueSerializer(SerializerProvider.java:713)
    at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:308)
    at com.fasterxml.jackson.databind.ObjectWriter$Prefetch.serialize(ObjectWriter.java:1396)
    at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:913)
    at com.fasterxml.jackson.jaxrs.base.ProviderBase.writeTo(ProviderBase.java:648)
    at com.sun.jersey.api.client.RequestWriter.writeRequestEntity(RequestWriter.java:300)
    at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:204)
    at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:147)
    at com.sun.jersey.api.client.filter.HTTPBasicAuthFilter.handle(HTTPBasicAuthFilter.java:81)
    at com.sun.jersey.api.client.Client.handle(Client.java:648)
    at com.sun.jersey.api.client.WebResource.handle(WebResource.java:670)
    at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
    at com.sun.jersey.api.client.WebResource$Builder.method(WebResource.java:623)
    at org.apache.atlas.AtlasBaseClient.callAPIWithResource(AtlasBaseClient.java:382)
    at org.apache.atlas.AtlasBaseClient.callAPIWithResource(AtlasBaseClient.java:353)
    at org.apache.atlas.AtlasBaseClient.callAPI(AtlasBaseClient.java:229)
    at org.apache.atlas.AtlasClientV2.createEntity(AtlasClientV2.java:320)
    at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerInstance(HiveMetaStoreBridge.java:450)
    at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerDatabase(HiveMetaStoreBridge.java:402)
    at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importDatabases(HiveMetaStoreBridge.java:281)
    at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importHiveMetadata(HiveMetaStoreBridge.java:251)
    at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:168)
    Failed to import Hive Meta Data!!!

解决:
1。下载 jackson-annotations-2.9.8.jar, jackson-core-2.9.8.jarjackson-databind-2.9.8.jar,可以通过https://mvnrepository.com下载
2。然后把jar放入到 ${atlas_project}/distro/target/apache-atlas-2.0.0-bin/apache-atlas-2.0.0/hook/hive/atlas-hive-plugin-impl/下面
apache atlas&hive

运行导入hive元数据

  • 运行:bin/import-hive.sh 成功
    apache atlas import-hive.sh

  • 浏览
    左边选框可以选择
    apache atlas web

  • 参考:
  1. https://blog.csdn.net/jeffiny/article/details/82841171
  2. https://blog.csdn.net/hhhh222222/article/details/77427769
  3. https://issues.apache.org/jira/browse/ATLAS-3172?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel&focusedCommentId=16827794#comment-16827794

当前网速较慢或者你使用的浏览器不支持博客特定功能,请尝试刷新或换用Chrome、Firefox等现代浏览器