Atlas--与Hive集成
本篇博文,讲解与配置Atlas与Hive的集成,以及其中遇到的一些问题,然后把hive的元数据信息导入到atlas里面,我们就可以直观的去看。
这边文章主要在于集成hive,其中的比如一下安装hadoop,hive等,可以查看我之前的博客。该集成不涉及到元数据血缘关系,主要通过atlas自身的导入脚本离线导入元数据,血缘以及实时导入将在后面的博文中介绍。
配置Hive
配置,Hive安装目录/conf/hive-site.xml 添加:
1
2
3
4<property>
<name>hive.exec.post.hooks</name>
<value>org.apache.atlas.hive.hook.HiveHook</value>
</property>配置,Hive安装目录/conf/hive-env.sh 添加:
1
export HIVE_AUX_JARS_PATH=$atlas编译好的安装目录/hook/hive
例如我这里修改为:
export HIVE_AUX_JARS_PATH=/opt/dev/idea/apache-atlas-sources-2.0.0/distro/target/apache-atlas-2.0.0-bin/apache-atlas-2.0.0/hook/hive
异常解决
- 异常1、在我们配置完上面的内容后,如果在运行hive,可能就会报如下的异常:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36show tables;
FAILED: Hive Internal Error: java.lang.ExceptionInInitializerError(null)
java.lang.ExceptionInInitializerError
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.atlas.hive.hook.HiveHook.initialize(HiveHook.java:72)
at org.apache.atlas.hive.hook.HiveHook.<init>(HiveHook.java:41)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.lang.Class.newInstance(Class.java:442)
at org.apache.hadoop.hive.ql.hooks.HookUtils.getHooks(HookUtils.java:61)
at org.apache.hadoop.hive.ql.Driver.getHooks(Driver.java:1685)
at org.apache.hadoop.hive.ql.Driver.getHooks(Driver.java:1669)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1941)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1526)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
Caused by: java.lang.NullPointerException
at org.apache.atlas.hook.AtlasHook.<clinit>(AtlasHook.java:81)
... 28 more
解决:
原因,hive缺少了相应的脚本atlas-application.properties、atlas-env.sh,需要做一个软连接1
2ln -s /opt/dev/idea/apache-atlas-sources-2.0.0/distro/target/apache-atlas-2.0.0-bin/apache-atlas-2.0.0/conf/atlas-application.properties /opt/local/apache-hive-2.3.5-bin/conf/atlas-application.properties
ln -s /opt/dev/idea/apache-atlas-sources-2.0.0/distro/target/apache-atlas-2.0.0-bin/apache-atlas-2.0.0/conf/atlas-env.sh /opt/local/apache-hive-2.3.5-bin/conf/atlas-env.sh
- 异常2、运行导入hive元数据脚本import-hive.sh,异常
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
382019-05-22T17:55:06,328 INFO [main] org.apache.atlas.AtlasBaseClient - method=GET path=api/atlas/v2/entity/uniqueAttribute/type/ contentType=application/json; charset=UTF-8 accept=application/json status=404
Exception in thread "main" java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.util.BeanUtil.okNameForSetter(Lcom/fasterxml/jackson/databind/introspect/AnnotatedMethod;)Ljava/lang/String;
at com.fasterxml.jackson.module.jaxb.JaxbAnnotationIntrospector.findNameForDeserialization(JaxbAnnotationIntrospector.java:1004)
at com.fasterxml.jackson.databind.introspect.AnnotationIntrospectorPair.findNameForDeserialization(AnnotationIntrospectorPair.java:749)
at com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector._addSetterMethod(POJOPropertiesCollector.java:620)
at com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector._addMethods(POJOPropertiesCollector.java:535)
at com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector.collectAll(POJOPropertiesCollector.java:309)
at com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector.getJsonValueAccessor(POJOPropertiesCollector.java:196)
at com.fasterxml.jackson.databind.introspect.BasicBeanDescription.findJsonValueAccessor(BasicBeanDescription.java:252)
at com.fasterxml.jackson.databind.ser.BasicSerializerFactory.findSerializerByAnnotations(BasicSerializerFactory.java:346)
at com.fasterxml.jackson.databind.ser.BeanSerializerFactory._createSerializer2(BeanSerializerFactory.java:216)
at com.fasterxml.jackson.databind.ser.BeanSerializerFactory.createSerializer(BeanSerializerFactory.java:165)
at com.fasterxml.jackson.databind.SerializerProvider._createUntypedSerializer(SerializerProvider.java:1388)
at com.fasterxml.jackson.databind.SerializerProvider._createAndCacheUntypedSerializer(SerializerProvider.java:1336)
at com.fasterxml.jackson.databind.SerializerProvider.findValueSerializer(SerializerProvider.java:510)
at com.fasterxml.jackson.databind.SerializerProvider.findTypedValueSerializer(SerializerProvider.java:713)
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:308)
at com.fasterxml.jackson.databind.ObjectWriter$Prefetch.serialize(ObjectWriter.java:1396)
at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:913)
at com.fasterxml.jackson.jaxrs.base.ProviderBase.writeTo(ProviderBase.java:648)
at com.sun.jersey.api.client.RequestWriter.writeRequestEntity(RequestWriter.java:300)
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:204)
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:147)
at com.sun.jersey.api.client.filter.HTTPBasicAuthFilter.handle(HTTPBasicAuthFilter.java:81)
at com.sun.jersey.api.client.Client.handle(Client.java:648)
at com.sun.jersey.api.client.WebResource.handle(WebResource.java:670)
at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
at com.sun.jersey.api.client.WebResource$Builder.method(WebResource.java:623)
at org.apache.atlas.AtlasBaseClient.callAPIWithResource(AtlasBaseClient.java:382)
at org.apache.atlas.AtlasBaseClient.callAPIWithResource(AtlasBaseClient.java:353)
at org.apache.atlas.AtlasBaseClient.callAPI(AtlasBaseClient.java:229)
at org.apache.atlas.AtlasClientV2.createEntity(AtlasClientV2.java:320)
at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerInstance(HiveMetaStoreBridge.java:450)
at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerDatabase(HiveMetaStoreBridge.java:402)
at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importDatabases(HiveMetaStoreBridge.java:281)
at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importHiveMetadata(HiveMetaStoreBridge.java:251)
at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:168)
Failed to import Hive Meta Data!!!
解决:
1。下载 jackson-annotations-2.9.8.jar
, jackson-core-2.9.8.jar
、 jackson-databind-2.9.8.jar
,可以通过https://mvnrepository.com下载
2。然后把jar放入到 ${atlas_project}/distro/target/apache-atlas-2.0.0-bin/apache-atlas-2.0.0/hook/hive/atlas-hive-plugin-impl/下面
运行导入hive元数据
运行:bin/import-hive.sh 成功
浏览
左边选框可以选择
- 参考: