Incompatibility with Google BigTable due to namespace descriptors #123
Description
We are trying to use SHC to connect to a BigTable instance. However, since 2.1, it seems that SHC makes use of HBase namespace descriptors, which are not supported by BigTable. As a result BigTable operations using SHC are failing with below exception:
java.lang.UnsupportedOperationException: getNamespaceDescriptor
at org.apache.hadoop.hbase.client.AbstractBigtableAdmin.getNamespaceDescriptor(AbstractBigtableAdmin.java:1264)
at org.apache.spark.sql.execution.datasources.hbase.HBaseRelation.createTable(HBaseRelation.scala:128)
at org.apache.spark.sql.execution.datasources.hbase.DefaultSource.createRelation(HBaseRelation.scala:60)
at org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:457)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:211)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:237)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:280)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:214)
at java.lang.Thread.run(Thread.java:745)
I understand that it may not be a goal of Hortonworks to support BigTable, but is there any workaround to this issue, except sticking to version 2.0? Can we plan a fix so that compatibility with BigTable is supported?