Willingness to contribute

Yes. I can contribute a fix for this bug independently.

Feathr version

0.10.4-rc1

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 20.0): Databricks 9.1
• Python version: 3.9
• Spark version, if reporting runtime issue: 3.1

Describe the problem

For Databricks Cluster, despite the azure.cosmos.spark jar being available in the feathr fat jar, still get this error for ClassNotFound.

2023-01-20 19:41:26.349 | ERROR | feathr.spark_provider._databricks_submission:wait_for_completion:290 - Feathr job has failed. Please visit this page to view error message: ***?o=5638037984879289#job/521696308781092/run/15238504
2023-01-20 19:41:26.349 | ERROR | feathr.spark_provider._databricks_submission:wait_for_completion:293 - Error Code: Cannot find catalog plugin class for catalog 'cosmosCatalog': com.azure.cosmos.spark.CosmosCatalog
2023-01-20 19:41:26.349 | ERROR | feathr.spark_provider._databricks_submission:wait_for_completion:295 - at org.apache.spark.sql.connector.catalog.Catalogs$.load(Catalogs.scala:73)
at org.apache.spark.sql.connector.catalog.CatalogManager.$anonfun$catalog$1(CatalogManager.scala:54)

Tracking information

https://github.com/feathr-ai/feathr/actions/runs/3970673005/jobs/6806673061

Code to reproduce bug

No response

What component(s) does this bug affect?

• Python Client: This is the client users use to interact with most of our API. Mostly written in Python.
• Computation Engine: The computation engine that execute the actual feature join and generation work. Mostly in Scala and Spark.
• Feature Registry API: The frontend API layer supports SQL, Purview(Atlas) as storage. The API layer is in Python(FAST API)
• Feature Registry Web UI: The Web UI for feature registry. Written in React