Skipping missing tables for leaf-collections #6397
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
1 Skipped Deployment
|
| @@ -30,7 +30,7 @@ def format_clause_for_query( | |||
| """Returns field names in clauses surrounded by quotation marks as required by Snowflake syntax.""" | |||
| return f'"{string_path}" {operator} (:{operand})' | |||
|
|
|||
| def _generate_table_name(self) -> str: | |||
| def generate_table_name(self) -> str: | |||
There was a problem hiding this comment.
Changing this from private to public
| try: | ||
| self.set_schema(connection) | ||
| if ( | ||
| query_config.partitioning | ||
| ): # only BigQuery supports partitioning, for now | ||
| return self.partitioned_retrieval(query_config, connection, stmt) | ||
|
|
||
| results = connection.execute(stmt) | ||
| return self.cursor_result_to_rows(results) | ||
| except Exception as exc: | ||
| # Check if table exists using qualified table name | ||
| if not self.table_exists(connection, node): | ||
| # Central decision point - will raise TableNotFound or ConnectionException | ||
| qualified_table_name = self.get_qualified_table_name(node) | ||
| self.handle_table_not_found( | ||
| node=node, | ||
| table_name=qualified_table_name, | ||
| operation_context="data retrieval", | ||
| original_exception=exc, | ||
| ) | ||
| # Table exists or can't check - re-raise original exception | ||
| raise |
There was a problem hiding this comment.
Central handling for missing tables. Instead of checking for table existence before each query, we check after any exception. The other way to check for missing tables would be to parse various exceptions for missing tables but that seemed too fragile.
| """ | ||
| return node.collection.name | ||
|
|
||
| def table_exists(self, connection: Connection, node: ExecutionNode) -> bool: |
There was a problem hiding this comment.
Generic way to check for table existence using SQLAlchemy. If this fails in any way we assume the table exists to maintain the current behavior.
Codecov Report❌ Patch coverage is ❌ Your patch status has failed because the patch coverage (36.36%) is below the target coverage (100.00%). You can increase the patch coverage or adjust the target coverage. Additional details and impacted files@@ Coverage Diff @@
## main #6397 +/- ##
==========================================
- Coverage 87.14% 87.00% -0.14%
==========================================
Files 455 455
Lines 29179 29252 +73
Branches 3244 3256 +12
==========================================
+ Hits 25427 25451 +24
- Misses 3025 3069 +44
- Partials 727 732 +5 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
fides
|
||||||||||||||||||||||||||||||
| Project |
fides
|
| Branch Review |
main
|
| Run status |
|
| Run duration | 00m 53s |
| Commit |
|
| Committer | Adrian Galvan |
| View all properties for this run ↗︎ | |
| Test results | |
|---|---|
Failures
|
0
|
Flaky
|
0
|
Pending
|
0
|
Skipped
|
0
|
Passing
|
5
|
Upgrade your plan to view test results. | |
| View all changes introduced in this branch ↗︎ | |
Closes ENG-1051
Description Of Changes
Added new behavior for
retrieve_datarequests with missing database tables.Code Changes
SQLConnector.retrieve_dataget_qualified_table_nametoSQLConnector(overridden byBigQueryConnectorandSnowflakeConnector)table_existstoSQLConnectorSteps to Confirm
python qa bigquery_table_not_found setupQA scenariocustomer-1@example.comcustomer_profile_missingcollection is skippedpython qa bigquery_table_not_found setup --table customerQA scenario (note thecustomerparam)customer-1@example.comcustomer_missingcollection has downstream dependenciesPre-Merge Checklist
CHANGELOG.mdupdatedmaindowngrade()migration is correct and works