Skip to content

BigQuery insert_data() fails with record type columns (nested objects) #2951

@tswast

Description

@tswast

BigQuery tabledata.insertAll supports inserting rows with columns that take record types (nested objects).

A customer reports that

My table structure has nested schemas. The [google-cloud-python] docs don't say how to handle nested structures. All we get is, "rows (list of tuples) – Row data to be inserted. Each tuple should contain data for each schema field on the current table and in the same order as the schema fields." A list of tuples. That's it. If I have nested data do I make sub-tuples? Is it simply not supported at this time?

I dug into the BQ code to find out what was going on. BQ does,

zip(table._schema, rows)

which looks like it won't support nested inserts (or could -- with sub-tuples, but who knows).

Example in the API explorer, with the following table schema:

image

Try it out in the API Explorer for bigquery.tabledata.insertAll. Enter the following for the request body (after creating a table with the same schema). You need to switch the editor to free-form mode

{
  "rows": 
  [
    {
      "insert_id": "12345",
      "json": 
      {
        "string_col": "Some value",
        "record_col": {
          "nested_string": "another string value",
          "nested_repeated": ["0", "1", "2"],
          "nested_record": {
            "nested_nested_string": "some deep insight"
          }
        }
      }
    }
  ]
}

I can verify that this works with a SQL query.

#standardSql
SELECT * FROM `swast-scratch.hello_world.nested` LIMIT 1000

image

It is not clear from the docs how to do a similar call from our client libraries (and it likely isn't possible).

Metadata

Metadata

Assignees

Labels

api: bigqueryIssues related to the BigQuery API.priority: p0Highest priority. Critical issue. P0 implies highest priority.release blockingRequired feature/issue must be fixed prior to next release.type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions