Skip to content

Poor performance when using rows to scan in data #119

@jimmiebtlr

Description

@jimmiebtlr

Describe the bug
When using rows to read row by row, the performance is around 10x worse than using raw sql.

Note, I also gave it a try using raw sql, and using qrm.ScanOneRowToDest, with the same poor performance seen there.

Some additional background

  • Using a struct with 3 not huge jet models (total probably 30 fields, 10 of which are duplicate with an AS clause, roughtly 10 fields used total out of them).
  • Reducing fields to 1 model with 11 fields cuts the time 50%
  • I'm returning 1M rows.
  • Similar code that uses Query or QueryContext instead of Rows is returning in around 25% the time.

My guess is the overhead of reflecting on every row is driving this behavior.

Environment (please complete the following information):

  • OS: macosx
  • Database: mysql
  • Database driver: mysql
  • Jet version 2.5

Code snippet

q, args := query.Sql()
rows, err := db.Query(q, args...)
// handle err and defer close

for rows.Next() {
	rowData := generator()
	err = qrm.ScanOneRowToDest(rows, rowData)
	// handle err
}

Expected behavior
I expect the results to take < 40% current time.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions