Skip to content

New batch migration causes error on sqlite #7348

@mrsdizzie

Description

@mrsdizzie

(cc @lunny)

With the changes in #7050, I now get these when testing a migration:

2019/07/03 11:52:30 routers/repo/repo.go:315:MigratePost() [E] MigratePost: too many SQL variables

My example happened for comments, based on this INSERT statement (I think):

gitea/models/migrate.go

Lines 121 to 127 in b5aa7f7

if _, err := sess.NoAutoTime().Insert(comments); err != nil {
return err
}
for issueID := range issueIDs {
if _, err := sess.Exec("UPDATE issue set num_comments = (SELECT count(*) FROM comment WHERE issue_id = ?) WHERE id = ?", issueID, issueID); err != nil {
return err
}

Here is the full gist:

https://gist.github.com/mrsdizzie/681ea0295c11350fea4244a4289665ef

This is just testing migrating the "tea" repo here: https://github.com/go-gitea/tea

But it could maybe happen other places too for similar reasons. For each comment there would be a few dozen variables in that SQL statement, so it breaks depending how many comments there are. Each comment adds about 22 variables to the statement now (one for each column), so even something like 50 total comments (say 10 issues with 5 comments each) is enough to trigger this error since I believe the default SQLITE_MAX_VARIABLE_NUMBER is 999 (22 * 50 = 1,100).

I know this is sort of a limit of sqlite and probably wouldn't happen with others -- but still an issue (and makes testing new migration features difficult since it is nice to use sqlite locally for development).

Maybe we can detect if sqlite and use the old method (or limit it to a known good number like 25 comments at a time)?

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions