This repository was archived by the owner on Dec 7, 2023. It is now read-only.
This repository was archived by the owner on Dec 7, 2023. It is now read-only.
Concurrent VM Startup fails #510
Description
When creating multiple vm's concurrently, there appears to be a race and/or contention that causes panics.
We've known about this for a while, but may have not ever created an issue.
Users have been working around this by doing strictly serial creation of vm's in their automation/integration.
This is potentially rrelated to our Mount code :))
This is easiest to reproduce with large numbers of vm's:
sudo ignite run weaveworks/ignite-ubuntu --name concurrent-${RANDOM} --ssh & \
sudo ignite run weaveworks/ignite-ubuntu --name concurrent-${RANDOM} --ssh & \
sudo ignite run weaveworks/ignite-ubuntu --name concurrent-${RANDOM} --ssh & \
sudo ignite run weaveworks/ignite-ubuntu --name concurrent-${RANDOM} --ssh & \
sudo ignite run weaveworks/ignite-ubuntu --name concurrent-${RANDOM} --ssh & \
[1] 7595
[2] 7596
[3] 7598
[4] 7599
[5] 7601
[6] 7604
~
❯ INFO[0000] Removed VM with name "concurrent-12507" and ID "c998fec1591a21ae"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x796b7f]
goroutine 1 [running]:
github.com/weaveworks/ignite/pkg/util.(*MountPoint).Umount(0x0, 0x1298b40, 0xc000206160)
/go/src/github.com/weaveworks/ignite/pkg/util/fs.go:108 +0x3f
github.com/weaveworks/ignite/pkg/dmlegacy.copyToOverlay(0xc000324000, 0x1298b40, 0xc000206160)
/go/src/github.com/weaveworks/ignite/pkg/dmlegacy/vm.go:95 +0x65e
github.com/weaveworks/ignite/pkg/dmlegacy.AllocateAndPopulateOverlay(0xc000324000, 0x0, 0x0)
/go/src/github.com/weaveworks/ignite/pkg/dmlegacy/vm.go:82 +0x5df
github.com/weaveworks/ignite/cmd/ignite/run.Create(0xc00000d5c0, 0x0, 0x0)
/go/src/github.com/weaveworks/ignite/cmd/ignite/run/create.go:125 +0x12b
github.com/weaveworks/ignite/cmd/ignite/run.Run(0xc000349710, 0xc00029d740, 0x1)
/go/src/github.com/weaveworks/ignite/cmd/ignite/run/run.go:31 +0x32
github.com/weaveworks/ignite/cmd/ignite/cmd/vmcmd.NewCmdRun.func1.1(0xc000500d00, 0xc00029d740, 0x1, 0x4, 0x0, 0x0)
/go/src/github.com/weaveworks/ignite/cmd/ignite/cmd/vmcmd/run.go:46 +0x7e
github.com/weaveworks/ignite/cmd/ignite/cmd/vmcmd.NewCmdRun.func1(0xc0002f6500, 0xc00029d740, 0x1, 0x4)
/go/src/github.com/weaveworks/ignite/cmd/ignite/cmd/vmcmd/run.go:47 +0x48
github.com/spf13/cobra.(*Command).execute(0xc0002f6500, 0xc00029d700, 0x4, 0x4, 0xc0002f6500, 0xc00029d700)
/go/src/github.com/weaveworks/ignite/vendor/github.com/spf13/cobra/command.go:830 +0x2ae
github.com/spf13/cobra.(*Command).ExecuteC(0xc000110f00, 0xc00000e010, 0x129a420, 0xc00000e018)
/go/src/github.com/weaveworks/ignite/vendor/github.com/spf13/cobra/command.go:914 +0x2fc
github.com/spf13/cobra.(*Command).Execute(...)
/go/src/github.com/weaveworks/ignite/vendor/github.com/spf13/cobra/command.go:864
main.Run(0xc0004e9f88, 0xc0000400b8)
/go/src/github.com/weaveworks/ignite/cmd/ignite/ignite.go:24 +0xb5
main.main()
/go/src/github.com/weaveworks/ignite/cmd/ignite/ignite.go:13 +0x22cc @darkowlzz