Storage bugfixes #1923

bensmrs · 2025-04-11T15:21:13Z

No description provided.

bensmrs · 2025-04-11T15:32:18Z

I’ve tested on one node, with a first pool not restricted to any linstor storage pool, and another restricted to a specific linstor storage pool.
However, I’d like a double-check for clustered environments, as I can’t do it right now (although the single-node test shows that the iteration works as expected).
@luissimas maybe?

root@test-incus:~# incus storage show lpool1 | grep storage_pool
root@test-incus:~# incus storage info lpool1
info:
  description: ""
  driver: linstor
  name: lpool1
  space used: 2.97GiB
  total space: 99.79GiB
used by: {}
root@test-incus:~# incus storage show lpool2 | grep storage_pool
linstor.resource_group.storage_pool: incus2
root@test-incus:~# incus storage info lpool2
info:
  description: ""
  driver: linstor
  name: lpool2
  space used: 0B
  total space: 49.89GiB
used by: {}

luissimas · 2025-04-11T16:38:50Z

Nice!

I'll set up a clustered environment to test the behavior later today.

luissimas · 2025-04-11T20:15:26Z

Did a test with the changes in a clustered environment with multiple storage pools. Both the usage and total space are reported correctly.

root@server01 ~# linstor sp l
╭──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ StoragePool          ┊ Node     ┊ Driver   ┊ PoolName                          ┊ FreeCapacity ┊ TotalCapacity ┊ CanSnapshots ┊ State ┊ SharedName                    ┊
╞══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ DfltDisklessStorPool ┊ server01 ┊ DISKLESS ┊                                   ┊              ┊               ┊ False        ┊ Ok    ┊ server01;DfltDisklessStorPool ┊
┊ DfltDisklessStorPool ┊ server02 ┊ DISKLESS ┊                                   ┊              ┊               ┊ False        ┊ Ok    ┊ server02;DfltDisklessStorPool ┊
┊ DfltDisklessStorPool ┊ server03 ┊ DISKLESS ┊                                   ┊              ┊               ┊ False        ┊ Ok    ┊ server03;DfltDisklessStorPool ┊
┊ DfltDisklessStorPool ┊ server04 ┊ DISKLESS ┊                                   ┊              ┊               ┊ False        ┊ Ok    ┊ server04;DfltDisklessStorPool ┊
┊ hdd                  ┊ server01 ┊ ZFS      ┊ linstor-hdd                       ┊    18.89 GiB ┊     19.50 GiB ┊ True         ┊ Ok    ┊ server01;hdd                  ┊
┊ hdd                  ┊ server02 ┊ ZFS      ┊ linstor-hdd                       ┊    18.89 GiB ┊     19.50 GiB ┊ True         ┊ Ok    ┊ server02;hdd                  ┊
┊ hdd                  ┊ server03 ┊ ZFS      ┊ linstor-hdd                       ┊    18.89 GiB ┊     19.50 GiB ┊ True         ┊ Ok    ┊ server03;hdd                  ┊
┊ hdd                  ┊ server04 ┊ ZFS      ┊ linstor-hdd                       ┊    18.89 GiB ┊     19.50 GiB ┊ True         ┊ Ok    ┊ server04;hdd                  ┊
┊ nvme                 ┊ server01 ┊ LVM_THIN ┊ linstor_linstor-nvme/linstor-nvme ┊    49.89 GiB ┊     49.89 GiB ┊ True         ┊ Ok    ┊ server01;nvme                 ┊
┊ nvme                 ┊ server02 ┊ LVM_THIN ┊ linstor_linstor-nvme/linstor-nvme ┊    49.89 GiB ┊     49.89 GiB ┊ True         ┊ Ok    ┊ server02;nvme                 ┊
┊ nvme                 ┊ server03 ┊ LVM_THIN ┊ linstor_linstor-nvme/linstor-nvme ┊    49.89 GiB ┊     49.89 GiB ┊ True         ┊ Ok    ┊ server03;nvme                 ┊
┊ nvme                 ┊ server04 ┊ LVM_THIN ┊ linstor_linstor-nvme/linstor-nvme ┊    49.89 GiB ┊     49.89 GiB ┊ True         ┊ Ok    ┊ server04;nvme                 ┊
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
root@server01 ~# incus storage ls
+-------+---------+--------------------+---------+---------+
| NAME  | DRIVER  |    DESCRIPTION     | USED BY |  STATE  |
+-------+---------+--------------------+---------+---------+
| all   | linstor |                    | 0       | CREATED |
+-------+---------+--------------------+---------+---------+
| hdd   | linstor |                    | 0       | CREATED |
+-------+---------+--------------------+---------+---------+
| local | zfs     | Local storage pool | 0       | CREATED |
+-------+---------+--------------------+---------+---------+
| nvme  | linstor |                    | 0       | CREATED |
+-------+---------+--------------------+---------+---------+
root@server01 ~# incus storage show all | grep storage_pool
root@server01 ~ [0|1]# incus storage info all
info:
  description: ""
  driver: linstor
  name: all
  space used: 2.44GiB
  total space: 277.58GiB
used by: {}
root@server01 ~# incus storage show hdd | grep storage_pool
  linstor.resource_group.storage_pool: hdd
root@server01 ~# incus storage info hdd
info:
  description: ""
  driver: linstor
  name: hdd
  space used: 2.44GiB
  total space: 78.00GiB
used by: {}
root@server01 ~# incus storage show nvme | grep storage_pool
  linstor.resource_group.storage_pool: nvme
root@server01 ~# incus storage info nvme
info:
  description: ""
  driver: linstor
  name: nvme
  space used: 0B
  total space: 199.58GiB
used by: {}

bensmrs · 2025-04-11T20:38:40Z

Great, thanks for your time!

stgraber · 2025-04-11T20:39:05Z

Any idea what's going on with the Linstor test failure?
It's basically the one test we care about here ;)

stgraber · 2025-04-11T20:39:27Z

The documentation thing I think is a server banning access from Azure, so I'll have to find a way to skip this particular check.

bensmrs · 2025-04-11T20:41:56Z

Just run it again, I’m sure it’s just one of those transient unreproducible bugs that we had identified when doing the PR. (pretty sure it’s a bug upstream)

bensmrs · 2025-04-11T20:42:56Z

Although it would be nice to find a way to reliably reproduce the FS locking problem, @luissimas

internal/server/storage/backend.go

Closes: lxc#1921 Signed-off-by: Benjamin Somers <benjamin.somers@imt-atlantique.fr>

Closes: lxc#1917 Signed-off-by: Benjamin Somers <benjamin.somers@imt-atlantique.fr>

luissimas · 2025-04-14T18:00:11Z

Although it would be nice to find a way to reliably reproduce the FS locking problem, @luissimas

@bensmrs sorry for the late reply. I think we've already reproduced this error and reported it upstream in LINBIT/linstor-server#443. The problem is not the FS locking itself, but the snapshot creation in LINSTOR that fails internally and results in a response never being sent back to the client.

Now that I'm revisiting the problem, it might make sense to implement a timeout on our side to ensure that we don't get stuck in an operation that might never finish.

bensmrs · 2025-04-14T18:04:15Z

The CI error wasn’t in snapshots unfortunately; see https://github.com/lxc/incus/actions/runs/14406481905/job/40404180051.

luissimas · 2025-04-14T21:28:03Z

Yeah, it's on a move operation. But in the end, this still triggers a snapshot creation because the generic move implementation takes a snapshot of the volume to ensure consistency. This is why I get the feeling that it's the same snapshot error that we've reported upstream.

bensmrs · 2025-04-14T21:46:50Z

Ooooh I forgot about that :)

This MR contains the following updates: | Package | Update | Change | |---|---|---| | [lxc/incus](https://github.com/lxc/incus) | minor | `v6.11.0` -> `v6.12.0` | MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot). **Proposed changes to behavior should be submitted there as MRs.** --- ### Release Notes <details> <summary>lxc/incus (lxc/incus)</summary> ### [`v6.12.0`](https://github.com/lxc/incus/releases/tag/v6.12.0): Incus 6.12 [Compare Source](lxc/incus@v6.11.0...v6.12.0) ### Announcement https://discuss.linuxcontainers.org/t/incus-6-12-has-been-released/23556 #### What's Changed - doc: Fix missing OCI section by [@stgraber](https://github.com/stgraber) in lxc/incus#1855 - doc: Fix config option reference on LINSTOR driver by [@luissimas](https://github.com/luissimas) in lxc/incus#1857 - Add support for server-side filtering by instance name by [@presztak](https://github.com/presztak) in lxc/incus#1856 - incusd/instance/lxc: Fix max gid when in a privileged container by [@stgraber](https://github.com/stgraber) in lxc/incus#1859 - Fix some static analysis nits by [@stgraber](https://github.com/stgraber) in lxc/incus#1860 - README: Fix typo by [@stgraber](https://github.com/stgraber) in lxc/incus#1861 - Docs: correct restriction on `path` option by [@gwenya](https://github.com/gwenya) in lxc/incus#1862 - lxd-to-incus: Fix typo in trigger by [@stgraber](https://github.com/stgraber) in lxc/incus#1865 - incusd/instance/edk2: Limit test to UEFI architectures by [@stgraber](https://github.com/stgraber) in lxc/incus#1866 - doc: Tweak ACME documentation by [@stgraber](https://github.com/stgraber) in lxc/incus#1868 - instances/drivers/qemu: update user parameter for QEMU v9.1+ by [@dnegreira](https://github.com/dnegreira) in lxc/incus#1871 - OCI improvements by [@stgraber](https://github.com/stgraber) in lxc/incus#1873 - Support server-side filtering by [@presztak](https://github.com/presztak) in lxc/incus#1872 - Enable filtering with the `all-projects` flag when listing images by [@presztak](https://github.com/presztak) in lxc/incus#1874 - Improve migration by [@stgraber](https://github.com/stgraber) in lxc/incus#1878 - incusd/storage: Add missing forwarding on snapshot list by [@stgraber](https://github.com/stgraber) in lxc/incus#1882 - incusd/instance/common: Fix concurrent restarts by [@stgraber](https://github.com/stgraber) in lxc/incus#1884 - Fix all static analysis in client/, shared/ and cmd/incus/ by [@stgraber](https://github.com/stgraber) in lxc/incus#1883 - generate-database: Fix documentation for `ignore` by [@breml](https://github.com/breml) in lxc/incus#1885 - incusd/response: Remove redundant line break in error by [@stgraber](https://github.com/stgraber) in lxc/incus#1886 - RFC 3442 compliance in forknet dhcp client by [@gwenya](https://github.com/gwenya) in lxc/incus#1887 - incus-agent: Retry mounts to avoid kernel races by [@stgraber](https://github.com/stgraber) in lxc/incus#1888 - Address sets for nftables and OVN by [@irhndt](https://github.com/irhndt) in lxc/incus#1728 - incusd/operations: Fix WaitGet on op failure by [@stgraber](https://github.com/stgraber) in lxc/incus#1894 - Update list of compresors by [@stgraber](https://github.com/stgraber) in lxc/incus#1892 - Add snapshot pre-fetching support by [@stgraber](https://github.com/stgraber) in lxc/incus#1891 - incusd/instance/lxc: Use pre-existing PATH when not overridden by [@stgraber](https://github.com/stgraber) in lxc/incus#1895 - incusd/acme: Include CA in generate certificate by [@stgraber](https://github.com/stgraber) in lxc/incus#1897 - Usability improvements to incus-migrate by [@stgraber](https://github.com/stgraber) in lxc/incus#1898 - client/incus: Fix non-constant format strings by [@c4t3l](https://github.com/c4t3l) in lxc/incus#1899 - docs: mDNS setup for cluster HA by [@MOZGIII](https://github.com/MOZGIII) in lxc/incus#1896 - Support filtering storage volumes by a single keyword by [@presztak](https://github.com/presztak) in lxc/incus#1915 - incusd/instance/qemu: Clean leftover sockets on startup by [@stgraber](https://github.com/stgraber) in lxc/incus#1916 - incusd: Implement Incus OS API forwarding by [@stgraber](https://github.com/stgraber) in lxc/incus#1918 - Add generated documentation for network bridge by [@NathanChase22](https://github.com/NathanChase22) in lxc/incus#1920 - doc: Use `$USER` instead of YOUR-USERNAME by [@bjackman](https://github.com/bjackman) in lxc/incus#1922 - doc: Ignore link that's blocking Azure by [@stgraber](https://github.com/stgraber) in lxc/incus#1924 - Storage bugfixes by [@bensmrs](https://github.com/bensmrs) in lxc/incus#1923 - incusd/patches: Refresh OpenFGA model for address sets by [@stgraber](https://github.com/stgraber) in lxc/incus#1925 - Add generated documentation for network forwards by [@tonyn10](https://github.com/tonyn10) in lxc/incus#1926 - Add support for configurable logging targets by [@presztak](https://github.com/presztak) in lxc/incus#1903 - Port tpm device documentation to gendoc by [@saahirN](https://github.com/saahirN) in lxc/incus#1929 - Allow basic connectivity under nftables by [@stgraber](https://github.com/stgraber) in lxc/incus#1930 - incusd/storage/zfs: Make CacheVolumeSnapshots failures non-fatal by [@stgraber](https://github.com/stgraber) in lxc/incus#1931 - incusd/instance/lxc: Restrict unprivileged ping to recent kernels by [@stgraber](https://github.com/stgraber) in lxc/incus#1934 - Implement SNAT as part of network forwards by [@stgraber](https://github.com/stgraber) in lxc/incus#1935 - incusd/apparmor/lxc: Allow write access to /proc/sys/user by [@zgttotev](https://github.com/zgttotev) in lxc/incus#1937 - incusd/instance/lxc: Defer calls to the scheduler by [@stgraber](https://github.com/stgraber) in lxc/incus#1938 - shared/archive: Prevent xattr errors from crashing unsquashfs by [@zgttotev](https://github.com/zgttotev) in lxc/incus#1939 - Extend use of ZFS pre-caching by [@stgraber](https://github.com/stgraber) in lxc/incus#1941 - Add common aliases for add/create remove/delete/rm in the CLI by [@joecwilson](https://github.com/joecwilson) in lxc/incus#1943 - feat: support access_token query parameter as JWT fallback by [@irtaza9](https://github.com/irtaza9) in lxc/incus#1940 - Memory hotplug support for VMs by [@presztak](https://github.com/presztak) in lxc/incus#1945 - incusd: Remove old routing logic by [@stgraber](https://github.com/stgraber) in lxc/incus#1947 - Fix refresh migrations in cluster and speed up ZFS startup by [@stgraber](https://github.com/stgraber) in lxc/incus#1946 - incusd/devices: Don't require a serial number for USB hotplug by [@stgraber](https://github.com/stgraber) in lxc/incus#1949 - Move tls testing functions to tlstest by [@nanjj](https://github.com/nanjj) in lxc/incus#1948 - Remove Rican7/retry dependency by [@nanjj](https://github.com/nanjj) in lxc/incus#1952 - Port `proxy` device documentation to `gendoc` by [@Abdomash](https://github.com/Abdomash) in lxc/incus#1953 - Port gpu device documentation to gendoc by [@kmxtn](https://github.com/kmxtn) in lxc/incus#1954 - Port nic device documentation to gendoc by [@rahafjrw](https://github.com/rahafjrw) in lxc/incus#1956 - Remove arping dependency by [@ahmetfturhan](https://github.com/ahmetfturhan) in lxc/incus#1958 - Remove gocapability dependency by [@nanjj](https://github.com/nanjj) in lxc/incus#1957 - Infiniband Device Documentation Ported to GenDoc by [@AbhinavTiruvee](https://github.com/AbhinavTiruvee) in lxc/incus#1962 - Replace rebfig/cron/v3 with adhocore/gronx by [@nanjj](https://github.com/nanjj) in lxc/incus#1959 - Update help of `incus storage list` by [@stgraber](https://github.com/stgraber) in lxc/incus#1968 - shared/api/scriptlet: Add yaml struct tags by [@breml](https://github.com/breml) in lxc/incus#1973 - incusd/storage/migration: Check instance size during migration by [@stgraber](https://github.com/stgraber) in lxc/incus#1971 - Logfile for forknet dhcp by [@gwenya](https://github.com/gwenya) in lxc/incus#1976 - Add dhcp static routes via 0.0.0.0 with link scope in forknet by [@gwenya](https://github.com/gwenya) in lxc/incus#1977 - incusd/device/disk: Fix registration of custom volumes by [@stgraber](https://github.com/stgraber) in lxc/incus#1980 - Add server side filtering for `incus profile list` by [@Abdomash](https://github.com/Abdomash) in lxc/incus#1982 - Fix reference passing when yaml unmarshal by [@nanjj](https://github.com/nanjj) in lxc/incus#1984 - Various fixes by [@stgraber](https://github.com/stgraber) in lxc/incus#1983 - scriptlet: Return proper error by [@breml](https://github.com/breml) in lxc/incus#1986 - incusd/instance: Also consider local CPU flags by [@stgraber](https://github.com/stgraber) in lxc/incus#1987 - Cap maximum VM memory to match host memory total by [@stgraber](https://github.com/stgraber) in lxc/incus#1988 #### New Contributors - [@dnegreira](https://github.com/dnegreira) made their first contribution in lxc/incus#1871 - [@c4t3l](https://github.com/c4t3l) made their first contribution in lxc/incus#1899 - [@MOZGIII](https://github.com/MOZGIII) made their first contribution in lxc/incus#1896 - [@NathanChase22](https://github.com/NathanChase22) made their first contribution in lxc/incus#1920 - [@bjackman](https://github.com/bjackman) made their first contribution in lxc/incus#1922 - [@tonyn10](https://github.com/tonyn10) made their first contribution in lxc/incus#1926 - [@saahirN](https://github.com/saahirN) made their first contribution in lxc/incus#1929 - [@zgttotev](https://github.com/zgttotev) made their first contribution in lxc/incus#1937 - [@joecwilson](https://github.com/joecwilson) made their first contribution in lxc/incus#1943 - [@irtaza9](https://github.com/irtaza9) made their first contribution in lxc/incus#1940 - [@Abdomash](https://github.com/Abdomash) made their first contribution in lxc/incus#1953 - [@kmxtn](https://github.com/kmxtn) made their first contribution in lxc/incus#1954 - [@rahafjrw](https://github.com/rahafjrw) made their first contribution in lxc/incus#1956 - [@ahmetfturhan](https://github.com/ahmetfturhan) made their first contribution in lxc/incus#1958 - [@AbhinavTiruvee](https://github.com/AbhinavTiruvee) made their first contribution in lxc/incus#1962 **Full Changelog**: lxc/incus@v6.11.0...v6.12.0 </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this MR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this MR, check this box --- This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).

bensmrs requested a review from stgraber as a code owner April 11, 2025 15:21

stgraber reviewed Apr 11, 2025

View reviewed changes

internal/server/storage/backend.go Outdated Show resolved Hide resolved

bensmrs added 2 commits April 11, 2025 21:49

incusd/storage: Avoid querying pending pool status

720b4ad

Closes: lxc#1921 Signed-off-by: Benjamin Somers <benjamin.somers@imt-atlantique.fr>

incusd/storage/linstor: Fix pool size reporting

681fa92

Closes: lxc#1917 Signed-off-by: Benjamin Somers <benjamin.somers@imt-atlantique.fr>

bensmrs force-pushed the main branch from aa7fb41 to 681fa92 Compare April 11, 2025 21:50

stgraber approved these changes Apr 11, 2025

View reviewed changes

stgraber merged commit edde717 into lxc:main Apr 11, 2025
37 of 38 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Storage bugfixes #1923

Storage bugfixes #1923

Uh oh!

bensmrs commented Apr 11, 2025

Uh oh!

bensmrs commented Apr 11, 2025

Uh oh!

luissimas commented Apr 11, 2025

Uh oh!

luissimas commented Apr 11, 2025 •

edited

Loading

Uh oh!

bensmrs commented Apr 11, 2025

Uh oh!

stgraber commented Apr 11, 2025

Uh oh!

stgraber commented Apr 11, 2025

Uh oh!

bensmrs commented Apr 11, 2025

Uh oh!

bensmrs commented Apr 11, 2025

Uh oh!

Uh oh!

Uh oh!

luissimas commented Apr 14, 2025

Uh oh!

bensmrs commented Apr 14, 2025

Uh oh!

luissimas commented Apr 14, 2025

Uh oh!

bensmrs commented Apr 14, 2025

Uh oh!

Uh oh!

Uh oh!

Storage bugfixes #1923

Storage bugfixes #1923

Uh oh!

Conversation

bensmrs commented Apr 11, 2025

Uh oh!

bensmrs commented Apr 11, 2025

Uh oh!

luissimas commented Apr 11, 2025

Uh oh!

luissimas commented Apr 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bensmrs commented Apr 11, 2025

Uh oh!

stgraber commented Apr 11, 2025

Uh oh!

stgraber commented Apr 11, 2025

Uh oh!

bensmrs commented Apr 11, 2025

Uh oh!

bensmrs commented Apr 11, 2025

Uh oh!

Uh oh!

Uh oh!

luissimas commented Apr 14, 2025

Uh oh!

bensmrs commented Apr 14, 2025

Uh oh!

luissimas commented Apr 14, 2025

Uh oh!

bensmrs commented Apr 14, 2025

Uh oh!

Uh oh!

luissimas commented Apr 11, 2025 •

edited

Loading