Skip to content

Strange display of volatile.cpu.nodes after setting nodes #2121

@evadne

Description

@evadne

Incus 6.12

Is there an existing issue for this?

  • There is no existing issue for this bug

Is this happening on an up to date version of Incus?

  • This is happening on a supported version of Incus

Incus system details

Content
config:
  core.https_address: '[::]:8443'
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_stop_priority
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
- network_sriov
- console
- restrict_dev_incus
- migration_pre_copy
- infiniband
- dev_incus_events
- proxy
- network_dhcp_gateway
- file_get_symlink
- network_leases
- unix_device_hotplug
- storage_api_local_volume_handling
- operation_description
- clustering
- event_lifecycle
- storage_api_remote_volume_handling
- nvidia_runtime
- container_mount_propagation
- container_backup
- dev_incus_images
- container_local_cross_pool_handling
- proxy_unix
- proxy_udp
- clustering_join
- proxy_tcp_udp_multi_port_handling
- network_state
- proxy_unix_dac_properties
- container_protection_delete
- unix_priv_drop
- pprof_http
- proxy_haproxy_protocol
- network_hwaddr
- proxy_nat
- network_nat_order
- container_full
- backup_compression
- nvidia_runtime_config
- storage_api_volume_snapshots
- storage_unmapped
- projects
- network_vxlan_ttl
- container_incremental_copy
- usb_optional_vendorid
- snapshot_scheduling
- snapshot_schedule_aliases
- container_copy_project
- clustering_server_address
- clustering_image_replication
- container_protection_shift
- snapshot_expiry
- container_backup_override_pool
- snapshot_expiry_creation
- network_leases_location
- resources_cpu_socket
- resources_gpu
- resources_numa
- kernel_features
- id_map_current
- event_location
- storage_api_remote_volume_snapshots
- network_nat_address
- container_nic_routes
- cluster_internal_copy
- seccomp_notify
- lxc_features
- container_nic_ipvlan
- network_vlan_sriov
- storage_cephfs
- container_nic_ipfilter
- resources_v2
- container_exec_user_group_cwd
- container_syscall_intercept
- container_disk_shift
- storage_shifted
- resources_infiniband
- daemon_storage
- instances
- image_types
- resources_disk_sata
- clustering_roles
- images_expiry
- resources_network_firmware
- backup_compression_algorithm
- ceph_data_pool_name
- container_syscall_intercept_mount
- compression_squashfs
- container_raw_mount
- container_nic_routed
- container_syscall_intercept_mount_fuse
- container_disk_ceph
- virtual-machines
- image_profiles
- clustering_architecture
- resources_disk_id
- storage_lvm_stripes
- vm_boot_priority
- unix_hotplug_devices
- api_filtering
- instance_nic_network
- clustering_sizing
- firewall_driver
- projects_limits
- container_syscall_intercept_hugetlbfs
- limits_hugepages
- container_nic_routed_gateway
- projects_restrictions
- custom_volume_snapshot_expiry
- volume_snapshot_scheduling
- trust_ca_certificates
- snapshot_disk_usage
- clustering_edit_roles
- container_nic_routed_host_address
- container_nic_ipvlan_gateway
- resources_usb_pci
- resources_cpu_threads_numa
- resources_cpu_core_die
- api_os
- container_nic_routed_host_table
- container_nic_ipvlan_host_table
- container_nic_ipvlan_mode
- resources_system
- images_push_relay
- network_dns_search
- container_nic_routed_limits
- instance_nic_bridged_vlan
- network_state_bond_bridge
- usedby_consistency
- custom_block_volumes
- clustering_failure_domains
- resources_gpu_mdev
- console_vga_type
- projects_limits_disk
- network_type_macvlan
- network_type_sriov
- container_syscall_intercept_bpf_devices
- network_type_ovn
- projects_networks
- projects_networks_restricted_uplinks
- custom_volume_backup
- backup_override_name
- storage_rsync_compression
- network_type_physical
- network_ovn_external_subnets
- network_ovn_nat
- network_ovn_external_routes_remove
- tpm_device_type
- storage_zfs_clone_copy_rebase
- gpu_mdev
- resources_pci_iommu
- resources_network_usb
- resources_disk_address
- network_physical_ovn_ingress_mode
- network_ovn_dhcp
- network_physical_routes_anycast
- projects_limits_instances
- network_state_vlan
- instance_nic_bridged_port_isolation
- instance_bulk_state_change
- network_gvrp
- instance_pool_move
- gpu_sriov
- pci_device_type
- storage_volume_state
- network_acl
- migration_stateful
- disk_state_quota
- storage_ceph_features
- projects_compression
- projects_images_remote_cache_expiry
- certificate_project
- network_ovn_acl
- projects_images_auto_update
- projects_restricted_cluster_target
- images_default_architecture
- network_ovn_acl_defaults
- gpu_mig
- project_usage
- network_bridge_acl
- warnings
- projects_restricted_backups_and_snapshots
- clustering_join_token
- clustering_description
- server_trusted_proxy
- clustering_update_cert
- storage_api_project
- server_instance_driver_operational
- server_supported_storage_drivers
- event_lifecycle_requestor_address
- resources_gpu_usb
- clustering_evacuation
- network_ovn_nat_address
- network_bgp
- network_forward
- custom_volume_refresh
- network_counters_errors_dropped
- metrics
- image_source_project
- clustering_config
- network_peer
- linux_sysctl
- network_dns
- ovn_nic_acceleration
- certificate_self_renewal
- instance_project_move
- storage_volume_project_move
- cloud_init
- network_dns_nat
- database_leader
- instance_all_projects
- clustering_groups
- ceph_rbd_du
- instance_get_full
- qemu_metrics
- gpu_mig_uuid
- event_project
- clustering_evacuation_live
- instance_allow_inconsistent_copy
- network_state_ovn
- storage_volume_api_filtering
- image_restrictions
- storage_zfs_export
- network_dns_records
- storage_zfs_reserve_space
- network_acl_log
- storage_zfs_blocksize
- metrics_cpu_seconds
- instance_snapshot_never
- certificate_token
- instance_nic_routed_neighbor_probe
- event_hub
- agent_nic_config
- projects_restricted_intercept
- metrics_authentication
- images_target_project
- images_all_projects
- cluster_migration_inconsistent_copy
- cluster_ovn_chassis
- container_syscall_intercept_sched_setscheduler
- storage_lvm_thinpool_metadata_size
- storage_volume_state_total
- instance_file_head
- instances_nic_host_name
- image_copy_profile
- container_syscall_intercept_sysinfo
- clustering_evacuation_mode
- resources_pci_vpd
- qemu_raw_conf
- storage_cephfs_fscache
- network_load_balancer
- vsock_api
- instance_ready_state
- network_bgp_holdtime
- storage_volumes_all_projects
- metrics_memory_oom_total
- storage_buckets
- storage_buckets_create_credentials
- metrics_cpu_effective_total
- projects_networks_restricted_access
- storage_buckets_local
- loki
- acme
- internal_metrics
- cluster_join_token_expiry
- remote_token_expiry
- init_preseed
- storage_volumes_created_at
- cpu_hotplug
- projects_networks_zones
- network_txqueuelen
- cluster_member_state
- instances_placement_scriptlet
- storage_pool_source_wipe
- zfs_block_mode
- instance_generation_id
- disk_io_cache
- amd_sev
- storage_pool_loop_resize
- migration_vm_live
- ovn_nic_nesting
- oidc
- network_ovn_l3only
- ovn_nic_acceleration_vdpa
- cluster_healing
- instances_state_total
- auth_user
- security_csm
- instances_rebuild
- numa_cpu_placement
- custom_volume_iso
- network_allocations
- zfs_delegate
- storage_api_remote_volume_snapshot_copy
- operations_get_query_all_projects
- metadata_configuration
- syslog_socket
- event_lifecycle_name_and_project
- instances_nic_limits_priority
- disk_initial_volume_configuration
- operation_wait
- image_restriction_privileged
- cluster_internal_custom_volume_copy
- disk_io_bus
- storage_cephfs_create_missing
- instance_move_config
- ovn_ssl_config
- certificate_description
- disk_io_bus_virtio_blk
- loki_config_instance
- instance_create_start
- clustering_evacuation_stop_options
- boot_host_shutdown_action
- agent_config_drive
- network_state_ovn_lr
- image_template_permissions
- storage_bucket_backup
- storage_lvm_cluster
- shared_custom_block_volumes
- auth_tls_jwt
- oidc_claim
- device_usb_serial
- numa_cpu_balanced
- image_restriction_nesting
- network_integrations
- instance_memory_swap_bytes
- network_bridge_external_create
- network_zones_all_projects
- storage_zfs_vdev
- container_migration_stateful
- profiles_all_projects
- instances_scriptlet_get_instances
- instances_scriptlet_get_cluster_members
- instances_scriptlet_get_project
- network_acl_stateless
- instance_state_started_at
- networks_all_projects
- network_acls_all_projects
- storage_buckets_all_projects
- resources_load
- instance_access
- project_access
- projects_force_delete
- resources_cpu_flags
- disk_io_bus_cache_filesystem
- instance_oci
- clustering_groups_config
- instances_lxcfs_per_instance
- clustering_groups_vm_cpu_definition
- disk_volume_subpath
- projects_limits_disk_pool
- network_ovn_isolated
- qemu_raw_qmp
- network_load_balancer_health_check
- oidc_scopes
- network_integrations_peer_name
- qemu_scriptlet
- instance_auto_restart
- storage_lvm_metadatasize
- ovn_nic_promiscuous
- ovn_nic_ip_address_none
- instances_state_os_info
- network_load_balancer_state
- instance_nic_macvlan_mode
- storage_lvm_cluster_create
- network_ovn_external_interfaces
- instances_scriptlet_get_instances_count
- cluster_rebalance
- custom_volume_refresh_exclude_older_snapshots
- storage_initial_owner
- storage_live_migration
- instance_console_screenshot
- image_import_alias
- authorization_scriptlet
- console_force
- network_ovn_state_addresses
- network_bridge_acl_devices
- instance_debug_memory
- init_preseed_storage_volumes
- init_preseed_profile_project
- instance_nic_routed_host_address
- instance_smbios11
- api_filtering_extended
- acme_dns01
- security_iommu
- network_ipv4_dhcp_routes
- network_state_ovn_ls
- network_dns_nameservers
- acme_http01_port
- network_ovn_ipv4_dhcp_expiry
- instance_state_cpu_time
- network_io_bus
- disk_io_bus_usb
- storage_driver_linstor
- instance_oci_entrypoint
- network_address_set
- server_logging
- network_forward_snat
- memory_hotplug
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
auth_user_name: evadne
auth_user_method: unix
environment:
  addresses:
  - 192.168.255.100:8443
  - 192.168.255.101:8443
  - 192.168.255.77:8443
  - 192.168.254.14:8443
  - 192.168.255.198:8443
  - 192.168.254.12:8443
  - 10.10.10.1:8443
  architectures:
  - x86_64
  - i686
  certificate: |
    -----BEGIN CERTIFICATE-----
    -----END CERTIFICATE-----
  certificate_fingerprint: 0c8157d132d99c8cfda35c8f60edb4a92253385472241bb3cf15998acc4d5396
  driver: lxc | qemu
  driver_version: 6.0.4 | 9.0.4
  firewall: nftables
  kernel: Linux
  kernel_architecture: x86_64
  kernel_features:
    idmapped_mounts: "true"
    netnsid_getifaddrs: "true"
    seccomp_listener: "true"
    seccomp_listener_continue: "true"
    uevent_injection: "true"
    unpriv_binfmt: "true"
    unpriv_fscaps: "true"
  kernel_version: 6.8.0-60-generic
  lxc_features:
    cgroup2: "true"
    core_scheduling: "true"
    devpts_fd: "true"
    idmapped_mounts_v2: "true"
    mount_injection_file: "true"
    network_gateway_device_route: "true"
    network_ipvlan: "true"
    network_l2proxy: "true"
    network_phys_macvlan_mtu: "true"
    network_veth_router: "true"
    pidfd: "true"
    seccomp_allow_deny_syntax: "true"
    seccomp_notify: "true"
    seccomp_proxy_send_notify_fd: "true"
  os_name: Ubuntu
  os_version: "24.04"
  project: default
  server: incus
  server_clustered: false
  server_event_mode: full-mesh
  server_name: hercules
  server_pid: 224411
  server_version: "6.12"
  storage: zfs
  storage_version: 2.2.2-0ubuntu9.1
  storage_supported_drivers:
  - name: btrfs
    version: 6.6.3
    remote: false
  - name: dir
    version: "1"
    remote: false
  - name: zfs
    version: 2.2.2-0ubuntu9.1
    remote: false
  - name: lvm
    version: 2.03.16(2) (2022-05-18) / 1.02.185 (2022-05-18) / 4.48.0
    remote: false

Instance details

Content
architecture: x86_64
config:
  cloud-init.user-data: |
    #cloud-config
    users:
      - name: evadne
        gecos: Evadne Wu
        sudo: ALL=(ALL) NOPASSWD:ALL
        ssh_import_id:
          - gh:evadne
        lock_passwd: true
        ssh_authorized_keys:
    package_update: true
    package_upgrade: true
    packages:
      - watchman
      - inotify-tools
      - libncurses-dev
      - libwxgtk3.2-dev
      - libwxgtk-webview3.2-dev
      - libgl1-mesa-dev
      - libglu1-mesa-dev
      - libpng-dev
      - libssh-dev
      - unixodbc-dev
      - unzip
      - xsltproc
      - fop
      - libxml2-utils
      - openjdk-17-jdk
      - autoconf
      - libssl-dev
      - libncurses-dev
      - avahi-daemon
      - build-essential
      - curl
      - git
      - m4
      - openssh-server
      - wget
      - zsh
    write_files:
      - path: /setup/install-eternal-terminal.sh
        permissions: '0744'
        owner: root
        content: |
          #!/usr/bin/env bash
          set -euxo pipefail
          echo "deb [signed-by=/etc/apt/keyrings/et.gpg] https://mistertea.github.io/debian-et/debian-source/ $(grep VERSION_CODENAME /etc/os-release | cut -d= -f2) main" | sudo tee -a /etc/apt/sources.list.d/et.list
          curl -sSL https://github.com/MisterTea/debian-et/raw/master/et.gpg | sudo tee /etc/apt/keyrings/et.gpg >/dev/null
          sudo apt update
          sudo apt install et
    runcmd:
      - /setup/install-eternal-terminal.sh
  image.architecture: amd64
  image.description: Debian bookworm amd64 (20250218_06:08)
  image.os: Debian
  image.release: bookworm
  image.serial: "20250218_06:08"
  image.type: squashfs
  image.variant: cloud
  limits.cpu: "16"
  limits.cpu.nodes: 10,11
  limits.memory: 64GiB
  volatile.base_image: 7b5d599bff45b24659f9ece25208032587bee0856968f19c6a232d16715d703b
  volatile.cloud-init.instance-id: 845e3bf6-ccdf-41b2-b509-4de6fbaf5536
  volatile.cpu.nodes: "12"
  volatile.eth0.host_name: veth3a005567
  volatile.eth0.hwaddr: 00:16:3e:98:e6:a3
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.power: RUNNING
  volatile.last_state.ready: "false"
  volatile.network-primary.host_name: enp65s0f0v3
  volatile.network-primary.hwaddr: 00:16:3e:a5:b6:f2
  volatile.network-primary.last_state.created: "false"
  volatile.network-primary.last_state.hwaddr: d6:ff:bd:71:f1:db
  volatile.network-primary.last_state.mtu: "1500"
  volatile.network-primary.last_state.vf.hwaddr: "00:00:00:00:00:00"
  volatile.network-primary.last_state.vf.id: "3"
  volatile.network-primary.last_state.vf.parent: enp65s0f0np0
  volatile.network-primary.last_state.vf.spoofcheck: "false"
  volatile.network-primary.last_state.vf.vlan: "0"
  volatile.network-primary.name: eth1
  volatile.network-sanctuary.host_name: enp65s0f1v3
  volatile.network-sanctuary.hwaddr: 00:16:3e:03:95:9c
  volatile.network-sanctuary.last_state.created: "false"
  volatile.network-sanctuary.last_state.hwaddr: ba:8e:70:f6:13:ce
  volatile.network-sanctuary.last_state.mtu: "1500"
  volatile.network-sanctuary.last_state.vf.hwaddr: "00:00:00:00:00:00"
  volatile.network-sanctuary.last_state.vf.id: "3"
  volatile.network-sanctuary.last_state.vf.parent: enp65s0f1np1
  volatile.network-sanctuary.last_state.vf.spoofcheck: "false"
  volatile.network-sanctuary.last_state.vf.vlan: "0"
  volatile.network-sanctuary.name: eth2
  volatile.uuid: 1ad0698d-c62c-4cc8-82c4-1d80f53765d8
  volatile.uuid.generation: 1ad0698d-c62c-4cc8-82c4-1d80f53765d8
devices:
  argus-data:
    path: /media/data
    pool: hercules
    source: argus-data
    type: disk
  disk-datasets-polygon:
    path: /media/datasets/polygon
    readonly: "true"
    source: /media/styx/polygon
    type: disk
  disk-scratch:
    path: /media/scratch
    source: /media/scratch
    type: disk
ephemeral: false
profiles:
- default
- infrastructure-network-cpu0
stateful: false
description: Application Development Target

Instance log

No response

Current behavior

After setting:

limit.cpus = 16
limit.cpu.nodes = a,b

I observe the following:

➜  ~ incus exec argus -- numactl -s                                  
policy: default
preferred node: current
physcpubind: 80 82 84 85 86 87 92 93 95 273 275 280 283 285 286 287 
cpubind: 10 11 
nodebind: 10 11 
membind: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 
preferred: 

So the container is correctly bound to 2 NUMA nodes. However, volatile.cpu.nodes only shows one node and it is not the correct one:

➜  ~ incus config get argus limits.cpu.nodes
10,11
➜  ~ incus config get argus volatile.cpu.nodes
12

Expected behavior

I would have expected volatile.cpu.nodes to be 10,11 but perhaps there is something going on that I am not knowledgeable enough about.

Steps to reproduce

  1. Make container
  2. Set limits.cpu.nodes to 2 NUMA nodes
  3. Set limits.cpu to corresponding number of cores

Metadata

Metadata

Assignees

Labels

EasyGood for new contributors

Type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions