Skip to content

Document memory handling in VM (ballooning) #1904

New issue
New issue
Closed
Task
Closed
Document memory handling in VM (ballooning)#1904
Task
Assignees
stgraber
Labels
DocumentationDocumentation needs updatingEasyGood for new contributors
Milestone
incus-6.12
@lnutimura

Description

@lnutimura
lnutimura
opened on Apr 7, 2025

Is there an existing issue for this?

  • There is no existing issue for this bug

Is this happening on an up to date version of Incus?

  • This is happening on a supported version of Incus

Incus system details

config:
  core.https_address: '[::]:8443'
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_stop_priority
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
- network_sriov
- console
- restrict_dev_incus
- migration_pre_copy
- infiniband
- dev_incus_events
- proxy
- network_dhcp_gateway
- file_get_symlink
- network_leases
- unix_device_hotplug
- storage_api_local_volume_handling
- operation_description
- clustering
- event_lifecycle
- storage_api_remote_volume_handling
- nvidia_runtime
- container_mount_propagation
- container_backup
- dev_incus_images
- container_local_cross_pool_handling
- proxy_unix
- proxy_udp
- clustering_join
- proxy_tcp_udp_multi_port_handling
- network_state
- proxy_unix_dac_properties
- container_protection_delete
- unix_priv_drop
- pprof_http
- proxy_haproxy_protocol
- network_hwaddr
- proxy_nat
- network_nat_order
- container_full
- backup_compression
- nvidia_runtime_config
- storage_api_volume_snapshots
- storage_unmapped
- projects
- network_vxlan_ttl
- container_incremental_copy
- usb_optional_vendorid
- snapshot_scheduling
- snapshot_schedule_aliases
- container_copy_project
- clustering_server_address
- clustering_image_replication
- container_protection_shift
- snapshot_expiry
- container_backup_override_pool
- snapshot_expiry_creation
- network_leases_location
- resources_cpu_socket
- resources_gpu
- resources_numa
- kernel_features
- id_map_current
- event_location
- storage_api_remote_volume_snapshots
- network_nat_address
- container_nic_routes
- cluster_internal_copy
- seccomp_notify
- lxc_features
- container_nic_ipvlan
- network_vlan_sriov
- storage_cephfs
- container_nic_ipfilter
- resources_v2
- container_exec_user_group_cwd
- container_syscall_intercept
- container_disk_shift
- storage_shifted
- resources_infiniband
- daemon_storage
- instances
- image_types
- resources_disk_sata
- clustering_roles
- images_expiry
- resources_network_firmware
- backup_compression_algorithm
- ceph_data_pool_name
- container_syscall_intercept_mount
- compression_squashfs
- container_raw_mount
- container_nic_routed
- container_syscall_intercept_mount_fuse
- container_disk_ceph
- virtual-machines
- image_profiles
- clustering_architecture
- resources_disk_id
- storage_lvm_stripes
- vm_boot_priority
- unix_hotplug_devices
- api_filtering
- instance_nic_network
- clustering_sizing
- firewall_driver
- projects_limits
- container_syscall_intercept_hugetlbfs
- limits_hugepages
- container_nic_routed_gateway
- projects_restrictions
- custom_volume_snapshot_expiry
- volume_snapshot_scheduling
- trust_ca_certificates
- snapshot_disk_usage
- clustering_edit_roles
- container_nic_routed_host_address
- container_nic_ipvlan_gateway
- resources_usb_pci
- resources_cpu_threads_numa
- resources_cpu_core_die
- api_os
- container_nic_routed_host_table
- container_nic_ipvlan_host_table
- container_nic_ipvlan_mode
- resources_system
- images_push_relay
- network_dns_search
- container_nic_routed_limits
- instance_nic_bridged_vlan
- network_state_bond_bridge
- usedby_consistency
- custom_block_volumes
- clustering_failure_domains
- resources_gpu_mdev
- console_vga_type
- projects_limits_disk
- network_type_macvlan
- network_type_sriov
- container_syscall_intercept_bpf_devices
- network_type_ovn
- projects_networks
- projects_networks_restricted_uplinks
- custom_volume_backup
- backup_override_name
- storage_rsync_compression
- network_type_physical
- network_ovn_external_subnets
- network_ovn_nat
- network_ovn_external_routes_remove
- tpm_device_type
- storage_zfs_clone_copy_rebase
- gpu_mdev
- resources_pci_iommu
- resources_network_usb
- resources_disk_address
- network_physical_ovn_ingress_mode
- network_ovn_dhcp
- network_physical_routes_anycast
- projects_limits_instances
- network_state_vlan
- instance_nic_bridged_port_isolation
- instance_bulk_state_change
- network_gvrp
- instance_pool_move
- gpu_sriov
- pci_device_type
- storage_volume_state
- network_acl
- migration_stateful
- disk_state_quota
- storage_ceph_features
- projects_compression
- projects_images_remote_cache_expiry
- certificate_project
- network_ovn_acl
- projects_images_auto_update
- projects_restricted_cluster_target
- images_default_architecture
- network_ovn_acl_defaults
- gpu_mig
- project_usage
- network_bridge_acl
- warnings
- projects_restricted_backups_and_snapshots
- clustering_join_token
- clustering_description
- server_trusted_proxy
- clustering_update_cert
- storage_api_project
- server_instance_driver_operational
- server_supported_storage_drivers
- event_lifecycle_requestor_address
- resources_gpu_usb
- clustering_evacuation
- network_ovn_nat_address
- network_bgp
- network_forward
- custom_volume_refresh
- network_counters_errors_dropped
- metrics
- image_source_project
- clustering_config
- network_peer
- linux_sysctl
- network_dns
- ovn_nic_acceleration
- certificate_self_renewal
- instance_project_move
- storage_volume_project_move
- cloud_init
- network_dns_nat
- database_leader
- instance_all_projects
- clustering_groups
- ceph_rbd_du
- instance_get_full
- qemu_metrics
- gpu_mig_uuid
- event_project
- clustering_evacuation_live
- instance_allow_inconsistent_copy
- network_state_ovn
- storage_volume_api_filtering
- image_restrictions
- storage_zfs_export
- network_dns_records
- storage_zfs_reserve_space
- network_acl_log
- storage_zfs_blocksize
- metrics_cpu_seconds
- instance_snapshot_never
- certificate_token
- instance_nic_routed_neighbor_probe
- event_hub
- agent_nic_config
- projects_restricted_intercept
- metrics_authentication
- images_target_project
- images_all_projects
- cluster_migration_inconsistent_copy
- cluster_ovn_chassis
- container_syscall_intercept_sched_setscheduler
- storage_lvm_thinpool_metadata_size
- storage_volume_state_total
- instance_file_head
- instances_nic_host_name
- image_copy_profile
- container_syscall_intercept_sysinfo
- clustering_evacuation_mode
- resources_pci_vpd
- qemu_raw_conf
- storage_cephfs_fscache
- network_load_balancer
- vsock_api
- instance_ready_state
- network_bgp_holdtime
- storage_volumes_all_projects
- metrics_memory_oom_total
- storage_buckets
- storage_buckets_create_credentials
- metrics_cpu_effective_total
- projects_networks_restricted_access
- storage_buckets_local
- loki
- acme
- internal_metrics
- cluster_join_token_expiry
- remote_token_expiry
- init_preseed
- storage_volumes_created_at
- cpu_hotplug
- projects_networks_zones
- network_txqueuelen
- cluster_member_state
- instances_placement_scriptlet
- storage_pool_source_wipe
- zfs_block_mode
- instance_generation_id
- disk_io_cache
- amd_sev
- storage_pool_loop_resize
- migration_vm_live
- ovn_nic_nesting
- oidc
- network_ovn_l3only
- ovn_nic_acceleration_vdpa
- cluster_healing
- instances_state_total
- auth_user
- security_csm
- instances_rebuild
- numa_cpu_placement
- custom_volume_iso
- network_allocations
- zfs_delegate
- storage_api_remote_volume_snapshot_copy
- operations_get_query_all_projects
- metadata_configuration
- syslog_socket
- event_lifecycle_name_and_project
- instances_nic_limits_priority
- disk_initial_volume_configuration
- operation_wait
- image_restriction_privileged
- cluster_internal_custom_volume_copy
- disk_io_bus
- storage_cephfs_create_missing
- instance_move_config
- ovn_ssl_config
- certificate_description
- disk_io_bus_virtio_blk
- loki_config_instance
- instance_create_start
- clustering_evacuation_stop_options
- boot_host_shutdown_action
- agent_config_drive
- network_state_ovn_lr
- image_template_permissions
- storage_bucket_backup
- storage_lvm_cluster
- shared_custom_block_volumes
- auth_tls_jwt
- oidc_claim
- device_usb_serial
- numa_cpu_balanced
- image_restriction_nesting
- network_integrations
- instance_memory_swap_bytes
- network_bridge_external_create
- network_zones_all_projects
- storage_zfs_vdev
- container_migration_stateful
- profiles_all_projects
- instances_scriptlet_get_instances
- instances_scriptlet_get_cluster_members
- instances_scriptlet_get_project
- network_acl_stateless
- instance_state_started_at
- networks_all_projects
- network_acls_all_projects
- storage_buckets_all_projects
- resources_load
- instance_access
- project_access
- projects_force_delete
- resources_cpu_flags
- disk_io_bus_cache_filesystem
- instance_oci
- clustering_groups_config
- instances_lxcfs_per_instance
- clustering_groups_vm_cpu_definition
- disk_volume_subpath
- projects_limits_disk_pool
- network_ovn_isolated
- qemu_raw_qmp
- network_load_balancer_health_check
- oidc_scopes
- network_integrations_peer_name
- qemu_scriptlet
- instance_auto_restart
- storage_lvm_metadatasize
- ovn_nic_promiscuous
- ovn_nic_ip_address_none
- instances_state_os_info
- network_load_balancer_state
- instance_nic_macvlan_mode
- storage_lvm_cluster_create
- network_ovn_external_interfaces
- instances_scriptlet_get_instances_count
- cluster_rebalance
- custom_volume_refresh_exclude_older_snapshots
- storage_initial_owner
- storage_live_migration
- instance_console_screenshot
- image_import_alias
- authorization_scriptlet
- console_force
- network_ovn_state_addresses
- network_bridge_acl_devices
- instance_debug_memory
- init_preseed_storage_volumes
- init_preseed_profile_project
- instance_nic_routed_host_address
- instance_smbios11
- api_filtering_extended
- acme_dns01
- security_iommu
- network_ipv4_dhcp_routes
- network_state_ovn_ls
- network_dns_nameservers
- acme_http01_port
- network_ovn_ipv4_dhcp_expiry
- instance_state_cpu_time
- network_io_bus
- disk_io_bus_usb
- storage_driver_linstor
- instance_oci_entrypoint
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
auth_user_name: ubuntu
auth_user_method: unix
environment:
  addresses:
  - 172.30.1.39:8443
  - '[2801:80:3ea1:c362::3ce]:8443'
  - 10.203.71.1:8443
  - '[fd42:b68f:e730:ff22::1]:8443'
  architectures:
  - x86_64
  - i686
  certificate: |
    -----BEGIN CERTIFICATE-----
    MIICBjCCAY2gAwIBAgIRAMV3z3gQcubrlKJPNmf6SBQwCgYIKoZIzj0EAwMwNDEZ
    MBcGA1UEChMQTGludXggQ29udGFpbmVyczEXMBUGA1UEAwwOcm9vdEBsYWItaW5j
    dXMwHhcNMjQxMjAyMTczNzMwWhcNMzQxMTMwMTczNzMwWjA0MRkwFwYDVQQKExBM
    aW51eCBDb250YWluZXJzMRcwFQYDVQQDDA5yb290QGxhYi1pbmN1czB2MBAGByqG
    SM49AgEGBSuBBAAiA2IABPAOGMVwXPbQiem8RAmka6Dfvr3IpzVjZq1AA5tfMsrP
    g/Y92PJ2+cUBWXGu8S9A8w6dtSrzPVwsI30P29ngT22mRLJI7q3F5SC5/WlAZgpv
    IOjXr8v2gKHVSgIhTFfmiKNjMGEwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQMMAoG
    CCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwLAYDVR0RBCUwI4IJbGFiLWluY3VzhwR/
    AAABhxAAAAAAAAAAAAAAAAAAAAABMAoGCCqGSM49BAMDA2cAMGQCMHv302zN13sN
    XZlaDrpSsN0Ra8ptoyWUMbYu4MTyovnBv/bHKjlKoIEWQZoMFVVsxgIwGAs6xavw
    IMXgl8Kw4e0XKGaUm8a9v4DZALtZYUo74/89AJ05i2zBVuc5dU9K5V0x
    -----END CERTIFICATE-----
  certificate_fingerprint: feb107c9458e0af77c0de3b11dfb398b9af7c672fa78748cd1654519e99c9de8
  driver: qemu | lxc
  driver_version: 9.0.4 | 6.0.3
  firewall: nftables
  kernel: Linux
  kernel_architecture: x86_64
  kernel_features:
    idmapped_mounts: "true"
    netnsid_getifaddrs: "true"
    seccomp_listener: "true"
    seccomp_listener_continue: "true"
    uevent_injection: "true"
    unpriv_binfmt: "true"
    unpriv_fscaps: "true"
  kernel_version: 6.8.0-49-generic
  lxc_features:
    cgroup2: "true"
    core_scheduling: "true"
    devpts_fd: "true"
    idmapped_mounts_v2: "true"
    mount_injection_file: "true"
    network_gateway_device_route: "true"
    network_ipvlan: "true"
    network_l2proxy: "true"
    network_phys_macvlan_mtu: "true"
    network_veth_router: "true"
    pidfd: "true"
    seccomp_allow_deny_syntax: "true"
    seccomp_notify: "true"
    seccomp_proxy_send_notify_fd: "true"
  os_name: Ubuntu
  os_version: "24.04"
  project: default
  server: incus
  server_clustered: false
  server_event_mode: full-mesh
  server_name: lab-incus
  server_pid: 2131137
  server_version: "6.11"
  storage: btrfs
  storage_version: 6.6.3
  storage_supported_drivers:
  - name: btrfs
    version: 6.6.3
    remote: false
  - name: dir
    version: "1"
    remote: false
  - name: lvm
    version: 2.03.16(2) (2022-05-18) / 1.02.185 (2022-05-18) / 4.48.0
    remote: false
  - name: lvmcluster
    version: 2.03.16(2) (2022-05-18) / 1.02.185 (2022-05-18) / 4.48.0
    remote: true

Instance details

architecture: x86_64
config:
  cloud-init.user-data: |-
    #cloud-config
    package_update: true
    package_upgrade: true
    package_reboot_if_required: true

    users:
      - default
      - name: ubuntu
        lock_passwd: true
        ssh_authorized_keys:
          - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDhURBc0F8D6As8UtI1lYeAYkEwgQpvZkUlgQtaWXlMTP9eR3YKf+IFgy6OrOzOwSmC7bPjUy51eUSYlz8rtcKlrP2fvCk0UCUfgiwWIxNZA8oAIkSQfWZBfLh1rPUN+sViu2knaugTwv/akbafhOZPqaHmLp1x735wvxhS4lEj320No6N28rOHmTbWM+e3wc7FBdE3ctm6MOQunYEfhxjTUngZs5QMlA/6Ww5VbfbmE5uBb1PRuO7Dxsjlb+hPi2YQR0GhS8kJYKSFAY3RLPyQBK7OTUqh+tCjvSdLUfK3D9eV417jvIr8uD7hpg+ql6jsQxbCkjwMsPzlcLy81uaokJF+TyMDgO5Pzoq778Rudy5c7yoOVlLW4PEo0R3Vmb4nBsAoqqD6OCiqqBsDYW6fE+fDPGNV8SbryB/ePnzkNrkY/tqOfQ5I2frY68QdDqCkX3qh8PXj2rYQTpOqq/1fvX8PLFD4DnhkflG4HX4BXBrh8r2QMFm8dKGfZQBIfqLcOs6vzKq9SgIZZ5rs/RCWuKZJqyjrRez6gL0gGWRsvOLARRn3x49KIGH6y3zJwUl/VL9OcMqPgmU7xWD29Mo037indbdfnp9ejewpuY4mkh4WCAe7DSAqpK9BSdpyrMnO+s4tXJx8k3MO3/YYzFLLWBB6d3VDQhwFPaRnx5d+/Q== poc

    packages:
      - openssh-server
  image.architecture: amd64
  image.description: Ubuntu jammy amd64 (20250402_07:42)
  image.os: Ubuntu
  image.release: jammy
  image.serial: "20250402_07:42"
  image.type: disk-kvm.img
  image.variant: cloud
  limits.cpu: "2"
  limits.memory: 4GB
  volatile.base_image: 276b718b01f3f3a60c3aa926413fe076aee8a3bb53e4b0c435bc8611640bedbb
  volatile.cloud-init.instance-id: cadf0b8b-a0a4-466a-aff2-bc20c600d8a3
  volatile.eth0.host_name: tap07e9fb48
  volatile.eth0.hwaddr: 10:66:6a:b7:03:46
  volatile.last_state.power: RUNNING
  volatile.uuid: 6cfe7733-346b-4d0d-96fa-1207779897ad
  volatile.uuid.generation: 6cfe7733-346b-4d0d-96fa-1207779897ad
  volatile.vm.definition: pc-q35-9.0
  volatile.vsock_id: "1646134706"
devices: {}
ephemeral: false
profiles:
- default
stateful: false
description: ""

Instance log

No response

Current behavior

I have a running instance with 4GB of memory. If I try to change its limits.memory to a lower value, e.g. incus config set <instance> limits.memory=1GB, the update fails due to timeout, but it does take effect on the instance.

Expected behavior

The update attempt should either:

  • Succeed without timeout, since the instance is actually changed; or
  • Fail completely, without changing the instance.

i.e. keep the instance memory in sync with its configuration in Incus.

Steps to reproduce

  1. Launch instance with 4GB memory limit:
ubuntu@lab-incus:~$ incus launch images:ubuntu/jammy/cloud my-vm \
    --vm \
    --config limits.memory=4GB \
    --config cloud-init.user-data="$(cat user_data.yml)"
Launching my-vm
  1. Check total instance memory:
root@my-vm:~# awk '$3=="kB"{$2=$2/1024^2;$3="GB";} 1' /proc/meminfo | head -n1
MemTotal: 3.52773 GB
  1. Try decreasing the total instance memory:
ubuntu@lab-incus:~$ incus config set my-vm limits.memory=1GB
Error: Failed updating memory limit: Failed setting memory to 953MiB (currently 3089MiB) as it was taking too long
  1. Check total instance memory (again):
root@my-vm:~# awk '$3=="kB"{$2=$2/1024^2;$3="GB";} 1' /proc/meminfo | head -n1
MemTotal: 0.734447 GB
  1. Check instance limits.memory:
ubuntu@lab-incus:~$ incus config get my-vm limits.memory
4GB

A desync occurred between the instance's total memory and its limits.memory configuration in Incus.

The memory limit update logic seems to be implemented here.

Metadata

Metadata

Assignees

Labels

DocumentationDocumentation needs updatingEasyGood for new contributors

Type

Task

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions