runtime-rs: Introduce PCIe Port devices in runtime-rs for qemu-rs #10578

Apokleos · 2024-11-25T09:28:26Z

To well support vfio devices in PCIe root ports or switch ports in case of qemu-rs, we instroduce port devices containing RootPort and SwitchPort mostly aligned with the implementations of kata-runtime.

src/runtime-rs/crates/runtimes/virt_container/src/sandbox.rs

src/runtime-rs/crates/hypervisor/src/device/topology.rs

BbolroC

Overall, it looks great to me! I've left a few minor questions.
Since I'm not an expert in this area, I would like to delegate the approval to @gkurz and @pmores. Thanks a lot!

zvonkok · 2024-12-04T15:41:23Z

There are limits per HV how many root/switch ports can be created. We need to make sure that we're not exceeding those limits otherwise we get cryptic memory errors.

For QEMU we have a limit of 16 root ports or 16 switch ports or 64KB IO memory each root or switch port taking up of default 4K IO memory.

zvonkok · 2024-12-04T15:42:26Z

@BbolroC One outstanding thing to clarify is can we attach VFIO-AP devices to a root-port? This way we can drop the support for legacy bridges altogether.

Apokleos · 2024-12-05T01:37:07Z

There are limits per HV how many root/switch ports can be created. We need to make sure that we're not exceeding those limits otherwise we get cryptic memory errors.

For QEMU we have a limit of 16 root ports or 16 switch ports or 64KB IO memory each root or switch port taking up of default 4K IO memory.

Yes, sir. without KEP-4113 support, we need introduce many related codes to do such work which seems hacking to me.
But I am thinking how to address this problem and will make it be more friendly for the future KEP-4113 or sandbox API.

I have pulled request for get bar max memory in another PR. And I am looking forward to your INPUT @zvonkok @BbolroC

Apokleos · 2024-12-06T12:58:52Z

I have tested it with related PRs and get the SWITHCH PORT output as below:

configuration

hotplug_vfio_on_root_bus = true
pcie_switch_port = 4

~/kata-cdi/apokleos/kata-containers/src/runtime-rs# ctr run -t --device /dev/vfio/75 --device /dev/vfio/76 --device /dev/vfio/214 --device /dev/vfio/215 --rm --runtime io.containerd.kata.v2 "$image" debug0456789 /bin/bash
[root@localhost /]# lspci -tvnn
-[0000:00]-+-00.0  Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller [8086:29c0]
           +-01.0  Red Hat, Inc. Virtio RNG [1af4:1005]
           +-02.0-[01]--
           +-03.0-[02]--
           +-04.0  Red Hat, Inc. Virtio SCSI [1af4:1004]
           +-05.0  Red Hat, Inc. Virtio socket [1af4:1053]
           +-06.0  Red Hat, Inc. Virtio file system [1af4:105a]
           +-07.0-[03-08]----00.0-[04-08]--+-00.0-[05]----00.0  NVIDIA Corporation TU104GL [Tesla T4] [10de:1eb8]
           |                               +-01.0-[06]----00.0  NVIDIA Corporation TU104GL [Tesla T4] [10de:1eb8]
           |                               +-02.0-[07]----00.0  NVIDIA Corporation TU104GL [Tesla T4] [10de:1eb8]
           |                               \-03.0-[08]----00.0  NVIDIA Corporation TU104GL [Tesla T4] [10de:1eb8]
           +-08.0  Red Hat, Inc. Virtio console [1af4:1003]
           +-1f.0  Intel Corporation 82801IB (ICH9) LPC Interface Controller [8086:2918]
           +-1f.2  Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] [8086:2922]
           \-1f.3  Intel Corporation 82801I (ICH9 Family) SMBus Controller [8086:2930]

get the root ports

           +-07.0-[03]--
           +-08.0-[04]--

the shell command

configuration

...
hotplug_vfio_on_root_bus = true

# Before hot plugging a PCIe device, you need to add a pcie_root_port device.
# Use this parameter when using some large PCI bar devices, such as Nvidia GPU
# The value means the number of pcie_root_port
# This value is valid when hotplug_vfio_on_root_bus is true and machine_type is "q35"
# Default 0
pcie_root_port = 2

[root@localhost /]# lspci -tvnn
-[0000:00]-+-00.0  Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller [8086:29c0]
           +-01.0  Red Hat, Inc. Virtio RNG [1af4:1005]
           +-02.0-[01]--
           +-03.0-[02]--
           +-04.0  Red Hat, Inc. Virtio SCSI [1af4:1004]
           +-05.0  Red Hat, Inc. Virtio socket [1af4:1053]
           +-06.0  Red Hat, Inc. Virtio file system [1af4:105a]
           +-07.0-[03]--
           +-08.0-[04]--
           +-09.0  Red Hat, Inc. Virtio console [1af4:1003]
           +-1f.0  Intel Corporation 82801IB (ICH9) LPC Interface Controller [8086:2918]
           +-1f.2  Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] [8086:2922]
           \-1f.3  Intel Corporation 82801I (ICH9 Family) SMBus Controller [8086:2930]
[root@localhost /]#

BbolroC · 2024-12-11T10:54:02Z

@BbolroC One outstanding thing to clarify is can we attach VFIO-AP devices to a root-port? This way we can drop the support for legacy bridges altogether.

Please take a look at the following:

@hbrueckner 's comment: Support basic capabilities to hotplug VFIO devices to root-port or switch-port for qemu-rs #10361 (comment)
Add subchannel and VFIO-AP hot-plugging support to qemu-runtime-rs for s390x #10573

Based on my understanding so far, there is no PCI support in s390x QEMU (so no concept like a root/switch port) and a bridge info is not required for VFIO-AP hot-plugging. I will raise a PR for VFIO-AP hot-plugging after this PR and #10362 are merged. Thanks.

zvonkok · 2024-12-11T15:02:43Z

hotplug_vfio_on_root_bus = true this is a very old unsupported option. We removed that some time ago. We should use

hot_plug_vfio = "root-port"
cold_plug_vfio = "root-port"

zvonkok · 2024-12-11T16:52:12Z

@Apokleos For the switch-port use-case I need to post a QEMU patch. Can you check if the BARs are correctly mapped?

dmesg | grep BAR

Apokleos · 2024-12-12T02:08:04Z

@BbolroC One outstanding thing to clarify is can we attach VFIO-AP devices to a root-port? This way we can drop the support for legacy bridges altogether.

Please take a look at the following:

@hbrueckner 's comment: Support basic capabilities to hotplug VFIO devices to root-port or switch-port for qemu-rs #10361 (comment)

Add subchannel and VFIO-AP hot-plugging support to qemu-runtime-rs for s390x #10573

Based on my understanding so far, there is no PCI support in s390x QEMU (so no concept like a root/switch port) and a bridge info is not required for VFIO-AP hot-plugging. I will raise a PR for VFIO-AP hot-plugging after this PR and #10362 are merged. Thanks.

Thx Choi @BbolroC, I'd like to know that there's some work in my PR I can do for you which can help your coming works ? If any ideas, please let me know.

Apokleos · 2024-12-12T02:09:59Z

hotplug_vfio_on_root_bus = true this is a very old unsupported option. We removed that some time ago. We should use
hot_plug_vfio = "root-port"
cold_plug_vfio = "root-port"

Sure. the coming updated PR will remove this option.

zvonkok · 2024-12-12T13:55:30Z

@Apokleos I am wondering how we're dealing with different archs, in the go-runtime we have different implementation for different architectures like x86 - s390x how are we dealing it with in runtime-rs?
A good example is that in x86 we do not need the bridge but s390x needs it. So we shouldn't introduce the bridge in x86 but only have it enabled in s390x.

pmores · 2025-04-25T10:40:26Z

src/runtime-rs/crates/hypervisor/src/device/topology.rs

+
+/// Represents an available node in the PCIe topology.
+#[derive(Clone, Debug)]
+pub enum AvailabledNode {


Should this be AvailableNode?

Thx Pavel. I think it should not be here now, I need remove this data structure as there's no reference of it in this PR. And I will add it back in a related PR(vfio device hotplug).

pmores

Thanks @Apokleos for this huge piece of work in a well-formatted PR! :-) I'm approving but frankly I'd prefer if you could deal with some of the remaining suggestions by @gkurz and @BbolroC either by incorporating them and resolving the pertinent conversations, or by documenting your reasons why not to incorporate them.

I've resolved my conversations for clarity that you've dealt with.

Apokleos · 2025-04-27T04:59:07Z

Thanks @Apokleos for this huge piece of work in a well-formatted PR! :-) I'm approving but frankly I'd prefer if you could deal with some of the remaining suggestions by @gkurz and @BbolroC either by incorporating them and resolving the pertinent conversations, or by documenting your reasons why not to incorporate them.

I've resolved my conversations for clarity that you've dealt with.

Thx Pavel @pmores.
Yes, absolutely I will do. As I have promissed @zvonkok I will clean up these left comments.

Apokleos · 2025-04-27T07:47:11Z

hotplug_vfio_on_root_bus = true this is a very old unsupported option. We removed that some time ago. We should use
hot_plug_vfio = "root-port"
cold_plug_vfio = "root-port"

I'd like to create an issue to do this work

hotplug_vfio_on_root_bus = true this is a very old unsupported option. We removed that some time ago. We should use
hot_plug_vfio = "root-port"
cold_plug_vfio = "root-port"

I think we should address it in another PR, ahead of it, I open an issue to track it

Apokleos · 2025-04-28T01:51:53Z

Hi @zvonkok @pmores @BbolroC I have updated the PR, and try to address the left comments. Is there any stuff I need address before it's merged ? It is with do-not-merge before your ACK.

BbolroC · 2025-04-28T07:14:21Z

Thanks for the update. I am fine with merging this. 👍

pmores · 2025-04-28T09:10:14Z

Hey @Apokleos , I'd still be in favour of mentioning the root/switch port counts limitation in the config file template but I'm fine merging this with or without that change. Thanks!

zvonkok · 2025-04-28T18:26:38Z

@Apokleos What happens if a VFIO device is hot-plugged to a root-port and the workload fails to start and the container is restarted. Is the root-port freed up? Can it be reused once he container restarts?

pmores · 2025-04-29T07:50:04Z

@zvonkok I don't think this PR deals with hotplugging at all.

Apokleos · 2025-05-15T11:38:31Z

Hey @Apokleos , I'd still be in favour of mentioning the root/switch port counts limitation in the config file template but I'm fine merging this with or without that change. Thanks!

Thx Pavel @pmores
I hope my explanation makes sense. When I initially designed this, the primary goal was to align with the runtime-go's implementation to ensure functional consistency.

Secondly, it's aimed for a simpler pcie topology that could still support advanced features for devices like GPUs. It's what the limitation only one port device type(RootPort or SwitchPort) exists in a pod/VM.

I also spent some time considering if there were scenarios requiring both a dedicated root port and a combined root/switch port or called switch port, but couldn't identify any clear use cases. That's what led to the current design.

However, I'm certainly open to making adjustments in future implementations if such scenarios do arise.

Apokleos · 2025-05-15T11:48:19Z

@Apokleos What happens if a VFIO device is hot-plugged to a root-port and the workload fails to start and the container is restarted. Is the root-port freed up? Can it be reused once he container restarts?

Thx @zvonkok This PR will not involve the vfio device hotplugging stuff, and the following one will do. But IMO, it's a point worth discussing. As I understand, continuously hot-plugging devices from VM isn't very reliable at the moment, so we might need to have a deep discussion about it based on the PR #10362.

(1) Introduce new field `pcie_switch_port` for switch ports. (2) Add related checking logics in vmms(dragonball, qemu) Fixes kata-containers#10361 Signed-off-by: alex.lyn <alex.lyn@antgroup.com>

Support setting switch ports with annotatation or configuration.toml Fixes kata-containers#10361 Signed-off-by: alex.lyn <alex.lyn@antgroup.com>

This commit introduces an implementation for managing PCIe topologies, focusing on the relationship between Root Ports and Switch Ports. The design supports two strategies for generating Switch Ports: Let's take the requirement of 4 switch ports as an example. There'll be three possible solutions as below: (1) Single Root Port + Single PCIe Switch: Uses 1 Root Port and 1 Switch with 4 Downstream Ports. (2) Multiple Root Ports + Multiple PCIe Switches: Uses 2 Root Ports and 2 Switches, each with 2 Downstream Ports. The recommended strategy is Option 1 due to its simplicity, efficiency, and scalability. The implementation includes data structures (PcieTopology, RootPort, PcieSwitch, SwitchPort) and operations (add_pcie_root_port, add_switch_to_root_port, add_switch_port_to_switch) to manage the topology effectively. Fxies kata-containers#10361 Signed-off-by: alex.lyn <alex.lyn@antgroup.com>

PortDevice is for handling root ports or switch ports in PCIe Topology. It will make it easy pass the root ports/switch ports information during create VM with requirements of PCIe devices. Fixes kata-containers#10361 Signed-off-by: alex.lyn <alex.lyn@antgroup.com>

Fixes kata-containers#10361 Signed-off-by: alex.lyn <alex.lyn@antgroup.com>

A new resource type `PortDevice` is introduced which is dedicated for handling root ports/switch ports during sandbox creation(VM). Fixes kata-containers#10361 Signed-off-by: alex.lyn <alex.lyn@antgroup.com>

Prepare pcie port devices before starting VM with the help of device manager and PCIe Topology. Fixes kata-containers#10361 Signed-off-by: alex.lyn <alex.lyn@antgroup.com>

Some data structures and methods are introduced to help handle vfio devices. And mothods add_pcie_root_ports and add_pcie_switch_ports follow runtime's related implementations of vfio devices. Fixes kata-containers#10361 Signed-off-by: alex.lyn <alex.lyn@antgroup.com>

Extract PortDevice relevant information, and then invoke different processing methods based on the device type. Fixes kata-containers#10361 Signed-off-by: alex.lyn <alex.lyn@antgroup.com>

Apokleos requested review from BbolroC, pmores, zvonkok and gkurz and removed request for BbolroC November 25, 2024 09:28

katacontainersbot added the size/huge Largest and most complex task (probably needs breaking into small pieces) label Nov 25, 2024

Apokleos requested a review from lifupan November 25, 2024 09:32

Apokleos force-pushed the pcie-port-devices branch from 6d29a59 to 4b1b6d2 Compare November 25, 2024 09:46

Apokleos added the ok-to-test label Nov 25, 2024

Apokleos mentioned this pull request Nov 25, 2024

runtime-rs: add support hotplugging vfio device for qemu-rs #10362

Merged

Apokleos requested a review from justxuewei November 25, 2024 12:10

BbolroC reviewed Nov 28, 2024

View reviewed changes

src/runtime-rs/crates/runtimes/virt_container/src/sandbox.rs Outdated Show resolved Hide resolved

src/runtime-rs/crates/runtimes/virt_container/src/sandbox.rs Outdated Show resolved Hide resolved

src/runtime-rs/crates/hypervisor/src/device/topology.rs Outdated Show resolved Hide resolved

BbolroC reviewed Nov 28, 2024

View reviewed changes

src/runtime-rs/crates/hypervisor/src/device/topology.rs Outdated Show resolved Hide resolved

BbolroC reviewed Nov 28, 2024

View reviewed changes

Apokleos force-pushed the pcie-port-devices branch 3 times, most recently from 12ffcc8 to 31a665b Compare December 6, 2024 12:46

Apokleos requested a review from BbolroC December 6, 2024 13:01

Apokleos force-pushed the pcie-port-devices branch from 31a665b to 87b9ebb Compare December 9, 2024 07:31

pmores reviewed Apr 25, 2025

View reviewed changes

pmores approved these changes Apr 25, 2025

View reviewed changes

Apokleos added the do-not-merge PR has problems or depends on another label Apr 27, 2025

Apokleos force-pushed the pcie-port-devices branch 2 times, most recently from 5fd6ef1 to bf0dbc3 Compare April 27, 2025 07:15

Apokleos mentioned this pull request Apr 27, 2025

runtime-rs: remove unsupported config of hotplug_vfio_on_root_bus #11198

Open

Apokleos added do-not-merge PR has problems or depends on another and removed do-not-merge PR has problems or depends on another labels Apr 27, 2025

Apokleos removed the do-not-merge PR has problems or depends on another label May 15, 2025

Apokleos added 9 commits May 15, 2025 20:10

kata-types: Introduce pcie_switch_port in configuration

a42d16a

(1) Introduce new field `pcie_switch_port` for switch ports. (2) Add related checking logics in vmms(dragonball, qemu) Fixes kata-containers#10361 Signed-off-by: alex.lyn <alex.lyn@antgroup.com>

kata-types: Support switch port config via annotation and configuration

2f5ee0e

Support setting switch ports with annotatation or configuration.toml Fixes kata-containers#10361 Signed-off-by: alex.lyn <alex.lyn@antgroup.com>

runtime-rs: handle useless Device match arms in dragonball vmm case

1d670bb

Fixes kata-containers#10361 Signed-off-by: alex.lyn <alex.lyn@antgroup.com>

runtime-rs: Introduce PortDevice in resource manager in sandbox

d435712

A new resource type `PortDevice` is introduced which is dedicated for handling root ports/switch ports during sandbox creation(VM). Fixes kata-containers#10361 Signed-off-by: alex.lyn <alex.lyn@antgroup.com>

runtime-rs: Prepare pcie port devices before start sandbox

47c7ba8

Prepare pcie port devices before starting VM with the help of device manager and PCIe Topology. Fixes kata-containers#10361 Signed-off-by: alex.lyn <alex.lyn@antgroup.com>

runtime-rs: Handle PortDevice devices when invoke start_vm with Qemu

0753352

Extract PortDevice relevant information, and then invoke different processing methods based on the device type. Fixes kata-containers#10361 Signed-off-by: alex.lyn <alex.lyn@antgroup.com>

Apokleos force-pushed the pcie-port-devices branch from bf0dbc3 to 0753352 Compare May 15, 2025 12:11

Apokleos merged commit 305a5f5 into kata-containers:main May 18, 2025
289 of 350 checks passed

Apokleos mentioned this pull request May 19, 2025

Support basic capabilities to hotplug VFIO devices to root-port or switch-port for qemu-rs #10361

Closed

3 tasks

runtime-rs: Introduce PCIe Port devices in runtime-rs for qemu-rs #10578

runtime-rs: Introduce PCIe Port devices in runtime-rs for qemu-rs #10578

Uh oh!

Conversation

Apokleos commented Nov 25, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

BbolroC left a comment

Choose a reason for hiding this comment

Uh oh!

zvonkok commented Dec 4, 2024

Uh oh!

zvonkok commented Dec 4, 2024

Uh oh!

Apokleos commented Dec 5, 2024

Uh oh!

Apokleos commented Dec 6, 2024

Uh oh!

BbolroC commented Dec 11, 2024

Uh oh!

zvonkok commented Dec 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zvonkok commented Dec 11, 2024

Uh oh!

Apokleos commented Dec 12, 2024

Uh oh!

Apokleos commented Dec 12, 2024

Uh oh!

zvonkok commented Dec 12, 2024

Uh oh!

pmores Apr 25, 2025

Choose a reason for hiding this comment

Uh oh!

Apokleos Apr 27, 2025

Choose a reason for hiding this comment

Uh oh!

pmores left a comment

Choose a reason for hiding this comment

Uh oh!

Apokleos commented Apr 27, 2025

Uh oh!

Apokleos commented Apr 27, 2025

Uh oh!

Apokleos commented Apr 28, 2025

Uh oh!

BbolroC commented Apr 28, 2025

Uh oh!

pmores commented Apr 28, 2025

Uh oh!

zvonkok commented Apr 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pmores commented Apr 29, 2025

Uh oh!

Apokleos commented May 15, 2025

Uh oh!

Apokleos commented May 15, 2025

Uh oh!

Uh oh!

Uh oh!

zvonkok commented Dec 11, 2024 •

edited

Loading

zvonkok commented Apr 28, 2025 •

edited

Loading