feat(power): add virtual machine power monitoring #2100

sthaha · 2025-05-26T05:06:25Z

Introduce support for monitoring power consumption of virtual machines (VMs) and expose those as Prometheus metrics kepler_vm_<rapl-zone>_{watts|joules_total}.

Key changes include:

Enhance a process type to the Process struct
Add VirtualMachine struct and VirtualMachines map to track VM metadata.
Detect VM processes using regex patterns for QEMU, KVM, and Libvirt hypervisors.
Enhance Process struct with Type and VirtualMachineID fields to associate processes with VMs.
Add VirtualMachines() method to Informer interface
Include mock support for VM testing in mock_utils.go.
Add VM power computation (in monitor) just like Container.
export VM prometheus metrics

NOTE: Only Qemu based VM detetion is supported. VirtualBox, VMware, and Xen hypervisor support are pending

sthaha · 2025-05-27T00:07:04Z

@vprashar2929 could you please help test this?

vimalk78

some initial comment. continuing to review

internal/exporter/prometheus/collector/power_collector.go

internal/resource/informer.go

Signed-off-by: Sunil Thaha <sthaha@redhat.com>

Add support for monitoring Virtual Machines (VMs). Key changes include: - Enhance a process type to the Process struct - Add `VirtualMachine` struct and `VirtualMachines` map to track VM metadata. - Detect VM processes using regex pattern for QEMU - Enhance `Process` struct with `Type` and `VirtualMachineID` fields to associate processes with VMs. - Add `VirtualMachines()` method to `Informer` interface - Include mock support for VM testing in `mock_utils.go`. *NOTE:* Only Qemu based VMs are supported. VirtualBox, VMware, and Xen hypervisor support are pending Signed-off-by: Sunil Thaha <sthaha@redhat.com>

Add VM power computation just like Container. Signed-off-by: Sunil Thaha <sthaha@redhat.com>

codecov · 2025-05-29T08:58:49Z

Codecov Report

Attention: Patch coverage is 95.58499% with 20 lines in your changes missing coverage. Please review.

Project coverage is 92.55%. Comparing base (c4aa86f) to head (ca85c1b).
Report is 6 commits behind head on reboot.

Files with missing lines	Patch %	Lines
internal/resource/informer.go	93.45%	5 Missing and 2 partials ⚠️
internal/monitor/monitor.go	0.00%	4 Missing and 2 partials ⚠️
internal/resource/vm.go	95.23%	4 Missing ⚠️
internal/monitor/vm.go	96.87%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           reboot    #2100      +/-   ##
==========================================
+ Coverage   91.97%   92.55%   +0.57%     
==========================================
  Files          30       32       +2     
  Lines        2143     2538     +395     
==========================================
+ Hits         1971     2349     +378     
- Misses        137      150      +13     
- Partials       35       39       +4

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Signed-off-by: Sunil Thaha <sthaha@redhat.com>

vimalk78 · 2025-05-29T13:26:30Z

internal/resource/informer.go

 	}

 	return nil
 }

+// Buffered channels prevent goroutine blocking


nit: incorrect code comment

vimalk78 · 2025-05-29T13:48:37Z

internal/monitor/process.go

@@ -44,6 +45,11 @@ func newProcess(proc *resource.Process, zones ZoneUsageMap) *Process {
 	if proc.Container != nil {
 		process.ContainerID = proc.Container.ID
 	}
+
+	// Add the container ID if available


vimalk78 · 2025-05-29T13:49:56Z

internal/monitor/types.go

+
+	CPUTotalTime float64 // CPU time in seconds
+
+	// Replace single Usage with ZoneUsageMap


nit: comment needed?

vimalk78 · 2025-05-29T13:51:47Z

internal/monitor/types.go

+
+// Container represents the power consumption of a container
+type VirtualMachine struct {
+	ID   string // VM ID


VM ID has a different meaning. better call it uuid

need not be uuid for other hypervisors ... Say in future, we support vmware, then the unique id (how ever they generate) will still be ID in kepler. vendor specific labels should be published as <vendor>_<label> . E.g. libvirt_id if we are to ever add it.

vimalk78 · 2025-05-29T13:55:20Z

internal/monitor/vm.go

+func (pm *PowerMonitor) calculateVMPower(prev, newSnapshot *Snapshot) error {
+	vms := pm.resources.VirtualMachines()
+
+	// Skip if no containers


nit: remove/change comment

vimalk78 · 2025-05-29T13:57:46Z

internal/monitor/vm.go

+	vmMap := make(VirtualMachines, len(vms.Running))
+
+	// For each VM, calculate power for each zone separately
+	for id, c := range vms.Running {


nit: change c to vm , the type is VirtualMachine

vimalk78 · 2025-05-29T14:01:47Z

internal/monitor/vm.go

+				continue
+			}
+
+			cpuTimeRatio := c.CPUTimeDelta / vms.NodeCPUTimeDelta


wouldn't it be simpler to use power usages from the vm's process power usage directly?

Thats assuming that any vm will have only a single process - right?

vimalk78 · 2025-05-29T14:07:09Z

internal/resource/informer.go

+	if vm == nil {
+		panic(fmt.Sprintf("process %d  of type %s has is nil virtual machine", proc.PID, proc.Type))


function panics if proc.VirtualMachine is nil, but the function's usage already invokes proc.VirtualMachine.ID

yes, you are right, but I explicitly wanted to assert and log the process that puts kepler in this specific state that is invalid.

This commit addresses comments on sustainable-computing-io#2100 Signed-off-by: Sunil Thaha <sthaha@redhat.com>

cleanup(vm): addresses comments from #2100

sthaha requested a review from vimalk78 May 26, 2025 13:44

sthaha force-pushed the feat-vm-power branch from 9d33449 to 0c6196a Compare May 26, 2025 13:51

sthaha force-pushed the feat-vm-power branch from 0c6196a to c75e4d4 Compare May 27, 2025 06:38

github-actions bot added the chore Routine tasks or maintenance label May 27, 2025

sthaha force-pushed the feat-vm-power branch from c75e4d4 to c2ab377 Compare May 27, 2025 07:17

vimalk78 reviewed May 27, 2025

View reviewed changes

internal/exporter/prometheus/collector/power_collector.go Outdated Show resolved Hide resolved

internal/resource/informer.go Show resolved Hide resolved

sthaha force-pushed the feat-vm-power branch 3 times, most recently from 23be1a3 to c0a6f4c Compare May 28, 2025 08:37

github-actions bot added the fix A bug fix label May 28, 2025

sthaha force-pushed the feat-vm-power branch from c0a6f4c to dda9c16 Compare May 29, 2025 05:29

sthaha added 4 commits May 29, 2025 04:25

fix: ensure resource informer is initilized

67dd6ec

Signed-off-by: Sunil Thaha <sthaha@redhat.com>

chore(testdata): add utils to copy procfs files

46e2ea3

Signed-off-by: Sunil Thaha <sthaha@redhat.com>

feat(monitor/vm): add VM power computation

87b0a62

Add VM power computation just like Container. Signed-off-by: Sunil Thaha <sthaha@redhat.com>

sthaha force-pushed the feat-vm-power branch from dda9c16 to b499140 Compare May 29, 2025 08:58

sthaha added feat A new feature or enhancement and removed fix A bug fix chore Routine tasks or maintenance labels May 29, 2025

sthaha force-pushed the feat-vm-power branch from b499140 to 2eba2a7 Compare May 29, 2025 08:59

github-actions bot added fix A bug fix chore Routine tasks or maintenance and removed feat A new feature or enhancement labels May 29, 2025

sthaha marked this pull request as ready for review May 29, 2025 10:29

sthaha added feat A new feature or enhancement and removed fix A bug fix chore Routine tasks or maintenance labels May 29, 2025

feat(exporter/vm): export VM prometheus metrics

ca85c1b

Signed-off-by: Sunil Thaha <sthaha@redhat.com>

sthaha force-pushed the feat-vm-power branch from 2eba2a7 to ca85c1b Compare May 29, 2025 10:47

github-actions bot added fix A bug fix chore Routine tasks or maintenance and removed feat A new feature or enhancement labels May 29, 2025

sthaha added feat A new feature or enhancement and removed fix A bug fix chore Routine tasks or maintenance labels May 29, 2025

vimalk78 approved these changes May 29, 2025

View reviewed changes

vimalk78 merged commit c8220d9 into sustainable-computing-io:reboot May 29, 2025
11 checks passed

sthaha added a commit to sthaha/kepler that referenced this pull request May 29, 2025

cleanup(vm): addresses comments from sustainable-computing-io#2100

809c343

This commit addresses comments on sustainable-computing-io#2100 Signed-off-by: Sunil Thaha <sthaha@redhat.com>

sthaha mentioned this pull request May 29, 2025

cleanup(vm): addresses comments from #2100 #2113

Merged

sthaha added a commit that referenced this pull request May 30, 2025

Merge pull request #2113 from sthaha/chore-vm-power

a719aae

cleanup(vm): addresses comments from #2100


		CPUTotalTime float64 // CPU time in seconds

		// Replace single Usage with ZoneUsageMap

		if vm == nil {
		panic(fmt.Sprintf("process %d of type %s has is nil virtual machine", proc.PID, proc.Type))

feat(power): add virtual machine power monitoring #2100

feat(power): add virtual machine power monitoring #2100

Uh oh!

Conversation

sthaha commented May 26, 2025

Uh oh!

sthaha commented May 27, 2025

Uh oh!

vimalk78 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

codecov bot commented May 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

codecov bot commented May 29, 2025 •

edited

Loading