Troubleshooting

This guide covers how to diagnose and resolve common vMetal failures. Start by identifying the state your server is in using BareMetalHost States, then jump to the matching section below.

Set the project, Machine, BareMetalHost namespace, and BareMetalHost name once. Every interpolated command on this page updates automatically.

Modify the following with your specific values to replace them across the whole page:

PROJECT

MACHINE

NAMESPACE

BMH_NAME

This page uses up to three Kubernetes contexts:

Context	Use it for
<management-context>	vCluster Platform resources, including NodeClaim, SSHKey, and the ipam-default ConfigMap.
<control-plane-context>	Metal3 resources, including BareMetalHost, BMC Secrets, webhooks, and component logs.
<tenant-context>	Kubernetes nodes and workloads in a tenant cluster.

The control plane context is the cluster referenced by NodeProvider.spec.metal3.clusterRef.cluster. Replace each placeholder with a name from kubectl config get-contexts.

Click to enlarge

Common failures

Stuck in registering

The BMH stays in registering when the BMO cannot reach the BMC.

BMC unreachable

Check BMO logs for connection errors:

kubectl --context <control-plane-context> logs statefulset/metal3 -n my-metal3-namespace -c metal3 | grep my-baremetalhost-name

Verify:

• The BMC address in spec.bmc.address is reachable from the control plane cluster.
• Firewall rules allow traffic from the Metal3 pod to the BMC port (623 for IPMI, 443 for Redfish).
• The Secret referenced by spec.bmc.credentialsName contains valid username and password keys.

Stuck in inspecting

The BMH stays in inspecting when the server cannot boot into the IPA ramdisk and phone home to Ironic. Work through these checks in order. Each one narrows the failure to a specific layer.

Check BMC console

The BMC console shows the server's boot sequence directly. Because it is out-of-band, it works even if the vMetal setup is misconfigured.

Access the console through the BMC web interface or using ipmitool:

ipmitool -I lanplus -H <bmc-address> -U <user> -P <pass> sol activate

You should see this sequence:

1. The server powers on or reboots (triggered by Metal3 over BMC).
2. The firmware attempts a network boot on the provisioning NIC.
3. The NIC gets a DHCP lease.
4. The server downloads a kernel and initramfs over TFTP or HTTP.

If the server does not power on or reboot, the problem is BMC access. Check the BMC address, network reachability from the Metal3 pod, and the credentials in the Secret referenced by spec.bmc.credentialsName.

If the server powers on but stalls during network boot, check the DHCP server logs next.

Check DHCP server logs

The DHCP server logs every request and response it sees. It is the most targeted tool for diagnosing a stuck network boot.

kubectl --context <control-plane-context> logs -n my-metal3-namespace -l app=dhcp-proxy

The logs include:

• DHCP requests from the server, including requests from MACs that do not match any registered BareMetalHost. These help identify a wrong spec.bootMACAddress.
• DHCP responses sent back to the server.
• TFTP requests for the bootloader.
• HTTP requests for the kernel and initramfs images.

If no entries appear for the server, the DHCP broadcast is not reaching the pod. Check that the DHCP server is attached to the provisioning network using a Multus NetworkAttachmentDefinition, or that it is running with hostNetwork: true for flat networks.

Check ramdisk logs

The ramdisk-logs container tails IPA log files as they arrive. If no logs appear, the server never contacted Ironic.

kubectl --context <control-plane-context> logs statefulset/metal3 -n my-metal3-namespace -c ramdisk-logs

Check Ironic logs

kubectl --context <control-plane-context> logs statefulset/metal3 -n my-metal3-namespace -c ironic

Look for errors related to the server's MAC address or IPMI/Redfish operations.

Other causes

• Wrong boot MAC. The spec.bootMACAddress field must match the NIC used for PXE boot. The DHCP server logs show requests from unmatched MACs, which makes this easy to spot.
• IPA ramdisk not downloaded. The ramdisk-downloader init container fetches the IPA image on startup. A failed init container blocks the entire Metal3 pod. Run kubectl --context <control-plane-context> describe pod metal3-0 -n <namespace> and check the init container status.

Stuck in provisioning

The BMH stays in provisioning when Ironic cannot complete the OS installation.

Check Ironic logs

kubectl --context <control-plane-context> logs statefulset/metal3 -n my-metal3-namespace -c ironic

Common causes

• Image URL unreachable. The server's IPA agent downloads the OS image directly. It must be reachable from the bare metal server, not just from the Metal3 pod. Test whether Ironic can reach the URL using an ephemeral debug container: kubectl --context <control-plane-context> debug -it metal3-0 -n <namespace> --image=curlimages/curl -- curl -I <image-url>.
• Checksum mismatch. Ironic verifies the downloaded image against spec.image.checksum. If the checksum is wrong, provisioning fails with a checksum error in the Ironic logs. Recheck the checksum against the actual file.
• Wrong rootDeviceHint. If a server has multiple disks, Ironic may target the wrong one. Set spec.rootDeviceHints on the BareMetalHost to target the correct disk by size, WWN, or device name.

Force a re-provision

If provisioning is stuck and you want Ironic to retry, delete the Machine from the vCluster Platform UI or API. The platform deprovisions the BMH and returns it to available. A new Machine request then triggers a fresh provisioning attempt.

Do not manually edit the BMH spec.image field while a Machine holds the claim. The platform manages that field and will reconcile it on the next cycle.

Stuck in deprovisioning

When a Machine is deleted, the platform removes the image reference from the BMH and sets it offline. The BMO then triggers automated cleaning and returns the server to available.

What cleaning does

The BMO powers on the server via BMC, boots a cleaning ramdisk, and runs the configured cleaning steps. With vMetal's default metadata cleaning mode, this removes disk metadata such as partition tables. It is not a full-disk secure erase. The disabled mode skips automated cleaning. See Reuse and reprovisioning before choosing a mode for sensitive workloads.

The BMC must remain reachable from Metal3 throughout cleaning. The cleaning ramdisk must also be able to reach Ironic and the DHCP proxy when booting.

Cleaning can take several minutes on servers with many disks or large disk capacities.

Check BMO and Ironic logs

kubectl --context <control-plane-context> logs statefulset/metal3 -n my-metal3-namespace -c metal3 | grep -i "clean\|deprovi"
kubectl --context <control-plane-context> logs statefulset/metal3 -n my-metal3-namespace -c ironic | grep -i "clean"

Server permanently unavailable

If a server is physically removed or permanently unavailable, delete the BareMetalHost, then delete its Machine from the platform. The Machine deletion treats the missing host as already removed and releases its allocated IP. This bypasses automated cleaning, so use it only when the server cannot complete normal deprovisioning.

Error state

A BMH in error state has stopped retrying. Read the error details first.

kubectl --context <control-plane-context> get baremetalhost my-baremetalhost-name -n my-metal3-namespace -o jsonpath='{.status.errorMessage}'
kubectl --context <control-plane-context> get baremetalhost my-baremetalhost-name -n my-metal3-namespace -o jsonpath='{.status.errorType}'

When to retry vs. when to recreate

• Retry when the error is transient: network blip or temporary BMC unresponsiveness. Annotate the BMH to trigger a retry:

kubectl --context <control-plane-context> annotate baremetalhost my-baremetalhost-name -n my-metal3-namespace \
baremetalhost.metal3.io/reboot='{"force":false}' --overwrite

• Fix credentials and retry when the error is wrong BMC credentials. Update the Secret with the correct values. The BMO picks up the change on its next reconcile without reprovisioning the server. Then annotate the BMH to trigger a retry using the command above.

• Abort and create a new Machine when the error is a checksum mismatch or image URL failure. The image URL and checksum are written to Ironic's database at provisioning time. Updating the OSImage resource does not affect an in-flight Machine. Delete the Machine from the vCluster Platform UI or API to abort provisioning. The platform deprovisions the BMH and returns it to available. Create a new Machine with corrected image configuration to restart. To avoid waiting for the full deprovisioning cycle, you can delete the Machine immediately after the BMH reaches deprovisioning.

• Correct the BMH configuration when the BMC address, credentials reference, or boot MAC is wrong. Edit the resource in place when the field permits it. If you must recreate the BMH, release its Machine first and wait for the NodeClaim to finish deleting before recreating the host. Deleting a claimed BMH does not by itself clean up the claim.

Outcome-specific failures

Use these checks after Metal3 reports the BareMetalHost as provisioned. First, map the Machine to its underlying host.

The NodeClaim records the BareMetalHost name in the metal3.vcluster.com/claimed-node annotation:

kubectl --context <management-context> get nodeclaim my-machine \
-n loft-p-my-project \
-o jsonpath='{.metadata.annotations.metal3\.vcluster\.com/claimed-node}'

To look up the mapping in the other direction, select the BareMetalHost by the metal3.vcluster.com/node-claim label:

kubectl --context <control-plane-context> get baremetalhost \
-n my-metal3-namespace \
-l 'metal3.vcluster.com/node-claim=my-machine,vcluster.com/project=my-project'

If the first command returns a host other than the page's BareMetalHost name, update that page variable before running the remaining interpolated commands.

Machine is available but SSH fails

An independent Machine can be Available even when SSH access fails. Available means the operating system was provisioned and the platform completed the claim. It does not test an SSH login.

1. Inspect the NodeClaim on the management cluster. Confirm that the IP annotation is present and that the claim references the expected node type.

   kubectl --context <management-context> get nodeclaim my-machine \
   -n loft-p-my-project -o yaml

2. Confirm that every cluster-scoped SSHKey named by the node type exists and contains the intended public key.

   kubectl --context <management-context> get sshkey <ssh-key-name> -o yaml

3. Inspect the cloud-init payload generated for the host. The Secret is on the control plane cluster. Treat the decoded payload as sensitive because it contains bootstrap configuration.

   kubectl --context <control-plane-context> get secret vcluster-userdata-my-baremetalhost-name \
   -n my-metal3-namespace -o jsonpath='{.data.userData}' | base64 --decode

4. Use the BMC console to inspect cloud-init when SSH is unavailable:

   sudo cloud-init status --long
   sudo tail -n 200 /var/log/cloud-init-output.log
   sudo journalctl -u cloud-final --no-pager

5. Verify that you are using the image's correct login user, TCP port 22 is reachable, and the reported IP belongs to the server. A successful provision does not validate routing or firewall policy between the operator and that IP.

Worker Machine is provisioned but not joined

For a tenant worker, Provisioned=True with Joined=False means the operating system installation finished but the bootstrap process did not register the node with the tenant cluster.

1. Read the NodeClaim conditions and events on the management cluster:

   kubectl --context <management-context> describe nodeclaim my-machine \
   -n loft-p-my-project

2. Use SSH or the BMC console to check cloud-init:

   sudo cloud-init status --long
   sudo tail -n 200 /var/log/cloud-init-output.log
   sudo journalctl -u cloud-final --no-pager

3. From the server, verify DNS and network access to the vCluster Platform/bootstrap endpoint and the tenant Kubernetes API endpoint. Bootstrap cannot finish if either endpoint is unreachable or its certificate name cannot be resolved.

4. Check whether the node registered in the tenant cluster:

   kubectl --context <tenant-context> get nodes -o wide

Control plane Machine is available but the cluster is offline

The platform can mark a control plane Machine Available after provisioning completes even if the vCluster service on the server fails to start. The cluster controller's VirtualClusterReady condition is not sufficient evidence either. For a standalone control plane, it can report that the requested Machine objects were created before the servers finish provisioning. Verify the service on the server and connect to the cluster API.

1. Use SSH or the BMC console to inspect cloud-init and the vCluster systemd service:

   sudo cloud-init status --long
   sudo tail -n 200 /var/log/cloud-init-output.log
   sudo systemctl status vcluster --no-pager
   sudo journalctl -u vcluster --no-pager -n 200

2. Verify that /etc/vcluster exists and that /etc/systemd/system/vcluster.service was created. If either is missing, use the cloud-init output to find the failed bootstrap step.

3. Confirm the host meets the control plane prerequisites: root access, systemd, curl, iptables, and the raised inotify limits listed in the standalone node requirements. Also verify DNS and network access to vCluster Platform and, for a multi-node control plane, the other control plane Machines.

4. Check the tenant cluster status and events from the management cluster. A ready Machine alone does not prove that the cluster API is online.

IP allocation issues

vMetal allocates an IP address per Machine from a CIDR or explicit IP range configured on the NodeProvider. The platform stores allocations in a ConfigMap named ipam-default in the loft management namespace (typically loft). If Machine creation fails with a failed to allocate IP error, check here.

"No available IPs" / CIDR exhausted

The error appears in the loft platform logs and in Machine events:

failed to allocate IP: CIDR exhausted; no free addresses left

or, for explicit IP ranges:

failed to allocate IP: all ranges exhausted; no free addresses left

Inspect current allocations

kubectl --context <management-context> get configmap ipam-default -n loft -o yaml

The ConfigMap data is a flat map of <ip>: NodeClaim/<namespace>/<name> entries. Each entry is one allocated address.

Example output:

data:
  10.10.1.2: NodeClaim/loft-p-myproject/machine-abc
  10.10.1.3: NodeClaim/loft-p-myproject/machine-def

Leaked IPs from failed deprovision

If the platform deleted a Machine but did not release its IP (for example, a crash during cleanup), the entry remains in the ConfigMap. The corresponding NodeClaim no longer exists. Remove stale entries manually:

1. List all active NodeClaims:

   kubectl --context <management-context> get nodeclaims -A

2. Compare against the ConfigMap entries. Any entry whose NodeClaim no longer exists is stale.

3. Edit the ConfigMap to remove stale entries:

   kubectl --context <management-context> edit configmap ipam-default -n loft

CIDR too small

If the number of Machines regularly exceeds the available address space, expand the IP range in the NodeProvider configuration. Set the range using the metal3.vcluster.com/network-cidr or metal3.vcluster.com/network-ip-range property on the NodeProvider or NodeType.

Webhook readiness

The Metal3 ValidatingWebhook validates every BareMetalHost create and update. The webhook runs inside the metal3 container of the metal3 StatefulSet and serves on port 9443. The API server routes requests through the metal3-webhook-service Service.

The webhook configuration uses failurePolicy: Fail by default. If the webhook endpoint is unreachable when you create a BMH, the API server rejects the request with a connection error. The BMH is never created. Servers appear unregistered until the webhook comes up.

Check webhook readiness

Check whether the BMO container is ready:

kubectl --context <control-plane-context> get pod metal3-0 -n my-metal3-namespace -o jsonpath='{.status.containerStatuses[?(@.name=="metal3")].ready}'

Check the readiness probe directly:

kubectl --context <control-plane-context> exec -n my-metal3-namespace metal3-0 -c metal3 -- \
wget -qO- http://localhost:9440/readyz

A healthy response returns ok.

Check the ValidatingWebhookConfiguration for the current CA bundle and service endpoint:

kubectl --context <control-plane-context> get validatingwebhookconfiguration metal3-validating-webhook-configuration -o yaml

Log locations

All Metal3 components run as containers inside the metal3 StatefulSet on the control plane cluster. The namespace matches the one configured in the NodeProvider's clusterRef.

Component	Container	Log command
Ironic	ironic	kubectl --context <control-plane-context> logs statefulset/metal3 -n <namespace> -c ironic
Bare Metal Operator	metal3	kubectl --context <control-plane-context> logs statefulset/metal3 -n <namespace> -c metal3
IPA ramdisk output	ramdisk-logs	kubectl --context <control-plane-context> logs statefulset/metal3 -n <namespace> -c ramdisk-logs

Platform-level events

Machine state and events are represented by NodeClaim resources on the management cluster. List claims across projects:

kubectl --context <management-context> get nodeclaims -A

Then inspect the claim for this Machine:

kubectl --context <management-context> describe nodeclaim my-machine \
-n loft-p-my-project

These events report why vMetal could not claim a BMH, such as no available hosts, an IPAM failure, or an image resolution error. You can check them before digging into component logs.