Cluster attachment

When a tenant cluster requests bare metal nodes, vMetal provisions servers and joins them to that cluster automatically. Each server appears in the cluster as a standard Kubernetes node. For the driver-neutral Machine model behind this, see Machine concept.

This page describes what happens when a Machine joins a tenant cluster as a worker. Two related cases are covered elsewhere. For Machines requested without any tenant cluster, see Independently provisioned machines. For Machines that run a tenant cluster's own control plane rather than joining one, see Control plane machines.

Click to enlarge

Private nodes

When vMetal provisions a server for a tenant cluster, that server is a private node. The platform allocates it exclusively to one cluster at a time. No other cluster or tenant shares the physical hardware.

Private nodes appear in the tenant cluster alongside any virtual nodes from vCluster. Schedule workloads to private nodes using standard Kubernetes node selectors and resource requests.

How joining works

The platform generates cloud-init user data for each Machine. For private nodes, the cloud-init runs a bootstrap script downloaded from the vCluster server. The script installs the container runtime and kubelet, then runs kubeadm join with a dedicated join token for the target tenant cluster. When the server finishes installing the OS and reboots, cloud-init runs the script automatically. For how user data and network configuration are resolved at provision time, see What the platform generates.

The server registers with the tenant cluster's API server and begins accepting workloads. The join happens without manual intervention.

Machine lifecycle and cluster lifecycle

A Machine exists for as long as the cluster needs that node. When scale-down removes a node or the cluster is deleted, the platform deletes the Machine. This triggers deprovisioning through Metal3. Metal3 cleans the server and returns it to the available pool.

If a Machine fails, it is not automatically replaced. Investigate the BareMetalHost status conditions, resolve the issue, and return the BareMetalHost to available before the platform can use it again.

Click to enlarge

Node resources

After joining, the server is a standard Kubernetes node. The resources it advertises depend on what software runs on it.

vMetal does not set GPU or accelerator resources. Device plugins running on the node advertise them to the kubelet. For GPU nodes, deploy the NVIDIA GPU Operator or equivalent. vMetal provisions the server with the correct OS. The device plugin discovers the hardware and reports it to the scheduler.

To target GPU nodes, use standard Kubernetes resource requests:

resources:
  limits:
    nvidia.com/gpu: "1"

Node profiles

A NodeProfile is reusable node configuration that the platform applies when a server joins a tenant cluster. Profiles are cluster-scoped and shared, so one profile can serve many clusters and many projects. The same profile applies whether the node arrives through a Machine, through Auto Nodes, or through a manual join.

A profile sets four things:

Field	Applies to the joined node
taints	Taints that stay on the node.
startupTaints	Temporary taints held while the node initializes, expected to be removed by something that tolerates them.
nodeLabels	Labels set on the node.
nodeAnnotations	Annotations set on the node.

apiVersion: management.loft.sh/v1
kind: NodeProfile
metadata:
  name: platform.gpu-training
spec:
  displayName: "GPU training nodes"
  description: "Reserved for training workloads that tolerate the gpu taint"
  nodeLabels:
    workload-class: training
  taints:
  - key: workload
    value: training
    effect: NoSchedule

Everything a profile sets is a Kubernetes node property. Profiles therefore do nothing for independently provisioned machines, which never become Kubernetes nodes. Use node type properties for anything those machines need at provision time.

Which profile a node gets

The platform resolves the profile in this order and takes the first one it finds:

1. An explicit spec.profileRef on the Machine, which is per-node selection.
2. The tenant cluster's own privateNodes.defaultProfile value.
3. The project's spec.defaultNodeProfile.

If none of the three is set, the node joins with no profile. The platform records which profile configured a node by labelling it vcluster.com/node-profile with the profile name. A node joined without a profile carries no such label, so that label is also how you tell the two apart after the fact.

A Machine referencing a profile that does not exist is rejected with nodeProfile "<name>" does not exist. A Machine referencing a profile the project is not allowed to use is rejected with nodeProfile "<name>" is not allowed in project "<project>". See Restricting which NodeProfiles a project can use.

Configuring private nodes

Private nodes are configured on the tenant cluster, not on the NodeProvider. See Tenant cluster nodes for the configuration and the task around it.