This page refers to the latest version of the provider.
If you are looking for older versions, you can find older reference documents here.

Immutable Provider

The Immutable provider installs Flatcar Container Linux on bare-metal or virtual machines over the network. It then creates a kubeadm Kubernetes cluster on those machines and installs the SD modules on top of it.

furyctl manages the operating system installation and the node life-cycle. You do not prepare an operating system, and you do not configure SSH access, before you start.

This document describes every aspect of the configuration file. You can find the reference schema for this provider here.

warning

The Immutable provider is in alpha status and is under active development. Its configuration and behavior can change between releases.

tip

If your machines already run an operating system, use the OnPremises provider instead.

For a complete, step-by-step installation, read the SIGHUP Distribution on Immutable guide. This page describes the configuration file, the guide describes the full procedure.

How the installation works

The Immutable provider starts from bare machines:

1. furyctl generates one iPXE script and two Ignition configurations for each machine. The MAC address of the machine selects the correct files.
2. furyctl starts an HTTP boot server and waits.
3. Each machine boots over the network, installs Flatcar to its disk, and reboots.
4. Each machine reports the booted status to furyctl.
5. After all the machines report booted, furyctl runs Ansible and builds the cluster.

Flatcar has a read-only /usr partition and an A/B update scheme. You do not patch packages on a node. Each node gets its full configuration from Ignition at first boot.

Automatic OS updates are off, because the installer masks update-engine and sets REBOOT_STRATEGY=off. furyctl drives the A/B update during a cluster upgrade.

Prerequisites

Machines

Each machine needs:

• An empty install disk, for example /dev/sda. The installer writes over its content. If the disk already holds a Flatcar installation, the installer stops instead.
• A stable IP address, from DHCP with a reservation or from a fixed address.
• The ability to boot over the network, and a boot order that puts the disk before the network.
• A way to start iPXE. See Network boot below.

For the minimum size of each node role, read the Requirements page.

warning

Put the disk before the network in the boot order. While the install disk is empty, the firmware finds no operating system and falls back to the network. This starts the installation. After the installation, the machine boots Flatcar from the disk.

If the network comes before the disk, the machine starts the installer again at every reboot. The installation never completes.

Network

• A DHCP server that you can modify. It must send option 67 to the machines, so that they reach the furyctl boot server.
• A DNS server that resolves the hostname of each machine to its IP address.
• One free IP address for the keepalived virtual IP, and a DNS record that points to it. This record becomes spec.kubernetes.controlPlane.address.
• A wildcard DNS record for the ingresses, that points to the same virtual IP.
• Access to the internet, to download the Flatcar images and the container images.

info

furyctl does not run DHCP, TFTP, or DNS. It serves only the boot script, the Ignition configurations, and the Flatcar assets. The DHCP and DNS setup is yours.

Network boot

Each machine must run iPXE, and then chain to the furyctl boot server. How iPXE starts depends on the machine:

• The network card runs iPXE already. iPXE sends the DHCP user class iPXE, and your DHCP server answers with the URL of the boot server. This path is the shortest one.
• The firmware does not run iPXE. The machine must load iPXE first. With UEFI HTTP Boot (UEFI 2.5+), DHCP gives the firmware the URL of an ipxe.efi file that you host. Virtual machines usually need this path.

The Immutable Installation Guide describes the four supported boot cases in detail.

Information to collect

Collect this information for each machine before you write the configuration file:

Item	Field in furyctl.yaml	Notes
Full host name	hostname	Your DNS must resolve this name to the IP address of the machine.
MAC address of the boot interface	macAddress	It selects the configuration of the machine. Your DHCP server also uses it.
CPU architecture	arch	x86-64 or arm64. It selects the Flatcar and sysext packages.
Install disk	storage.installDisk	The device path, for example /dev/sda.
Name of the network interface	the key below network.ethernets	For example eth0 or enp1s0. A wrong name leaves the machine offline.

For the load balancer machines, you also need the name of the interface that receives the virtual IP. It goes into loadBalancers.keepalived.interface.

tip

A name such as /dev/sda can change between boots. To always select the same disk, use a stable path from /dev/disk/by-path/ or /dev/disk/by-id/.

SSH key pair

furyctl writes a public key into the chosen user account on every machine. Then it uses the private key to connect to the machines. Create the key pair before the installation:

ssh-keygen -t ed25519 -f ./ssh-key -C "sd-cluster" -N ""

Both paths go into the spec.infrastructure.ssh section.

PKI

The Immutable provider needs a set of certificate authorities for the Kubernetes control plane and for the etcd database. furyctl creates them with one command:

furyctl create pki

The command writes the files into the folder that spec.kubernetes.pkiPath selects. To see the advanced options, run furyctl create pki --help.

Configuration file creation

To create a configuration file with default values and explanatory comments, run:

furyctl create config --kind Immutable --version v1.35.1 --name <cluster-name>

spec.infrastructure section

This section describes the machines and how furyctl reaches them. It is required for this provider, and the other providers do not have it.

ssh and ipxeServer

spec:
  infrastructure:
    ssh:
      username: core
      privateKeyPath: ./ssh-key
      publicKeyPath: ./ssh-key.pub
    ipxeServer:
      url: http://furyctl.example.com:8080

ssh.username selects the account that furyctl uses to reach the machines. The example uses core, the built-in Flatcar account, but you can name a different account.

ipxeServer.url is the address of the boot server that furyctl starts. Every machine downloads its boot script and its Ignition configurations from this address. Use the name or the IP address of the machine that runs furyctl.

warning

The machines must reach this URL during the boot. A firewall or a proxy between the machines and the boot server makes the installation stop.

tip

By default, the boot server binds to the host and the port of this URL. If the host does not resolve to a local address, add bindAddress: 0.0.0.0 below ipxeServer. You can also change the port with bindPort.

To add more user accounts, use the passwd.users section of a node. It follows the Butane passwd schema.

nodes

spec:
  infrastructure:
    nodes:
      - hostname: cp1.example.com
        macAddress: "52:54:00:01:00:01"
        arch: x86-64
        storage:
          installDisk: /dev/sda
        network:
          ethernets:
            eth0:
              dhcp4: true

The nodes list holds every machine of the cluster: the load balancers, the control-plane machines, and the workers. This list gives the hardware description of each machine.

The role of a machine comes later, from the spec.infrastructure.loadBalancers and spec.kubernetes sections.

warning

storage.installDisk selects the disk that receives Flatcar. The installer writes over the content of this disk.

warning

The key below ethernets is the real name of the network interface, for example eth0 or enp1s0. furyctl writes it into a systemd-networkd match rule. A wrong name leaves the machine without network.

Set arch on every node. The field accepts x86-64 and arm64, and one cluster can mix the two architectures.

To give a machine a fixed address instead of DHCP, replace dhcp4: true with:

            eth0:
              addresses:
                - 192.168.1.181/24
              gateway: 192.168.1.1
              nameservers:
                addresses:
                  - 192.168.1.1

loadBalancers

spec:
  infrastructure:
    loadBalancers:
      members:
        - hostname: lb1.example.com
        - hostname: lb2.example.com
      keepalived:
        enabled: true
        interface: eth0
        ip: 192.168.1.179
        virtualRouterId: "201"
        passphrase: "b16cf069"
      haproxy:
        configuration: "{file://./haproxy.cfg}"

The load balancer machines run HAProxy in a container. keepalived moves a virtual IP address between them, so one machine can fail without loss of service.

The haproxy.configuration field replaces the default HAProxy configuration. The default one load balances only the Kubernetes API server. To also load balance the ingress controller, supply your own file. The HAProxy Ingress Controller of SD listens on the node ports 30080 and 30443.

This section is optional. If you manage your own load balancer, omit it.

warning

The interface field must hold the real name of the interface that receives the virtual IP.

If two keepalived clusters share a network, give each one a different virtualRouterId. The installer always adds a prometheus frontend on port 8405, so do not add it to your own file.

spec.kubernetes section

This section drives the Kubernetes provisioning. The lists in this section select machines from spec.infrastructure.nodes by hostname.

spec:
  kubernetes:
    pkiPath: ./pki
    networking:
      podCIDR: 172.16.128.0/17
      serviceCIDR: 172.16.0.0/17
    controlPlane:
      address: control-plane.example.com:6443
      members:
        - hostname: cp1.example.com
        - hostname: cp2.example.com
        - hostname: cp3.example.com
    nodeGroups:
      - name: worker
        nodes:
          - hostname: worker1.example.com
          - hostname: worker2.example.com
          - hostname: worker3.example.com

• pkiPath: the folder that holds the certificate authorities from the PKI step.
• networking: the networks of the Pods and of the Kubernetes services. These networks must not overlap the IP addresses of the machines, or other networks that the cluster must reach.
• controlPlane.address: the DNS record that points to the virtual IP of the load balancers, with its port.
• controlPlane.members: the machines that run the Kubernetes control plane.
• nodeGroups: the groups of worker nodes. The name parameter becomes a role label that you can use to select those nodes. You can also set labels, annotations, taints, and kubeletConfiguration for each group.

tip

To run etcd on separate machines, add a spec.kubernetes.etcd.members list. Without this list, etcd runs on the control-plane machines. Read the dedicated etcd nodes page.

warning

spec.kubernetes.etcd, spec.kubernetes.controlPlane.address and spec.kubernetes.controlPlane.members are immutable after deployment. Plan the topology of your cluster before the installation.

Custom registry for the Kubernetes core components

You can change the registry that supplies the images of the Kubernetes core components. This registry gives kubeadm its images, and also the pause and kubelet-csr-approver images. The host is mandatory, the port is optional.

spec:
  kubernetes:
    advanced:
      registry: <registry-host>[:<registry-port>]

spec.distribution section

This section configures the SD core modules that are installed inside the cluster. It works in the same way as in the other providers.

For the description of each field, read the reference schema. For the module documentation, read the Modules section.

Installation

When the configuration file is ready, run:

furyctl apply

furyctl runs the infrastructure, kubernetes, distribution and plugins phases in this order. The infrastructure phase starts the boot server and waits for the machines. Read the SD Installation page for the options of this command.