Installing etcd on dedicated nodes

Since version 1.31.1, 1.30.2 and 1.29.7 SD supports deploying etcd on dedicated nodes, separate from the Kubernetes control plane nodes.

note

This feature applies to the OnPremises provider and, since SD v1.35.0, to the Immutable provider. The two providers use different field names, so every configuration example below has a tab for each of them.

By default, in SD etcd runs on the control plane nodes alongside the Kubernetes API server, controller manager, and scheduler. This guide explains how to configure etcd to run on separated nodes instead.

Planning

Before starting you need to plan your infrastructure carefully. For development or testing environments, etcd typically works well with minimal resources. However, production clusters running etcd on dedicated nodes need proper hardware planning. Refer to the etcd hardware recommendations for detailed guidelines on sizing your production cluster.

Installation

Create the configuration file

Create a SD cluster furyctl.yaml file with default values using the OnPremises or the Immutable provider. You'll now customize it for this use case.

Configure etcd nodes

The key to deploying etcd on dedicated nodes is the .spec.kubernetes.etcd section. When this section is present, etcd will be deployed on the specified nodes instead of on control plane nodes.

Edit your furyctl.yaml and add the etcd configuration:

OnPremises

spec:
  kubernetes:
    etcd:
      hosts:
        - name: etcd1
          ip: 192.168.1.50
        - name: etcd2
          ip: 192.168.1.51
        - name: etcd3
          ip: 192.168.1.52

Parameters:

• hosts: Array of etcd node definitions (minimum 1, recommended 3 or 5)
  • name: Hostname for the etcd node (used for identification in the cluster)
  • ip: IP address of the etcd node

Immutable

spec:
  kubernetes:
    etcd:
      members:
        - hostname: etcd1.example.com
        - hostname: etcd2.example.com
        - hostname: etcd3.example.com

Parameters:

• members: Array of etcd node references (minimum 1, recommended 3 or 5)
  • hostname: Full host name of the etcd node

In the Immutable provider you declare every machine of the cluster once, in the .spec.infrastructure.nodes list. Each hostname here must be the same as the hostname of a machine in that list.

warning

The etcd node list is immutable after deployment. You cannot add, remove, or change etcd nodes after the cluster is created. Plan your etcd infrastructure carefully before installation.

Configure control plane nodes

Continue configuring your control plane nodes as usual.

OnPremises

spec:
  kubernetes:
    masters:
      hosts:
        - name: master1
          ip: 192.168.1.10
        - name: master2
          ip: 192.168.1.11
        - name: master3
          ip: 192.168.1.12

Immutable

spec:
  kubernetes:
    controlPlane:
      address: control-plane.example.com:6443
      members:
        - hostname: cp1.example.com
        - hostname: cp2.example.com
        - hostname: cp3.example.com

address is the DNS record that points to the virtual IP of the load balancers, with its port.

warning

The control plane node list is also immutable after deployment when using dedicated etcd nodes.

Complete the setup

Continue configuring the rest of your cluster:

OnPremises

1. SSH configuration: set up the SSH user and key path in the .spec.kubernetes.ssh section
2. Load balancers: configure load balancers in .spec.kubernetes.loadBalancers, or disable them if managing your own
3. Worker nodes: define your worker node groups in .spec.kubernetes.nodes
4. Networking: configure Pod and Service CIDRs if needed
5. Distribution modules: configure the SD modules you prefer to install

Immutable

1. Machines: describe every machine of the cluster in .spec.infrastructure.nodes, the etcd machines included
2. SSH configuration: set up the SSH user and key paths in the .spec.infrastructure.ssh section
3. Load balancers: configure load balancers in .spec.infrastructure.loadBalancers, or omit the section if managing your own
4. Worker nodes: define your worker node groups in .spec.kubernetes.nodeGroups
5. Networking: configure Pod and Service CIDRs in .spec.kubernetes.networking if needed
6. Distribution modules: configure the SD modules you prefer to install

Example configuration

Here's an example showing the key sections:

OnPremises

apiVersion: kfd.sighup.io/v1alpha2
kind: OnPremises
metadata:
  name: my-cluster

spec:
  kubernetes:
    ssh:
      username: example
      keyPath: ~/.ssh/example-key

    podCidr: 10.244.0.0/16
    svcCidr: 10.96.0.0/16

    etcd:
      hosts:
        - name: etcd1
          ip: 192.168.1.50
        - name: etcd2
          ip: 192.168.1.51
        - name: etcd3
          ip: 192.168.1.52

    masters:
      hosts:
        - name: master1
          ip: 192.168.1.10
        - name: master2
          ip: 192.168.1.11
        - name: master3
          ip: 192.168.1.12

    nodes:
      - name: worker
        hosts:
          - name: worker1
            ip: 192.168.1.20
          - name: worker2
            ip: 192.168.1.21
          - name: worker3
            ip: 192.168.1.22

  distribution:
    modules:
      networking:
        type: calico

      ingress:
        baseDomain: example.com
        nginx:
          type: dual

Immutable

apiVersion: kfd.sighup.io/v1alpha2
kind: Immutable
metadata:
  name: my-cluster

spec:
  infrastructure:
    ssh:
      username: core
      privateKeyPath: ./ssh-key
      publicKeyPath: ./ssh-key.pub

    nodes:
      # Declare every machine of the cluster here, the etcd machines included.
      # Each machine also needs its macAddress, network and storage configuration.
      - hostname: etcd1.example.com
        # ...

  kubernetes:
    pkiPath: ./pki

    networking:
      podCIDR: 10.244.0.0/16
      serviceCIDR: 10.96.0.0/16

    etcd:
      members:
        - hostname: etcd1.example.com
        - hostname: etcd2.example.com
        - hostname: etcd3.example.com

    controlPlane:
      address: control-plane.example.com:6443
      members:
        - hostname: cp1.example.com
        - hostname: cp2.example.com
        - hostname: cp3.example.com

    nodeGroups:
      - name: worker
        nodes:
          - hostname: worker1.example.com
          - hostname: worker2.example.com
          - hostname: worker3.example.com

  distribution:
    modules:
      networking:
        type: calico

      ingress:
        baseDomain: example.com
        nginx:
          type: dual

For the full machine definition, read the SIGHUP Distribution on Immutable guide.

Important considerations

No migration path

warning

No migration path is supported, neither from control plane to dedicated etcd nodes, nor viceversa.

Once you deploy a cluster with a specific etcd configuration, you cannot change it. If you need to switch between architectures, you must create a new cluster and migrate your workloads.

Apply the configuration

Before applying, make sure you have:

• Created the PKI with furyctl create pki (see the OnPremises or the Immutable provider page)
• Configured SSH access to your machines setting up the spec.kubernetes.ssh section of the config file (spec.infrastructure.ssh for the Immutable provider)

Then apply the configuration:

furyctl apply

After installation completes, verify etcd is running correctly:

sudo ETCDCTL_API=3 etcdctl \
  --endpoints=https://192.168.1.50:2379,https://192.168.1.51:2379,https://192.168.1.52:2379 \
  --cacert=/etc/etcd/pki/etcd/ca.crt \
  --cert=/etc/etcd/pki/etcd/server.crt \
  --key=/etc/etcd/pki/etcd/server.key \
  endpoint health

All etcd endpoints will report healthy.

tip

Follow the full installation guide in the OnPremises or the Immutable provider documentation for complete details on SSH setup, load balancers, and other required configurations.