3.3. Running the LOCKSS Installer

Section Summary

The next task is to run the LOCKSS Installer. The installation process goes through various phases -- checking system prerequisites, configuring firewall and DNS settings, installing the K3s Kubernetes distribution, and testing the K3s node. After the LOCKSS Installer succeeds, you can also optionally run the K3s Configuration Checker.

Note

Commands in this section are run as root [1].

3.3.1. Invoking the LOCKSS Installer

To start the installation process, run this command as root [1], relative to the LOCKSS Installer Directory:

scripts/install-lockss

The installer will run through its phases, each of which is described in its own section below from Checking K3s Prerequisites (Section 3.3.2) to Completion of the LOCKSS Installation Process (Section 3.3.10).

Tip

Skipping install-lockss phases

You may need to skip some of the phases of install-lockss, for example to overcome an incompatibility with the specifics of your host system. If this is necessary, invoke install-lockss with one or more of the following options:

Option	Phase(s) skipped
--skip-check-prerequisites	Checking K3s Prerequisites (Section 3.3.2)
--skip-check-system-user	Checking the System User and Group (Section 3.3.3)
--skip-configure-iptables	Configuring iptables for K3s (Section 3.3.4)
--skip-configure-firewalld	Configuring firewalld for K3s (Section 3.3.5)
--skip-configure-ufw	Configuring ufw for K3s (Section 3.3.6)
--skip-configure-coredns	Configuring CoreDNS for K3s (Section 3.3.7)
--skip-install-k3s	Checking K3s Prerequisites (Section 3.3.2) Configuring iptables for K3s (Section 3.3.4) Configuring firewalld for K3s (Section 3.3.5) Configuring ufw for K3s (Section 3.3.6) Configuring CoreDNS for K3s (Section 3.3.7) Installing K3s (Section 3.3.8)
--skip-test-k3s	Testing the K3s Node (Section 3.3.9)

When a phase is skipped as a result of one of these options, you will see a message similar to this during the corresponding phase:

[success] Skipping (--skip-configure-firewalld)

Running only one install-lockss phase

Conversely, you may need to run or re-run only one phase of install-lockss, for example re-running the Testing the K3s Node phase after it fails and you perform some troubleshooting. If this is necessary, invoke install-lockss with exactly one of the following options:

Option	Phase executed
--check-prerequisites (or -P)	Checking K3s Prerequisites (Section 3.3.2)
--check-system-user (or -L)	Checking the System User and Group (Section 3.3.3)
--configure-iptables (or -I)	Configuring iptables for K3s (Section 3.3.4)
--configure-firewalld (or -F)	Configuring firewalld for K3s (Section 3.3.5)
--configure-ufw (or -U)	Configuring ufw for K3s (Section 3.3.6)
--configure-coredns (or -C)	Configuring CoreDNS for K3s (Section 3.3.7)
--install-k3s (or -K)	Installing K3s (Section 3.3.8)
--test-k3s (or -T)	Testing the K3s Node (Section 3.3.9)

Running install-lockss on auto-pilot

If you invoke install-lockss with the --assume-yes (or -y) option, it will attempt to run without asking any questions interactively, by assuming that the answer to any yes/no question is "yes" and that the answer to other interactive questions is the suggested default value. This is only appropriate for advanced users who understand the implications of the default code paths in Configuring iptables for K3s (Section 3.3.4), Configuring firewalld for K3s (Section 3.3.5), Configuring ufw for K3s (Section 3.3.6), Configuring CoreDNS for K3s (Section 3.3.7) and Installing K3s (Section 3.3.8) on the host system, for example after previous experience installing the LOCKSS system.

3.3.2. Checking K3s Prerequisites

During this phase, install-lockss will check that certain prerequisites to installing K3s are met. This phase begins with this heading:

Checking K3s prerequisites...

No user interaction is expected; if everything goes well, you will see this message:

[success] K3s prerequisites checked

and install-lockss will successfully proceed to the next phase, Checking the System User and Group (Section 3.3.3).

Error conditions and what to do about them

User namespaces must be enabled in RHEL/CentOS 7

In some RHEL 7 and CentOS 7 systems, user namespaces are not enabled by default. If this is the case, you will see the error message:

[ERROR] User namespaces must be enabled in RHEL/CentOS 7; see manual

and install-lockss will fail. See Enabling User Namespaces in RHEL 7 and CentOS 7 for troubleshooting, then go back to Invoking the LOCKSS Installer to try again.

Apparmor enabled but apparmor_parser missing

In some systems, Apparmor is enabled but apparmor_parser is not installed. If this is the case, you will see the error message:

[ERROR] apparmor enabled but apparmor_parser missing; see manual

and install-lockss will fail. See Installing apparmor_parser for troubleshooting, then go back to Invoking the LOCKSS Installer to try again.

K3s Installer will fail on Debian/Ubuntu with SELinux enabled

On Debian and Ubuntu, the K3s Installer will fail if SELinux is installed and enabled. If this is the case, you will see the error message:

[ERROR] K3s Installer will fail on Debian/Ubuntu with SELinux enabled; see manual

and run install-lockss again.

K3s Installer will fail on Debian/Ubuntu with /usr/local/bin/k3s labeled by SELinux

On Debian and Ubuntu, the K3s Installer will fail if SELinux was previously enabled, and /usr/local/bin/k3s has already been installed and labeled by SELinux. If this is the case, you will see the error message:

[ERROR] K3s Installer will fail on Debian/Ubuntu with /usr/local/bin/k3s labeled by SELinux; see manual

and install-lockss will fail. To fix this, run the following as root:

chcon -t unlabeled_t /usr/local/bin/k3s

and run install-lockss again.

3.3.3. Checking the System User and Group

During this phase, install-lockss will check that the lockss user and group exist on the host system. This phase begins with the heading:

Checking the system user and group...

No user interaction is expected; if everything goes well, you will see this message:

[success] System user and group present

and install-lockss will successfully proceed to the next phase, Configuring iptables for K3s (Section 3.3.4).

Error conditions and what to do about them

lockss user or group does not exist

If the lockss user or group does not exist on the host system, you will see one of these error messages:

[ERROR] The lockss user does not exist

[ERROR] The lockss group does not exist

and install-lockss will fail. Go back to the Creating the lockss User section to create the lockss user and group, then return to Invoking the LOCKSS Installer to try again.

3.3.4. Configuring iptables for K3s

During this phase, install-lockss will configure iptables to work with K3s, if applicable. This phase begins with the heading:

Configuring iptables for K3s...

In many situations, no configuration of iptables is needed; you will see one of these messages:

[success] Skipping (iptables is not on the PATH nor run via Alternatives)

[success] Skipping (iptables version is older than 1.8.0)

[success] Skipping (iptables version is newer than 1.8.3)

[success] Skipping (iptables is in legacy mode)

[success] Skipping (iptables is not run via Alternatives)

and install-lockss will successfully proceed to the next phase, Configuring firewalld for K3s (Section 3.3.5).

Otherwise, you will receive the following prompt:

Switch iptables to legacy mode via Alternatives?

Enter Y to accept the proposed iptables configuration, or enter N to bypass, or hit Enter to accept the default in square brackets [2]. (You may be prompted for your sudo password.)

Caution

If you choose to bypass the proposed iptables configuration, you will see the warning:

[Warning] Leaving iptables unchanged; see manual for details

and install-lockss will keep going. But K3s may malfunction without further intervention; see Troubleshooting iptables for details.

Error conditions and what to do about them

iptables configuration attempt fails

If the iptables configuration attempt fails, you will see one of these error messages:

[ERROR] Error deactivating ufw

[ERROR] Error applying update-alternatives to iptables

[ERROR] Error applying update-alternatives to ip6tables

[ERROR] Error flushing iptables

[ERROR] Error reactivating ufw

and install-lockss will fail. See Troubleshooting iptables for remediation details.

3.3.5. Configuring firewalld for K3s

During this phase, install-lockss will configure firewalld to work with K3s, if applicable. This phase begins with the heading:

Configuring firewalld for K3s...

In many situations, no configuration of firewalld is needed; you will see one of these messages:

[success] Skipping (firewall-cmd is not on the PATH)

[success] Skipping (firewalld is not running)

and install-lockss will successfully proceed to the next phase, Configuring ufw for K3s (Section 3.3.6).

Otherwise, you will receive the following prompt:

Add 10.42.0.0/16 and 10.43.0.0/16 to firewalld's trusted zone?

Enter Y to accept the proposed firewalld configuration, or enter N to bypass, or hit Enter to accept the default in square brackets [2]. (You may be prompted for your sudo password.)

Caution

If you choose to bypass the proposed firewalld configuration, you will see the warning:

[Warning] Leaving firewalld unchanged; see manual for details

and install-lockss will keep going. But K3s may malfunction without further intervention; see Troubleshooting firewalld for details.

Error conditions and what to do about them

firewalld configuration attempt fails

If the firewalld configuration attempt fails, you will see one of these error messages:

[ERROR] Could not add 10.42.0.0/16 to firewalld's trusted zone

[ERROR] Could not add 10.43.0.0/16 to firewalld's trusted zone

[ERROR] Could not reload firewalld

and install-lockss will fail. See Troubleshooting firewalld for remediation details.

3.3.6. Configuring ufw for K3s

During this phase, install-lockss will configure ufw to work with K3s, if necessary. This phase begins with the heading:

Configuring ufw for K3s...

In many situations, no configuration of firewalld is needed; you will see one of these messages:

[success] Skipping (ufw is not on the PATH)

[success] Skipping (ufw is not active)

and install-lockss will successfully proceed to the next phase, Configuring CoreDNS for K3s (Section 3.3.7).

Otherwise, you will receive the following prompt:

Allow traffic from 10.42.0.0/16 and 10.43.0.0/16 via ufw?

Enter Y to accept the proposed ufw configuration, or enter N to bypass, or hit Enter to accept the default in square brackets [2]. (You may be prompted for your sudo password.)

Caution

If you choose to bypass the proposed ufw configuration, you will see the warning:

[Warning] Leaving ufw unchanged; see manual for details

and install-lockss will keep going. But K3s may malfunction without further intervention. See Troubleshooting ufw for details.

Error conditions and what to do about them

ufw configuration attempt fails

If the ufw configuration attempt fails, you will see one of these error messages:

[ERROR] Could not allow traffic from 10.42.0.0/16 via ufw

[ERROR] Could not allow traffic from 10.43.0.0/16 via ufw

[ERROR] Could not reload ufw

and install-lockss will fail. See Troubleshooting ufw for remediation details.

3.3.7. Configuring CoreDNS for K3s

During this phase, install-lockss will configure CoreDNS to work with K3s, if necessary. This phase begins with the heading:

Configuring CoreDNS for K3s...

In many situations, no configuration of firewalld is needed; you will see this message:

[success] Using system resolv.conf files

and install-lockss will successfully proceed to the next phase, Installing K3s (Section 3.3.8).

Otherwise [4], you will receive a message including CoreDNS does not allow a loopback address to be given to Kubernetes pods as an upstream DNS server, and the following prompt:

IP address(es) of DNS resolvers, separated by ';'

Enter a semicolon-separated list of DNS server IP addresses that are not loopback addresses. A suggested value will be offered to you in square brackets, consisting of non-loopback IP addresses collected from your machine's DNS configuration; you can simply hit Enter to accept the suggested value [3].

Error conditions and what to do about them

CoreDNS configuration attempt fails

If the CoreDNS configuration attempt fails, you will see one of these error messages:

[ERROR] Could not create /etc/lockss

[ERROR] Error rendering config/templates/k3s/resolv.conf.mustache to config/resolv.conf

[ERROR] Could not copy config/resolv.conf to /etc/lockss/resolv.conf

and install-lockss will fail. See Troubleshooting CoreDNS for remediation details.

3.3.8. Installing K3s

During this phase, install-lockss will install K3s 1.21.5+k3s1, if applicable. This phase begins with the heading:

Installing K3s...

This phase consists of these steps:

1. First, install-lockss will determine if K3s 1.21.5+k3s1 needs to be installed:

   • If K3s is not present, install-lockss will display K3s is not present, and will install K3s 1.21.5+k3s1 in the next step.

   • If an older version of K3s is present, install-lockss will display Detected K3s version <installed version of K3s> is older than expected version <target version of K3s>, and you will receive the following prompt:

     Upgrade K3s from <installed version of K3s> to <target version of K3s>?

     Enter Y and install-lockss will install K3s 1.21.5+k3s1 in the next step, or enter N and install-lockss will not install K3s 1.21.5+k3s1 in the next step, or hit Enter to accept the default in square brackets [2].

   • If the expected version of K3s is already present, install-lockss will display K3s version <installed version of K3s> is already installed; skipping, and will not install K3s 1.21.5+k3s1 in the next step.

   • If a more recent version of K3s is present, install-lockss will display Detected K3s version <installed version of K3s> is more recent than expected version <target version of K3s>, and will not install K3s 1.21.5+k3s1 in the next step.

   • If K3s is detected but the installed and expected version numbers cannot be compared automatically, install-lockss will display [Warning] Detected K3s version <installed version of K3s>, expected version <target version of K3s>, comparison failure, skipping, and install-lockss will not install K3s in the next step.

2. If install-lockss determined in the previous step that it will not install K3s 1.21.5+k3s1, you will see the confirmation Not installing K3s, and nothing will happen in this step.

   But if install-lockss determined in the previous step that it will install K3s 1.21.5+k3s1, you will see the confirmation Installing K3s version <target version of K3s>, and this step will proceed as follows:

   1. First, install-lockss will ask you to specify the K3s state data directory (the directory K3s uses to store state data), with this prompt:

      K3s state data directory

      By default, this is /var/lib/rancher/k3s. However, if /var is space-limited, you should specify a different directory that has ample space, and is not backed by NFS or by XFS with legacy ftype=0.

      Enter a suitable directory path for the K3s state data directory, or hit Enter to accept the default in square brackets [3] [5].

   2. Then install-lockss will attempt to determine the filesystem type of the specified K3s state data directory. In many situations, it will simply display the filesystem type in a message similar to this (for example, <filesystem type> might be ext4):

      Filesystem type of <path of K3s state data directory> (<mountpoint of K3s state data directory>) is <filesystem type>; proceeding

      Error conditions and warnings, and what to do about them

      Filesystem type of K3s state data directory is NFS

      If the filesystem type backing the K3s state data directory is NFS, you will see the error message:

      [ERROR] Filesystem type of <path of K3s state data directory> (<mountpoint of K3s state data directory>) is NFS; see manual

      and install-lockss will fail. It is not possible to run K3s with a state data directory backed by NFS [6]. Re-run install-lockss and designate a different K3s state data directory that is not backed by NFS.

      Filesystem type of K3s state data directory is XFS with legacy ftype=0

      If the filesystem type backing the K3s state data directory is XFS with legacy ftype=0, you will see the error message:

      [ERROR] Filesystem type of <path of K3s state data directory> (<mountpoint of K3s state data directory>) is XFS with legacy ftype=0; see manual for workaround

      and install-lockss will fail. Contemporary XFS filesystems with modern ftype=1 work well with K3s, but older XFS filesystems with legacy ftype=0 are not compatible [7]. Ideally, re-run install-lockss and designate a different K3s state data directory that is not backed by XFS with legacy ftype=0. Alternatively, you can read about a workaround in Troubleshooting OverlayFS with XFS.

      Filesystem type of K3s state data directory unknown

      If the filesystem type backing the K3s state data directory cannot be inferred automatically, you will see the warning:

      [Warning] Filesystem type of <path of K3s state data directory> unknown (findmnt not present); proceeding

      and install-lockss will keep going. But K3s may malfunction if the actual filesystem type backing the selected K3s state data directory is one that does not work with K3s, such as NFS, or XFS with legacy ftype=0; see the error conditions above.

      Filesystem type of K3s state data directory is XFS but ftype unknown

      If the ftype of the XFS filesystem backing the K3s state data directory cannot be inferred automatically, you will see the warning:

      [Warning] Filesystem type of <path of K3s state data directory> (<mountpoint of K3s state data directory>) is XFS but ftype unknown (xfs_info not present); proceeding

      and install-lockss will keep going. But K3s may malfunction if the actual filesystem type backing the selected K3s state data directory is XFS with legacy ftype=0; see the corresponding error condition above.

   3. Then install-lockss will download the K3s Installer from https://get.k3s.io/ and invoke it with suitable options. This may take several minutes, during which the output to the console will be from the K3s Installer, not from install-lockss.

      Depending on your operating system and other factors, the K3s Installer may install additional software packages or configure system components, using sudo if necessary (which may prompt for the user's sudo password).

      Error conditions and what to do about them

      Error messages from the K3s Installer

      If the K3s Installer does not succeed, it will display its own error messages, then install-lockss will fail. See Troubleshooting the K3s Installer for remediation details.

      Error messages that the K3s Installer may display include:

      [ERROR]  Failed to apply container_runtime_exec_t to /usr/local/bin/k3s, please install:
          yum install -y container-selinux selinux-policy-base
          yum install -y https://rpm.rancher.io/k3s/stable/common/centos/8/noarch/k3s-selinux-0.3-0.el8.noarch.rpm
      

      Error: Package: k3s-selinux-0.3-0.el7.noarch (rancher-k3s-common-stable)
                Requires: container-selinux >= 2.107-3
      You could try using --skip-broken to work around the problem
      You could try running: rpm -Va --nofiles --nodigest
      

3. Finally, whether or not K3s was installed, install-lockss will store Kubernetes configuration data as the lockss user in the file config/k8s.cfg (relative to the LOCKSS Installer Directory).

   Error conditions and what to do about them

   Could not write or append to k8s.cfg

   If the creation of the file fails, you will see one of these error messages:

   [ERROR] Could not write k8s.cfg
   
   [ERROR] Could not append to k8s.cfg
   

   and install-lockss will fail. Check for file permission mismatches between the user running install-lockss and the lockss-installer/config directory, then try again.

3.3.9. Testing the K3s Node

During this phase, install-lockss runs a series of tests to verify that the K3s node is operational. This phase begins with the heading:

Testing the K3s node...

No user interaction is expected. If all tests pass, you will see the message:

[success] Tested the K3s node

and install-lockss will successfully proceed to the next phase, Completion of the LOCKSS Installation Process (Section 3.3.10).

Otherwise, you will see an error message corresponding to the test that did not pass, and install-lockss will fail.

Error conditions and what to do about them

Problems with config/k8s.cfg

At the end of Section 3.3.8 (Installing K3s), some Kubernetes-related data is stored in config/k8s.cfg (relative to the LOCKSS Installer Directory). If the file cannot be found or read, or if it contains invalid or unexpected data, you may see one of these error messages:

[ERROR] k8s.cfg not found

[ERROR] Error reading K8S_FLAVOR

[ERROR] K8S_FLAVOR is not set

[ERROR] K8S_FLAVOR is not k3s

[ERROR] Error reading KUBECTL_CMD

[ERROR] KUBECTL_CMD is not set

[ERROR] k3s command of KUBECTL_CMD is not on the PATH

Check the contents of config/k8s.cfg and contact us () for troubleshooting if necessary.

Problems with the K3s node

If the K3s node is not behaving as expected, you may see one of these errors:

[ERROR] Command failed (kubectl version)

[ERROR] Timeout waiting for the K3s node to be ready

[ERROR] Command failed (kubectl get node)

[ERROR] Unexpected number of K3s nodes

If the K3s node is newly installed, it may simply be that there has not yet been enough time for it to come up; you can re-run this phase with scripts/install-lockss --test-k3s (or scripts/install-lockss -K) to retry. Contact us () for troubleshooting if necessary.

Problems with DNS

If the K3s node's DNS environment is not working properly, you may see one of these errors:

[ERROR] Timeout waiting for the CoreDNS pod to be running and ready

[ERROR] Command failed (kubectl get pod)

[ERROR] Unexpected number of CoreDNS pods

[ERROR] Timeout waiting for the DNS service to be present

[ERROR] Command failed (kubectl get service)

[ERROR] Unexpected number of kube-dns services

[ERROR] Unexpected kube-dns service type

[ERROR] Timeout waiting for DNS resolution

[ERROR] Unexpected Cluster-IP

If the K3s node is newly installed, it may simply be that there has not yet been enough time for CoreDNS to come up; you can re-run this phase with scripts/install-lockss --test-k3s (or scripts/install-lockss -T) to retry. You can also use the install-lockss options --retries=N (to increase the number of retries in DNS lookup tests to N from 5) and/or --wait=S (to increase the delay between retries in DNS lookup tests to S seconds from 10). Contact us () for troubleshooting if necessary.

3.3.10. Completion of the LOCKSS Installation Process

If all phases completed successfully, you will see the message:

[success] Successful completion of the LOCKSS installation process

and install-lockss will terminate.

3.3.11. Checking the K3s Configuration

Tip

This section is optional.

K3s comes with k3s check-config, a configuration checker tool. The K3s configuration checker is capable of detecting complex underlying system situations that definitely require fixing (or applications running in the K3s cluster will not be able to function properly). On the other hand, the versions of the K3s configuration checker available at the time LOCKSS 2.0-beta1 was released contained bugs that reported spurious issues that are either inaccurate or moot. As a result, we have decided against running k3s check-config as part of install-lockss at this time, to avoid unnecessary interruptions in the installation of the LOCKSS system in many cases where there is no particular cause for concern.

That being said, we still recommend running k3s check-config and interpreting the results using the Troubleshooting the K3s Configuration Checker section of the manual:

1. Run this command:

   k3s check-config
   

2. The following error messages in the output are indicative of system situations that require attention:

   /usr/sbin iptables v1.8.2 (nf_tables): should be older than v1.8.0, newer than v1.8.3, or in legacy mode (fail)
   

   RHEL7/CentOS7: User namespaces disabled; add 'user_namespace.enable=1' to boot command line (fail)
   

   apparmor: enabled, but apparmor_parser missing (fail)
   

   Troubleshooting

   See Troubleshooting the K3s Configuration Checker for details.

3. The following error messages in the output can be ignored:

   cgroup hierarchy: nonexistent?? (fail)
     (see https://github.com/tianon/cgroupfs-mount)
   

   links: aux/ip6tables should link to iptables-detect.sh (fail)
   links: aux/ip6tables-restore should link to iptables-detect.sh (fail)
   links: aux/ip6tables-save should link to iptables-detect.sh (fail)
   links: aux/iptables should link to iptables-detect.sh (fail)
   links: aux/iptables-restore should link to iptables-detect.sh (fail)
   links: aux/iptables-save should link to iptables-detect.sh (fail)
   

   swap: should be disabled
   

   CONFIG_INET_XFRM_MODE_TRANSPORT: missing
   

   Troubleshooting

   See Troubleshooting the K3s Configuration Checker for details.

4. For other error messages, check the official K3s documentation, search for K3s issues database on GitHub or the Web for resources matching your error message or operating system, and/or contact us so we can help investigate and document for future reference.

---

Footnotes

[1]

See Running Commands as root.

[2]

If install-lockss was invoked with the --assume-yes option, Y is automatically entered for you.

[3]

If install-lockss was invoked with the --assume-yes option, the suggested value is automatically accepted for you.

[4]

Or if your install-lockss was invoked with the --force-dns-prompt option.

[5]

If install-lockss was invoked with the --k3s-data-dir=DIR option, DIR will automatically be used without the prompt.

[6]

See https://github.com/containerd/containerd/discussions/6140.

[7]

See https://docs.docker.com/storage/storagedriver/overlayfs-driver/#prerequisites.