3.3. Running the LOCKSS Installer

Note

Commands in this section are run as root 1.

The next task is to run the LOCKSS Installer.

3.3.1. Overview of the LOCKSS Installer

When you invoke the LOCKSS Installer, the installation process goes through various phases:

  • Checking that some prerequisites to install K3s are met. No user interaction is expected.

  • Checking that the lockss system user and group exist. No user interaction is expected.

  • Configuring iptables, firewalld and ufw for K3s. If applicable, you will be prompted to confirm before your system configuration is modified. You may incidentally be prompted for your sudo password.

  • Configuring CoreDNS for K3s. If applicable, you will be prompted to enter non-loopback IP addresses of DNS servers.

  • Installing K3s. If applicable, you will be prompted for a Kubernetes state data storage directory.

  • Testing the K3s node. No user interaction is expected.

After the LOCKSS Installer succeeds, you can also optionally run the K3s Configuration Checker.

3.3.2. Invoking the LOCKSS Installer

To start the installation process, run this command (relative to the lockss-installer-directory) as root 1:

scripts/install-lockss

The installer will run through its phases, each of which is described in its own section below. The first phase is Checking K3s Prerequisites (Section 3.3.3).

3.3.3. Checking K3s Prerequisites

Heading

This phase begins with the heading Checking K3s prerequisites....

Description

During this phase, install-lockss will check that certain prerequisites to installing K3s are met.

Steps

  1. If install-lockss was invoked with the --skip-check-prerequisites option (implied by --skip-install-k3s), you will see one of these messages:

    [success] Skipping (--skip-install-k3s)
    
    [success] Skipping (--skip-check-prerequisites)
    

    and install-lockss will successfully proceed to the next phase, Checking the System User and Group (Section 3.3.4).

  2. Next, install-lockss will check that user namespaces are enabled. In some RHEL 7 and CentOS 7 systems, user namespaces are not enabled by default; if this is the case, you will see the error message:

    [ERROR] User namespaces must be enabled in RHEL/CentOS 7; see manual
    

    and install-lockss will fail.

    Troubleshooting

    See Enabling User Namespaces in RHEL 7 and CentOS 7 for troubleshooting, then go back to Invoking the LOCKSS Installer to try again.

  3. Then install-lockss will check that apparmor_parser is installed if Apparmor is enabled. If Apparmor is enabled but apparmor_parser is not installed, you will see the error message:

    [ERROR] apparmor enabled but apparmor_parser missing; see manual
    

    and install-lockss will fail.

    Troubleshooting

    See Installing apparmor_parser for troubleshooting, then go back to Invoking the LOCKSS Installer to try again.

  4. Finally, you will see the message:

    [success] K3s prerequisites checked
    

    and install-lockss will successfully proceed to the next phase, Checking the System User and Group (Section 3.3.4).

3.3.4. Checking the System User and Group

Heading

This phase begins with the heading Checking the system user and group....

Description

During this phase, install-lockss will check that the lockss user and group exist on the host system.

Steps

  1. If install-lockss was invoked with the --skip-check-system-user option, you will see the message:

    [success] Skipping (--skip-check-system-user)
    

    and install-lockss will successfully proceed to the next phase, Configuring iptables for K3s (Section 3.3.5).

  2. If the lockss user or group does not exist on the host system, you will see one of these error messages:

    [ERROR] The lockss user does not exist
    
    [ERROR] The lockss group does not exist
    

    and install-lockss will fail.

    Troubleshooting

    See the Creating the lockss User section to create the lockss user and group, then go back to Invoking the LOCKSS Installer to try again.

  3. Finally, you will see the message:

    [success] System user and group present
    

    and install-lockss will successfully proceed to the next phase, Configuring iptables for K3s (Section 3.3.5).

3.3.5. Configuring iptables for K3s

Heading

This phase begins with the heading Configuring iptables for K3s....

Description

During this phase, install-lockss will configure iptables to work with K3s, if applicable.

Steps

  1. If install-lockss was invoked with the --skip-configure-iptables option (implied by --skip-install-k3s), or if no changes to the configuration of iptables are necessary, you will see one of these messages:

    [success] Skipping (--skip-install-k3s)
    
    [success] Skipping (--skip-configure-iptables)
    
    [success] Skipping (iptables is not on the PATH nor run via Alternatives)
    
    [success] Skipping (iptables version is older than 1.8.0)
    
    [success] Skipping (iptables version is newer than 1.8.3)
    
    [success] Skipping (iptables is in legacy mode)
    
    [success] Skipping (iptables is not run via Alternatives)
    

    and install-lockss will successfully proceed to the next phase, Configuring firewalld for K3s (Section 3.3.6).

  2. Otherwise, you will receive the following prompt:

    Switch iptables to legacy mode via Alternatives?

    Enter Y to accept the proposed iptables configuration or N to bypass (or hit Enter to accept the default in square brackets).

    • If install-lockss was invoked with the --assume-yes option, Y is automatically entered for you.

    • You may be prompted for your sudo password.

    Warning

    If you bypass the proposed iptables configuration, you will see the warning:

    [Warning] Leaving iptables unchanged; see manual for details
    

    and install-lockss will immediately proceed to the next phase, Configuring firewalld for K3s (Section 3.3.6). But K3s may malfunction without further intervention; see Troubleshooting iptables for details.

  3. If the iptables configuration attempt fails, you will see one of these error messages:

    [ERROR] Error deactivating ufw
    
    [ERROR] Error applying update-alternatives to iptables
    
    [ERROR] Error applying update-alternatives to ip6tables
    
    [ERROR] Error flushing iptables
    
    [ERROR] Error reactivating ufw
    

    and install-lockss will fail.

    Troubleshooting

    See Troubleshooting iptables for remediation details.

  4. Finally, you will see the message:

    [success] Configured iptables for K3s
    

    and install-lockss will successfully proceed to the next phase, Configuring firewalld for K3s (Section 3.3.6).

3.3.6. Configuring firewalld for K3s

Heading

This phase begins with the heading Configuring firewalld for K3s....

Description

During this phase, install-lockss will configure firewalld to work with K3s, if applicable.

Steps

  1. If install-lockss was invoked with the --skip-configure-firewalld option (implied by --skip-install-k3s), or if firewalld is not present or is not running, you will see one of these messages:

    [success] Skipping (--skip-install-k3s)
    
    [success] Skipping (--skip-configure-firewalld)
    
    [success] Skipping (firewall-cmd is not on the PATH)
    
    [success] Skipping (firewalld is not running)
    

    and install-lockss will successfully proceed to the next phase, Configuring ufw for K3s (Section 3.3.7).

  2. If firewalld is running, you will receive the following prompt:

    Add 10.42.0.0/16 and 10.43.0.0/16 to firewalld's trusted zone?

    Enter Y to accept the proposed firewalld configuration or N to bypass (or hit Enter to accept the default in square brackets).

    • If install-lockss was invoked with the --assume-yes option, Y is automatically entered for you.

    • You may be prompted for your sudo password.

    Warning

    If you bypass the proposed firewalld configuration, you will see the warning:

    [Warning] Leaving firewalld unchanged; see manual for details
    

    and install-lockss will immediately proceed to the next phase, Configuring ufw for K3s (Section 3.3.7). But K3s may malfunction without further intervention; see Troubleshooting firewalld for details.

  3. If the firewalld configuration attempt fails, you will see one of these error messages:

    [ERROR] Could not add 10.42.0.0/16 to firewalld's trusted zone
    
    [ERROR] Could not add 10.43.0.0/16 to firewalld's trusted zone
    
    [ERROR] Could not reload firewalld
    

    and install-lockss will fail.

    Troubleshooting

    See Troubleshooting firewalld for remediation details.

  4. Finally, you will see the message:

    [success] Configured firewalld for K3s
    

    and install-lockss will successfully proceed to the next phase Configuring ufw for K3s (Section 3.3.7).

3.3.7. Configuring ufw for K3s

Heading

This phase begins with the heading Configuring firewalld for ufw....

Description

During this phase, install-lockss will configure ufw to work with K3s, if necessary.

Steps

  1. If install-lockss was invoked with the --skip-configure-ufw option (implied by --skip-install-k3s), or if ufw is not present or is not active, you will see one of these messages:

    [success] Skipping (--skip-install-k3s)
    
    [success] Skipping (--skip-configure-ufw)
    
    [success] Skipping (ufw is not on the PATH)
    
    [success] Skipping (ufw is not active)
    

    and install-lockss will successfully proceed to the next phase, Configuring CoreDNS for K3s (Section 3.3.8).

  2. If ufw is active, you will receive the following prompt:

    Allow traffic from 10.42.0.0/16 and 10.43.0.0/16 via ufw?

    Enter Y to accept the proposed ufw configuration or N to bypass (or hit Enter to accept the default in square brackets).

    • If install-lockss was invoked with the --assume-yes option, Y is automatically entered for you.

    • You may be prompted for your sudo password.

    Warning

    If you bypass the proposed ufw configuration, you will see the warning:

    [Warning] Leaving ufw unchanged; see manual for details
    

    and install-lockss will immediately proceed to the next phase, Configuring CoreDNS for K3s (Section 3.3.8). But K3s may malfunction without further intervention. See Troubleshooting ufw for details.

  3. If the ufw configuration attempt fails, you will see one of these error messages:

    [ERROR] Could not allow traffic from 10.42.0.0/16 via ufw
    
    [ERROR] Could not allow traffic from 10.43.0.0/16 via ufw
    
    [ERROR] Could not reload ufw
    

    and install-lockss will fail.

    Troubleshooting

    See Troubleshooting ufw for remediation details.

  4. Finally, you will see the message:

    [success] Configured ufw for K3s
    

    and install-lockss will successfully proceed to the next phase, Configuring CoreDNS for K3s (Section 3.3.8).

3.3.8. Configuring CoreDNS for K3s

Heading

This phase begins with the heading Configuring CoreDNS for K3s....

Description

During this phase, install-lockss will configure CoreDNS to work with K3s, if necessary.

Steps

  1. If install-lockss was invoked with the --skip-configure-coredns option (implied by --skip-install-k3s), or if your system's DNS configuration will simply work with CoreDNS, you will see one of these messages:

    [success] Skipping (--skip-install-k3s)
    
    [success] Skipping (--skip-configure-dns)
    
    [success] Using system resolv.conf files
    

    and install-lockss will successfully proceed to the next phase, Installing K3s (Section 3.3.9).

  2. If your system's DNS configuration will not work with CoreDNS, or if install-lockss was invoked with the --force-dns-prompt option, you will receive a message including CoreDNS does not allow a loopback address to be given to Kubernetes pods as an upstream DNS server, and the following prompt:

    IP address(es) of DNS resolvers, separated by ';'

    Enter a semicolon-separated list of DNS server IP addresses that are not loopback addresses. A suggested value will be offered to you in square brackets, consisting of non-loopback IP addresses collected from your machine's DNS configuration; you can simply hit Enter to accept the suggested value.

    • If install-lockss was invoked with the --assume-yes option, the suggested value is automatically accepted witout the prompt.

  3. If the creation of the CoreDNS configuration file fails, you will see error messages similar to these:

    [ERROR] Could not create /etc/lockss
    
    [ERROR] Error rendering config/templates/k3s/resolv.conf.mustache to config/resolv.conf
    
    [ERROR] Could not copy config/resolv.conf to /etc/lockss/resolv.conf
    

    and install-lockss will fail.

    Troubleshooting

    See Troubleshooting CoreDNS for remediation details.

  4. Finally, you will see the message:

    [success] Configured CoreDNS for K3s
    

    and install-lockss will successfully proceed to the next phase, Installing K3s (Section 3.3.9).

3.3.9. Installing K3s

Heading

This phase begins with the heading Installing K3s....

Description

During this phase, install-lockss will install K3s.

Steps

  1. If install-lockss was invoked with the --skip-install-k3s option, you will see the message:

    [success] Skipping (--skip-install-k3s)
    

    and install-lockss will successfully proceed to the next phase, Testing the K3s Node (Section 3.3.10).

  2. Next, install-lockss will determine if K3s needs to be installed or upgraded. There are five cases:

    • Case 1: If K3s is not present, install-lockss will display K3s is not present, and will install K3s.

    • Case 2: If the expected version of K3s is already present, install-lockss will display K3s version installed_version is already installed; skipping, and will not install K3s.

    • Case 3: If a more recent version of K3s is present, install-lockss will display Detected K3s version installed_version is more recent than expected version expected_version, and will not install K3s.

    • Case 4: If an older version of K3s is present, install-lockss will display Detected K3s version installed_version is older than expected version expected_version and you will receive the following prompt:

      Upgrade K3s from {installed_version} to {expected_version}?

      Enter Y and install-lockss will install the newer K3s version, or N and install-lockss will not install the newer K3s version (or hit Enter to accept the default in square brackets).

      • If install-lockss was invoked with the --assume-yes option, Y is automatically entered for you.

    • Case 5: If K3s is detected but the installed and expected version numbers cannot be compared automatically, you will see the following warning:

      [Warning] Detected K3s version installed_version, expected version expected_version, comparison failure, skipping

      and install-lockss will not install K3s.

    If install-lockss determines that it will not install K3s, it will confirm Not installing K3s, then will skip the next 3 steps.

    Otherwise, install-lockss will confirm Installing K3s version expected_version and will simply proceed to the next step.

  3. First, install-lockss will warn you that if the directory K3s uses to store state data (by default /var/lib/rancher/k3s) is space-limited, you should specify a different directory. You will see the following prompt:

    K3s state data directory

    Enter a suitable directory path for the K3s state data directory, or simply hit Enter to accept the default in square brackets.

    • If install-lockss was invoked with the --k3s-data-dir=DIR option, DIR will automatically be used without the prompt.

    • If install-lockss was invoked with the --assume-yes option, the default is automatically used without the prompt.

  4. Next, install-lockss will detect the filesystem type backing the K3s state data directory, because some filesystem types (like NFS, or XFS with legacy ftype=0) are not suitable or require special handling:

    1. If the filesystem type backing the K3s state data directory cannot be inferred automatically, you will see the warning:

      [Warning] Filesystem type of k3s_dir unknown (findmnt not present); proceeding

      and install-lockss will proceed to the next step.

    2. If the filesystem type backing the K3s state data directory is NFS, you will see the error message:

      [ERROR] Filesystem type of k3s_dir (k3s_mountpoint) is NFS; see manual

      and install-lockss will fail.

    3. If the filesystem type backing the K3s state data directory is XFS, install-lockss will determine its ftype (an internal characteristic):

      1. If the XFS filesystem backing the K3s state data directory has legacy ftype=0, you will see the error message:

      [ERROR] Filesystem type of k3s_dir (k3s_mountpoint) is XFS with legacy ftype=0; see manual for workaround

      and install-lockss will fail.

      1. If the XFS filesystem backing the K3s state data directory does not have legacy ftype=0, you will see one of these messages:

      Filesystem type of k3s_dir (k3s_mountpoint) is XFS with ftype=1; proceeding

      Filesystem type of k3s_dir (k3s_mountpoint) is XFS but not with legacy ftype=0; proceeding

      and install-lockss will proceed to the next step.

      1. If the ftype of the XFS filesystem backing the K3s state data directory cannot be inferred automatically, you will see the warning:

        [Warning] Filesystem type of k3s_dir (k3s_mountpoint) is XFS but ftype unknown (xfs_info not present); proceeding

      and install-lockss will proceed to the next step.

      1. Otherwise, install-lockss will display the filesystem type backing the K3s state data directory in a message similar to this:

      Filesystem type of k3s_dir (k3s_mountpoint) is fs_type; proceeding

      (for some filesystem type fs_type, for example ext4), and will proceed to the next step.

  5. Finally, the K3s Installer will be downloaded from https://get.k3s.io/ and invoked with suitable options.

    Depending on your operating system and other factors, the K3s Installer may install additional software packages or configure system components, using sudo if necessary (which may prompt for the user's sudo password).

    If the K3s Installer does not succeed, it will display its own error messages, then install-lockss will fail.

    Troubleshooting

    Error messages that the K3s Installer may display include:

    [ERROR]  Failed to apply container_runtime_exec_t to /usr/local/bin/k3s, please install:
        yum install -y container-selinux selinux-policy-base
        yum install -y https://rpm.rancher.io/k3s/stable/common/centos/8/noarch/k3s-selinux-0.3-0.el8.noarch.rpm
    
    Error: Package: k3s-selinux-0.3-0.el7.noarch (rancher-k3s-common-stable)
               Requires: container-selinux >= 2.107-3
     You could try using --skip-broken to work around the problem
     You could try running: rpm -Va --nofiles --nodigest
    

    See Troubleshooting the K3s Installer for remediation details.

  1. Whether or not K3s was installed, install-lockss will store Kubernetes configuration data as the lockss user in the file config/k8s.cfg, relative to the LOCKSS Installer home directory. If the creation of the file fails, you will see one of these error messages:

    [ERROR] Could not write k8s.cfg
    
    [ERROR] Could not append to k8s.cfg
    

    and install-lockss will fail.

    Troubleshooting

    Check file permission mismatches between the user running install-lockss and the lockss-installer/config directory, then try again.

  2. Finally, you will see the message:

    [success] Installed K3s
    

    and install-lockss will successfully proceed to the next phase (Testing the K3s Node).

3.3.10. Testing the K3s Node

Heading

This phase begins with the heading Testing the K3s node....

Description

During this phase, install-lockss runs a series of tests to verify that the K3s node is operational.

Steps

  1. If install-lockss was invoked with the --skip-test-k3s option (implied by --skip-install-k3s), you will see one of these messages:

    [success] Skipping (--skip-install-k3s)
    
    [success] Skipping (--skip-test-k3s)
    

    and install-lockss will successfully proceed to the next phase, Completion of the LOCKSS Installation Process (Section 3.3.11).

  2. Next, install-lockss will run a series of tests. If a test fails, you will see one of these error messages:

    [ERROR] k8s.cfg not found
    
    [ERROR] Error reading K8S_FLAVOR
    
    [ERROR] K8S_FLAVOR is not set
    
    [ERROR] K8S_FLAVOR is not k3s
    
    [ERROR] Error reading KUBECTL_CMD
    
    [ERROR] KUBECTL_CMD is not set
    
    [ERROR] k3s command of KUBECTL_CMD is not on the PATH
    
    [ERROR] Command failed (kubectl version)
    
    [ERROR] Timeout waiting for the K3s node to be ready
    
    [ERROR] Command failed (kubectl get node)
    
    [ERROR] Unexpected number of K3s nodes
    
    [ERROR] Timeout waiting for the CoreDNS pod to be running and ready
    
    [ERROR] Command failed (kubectl get pod)
    
    [ERROR] Unexpected number of CoreDNS pods
    
    [ERROR] Timeout waiting for the DNS service to be present
    
    [ERROR] Command failed (kubectl get service)
    
    [ERROR] Unexpected number of kube-dns services
    
    [ERROR] Unexpected kube-dns service type
    
    [ERROR] Timeout waiting for DNS resolution
    
    [ERROR] Unexpected Cluster-IP
    

    and install-lockss will fail.

    Troubleshooting

    The reasons for some of these tests failing vary. Some wait for K3s to start up and retry a number of times but eventually give up, even though K3s will eventually come up fully. You can invoke just this portion of lockss-install by invoking:

    install-lockss --test-k3s
    

    or equivalently:

    install-lockss -T
    

    You can also alter the number of retries and the number of seconds between retries with --retries=N and --wait=S respectively.

    Other problems may require reaching out to the LOCKSS support team at for assistance.

  3. Finally, you will see the message:

    [success] Tested the K3s node
    

    and install-lockss will successfully proceed to the next phase, Completion of the LOCKSS Installation Process (Section 3.3.11).

3.3.11. Completion of the LOCKSS Installation Process

If all phases completed successfully, you will see the message:

[success] Successful completion of the LOCKSS installation process

and install-lockss will terminate.

3.3.12. Checking the K3s Configuration

Tip

This section is optional.

K3s comes with k3s check-config, a configuration checker tool. The K3s configuration checker is capable of detecting complex underlying system situations that definitely require fixing (or applications running in the K3s cluster will not be able to function properly). On the other hand, the versions of the K3s configuration checker available at the time LOCKSS 2.0-alpha4 and LOCKSS 2.0-alpha5 were released contained bugs that reported spurious issues that are either inaccurate or moot. As a result, we have decided against running k3s check-config as part of install-lockss at this time, to avoid unnecessary interruptions in the installation of the LOCKSS system in many cases where there is no particular cause for concern.

That being said, we still recommend running k3s check-config and interpreting the results using the Troubleshooting the K3s Configuration Checker section of the manual:

  1. Run this command:

    k3s check-config
    
  2. The following error messages in the output are indicative of system situations that require attention:

    /usr/sbin iptables v1.8.2 (nf_tables): should be older than v1.8.0, newer than v1.8.3, or in legacy mode (fail)
    
    RHEL7/CentOS7: User namespaces disabled; add 'user_namespace.enable=1' to boot command line (fail)
    
    apparmor: enabled, but apparmor_parser missing (fail)
    

    Troubleshooting

    See Troubleshooting the K3s Configuration Checker for details.

  3. The following error messages in the output can be ignored:

    cgroup hierarchy: nonexistent?? (fail)
    
    links: aux/ip6tables should link to iptables-detect.sh (fail)
    links: aux/ip6tables-restore should link to iptables-detect.sh (fail)
    links: aux/ip6tables-save should link to iptables-detect.sh (fail)
    links: aux/iptables should link to iptables-detect.sh (fail)
    links: aux/iptables-restore should link to iptables-detect.sh (fail)
    links: aux/iptables-save should link to iptables-detect.sh (fail)
    
    swap: should be disabled
    
    CONFIG_INET_XFRM_MODE_TRANSPORT: missing
    

    Troubleshooting

    See Troubleshooting the K3s Configuration Checker for details.

  4. For other error messages, check the official K3s documentation, search for K3s issues database on GitHub or the Web for resources matching your error message or operating system, and/or contact us so we can help investigate and document for future reference.


Footnotes

1

See Running Commands as root.

2

See https://github.com/containerd/containerd/discussions/6140.

3

See https://docs.docker.com/storage/storagedriver/overlayfs-driver/#prerequisites.