3.3. Running the LOCKSS Installer
Note
Commands in this section are run as root
1.
The next task is to run the LOCKSS Installer.
The installation process goes through various phases:
Checking that some prerequisites to install K3s are met. No user interaction is expected.
Checking that the
lockss
system user and group exist. No user interaction is expected.Configuring iptables, firewalld and ufw for K3s. If applicable, you will be prompted to confirm before your system configuration is modified. You may incidentally be prompted for your sudo password.
Configuring CoreDNS for K3s. If applicable, you will be prompted to enter non-loopback IP addresses of DNS servers.
Installing K3s. If applicable, you will be prompted for a Kubernetes state data storage directory.
Testing the K3s node. No user interaction is expected.
After the LOCKSS Installer succeeds, you can also optionally run the K3s Configuration Checker.
3.3.1. Invoking the LOCKSS Installer
To start the installation process, run this command (relative to the LOCKSS Installer Directory) as root
1:
scripts/install-lockss
The installer will run through its phases, each of which is described in its own section below from Checking K3s Prerequisites (Section 3.3.2) to Completion of the LOCKSS Installation Process (Section 3.3.10).
Tip
Below are some advanced tips for this section.
Skipping install-lockss phases
You may need to skip some of the phases of install-lockss, for example to overcome an incompatibility with the specifics of your host system. If this is necessary, invoke install-lockss with one or more of the following options:
Option |
Phase(s) skipped |
---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
When a phase is skipped as a result of one of these options, you will see a message similar to this during the corresponding phase:
[success] Skipping (--skip-configure-firewalld)
Running only one install-lockss phase
Conversely, you may need to run or re-run only one phase of install-lockss, for example re-running the Testing the K3s Node phase after it fails and you perform some troubleshooting. If this is necessary, invoke install-lockss with exactly one of the following options:
Option |
Phase executed |
---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Running install-lockss on auto-pilot
If you invoke install-lockss with the --assume-yes
(or -y
) option, it will attempt to run without asking any questions interactively, by assuming that the answer to any yes/no question is "yes" and that the answer to other interactive questions is the suggested default value. This is only appropriate for advanced users who understand the implications of the default code paths in Configuring iptables for K3s (Section 3.3.4), Configuring firewalld for K3s (Section 3.3.5), Configuring ufw for K3s (Section 3.3.6), Configuring CoreDNS for K3s (Section 3.3.7) and Installing K3s (Section 3.3.8) on the host system, for example after previous experience installing the LOCKSS system.
3.3.2. Checking K3s Prerequisites
During this phase, install-lockss will check that certain prerequisites to installing K3s are met. This phase begins with this heading:
Checking K3s prerequisites...
No user interaction is expected; if everything goes well, you will see this message:
[success] K3s prerequisites checked
and install-lockss will successfully proceed to the next phase, Checking the System User and Group (Section 3.3.3).
Error
Below are some error conditions you may encounter here and what to do about them.
User namespaces must be enabled in RHEL/CentOS 7
In some RHEL 7 and CentOS 7 systems, user namespaces are not enabled by default. If this is the case, you will see the error message:
[ERROR] User namespaces must be enabled in RHEL/CentOS 7; see manual
and install-lockss will fail. See Enabling User Namespaces in RHEL 7 and CentOS 7 for troubleshooting, then go back to Invoking the LOCKSS Installer to try again.
Apparmor enabled but apparmor_parser missing
In some systems, Apparmor is enabled but apparmor_parser is not installed. If this is the case, you will see the error message:
[ERROR] apparmor enabled but apparmor_parser missing; see manual
and install-lockss will fail. See Installing apparmor_parser for troubleshooting, then go back to Invoking the LOCKSS Installer to try again.
3.3.3. Checking the System User and Group
During this phase, install-lockss will check that the lockss
user and group exist on the host system. This phase begins with the heading:
Checking the system user and group...
No user interaction is expected; if everything goes well, you will see this message:
[success] System user and group present
and install-lockss will successfully proceed to the next phase, Configuring iptables for K3s (Section 3.3.4).
Error
Below are some error conditions you may encounter here and what to do about them.
lockss
user or group does not exist
If the lockss
user or group does not exist on the host system, you will see one of these error messages:
[ERROR] The lockss user does not exist
[ERROR] The lockss group does not exist
and install-lockss will fail. Go back to the Creating the lockss User section to create the lockss
user and group, then return to Invoking the LOCKSS Installer to try again.
3.3.4. Configuring iptables for K3s
During this phase, install-lockss will configure iptables to work with K3s, if applicable. This phase begins with the heading:
Configuring iptables for K3s...
In many situations, no configuration of iptables is needed; you will see one of these messages:
[success] Skipping (iptables is not on the PATH nor run via Alternatives)
[success] Skipping (iptables version is older than 1.8.0)
[success] Skipping (iptables version is newer than 1.8.3)
[success] Skipping (iptables is in legacy mode)
[success] Skipping (iptables is not run via Alternatives)
and install-lockss will successfully proceed to the next phase, Configuring firewalld for K3s (Section 3.3.5).
Otherwise, you will receive the following prompt:
Switch iptables to legacy mode via Alternatives?
Enter Y to accept the proposed iptables configuration, or enter N to bypass, or hit Enter to accept the default in square brackets 2. (You may be prompted for your sudo password.)
Caution
If you choose to bypass the proposed iptables configuration, you will see the warning:
[Warning] Leaving iptables unchanged; see manual for details
and install-lockss will keep going. But K3s may malfunction without further intervention; see Troubleshooting iptables for details.
Error
Below are some error conditions you may encounter here and what to do about them.
iptables configuration attempt fails
If the iptables configuration attempt fails, you will see one of these error messages:
[ERROR] Error deactivating ufw
[ERROR] Error applying update-alternatives to iptables
[ERROR] Error applying update-alternatives to ip6tables
[ERROR] Error flushing iptables
[ERROR] Error reactivating ufw
and install-lockss will fail. See Troubleshooting iptables for remediation details.
3.3.5. Configuring firewalld for K3s
During this phase, install-lockss will configure firewalld to work with K3s, if applicable. This phase begins with the heading:
Configuring firewalld for K3s...
In many situations, no configuration of firewalld is needed; you will see one of these messages:
[success] Skipping (firewall-cmd is not on the PATH)
[success] Skipping (firewalld is not running)
and install-lockss will successfully proceed to the next phase, Configuring ufw for K3s (Section 3.3.6).
Otherwise, you will receive the following prompt:
Add 10.42.0.0/16 and 10.43.0.0/16 to firewalld's trusted zone?
Enter Y to accept the proposed firewalld configuration, or enter N to bypass, or hit Enter to accept the default in square brackets 2. (You may be prompted for your sudo password.)
Caution
If you choose to bypass the proposed firewalld configuration, you will see the warning:
[Warning] Leaving firewalld unchanged; see manual for details
and install-lockss will keep going. But K3s may malfunction without further intervention; see Troubleshooting firewalld for details.
Error
Below are some error conditions you may encounter here and what to do about them.
firewalld configuration attempt fails
If the firewalld configuration attempt fails, you will see one of these error messages:
[ERROR] Could not add 10.42.0.0/16 to firewalld's trusted zone
[ERROR] Could not add 10.43.0.0/16 to firewalld's trusted zone
[ERROR] Could not reload firewalld
and install-lockss will fail. See Troubleshooting firewalld for remediation details.
3.3.6. Configuring ufw for K3s
During this phase, install-lockss will configure ufw to work with K3s, if necessary. This phase begins with the heading:
Configuring ufw for K3s...
In many situations, no configuration of firewalld is needed; you will see one of these messages:
[success] Skipping (ufw is not on the PATH)
[success] Skipping (ufw is not active)
and install-lockss will successfully proceed to the next phase, Configuring CoreDNS for K3s (Section 3.3.7).
Otherwise, you will receive the following prompt:
Allow traffic from 10.42.0.0/16 and 10.43.0.0/16 via ufw?
Enter Y to accept the proposed ufw configuration, or enter N to bypass, or hit Enter to accept the default in square brackets 2. (You may be prompted for your sudo password.)
Caution
If you choose to bypass the proposed ufw configuration, you will see the warning:
[Warning] Leaving ufw unchanged; see manual for details
and install-lockss will keep going. But K3s may malfunction without further intervention. See Troubleshooting ufw for details.
Error
Below are some error conditions you may encounter here and what to do about them.
ufw configuration attempt fails
If the ufw configuration attempt fails, you will see one of these error messages:
[ERROR] Could not allow traffic from 10.42.0.0/16 via ufw
[ERROR] Could not allow traffic from 10.43.0.0/16 via ufw
[ERROR] Could not reload ufw
and install-lockss will fail. See Troubleshooting ufw for remediation details.
3.3.7. Configuring CoreDNS for K3s
During this phase, install-lockss will configure CoreDNS to work with K3s, if necessary. This phase begins with the heading:
Configuring CoreDNS for K3s...
In many situations, no configuration of firewalld is needed; you will see this message:
[success] Using system resolv.conf files
and install-lockss will successfully proceed to the next phase, Installing K3s (Section 3.3.8).
Otherwise 4, you will receive a message including CoreDNS does not allow a loopback address to be given to Kubernetes pods as an upstream DNS server
, and the following prompt:
IP address(es) of DNS resolvers, separated by ';'
Enter a semicolon-separated list of DNS server IP addresses that are not loopback addresses. A suggested value will be offered to you in square brackets, consisting of non-loopback IP addresses collected from your machine's DNS configuration; you can simply hit Enter to accept the suggested value 3.
Error
Below are some error conditions you may encounter here and what to do about them.
CoreDNS configuration attempt fails
If the CoreDNS configuration attempt fails, you will see one of these error messages:
[ERROR] Could not create /etc/lockss
[ERROR] Error rendering config/templates/k3s/resolv.conf.mustache to config/resolv.conf
[ERROR] Could not copy config/resolv.conf to /etc/lockss/resolv.conf
and install-lockss will fail. See Troubleshooting CoreDNS for remediation details.
3.3.8. Installing K3s
During this phase, install-lockss will install K3s 1.21.5+k3s1, if applicable. This phase begins with the heading:
Installing K3s...
This phase consists of these steps:
First, install-lockss will determine if K3s 1.21.5+k3s1 needs to be installed:
If K3s is not present, install-lockss will display
K3s is not present
, and will install K3s 1.21.5+k3s1 in the next step.If an older version of K3s is present, install-lockss will display
Detected K3s version <installed_version> is older than expected version <expected_version>
, and you will receive the following prompt:Upgrade K3s from <installed_version> to <expected_version>?
Enter Y and install-lockss will install K3s 1.21.5+k3s1 in the next step, or enter N and install-lockss will not install K3s 1.21.5+k3s1 in the next step, or hit Enter to accept the default in square brackets 2.
If the expected version of K3s is already present, install-lockss will display
K3s version <installed_version> is already installed; skipping
, and will not install K3s 1.21.5+k3s1 in the next step.If a more recent version of K3s is present, install-lockss will display
Detected K3s version <installed_version> is more recent than expected version <expected_version>
, and will not install K3s 1.21.5+k3s1 in the next step.If K3s is detected but the installed and expected version numbers cannot be compared automatically, install-lockss will display
[Warning] Detected K3s version <installed_version>, expected version <expected_version>, comparison failure, skipping
, and install-lockss will not install K3s in the next step.
If install-lockss determined in the previous step that it will not install K3s 1.21.5+k3s1, you will see the confirmation
Not installing K3s
, and nothing will happen in this step.But if install-lockss determined in the previous step that it will install K3s 1.21.5+k3s1, you will see the confirmation
Installing K3s version <expected_version>
, and this step will proceed as follows:First, install-lockss will ask you to specify the K3s state data directory (the directory K3s uses to store state data), with this prompt:
K3s state data directory
By default, this is
/var/lib/rancher/k3s
. However, if/var
is space-limited, you should specify a different directory, that has ample space and is not backed by NFS or by XFS with legacyftype=0
.Enter a suitable directory path for the K3s state data directory, or hit Enter to accept the default in square brackets 3 5.
Then install-lockss will attempt to determine the filesystem type of the specified K3s state data directory. In many situations, it will simply display the filesystem type in a message similar to this (for example,
<fs_type>
might beext4
):Filesystem type of <k3s_dir> (<k3s_mountpoint>) is <fs_type>; proceeding
Error
Below are some error conditions you may encounter here and what to do about them.
Filesystem type of K3s state data directory is NFS
If the filesystem type backing the K3s state data directory is NFS, you will see the error message:
[ERROR] Filesystem type of <k3s_dir> (<k3s_mountpoint>) is NFS; see manual
and install-lockss will fail. It is not possible to run K3s with a state data directory backed by NFS 6. Re-run install-lockss and designate a different K3s state data directory that is not backed by NFS.
Filesystem type of K3s state data directory is XFS with legacy
ftype=0
If the filesystem type backing the K3s state data directory is XFS with legacy
ftype=0
, you will see the error message:[ERROR] Filesystem type of <k3s_dir> (<k3s_mountpoint>) is XFS with legacy ftype=0; see manual for workaround
and install-lockss will fail. Contemporary XFS filesystems with modern
ftype=1
work well with K3s, but older XFS filesystems with legacyftype=0
are not compatible. Ideally, re-run install-lockss and designate a different K3s state data directory that is not backed by XFS with legacyftype=0
. Alternatively, you can read about a workaround in Troubleshooting OverlayFS with XFS.Warning
Below are some warning messages you may see here and how to respond to them.
Filesystem type of K3s state data directory unknown
If the filesystem type backing the K3s state data directory cannot be inferred automatically, you will see the warning:
[Warning] Filesystem type of <k3s_dir> unknown (findmnt not present); proceeding
and install-lockss will keep going. But K3s may malfunction if the actual filesystem type backing the selected K3s state data directory is one that does not work with K3s, such as NFS, or XFS with legacy
ftype=0
; see the error conditions above.Filesystem type of K3s state data directory is XFS but
ftype
unknownIf the
ftype
of the XFS filesystem backing the K3s state data directory cannot be inferred automatically, you will see the warning:[Warning] Filesystem type of k3s_dir (k3s_mountpoint) is XFS but ftype unknown (xfs_info not present); proceeding
and install-lockss will keep going. But K3s may malfunction if the actual filesystem type backing the selected K3s state data directory is XFS with legacy
ftype=0
; see the corresponding error condition above.Then install-lockss will download the K3s Installer from https://get.k3s.io/ and invoke it with suitable options. This may take several minutes, during which the output to the console will be from the K3s Installer, not from install-lockss.
Depending on your operating system and other factors, the K3s Installer may install additional software packages or configure system components, using sudo if necessary (which may prompt for the user's sudo password).
Error
If the K3s Installer does not succeed, it will display its own error messages, then install-lockss will fail. See Troubleshooting the K3s Installer for remediation details.
Sample error messages from the K3s Installer
Error messages that the K3s Installer may display include:
[ERROR] Failed to apply container_runtime_exec_t to /usr/local/bin/k3s, please install: yum install -y container-selinux selinux-policy-base yum install -y https://rpm.rancher.io/k3s/stable/common/centos/8/noarch/k3s-selinux-0.3-0.el8.noarch.rpm
Error: Package: k3s-selinux-0.3-0.el7.noarch (rancher-k3s-common-stable) Requires: container-selinux >= 2.107-3 You could try using --skip-broken to work around the problem You could try running: rpm -Va --nofiles --nodigest
Finally, whether or not K3s was installed, install-lockss will store Kubernetes configuration data as the
lockss
user in the fileconfig/k8s.cfg
(relative to the LOCKSS Installer Directory).Error
Below are some error conditions you may encounter here and what to do about them.
Could not write or append to
k8s.cfg
If the creation of the file fails, you will see one of these error messages:
[ERROR] Could not write k8s.cfg [ERROR] Could not append to k8s.cfg
and install-lockss will fail. Check for file permission mismatches between the user running install-lockss and the
lockss-installer/config
directory, then try again.
3.3.9. Testing the K3s Node
During this phase, install-lockss runs a series of tests to verify that the K3s node is operational. This phase begins with the heading:
Testing the K3s node...
No user interaction is expected. If all tests pass, you will see the message:
[success] Tested the K3s node
and install-lockss will successfully proceed to the next phase, Completion of the LOCKSS Installation Process (Section 3.3.10).
Otherwise, you will see an error message corresponding to the test that did not pass, and install-lockss will fail.
Error
Below are some error conditions you may encounter here and what to do about them.
Problems with config/k8s.cfg
At the end of Section 3.3.8 (Installing K3s), some Kubernetes-related data is stored in config/k8s.cfg
(relative to the LOCKSS Installer Directory). If the file cannot be found or read, or if it contains invalid or unexpected data, you may see one of these error messages:
[ERROR] k8s.cfg not found
[ERROR] Error reading K8S_FLAVOR
[ERROR] K8S_FLAVOR is not set
[ERROR] K8S_FLAVOR is not k3s
[ERROR] Error reading KUBECTL_CMD
[ERROR] KUBECTL_CMD is not set
[ERROR] k3s command of KUBECTL_CMD is not on the PATH
Check the contents of config/k8s.cfg
and contact us () for troubleshooting if necessary.
Problems with the K3s node
If the K3s node is not behaving as expected, you may see one of these errors:
[ERROR] Command failed (kubectl version)
[ERROR] Timeout waiting for the K3s node to be ready
[ERROR] Command failed (kubectl get node)
[ERROR] Unexpected number of K3s nodes
If the K3s node is newly installed, it may simply be that there has not yet been enough time for it to come up; you can re-run this phase with scripts/install-lockss --test-k3s
(or scripts/install-lockss -K
) to retry. Contact us () for troubleshooting if necessary.
Problems with DNS
If the K3s node's DNS environment is not working properly, you may see one of these errors:
[ERROR] Timeout waiting for the CoreDNS pod to be running and ready
[ERROR] Command failed (kubectl get pod)
[ERROR] Unexpected number of CoreDNS pods
[ERROR] Timeout waiting for the DNS service to be present
[ERROR] Command failed (kubectl get service)
[ERROR] Unexpected number of kube-dns services
[ERROR] Unexpected kube-dns service type
[ERROR] Timeout waiting for DNS resolution
[ERROR] Unexpected Cluster-IP
If the K3s node is newly installed, it may simply be that there has not yet been enough time for CoreDNS to come up; you can re-run this phase with scripts/install-lockss --test-k3s
(or scripts/install-lockss -T
) to retry. You can also use the install-lockss options --retries=N
(to increase the number of retries in DNS lookup tests to N
from 5) or --wait=S
(to increase the delay between retries in DNS lookup tests to S
seconds from 10). Contact us () for troubleshooting if necessary.
3.3.10. Completion of the LOCKSS Installation Process
If all phases completed successfully, you will see the message:
[success] Successful completion of the LOCKSS installation process
and install-lockss will terminate.
3.3.11. Checking the K3s Configuration
Tip
This section is optional.
K3s comes with k3s check-config, a configuration checker tool. The K3s configuration checker is capable of detecting complex underlying system situations that definitely require fixing (or applications running in the K3s cluster will not be able to function properly). On the other hand, the versions of the K3s configuration checker available at the time LOCKSS 2.0-alpha7 was released contained bugs that reported spurious issues that are either inaccurate or moot. As a result, we have decided against running k3s check-config as part of install-lockss at this time, to avoid unnecessary interruptions in the installation of the LOCKSS system in many cases where there is no particular cause for concern.
That being said, we still recommend running k3s check-config and interpreting the results using the Troubleshooting the K3s Configuration Checker section of the manual:
Run this command:
k3s check-config
The following error messages in the output are indicative of system situations that require attention:
/usr/sbin iptables v1.8.2 (nf_tables): should be older than v1.8.0, newer than v1.8.3, or in legacy mode (fail)
RHEL7/CentOS7: User namespaces disabled; add 'user_namespace.enable=1' to boot command line (fail)
apparmor: enabled, but apparmor_parser missing (fail)
Troubleshooting
See Troubleshooting the K3s Configuration Checker for details.
The following error messages in the output can be ignored:
cgroup hierarchy: nonexistent?? (fail)
links: aux/ip6tables should link to iptables-detect.sh (fail) links: aux/ip6tables-restore should link to iptables-detect.sh (fail) links: aux/ip6tables-save should link to iptables-detect.sh (fail) links: aux/iptables should link to iptables-detect.sh (fail) links: aux/iptables-restore should link to iptables-detect.sh (fail) links: aux/iptables-save should link to iptables-detect.sh (fail)
swap: should be disabled
CONFIG_INET_XFRM_MODE_TRANSPORT: missing
Troubleshooting
See Troubleshooting the K3s Configuration Checker for details.
For other error messages, check the official K3s documentation, search for K3s issues database on GitHub or the Web for resources matching your error message or operating system, and/or contact us so we can help investigate and document for future reference.
Footnotes
- 1
- 2
If install-lockss was invoked with the
--assume-yes
option, Y is automatically entered for you.- 3
If install-lockss was invoked with the
--assume-yes
option, the suggested value is automatically accepted for you.- 4
Or if your install-lockss was invoked with the
--force-dns-prompt
option.- 5
If install-lockss was invoked with the
--k3s-data-dir=DIR
option,DIR
will automatically be used without the prompt.- 6
See https://github.com/containerd/containerd/discussions/6140.
- 7
See https://docs.docker.com/storage/storagedriver/overlayfs-driver/#prerequisites.