Frequently Asked Questions

How do I know if watchmaker has installed?

To determine whether watchmaker is installed, the simplest method is to run the command watchmaker --help. If it displays the cli help page, watchmaker is installed. Another option is to check pip list | grep watchmaker.

What do I do if watchmaker failed to install?

First, review the installation document. Then double-check the output of a failed installation. Usually, the output points pretty clearly at the source of the problem. Watchmaker can be re-installed over itself with no problem, so once the root cause is resolved, simply re-install watchmaker.

Why does watchmaker immediately fail to run on my spel-based system?

System-images created using spel enable SELinux role-transitions for all local/baked-in user accounts. This includes the provisioning-users created at launchtime by the cloud-init utility. These users are created with the default SELinux context of staff_u:staff_r:staff_t:s0-s0:c0.c1023. Their default role-transition can result in the user not having enough privileges to run watchmaker. To work around any watchmaker-killing “permission denied” errors this may cause, escalate privileges using sudo -i -r unconfined_r -t unconfined_t.

Note: this should only impact processes that are not initiated via systemd (i.e., should not impact provisioning-processes kicked off via userData, CloudFormation or the like).

Why does the watchmaker install fail if my system is FIPS enabled?

This is primarily a question for Red Hat (and derived distributions). As of this writing, the pip utility in all Red Hat releases up through 7.4.1708, default to looking for pypi packages signed with MD5 signatures. If you’ve enabled FIPS (or are using a build that has FIPS pre-enabled), MD5 is disabled in the kernel (due to being a weak hashing-method). You can either disable FIPS (not recommended) or explicitly force pip to use a different signature-index. The latter is detailed in the Linux section of the usage document.

How do I know if watchmaker has completed without errors?

By default, watchmaker will reboot the system after a sucessful execution. Therefore, if the system reboots, watchmaker executed successfully. If you are investigating sometime after watchmaker completed, check the logs for errors. If anything fails, watchmaker will suppress the reboot. (Though note that the --no-reboot flag can be used to suppress the reboot even after a successful execution.)

You can also test the watchmaker exit code programmatically. If watchmaker fails, it will return a non-zero exit code. If watchmaker completes successfully, it will return an exit code of zero. You would typically pass the --no-reboot flag if you intend to test the exit code and determine what to do from there.

What do I do if watchmaker failed to complete or completes with errors?

Start by checking the logs generated by watchmaker. The logs are stored in the directory specified by the --log-dir argument. Search the log for entries that have [ERROR], this will give you a starting point to begin troubleshooting. Also, if a salt state failed, look for the pattern Result: False. If it is not an obvious or simple issue, feel free to create an issue on the watchmaker github page. If there is a salt_call.debug.log in the watchmaker log directory, you can look for [ERROR] messages in there as well. However, this log file can be very noisy and a message with the error label may not be related to the error you are encountering.

Does watchmaker support Enterprise Linux 7?

Watchmaker is supported on RedHat 7 and CentOS 7. See the index page for a list of all supported operating systems.

Does watchmaker support Enterprise Linux 8?

Watchmaker is supported on RedHat 8, CentOS 8 Stream, and Oracle Linux 8. See the index page for a list of all supported operating systems.

Does watchmaker support Enterprise Linux 9?

As of today’s date (2024-04-10), watchmaker’s hardening-modules do not yet support Enterprise Linux 9 or related distros. This is currently a to-be-started project-task. Action on support for Enterprise Linux 9-based distros can be tracked through ash-linux-formula issue #496.

How can I exclude salt states when executing watchmaker?

The Salt worker in Watchmaker supports an exclude_states argument. When present, the value is passed directly to the exclude option of the salt highstate execution module. To use this option with watchmaker from the command line, pass the argument --exclude-states <sls_glob>. For example:

# Exclude the state "foo" with an exact match
watchmaker --exclude-states foo

# Exclude all state names that begin with "foo"
watchmaker --exclude-states foo*

# Exclude multiple states "foo" and "bar" with an exact match
watchmaker --exclude-states foo,bar

Can I use the underlying salt functionality directly?

Yes, by passing watchmaker’s salt configuration directory to the salt command, using the -c|--config-dir argument:

• Linux: /opt/watchmaker/salt

• Windows: C:\Watchmaker\salt\conf

For example:

# -c|--config-dir
salt-call -c /opt/watchmaker/salt state.show_top

Can I use watchmaker to toggle my RedHat/Centos host’s FIPS mode?

For Enterprise Linux, “yes, indirectly”. Because watchmaker implements most of its functionality via SaltStack modules, you can directly-use the underlying SaltStack functionality to effect the desired change. This is done from the commandline - as root - by executing:

• Disable FIPS-mode: salt-call -c /opt/watchmaker/salt ash.fips_disable

• Enable FIPS-mode: salt-call -c /opt/watchmaker/salt ash.fips_enable

And then rebooting the system.

How do I get Watchmaker release/project notifications?

Users may use an RSS reader of their choice to subscribe to the Watchmaker Release feed to get notifications on Watchmaker releases. The Watchmaker RSS release feed is https://github.com/plus3it/watchmaker/releases.atom.

Users can also “watch” the GitHub project to receive notifications on all project activity, https://github.com/plus3it/watchmaker/subscription.

Why do I get warnings in my system logs about the rsyslog service not being able to connect to the logcollector host

Watchmaker leverages the oscap utility and associated OSCAP content for a significant percentage of its hardening-actions. Recent updates to this content have included (inappropriately) adding:

# Per CCE-80863-4: Set *.* @@logcollector in /etc/rsyslog.conf
*.* @@logcollector

To the /etc/rsyslog.conf file. The *.* @@logcollector line results in the rsyslog service attempting to do remote-logging to the external host logcollector. If no such hostname exists in the hardened-system’s DNS domain, “host not found” error-types will be logged. If using a log-offloader tool (such as the Splunk forwarder-agent), it is safe to either comment out the *.* @@logcollector. If data should be sent directly from rsyslog to a remote syslog-host, change the string logcollector to the FQDN of your site’s actual log-collector host.

Why isn’t the domain-join functionality doing DDNS updates

Starting in early 2023, the default method for joining Linux hosts to Active Directory domains was changed to leveraging the native, sssd-based integrations. While sssd has the capability of doing DDNS updates (and doing regular refreshes to prevent record-loss due to record-scavenging), it has a couple of requirements in order to be able to do so. First and foremost is that the hosts providing DNS service to the Linux system have DDNS enabled for the A and PTR zones the Linux system is a member of. Similarly, sssd defaults to attempting to use GSS with TSIG for its DDNS update-requests: even if the Linux system’s A and PTR zones are DDNS-enabled, if those zones don’t allow DDNS clients to negotiate updates with GSS and TSIG, the updates will fail. Watchmaker’s join-domain-formula (for sssd) does allow overriding the attempt to negotiate updates with GSS and TSIG. This is done by appending dyndns_auth: none (and, optionally, adding dyndns_auth_ptr: none) to the Salt pillar’s sssd_conf_parameters parameter-value.

Why are my security-scans failing due to lack of nosuid/noguid mount-options for /boot

Depending on how your OS was provisioned, the /boot directory-tree may or may not be on its own partition. If it’s not on its own partition, it will be part of the / partition. As a result, it will not be possible to set partition-specific mount-options.

Note: it is known that spel AMIs created for EL8 prior to April of 2024 will typically not have /boot on its own partition. Suport for EFI-boot was only added to the EL8 automation in February of 2024 and only implemented (in some regions) in April of 2024. spel AMIs not built to support EFI-boot typically will not have /boot on its own partition.

How can I see what hardening-actions are implemented through oscap

Watchmaker’s use of oscap (for Linux hosts) is not transparently-implemented. While the oscap activities are logged to the /var/log/oscap.log file, that file’s logged-activity is created at an informationl level rather than a detailed, “debug”-style level. In order to see exactly what the oscap utility does, one can make oscap generate a script-file containing all of its hardening-actions. This can be done by doing (as the root user):

oscap xccdf generate fix \
  --fix-type bash \
  --output ~/fixes.sh \
  --profile xccdf_org.ssgproject.content_profile_stig
  /root/scap/content/openscap/ssg-<PLATFORM>-ds.xml

Where “PLATFORM” will be something like rhel7 on a Red Hat l system or ol8 on an Oracle Linux 8 system. Once the ~/fixes.sh script is generated, one can grep it for specific actions or open it up for viewing and peruse it for (presumably problematic) hardening actions (such as the logcollector setting noted in a prior section of this FAQ).