Linux SD Troubleshooting and Solutions

Prev Next

Linux SD Troubleshooting and Solutions

Technical Reference — SDLinux Deployment & Recovery

Table of Issues

SDLinux Frequently Asked Questions[ Error ] Unable to find a matching driver to update the system

Environment

Applies to all SDLinux versions 8.2SR1 and above and all Linux Flavors.

Symptom

During deployment, the installation fails with an error: [ Error ] Unable to find a matching driver to update the system.

Cause

Case 1

Kernel driver is not supported and does not exist on WinMagic's AWS S3 account.

Solution 1

STEP 1  Check kernel support

  1. Check the SDLinux Kernel List page to confirm your kernel is supported.

  2. If the driver exists, proceed to other solutions.

  3. If the driver does not exist, submit a request to the support team.

Case 2

Date on local machine is not set, incorrect, or not synced with the current time.

Solution 2

STEP 2  Verify and set system date

  1. Verify date on the machine with command:

[root@localhost]# date

  1. Once confirmed that date is incorrect, set the current date with command:

[root@localhost]# date --set="datestring"

where datestring is the current date.

Case 3

Local machine is unable to reach AWS S3 or is having issues downloading the driver directly.

Solution 3-1: Manually download the specific kernel driver

STEP 3  Manually download the specific kernel driver

  1. Download the drivers using the link below:

https://s3.amazonaws.com/CloudVM/Linux/drivers/v2/SDLinuxDrv-<kernel_version>.ko

  1. Rename the downloaded file to SDLinuxDrv.ko

rename file from SDLinuxDrv-3.10.0-862.ko to SDLinuxDrv.ko

  1. On the SDLinux package navigate to:

- For 8.2 SR1 (and earlier): wmsd > drivers

- For 8.3+: winmagic > drivers

  1. Create a new folder and rename it to the kernel version number (e.g. 3.10.0-862).

  2. Copy the SDLinuxDrv.ko file to the new folder. You should now have:

- For 8.2 SR1 (and earlier): wmsd\drivers\3.10.0-862\SDLinuxDrv.ko

- For 8.3+: winmagic\drivers\3.10.0-862\SDLinuxDrv.ko

Solution 3-2 (Ubuntu only): Manually download all kernel drivers

STEP 4  Download all Ubuntu kernel drivers

  1. Download the drivers using the following link:

https://s3.amazonaws.com/CloudVM/Linux/drivers/v2/ubuntu_drivers.tar.gz

  1. Copy both the SDLinux package and the downloaded ubuntu_drivers.tar.gz file to the Linux system.

  2. Untar the wmsd-sdlinux.tar.gz file to extract the winmagic folder.

  3. Untar the ubuntu_drivers.tar.gz file to extract the drivers folder.

  4. Run the following command to copy files:

sudo cp -avr drivers/* package_name/winmagic/drivers/

  1. You should now be able to deploy SDLinux without connecting to AWS S3.

Unable to download libpwquality, libpwquality-tools and cryptsetup

Environment

Applies to RHEL OS's only.

Symptom

During SDLinux installation, at the prerequisite installation stage, the user sees an error that it is unable to download one or all of the following libraries/packages: libpwquality, libpwquality-tools, and cryptsetup.

Cause

(Redhat only) No subscription or alternate repository configured.

Solution

If you have purchased a subscription to Redhat, follow the instructions to register and subscribe your system using subscription-manager:

https://access.redhat.com/solutions/253273

Error 23: Error while parsing number

Environment

Applies to RHEL6.x OS's.

Symptom

After running the rhel6_chgconfig script to set the display resolution, when the system is rebooted, Error 23: Error while parsing number is displayed on screen.

Cause

An incorrect vga mode was entered when running the rhel6_chgconfig script (e.g. ./rhel6_chgconfig vga=0xg).

Solution

  • Upon seeing the error, follow the prompt and press any key to continue.

  • On the GNU grub screen (with Red Hat Enterprise Linux 6 selected), press 'e' to edit.

  • Select the appropriate Kernel and press 'e'.

  • Remove the invalid vga entry.

  • Press Enter to save the change.

  • Press 'b' to boot.

Error: Failed to find SD partition

Environment

Applies to all SDLinux versions and Linux OS's.

Symptom

Installation fails due to SDLinux being unable to create/find the SecureDoc partition.

2018-07-05 04:37:16 SecureDoc partition check starting ...

SecureDoc requires separate partitions for boot plus extra encryption management space

a total of about 6 MB of space will be required

Creating SecureDoc partition...

Error while creating partition

ERROR: failed to find sd part

ERROR: setting up partitions , exiting...

Cause

When re-imaging a physical Linux machine from a bootable USB device, after the re-image is completed and the USB is unplugged, the machine is not rebooted prior to starting the SDLinux deployment. A reboot is required to ensure that no files on the system are still pointing to the USB device.

Solution

  • After re-imaging the system, remove the USB drive (containing the image) and manually reboot the system.

  • After the system starts up, you can run the installation once more.

Permission denied error when running install script or ./sdot commands

Environment

Applies to all SDLinux versions and Linux OS's.

Symptom

User gets a "Permission denied error" when running SDLinux scripts or ./sdot commands.

Cause

Case 1

User runs the commands as "sudo" however this may have only been configured to provide super user access and is not the same as actual root permissions.

Solution 1

Instead of running the command as sudo, switch to root first and then run the command:

vmadmin@localhost:~$ sudo su

[sudo] password for vmadmin:

root@localhost:/home/vmadmin# ./install -s

Case 2

The SDLinux package scripts/files permissions have changed due to copy operations, normally when performed between Windows and Linux environments. This can especially happen when copying files to a FAT32 USB stick since FAT32 does not retain these permissions.

Solution 2

  • Do not extract the wmsd-sdlinux.tar.gz (or rhel6_wmsd.tar.gz for rhel6.x) installer on Windows.

  • Use SFTP software to transfer the files from Windows to Linux instead of using a USB (e.g. Filezilla, WinSCP).

Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

Environment

Applies to SDLinux version 8.3 and below.

Symptom

During deployment, after installation completes and boot logon is installed, the system is rebooted and is unable to boot normally. A Kernel Panic issue is displayed.

Cause

  • During deployment, the SDLinux installer failed to create a loop device to mount SDspace. This was because the loop modules (needed by the losetup command) were not enabled by default on the server and this is not handled appropriately by the installer.

  • The root in grub was identified by name (/dev/mapper/sd1) instead of by UUID. This is handled appropriately on Ubuntu OS's but not for RHEL OS's.

Solution

Note: This issue has been fixed in version 8.3SR1. See Jira SD-29462 / SD-29463.

./sdot: error while loading shared libraries

Environment

Applies to SDLinux version 8.3 and below.

Symptom

  • The Linux system halts at start-up with a FATAL ERROR reported on the console.

  • The following errors are seen:

dracut-pre-mount[3068] ./sdot: error while loading

shared libraries: libcryptsetup.so.4: cannot open shared object file: No such file or directory

wm_dracut_preboot: FATAL ERROR: wm-- (status 127)

dracut: FATAL: wm-- (status 127)

dracut: Refusing to continue

Cause

Security patches caused a change to the current system's library which is incompatible with the preboot that is installed.

Solution

Note: Issue has been resolved in 8.3SR1+.

Workaround (for 8.3 and lower versions):

STEP 1  Boot to shell and create symbolic link

  1. Boot the VM, stopping at the grub screen, and edit the kernel boot line (press 'e').

  2. Add rd.debug=1 rd.shell rd.break=pre-mount to the end of the kernel line, also remove the keyword 'quiet' if present, then press Ctrl-X to continue.

  3. The VM will boot and then stop, giving you shell access. cd to /usr/lib/sdot folder.

  4. Find the name of the new libcryptsetup in the system (most likely under /usr/lib, e.g. libcryptsetup.12.so.0.1.5) and create a symbolic link:

ln -s /usr/lib/libcryptsetup.12.so.0.1 libcryptsetup.so.4

  1. Run ./sdot version while in the same folder. If you get a valid version output, type exit and let it boot normally. If you get an error of a missing library, verify the symbolic link is correct.

An Encrypted Linux virtual machine is stuck at the SecureDoc pre-boot screen after reverting from a snapshot

Environment

Applies to VMware vSphere virtual machines.

Symptom

The Linux virtual machine (VMware vSphere) is unable to connect to SES and auto-boot at pre-boot after being reverted to a snapshot taken at an earlier state.

Cause

After reverting the virtual machine from a snapshot, the network adapters were not set by default to be "Connected" and to "Connect at power on".

Solution

  • On your vSphere client, select the virtual machine and edit its settings.

  • On the Edit Settings > Virtual Hardware window, expand Network Adapter and ensure both "Connected" and "Connect At Power On" checkboxes are selected.

Unable to perform OS updates on Linux VM on Azure after SDLinux is deployed

Environment

Applies to Azure Linux VMs (Redhat, Ubuntu).

Symptom

Linux VM is unable to perform OS updates (e.g. yum update, apt update) after SDLinux is installed. The user will see errors regarding the system being unable to connect to the repository.

Cause

After installing SDLinux on the Linux client, an automatic "guest" reboot is performed to complete the installation. While the system resumes normally and is accessible via the original IP address, in some cases the DNS config on the system no longer works, causing OS updates to fail when trying to connect to repos via the mirrors.

Solution

  • Stop (deallocate) the VM via the Azure portal, then start it back again.

  • Log in to the system and verify that you can connect to the repo mirrors and update your repo list (e.g. yum repolist, apt update).

Failed to create loopback device / Could not find any loop device during deployment

Environment

Applies to RedHat 6.x Linux systems.

Symptoms

Deploying on a Linux system with no free space but with an existing boot partition fails to create sdspace as a file due to failing to create a loopback device.

Cause

This is a bug with RedHat 6.x systems and how loop device nodes are created in this kernel version. It is also possible that util-linux is missing on the system (this package includes important libraries like mount, libmount, libblkid and libuuid).

Solution

STEP 1  Install util-linux and verify loop device

  1. Install util-linux on the system:

$ yum install util-linux

  1. Reboot the client machine.

  2. On start-up, run the command below to verify that loop device creation works:

$ losetup -f

/dev/loop0

Duplicate LVM drives are displayed in the Compliance section in SESWeb after upgrading SDLinux client from 8.3 to 8.3SR1

Environment

  • Applies to all Ubuntu and RedHat/CentOS operating systems configured with LVM partitions.

  • Affects Linux systems that have been installed with the 8.3 release version of SDLinux.

Symptom

After upgrading the SDLinux client from 8.3 to 8.3SR1, duplicate drive info is displayed for the device under the Compliance section of SESWeb.

Cause

SDLinux 8.3 used a different diskId format which was fixed in 8.3SR1 to be consistent with Windows diskId format. After upgrading, SES identifies these as different drives and adds new records WITHOUT removing the old compliance records, resulting in duplicated drive compliance info.

Solution

Warning: Because the queries below delete multiple records from your database, it is highly recommended to back up your database prior to running the queries.

The queries only remove drive info from the Compliance section of SESWeb and do not affect drive info used for recovery.

Case 1 — Upgrading ALL Linux clients from 8.3 to 8.3SR1

Running the query below for tblAsset_DeviceDisk will clean out all compliance device disk data for all Linux devices on the database. Use this method when upgrading all Linux devices, or when automation is handling upgrades.

DELETE FROM tblAsset_DeviceDisk WHERE PC_Index IN (SELECT PC_Index FROM Computers WHERE Node_Type_ID=24)

Case 2 — Upgrading only select Linux Clients from 8.3 to 8.3SR1

  1. Run the command below to view all current Linux devices:

SELECT * FROM Computers WHERE Node_Type_ID=24

  1. From the result, obtain the PC_Index from the specific device you want to upgrade.

  2. Run the following command replacing the PC_Index obtained from step 2:

DELETE FROM tblAsset_DeviceDisk WHERE PC_Index IN (replace_with_PC_Index)

Error: Cannot retrieve repository metadata (repomd.xml) for repository: epel during SDLinux deployment

Environment

  • Applies to RedHat Enterprise Linux/CentOS versions 6.6 and 6.7.

  • Applies to new installations of SDLinux.

Symptom

SDLinux deployment fails with: "Cannot retrieve repository metadata (repomd.xml) for repository: epel. Please verify its path and try again."

Cause

Legacy RHEL systems (version 6.7 and 6.6 or older) use TLS 1.0 or 1.1, which are now deprecated for SSL. TLS v1.2 is required in order to communicate with the Fedora site.

Solution

The following packages plus their associated dependencies will need to be updated to force SSL on your system to use TLS v1.2:

  • yum

  • curl

  • openssl

  • nss

For RedHat

  • Run subscription-manager status to verify that your device is registered and has access to RedHat repositories.

  • Run the yum repolist command to list repos available for update.

For CentOS

  • Run the yum repolist command to list packages in the repo that are available for update.

Note: If you are getting the error while trying to run the yum repolist command, you can run: yum --disablerepo="epel" repolist to bypass this issue.

Run the following commands to update the packages:

$ yum clean all

$ yum update yum

$ yum update curl

$ yum update openssl

$ yum update nss

Or alternatively with epel disabled:

$ yum clean all

$ yum --disablerepo="epel" update yum

$ yum --disablerepo="epel" update curl

$ yum --disablerepo="epel" update openssl

$ yum --disablerepo="epel" update nss

You should now be able to deploy SDLinux once more.

Primary Account Set-up window crashes on log-in

Environment

Ubuntu Desktop 18.04 LTS.

Symptom

The SecureDoc Primary Account Set-up window crashes on log-in to the Ubuntu system.

Cause

Under investigation.

Work Around

Trigger the SecureDoc Primary Account Set-up window again to take ownership of the system.

  1. Run the script below with the current account that will take ownership of the system (no need to run this as root):

$ /etc/profile.d/wm_housekeeper.sh

  1. Once the SecureDoc Primary Account Set-up window is displayed again, complete the set-up to take ownership of the system.

insmod Error: could not insert module ./SDLinuxDrv.ko: Required key not available

Environment

Applies to all Linux OS.

Symptom

SDLinux installation fails with error: "insmod: Error: could not insert module ./SDLinuxDrv.ko Required key not available".

Cause

SecureBoot is enabled on the BIOS of the client machine.

Solution

Disable SecureBoot on the client machine's BIOS and then try to deploy SDLinux once more. The user can re-enable SecureBoot once encryption has completed on the system.

Installation failed, lack of network support when booting up

Environment

Applies to Red Hat and CentOS.

Symptom

SDLinux Installation fails with error: "Installation failed, lack of network support when booting up".

Cause

The system's network interface file ONBOOT setting is disabled.

Solution

Enable the ONBOOT setting on the system's network interface file.

STEP 1  Enable ONBOOT on network interface

  1. Navigate to the systems network interface file folder:

$ cd /etc/sysconfig/network-scripts

  1. Open the network interface file with your preferred editor (note: interface file name is different for each system):

$ nano ifcfg-eth0

  1. Modify the ONBOOT setting to yes and save your changes.

  2. You can now try to re-install SDLinux.

Trackpad and Trackpoint issues on HP ZBook Devices

Environment

Applies to HP ZBook HW running RHEL 7.x.

Symptom

Trackpad and Trackpoint are unresponsive at preboot.

Cause

Trackpoint and trackpad drivers are not loaded correctly.

Solution

Before deploying SDLinux on the client machine, ensure driver loading is enabled.

STEP 1  Enable driver loading for HP ZBook

  1. Untar the install package:

tar xvf wmsd-sdlinux.tar.gz

  1. Navigate to the 'winmagic' folder.

  2. Rename the file 'hpz-g6.load' to 'drivers.load':

mv hpz-g6.load drivers.load

  1. Run the install.

Unable to boot to OS after SDLinux deployment

Environment

Applies to all Linux OS.

Symptom

After SDLinux is deployed, the system automatically reboots but is unable to boot back to the OS.

Cause

General issues where the system is unable to boot back to the OS.

Solution

STEP 1  Add nosecdoc parameter at grub

  1. Force reboot the machine.

  2. On the grub menu, press 'e'.

  3. Add the nosecdoc parameter to the linux command line.

  4. Press Ctrl-X to start.

  5. If the system boots to dracut shell, enter the following command:

systemctl --default

Running install script again when SDLinux was not properly uninstalled

Environment

Applies to all Linux OS.

Symptoms

Installation fails with an error after SDLinux was recently uninstalled.

Cause

Uninstall may have only been run once on the system and then the install script was run again before the second uninstall command was run. The uninstall command needs to be run twice: (1) to perform decryption, then (2) after decryption completes, to remove the installation path, roll back fstab, and remove sdspace. Running the install command before the second uninstall deletes the existing install path (including the fstab backup) but does not remove the existing sdspace.

Solution

STEP 1  Clean up and re-register

  1. Go to the SES server and manually remove the registration of the device from SES.

  2. Remove the sdspace partition manually by running:

parted /dev/sdX rm N

Where /dev/sdX is the disk where sdspace was created and N is the partition number (sdspace is approximately 6MB in size).

System fails to boot after uninstall is performed

Environment

Applies to all Linux OS.

Symptoms

After decryption is completed and the second uninstall command is run, after the machine is manually rebooted, the client machine is unable to boot back to the system.

Cause

Possibly caused by a timing issue on certain environments during uninstall.

Workaround 1

STEP 1  Uninstall with sync before reboot

  1. Run uninstall with the following parameters:

/usr/local/WinMagic/secdoc.py uninstall -s -n

  1. After the uninstall completes, run the following command:

sync;sync

  1. Then manually reboot:

reboot

Workaround 2

STEP 2  Uninstall with delayed reboot

  1. Run uninstall with the following parameters:

/usr/local/WinMagic/secdoc.py uninstall -s -n

  1. After the uninstall completes, run the command to reboot machine with a 30 second delay:

sleep 30s; reboot

A Partition fails to decrypt on Ubuntu during uninstall

Environment

Applies to Ubuntu 22.04.3 — Affects SDLinux version 9.0SR4 and 9.0SR4 HF1.

Symptoms

After running the initial uninstall and after decryption completes, upon login to the system one of the partitions appears to be still encrypted.

Cause

Initramfs appears to have not been modified and preboot does not know it needs to start decrypting.

Workaround

STEP 1  Uninstall without reboot, then fsync

  1. Run uninstall command without automatic reboot:

./secdoc.py uninstall -s -n

  1. After running the command above, run the fsync command:

fsync

  1. Reboot the machine and observe that all partitions are decrypted.

System is unable to boot to OS after install and is in a constant boot loop (Reboot due to panic= boot argument)

Environment

Appears to affect some Ubuntu 20.04 virtual machines using FIPS kernel.

Symptoms

After the system reboots from the initial SDLinux installation, the system is unable to boot back and after some time reboots again, constantly repeating. On the console output the following error is displayed related to mounting the boot device.

Cause

At boot, the system is trying to mount the boot device but fails, or more accurately does not know what the boot device is.

Workaround

Adding bootdev=/dev/sdX (/dev/sdX being the location of your boot partition) to the kernel command line fixes the issue.

Adding the parameter at the grub menu

STEP 1  Add bootdev parameter to grub

  1. On start-up, Press 'e' on the grub menu.

  2. Add the bootdev=/dev/sdX parameter to the linux command line.

  3. Once booted back to the system, open terminal and add the bootdev parameter to the grub file permanently:

/etc/default/grub

  1. Save the file and run command:

update-grub

  1. Reboot system and observe on start-up.

If you do not know the sdspace location for bootdev

STEP 2  Bypass via nosecdoc and use lsblk

  1. On start-up, Press 'e' on the grub menu.

  2. Add the 'nosecdoc' parameter to the linux command line kernel.

  3. Press Ctrl-X to continue. This should allow you to bypass the issue and boot back to the OS.

  4. Run lsblk command and take note of the boot partition.

WinMagic Technical Solutions  ·  SDLinux Troubleshooting Reference