Generate an sos report on OCP Master Nodes

#devops #redhat #openshift #sosreport

Generate an sos report on OCP Master Nodes

In Red Hat OpenShift Container Platform 4, Red Hat uses Red Hat Enterprise Linux (RHEL) CoreOS.
RHEL CoreOS represents the next generation of single-purpose container operating system technology. RHEL CoreOS was created by the same development teams that created Red Hat Enterprise Linux Atomic Host and CoreOS Container Linux and combines the quality standards of Red Hat Enterprise Linux (RHEL) with the automated, remote upgrade features from Container Linux. Additionally, RHEL CoreOS is the new underlying operating system required for the master nodes along with the default operating system for the worker nodes.

While the underlying operating system consists primarily of RHEL components and the same quality, security, and control measures that support RHEL, there is one key feature that is not included in RHCOS: the sos package.

With the exclusion of the sos package, which provides the sosreport application, Red Hat provides two options to generate a sosreport from an RHCOS host. The first, which is the preferred method, is to debug the node using the oc debug command and the second is to connect to the host via SSH.

oc debug

Connect to master nodes via oc debug
List all the nodes in your cluster with the oc get nodes command and connect to the node required using oc debug
This way you would be connecting to a RHEL CoreOS 4.x node without SSH access

[ocpadmin@bastion suyi]$ oc get nodes
NAME                        STATUS   ROLES                  AGE    VERSION
ocpmn01.ocpcl.suyi.local   Ready    control-plane,master    21d   v1.25.14+20cda61
ocpwn01.ocpcl.suyi.local   Ready    app,worker              21d   v1.25.14+20cda61
ocpwn02.ocpcl.suyi.local   Ready    infra,worker            21d   v1.25.14+20cda61
[ocpadmin@bastion suyi]$ oc debug node/ocpmn01.ocpcl.suyi.local
A temporary namespace openshift-debug-knr4j is created for debugging node...
Starting pod/ocpmn01ocpclsuyilocal-debug ...
To use host binaries, run `chroot /host`
Pod IP: 10.1.1.69
If you don't see a command prompt, try pressing enter.
sh-4.4# chroot /host bash
[root@ocpmn01 /]#

The cluster will spawn a debug pod using the image registry.redhat.io/rhel7/support-tools.
Now chroot into the underlying host by running the chroot /host bash command.
Then run the toolbox command.

toolbox

Use the toolbox command to run the sosreport
Toolbox launches a container that brings debugging or admin tools. This starts a container using the registry.redhat.io/rhel8/support-tools:latest image if you are using RHEL CoreOS.
Once the toolbox container launches, you can now run the sosreport application to generate the sosreport for your support ticket.

sh-4.4# chroot /host bash
[root@ocpmn01 /]#
[root@ocpmn01 /]# toolbox
Trying to pull registry.redhat.io/rhel8/support-tools:latest...
Getting image source signatures
Checking if the image destination supports signatures
Copying blob ab212333ae4d done
Copying blob c2650fe947f6 skipped: already exists
Copying config 48112b036a done
Writing manifest to image destination
Storing signatures
48112b036ac8ad0f639d8703c9bcd00a91f1c3803c066a00dee28e53a43db21e
Spawning a container 'toolbox-root' with image 'registry.redhat.io/rhel8/support-tools'
Detected RUN label in the container image. Using that as the default...
aa01de6d5482bb6d120508a82e2e67c5fe5ff0f9379d20422b0c3d5320e1040c
toolbox-root
The container started successfully. To exit, type 'exit'.
[root@ocpmn01 /]# sos report

sos report

Earlier known by a single command sosreport, the latest command version is sos report
Simply run the command and enter the details requested.
After sos report has been generated the file will be saved in /var/tmp on the RHEL CoreOS node.

[root@ocpmn01 /]# sos report
sosreport (version 4.6.0)

This command will collect diagnostic and configuration information from
this Red Hat CoreOS system.

An archive containing the collected information will be generated in
/host/var/tmp/sos.u3jnn7nk and may be provided to a Red Hat support
representative.

Any information provided to Red Hat will be treated in accordance with
the published support policies at:

        Distribution Website: https://www.redhat.com/
        Commercial Support: https://access.redhat.com/

The generated archive may contain data considered sensitive and its
content should be reviewed by the originating organization before being
passed to any third party.

No changes will be made to system configuration.

Press ENTER to continue, or CTRL-C to quit.

Optionally, please enter the case ID that you are generating this report for []: 02343321

 Setting up archive ...
 Setting up plugins ...
[plugin:auditd] Could not open conf file /etc/audit/auditd.conf: [Errno 2] No such file or directory: '/etc/audit/auditd.conf'
[plugin:networking] skipped command 'ip -s macsec show': required kmods missing: macsec.   Use '--allow-system-changes' to enable collection.
[plugin:networking] skipped command 'ss -peaonmi': required kmods missing: xsk_diag, unix_diag, udp_diag, af_packet_diag, netlink_diag.   Use '--allow-system-changes' to enable collection.
[plugin:networking] skipped command 'ip netns exec 29dcbacd-ceb0-4201-a0a6-f0bcc8b265aa ss -peaonmi': required kmods missing: xsk_diag, unix_diag, udp_diag, af_packet_diag, netlink_diag.
[plugin:networking] skipped command 'ip netns exec 6d7a009c-516c-4618-b330-6ba78c985892 ss -peaonmi': required kmods missing: xsk_diag, unix_diag, udp_diag, af_packet_diag, netlink_diag.
caught exception in plugin method "system.setup()"
writing traceback to sos_logs/system-plugin-errors.txt
[plugin:systemd] skipped command 'resolvectl status': required services missing: systemd-resolved.
[plugin:systemd] skipped command 'resolvectl statistics': required services missing: systemd-resolved.
 Running plugins. Please wait ...

  Finishing plugins              [Running: networking]
  Finished running plugins
Creating compressed archive...

Your sosreport has been generated and saved in:
        /host/var/tmp/sosreport-ocpmn01-02343321-2024-01-09-jbzltfe.tar.xz

 Size   76.99MiB
 Owner  root
 sha256 14c9628027a8c7ed1188bd722e25df936d3952778cfb159556b401160d689d5b

Please send this file to your support representative.

redhat-support-tool

Considering that the nodes have internet access, it's possible to directly upload the sosreport to your case using the redhat-support-tool application.

[root@ocpmn01 /]# redhat-support-tool addattachment -c 02343321 /host/var/tmp/sosreport-ocpmn01-02343321-2024-01-09-jbzltfe.tar.xz
Please enter your RHN user ID: suyash.sambhare
Save the user ID in /root/.redhat-support-tool/redhat-support-tool.conf (y/n): y
Please enter the password for suyash.sambhare:
Save the password for suyash.sambhare in /root/.redhat-support-tool/redhat-support-tool.conf (y/n): y
Uploading sosreport-ocpmn01-02343321-2024-01-09-jbzltfe.tar.xz to the case ...
 100.00% completed successfully.

[root@ocpmn01 /]#

Congratulations!! 🫠👩🏽‍🚀🪮🫖⛰️〽️
You have successfully generated the sos report on the master nodes of Red Hat OpenShift Cluster!

Ref: https://www.redhat.com/en/blog/how-generate-sosreport-red-hat-enterprise-linux-coreos