Aria Operations - FSDB file corruption detected

INTRO

A few days ago, while testing VMware ARIA Operations 8.14.1, I came across this alert: "FSDB file corruption detected." Typically, this suggests potential corruption within the directory /storage/db/vcops/data/. Searching online, I found VMware's KB2126139, which advises deleting the alert and contacting support if it reoccurs. I've decided to write this article to share some insights on the matter.

ISSUE

This message suggests potential corruption within the directory /storage/db/vcops/data/. The corruption of these files could have been caused by an unexpected poweroff, a VM crash, or the filling of the database partition. Therefore, the first step is to ensure that the disk space of the database is not exhausted.

To proceed with cleaning or expanding the disk space of VMware Aria Operations, you can follow the KB2093903 and KB2016022.

After verifying that there is sufficient disk space, we can check for corrupted files on the filesystem. Once connected via SSH to the node with the analytics service, we can use the following command:

1find /storage/db/vcops/data/ -type f -not -regex '.*/[2][0][2][0-3]_[0-9][0-9]_.*.dat' -and ! -regex '.*/[2][0][1][7-9]_[0-9][0-9]_.*.dat' -and ! -name '*dtr' -and ! -name 'mps_*'

1/storage/db/vcops/data/cache/0.cache
2/storage/db/vcops/data/cache/.lock

An example of a corrupted file name is as follows:

1/usr/lib/vmware-vcops/data/8/8584/163603858_02_8584.dat

Note the future timestamp.

An example of a valid file name is as follows:

1/usr/lib/vmware-vcops/data/10/10718/2020_10_10718.dat

Note the relatable timestamp.

Let's make a note of all the names of the files that appear corrupted.

RESOLUTION

After noting down the files to be deleted, we proceed by taking the cluster offline from the Admin UI and then shutting down the nodes from vCenter. We create a snapshot of the powered-off VM. Immediately after powering on the VM, we log in via SSH while the cluster is still offline and proceed with the removal of the corrupted files using the rm command.

Example

1rm /usr/lib/vmware-vcops/data/8/8584/163603858_02_8584.dat

Next, we proceed by clearing the cache on all analytics nodes using the command:

1rm /usr/lib/vmware-vcops/data/cache/*.cache

Now we can restart the services and verify that everything is okay. If the alert persists, it might be necessary to manually delete the alert.

In some cases, the corrupted files might not be present, and simply cleaning the cache of the offline cluster nodes is sufficient.

By following these steps, you can efficiently address and resolve the issue of corrupted FSDB files in VMware Aria Operations, ensuring your analytics nodes are functioning correctly.

Have you faced a similar FSDB file corruption issue in VMware ARIA Operations, or do you have any troubleshooting tips to share? I would love to hear about your experiences and any advice you might have. Please leave your thoughts and questions in the comments below, or if you found this post helpful, consider sharing it on social media. Your feedback not only helps me improve but also assists others in the community dealing with similar challenges.