File system /storage/seat is low on database storage space.

VCSA 6.7 error related to disk space  

VMware has this KB with the steps to increase the Disk on the VCSA

https://kb.vmware.com/s/article/2145603

=================================================

below the same steps for your convenience, 

• when we checked the VAMI interface we saw below error

File system /storage/seat is low on database storage space.

Increase the size of disk /storage/seat or decrease the data retention.

when we checked the VCSA /storage/seat reached 98%, which caused vpxd services to stop by the design to prevent further data corruption.

root@vcsa001 [ ~ ]# df -h

Filesystem                Size Used Avail Use% Mounted on

devtmpfs                 7.9G   0 7.9G  0% /dev

tmpfs                   7.9G 932K 7.9G  1% /dev/shm

tmpfs                   7.9G 680K 7.9G  1% /run

tmpfs                   7.9G   0 7.9G  0% /sys/fs/cgroup

/dev/sda3                 11G 6.4G 3.8G 63% /

tmpfs                   7.9G 1.5M 7.9G  1% /tmp

/dev/mapper/netdump_vg-netdump      985M 1.3M 916M  1% /storage/netdump

/dev/mapper/imagebuilder_vg-imagebuilder 9.8G  23M 9.2G  1% /storage/imagebuilder

/dev/mapper/db_vg-db           9.8G 493M 8.8G  6% /storage/db

/dev/mapper/autodeploy_vg-autodeploy   9.8G  23M 9.2G  1% /storage/autodeploy

/dev/mapper/archive_vg-archive       50G  47G  15M 100% /storage/archive

/dev/mapper/dblog_vg-dblog         15G 9.3G 4.7G 67% /storage/dblog

/dev/sda1                 120M  34M  78M 31% /boot

/dev/mapper/seat_vg-seat          25G  23G 500M 98% /storage/seat

/dev/mapper/core_vg-core          50G 2.4G  45G  6% /storage/core

/dev/mapper/log_vg-log          9.8G 3.6G 5.7G 39% /storage/log

/dev/mapper/updatemgr_vg-updatemgr     99G 1001M  93G  2% /storage/updatemgr

Resolution

• identify the disk -

df -h

/dev/mapper/seat_vg-seat          25G  23G 500M 98% /storage/seat

lsblk

sdh               8:112 0  25G 0 disk

└─seat_vg-seat         252:8  0  25G 0 lvm /storage/seat

lsscsi

[2:0:8:0]  disk  VMware  Virtual disk   1.0  /dev/sdh

• increased the disk space by 15GB
• used /usr/lib/applmgmt/support/scripts/autogrow.sh to grow the file system

• restart the appliance and confirm vCenter is working

autogrow.sh makes things really easy with VCSA 6.7 disk expansion 

dnsmasq.log - root partition is full VCSA 6.5

In the Morning got alerts regarding the VCSA Health ,I logged to VCSA Management Interface 

https://vcentercerver:5480

I can see some Helth warning on VCSA, SSH to VCSA using putty and immediately Noticed that  /root partition is full..



Then I dig deeper to find which files causing the issue and find the culprit bunch of dnsmasq.log files were created in the /var/log directory.  



Found KB related the issue that I'm having .. seems like it is safe to remove larger log files as a quick solution, it cleared most of the space and now I have free space.. 




Still I need a permanent solution for this .. Time to follow the KB and add parameters to Log rotate..   

First edit Log rotate for dnsmasq

vi /etc/logrotate.d/dnsmasq
Update the file to contain these lines and save the file: /var/log/vmware/dnsmasq.log {
nodateext
daily
missingok
notifempty
compress
maxsize 5M
rotate 5
sharedscripts
postrotate
[ ! -f /var/run/dnsmasq.pid ] || kill -USR2 `cat /var/run/dnsmasq.pid`
endscript
create 0640 dnsmasq dnsmasq
}

Then update the dnsmasq.conf

vi /etc/dnsmasq.conf

Update the file to contain these lines and save the file:

listen-address=127.0.0.1
bind-interfaces
user=dnsmasq
group=dnsmasq

no-negcache
no-hosts
log-queries
log-facility=/var/log/vmware/dnsmasq.log
domain-needed
dns-forward-max=150
cache-size=8192
neg-ttl=3600 

finally restart dnsmasq service

service dnsmasq restart 

new log files are much smaller and they are zipped.

 

VMware KB shows that this issue resolved in below version.

• vCenter Server 6.0 U3g, available at VMware Patch Downloads.
• vCenter Server 6.5 U2, available at VMware Downloads.