as the title says, there was some helpdesk issues that arose. particularly with the uvdesk virtual machine. i was going through npm and clicked on my helpdesk url when it gave me a nasty page about how there was errors. specifically this one:

Notice: fwrite(): Write of 442 bytes failed with errno=28 No space left on device in /var/www/uvdesk/vendor/monolog/monolog/src/Monolog/Handler/StreamHandler.php on line 181

this was a gut hitter as i did not know what was going on. meaning that my helpdesk was down. i obviously did not take the time to read it clearly because it says “no space left on device in …”. i kind of glossed over it because i set aside one hundred twenty eight gigabytes of storage for that vm. upon further research, it meant that there was no more space in the /var partition. so it was because it logged a lot of things that it filled up the partition, which is no good. luckily i had a backup i did at the beginning of the month, and this was a full backup. reasoning was so that i had a clean baseline before i deleted all my backups and started on pbs.

which reinforced the importance of backups, full and incremental. i would assume that this has been happening for the past few days, so it might have screwed with my incremental backups. and after checking the logs after each backup, i saw that it took the backup a bit longer than usual, so it must have been recent. currently as i am writing this, i am fixing the issue.

14:24 – realized help desk page was down and saw the error
14:29 – ran a full backup of the vm to get the current status baseline
14:31 – restored to known full backup
14:40 – started trying to fix the partition
15:05 – created this page to document
15:12 – continued fixing
15:17 – accidentally nuked the vm disk, restored the backup again
15:35 – finished the partition fixes, and ran an lsblk to confirm.

so what did i learn? provision the /var partition for production servers because logs will build up overtime, especially from php. so lesson learned well, i will have to anticipate that for my web apps. i might restore the backup that i did to get a baseline of the issue, but we will see. probably not since there has not been too much activity between the baseline backup and now.

i decided to install the gparted live media iso and boot into it from my vm. changed the boot order and moved around the partitions so that i could extend the /var partition. process was very easy, but did take a bit of research since it was a bit since i used gparted.