A production Ubuntu server could not create anymore files claiming that no space was left. Strangely enough it was not about storage space but inode availability. The first response might be to increase disk size but it will not help unless you format the drive to assign a new number of inodes. Below is the sample output that highlights the problem.
# df -h Filesystem Size Used Avail Use% Mounted on /dev/xvda1 17G 7.7G 9.0G 46% / devtmpfs 1001M 4.0K 1001M 1% /dev none 201M 168K 201M 1% /run none 5.0M 0 5.0M 0% /run/lock none 1001M 76K 1001M 1% /run/shm /dev/xvdg 18G 11G 5.9G 66% /srv/mobi_BACKUP
# df -i Filesystem Inodes IUsed IFree IUse% Mounted on /dev/xvda1 1066240 1061813 4427 100% / devtmpfs 256189 333 255856 1% /dev none 256229 250 255979 1% /run none 256229 2 256227 1% /run/lock none 256229 29 256200 1% /run/shm /dev/xvdg 1179648 68692 1110956 6% /srv/mobi_BACKUP
It should be very hard to exhaust the inodes unless huge amount of 0-sized or very small files are created. If you look in the image above, more than 1 million inodes have been used.
It’s necessary to locate all those small files in the partition to get to the source of the problem. We want to find folders with unusual large number of files. A bash command can be of help starting at the root / (borrowed from Ivan).
for i in /*; do echo $i; find $i | wc -l; done
The command will list the directories with the number of files inside, from that point on you can keep on looking up some sub-folders until you find the source. If a sub-folder has too many files, the script might just hang for a long time, so you might have found the troublemaker.
for i in /var/lib/*; do echo $i; find $i | wc -l; done
In this particular case it turned out that PHP sessions were at the source of the issue.
Reason: PHP Sessions
This server is configured to keep the sessions using files thus the Garbage Collection does not take place. Therefore, the sessions have to be cleaned manually. From the previous Stack Overflow link, a cron task can be used for the cleanup:
# /etc/cron.d/php5: crontab fragment for php5 # This purges session files older than X, where X is defined in seconds # as the largest value of session.gc_maxlifetime from all your php.ini # files, or 24 minutes if not defined. See /usr/lib/php5/maxlifetime # Look for and purge old sessions every 30 minutes 09,39 * * * * root [ -d /var/lib/php5 ] && find /var/lib/php5/ -type f -cmin +$(/usr/lib/php5/maxlifetime) -print0 | xargs -r -0 rm
In order to work, the previous code has to be located inside the file /etc/cron.d/php5
The time can be adjusted according to the actual PHP configuration.
In this particular case there were about 1 million PHP session files, running the cron script would just take too much time. Therefore, a manual delete did the work.