Posts

Linux: No space left on device – running out of Inodes – PHP sessions

A production Ubuntu server could not create anymore files claiming that no space was left. Strangely enough it was not about storage space but inode availability. The first response might be to increase disk size but it will not help unless you format the drive to assign a new number of inodes. Below is the sample output that highlights the problem.

The Problem

It should be very hard to exhaust the inodes unless huge amount of 0-sized or very small files are created. If you look in the image above, more than 1 million inodes have been used.

The Solution

It’s necessary to locate all those small files in the partition to get to the source of the problem. We want to find folders with unusual large number of files. A bash command can be of help starting at the root / (borrowed from Ivan).

The command will list the directories with the number of files inside, from that point on you can keep on looking up some sub-folders until you find the source. If a sub-folder has too many files, the script might just hang for a long time, so you might have found the troublemaker.

In this particular case it turned out that PHP sessions were at the source of the issue.

Reason: PHP Sessions

This server is configured to keep the sessions using files thus the Garbage Collection does not take place. Therefore, the sessions have to be cleaned manually. From the previous Stack Overflow link, a cron task can be used for the cleanup:

In order to work, the previous code has to be located inside the file  /etc/cron.d/php5

The time can be adjusted according to the actual PHP configuration.

In this particular case there were about 1 million PHP session files, running the cron script would just take too much time. Therefore, a manual delete did the work.

 

Portfolio Items