Posts

Linux: No space left on device – running out of Inodes – PHP sessions

A production Ubuntu server could not create anymore files claiming that no space was left. Strangely enough it was not about storage space but inode availability. The first response might be to increase disk size but it will not help unless you format the drive to assign a new number of inodes. Below is the sample output that highlights the problem.

The Problem

It should be very hard to exhaust the inodes unless huge amount of 0-sized or very small files are created. If you look in the image above, more than 1 million inodes have been used.

The Solution

It’s necessary to locate all those small files in the partition to get to the source of the problem. We want to find folders with unusual large number of files. A bash command can be of help starting at the root / (borrowed from Ivan).

The command will list the directories with the number of files inside, from that point on you can keep on looking up some sub-folders until you find the source. If a sub-folder has too many files, the script might just hang for a long time, so you might have found the troublemaker.

In this particular case it turned out that PHP sessions were at the source of the issue.

Reason: PHP Sessions

This server is configured to keep the sessions using files thus the Garbage Collection does not take place. Therefore, the sessions have to be cleaned manually. From the previous Stack Overflow link, a cron task can be used for the cleanup:

In order to work, the previous code has to be located inside the file  /etc/cron.d/php5

The time can be adjusted according to the actual PHP configuration.

In this particular case there were about 1 million PHP session files, running the cron script would just take too much time. Therefore, a manual delete did the work.

 

preg_replace compilation failed (Amazon EC2, WordPress)

We have an Amazon EC2 AMI (Linux) hosting the latest WordPress installation (3.4.1) at this point in time. All the configuration was running fine till an upgrade that introduced a PHP warning:

“Warning: preg_replace(): Compilation failed: unknown option bit(s) set at offset 0”

The first time I noticed it, I didn’t pay that much attention since the WordPress administration was still working besides the warning but some other functionality like the shortcodes was not.

The problem arose when we needed the GD library to generate png and jpg images on the fly which, is used in a WordPress theme. By doing a simple install of the library (php-gd.x86_64, for our system) several dependencies were upgraded to newer versions (php, php-cli, php-common, php-mysql, php-pdo). The install finished normally without any warnings to be worried about.

After a while digging in Google, I found this post which was having a similar problem and most importantly it hinted that the problem was the PCRE library (libpcre), which was being loaded differently (different versions) in the apache and commandline modes. Running the php code below in command line and a web page would show that:

The same post mentioned that a libpcre upgrade would fix the problem. Thus, I went for the upgrade from version 7.8-3.1.8 to version 8.21-5.3 in our Amazon Linux instance running the following commands:

If you’re running a different linux system other than the Amazon Linux AMI or CentOS you might refer to your docs. For example in Ubuntu the package manager is apt-get.

This should never have happened if the dependencies were checked correctly during the GD library install. Maybe the libpcre version was not considered as needing an upgrade. Anyhow, I hope this helps others to save some time fixing this problem.

If you have any comments or further information on this issue, I’d appreciate if you let me know.

Portfolio Items