Always check your logs

Brad’s #1 rule of debugging web apps: always check your log files. All of them.

A friend of mine just ran into a problem where a php file upload script that had been working fine for months suddenly started barfing on large files. He was getting a “partial upload” error. I told him to check his web server error logs and his system message log. He did the former, and after hours of pounding his head against the wall, I finally went in and poked around myself. I actually found the problem before looking in the log files, but there it was in /var/log/messages:

Jul 7 15:27:42 xxxx /kernel: pid 46891 (httpd), uid 80 on /var: file system full
Jul 7 15:30:02 xxxx /kernel: pid 46875 (httpd), uid 80 on /var: file system full
Jul 7 15:49:07 xxxx /kernel: pid 46872 (httpd), uid 80 on /var: file system full

The temp directory that php was using for file uploads was /var/tmp, and /var only had 216K left. This is why small file uploads were working, but larger ones weren’t. (As an aside, be careful of what you’re storing in /var. On some systems, log files are kept in /var/log, and /var is often set up as a separate partition, and often relatively small at that. In his case, the whole partition was only 252M, and /var/log was eating up 195M of that…and growing. We moved /var/log to /usr/var/log and created a symlink.)