Tuesday, February 21, 2012

Where Did My Files Go: Root, NAS, and Linux Heartache

Well, this is interesting.

I wanted to set up a share on the NSS4000 to use as a holding spot for the homes folders I would be creating for my virtual FTP users (another post for another time). I was having difficulty adjusting the permissions on the share; I couldn't change the owner to the FTP user, which meant the whole setup wouldn't work. I found that the NSS4000 has the root_squash option hard-coded NSS4000 has the root_squash option hard-coded and the only way to fix this (other than some hacky methods to enable SSH) was to do a firmware upgrade. The instructions for the upgrade were not encouraging either. Not only do you have to run the firmware upgrade procedure twice, but it is also common for the first run-through to hang and require a hard reset after waiting 15 minutes or so. If your unit is in a data center you see the difficulty with this method.

I bit the bullet, scheduled downtime, and went out to our data center in the cold, dark, wee hours of the night to do the upgrade. Sure enough I did have to pull the plug on the thing. Luckily I had other things to do in the meantime (again, another post) so I wasn't bored.

It wasn't too long after (i.e. the next day) that I started getting backup failure reports from our provider.
See, our MySQL server performs a dump of all its databases and puts them on the NAS. They then get zipped by the same cron job and our offsite backup picks up the compressed file. I was getting error notifications that the backup process could not grab the actual individual database files, which was confusing because it wasn't supposed to be getting those files -- it was supposed to be grabbing the zip file.

After checking logs and verifying that the script hadn't changed, the clues lead me to the realization that somehow the firmware upgrade had adjusted permissions and now they were fubar'd. Attempts to access any of the zip files from January and earlier via Windows Explorer worked fine. Any attempt to access zip files from February resulted in an error that 'Windows cannot open the folder.' and that the compressed file is invalid. What was odd was that the firmware upgrade happened mid-month, so if that was the problem I should still be able to access the rest of February's files.

Viewing the permissions from the Linux box as root:

root@database:/mnt/nas# ls -ll | less
total 0
d????????? ? ? ? ?                ? \2012-02-16
-????????? ? ? ? ?                ? \2012-02-16\dbbackup.zip
-????????? ? ? ? ?                ? \2012-02-16\database1.sql
-????????? ? ? ? ?                ? \2012-02-16\database2.sql
-????????? ? ? ? ?                ? \2012-02-16\database3.sql
-????????? ? ? ? ?                ? \2012-02-16\database4.sql

The question marks mean the user can't stat the file in question, i.e. doesn't have execute permissions (although it does have read permissions).

Running this exact same ls command as root immediately after however has a completely different output, showing the actual permissions entries and the owner and group. I verified this by piping the output of each go into two different files and comparing. I also could not navigate into any of the folders. For example in the above snippet Linux sees the directory 2012-02-16 and it shows in Windows Explorer, but I couldn't cd into it.

I decided to check out the NAS itself since I hadn't investigated it since the firmware upgrade. Beyond some basic cosmetic differences (like the fancy Cisco branding that wasn't there before, in a new fetching shade of blue) there didn't seem to be anything too different...until I looked at the share properties. There's a new option here


It was checked. It seemed to only be related to NFS, but I unchecked it anyway and remounted the share on my Linux server. I didn't see a repeat of the mysterious question marks, but I still was unable to access any mounted folders created after I updated the firmware. I stared at my ls output until it finally hit me. I was completely ignoring the fact that the files were preceded with backslashes, the Windows path delimiter. If I escaped that I could in fact cd into the directory. Of course, once there it claimed to not see any files, but the general issue turned out to be the fact that the NAS shares were showing in a Windows directory listing format. This was some kind of side effect of the firmware upgrade. Once I remounted the share I verified that the new directories and files I created showed up as normal.

So what to do about the files that are still there? I needed them for backups, but they were useless as they stood. I couldn't access them via Windows Explorer; they didn't even show by their proper names. The files listed above in the ls command had some sort of truncated filename in Windows, like "_0B5OV.sql". I could access the files via the Linux box by making sure to use the escape character (\), but what a pain to do!

There were approximately 4 days worth of goofy files. I created the appropriate root directories and moved the corresponding .sql files into them. Next I needed to rename the files, stripping the Windows file path. Of course I didn't want to go through and do this one by one so with a little research I acquainted myself with substring manipulation using the # and % bash operators. My resulting script looked like this:


#!/bin/bash

for f in *.sql;

do

mv "$f" ${f#\\2012-02-16\\};

done


I ran this script in each of the new directories I'd created, changing the directory string of course, and came back with simple .sql files that I could then zip up. 

Thanks heaps to stackoverflow.com and linuxquestions.org.






No comments:

Post a Comment