Troubleshooting: Method "/lib/svc/method/fs-local" failed with exit status 95 in Solaris - Possible Solution

Friday, August 27, 2010

Yesterday, when I was working with my Solaris system, after reboot, I couldn't ssh into the system, the ssh command failed with the "Connection refused" error, after some research (by logging into the serial console of the server and rebooting the system again through the serial console), I found the below system error message.

svc:/system/filesystem/local:default: Method "/lib/svc/method/fs-local" failed with exit status 95

Then everything came to standstill till I figured out the solution.

Solution:

1. The possible reason for ssh "Connection Refused" may be due to the fact that some filesystem, whose entry is in /etc/vfstab would have failed to mount properly.

A typical vfstab entry looks like the below one.

$ cat /etc/vfstab
#device         device          mount           FS      fsck    mount   mount
#to mount       to fsck         point           type    pass    at boot options
#
fd      -       /dev/fd fd      -       no      -
/proc   -       /proc   proc    -       no      -
/dev/dsk/c0t2d0s1       -       -       swap    -       no      -
/dev/dsk/c0t2d0s0       /dev/rdsk/c0t2d0s0      /       ufs     1       no      logging
...
...

2. To find which entry was causing the problem, I checked the system log, the entries were as follows.

$ cat /var/svc/log/system-filesystem-local:default.log
"/var/svc/log/system-filesystem-local:default.log" 149 lines, 9256 characters
[ Dec  3 01:25:35 Rereading configuration. ]
[ Dec  3 01:25:52 Executing start method ("/lib/svc/method/fs-local") ]
[ Dec  3 01:25:52 Method "start" exited with status 0 ]
[ Dec  3 01:31:54 Executing start method ("/lib/svc/method/fs-local") ]
....
....
WARNING: /sbin/mountall -l failed: exit status 33
[ Aug 27 07:33:14 Method "start" exited with status 95 ]
[ Aug 27 07:35:37 Leaving maintenance because clear requested. ]
[ Aug 27 07:35:37 Enabled. ]
[ Aug 27 07:35:37 Executing start method ("/lib/svc/method/fs-local") ]
mount: /var/js/1/boot: No such file or directory
WARNING: /sbin/mountall -l failed: exit status 1
[ Aug 27 07:35:37 Method "start" exited with status 95 ]
[ Aug 27 07:37:19 Leaving maintenance because clear requested. ]
[ Aug 27 07:37:19 Enabled. ]
[ Aug 27 07:37:19 Executing start method ("/lib/svc/method/fs-local") ]
mount: /var/js/1/boot: No such file or directory
WARNING: /sbin/mountall -l failed: exit status 33
[ Aug 27 07:37:19 Method "start" exited with status 95 ]
svc:/system/filesystem/local:default: Method "/lib/svc/method/fs-local" failed with exit status 95

Looks like there was an entry /var/js/1/boot in /etc/vfstab which didn't mount because the filesystem was not present, hence the error.

3. After finding the root cause for the problem, I removed the line which mounted the /var/js/1/boot filesystem in /etc/vfstab and rebooted the system, now I could login through ssh.

Therefore when you get ssh "Connection refused" error, chances are that some filesystem whose entry is in /etc/vfstab would have failed to mount, in which case check the system log, remove the line causing the problem from /etc/vfstab and ssh should work fine.

3 comments:

Anonymous said...

Thank you very much for posting this problem and solution! Somebody left bad info in /etc/vfstab and that was preventing the system from bringing up sshd. Stupid move on Sun's part!

Prasanna Seshadri said...

I don't think it has to do with Sun, any file where there are stale entries would cause this.

Anonymous said...

It's really dumb of Sun to keep ssh from coming up if a mount fails. If you can't get in through the serial port to fix the problem, you are screwed!


Copyright © 2008 Prasanna Seshadri, www.prasannatech.net, All Rights Reserved.
No part of the content or this site may be reproduced without prior written permission of the author.