Menu

Subscribe

ZFS, it's sometimes good to know how screwed you are.

on opensolaris, zfs

I have just had a disk fail on my NAS, actually it happened ages ago but I was too broke to replace it. At the same time as one being faulted, another was degraded through having too many errors. Below is my interaction with ZFS to discover the extent of the problem and “fix” it.

(this is after I had swapped out c2t1d0, only then did I know I had data corruption, should have periodically scrubed the pool!)

Lets see how bad it is ` root@nas:~# zpool status pool: Toms state: DEGRADED status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scrub: resilver completed after 4h57m with 163802 errors on Sun Feb 8 16:47:46 2009 config

NAME STATE READ WRITE CKSUM Toms DEGRADED 35 0 320K raidz1 DEGRADED 35 0 320K c2t0d0 DEGRADED 0 0 0 too many errors c2t2d0 DEGRADED 89 0 3 too many errors c2t1d0 ONLINE 0 0 0 c2t4d0 DEGRADED 12 0 243 too many errors raidz1 ONLINE 0 0 0 c2t5d0 ONLINE 0 0 0 c2t7d0 ONLINE 0 0 0 c2t3d0 ONLINE 0 0 0 c2t6d0 ONLINE 0 0 0

errors: 140338 data errors, use ’-v’ for a list

pool: syspool state: ONLINE scrub: none requested config:

NAME STATE READ WRITE CKSUM syspool ONLINE 0 0 0 c1t1d0s0 ONLINE 0 0 0

errors: No known data errors `

Ok, so what files are affected root@nas:~# zpool status -v ..................(cut) /share/TalkingBooks/TTC/Concert Masterworks/CD10-Track07.mp3 /share/TalkingBooks/TTC/Science in the Twentieth Century A Social-Intellectual Survey/09-10 - Subterranean Fury.mp3 /share/TalkingBooks/TTC/Foundations of Western Civilization/45-2 - The Protestant Reformation - John Calvin.mp3 /share/TalkingBooks/TTC/The Long 19th Century, European History from 1789 to 1917/05.05 The Napoleonic Era, 1799-1815.mp3 /share/TalkingBooks/TTC/Great World Religions - Islam/Lecture 6d Paths To God -- Islamic Law And Mysticism.mp3 /share/TalkingBooks/TTC/History of Ancient Egypt/Lecture 23 The Murder of Tutankhamen.mp3 /share/TalkingBooks/TTC/The Symphony/00-02 Page 20-21.jpg /share/TalkingBooks/TTC/The United States And The Middle East 1914- 9-11/19-3 The First Palestinian Intifada.mp3 /share/TalkingBooks/TTC/World of Byzantium/TTC - World of Byzantium - Guidebook Part 2/[World.of.Byzantium.part.II]16-17.jpg /share/TalkingBooks/TTC/How to Read and Understand Poetry/13 Free Verse Track 2.mp3 /share/Tunes/Tom Jones - Reload/04 All Mine with Divine Comedy.mp3 /share/Tunes/Tom Jones - Reload/10 Sometimes We Cry with Van Morrison.mp3 /share/Tunes/Tom Jones - Reload/08 You Need Love Like I Do with Heather Small.mp3 /share/Tunes/Tom Jones - Reload/07 Sexbomb with Mousse T.mp3 /share/Tunes/Tom Jones - Reload/06 I'm Left, You're Right, She's Gone with James Dean Bradfield.mp3 ..................(cut)

Lets take a snapshot before we delete anything root@nas:~# zfs list NAME USED AVAIL REFER MOUNTPOINT Toms 1.78T 421G 40.4K none Toms@one 0 - 40.4K - Toms/share 1.43T 421G 31.3G /share Toms/share/500G 336G 421G 336G /share/500G Toms/share/Notes 38.9G 421G 38.9G /share/Notes Toms/share/SetupFiles 72.1G 421G 72.1G /share/SetupFiles Toms/share/TalkingBooks 144G 421G 144G /share/TalkingBooks Toms/share/Tunes 61.5G 421G 61.5G /share/Tunes Toms/share/Videos 337G 421G 337G /share/Videos Toms/share/apps 28.4K 421G 28.4K /share/apps Toms/share/downloads 359G 421G 359G /share/downloads Toms/share/files 34.6G 421G 34.6G /share/files Toms/share/xbox 52.8G 421G 52.8G /share/xbox Toms/tom 7.59G 421G 7.59G /tom syspool 1.17G 225G 23K none syspool/rootfs-nmu-000 1.17G 225G 1.12G legacy syspool/rootfs-nmu-000@initial 49.8M - 631M

root@nas:~# zfs snapshot Toms@b4delete root@nas:~# zfs list NAME USED AVAIL REFER MOUNTPOINT Toms 1.78T 421G 40.4K none Toms@one 0 - 40.4K - Toms@b4delete 0 - 40.4K - Toms/share 1.43T 421G 31.3G /share Toms/share/500G 336G 421G 336G /share/500G Toms/share/Notes 38.9G 421G 38.9G /share/Notes Toms/share/SetupFiles 72.1G 421G 72.1G /share/SetupFiles Toms/share/TalkingBooks 144G 421G 144G /share/TalkingBooks Toms/share/Tunes 61.5G 421G 61.5G /share/Tunes Toms/share/Videos 337G 421G 337G /share/Videos Toms/share/apps 28.4K 421G 28.4K /share/apps Toms/share/downloads 359G 421G 359G /share/downloads Toms/share/files 34.6G 421G 34.6G /share/files Toms/share/xbox 52.8G 421G 52.8G /share/xbox Toms/tom 7.59G 421G 7.59G /tom Toms/vffs 349G 421G 349G /vffs syspool 1.17G 225G 23K none syspool/rootfs-nmu-000 1.17G 225G 1.12G legacy syspool/rootfs-nmu-000@initial 49.8M - 631M - Just how many file are effected? root@nas:~# zpool status -v | grep " /share/" | wc -l 2618

Lets get rid (probably can be done neater, I had to google for the sed syntax to strip whitespace.) root@nas:~# zpool status -v | grep " /share" | sed 's/^[ \t]*//;s/[ \t]*$//' | xargs -d "\n" rm root@nas:~# zpool status -v | grep " /share/" | wc -l 0 No bad files now

Clear the errors on the disks (hopefully they are not really ALL bad) root@nas:~# zpool clear Toms root@nas:~# zpool status pool: Toms state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scrub: resilver completed after 4h57m with 163802 errors on Sun Feb 8 16:47:46 2009 config:

NAME STATE READ WRITE CKSUM Toms ONLINE 0 0 0 raidz1 ONLINE 0 0 0 c2t0d0 ONLINE 0 0 0 c2t2d0 ONLINE 0 0 0 c2t1d0 ONLINE 0 0 0 c2t4d0 ONLINE 0 0 0 raidz1 ONLINE 0 0 0 c2t5d0 ONLINE 0 0 0 c2t7d0 ONLINE 0 0 0 c2t3d0 ONLINE 0 0 0 c2t6d0 ONLINE 0 0 0

errors: 140338 data errors, use ’-v’ for a list

pool: syspool state: ONLINE scrub: none requested config:

NAME STATE READ WRITE CKSUM syspool ONLINE 0 0 0 c1t1d0s0 ONLINE 0 0 0

errors: No known data errors Still some errors, need to figure out what this means root@nas:/share# zpool status -v ..................(cut) Toms/share/TalkingBooks:<0x61ff> Toms/share/TalkingBooks:<0x62ff> Toms/share/TalkingBooks:<0x64ff> Toms/share/TalkingBooks:<0x65ff> Toms/share/Tunes:<0x239> Toms/share/Tunes:<0x247> Toms/share/Tunes:<0x34e> Toms/share/Tunes:<0x47f> ..................(cut)