<p>Justin, <br>
Taking this idea to production on network servers would be interesting. I am working on a networked backup script and I like the thought of automating a scoring system for backup priority.</p>
<p>Wolf</p>
<p><a href="http://evergreen-community-01.lyrasistechnology.org">http://evergreen-community-01.lyrasistechnology.org</a><br>
<a href="http://sourcefreedom.com">http://sourcefreedom.com</a><br>
Apache developer:<br>
<a href="mailto:wolfhalton@apache.org">wolfhalton@apache.org</a></p>
<div class="gmail_quote">On May 14, 2012 10:51 PM, "Justin Goldberg" <<a href="mailto:justgold79@gmail.com">justgold79@gmail.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
That's a good argument for distributed, intelligent, encrypted backup.<br>
Think of a private Freenet.<br>
<br>
We all have tons of drives and computers but why do we have to backup<br>
everything? For example, we edit a document that we downloaded, we<br>
keep a few revisions/backups. We create a document, score that file<br>
higher to keep more backups of that file. Mp3 files from bittorrent,<br>
don't back them up or score really low. Mp3 files we ripped from a cd,<br>
back them up, scored higher than the downloaded ones, but lower than a<br>
document or our bookmarks file. Or you could say that This would be<br>
best created by a filesystem like BFS, where custom database<br>
attributes can be kept. I guess a fully journaled filesystem would<br>
accomplish this as well, but I'm dreaming of something network-wide.<br>
<br>
On 5/14/12, Ron Frazier (ALE) <<a href="mailto:atllinuxenthinfo@techstarship.com">atllinuxenthinfo@techstarship.com</a>> wrote:<br>
> Hi Mike T,<br>
><br>
> Thanks for all the info you shared in this post. It's taught me several<br>
> things I didn't know about the low level operation of a PC. I may have<br>
> mis-spoken about the reliance of SpinRite on BIOS. I think I read somewhere<br>
> on his website (later) that he bypasses BIOS, but I don't know how all the<br>
> details are worked out. I try to do regular backups, but always seem to be<br>
> behind on doing so. I did eventually manage to boot a Ubuntu live CD on my<br>
> old pc and run a badblocks non destructive read write test. It finished the<br>
> Linux partitions fine and is chugging away on one of two windows partitions.<br>
> This test takes a LONG time, just as it likewise does with SpinRite. I found<br>
> an interesting quirk you have to account for when specifying the block<br>
> numbers for badblocks to use, which I commented on in another thread.<br>
> Essentially, if you get the number of blocks from fdisk, you have to<br>
> subtract 1 from the number to feed it to badblocks. That drove me crazy for<br>
> half a day or so.<br>
><br>
> You mention data scrubbing in your post. I've run across that in my<br>
> research, as well as some very interesting info on latent defects. My<br>
> research is ongoing, but my current conclusion is that data scrubbing is<br>
> beneficial for personal drives too, hence my willingness to run 36 - 48<br>
> hours of diagnostics on my drives, 2 - 3 times / year. Data scrubbing may be<br>
> more valuable on personal drives than on raid, since there is no drive to<br>
> auto failover to. If the sectors on my drive fail to read, unless SpinRite<br>
> can recover them, which I've seen it do on occasion, then my only solution<br>
> is to resort to backups, which are probably out of date.<br>
><br>
> Another, unrelated, problem is that, about half the time, that old computer<br>
> will not boot the GUI Unity interface on the live CD of Ubuntu 11.10. At<br>
> other times, it works. Totally insane. I hate Unity anyway, but it would be<br>
> nice to have SOME kind of GUI running. To run the recent badblocks test, I<br>
> had to resort to hitting ALT-F1 to get to a terminal and use that. The GUI<br>
> never did start up. I tried the startx command, but that didn't do anything<br>
> either.<br>
><br>
> The saga to preserve ever so fragile data continues ...<br>
><br>
> Sincerely,<br>
><br>
> Ron<br>
><br>
><br>
> --<br>
><br>
> Sent from my Android Acer A500 tablet with bluetooth keyboard and K-9 Mail.<br>
> Please excuse my potential brevity.<br>
><br>
> (To whom it may concern. My email address has changed. Replying to former<br>
> messages prior to 03/31/12 with my personal address will go to the wrong<br>
> address. Please send all personal correspondence to the new address.)<br>
><br>
> (PS - If you email me and don't get a quick response, you might want to<br>
> call on the phone. I get about 300 emails per day from alternate energy<br>
> mailing lists and such. I don't always see new email messages very<br>
> quickly.)<br>
><br>
> Ron Frazier<br>
> <a href="tel:770-205-9422" value="+17702059422">770-205-9422</a> (O) Leave a message.<br>
> linuxdude AT <a href="http://techstarship.com" target="_blank">techstarship.com</a><br>
><br>
><br>
> "<a href="mailto:mike@trausch.us">mike@trausch.us</a>" <<a href="mailto:mike@trausch.us">mike@trausch.us</a>> wrote:<br>
><br>
> On 05/11/2012 09:09 PM, Ron Frazier (ALE) wrote:<br>
>> Hi Mike,<br>
>><br>
>> I don't think that discredits the project, I think it's a wise design.<br>
>> Here's why. The most recent version of SpinRite came out in 2004 and has<br>
>> a history going back to 1988. The program designer is planning an<br>
>> update, but it's not out yet. Nevertheless, that version will work on<br>
>> modern drives. I don't know how deeply SpinRite interacts with the bios.<br>
><br>
> If it has the limitations of BIOS, then it is using the BIOS functions<br>
> as published in Ralf Brown's Interrupt List (RBIL), which was (and for<br>
> the real-mode programmer, still is) the definitive source of BIOS<br>
> interrupt interfaces. The BIOS supports only a limited set of<br>
> functionality, which has been extended over the years to cater to larger<br>
> drives and so forth. It supports only limited error handling (for<br>
> example, INT 0x13/00 is "RESET DISK SYSTEM", but that only seeks the<br>
> drives to track 0, it doesn't actually perform a bus reset).<br>
><br>
> The INT 13 interface is horrible, and unsuitable for the implementation<br>
> of any serious software these days. And has been for many years, which<br>
> is why modern operating systems no longer use that interface. Even<br>
> Windows 3.11 had a dedicated driver for bypassing the BIOS routines for<br>
> disk access (they called it "32-bit disk access" or something along<br>
> those lines, IIRC).<br>
><br>
>> I do know it boots from its own copy of freedos and runs from there. The<br>
>> product was designed to work with any PC compatible computer, be it PC,<br>
>> Linux box, or Mac. It can even work with Tivo or iPod drives, etc. if<br>
>> you take the drive out and attach it to a PC. It needs to be able to<br>
><br>
> Well, yes. They all speak the same language, regardless of the type of<br>
> computer they are plugged into. The only thing that is really different<br>
> between disks in a Mac and a PC is the convention used to store data<br>
> (e.g., partition tables or maps and filesystems).<br>
><br>
>> have total control over the drive, including disabling some of the<br>
>> drive's normal error correction, so it can do analysis and detect<br>
>> problems. It can't have the OS in the way and interfering with it's<br>
>> operation. The primary target machine it was designed to run on was no<br>
><br>
> ... but it's perfectly happy having the limited, inconsistently designed<br>
> various BIOS programs get in the way and do its work for it?<br>
><br>
> I'll grant that the Linux generic SCSI interface did not exist when<br>
> SpinRite was first created. I'll grant even further that it was<br>
> impractical in that time period to actually write dedicated drivers for<br>
> the various disk controllers which existed at the time. The reason that<br>
> BIOS existed was to simplify the creation of relatively simple systems<br>
> so that they did not need to know anything more than the generic BIOS<br>
> interface.<br>
><br>
> However, with that generic BIOS interface comes a<br>
> lowest-common-denominator approach to handling the disk controller and<br>
> therefore the disks themselves. This would also be the primary reason<br>
> why it's a horrible idea to use the BIOS interface.<br>
><br>
>> doubt Windows machines. Back in 2004, those machines were running<br>
>> various combinations of Win 95, Win98, Win ME, Win 2000, Win NT, and Win<br>
>> XP. I don't think any of the Windows systems allow the kind of<br>
>> unfettered access to the drive that SpinRite needs. Also, as far as I<br>
><br>
> Windows systems not built on the Windows NT kernel (e.g., Win9x and<br>
> earlier) do allow direct hardware access because at their core they were<br>
> still 16-bit operating systems. Calls to the Win32 API were largely<br>
> thunked to 16-bit modules that were preexisting. There were large<br>
> components of the system that ran in 32-bit mode, but a lot of it did<br>
> not. There was therefore a heavy cost associated with the constant<br>
> changing of the CPU mode as part of context switches and so forth, which<br>
> made Win9x both clunky and relatively unstable.<br>
><br>
> Starting with Windows NT, direct hardware access is prohibited to most<br>
> applications. The exceptions are those that create kernel drivers that<br>
> enable an application to bypass such restrictions. I don't know if NT<br>
> has something like the Linux generic SCSI interface, but if it doesn't<br>
> and it were necessary for an application to be implemented, it would<br>
> certainly be possible to do.<br>
><br>
> Building a utility directly on top of a SCSI interface would be far<br>
> superior to building it on BIOS. Building it to talk directly to<br>
> hardware would be the only way to get closer to the metal than what SCSI<br>
> interfaces would allow, but that's unnecessary for most applications,<br>
> and if it were necessary for an application the preferred way to do it<br>
> would ideally be a 32-bit extended DOS program that performs direct<br>
> hardware access. Of course a real-mode program would also work, but<br>
> there's little to no point to writing real mode code, since it's quite<br>
> easy to get a 32-bit DOS compiler anyway (GCC, for example, supports<br>
> MS-DOS via DJGPP).<br>
><br>
>> know, there isn't a way to dismount an internal drive in Windows and<br>
>> work on it, as you potentially can in Linux. Even if you could dismount<br>
>> a drive, the system needs to run on the system drive, and the average<br>
>> user doesn't have any way to boot a Windows machine without doing so<br>
>> from the system drive. So, the best design choice was to make a product<br>
><br>
> You can absolutely unmount drives in Windows. Use the Disk Manager in<br>
> the Microsoft Management Console for a GUI way to do it. There are of<br>
> course API calls that can be used by custom software to do it as well.<br>
><br>
> You are right in that you cannot unmount the system drive, but that<br>
> problem is common to all operating systems, not just Windows. DOS had a<br>
> sort of exception, but DOS was also small enough to remain completely<br>
> resident in RAM when there was more than 1 MB of memory present and usable.<br>
><br>
>> that booted itself. That way, the OS isn't running, all the drives are<br>
>> dismounted, and he didn't have to wonder whether the user would be able<br>
>> to boot their pc so they could use his software. It was probably the<br>
>> best solution to the problems he had to deal with. On my machines where<br>
>> the bios is new enough to match the hard drive capacity, I can run<br>
><br>
> Most, if not all, modern BIOS firmware provides what are known as the<br>
> INT 13 extensions, which provide a portable interface to address up to<br>
> one or many different sizes. There was a series of progressions in the<br>
> maximum disk size that BIOS could support; 500 MB, 1 GB, 4 or 8 GB, 128<br>
> GB and I don't remember the current one but it is pretty large relative<br>
> to the time period it was introduced in. Maybe 500 or 800 GB or something.<br>
><br>
> Anyway, given a disk that fits within the support of the BIOS, you can<br>
> read and write sectors by identifying the C/H/S (very old API) or LBA<br>
> address of the first sector and the count of sectors. The BIOS will of<br>
> course return an error condition in the CPU's registers if it<br>
> encountered an error while reading one of the sectors.<br>
><br>
> Just as a side note, DOS called upon BIOS to do the work, but<br>
> well-behaved programs that were written for DOS and did not require the<br>
> ability to go beyond DOS' support of storage would simply call DOS<br>
> interrupts to get the job done, which provided a slightly higher-level<br>
> API since you did not need, for example, to worry about sectors but<br>
> instead clusters in named files.<br>
><br>
> SpinRite doesn't need to use any of the DOS filesystem facilities,<br>
> though, and DOS is inherently a single-tasking operating system, so it<br>
> is safe for SpinRite to assume that it can have exclusive control of the<br>
> disk while it is running.<br>
><br>
> Also note that there are DOS implementations that have Windows NT-like<br>
> restrictions on direct hardware access for certain things. Such<br>
> versions are usually ones that provide some sort of task switching or<br>
> multitasking ability, for example taking advantage of the functionality<br>
> and features of the 386 or newer CPUs and providing DOS applications<br>
> with a V86 environment instead of a real one.<br>
><br>
>> SpinRite on both the Windows partitions and the Linux partitions. It<br>
>> doesn't care. It works strictly at the sector level and is non<br>
>> destructive. Even to use the badblocks command as you and Jim have<br>
>> suggested on my old PC, I have to shut it down and boot a foreign OS, ie<br>
>> a live Linux CD, in order to run the test. That's exactly the same thing<br>
>> SpinRite is doing. It just happens to be booting freedos rather than<br>
>> Linux.<br>
><br>
> No, you can do it while the system is running, you just cannot use the<br>
> read-write test mode.<br>
><br>
> The read-write test mode in badblocks is superior because it does not<br>
> depend on the data currently stored in the sector. Certain data<br>
> patterns can hide error conditions which may exist on the platter; for<br>
> example, a particular bit may be stuck "on" or "1", but you'd never know<br>
> that if the value that is there is legitimately 0xFF. But you'd detect<br>
> it if you wrote 0x00 there, and when you read it back it was, for<br>
> example, 0x01 or 0x80, because the stuck bit didn't get cleared.<br>
><br>
> SpinRite does this at the disk sector level (presumably, since it is an<br>
> ancient program, with a fixed value of 512 for the sector size). The<br>
> badblocks command works on blocks, too, but you can specify the size of<br>
> what it considers a block. A common value is 4,096 bytes for a block<br>
> when running badblocks, though virtually any block size that is a<br>
> multiple of 512 will work for older drives; make that a multiple of<br>
> 4,096 for modern so-called "advanced format" drives.<br>
><br>
>> I may run the non destructive rw test on the old pc using badblocks as<br>
>> you and Jim suggested in other messages. It already passed the long<br>
>> smart test and it says the drive is healthy with no bad sectors. I just<br>
>> have to figure out how much additional time I want to spend on it.<br>
><br>
> For a non-critical (e.g., personal) system, SMART should be sufficient.<br>
> You of course take regular (monthly or weekly) backups of your ${HOME},<br>
> right? If that's the case then recovery is possible within hours of new<br>
> drive installation, and for an individual, particularly one who has<br>
> multiple computers, that is an acceptable thing. If not, you can employ<br>
> other steps to try to delay or defer the restoration process, such as<br>
> using RAID, but backups are backups and still quite necessary.<br>
><br>
> I have one array I manage that it would take approx. 30 hours to restore<br>
> from backup. In an attempt to avoid that in all but the most<br>
> devastating situations, I have it on a RAID array. As long as the RAID<br>
> array's health is maintained, it is possible for me to keep taking and<br>
> testing backups and knowing that I can, if need be, recreate the<br>
> configuration as it exists in the office today along with all the data,<br>
> but I don't have to if only one or two drives fail, because I can<br>
> replace them almost immediately upon failure. I'm a bit on the paranoid<br>
> side, too, I will start replacing drives as soon as they stop behaving<br>
> 100% perfectly as opposed to wait for a hard failure.<br>
><br>
> So far that seems to have been a good way to ensure that things stay<br>
> running... the drives I've removed were all used again in my home<br>
> (after, of course, being wiped) and failed within a month of removal<br>
> from the array. One of them failed only two days after removal. Even<br>
> more than smart, the Linux kernel ring buffer is a great source to<br>
> monitor for disk trouble, especially on a system that has many disks<br>
> with lots of activity on them. The kernel will notice errors as soon as<br>
> they are encountered.<br>
><br>
> Additionally, RAID devices get "scrubbed" once per month anyway (at<br>
> least by default on Debian and Ubuntu). The "scrub" process is<br>
> *exactly* what SpinRite does, reading everything on all the disks. It<br>
> doesn't re-write every sector, but it doesn't need to: if a sector<br>
> cannot be read it is reassembled from parity information and an attempt<br>
> is made to re-write it. If the drive was able to re-map the sector, the<br>
> write will work and things continue. If the drive was unable to re-map<br>
> the sector (say, because it ran out of sectors in the spare sector area)<br>
> then the write will most likely fail and the disk will be marked<br>
> "failed" by the virtual RAID controller.<br>
><br>
> That's robust enough for me, at least with the requirements I have in<br>
> the environments that I am managing for the moment. I would like,<br>
> however, to have a beefier system for the RAID. Not because of the CPU,<br>
> but because it would really do well to have a veritable buttload of RAM.<br>
> A lot of the operations would be faster if the system had 4 GB of RAM<br>
> to use for caching and buffers...<br>
><br>
> --- Mike<br>
><br>
> --<br>
> A man who reasons deliberately, manages it better after studying Logic<br>
> than he could before, if he is sincere about it and has common sense.<br>
> --- Carveth Read, “Logic”<br>
><br>
> _____________________________________________<br>
><br>
> Ale mailing list<br>
> <a href="mailto:Ale@ale.org">Ale@ale.org</a><br>
> <a href="http://mail.ale.org/mailman/listinfo/ale" target="_blank">http://mail.ale.org/mailman/listinfo/ale</a><br>
> See JOBS, ANNOUNCE and SCHOOLS lists at<br>
> <a href="http://mail.ale.org/mailman/listinfo" target="_blank">http://mail.ale.org/mailman/listinfo</a><br>
><br>
><br>
<br>
<br>
--<br>
<br>
Justin Goldberg<br>
<br>
*<a href="mailto:justgold79@gmail.com">justgold79@gmail.com</a>*<br>
<a href="tel:%28504%29%20208-1158" value="+15042081158">(504) 208-1158</a><br>
<a href="http://gplus.to/goldberg" target="_blank">http://gplus.to/goldberg</a><br>
<a href="http://twitter.com/justingoldberg" target="_blank">http://twitter.com/justingoldberg</a><br>
<br>
_______________________________________________<br>
Ale mailing list<br>
<a href="mailto:Ale@ale.org">Ale@ale.org</a><br>
<a href="http://mail.ale.org/mailman/listinfo/ale" target="_blank">http://mail.ale.org/mailman/listinfo/ale</a><br>
See JOBS, ANNOUNCE and SCHOOLS lists at<br>
<a href="http://mail.ale.org/mailman/listinfo" target="_blank">http://mail.ale.org/mailman/listinfo</a><br>
</blockquote></div>