Win7 how to prevent local files from being null-corrupted after a BSOD

[MtheK]

  Split off from here:

http://masm32.com/board/index.php?topic=4229.0

-----

Win7 how to prevent local files from being null-corrupted after a BSOD

 
  Well, I declare my "commit" program a success (probably).  Surely
there's gotta be something somewhere that does this?! Some DOS cmd?
FSUTIL (ADMIN or not)? What's the PC word for "commit"; "resource"?
"flush"? "transactional"? "atomicity"? And don't call me Shirley.

  I concentrated on 3 .BAT output files:
. the simple .BAT ECHO
. 2 DSNTODAY .BATs: LINKWTS ECHO ("calling...") and DSNTODAY *AB

  Only the DSNTODAY *AB file was left "un-committed" (un-TOUCHed).

  In about 40 manual BSODs (56 total so far), the 2 "committed" files
both survived intact, every time, with no null-corruption, kinda
verifying my TOUCH program is flushing the .BAT O/P files correctly.

  For the "un-committed" file, it seems that if I wait apx 12sec
(all human-time used in these tests based on the taskbar clock;
probably +- 1sec), I can MOSTLY get it to end up null-corrupted,
but not always. A dozen +5sec runs all got null-corrupted, while
a dozen +30sec runs MOSTLY survived, yet a few runs still got
null-corrupted, seemingly ruling out that LanMan registry setting
(which I don't have specified).

  Apparently, there are no tuning parameters for Cache Manager except
the 'Policies' tab for a drive, which doesn't work for my SATA boot drive
(response is apx 5x (.dmp, logon) but files still get null-corrupted)? 
With all the many articles I've found discussing all this (I like the
Win95 "bug", the lazy writer, a neat tester), I'm really surprised there
isn't a command/program built into the OS for the normal files to
guarantee data integrity (ie: do a flush), unless it's buried so deep
somewhere that I just can't find it. Oh well, at least now I can control
it myself for my own files.

---

  In summary, to prevent null-corruption of local files after a BSOD,
do one of these before the BSOD:

. open a file with FILE_FLAG_NO_BUFFERING
in my case, I write a single 4K (page) block, page-aligned in memory
(I use my $ALIGN just before .DATA? and let the OS deal w/the allocation)

. add an 'INVOKE FlushFileBuffers' before the close, and/or periodically

. write a separate program that opens a file for output and does
'INVOKE FlushFileBuffers' before the close, then run that whereever
you want a file "committed" after: any program ends, any .BAT command
that writes to a file (ie: say, an ECHO for a log), etc.

  The tradeoff is performance. In my case, I see a 3-6% increase in
wait I/O across my 2 machines in my "heartbeat" file which
opens/clears/writes/(and now flushes)/closes every second.
Specifically:

Code Select Expand



2015/05/26 14:34:22.801 ^211/000499,024/000499 94 0000499 017/000500;0059h+++
2015/05/26 14:35:22.799 ^212/000499,024/000499 94 0000499 017/000500;0060h+++




A lot of Mxyzptlk, but basically:
. wait I/O (ticks) took 2.12sec (avg over 1 hour) per minute
. peak wait I/O per heartbeat is .17sec, tho I've seen peaks over 1sec before.

[Magnum]

I would think that Win 7 would be immune to most BSODs.

How do they occur ?

[sinsi]

One problem is that the drive itself will also have a buffer, when Windows tells it to flush to disk it pretends to but doesn't.
http://blogs.msdn.com/b/oldnewthing/archive/2010/09/09/10059575.aspx

[Magnum]

Thanks sinsi.

I will not try to encourage you to try Linux.

:-)

[jj2007]

Raymond Chen on TechNet:

Windows 3.11 introduced "32-bit file access," which was a 32-bit implementation of the low-level file I/O interface. But the implementation of the Commit function contained a bug that effectively ignored requests to flush file buffers. If you took a program that flushed its file buffers and ran it on Windows 3.11, the flush call had no effect. As a result, if you lost power at just the wrong time, you ended up with a corrupted database.
The folks working on the Windows 95 file system fixed this bug, but new bug reports started to trickle in. Somebody's accounts payable program started running really slowly. Then somebody else's database program did the same. What was going on?
It turned out that these programs constantly issued flush calls. The programmers noticed that flush calls were really fast on Windows 3.11, so they sprinkled them liberally throughout their program. Write a byte, flush. Write a string, flush. Since the flushes were so fast, the app could commit the data to disk after every operation with no noticeable performance degradation. But once Windows 95 fixed the bug, these programs started to run very slowly since the Commit calls were suddenly doing actual work.

[rrr314159]

I will not try to encourage you to try Linux.

-You don't have to, Windows itself encourages us to try Linux

[sinsi]

Thanks sinsi.

I will not try to encourage you to try Linux.

:-)

I try linux every year or so but it just doesn't interest me because I'm mostly a gamer/programmer from the DOS 3.2 days (and CP/M on a Z80 before that).
All I usually do with linux is browse the internet, might as well use my mobile for that 

Funnily enough, my boot CD of utilities uses linux.

[rrr314159]

CP/M - Z80 - WordStar - Turbo Pascal, GWBasic, rudimentary assembler, debugger: as fast as today's environments, and more fun, with a chip clocked a thousand times slower! those were the days. Plus u could knock out a burglar with dual 8-inch floppy drives (282K!), try that with your 2-Tera flash drive

[MtheK]

  Addendums:

. some articles suggest that 'INVOKE FlushFileBuffers' is necessary even
when FILE_FLAG_NO_BUFFERING is used, to flush the metadata also.

. for the 'opens a file for output', I open it exclusive; I use the most
to least aggressive strategy, using the 1st handle (of 3) that opens OK.

  The DSNTODAY system now can protect its' .bat ECHO logs (my "commit" "command" I was looking for) with its' STDOUT.bat w/the
# prefix. A few dozen more manual BSODs (77 total), waiting +5, +12, and +30sec, and everything I "committed" (via program or .bat) now all survive
intact. So even if the SATA disk IS lying to the OS about flushing (the neat tester):

http://brad.livejournal.com/2094221.html

at least, using these methods, it appears to honor them. In fairness to the PC,
I had a similar problem in the mainframe quite awhile back in our test DRP
(Disaster Recovery Plan). A header record appeared before the data records
in an online journal pair, which, per IBM, s/not be possible since the header has
a count of the number of data records written first. It appeared that the mirroring hardware, which was using "fancy" newer tech (RAID?), was at fault.   


  Thankx to all who commented on this important (to me) problem.

[Magnum]

I will not try to encourage you to try Linux.

-You don't have to, Windows itself encourages us to try Linux

You are good. :-)

[hutch--]

 

Ha ha, there is a place for Linux, use it as your web server but do not delude yourself about it being a desktop operating system. There is a good reason why a pig like Windows is still the dominant PC operating system, Linux is a flop as a desktop OS and it has to do with the assumptions behind it. BASH (yo ho ho), COMMAND.COM was more user friendly, file permissions in OCTAL (U gotta bee kidding !), SUDO, (how to prevent anyone from learning the command line interface) and a collection of incompatible GUI interfaces that hilite just how good the Windows interface is in comparison.

The Linux assault on the desktop domain is effectively "fantasies in DISTRO land". It is unfortunate because behind the antiquated 1960s command line interface, there is a lot of grunt, security options, the best web server on the market and a host of other high powered components but it is crippled by the appalling interface and a lack of willingness to modernise it into something that a human being could learn and use.

Long ago on IRC there was high mileage of an application called BITCHX which had a slogan "Keep it to yourself". Sad to say Linux has succeeded here and ensured that Windows will remain the dominant OS for many years to come.

[rrr314159]

@MtheK,

Thankx to all who commented on this important (to me) problem.

- congratulations MtheK. I sense u feel a lack of support from many of us, but that's not the case. I've followed your travails with interest but couldn't help, it's a tough problem I know nothing about. U stuck to it and nailed it :t

@Andy, thankx!

@hutch,

Matter of opinion, but can't say you're wrong. "Linux itself encourages us to stick with Windows". But I look forward to some new alternative, maybe coming out of China or someplace, to blow them both away ... Android? ARM? dunno. Sooner the better

[Magnum]

Ha ha, there is a place for Linux, use it as your web server but do not delude yourself about it being a desktop operating system. There is a good reason why a pig like Windows is still the dominant PC operating system, Linux is a flop as a desktop OS and it has to do with the assumptions behind it. BASH (yo ho ho), COMMAND.COM was more user friendly, file permissions in OCTAL (U gotta bee kidding !), SUDO, (how to prevent anyone from learning the command line interface) and a collection of incompatible GUI interfaces that hilite just how good the Windows interface is in comparison.

The Linux assault on the desktop domain is effectively "fantasies in DISTRO land". It is unfortunate because behind the antiquated 1960s command line interface, there is a lot of grunt, security options, the best web server on the market and a host of other high powered components but it is crippled by the appalling interface and a lack of willingness to modernise it into something that a human being could learn and use.

Long ago on IRC there was high mileage of an application called BITCHX which had a slogan "Keep it to yourself". Sad to say Linux has succeeded here and ensured that Windows will remain the dominant OS for many years to come.

Hutch,

Windows has been losing market share for many years.

Many other countries and general users have gone to Linux because it is a superior Operating System.

You have free will and can stay in your "fantasy land."

Your posts have indicated that you have very limited experience with Linux.

I have no animosity against you.

[hutch--]

 

> Your posts have indicated that you have very limited experience with Linux.

Yeah, this inexperience comes from installing Linux on the web server, setting up this forum and the old one on it and endlessly maintaining it to avoid the near endless supply of security holes and other related hacks.  :P

[Magnum]

If you will give me more details, I will do some research.

I have many friends in Linux forums that will gladly help.