More Megapixels per Megabyte

<p dir="ltr"><a href="http://www.rawzor.com/">http://www.rawzor.com/</a></p>

<p dir="ltr"> </p>

<p dir="ltr"> </p>

<p dir="ltr"> </p>

<p dir="ltr">Looks very interesting.</p>

<p dir="ltr"> </p>

<p dir="ltr"> </p>

<p dir="ltr"> </p>

<p dir="ltr">Happy shooting,</p>

<p dir="ltr">Yakim.</p>

<p>.</p>

<p>Have you used it? Are the files immediately compatible, or do they require uncompressing before re-use? Hard drives are so cheap, I hardly see the need to spend my time to make files smaller, then larger, then smaller, then larger ... just to actually use them. Can Picasa still load them on a hard drive search? Please share some more information and user experience, thanks!</p>

<p>,</p>

<p>Computers with 2-8 cores have so much oomph that real time pack-repack isn't a problem.<br>

Then again, Rawzor doesn't mention anything like that. Looks like a normal archiver app. And version 0.4b really isn't something I'd feel comfortable with packing all my images.</p>

<p>What a complete waste of time. I can't see it being much of a success. You may as well use Winrar unless there's something startling about this new software. As already mentioned, hard drive space is so cheap and it will only get cheaper.</p>

<p>This might not be a waste. RAW images are probably not that easy to compress, and optimising your compression routines to RAW images could yield real gains.</p>

<p>As someone with 50,000 images in a couple of Terabyte drives (and now in need of more Terabyte drives), this would be of some interest to me if the process is transparent. Terabytes are not all that cheap. Every time I come back from a shoot it's 20 GB (1Ds MkIII), so I use up the space quickly.</p>

<p>But it's Windows only - so, no cigar.</p>

<p>Michael</p>

<p>They are working on a filesystem driver so that compression/uncompress would be transparent and programs would not have to be modified to read the format.</p>

<p>It is not Windows only, there is a Linux shared lib on the web site and it says Mac support is coming.</p>

<p>Of course hard drives get cheaper and larger but RAW files also get larger from more MP, 12-bit to 14-bit, etc. </p>

<p>If you don't think this software is useful nobody is making you use it.</p>

<p>.</p>

<p>Space wise, I find the slack after the end of the file is more of a waste than any standard compression algorithm recovers. I use Windows NTFS compression and find it sucks and does little of benefit. Old Stacker would create a zip-based operating system where the end of one file was immediately occupied by the beginning of the next file -- no slack in Stack! THAT is were future (and past) space-saving is hiding. So, when we Zip-compress entire directories, we may not gain much per-file because the files are already dense, but we gain more because there is no more wasted space in slack between files if all files are in one Zip archive per directory. So, I'm not looking for another zipper, I'm looking for a slack-remover enhancement to the operating system, and if Microsoft doesn't come up with one, we're screwed, and we will fill our 2 terabyte drives with only 1.5 terabytes or less of data.</p>

<p>My Raw files are already smaller than any other lossless image file format. I'm curious what problem Rawzor thinks thay are addressing? I imagine the goal is to mature the technology then sell it to either Microsoft or Apple or Nikon or some other one-sale profit, or multiple sales the way Apical sells it image processing chips to each camera maker wanting D-Lighting/DRO/and so on. Like Sysinternals, perhaps Sachin is looking for a career within one of those companies, or is looking to build a suite of products of similar focus?</p>

 

<blockquote>

<p>http://www.rawzor.com/<br>

Sachin Garg<br>

B-442, Meera Bagh<br>

Paschim Vihar, Outer Ring Road<br>

New Delhi, New Delhi 110063 India</p>

</blockquote>

<p>.</p>

<p>Peter, what you are talking about is a problem due to the block size of the filesystem. Assuming 4K blocks and a 5K file you use up 2 full blocks. Another file that is just 1 byte would also use an entire 4K block. If you want to store data more efficiently you need a filesystem with block suballocation or tail packing. The extra sub-blocks or "tails" of different files like the 1K extra in my example would be stored together. I used to use ReiserFS on Linux which supported this. I think the disk space used was about 10% less for the type of data I had. More at wikipedia here: http://en.wikipedia.org/wiki/Block_suballocation</p>

<p>As for what the Rawzor creator is trying to do, he says the software saves 20-60%. You're either interested or not. Since he hasn't open sourced his software I'm assuming that he plans to try and sell it to some other software companies. He does have a forum on his web site so if you're really interested then ask there.</p>

<p>

<p>Peter, I have not tried it myself but a friend of mine is very enthusiastic about it. He passed it to me and I thought I'd share it.</p>

<p>Happy shooting,<br>

Yakim.</p>

</p>

<p>As a REAL TEST I took a directory that has 8 raw files and the directory is 77.1 megabytes total<br>

<br /> (1) WINRAR . Then I compressed these to one BIG file using winrar and the directory is now 53.0 megabytes for the one file; this took 252 seconds; this made one file of 53 megs<br>

<br /> (2) PK ZIP . Then I compressed the raw files with old PK ZIP for Windows 4.00 and it makes one BIG file of 63.2 megabytes; 19 seconds time; this made one zipped file of 63.2 megs<br>

<br /> (3)RAWZOR .Then I tried Rawzor 0.41 with the same 8 files and one has 37.1 megabytes with the 8 individual files; 56 seconds time</p><p>****All this was done on an old computer at home that has many versions of Photoshop and Illustrator; a Pentium III with a 1 Ghz CPU and 1 gig of ram; bus is a 133Mhz speed. During the three compression runs the CPU is at 100 percent; used ram is zilch say 30 to 50 megs. Thus a conversion probably speed probably varies with CPU speed.<br>

****A modern computer would probably convert these is say 1/3 to 1/4 the time.</p><p>****During the test there was 14 firefox windows open; I am downloading some stuff; wordpad; PK zip and winrar are open; plus task manager; plus Photoshop CS2 ;plus Bridge; plus OSLO. Unit has windows 2000.</p><p>The 8 converted raw files are from a Epson RD-1/S and are this type roof image BELOW and variants. The compression of Rawzor of a raw file varies with the images content; the 8 raw files are each 9.60 to 9.65 megs; the compressed created new 8 Rawzor files vary between 4.28 to 5.01 megs; imges with more details and tones make the bigger Rawzor files.</p><p><img src="http://i4.photobucket.com/albums/y148/ektar/15mm%20VC%20lens/_EPS7467roofone250secondiso200F8.jpg?t=1240148216" alt="" width="640" height="426" /></p><p>cropped section of above; 15mm VC lens was used; raw allows reducing the vigneting.</p><p><img src="http://i4.photobucket.com/albums/y148/ektar/15mm%20VC%20lens/_EPS7467roofone250secondiso200F8CRO.jpg?t=1240148377" alt="" /></p><p>Old Piii box has 6 versions of full bore Photoshop; 3 versions of Elements; 2 versions of Illustrator; plus Photostyler 2. The disc space *consumed* by Photoshop 3 equals one raw file off a Epson RD-1. Version 3 will open rare obscure variants of files that will not open with newer Photoshop.<br>

<br /> <img src="http://i4.photobucket.com/albums/y148/ektar/PNdesktop/desktopcrop14.gif?t=1240149085" alt="" width="299" height="600" /></p>

<p>RAWZOR compressed the group of 8 images ie 77.1 megs to 37.1 megs in 56 seconds<br /> <br /> WINRAR compressed the group of 8 images ie 77.1 megs to 53 megs in 252 seconds.<br /> <br /> RAWZOR might be usefull here where I want to email a buddy a 9.65meg raw file to his 10 meg box; ie rawzor compresses it to 4.60 megs; winrar makes it 6.42 megs. A super compressed file thats 7 megs can breach a 10 meg limit; when one adds headers; overhead; error correction; and/or when folks add comments; or reforward it again and again :)<br /> <br /> As far as storage here I am not going to use a new program to store images.<br /> <br /> The RAWZOR program looks and worked well and is worth experimenting with<br /> Later I will try RAWZOR on a dual core and dual CPU boxes to see if the extra core or cpus is idle during a conversion (boar hog knockers); or active. Many programs really do not use a second core of cpu; and many do too.</p>

<p>To EXTRACT the 8 files back to raw with RAWZOR took 32 seconds</p>

<p>To EXTRACT the 8 RAWZOR files (37.1 megs total) back to raw with RAWZOR took 32 seconds<br>

To EXTRACT the BIG rar-'ed 53 megs file back to the 8 raw files took 12 seconds<br>

To EXTRACT the BIG zip-'ed 63 megs file back to the 8 raw files took 8 seconds</p>

 

<p>.</p>

<p>PKZip has much more aggressive compression options than default, so I suggest comparing best to best. Windows OS has native ability to "see" into zip files, so that's a plus, though few applications or accessories I've used can "see" inside zips, as well as the thumbnail feature of the Windows OS seems to NOT function within zip files, even though the contents are displayable in Windows Explorer.</p>

<p>Also, I often use ancient (Win 95) InSo (Inside Out) QuickView Plus v5 (distributed by WordPerfect) to open and expand UFO zip files, by the way, when other programs fail (it also uniquely shows me inside MSOffice2007 XML file contents, allowing me to expand the document inside -- an example of a 1995 program that's well-written working over a 2007 document -- cool!).</p>

<p>See the PKZip v2.5 for DOS command line reference:<br /> <a href="http://www.uwm.edu/~dtoth/pkzip.html">http://www.uwm.edu/~dtoth/pkzip.html</a></p>

<table border="1" width="600" align="center" bordercolor="#808080">

<tbody>

<tr>

<td width="26%" valign="top"><strong>level </strong>

<p>set the level of compression <br /> configurable</p>

</td>

<td width="25%" valign="top">any digit from 0 through 9, with 0 being no compression at the fastest speed, and 9 being the most compression at the slowest speed

<p>----------------------------- <br /> default = level 5 (normal)</p>

</td>

<td width="33%" valign="top">pkzip25 -add -lev=9 save.zip *.doc</td>

<td width="16%" valign="top">

<p>add</p>

</td>

</tr>

</tbody>

</table>

<table border="1" width="600" align="center" bordercolor="#808080">

<tbody>

<tr>

<td width="26%" valign="top"><strong>maximum </strong>

<p>set the level of compression to the highest level, but lowest speed <br /> configurable</p>

</td>

<td width="25%" valign="top">no suboptions

<p>----------------------------- <br /> no default value</p>

</td>

<td width="33%" valign="top">pkzip25 -add -max save.zip *.doc

<p>pkzip25 -config -max</p>

</td>

<td width="16%" valign="top">add</td>

</tr>

</tbody>

</table>

<p>See also PKZip for Windows:<br /> <a href="http://download.pkzip.com/pub/pkware/products/win/pkz90010.exe ">http://download.pkzip.com/pub/pkware/products/win/pkz90010.exe </a> <br /> PKZIP for Windows Command Line Interface:<br /> <a href="http://pkware.cachefly.net/products/pkzip/win/pkzc84007en.exe">http://pkware.cachefly.net/products/pkzip/win/pkzc84007en.exe </a></p>

<p>If we could only get applications to look inside Zips and act on their individual contests as they do on individual files, THAT would be great. Then we could zip and compress our image directories and drives to eliminate slack, yet still be able to act on individual "files". This is an OS problem, and with Microsoft averse to enhance their OS savvy, and with the price of hard drive's dollars- per-megabyte forever falling, I do not see this ever happening, sadly. How do I know? Because Stac Electronics's Stacker, which did all this, came and went and was quashed by the Microsoft monopoly juggernaut:</p>

<ul>

<li><a href="http://vaxxine.com/lawyers/articles/stac.html">http://vaxxine.com/lawyers/articles/stac.html</a> </li>

</ul>

<ul>

<li><a href="http://support.microsoft.com/kb/71787">http://support.microsoft.com/kb/71787</a> </li>

</ul>

<ul>

<li>MS-DOS - History ... Added DoubleSpace disk <em>compression</em> (copied from <em>Stacker</em> ) and other features <strong>...</strong> 1994 - Following <em>Stac</em> lawsuit, removed DoubleSpace disk <em>compression</em> ... <a href="http://www.spiritus-temporis.com/ms-dos/history.html">http://www.spiritus-temporis.com/ms-dos/history.html</a> </li>

</ul>

<p>Oh well, we get what we pay for, and if we don't pay to prosecute and break up Microsoft's monopoly, then we get to complain forever about their lack of <em>customer service innovation</em> (what's <em>that</em> ?) in public commerce and in the public marketplace. Drat!<br /> .</p>

<p>PKZIP compressed the 8 raw files to 63.2 megabytes in 27 seconds with the slider set to 9 ie maximum</p>

<p>PKZIP compressed the 8 raw files to 66.7 megabytes in 17 seconds with the slider set to 1 ie SUPERFAST</p>

<p>PKZIP compressed the 8 raw files to 63.1 megabytes in 19 seconds with the slider set to 5 ie NORMAL</p>

<p>Thus in this real world test moving changing setting the compression level from NORMAL -5 to MAXIMUM-9 saved one 0.1 megabytes; and it took 8 seconds longer. ie a group of 77.1 meg files was compressed to either 63.1 or 63.2 megs; in the noise as a betterment.</p>

<p>changing the settings to from (stock/default) *deflate* to *deflate64* and setting the compression to maximum -9 gives a file of 62.9 megs in 42 seconds; here we saved abit of a floppy.:)</p>

<p>changing the settings to from (stock/default) *deflate* to *DCL implode* and setting the compression to 1024 ascii gives a file of 77.1 megs in 28 seconds; no compression.</p>

<p>Thus the default PK ZIP settings gave results within 0.15 percent of the fiddled with compression settings ; and got the job done quicker by 30 percent less time.</p>

<p>.</p>

<p>Thanks for doing our work and testing for us, Kelly. ;-) </p>

<p>I guess some Raw image files are pretty dense or compressed already (what Raw are you using, by the way? and 10, 12, or 14-bit? Leica offers square-root compression, does anyone wanna test them?), so the only advantage to PKZip is native Windows compatibility. Rawzor must be doing something intelligent, Raw image contents wise. Does anyone have Sigma/Foveon Raw non-Bayer RGB to compare -- maybe Rawzor is biased towards intelligent patterning within standard RGGB Bayer sensor arrays?</p>

<p>By the way, specific timing and file sizes are useless and means I have to compare your results to each other. Percentages, however, are useful to any reader ... but still we need to know more about your specific source files, system, and system resources.</p>

<p>I've just got three 500 GB PATA drives in my older system, and three 1.5 GB SATA drives in my newer system -- so of course they're announcing 2 GB SATA! At least they're incredibly cheap. To some extent, unless Rawzor sells their technology to an operating system vendor or a camera maker (lossless compression will be VERY important for the next generation of 34 million pixel full frame DSLR images, by the way, just to get write rates quick from camera buffer to storage card, Sony are you listening?), Rawzor will be a no-go for me. I absolutely depend on my files being readable anywhere, and standard native format and PKZip do that, Rawzor does not ... yet. I even am tempted to use FAT32 formatting rather than NTFS just so the drives can be read in any system I swap them into in an emergency, even my good ol', still working, Windows 98SE and ME systems. =8^o NTFS is such a bear, though Linix/Unix utilities are learning to make them addressable ...</p>

<p>Thanks for the thought provoking tease, Yakim.</p>

<p>.</p>

<p>

<p>Always glad to serve the community.</p>

<p>Happy shooting,<br>

Yakim.</p>

</p>