Memory Cards Protocol Query .....

[matt_t_butler]

It is 'good practice' not to completely fill up a computer hard drive to its maximum capacity.

The suggested practice is to leave at least 10% to 20% of the capacity of the card empty of data

so as not to affect the speed of data retrieval and usage.

Is this the case for both HDDs and SSDs?

Does the same apply to camera memory cards?

[steve_gallimore1]

Does the same apply to camera memory cards?

No, cameras write sequentially, from the beginning of the card to the end.

 

The 10-20% rule is for computers with operating systems that need space for temporary files, defragmentation and such.

 

(It's also, arguably, rubbish, and highly operating system dependent)

[matt_t_butler]

[uSER=8475649]@steve_gallimore|1[/uSER] ... Thanks!

[Jochen_S]

Memory cards might generate an unwriteable last image or burst in a wrong moment, otherwise fill them up.

(System)SSDs might benefit from a bit of vacant space, to distribute the read and write cycles evenly between their clusters? But I 'd fill an SSD that is only meant to sit around, storing images.

[rodeo_joe1]

Hmmm. Not so sure. I recently almost filled the removable 256gig micro-SD card in my phone (with about 5 gig to spare) and the read/write speeds became slower to the point of almost freezing the phone. There was still plenty of internal storage (>100GB) for the OS to work with.

 

Transferring those surplus files to a backup HDD restored the read/write speed to full usability again.

 

Not sure if the issue was fragmentation, with the card 'hunting' for spare storage space and file fragments, or what. But lesson learned; those bulky videos will get offloaded before the card gets near-full in future.

[matt_t_butler]

I have had occasional issues when I saw the 'CARD FULL' message, where I could not immediately view picture preview files on the camera's LCD.

Turning the camera OFF and re-inserting the card solved the problem. Thanks again for the answers.

[glen_h]

It is 'good practice' not to completely fill up a computer hard drive to its maximum capacity.

The suggested practice is to leave at least 10% to 20% of the capacity of the card empty of data

so as not to affect the speed of data retrieval and usage.

Is this the case for both HDDs and SSDs?

Does the same apply to camera memory cards?

 

Unix, and unix-like systems, set the "full" value to 90% of capacity, except for root user.

 

That mostly allows files to be in more contiguous disk blocks, especially as files are

written and deleted, and new files written. And for magnetic disks, contiguous blocks

are read faster.

 

As well as I know, systems haven't changed the way they write file systems on SSD,

though internally SSD usually have some system to distribute writes, such that the

same blocks aren't written many times. (This is especially true for systems with a

virtual memory swap file, which can be written over very often.)

 

Since SSD don't have seek and latency delays, those reasons shouldn't apply.

[Ed_Ingold]

Data does not have to be stored in contiguous blocks, but are faster if they are. That's why it's important to reformat cards in the camera rather than deleting files, which causes fragmentation (a mix of filled and open blocks). The camera will store new files end to end on a reformatted disk or memory card, if possible, but break the file into smaller chunks if the disk is fragmented, and store the pieces where space is available. If the spaces aren't large enough, the disk may register insufficient space, even if overall it isn't completely filled.

[matt_t_butler]

"That's why it's important to reformat cards in the camera rather than deleting files, which causes fragmentation (a mix of filled and open blocks)."

Does this also apply to deleting single unwanted photos in camera folders using the camera's own ERASE or DELETE function?

Or rather deleting some files when an external card reader is connected to a computer before reinserting the same card back into the camera?

[glen_h]

Fragmentation is important for moving head (magnetic) disk drives,

as it requires more seeks to read all of a file.

 

Flash memory is a little complicated, but seek time is one that it

doesn't have. The main complication that flash has, is a limited

number of write cycles. Mostly that isn't a problem, except for files

that get written a very large number of times. For most systems, that

is only virtual memory swap files.

 

More writes increases the probability of a bad block, where the data

stored isn't what it is supposed to be. Systems have a way to spread

writes around, to reduce problems. It seems that the number is

about 10,000 for some older ones, up to 35,000 to 100,000 or

even 1,000,000 for newer ones.

 

One other effect of fragmenting is that it makes recovery of data

accidentally erased, or otherwise needing to be recovered,

harder.

[Tony Parsons]

Must admit that my standard practice is never to completely fill an SD card, just in case. With a 64 GB card, that usually means between 1600 and 1800 RAW images before it is retired and replaced. I NEVER re-use SD cards - they are so cheap I prefer to use a new one each time, and it also means that I have the original images (as well as a working copy and backups) - with 35mm film negatives they were always kept and filed, after all.

[q.g._de_bakker]

Must admit that my standard practice is never to completely fill an SD card, just in case. With a 64 GB card, that usually means between 1600 and 1800 RAW images before it is retired and replaced. I NEVER re-use SD cards - they are so cheap I prefer to use a new one each time, and it also means that I have the original images (as well as a working copy and backups) - with 35mm film negatives they were always kept and filed, after all.

Is there a good reason to keep your files on the memory card you first recorded them on? I don't know of any, really.

There is no analogy with negatives, since you can't have exact copies, let alone multiple exact copies. And you can't transfer a negative from the film it is on to another medium either. With digital files, you can. On a much cheaper medium too. Multiple exact copies. Negatives must be processed and stored well, and will then keep for a very long time. Memory cardshave a limited life span, and you can't tell by looking at them that they are no longer what they are supposed to be.

So why would you keep them on expensive memory card, and never reuse those cards?

[glen_h]

(snip)

 

So why would you keep them on expensive memory card, and never reuse those cards?

 

A 64GB SDXC card costs less than one roll of film, not including

processing, and as noted holds over 1600 raw images.

 

I mostly shoot JPEG, though at the highest quality mode for the camera, so maybe 6MB each.

So maybe 10,000 picture for a 64GB card. (Though I more often use smaller cards.)

 

You can get 1TB SDXC for about $26, maybe a roll of Ektachrome + processing.

 

Now, if it is some brand new card type, not readily available yet, then they

might be more expensive. But the quote mentions SD. (Likely SDXC.)

 

When I was young, the best data storage was 40 MB on a 12 inch reel

of magnetic tape. I had a few reels for some of my files.

[glen_h]

(snip)

 

There is no analogy with negatives, since you can't have exact copies, let alone multiple exact copies. And you can't transfer a negative from the film it is on to another medium either.

(snip)

 

This reminds me that in the early days of photography, with glass plate negatives,

it was usual to reuse plates. Many negatives that probably should have been

kept, were instead reused.

 

In the case of digital data, there is a (hopefully small) chance of errors in any copy operations.

More likely it is a defect in one card or the other.

 

As well as I know, though, a bigger problem is losing cards. Having a good file method for

storing them probably helps. Not requiring time to copy, and for verifying that the right

things were copied, saves time.

[q.g._de_bakker]

The price per MB of memory cards is high. Ye olde spinning discs are much cheaper (allows for redundant storage) and hold much more.

 

The time it takes to copy from card to other medium is short, and not time you need to spend watching and waiting.

 

The chance of errors during copying is real, small and taken care of by just about all software you can use to copy files.

 

And unless you accept the bigger risk of corrupt or lost memory cards, you will have to copy anyway.

[Tony Parsons]

I have never stated that I do not copy the cards. As I stated, I keep the cards, creating one working copy of each day's shooting on my data drive as soon as I arrive home, and have two further drives, to which I backup weekly the original images and those that have been processed. This is the way which suits me - others may wish to do differently, up to them. I can recall friends when having colour negative film processed, and being supplied with en-prints, or 6x4 enlargements, throwing away the negs - then wanting further copies, or having the original prints damaged, lost or destroyed, and wishing they had kept the negatives.

 

Furthermore, although I have seen references to 'Memory cards have a limited life span', I cannot recall this ever being backed up with data showing the projected life span of an SD or SDXC card, or even anecdotal evidence of this phenomenon occurring. Perhaps someone can point me in the direction of any such published information, please ? And yes, my completed SD cards are kept filed in order, along with DVD copies of them. Current price of 64GB SDXC card on Ebay - £7.99. Hardly likely to break the bank, even using three per year. And, should it so happen that a memory card becomes corrupted, software is available that will recover data. I do not have any evidence to back up my views, but I feel it is more likely that a card will become corrupted through frequent formatting and re-use than in my own system of 'Write once. read once'.

 

Finally, may I enquire why it is so important to you that I relinquish a system that has worked well for me since the days of my Pentax K10D (whose cards are still able to be read, by the way) to adopt a system which to me is less secure, and will save me less than the price of a pizza per year ? Thank you for your suggestions - they have been securely filed.

[Tony Parsons]

I have stated my point - my contribution to this thread is now ended.

[q.g._de_bakker]

[...]

 

Finally, may I enquire why it is so important to you that I relinquish a system that has worked well for me since the days of my Pentax K10D (whose cards are still able to be read, by the way) to adopt a system which to me is less secure, and will save me less than the price of a pizza per year ? Thank you for your suggestions - they have been securely filed.

 

The question i asked was what reason there would be (besides personal preference, sure) to keep files on memory cards.

 

The emphasis you put on NEVER certainly suggests that there is some urgency to at least not reusing them. And yes, increasing the number of write cycles will not help keep a card good. But is it that problem that BIG that we should take note of your NEVER?

You raised the question. Why is it so important to you that we take note of your habits?

 

And i'll contest again that keeping them on cards is more secure (re your "less secure"). It certainly is not, so no good reason.

[glen_h]

There is much written on the number of write cycles for flash memory, with numbers around 10,000, and more recent ones maybe 100,000.

 

That is mostly a problem when they are used for virtual memory swap space. The MacBook Air that I write this on has flash for its

internal storage, but I mostly keep data on an NFS mounted network disk drive (at home).

 

There is less written about data retention time when not being accessed.

 

Storage is based on charge on a SiO2 capacitor, which does stay for an amazingly long time.

 

Write and erase cycles damage the SiO2, which is the reason it fails after some number

of write cycles. There are, then, physics reasons that repeated write cycles reduce

the data retention time for data.

 

For many forms of data storage media, the data lasts longer than the read/write devices.

 

I have film negatives of mine from 55 years ago, and others much older than that.

 

Much computer data storage 55 years ago was on 7 track magnetic tape, for which

it is very difficult to find drives to read. Not so hard for 9 track tape, but not easy, either.

 

I suspect CD-R and DVD-R will be not so easy to read in 55 years, even if the data

is still just fine.

[glen_h]

More than you wanted to know:

 

https://www.ni.com/en-us/support/documentation/supplemental/12/understanding-life-expectancy-of-flash-storage.html

[matt_t_butler]

After reading all about spinning disks and write cycles my head is spinning .... ;)

I have CF camera cards from the last century - circa the late nineties - that are still readable.

Do any forum readers have any anecdotes about CF card failure when used for long term storage?

[glen_h]

After reading all about spinning disks and write cycles my head is spinning .... ;)

I have CF camera cards from the last century - circa the late nineties - that are still readable.

Do any forum readers have any anecdotes about CF card failure when used for long term storage?

 

We are used to keeping film cold, but note that keeping flash cold, also helps.

 

The graphs I saw seem to indicate that the mentioned 10 years is at 40C.

The longer times they state at 25C or so. Note that the chips warm up

in use, but for long term storage, that isn't so important.

 

But also they have changed a lot over the years, so older ones might not be

representative of ones today.

 

The early ones store one bit per cell, that is per transistor. They store

two different charge levels. Newer ones store two or three bits, with

four or eight charge levels. That makes them more sensitive to changes

in stored charge.

 

In any case, they mostly follow Arrhenius, so storage at higher temperature

allows for speeding up testing.

 

In any case, the oldest I have is 2006, and I haven't actually tried reading

them recently to test.

[AlanKlein]

11 hours ago, EllinorWilliam said:

Once your disk(s) are 80% full, you should consider them full, and you should immediately be either deleting things or upgrading. SSDs and SMR disks without enough free space will show decreased throughput and - even worse - increased latency (sometimes by many orders of magnitude).

SSDs with a Sandforce controller can fall back in free-block erasure on high-write workloads, leading to write latencies going up by thousands of percent. We usually leave 25% free on the partition level, then observe a fill rate of below 80%.

What does that mean?

[rodeo_joe1]

On 12/12/2022 at 5:54 AM, EllinorWilliam said:

SSDs with a Sandforce controller can fall back in free-block erasure on high-write workloads, leading to write latencies going up by thousands of percent.

Any way of finding out what type of controller is used in a particular SSD? 

I tend to only use SSD/NVME for storing an OS and programs. So far I haven't let any 'C' drive get much more than 2/3rds full, so haven't experienced noticeable slowing. But overfilling is definitely something I'll keep an eye on.

If it's a known issue, then why isn't a fast volatile RAM cache incorporated in the design? Because I see no point in supplying a 1TB drive that has 200GB of effectively unusable space. 

[glen_h]

Unix tradition is to fill drives only 90% full.  Unless you are root, it won't let you write more.

 

In the case of rotating drives, you want to keep files on consecutive

blocks, for faster access.  As disks get fragmented, that is harder.

 

But SSD don't have the rotation latency that magnetic drives have.