Too Much Data

When you think about it, which is what I’ve been doing today, there’s a single attribute the Internet shares with the NSA. They both harvest a gazillion bits and bytes every hour. Electronic storage capacities are constantly increasing.

Think about filling every inch of your house with sealed envelopes containing a note. Some of those notes are worthless, recipes or cartoons. But some are important, and some are vital to your survival.

How do you find what you need when you need it?

The common answer is algorithms. You tell your search bot what you’re looking for and let it sort through everything to find it. But algorithms are written by humans, and like humans are imperfect. So too often what you want is missed, overlooked, never found until it’s too late, if at all.

And the problem is only getting worse. More surveillance, more data mining, it’s adding to an already unmanageable pile of data. Search engines will eventually fail due to an overabundance of stuff to sort through. Maybe quantum computers will solve the problem, maybe not.

Meanwhile billions of bits of information, some of it vital to a person or the nation, will go unseen. It will disappear into the ocean of bits and bytes, of no use to anyone.

And still we gather more every second.