From 1 thousand to 1 Billion – Draft Extract from forthcoming new “Spheres Book”

Extract from forthcoming “Spheres Book”

The phenomena of digitization has emerged connecting and defining information and relationships in the information era. In the last fifty years the scale of digital data is perhaps the single most extraordinary fact that has grown in unprecedented size.

To put this in perspective the library of congress in 1997 had an estimated 3 Petabytes of data in the form of paper books, microfiche and other records (1).. Just ten years later Google MapReduce cluster systems was reported to be processing approx. twenty Petabytes per day.(2). Today research programs such as the Brain Research through Advancing Innovative Neurotechnologies (BRAIN) Initiative to map the human brain is stated as potentially creating Yottabytes of data. (3). Typically today many defense and commercial analysis programs can generate Exobytes of storage.

Perhaps this technological journey is no better exemplified than in comparing two space satellites that were launched approximately at the start and current period. Voyager and Gaia satellites represent a technological shift of forty years. While both remain monumental achievements, the state of technology and the rate and volume of data captured illustrates the orders of magnitude change made possible in that time period. While the famous Hubble telescope launched in 1990 has collected over forty five Terabytes of data (4), the potential of the Gaia Satellite in terms of data collection dwarfs this by several magnitudes.

Voyager 1 and 2 where a product of the 1970’s engineering skill and design. Launched consecutively in 1977 their historic mission continues today and last September 2013, Voyager 1 officially entered interstellar space having travelled 19 billion kilometres and 36 years to reach this point. (5.)

On board Voyager 1 the wide angle camera system had a resolution of 800 x 800 pixels with a field of view of approx. 3 degrees, largely a basic 1 Kilopixel camera and approximately a small corner of a 128 Megabyte Digital Camera today.(6.) The on board data processing system is little more than an eight track tape recorder and basic streaming system. The tape has a storage capacity of 12 images with approximately 6 megbtyes per image. The Voyager space craft took estimated 19,000 images representing an estimating 100 to 200 megabytes of data (7).

By comparison the one billion pixel Gaia camera space telescope launched in December 2013 is the latest satellite from ESA with the mission to map the Milky way in 3D covering one thousand million stars. The Gaia camera is made up of 106 charge couple devices (CCDs) that each feature 4500 pixels by 1966 pixels representing over thousand million pixels. It has a field of view of 212 x 212 arcseconds with an image integration time of 2.85 Seconds. (8) Download Link speed is 10 mbps from L2 Lagrangian orbit injection point. (9)

The Voyager and Gaia mission roles and platform designs are different yet exemplify the change in technology. Voyager camera uses vector scanning to build up the image the contrast in data collected can be seen in this table.

These two satellites with forty years separation between their launch dates can be seen as representative of the massive shift in information technological prowess that shift from resolutions , speed and storage scaling 103 to 105

Back on earth in today’s modern world of internetworking these effects are further increased by the power of the number of nodes and devices connected together.

The era of super massive data is upon us.

Petabytes, Exabytes, Zetabytes and Yottabytes.

Not long ago the benchmark for databases was the a terabyte, or one trillion bytes, a Petabyte of data representing 1015 bytes are now common commercially realistic sizes of data for many larger enterprise systems seeking to store and analyse data. The rapid development of transactional data process has grown at an even larger pace as exemplified by Google processing about 24 Petabytes of data per day back in 2009. (10.) One Yottabyte is a billion Petabytes while these remain extraordinary in scale, the increasing ability to capture , store and importantly manipulate and process this data is perhaps the realization of a digital representation of the world around us. (11.)

#Book #BigData #Digital