Imagine this….

[Grr! While I finished my previous post, I didn’t publish it. Darn it.]

Since I’ve been playing around with computer generated images recently, my thoughts turned to how we see images. When you look at a computer or television screen these days, you are looking at a matrix of pixels. A pixel can be thought of as a very tiny point of light, or a location that can be switched on and off very rapidly.

Pixels are small. There’s 1920 across my screen at the current resolution, and while I can just about see the individual pixels if I look up close, they are small. To get the same resolution with an array of 5cm light bulbs, the screen would need to be 96 metres in size! You’d probably want to sit at about 150m from the screen to watch it.

The actual size of a pixel is a complicated matter, and depends on the resolution setting of your screen. However, the rating of a camera sensor is a different matter entirely. When I started looking into this, I thought that I understood it, but I discovered that I didn’t.

What complicates things as regards camera sensor resolutions is that typically a camera will store an image as a JPG/JPEG image file, though some will save the image as a RAW image file. The JPG format is “lossy” so some information is lost in the process (though typically not much). RAW image file are minimally processed from the sensor data so contain as much information about what the sensor sees as is possible. Naturally they are larger than JPG format images.

When we look at a screen we don’t see an array of dots. We pretty much see a smooth image. If the resolution is low, we might consider the image to be grainy, or fuzzy, but we don’t actually “see” the individual pixels as such, unless we specifically look closely. This is because the brain does a lot of processing of an image before we “see” it.

I’ve used the scare quotes around the word “see”, because seeing is very much a mental process. The brain cells extend right out to the eye, with the nerves from the eye being connected directly into the brain.

The eye, much like a camera, consists of a hole to let in the light, a lens to focus it, and sensor at the back of the eye to capture the image. Apparently the measured resolution of the eye is 576 megapixels, but the eye has a number of tricks to improve its apparent resolution. Firstly, we have two eyes and the slightly different images are used to deduce detail that one eye alone will not resolve. Secondly, the eye moves slightly and this also enables it to deduce more detail than would be apparent otherwise.

That said, the eye is not made of plastic metal and glass. It is essentially a ball of jelly, mostly opaque but with a transparent window in it. The size of the window or pupil is controlled by small muscles which contract or expand the size of the pupil depending on the light level (and other factors, such as excitement).

The light is focused on to an area at the back of the eye, which is obviously not flat, but curved. Most the focusing is done by the cornea, the outermost layer of the eye, but the lens is fine tuned by muscles which stretch and relax the lens as necessary. This doesn’t on the face of it seem as accurate as a mechanical focusing system.

In addition to these factors, human eyes are prone to various issues where the eye cannot focus properly, such as myopia (short sightedness) or hyperopia (long sightedness) and similar issues. In addition the jelly that forms the bulk of the eye is not completely transparent, with “floaters” obstructing vision. Cataracts may cloud the front of the cornea, blurring vision.

When all this is considered, it’s amazing that our vision works as well as it does. One of the reasons that it does so well is, as I mentioned above, the amazing processing that our brains. Interestingly, what it works with is the rods and cones at the back of the eye, which may or may not be excited by light falling on them. This in not exactly digital data, since the associated nerve cells may react when the state of the receptor changes, but it is close.

It is unclear how images are stored in the brain as memories. One thing is for sure, and that is that it is not possible to dissect the brain and locate the image anywhere in the brain. Instead an image is stored, as it is in a computer, as a pattern. I suspect that the location of the pattern may be variable, just as a file in a computer may move as files are moved about.

The mind processes images after the raw data is captured by the eye and any gaps (caused by, for example, blood vessels in the eye blocking the light). This is why, most of the time, we don’t notice floaters, as the mind edits them out. The mind also uses the little movements of the eye to refine information that the mind uses to present the image to our “mind’s eye“. The two eyes, and the difference between the images on the backs of them also helps to build up the image.

It seems likely to me that memories that come in the form of images are not raw images, but are memories of the image that appears in the mind’s eye. If it were otherwise the image would lacking the edits that are applied to the raw images. If I think of an image that I remember, I find that it is embedded in a narrative.

That is, it doesn’t just appear, but appears in a context. For instance, if I recall an image of a particular horse race, I remember it as a radio or television commentary on the race. Obviously, I don’t know if others remember images in a similar way, but I suspect that images stored in the brain are not stored in isolation, like computer files, but as part of a narrative. That narrative may or may not relate to the occasion when the image was acquired. Indeed the narrative may be a total fiction and probably exists so that the mental image may be easily retrieved.