May 21, 2010
In a few earlier posts I have mentioned the new generation of Sony sensors boasting “back-side illumination,” and marketed as Exmor-R (as distinct from Sony’s conventional sensors, just branded Exmor).
Back-side illumination (BSI in the industry jargon) is a tricky and costly chip-fabrication technique, where after depositing all the wiring traces on a silicon wafer, the substrate is flipped over and almost entirely thinned away. This leaves the wiring on the underside of the light-sensitive photodiodes (as Sony describes here), so these unobstructed pixels will theoretically collect more light.
BSI is promoted as one of the technological breakthroughs which might help save image quality, even as manufacturers race to cram more megapixels into tiny sensor areas. In fact, the IMX050CQK actually scaled back its pixel count to 10 Mp, compared to the 12 and 14 that have been becoming increasingly common in the point & shoot market.
Sony introduced the chip in its own first models in the fall of 2009, for example in the WX1. But clearly Sony found it advantageous to spread the sensor development costs over a larger production run, and apparently they’ve aggressively marketed the chip to other camera makers as well. Pretty much any 10 Mp camera sold this year advertising a backside-illuminated sensor uses it. It seems particularly popular in today’s nutty “Ultra Zoom” market segment.
As reviews of these new BSI-based cameras filter out, the word seems to be that they do offer decent image quality—but hardly anything revolutionary. If their high-ISO images look smooth, it seems to be partly thanks to noise reduction processing, which can destroy detail and add unnatural, crayon-like artifacts.
March 23, 2010
I can’t think of any greater achievement in press-release puffery than having your claims uncritically repeated in the New York Times.
As many of you have heard, a company called InVisage has announced a “disruptive” improvement in imager sensitivity, through applying a film of quantum dot material. The story has been picked up by lots of photography blogs and websites, including DP Review.
But don’t throw your current camera in the trash quite yet. The Times quoted InVisage’s Jess Lee as saying, “we expect to start production 18 months from now”—with the first shipping sensors designed for phone-cam use.
I have no way of knowing if InVisage’s claims will pan out in real-world products. But it’s interesting that a few people who work in the industry have skeptical things to say.
I do find it exaggerated to claim that the new technology is “four times” better than conventional sensors (95% versus 25% efficiency). Backside-illuminated sensors are shipping today which have much higher sensitivity; and refinements to microlenses and fill-factors are continuing.
However one true advantage of the quantum-dot film is that incoming photons slam to a stop within a very shallow layer (just half a micron thick). This is in contrast to conventional photodiodes, where longer-wavelength (redder) photons might need to travel through 6-8 microns of silicon before generating an electron.
That difference might enable sensors without microlenses to absorb light efficiently even from very oblique angles. It would permit lens designs with shorter back focus (as with rangefinder cameras); and thus we could get more compact cameras overall.
Kodak’s full-frame KAF-18500 CCD, used in the Leica M9, could only achieve the same trick by using special offset microlenses. (And if we are to believe this week’s DxO Mark report, that sensor may have compromised image quality in other ways.)
But I’m still slapping my head at the most ridiculous part of this whole story:
To give an example of what the “quantum film” technology would enable, Mr. Lee noted that we could have an iPhone camera with a 12-megapixel sensor.
Can I scream now? Is the highest goal of our civilization trying to cram more megapixels into a phone-cam? And WHY? But the über-iPhone is just nonsense for other reasons, even if we desired it.
As near as I can tell, an iPhone sensor is about 2.7 x 3.6 mm (a tiny camera module needs an extremely tiny chip). Space is so tight that a larger module would be unacceptable. So, to squish 12 Mp into the current sensor size, each pixel would need to be 0.9 microns wide!
An iPhone’s lens works at f/2.8. At that aperture, the size of the Airy disk (for green light) is 3.8 microns. This is the theoretical limit to sharpness, even if the lens is absolutely flawless and free from aberrations. At the micron scale, any image will be a mushy, diffraction-limited mess.
Also, remember that noise is inherent in the randomness of photon arrival. No matter what technology is used, teensy sensors will always struggle against noise. (The current “solution” is processing the image into painterly color smudges.)
And the dynamic range of a sensor is directly related to pixel size. Micron-scale pixels would certainly give blank, blown-out highlights at the drop of a hat.
But let’s be optimistic that, eventually, this technology will migrate to more sensible pixel sizes. Even if the sensitivity increase only turns out to be one f/stop or so, it would still be welcome. A boost like that could give a µ4/3-sized sensor very usable ISO-1600 performance.
But before we proclaim the revolution has arrived, we’ll need to wait for a few more answers about InVisage’s technology:
- Is the quantum-dot film lightfast, or does it deteriorate over time?
- Is there crosstalk/bleeding between adjacent pixels?
- Is the sensitivity of the quantum-dot film flat across the visible spectrum, or significantly “peaked”? (This would affect color reproduction)
- How reflective is the surface of the film? (Antireflection coatings could be used—but they affect sharpness and angular response)
So, stay tuned for more answers, maybe in the summer of 2011…
March 5, 2010
If you’re interested in a behind-the-scenes peek into the imaging-chip industry, check out the blog “Image Sensors World.”
Much of this revolves around cell-phone cameras, which today are by far the largest consumer of imaging chips. And that’s a market where the drive for miniaturization is even more extreme than with point & shoot cameras. For a phone-cam to boast 2 megapixels, 4 megapixels, or more, each pixel must be tiny.
At that scale, the light-gathering area of each pixel is so minuscule that back-side illumination practically becomes mandatory. The reasons are well explained in OmniVision’s “technology backgrounder” PDF.
This document’s introduction says,
“Evidently, pixels are getting close to some fundamental physical size limits. With the development of smaller pixels, engineers are asked to pack in as many pixels as possible, often sacrificing image quality.”
Which is an amusingly candid thing to say—considering that they are selling the aforementioned chips packed with “as many pixels as possible.”
What are these “fundamental limits”? Strangely, OmniVision’s document never once mentions the word “diffraction.” But as I’ve sputtered about before, with pixels the size of bacteria, diffraction becomes a serious limitation.
Because of light’s wavelike nature, even an ideal, flawless lens cannot focus light to a perfect point. Instead, you get a microscopic fuzzy blob called the Airy disk.
Now, calling it a “disk” is slightly deceptive: It is significantly brighter in the center than at the edge. Thus, there is still some information to extract by having pixels smaller than the Airy disk. But by the time the Airy disk covers many pixels, no further detail is gained by “packing in” additional ones.
Our eyes are most sensitive to light in the color green. For this wavelength, the Airy disk diameter in microns is the f/ratio times 1.35. (In practice, lens aberrations will make the blur spot larger than this diffraction limit.)
But even using a perfect lens that is diffraction-limited at f/2.3, the Airy disk would cover four 1.1 micron pixels.
A perfect lens working at f/3.5 (which is more realistic for most zooms) will have an Airy disk covering nine pixels of 1.1 micron width. This is one of the “fundamental physical size limits” mentioned in OmniVision’s document.
Manufacturing a back-illuminated chip is quite complex. And for OmniVision to be able to crank them out in quantity is a technological tour de force. As I wrote earlier, there are still a few tweaks left to make imaging chips more sensitive per unit area; this is one of them.
Perhaps this helps explain another curiously candid statement I saw recently. Sony executive Masashi “Tiger” Imamura was discussing the “megapixel race” in a PMA interview with Imaging Resource. And he said,
” …making the pixel smaller on the imager, requires a lot of new technology development. [...] So, as somebody said, the race was not good for the customers, but on the other hand, good for us to develop the technologies. Do you understand?”
February 19, 2010
The stats show that my post, “the Great Megapixel Swindle,” continues to get quite a lot of traffic. That’s a little unnerving, given that it was written as a quick, off-the-cuff tantrum. If I’d known how many folks would read it, I would have said several things more precisely.
Around the internet, the “Swindle” spawned many, many discussion threads—I can’t keep track of them all. I did try to respond to many of the questions and misunderstandings that I was seeing, in a followup post here. And I’ve expanded on the same issues in many other posts as well.
Today I’ve noticed a thread over at Rangefinder Forum which raises the question, “isn’t it unfair to use a crop from the background? Naturally that looks bad, since it’s out of focus.”
First, the real point this example makes is this: Cameras with tiny pixels must use aggressive post-processing to reduce noise; and this can cause strange, unnatural-looking artifacts. (There’s more on the subject here.)
But on the question of depth of field, I should clarify a bit.
The EXIF data for this shot shows the Olympus FE-26 was set at the wide end of its zoom range—namely 6.3mm. Such a short focal length implies extreme depth of field. The f/stop used was f/4.6.
The H. Lee White is a 700-foot-long Great Lakes freighter. I’m not exactly sure how far the camera was from the subject; but it’s doubtful that it was closer than 15 feet.
Now, we can’t blindly apply standard depth-of-field tables here. The standard calculation (e.g. if you scroll down to Olympus FE-26 here) uses a circle of confusion of 0.005 mm for this sensor format.
But when you are looking at such an extreme enlargement, the assumptions behind that break down (CoC is generally referenced to viewing an 8×10 print at a moderate distance).
But consider that this camera has a sensor size of about 6 x 4.5mm, and so each individual pixel is 1.53 microns wide. In other words, 0.00153 mm.
Clearly, the meaningful circle of confusion can’t be smaller than one pixel. Given the resolution loss that happens with Bayer interpolation, 0.002 mm seems a realistic CoC.
So the out-of-focus blur is actually negligible compared to the pixel size in this case.
You can use an alternate depth of field calculator which lets you input arbitrary values if you’d like to explore this further yourself.
February 16, 2010
Today, some musings that are a bit more (har har) abstract.
Sometimes we become numbed with the perpetual escalation of tech specs. The mindset of the computer industry, where each new generation promises more, faster & bigger, seems to be the new normal.
And it can become a self-fulfilling prophecy. If it is technically possible to ratchet up some specs number, inevitably we’ll choose to. That’s what keeps people buying!
But there’s one nice example of an industry that dangled ever-increasing numbers in front of consumers—who then yawned and said “no thanks.”
How many DVD-Audio or “Super Audio CD” titles do you own? (Okay, I’m sure someone out there is an enthusiastic adopter. But I mean, the average person.)
The original standard for music CDs uses 44,100 samples per second. Each sound sample has 16 bits; meaning it encodes the full range from soft to loud with about 65,000 discrete levels.
The CD standard was adopted around 1980; so at that time there was pressure to keep the bit rate low enough so that the player electronics would not be prohibitively expensive.
No one knew how data-handling ability would explode over the following decades. When you see a speed “60x” or “133x” on a camera memory card, those are referenced against the original CD-player (or, 1x CD-ROM) bit rate.
So in the audiophile world there were grumbles from the start that the CD bit rate was insufficient.
With only 16 bits, the quietest elements of music (like note decays and room ambience) must be recorded with somewhat coarse resolution. It’s similar to how shadow areas in digital photos can look noisier than highlights. A standard that used 20 bits or so would have preserved more fine “texture” in low-amplitude sounds.
And before digitizing sound for CD, any frequency of 22,050 cycles/second or higher must get chopped off. A high-pitched rising chirp that went past that frequency would make the A/D converter miss the true peaks and troughs of the wave; it would misrepresent it as a falling note, back down in the audible spectrum.
For CD audio, 22 kHz is the “Nyquist Limit,” and you need an “antialias filter” to block higher frequencies (yes, these exact same principles apply to digital-camera sensors). But there were complaints that the antialias filters degraded sound quality.
And while most adults can’t hear steady pitches much above 16 kHz, very brief clicks at a higher theoretical frequency might contribute some “edge” to percussion and note attacks. A lot of professionals preferred to record at 48 kHz instead. (You can go higher today, though bandwidth limitations in the analog realm become significant.)
Folks who produce music may have reasons for 24-bit sampling (tracks are often put through computation-intensive effects; you don’t want rounding errors), but 20-bit delivery covers an excellent dynamic range. Even taking a 20-bit master and dithering it down to a 16-bit release version can work well.
I apologize to all of you whose eyes glazed over during those past few paragraphs. The truth is, most of us found CD sound quality perfectly adequate.
If you do the math, the bit rate used for CD sound is about 1.4 Mbit/sec (in stereo). A reasonable standard that would have handled any outstanding quality issues might have been 20 bits at 48 kHz. That works out to 1.9 Mbit/sec.
But the arrival of DVD technology offered a huge increase in disc capacity. It was a great opportunity to sell a newer, zingier, whizzier-spec music format too.
So a “format war” broke out, but based on bit rates that were sort of crazy. Stereo in Sony’s SACD standard burns up 5.6 Mbit/sec—quadrupling the CD rate. DVD-A is a “family” of standards (a standard that doesn’t standardize); but its highest supported stereo rate is 9.2 Mbits/sec!
A leading writer in the digital-audio field once told me in an email that the reasons for these bit rates had more to do with “quieting the lunatic fringe” than with any technical justification.
But the public treated these new audio formats with indifference. SACD has found a niche in classical music, but most folks are completely unaware of it. (Even though, soon enough, millions did go out to buy a new disc format: Blu-ray.)
You all know what actually happened with music: Buying it online, and being able to take it everywhere in your pocket, totally changed the game.
So with downloadable music, the bit rate actually plunged instead. Today we’re buying music that uses 1/5th or even 1/8th the old CD standard! (Psychoacoustically intelligent data compression makes it possible.)
So… does this have anything to do with photography?
Camera manufacturers today (even those making enthusiast models) continue to use megapixels as the spec that defines “improvement.” Every year, more bits!
This shows a depressing lack of creativity. Past the point where this offers any real value, it’s just mindlessly chasing a number.
What we need is a serious rethink—to create something so novel and desirable that any talk about pixel count become irrelevant.
My feeling is that a game-changer for digital cameras is radical improvements in low-light capability. (This parallels the opinions of that recent Gizmodo article.)
We’ve suffered through decades of terrible point & shoots—whose slow zooms and limited sensitivity demanded nuclear-blast electronic flash for every indoor shot.
Flash is blinding, conspicuous, annoying to bystanders, and quite rightly prohibited at most museums and concerts. It drags out the lag time before shots.
It’s also a form of lighting which makes people look like shit.
Consider the fraction of our days we spend indoors, often under marginal illumination. But living rooms, restaurants, etc.—isn’t this where our real lives happen? Wouldn’t it be amazing to record those moments realistically, accurately, but without blinding and ugly flash?
What if you had a camera that could shoot at ISO 1600, cleanly? What about a camera where anti-shake let you trust shooting at 1/15th sec.? Plus a lens of f/1.7 or f/1.4—scooping up four times as much light?
Then, people could take photos freaking anywhere. Without flash. The light of a single candle is enough! (That’s LV 2, if you’re wondering.)
Now, to get a lens that fast, we’d probably need to lose the zoom.
Oh noes! Cameramakers’ second most-flogged spec number is zoom range. Our 12x is better than their 10x! I can hear the howls already: “Who would buy a camera without a zoom?”
Well, how many people use cell phones as their main camera today? Those have no optical zoom.
Some used to ask, “who would pay for a compressed MP3 when you can own the real physical disc?”
People who appreciate convenience. People who want technology to fit into their real lives. Us.
February 14, 2010
The first thing to understand about picture noise (aka grain, speckles) is that it’s already present in the optical image brought into focus on the sensor.
Even when you photograph something featureless and uniform like a blank sky, the light falling onto the sensor isn’t creamy and smooth, like mayonnaise. At microscopic scales, it’s lumpy & gritty.
This is because light consists of individual photons. They sprinkle across the sensor at somewhat random timings and spacings. And eventually you get down to the scale where one tiny area might receive no photons at all, even as its neighbor receives many.
Perhaps one quarter of the photons striking the sensor release an electron—which is stored, then counted by the camera after the exposure. This creates the brightness value recorded for each pixel. (There is also a bit of circuit noise sneaking in, mostly affecting the darkest parts of the image.)
But no matter how carefully a camera is constructed, it is subject to photon noise—sometimes called “shot noise.” You might also hear some murmurs about Poisson statistics, the math describing how this noise is distributed.
When you start from a focused image that’s tiny (as happens with point & shoot cameras), then magnify it by dozens of times, the more noticeable this inherent noise becomes:
In fact, the only way to reduce the speckles is to average them out over some larger area. However, the best method for doing this requires some consideration.
The most obvious solution is this: You decide what is the minimum spatial resolution needed for your uses (i.e., what pixel count), then simply make each pixel the largest area permissible. Bigger pixels = more photon averaging.
Lets recall that quite a nice 8 x 10″ print can be made from a 5 Mp image. The largest inkjet printers bought by ordinary citizens print at 13 inches wide; 9 Mp suffices for this, at least at arm’s length. And any viewing on a computer screen requires far fewer pixels still.
The corollary is that when a photographer does require more pixels (and a few do), you increase the sensor size, not shrink the pixels. For a given illumination level (a particular scene brightness and f/stop) the larger sensor will simply collect more photons in total—allowing better averaging of the photon noise.
But say we take our original sensor area, then subdivide it into many smaller, but noisier, pixels. Their photon counts are bobbling all over the place now! The hope here is that later down the road, we can blend them in some useful way that reduces noise.
One brute-force method is just applying a small-radius blur to the more pixel-dense image. However this will certainly destroy detail too. It’s not clear what advantage this offers compared to starting from a crisp, lower-megapixel image (for one thing, the file size will be larger).
Today, the approach actually taken is to start with the noisier high-megapixel image, then run sophisticated image-processing routines on it. Theoretically, smart algorithms can enhance true detail, like edges; while smoothing shot noise in the areas that are deemed featureless.
This is done on every current digital camera. Yet it must be done much more aggressively when using tiny sensors, like those in compact models or phone-cams.
One argument is that by doing this, we’ve simply turned the matter into a software problem. Throw in Moore’s law, plus ever-more-clever programming, and we may get a better result than the big-pixel solution. I associate the name Nathan Myrvold with this form of techno-optimism (e.g. here). Assuming the files were saved in raw format, you might even go back and improve photos shot in the past.
But it’s important to note the limits of this image-processing, as they apply to real cameras today.
Most inexpensive cameras do not give the option of saving raw sensor files. So before we actually see the image, the camera’s processor chip puts it through a series of steps:
Bayer demosaicing —> Denoising & Sharpening —> JPEG compression
The problem is that photon noise affects pixels randomly—without regard to their assigned color. If it happens (and statistically, it will) that several nearby “red” pixels end up too bright (because of random fluctuations), the camera can’t distinguish this from a true red detail within the subject. So, false rainbow blobs can propagate on scales much larger than individual pixels:
The next problem is that de-noising and sharpening actually tug in opposite directions. So the camera must make an educated guess about what is a true edge, sharpen that, then blur the rest.
This works pretty well when the processor finds a crisp, high-contrast outline. But low-contrast details (which are very important to our subjective sense of texture) can simply be smudged away.
The result can be a very unnatural, “watercolors” effect. Even when somewhat sharp-edged, blobs of color will be nearly featureless inside those outlines.
Or, combined with a bit more aggressive sharpening, you might get this,
Clearly, the camera’s guesses about what is true detail can fail in many real-world situations.
There’s an excellent technical PDF from the DxO Labs researchers, discussing (and attempting to quantify) this degradation. Their research was originally oriented towards cell-phone cameras (where these issues are even more severe); but the principles apply to any small-sensor camera dependent on algorithmic signal recovery.
Remember that image processing done within the camera must trade off sophistication against speed and battery consumption. Otherwise, camera performance becomes unacceptable. And larger files tax the write-speed and picture capacity of memory cards; they also take longer to load and edit in our computers.
So there is still an argument for taking the conservative approach.
We can subdivide sensors into more numerous, smaller pixels. But we should stop at the point which is sufficient for our purposes, in order to minimize reliance on this complex, possibly-flawed software image wrangling.
And when aberrations & diffraction limit the pixel count which is actually useful, the argument becomes even stronger.
February 12, 2010
January and February are months when the air hangs thick with new-camera introductions.
We’re also approaching this country’s wildest Lost Weekend of photo-equipment marketing, PMA 2010, which starts February 19th.
So, it’s the right moment to look our camera industry straight in the bleary eye, and ask the hard question. Are you on drugs?
Regular readers of this blog know my arguments well: Overdosing cameras with millions of teensy pixels is risky behavior—in fact, irrational and damaging.
Not unlike drug abuse.
But to survey the full breadth of this scourge, I’ve needed to pore over DP Review’s specs listings—noting the pixel density of every model introduced since January, 2008. (There were almost 400 in total.)
No doubt I’ve missed some models somewhere, or copied some numbers wrong. But I’ve made a sincere attempt to find out: Which brand has the worst megapixel-monkey on its back?
Here’s how it works:
- Camera models of 35 Mp/sq. cm or more (but less than 40) earn one crack pipe
- Models having 40 Mp/sq. cm or above (but below 45) get two crack pipes
- Any model with 45 Mp/sq. cm or “higher” is awarded the unprecedented three crack pipes.
But a camera maker can redeem themselves, somewhat. All I want to see is evidence they’re entering rehab and doing community service.
- Any model with a pixel density of 5 to 15 Mp/sq. cm subtracts one crack pipe
- A camera having 2.5 to 5 Mp/sq. cm knocks off two pipes
- A model of less than 2.5 Mp/sq. cm expunges three whole pipes.
[Note: Currently, the last category only includes Nikon's high-end D700 and D3s. The Micro Four Thirds models included are all a whisker over 5 Mp/sq. cm.]
First, we must single out Sigma—Boy Scout among camera makers.
Better known for their lenses, they have a small lineup of cameras using the Foveon sensor, which is 20.7 x 13.8mm.
This makes them the only current camera manufacturer to be 100% CRACK FREE. We may find their products a bit geeky and lacking in social graces, but at least they’re leading the clean life.
But for the others, it’s a grim tragedy. In reverse order of crackheadedness, here is the ranking:
- Ricoh: 2 crack pipes
- Pentax: 12 crack pipes
- Tied, with 24 crack pipes each: Sony and Kodak
I must interrupt here to mention that Sony’s crack score should have been 20 points higher—except that, like an agitated street person muttering “I’m getting my life together!”, Sony somehow introduced eleven different DSLR models in the past two years.
But I’m going to let that slide. Sony has at least admitted a problem. Their new (and unproven) detox plan involves a medication named “Exmor R,” and a risky procedure called “back illumination.”
Sadly, we all know how fragile recovery can be.
- Okay. Back to Nikon: 25 crack pipes
- Fujifilm: 31
- Casio: 34
- Canon: 37
(Canon does earn a special “we’re getting help” mention—for tapering off their S90 and G11 models to a slightly lower dose of 10 mg. Er… Mp.)
- Samsung: 39
Samsung! Snap out of it! There’s still time to go home to your family, bringing more NX-mount cameras. I am speaking to you as a concerned friend.
And finally—we get to the two saddest cases in the whole megapixel ward.
Like many addicts, they always seemed able to hold it together in public. But the numbers don’t lie.
- Yes, Olympus: 54 crack pipes
…and perhaps most shocking,
- Panasonic: 67 crack pipes
Tell me it’s not true. The two leaders of Micro Four Thirds? The upstanding citizens who gathered us all in the church basement, to spread the good news about large chips in compact cameras—living a lie?
It’s tragic. One day, you’re a respected member of your industry. Then suddenly, you’re passed out in a seedy motel, wearing nothing but a frayed terry-cloth robe, surrounded by crumpled marketing plans.
Camera makers: There is still time to clean up, and save yourselves.
February 11, 2010
During the past decade, the world of digital cameras has obviously gone through numerous changes.
Now, the aspect I’ve written about most here is the endless (and problematic) escalation of pixel counts. But we should remember that many other facets of camera evolution have been going on in parallel.
Today we can only shake our heads at the postage-stamp LCD screens which were once the norm on digital cameras. And autofocus technology continues to improve (although cameras can still frustrate us, making us miss shots of moving subjects).
Moore’s Law has raced onwards. The result is that the proprietary image-processing chips used in each camera get increased “horsepower” with each new generation.
Besides keeping up with the growing image-file sizes, this allows more elaborate sharpening and noise-reduction methods to be applied to each photo. (Whether this noise suppression creates weird and unnatural artifacts is still a question.)
And there are other changes which have helped offset the impact of megapixel escalation. Chip design has improved, reducing the surface area lost to wiring connections. Sensors today are also usually topped with a grid of microlenses, helping funnel most of the light striking the chip onto the active photodetectors.
At the beginning of the digital-camera revolution, CMOS sensors were a bit less developed than CCDs (which had been used in scientific applications for some time). But eventually the new challenges of CMOS technology got ironed out. Today, the DSLRs which lead their market segments all use CMOS sensors.
Not every camera maker is on the same footing, technologically. Companies control different patent portfolios. Many lack their own in-house chip fabs, which can help move innovations to market faster.
So within a given class of cameras (e.g. a particular pixel size), you can still discover performance differences.
But the sum total of all this technology change has been that the better-designed cameras have been able to maintain and even improve image quality, even as pixel pitch continued to shrink.
Can technology keep saving us? Will progress continue forever?
I dispute that it’s even desirable to decrease pixel size further still. But one question is whether there is still some headroom left in sensor technology—allowing sensitivity per unit area to keep increasing. That could compensate for the shrinking area of each pixel.
Well, there are some important things to remember.
The first is that every pixel in a camera sensor is covered by a filter in one of three colors (the Bayer array). And these exist for the purpose of blocking roughly two-thirds of the visible light spectrum.
(There was a Kodak proposal from 2007 for sensors including some “clear” pixels, which would avoid this. But that creates other problems, and I’m not aware of any shipping product based on it.)
The other issue is that there’s a hard theoretical ceiling on how sensitive any photoelectric element can be, no matter its technology. How close a chip approaches that limit is called its quantum efficiency. And out of a theoretically perfect 100%, real sensors today get surprisingly close.
Considering monochrome scientific chips (i.e., no Bayer array), the best conventional microlensed models can average roughly 60% QE in the visible spectrum.
Astrophotographers worry quite a lot about QE. And a well-known one based in France, Christian Buil, actually tested and graphed the QE of various Canon DSLRs. Note, for example, that the Canon 5D did improve its green-channel sensitivity quite a bit in the Mk II version.
In Buil’s bottom graphs notice how much the Bayer color filters limit Canon’s QE compared to one top-of-the-line Kodak monochrome sensor. (The KAF-3200ME has microlenses and 6.8 micron pixels.)
So, seemingly, one area of possible improvement could be improving the color filters used in the Bayer array.
But tri-color filters are a mature technology—having had numerous uses in photography in the many decades before digital. To insure accurate color response, you must design a dye which attenuates little of the desired band; but blocks very effectively outside it. Dyes must also remain colorfast as they age or are exposed to light. Basically, it’s a chemistry problem—and a surprisingly difficult one.
Considering all this, the ability to reach 35% QE (on a pixel basis) in a color-filtered chip is a pretty decent showing already.
Now for years, scientific imagers have used a special trick of back-illuminating a CCD. This can push QE up to 90% in the photographic spectrum (roughly 400-700 nm) on an unfiltered chip.
And suddenly, camera makers have “invented” the same idea for photography applications. Sony is talking about tiny, point & shoot pixels here, which lose significant area to their opaque structures. So a “nearly twofold” efficiency boost might be feasible in that case.
But we saw that when pixels are larger, back illumination can only improve QE from about 60% to 90% (before filtering).
And it’s much more expensive to fabricate a chip right-side up, flip it over, and carefully thin away the substrate to the exact dimension required. Yields are lower; so when you try to scale it up to larger chips, costs are high. It’s not clear whether this will really be an economical option for DSLR-sized sensors.
But wouldn’t it be a massive breakthrough to add 50% more light-gathering ability?
Actually, less than you might think. Remember, that’s only half an f/stop. You get more improvement e.g. in switching from a Four Thirds sensor to an APS-C model, just from the area increase.
So back-illumination is an improvement worth pursuing—especially for cameras using the teeniest chips, which are the most handicapped by undersized pixels.
But beyond that, we start to hit serious limits.
Pure sensor size remains the most important factor in digital-photo image quality. And no geeky wizardry is likely to change that soon.
February 9, 2010
For example, he once ran a demonstration showing that random viewers couldn’t see much difference in a row of enormous, 16 x 24″ prints, even when the pixel counts varied wildly.
But Pogue made an odd aside last week, at the conclusion of his compact-camera buyer’s guide:
“As the ridiculous megapixel race winds down at last, …”
…a comment which left me scratching my head in confusion.
Perhaps he’s been busy—avalanched under press releases for all those new tablet e-readers. Or maybe he’s aggravated that the megapixel race didn’t stop at 7 Mp, as he hoped in 2006?
Believe me, I understand the frustration; the desire to throw up your hands, declare victory, and retreat.
But in reality, the megapixel war still rages—most obviously among point & shoot cameras. (And it’s the buyers of these mass-market models who are most likely to take advice from newspaper articles, rather than from some specialist geek website.)
Now, Pogue begins his compact-model roundup by noting some limitations inherent to all small cameras: shutter lag, grain, and blown highlights. But he hasn’t much followed his own oft-stated advice: Choose a camera based on its sensor size, not pixel count.
Seven of Pogue’s nine selections have pixels smaller than 1.54 microns. (The Nikon’s are a ludicrous 1.43 microns.) His Panasonic pick does a smidge better, at 1.56 µm.
But compare these to the 2.0 microns of the (still fairly compact) Canon S90—each of its pixels can collect about 70% more light.
His one choice I might grudgingly accept is the 10 Mp Fujifilm F70EXR. Besides having 1.77 micron pixels, this model offers a special low-light mode. Ironically, it works by pairing up pixels, turning it into a 5 megapixel camera! Hey, it’s a Pyrrhic victory, but I’ll take it.
But Pogue’s other picks simply have pixels that are too small, by any reasonable criterion.
I do admit that anyone forced to buy a compact digicam today—lets say your old one just died, and you’re leaving on a trip tomorrow—faces very limited choices.
If need be, you might hunt for a model using one of the new generation of 10 Mp back-illuminated CMOS sensors. For example, Sony’s “Exmor R” chip (versus the regular, non-R kind) works some special tricks to wring the most out of its 1.7 micron pixels.
This is actually rather worrying. Aren’t “enthusiast” photographers supposed to know better? That smaller pixels compromise other aspects of performance, like dynamic range and noise?
Stuffing 18 million pixels into the same 22.3 x 14.9mm sensor area makes each pixel 4.3 microns wide. This is the same pixel pitch that causes Micro Four Thirds cameras to struggle with noise when pushed up to ISO 800.
Consider the 12 Mp Pentax K-x, praised for its high-ISO performance. It uses 5.5 micron pixels instead. This gives each pixel 63% more light-gathering area.
Also remember that on the T2i’s sensor, each millimeter of sensor width contains 232 pixels.
But it is very rare for a real-world lens to resolve detail at that scale with reasonable contrast. If one can do so, it will only be at a single, optimum, middle f/stop. That’s not especially practical.
(Aberrations limit sharpness at wide f/stops; diffraction creates blur at smaller ones—in APS-C cameras, typically f/8 or smaller. For a more technical discussion, start here.)
I wish we could say that megapixel marketing madness had finally ended.
But I’m not seeing any evidence this is true.
February 8, 2010
Why jam extra megapixels into a compact camera, if its lens can’t resolve enough detail to use them?
Sampling a fuzzy image with an ever more finely-spaced pixel grid eventually stops adding information. After that, it merely balloons file sizes needlessly.
So it’s useful to check whether all of a camera’s pixels are capturing something real. Or do they simply hit a wall of lens aberrations, diffraction, and sensor noise?
I’ve had a chance to take some sample shots with the 8-megapixel Nikon Coolpix P60, using the resolution test target I posted last week. (Open a tab to remind yourself how the target is supposed to look.)
The P60 is assembled in China—perhaps even in a factory that doesn’t say “Nikon” over the door. Nonetheless, Nikon’s lens designers have an enviable reputation. And using a 9-element, 7-group design, its lens aberrations ought to be reasonably well controlled. So how well did the little Coolpix do?
As with my earlier post, I set up the target so that its squares with 40 divisions per inch match the pixel pitch of the sensor. At that magnification, ideally the camera should form an image of the 40-line target as a row of black pixels, then a row of white pixels, then black, etc.
The P60 shoots images that are 3,264 pixels wide. Dividing this by 40 tells us the subject field needs to be 81.6 inches wide overall—6 feet, 9–5/8 inches. Two reference marks at the proper spacing (black electrical tape on a sheet of plywood) helped me frame each shot with the right magnification.
Here’s a full-rez sample of what one complete test frame looks like. I turned off as many automatic settings as possible, to improve consistency (see notes* at end).
We’ll start with the best-case scenario: The lens is set at its sharpest focal length (at the wide end of the zoom range) and the target is in the center of the frame. The ISO is 80 (its lowest setting), for minimum noise. The aperture is wide open, for lowest diffraction:
Yes, this is the sharpest image I got in my tests.
While it’s startling to see the rainbow patterning in the 40-line sample, this is actually the “good” news. It means that enough resolution is being focused on the sensor for the test pattern to completely confuse the demosaicing algorithm.
We also see vertical and horizontal texture in the 50-line squares; but I believe this is “false texture” (aliasing), rather than true resolution.
(And please remember that most real-world subjects lack the kind of repeating patterns which make demosaicing totally freak out like this.)
The sharpness is not quite as good at longer focal lengths. Zooming to 14.3mm (corresponding to 81e) and backing away to maintain field size, things look like this:
The 50- and 40-line samples have lost most of their detail; also, the 30-line sample has begun to look a bit rougher. Note that the hairline border around the number boxes is virtually gone here—unlike the first shot which showed a hint of it.
We can also look at what happens towards the edges of the frame (where lens aberrations are generally not as well controlled). At a longer zoom setting of 23.3mm, a target at the photo’s right edge looked like this:
Well, there’s some rather troubling green fringing here. And even the 10-line sample has lost contrast noticeably.
But the other thing to notice is how soft the vertical 30-line square has gotten. It’s hard to avoid the conclusion that 8 megapixels is plenty at this point; more finely-spaced pixels would not capture any additional detail.
Now, traditionally photographers have enjoyed the creative control of trading off shutter speed against aperture; e.g. using longer exposures at smaller f/stops, to yield a deeper zone of sharp focus.
And the P60 is theoretically aimed at the enthusiast end of the point & shoot market—folks who would appreciate manual controls like this.
But, in fact, its aperture is only “sort of” adjustable.
Nikon’s manual notes (somewhat cryptically),
- Aperture: Electronically-controlled preset aperture and ND filter (–0.9 AV) selections
- Range: 2 steps (f/3.6 and f/8.5 [W])
What happens when you “stop down” this lens is that an arm swings into place with a smaller hole in it. And after inspecting this with a magnifier, the hole does appear to be covered by a rectangle of neutral-density filter material.
Combined, the filter and the hole cut out about 2.4 f/stops worth of light. But diameter-wise, the aperture is seemingly just f/6.0 or so—not the f/8.5 stated (at the zoom wide end).
Why on earth did Nikon do this? Well, it’s because stopping down the lens increases diffraction, that’s why. (And given compact cameras’ teeny focal lengths, you rarely need more depth of field.)
Despite this throttled aperture range, we can still see diffraction having a blurring effect:
First, notice the overall drop in contrast. The 50- and 40-line samples are completely featureless. And the 30-line sample has slipped past the limit of resolution—you can no longer count all of the lines.
At this zoom setting, the (physical) aperture might measure f/7.0; this means an Airy disk more than 9 microns wide. With the P60’s sensor size, those blur disks spill across many pixels.
While sharpening by the camera’s processor can accentuate the bright peak at the center of the Airy disk, it can’t pull back detail that never existed. So, if we want the ability to close down the lens even by two stops, then a sensor with larger pixels, not smaller ones, is needed.
Note that we’ve been looking exclusively at ISO 80 here—the camera’s lowest sensitivity setting. But that’s not very realistic, considering how people actually use their cameras.
With a shirt-pocket compact, we would rarely feel like lugging around a tripod! So under anything but bright daylight, we’ll often need to use a higher ISO setting.
Fifteen years ago, films of ISO 400 were the most commonly purchased speed. So how does ISO 400 look here?
The 40-line sample does show some color tint from demosaicing; but the noise (and noise-reduction processing) are severe here. Neither the 40- or 50-line samples give any hint which direction the lines run.
The 30-line squares have once again passed the point where lines cannot be resolved completely. And there’s no sign of the hairlines bordering the number boxes.
At this level of resolution, we would hardly lose any detail if we substituted a 5 megapixel sensor of the same size. Plus in that case, each pixel would have 60% more light-gathering area—helping tame the noise.
In conclusion, the P60’s lens somewhat out-resolves the sensor under the most favorable circumstances. This is seen in the form of colored, “gritty” demosaicing artifacts.
But it doesn’t take long before real-world complications undercut the sensor’s inherent resolving power. And while we’ve treated aberrations, diffraction, and noise as separate, in practice several of these handicaps often come together in the same photograph (along with other factors such as camera shake).
This test is not definitive; it merely represents the performance of one, very average compact camera. However if we are seeing such flaws at “only” 8 megapixels, what sense does it make to drive up pixel counts even higher—to today’s 10, 12 or 14 Mp?
You can download a PDF of the target here. If you own a compact camera, I encourage you to try this test for yourself.
* Test setup: The camera was mounted on a tripod, with VR turned off. Unless noted otherwise, enlarged details are from the center of the frame, with the camera set at ISO 80 (best-case conditions).
I set ISO, aperture, and shutter speed manually. The white balance was set to “cloudy,” and the contrast setting was turned up to +1. I left sharpening and saturation at their mid setting, 0. A 2-second self-timer allowed vibrations to die out after I pressed the shutter release.
Autofocus used the central spot only; this included several of the target’s black squares. Two shots were taken at each setting, allowing the camera to re-focus each time (I never noticed any inconsistency between pairs of photos).
The 200% samples shown here were upsized using Photoshop’s “nearest neighbor” method, to avoid any additional artifacts. Any sharpening halos are from the camera’s own processing.