by Barry Green
The trade-offs inherent in CMOS and CCD sensor technology manifests themselves in sensor artifacts. Educating yourself on this issue is important because it can have a potentially detrimental effect on your footage. The four main sensor artifacts we're exploring are smear, skew, wobble and partial exposure. Your type of shooting mayor may not encounter these artifacts, but any prudent shooter should be educated as to what they are, how they affect your footage and what circumstances cause them to happen. "Forewarned is forearmed," and having an understanding of typical sensor artifacts may keep you from compromising footage that otherwise may have turned out well.
THE INFAMOUS ROLLING-SHUTTER EFFECT
Before we examine various artifacts, let's look at the concept of a "rolling shutter" versus a "global shutter," and why it makes such a difference in the images that each produces. For clarification, CCDs use a global shutter, while CMOS sensors use either a rolling or global shutter; as a practical matter, all of the current camcorders on the market at the time of this writing use rolling-shutter technology. A global shutter exposes the entire imager simultaneously. The entire frame is exposed and begins gathering light; when the predetermined "shutter speed" has elapsed, the sensor stops gathering light and turns its current exposure into an electronic image.
There's no physical "shutter" that covers and uncovers the sensor; it's all done with timing. At the start of exposure, the entire sensor starts gathering light. At the end of exposure, the light-gathering circuitry is turned off, and the contents of the sensor are "read out" to become an image.
A rolling shutter is very different. The rolling shutter actually exposes a small window of the frame, "rolling"
this window through the frame. Again, it's not an actual physical moving shutter that's doing this (as opposed to a movie camera, which has a moving physical shutter). Instead, the sensor is telling different portions to become light-sensitive at different moments in time, and this process proceeds down the course of the full frame, until the entire frame is exposed. If you've ever watched a photocopier scanning a document, that's pretty much how a rolling shutter works.In order to understand the various image artifacts that arrive, the important thing to note here is that in the rolling shutter, different portions of the frame are exposed at different times than other portions. The top of the frame will be exposed before the bottom of the frame, for example.
this window through the frame. Again, it's not an actual physical moving shutter that's doing this (as opposed to a movie camera, which has a moving physical shutter). Instead, the sensor is telling different portions to become light-sensitive at different moments in time, and this process proceeds down the course of the full frame, until the entire frame is exposed. If you've ever watched a photocopier scanning a document, that's pretty much how a rolling shutter works.In order to understand the various image artifacts that arrive, the important thing to note here is that in the rolling shutter, different portions of the frame are exposed at different times than other portions. The top of the frame will be exposed before the bottom of the frame, for example.If the subject or the camera were to move during the exposure, the result would be reflected in the frame as one of the three rolling shutter artifacts-skew, wobble or partial exposure. A global shutter avoids those problems; since all areas are exposed simultaneously with a CCD's global shutter, your footage will be immune from skew, wobble or partial exposure. Both systems are subject to motion: blur, however. If an object moves during the exposure, any motion during the frame results in blurring of that motion.
THE EFFECT OF SHUTTER SPEED
Most shooters are aware that shortening the shutter speed reduces the amount of motion blur. Indeed, when an extremely short shutter speed (such as 1/2000 see.), there likely will be no motion blur whatsoever. Even so, there's still something new to learn here: With a rolling-shutter camera, there actually are two factors at work, and they're not related to each other. There's a general shutter speed, but there's also the scanning speed (or the read/reset time. The shutter speed controls how long each line exposes for and, therefore, can control motion blur as on a global shutter. However, the shutter speed has no effect on how quickly the rolling shutter "rolls through" the frame-which means that shorter shutter speeds have no effect on skew or partial exposure. The faster the read/reset time, the milder the skew/wobble/partial-exposure artifacts should be.
SENSOR ARTIFACTS: SMEAR, SKEW, WOBBLE AND PARTIAL EXPOSURE
For the purpose of this article, we're classifying the four main artifacts that imaging sensors may exhibit. CCDs can suffer from vertical smearing on bright light sources, while CMOS sensors are immune to that artifact. But rolling-shutter CMOS sensors can exhibit skew, wobble and partial exposure; CCD sensors are immune to those effects. (Note: A CMOS sensor with a global shutter also would be immune to them, but since no current CMOS camcorders are equipped with global shutters, shooters need to be aware of what the implications of a rolling shutter would be.)
CCD: Smear. Smear occurs when a very bright portion of an image causes an entire column of pixels to overload and bloom to white. Any bright point of light can potentially cause smear; common offenders include street lights and car headlights. It also can happen
if the camera is shooting footage of a camera flash or even the sun. Avoiding smear involves lowering the exposure enough that the bright lights don't bloom and trigger a column of smearing. Stopping down the iris to bring down the brightness of the bright lights can eliminate smear entirely, but also may cause. the overall picture to be too dark. Smear also is one of the dead giveaways that your production was shot on video rather than on film; film doesn't "smear" like this. Controlled lighting can eliminate all traces of smear, but in uncontrolled circumstances, it will happen. CMOS sensors function differently and are immune to smear.
if the camera is shooting footage of a camera flash or even the sun. Avoiding smear involves lowering the exposure enough that the bright lights don't bloom and trigger a column of smearing. Stopping down the iris to bring down the brightness of the bright lights can eliminate smear entirely, but also may cause. the overall picture to be too dark. Smear also is one of the dead giveaways that your production was shot on video rather than on film; film doesn't "smear" like this. Controlled lighting can eliminate all traces of smear, but in uncontrolled circumstances, it will happen. CMOS sensors function differently and are immune to smear. The propensity of a CCD to smear is directly related to the size of the pixels on the chip. The smaller the pixels, the more likely that smear will happen. Since smear is the one artifact that affects CCDs, and larger pixel size minimizes smear, it's more understandable why the larger-chip digital cinema cameras from DALSA, Panasonic and Sony all use CCD sensors-their pixels are large enough to minimize or even overcome any smear issues. Smear is much more likely to happen on a smaller-chip camera where the pixels are correspondingly smaller.
CMOS: Skew. The easiest way to understand skew is to think about taking a photo of a tall building. As the sensor scans down the frame, the building is "drawn" onto the chips and is represented accurately. But if the camera were to pan horizontally while the sensor was still exposing, what would happen? The top of the building would be on one side the frame, and the bottom of the building would be on the other.The whole building would look like the Leaning Tower of Pisa!
Skew is probably the most well-known rolling-shutter effect. With horizontal camera motion, vertical objects can look l
ike they're leaning one way or the other, depending on which way the camera (or the subject) is moving. Skew perhaps may be the least objetionable of the rolling-shutter artifacts, since it typically only exhibits during a fast pan or with a slower frame rate. Also, skew can happen if the camera is stationary; fast-moving objects through the frame will exhibit just as much skew. It doesn't matter whether it's the subject that's moving or the camera-all that matters is the amount of motion relative to the frame; the more motion, the more skewing. The faster the read/reset time, the milder the skew should be.
The amount of skew that occurs is dependent on how much motion there is relative to the frame. Skew is at its worst when you zoom in to full telephoto and pan rapidly; the more telephoto the lens and/or the faster the pan, the quicker the relative motion of the subject through the frame, and therefore the more skewed it will be. And it becomes even more noticeable if you reverse direction. Exercise caution when using a rolling-shutter CMOS camera to do motion tracking for postproduction work! Skew can play havoc with motion tracking for special-effects compositing, so if you plan on doing compositing and motion tracking, it's vital to be aware of the effect that skew can have and take efforts to minimize it.
CMOS: Wobble. This is probably the most talked about rolling-shutter effect. Wobble is related to skew and is caused by the same root causes, but wobble is far more troublesome. Whereas skew represents a leaning of vertical objects, wobble is a stretchy/rubbery look that happens to the video. Wobble is more likely to occur in handheld footage or in situations where the camcorder is subject to vibration or sudden motion. To imagine how wobble might affect the scanning of an image, picture the idea of shooting a scene while tilting the camera down. If the camera moved fast enough, it may be possible for you to "race" the rolling shutter down the frame, and thus the camera would "stretch" any object that was being filmed-it would keep repeating that same section over and over, pulling and stretching the object down the frame.
Or when moving upward, the image might become artificially squashed or compressed. Now, let's throw in the concept of handheld motion (a little up, a little down, some tilting or diagonal movement, etc.), and you can see where the "wobble" comes in. As the camcorder is moving downward, the image becomes stretched out. When it moves upward, the image gets scrunched down, and as it moves side to side, it leans one way or the other. The result is that your entire frame becomes "wobbly." A way to describe it might be to say that it looks like you're shooting through a layer of gelatin. For a relatively static shot, this obviously isn't a big deal. For a camera mounted on a tripod, it shouldn't be a problem, but the more motion that's involved, and especially motion that reverses itself (like a bump or jolt), the worse it can become.
It can become a problem for unintentional motion, as well. A lightweight camera being jiggled by the wind, for example, or mounted on a car or in a helicopter, or rapid handheld running or bouncing all could create conditions where the footage turns out rubbery or wobbly. Not all footage is ruined by excessive wobbling, as it depends largely on the type of motion of the camera and the speed and direction of that motion. Direct linear motion (such as a tripod pan) shouldn't cause any wobble at all (just skew), but handheld footage or footage subject to vibration is just going to wobble. Slower frame rates seem to accentuate the wobble; faster frame rates seem to minimize it. Higher shutter speeds make it more distinct and slower shutter speeds mask the wobble under motion blur, but the wobbling effect is equivalent regardless of shutter speed.
The amount of wobble you get all depends on how the camera moves during a shot and how quickly it moves (or how quickly the subject is moving; the wobble doesn't distinguish between camera movement and subject movement. Under most circumstances, you shouldn't see or notice wobble at all, but there are just certain types of shots that wobble will ruin.
CMOS: Partial Exposure. One of the more pernicious rolling-shutter artifacts, partial exposure can affect the footage from many different shooting scenarios. It usually occurs under circumstances where there's a flash of light that significantly alters the exposure-a camera flash, a bolt of lightning, fireworks, strobe lights, etc. In these cases, if Y<:JU look at how the rolling-shutter system works, you can easily” see how part of the image would be dark and the rest of the image would be bright-as the shutter "rolls" through the frame, it exposes portions of the frame at the prevailing light conditions, and then when the "flash" occurs, the current portion of the frame that the shutter's exposing will be brightly lit. The result is that when you use a rolling-shutter camera in a scenario where flash photography is taking place (such as a wedding, a press conference, a red-carpet premiere, etc.), it's possible that you'll encounter black or dark bands, or even bright bands, in your video. Another situation of concern for rolling shutters is when shooting under fluorescent or HMI lighting.
Modern fluorescent lighting uses high-frequency ballasts and produces a largely continuous stream of light, but magnetic-ballast fluorescents use slow ballasts and, as such, the intensity of the light could vary significantly over the course of 1/60 sec. This varying intensity can play havoc with a rolling-shutter camcorder. Depending on your shutter speed, you could see rolling, rippling waves running through your footage or it can even result in thick orange bands scrolling through your footage (at very short shutter speeds)! The same applies to HMI lighting; when using high-speed electronic ballasts, the rolling shutter should perform well, but when using magnetic ballast, the same restrictions need to be observed. Be cautious about shooting under fluorescent or HMI lighting with a rolling-shutter camcorder, and try to use only high-frequency-ballast fixtures. If you can't avoid magnetic-ballast fixtures and you have to shoot with a rolling-shutter CMOS camcorder, you can try using a shutter speed that's a multiple of the light's refresh rate to eliminate the bands (I recommend using 1/40 or 1/60 see.). Don't use 1/48 sec. with a magnetic-ballast HMI or fluorescent, and don't use a relative shutter speed in angles. Slow ballasts also could prove problematic for global shutters if the shutter speed isn't in phase with the cycle of the light.
That's why it's recommended to shoot using the same shutter speed as the lights (ie., in PAL territories, you'd use a 1/50 sec. exposure to match the 50 Hz lights; in NTSC territories, you'd use 1/60 sec. to match the 60 Hz lights). A global-shutter system running with a shutter speed that's out of phase with the lights (50 Hz shutter under 60 Hz lights, for example) can exhibit some general flickering or color shifting under fluorescent lights. The rolling-shutter system exhibits a very different artifact bands of shifting color scrolling in the image or darker lines flickering in and out on the image.
Magnetic HMls also can cause flicker with film and have resulted in people publishing "HMI-safe" filming speeds for film cameras. These charts show certain frame rates that are immune to the HMI pulsing effect for film. Understand that these charts have no applicability to a rolling-shutter camera. There's no such thing as an HMI-safe frame rate when shooting with a rolling shutter! Instead, it's vital to control the shutter speed. There are HMI-safe shutter speeds, but there aren't HMI-safe frame rates when working with a rolling-shutter or read-reset shutter.
SUMMARY:
The CCD artifact of smear is well known, and for the most part, shooters know how to deal with it and minimize or avoid it. But the new artifacts/ characteristics that come from the CMOS rolling-shutter technology may provide some nasty challenges for unprepared shooters. Camera manufacturers likely will continue to offer more and more CMOS camcorders. Shooters, educate yourselves as to what the issues are, how they affect your footage and prepare yourselves for whatever steps you need to take to make sure that there are no unwelcome surprises.
ike they're leaning one way or the other, depending on which way the camera (or the subject) is moving. Skew perhaps may be the least objetionable of the rolling-shutter artifacts, since it typically only exhibits during a fast pan or with a slower frame rate. Also, skew can happen if the camera is stationary; fast-moving objects through the frame will exhibit just as much skew. It doesn't matter whether it's the subject that's moving or the camera-all that matters is the amount of motion relative to the frame; the more motion, the more skewing. The faster the read/reset time, the milder the skew should be. The amount of skew that occurs is dependent on how much motion there is relative to the frame. Skew is at its worst when you zoom in to full telephoto and pan rapidly; the more telephoto the lens and/or the faster the pan, the quicker the relative motion of the subject through the frame, and therefore the more skewed it will be. And it becomes even more noticeable if you reverse direction. Exercise caution when using a rolling-shutter CMOS camera to do motion tracking for postproduction work! Skew can play havoc with motion tracking for special-effects compositing, so if you plan on doing compositing and motion tracking, it's vital to be aware of the effect that skew can have and take efforts to minimize it.
CMOS: Wobble. This is probably the most talked about rolling-shutter effect. Wobble is related to skew and is caused by the same root causes, but wobble is far more troublesome. Whereas skew represents a leaning of vertical objects, wobble is a stretchy/rubbery look that happens to the video. Wobble is more likely to occur in handheld footage or in situations where the camcorder is subject to vibration or sudden motion. To imagine how wobble might affect the scanning of an image, picture the idea of shooting a scene while tilting the camera down. If the camera moved fast enough, it may be possible for you to "race" the rolling shutter down the frame, and thus the camera would "stretch" any object that was being filmed-it would keep repeating that same section over and over, pulling and stretching the object down the frame.
Or when moving upward, the image might become artificially squashed or compressed. Now, let's throw in the concept of handheld motion (a little up, a little down, some tilting or diagonal movement, etc.), and you can see where the "wobble" comes in. As the camcorder is moving downward, the image becomes stretched out. When it moves upward, the image gets scrunched down, and as it moves side to side, it leans one way or the other. The result is that your entire frame becomes "wobbly." A way to describe it might be to say that it looks like you're shooting through a layer of gelatin. For a relatively static shot, this obviously isn't a big deal. For a camera mounted on a tripod, it shouldn't be a problem, but the more motion that's involved, and especially motion that reverses itself (like a bump or jolt), the worse it can become.
It can become a problem for unintentional motion, as well. A lightweight camera being jiggled by the wind, for example, or mounted on a car or in a helicopter, or rapid handheld running or bouncing all could create conditions where the footage turns out rubbery or wobbly. Not all footage is ruined by excessive wobbling, as it depends largely on the type of motion of the camera and the speed and direction of that motion. Direct linear motion (such as a tripod pan) shouldn't cause any wobble at all (just skew), but handheld footage or footage subject to vibration is just going to wobble. Slower frame rates seem to accentuate the wobble; faster frame rates seem to minimize it. Higher shutter speeds make it more distinct and slower shutter speeds mask the wobble under motion blur, but the wobbling effect is equivalent regardless of shutter speed.
The amount of wobble you get all depends on how the camera moves during a shot and how quickly it moves (or how quickly the subject is moving; the wobble doesn't distinguish between camera movement and subject movement. Under most circumstances, you shouldn't see or notice wobble at all, but there are just certain types of shots that wobble will ruin.
CMOS: Partial Exposure. One of the more pernicious rolling-shutter artifacts, partial exposure can affect the footage from many different shooting scenarios. It usually occurs under circumstances where there's a flash of light that significantly alters the exposure-a camera flash, a bolt of lightning, fireworks, strobe lights, etc. In these cases, if Y<:JU look at how the rolling-shutter system works, you can easily” see how part of the image would be dark and the rest of the image would be bright-as the shutter "rolls" through the frame, it exposes portions of the frame at the prevailing light conditions, and then when the "flash" occurs, the current portion of the frame that the shutter's exposing will be brightly lit. The result is that when you use a rolling-shutter camera in a scenario where flash photography is taking place (such as a wedding, a press conference, a red-carpet premiere, etc.), it's possible that you'll encounter black or dark bands, or even bright bands, in your video. Another situation of concern for rolling shutters is when shooting under fluorescent or HMI lighting.
Modern fluorescent lighting uses high-frequency ballasts and produces a largely continuous stream of light, but magnetic-ballast fluorescents use slow ballasts and, as such, the intensity of the light could vary significantly over the course of 1/60 sec. This varying intensity can play havoc with a rolling-shutter camcorder. Depending on your shutter speed, you could see rolling, rippling waves running through your footage or it can even result in thick orange bands scrolling through your footage (at very short shutter speeds)! The same applies to HMI lighting; when using high-speed electronic ballasts, the rolling shutter should perform well, but when using magnetic ballast, the same restrictions need to be observed. Be cautious about shooting under fluorescent or HMI lighting with a rolling-shutter camcorder, and try to use only high-frequency-ballast fixtures. If you can't avoid magnetic-ballast fixtures and you have to shoot with a rolling-shutter CMOS camcorder, you can try using a shutter speed that's a multiple of the light's refresh rate to eliminate the bands (I recommend using 1/40 or 1/60 see.). Don't use 1/48 sec. with a magnetic-ballast HMI or fluorescent, and don't use a relative shutter speed in angles. Slow ballasts also could prove problematic for global shutters if the shutter speed isn't in phase with the cycle of the light.
That's why it's recommended to shoot using the same shutter speed as the lights (ie., in PAL territories, you'd use a 1/50 sec. exposure to match the 50 Hz lights; in NTSC territories, you'd use 1/60 sec. to match the 60 Hz lights). A global-shutter system running with a shutter speed that's out of phase with the lights (50 Hz shutter under 60 Hz lights, for example) can exhibit some general flickering or color shifting under fluorescent lights. The rolling-shutter system exhibits a very different artifact bands of shifting color scrolling in the image or darker lines flickering in and out on the image.
Magnetic HMls also can cause flicker with film and have resulted in people publishing "HMI-safe" filming speeds for film cameras. These charts show certain frame rates that are immune to the HMI pulsing effect for film. Understand that these charts have no applicability to a rolling-shutter camera. There's no such thing as an HMI-safe frame rate when shooting with a rolling shutter! Instead, it's vital to control the shutter speed. There are HMI-safe shutter speeds, but there aren't HMI-safe frame rates when working with a rolling-shutter or read-reset shutter.
SUMMARY:
The CCD artifact of smear is well known, and for the most part, shooters know how to deal with it and minimize or avoid it. But the new artifacts/ characteristics that come from the CMOS rolling-shutter technology may provide some nasty challenges for unprepared shooters. Camera manufacturers likely will continue to offer more and more CMOS camcorders. Shooters, educate yourselves as to what the issues are, how they affect your footage and prepare yourselves for whatever steps you need to take to make sure that there are no unwelcome surprises.
