Login or Join Now

Upload your photos, chat, win prizes and much more

Username:
Password:
Remember Me

Can't Access your Account?

New to ePHOTOzine? Join ePHOTOzine for free!

Like 0

Misleading ISO testing

Join Now

Join ePHOTOzine, the friendliest photography community.

Upload photos, chat with photographers, win prizes and much more for free!

pulsar69
pulsar69  101611 forum posts United Kingdom6 Constructive Critique Points
15 Jan 2013 - 10:38 PM

I would like to open a can of worms here and see what pops out.

Including EPZ all the magazines and websites love to display colour checker charts of the ISO performance from the latest cameras - all of which are shot in natural daylight or artificial lighting which recreates the same ( i presume - please correct me if wrong )

ISO performance in my field in particular, wedding photography, is extremely important for LOW LIGHT situations , and i cannot say i have ever had any need whatsoever for ISO 3200 , 12800 etc etc in normal lighting situations ...

Commercially for the camera companies it looks fab, EPZs very own test of the Canon 1DX is marvellous at 51200 under normal light .... however ( and hats off to EPZ for putting this on ) the photo at ISO51200 taken of a night scene in a real low light situation is comparatively rubbish and is not to me a clean useable image.

So why do people insist on testing low light capability in good light - there may be a completely reasonable explanation for this from sports wildlife togs etc - do you guys need to shoot ISOs that high during the day ?

I still get the feeling with all the new cameras coming out that the era of the 5D2 and the D700 were in ISO terms so massively groundbreaking and everything that has followed so far just hasn't been able to make such big leaps and has lots of new bells and whistles but the real meat of the cameras isnt that much better ?

Sponsored Links
Sponsored Links 
15 Jan 2013 - 10:38 PM

Join ePHOTOzine for free and remove these adverts.

Pete
Pete Site Moderator 1318433 forum postsPete vcard ePz Advertiser England96 Constructive Critique Points
15 Jan 2013 - 10:51 PM

Interesting point.
We do it because it's a comparison of ISOs at different settings - so you can get a standard feel for each ISO setting and the level of noise introduced as the ISO increases. As the set up is consistent you can also then compare camera A with camera B in a like for like environment. I truly believe the version we do is valid, but there's nothing stopping us adding a low light low noise test as an addition.

Sooty_1
Sooty_1 Critique Team 41177 forum posts United Kingdom196 Constructive Critique Points
15 Jan 2013 - 11:01 PM

It might be just as valid to show what can be achieved with PP noise reduction too, because shooting in very low light will create noise whatever, but different software and different settings will make at least as much differrence to your final output as the marginal difference between new cameras in the same market area.
Skill of using said software will also make a difference.

Oh for the simple days of shooting 100 for outdoors and 400 indoors.

Tongue

Nick

pulsar69
pulsar69  101611 forum posts United Kingdom6 Constructive Critique Points
15 Jan 2013 - 11:08 PM

I totally agree with you Pete the version you do is definitely valid and it gives us a comparison between the cameras, I do believe personally for my type of work a new testing standard in some sort of low light would be very useful indeed. I suppose the question will be what the testing conditions would need to be and how to keep them consistent.

The low light shooting in a wedding varies from a maybe cloudy dull day in a room with narrow windows and curtains and poor lighting in a ceremony , to full blow near darkness depending on the disco lighting during the first dance , I have even photographed a candlelit ceremony on one occasion.

We should ask the many knowledgeable togs on here and try and develop a system which can test it consistently do you think ?

pulsar69
pulsar69  101611 forum posts United Kingdom6 Constructive Critique Points
15 Jan 2013 - 11:11 PM

I suppose it could be as simple as performing the test in a black out room or in darkness to cut out ambience and then using a lighting system powered down to a low enough light ?

Pete
Pete Site Moderator 1318433 forum postsPete vcard ePz Advertiser England96 Constructive Critique Points
15 Jan 2013 - 11:20 PM

Yes - whatever we do needs to be controlled other wise it becomes a meaningless comparison.

janeez
janeez e2 Member 51174 forum postsjaneez vcard United Kingdom8 Constructive Critique Points
15 Jan 2013 - 11:58 PM

I would think that using controlled dimmable daylight lamps in a room to simulate, all be it, artificial daylight could create at least a consistent lighting condition.

User_Removed
16 Jan 2013 - 10:28 AM

Purely a personal observation:

For me, there are two main reasons for using a high ISO setting:

1. To allow hand-held shooting in low light (e.g. street scenes at night)

2. To allow a fast shutter speed in normal light without having to use a very fast aperture. (e.g. for birds in flight)

Only very rarely, for either of those purposes, do I have to exceed ISO6400 and, with my current Nikon D800, there is virtually no discernible noise up to that setting.

However, with previous Nikons (D80, D300, D3s) I have noticed that high-ISO noise was more noticeable at any given setting when used for Purpose 1 (above) than when used for Purpose 2. In other words, the lower the lighting level, the more noticeable high-ISO noise appeared to be.

As an example, with the D300, a good-light image of a bird in flight taken at ISO1600, 1/2000th and f/8 would show much less noise than a poor light image of a street scene shot at ISO1600, 1/125, f5.6.

.

pulsar69
pulsar69  101611 forum posts United Kingdom6 Constructive Critique Points
16 Jan 2013 - 10:56 AM

I agree and that is my point ,to make a truly accurate determination of ISO performance requires a test which is done in low light as this is when the noise is most prevalent and noticeable, there are not many times that cameras of todays quality need to shoot at iso 6400 or above during the day and some cameras as you say can exhibit bad noise right from 3200 up in lower light ..

It would be great if EPZ could lead the way and start performing these tests - they would i think reading the thousands of reviews and forums out there be of so much more use and interest ..

samueldilworth
16 Jan 2013 - 11:35 AM

At handheld shutter speeds, and considerably beyond, thermal noise – the type that gets worse with longer exposure – is negligible. So if you halve your scene brightness but double your exposure time, you end up with the same amount of noise.

From this it follows that the brightness of the scene doesn’t matter for noise. What matters is exposure: the amount of light falling on the sensor. You can give any exposure you want by adjusting shutter speed and f-number.

The reason that cameras perform much worse in real-world low-light conditions than they typically do in studio noise comparisons is not that the brightness is lower in the real world (although it often is), but that real-world low-light scenes typically come with two other problematic characteristics that studio scenes avoid: high contrast and skewed spectrum. These characteristics prevent you from giving the whole scene as much exposure as you’d like.

You can make this observation for yourself by doing a search on Google Images for ‘low light’, like this.

There you see hundreds of high-contrast photos, most with a small tungsten-lit (or worse!) area amid a sea of deep shadows. Those shadows have little exposure, and thus high noise. And because they’re often large and featureless, the noise is easy to see.

The bright areas, if lit by tungsten, have low exposure in the blue channel. If lit by sodium-vapour lamps, fluorescent lamps, LED lamps, laser stage lighting, etc., then there will be chunks of the spectrum that are dimly or not-at-all present (or greatly ‘underexposed’). Visible noise results.

So to test low-light performance in a studio you need to do two things: first, produce a scene with high contrast and large deep-shadow areas (you could do this by raking a collection of objects with hard light from the side, if necessary by using a flag (gobo); and second, light your scene with tungsten-balanced light, sodium-vapour light, or ideally a variety of light sources commonly encountered in real-world low-light environments.

pulsar69
pulsar69  101611 forum posts United Kingdom6 Constructive Critique Points
16 Jan 2013 - 3:02 PM

Often in the real world low light photography is much more restrictive than that and the ability to change f stop shutter speed or ISO are either small or non existent. Take for an instance a typical ceremony room with tungsten lights 20 or so feet up narrow georgian windows and a typically overcast british day.

Shooting below say F4 will mean at the angle you normally end up either a bride or groom will be out of focus , so F4 could be your maximum aperture , you know your camera through experience performs poory after iso3200 so that is the maximum you can use that at , given those variable are maxed out the shutter speed then has to correct for the exposure which could quite often end up at less than 1/100 sec in these circumstances. Using a 24-70 lens at 70mm you are then on the edges of a sharp or not sharp shot.

Hence the reason for needing to know exactly how well the iso is handled on new cameras for me because all else aside a usable iso a couple of stops higher makes a massive amount of difference.

Your studio test sounds very good and i would expect a simple tungsten light from above would be a great starting point ?

LenShepherd
LenShepherd e2 Member 62434 forum postsLenShepherd vcard United Kingdom
17 Jan 2013 - 7:54 AM

Perhaps the comment behind the comment is to do with modern DSLR's significantly increasing the range of photographs that can be taken without flash in low light, and many test reports not taking sufficient account of new camera ability to keep keep advanced users happy.
Should there be a tungsten light test? Most modern cameras can perform much better than in the film area, so why not?
Should there be a test to measure the reduction in resolution and dynamic range as ISO is increased? While each new generation of camera is improving in this respect why is it usual only to test for noise at higher ISO's? I regard all three as useful information.
Should there be an autofocus test? Generally each new generation of cameras does this better, and high-end cameras do this better still, so why not?
There is a balance between the time and cost of producing more in-depth reviews, and increasing numbers of photographers not reading reviews that do not go into more detail than that at ephotozine.
A two stage test might increase site traffic. The first stage test might be "quick to complete" similar to the present one; followed by a more in-depth review taking into account the low light tungsten performance and autofocus inability.
What do other readers think?

mikehit
mikehit  46171 forum posts United Kingdom9 Constructive Critique Points
17 Jan 2013 - 8:55 AM

The other thing to consider is final output. Prints are far more forgiving of noise than the computer screen image and as most professional pictures are printed at some point you may have differing sets of variables between users.

User_Removed
17 Jan 2013 - 10:26 AM


Quote: Perhaps the comment behind the comment is to do with modern DSLR's significantly increasing the range of photographs that can be taken without flash in low light, and many test reports not taking sufficient account of new camera ability to keep keep advanced users happy.
Should there be a tungsten light test? Most modern cameras can perform much better than in the film area, so why not?
Should there be a test to measure the reduction in resolution and dynamic range as ISO is increased? While each new generation of camera is improving in this respect why is it usual only to test for noise at higher ISO's? I regard all three as useful information.
Should there be an autofocus test? Generally each new generation of cameras does this better, and high-end cameras do this better still, so why not?
There is a balance between the time and cost of producing more in-depth reviews, and increasing numbers of photographers not reading reviews that do not go into more detail than that at ephotozine.
A two stage test might increase site traffic. The first stage test might be "quick to complete" similar to the present one; followed by a more in-depth review taking into account the low light tungsten performance and autofocus inability.
What do other readers think?

As always, Len, a lot of sound common sense in what you say.

Ever since the advent of digital photography on a consumer scale, my over-riding ambition was to find a digital camera (affordable) that would give me as good photographs as I used to get from film.

That took a very long time coming - but, with the Nikon D800, I think that I have finally got there.

Now there is a lot of subjectivity in that assessment. With the D300 and D3s it was beginning to be only fairly marginal deficiencies that still had me thinking that digital had not quite caught up with film for image quality. With the D800 I cannot honestly say that I achieved even better image quality from film than I now can get with digital.

What a subjective assessment of image quality does not take account of, of course, is improvements (or disimprovements) in camera functions over the transition from film SLRs to modern dSLRs. The one major disadvantage that dSLRs still haven't cracked is the superb manual focussing capabilities of 1980s SLR cameras. But in most other respects we are better off now. High ISO performance is one such area.

So, what am I really saying? I think that it is, for me, I prefer to assess camera performance according to my own personal criteria rather than paying very much attention to what magazines or manufacturers might publish in the way of "test results". Basically I trust my own judgement far more than I would trust that of a jumped-up journalist conducting tests for a magazine or website or some marketing/PR jobsworth (mis)interpreting the results obtained by some low-level technician at a test-bench. Even, in the extremely unlikely event that what we were seeing was a genuine scientific test, with all the necessary controls, by a multi-PhD scientist, I would still worry that maybe the things she was testing were not the things that were of most importance to my pursuit of a photography hobby.

Or maybe I just don't want to think that some plonker in a laboratory or a magazine office has any influence over my art.

Last Modified By User_Removed at 17 Jan 2013 - 10:28 AM
samueldilworth
17 Jan 2013 - 7:45 PM


Quote: Often in the real world low light photography is much more restrictive than that and the ability to change f stop shutter speed or ISO are either small or non existent.

Oh, I agree. I just meant to reinforce the fact that noise doesn’t depend on the scene brightness itself, as is commonly thought, but instead on the exposure (the total amount of light collected at the sensor).


Quote: i would expect a simple tungsten light from above would be a great starting point ?

It would. If I were designing the test I’d make sure there were both shadow areas with fine detail, and pitch-black shadow areas to reveal pattern noise (banding, etc.).

Add a Comment

You must be a member to leave a comment

Username:
Password:
Remember me:
Un-tick this box if you want to login each time you visit.