[Leica] Comments on FOM2 judging: some details

[Alastair Firkin <firkin@ncable.net.au>]

G'day Andrew,

>I really think there has been a colosal mis-communication
>here.
>
>The original 8 Dec 2001 email I received from you clearly shows
>that only one of my submissions - the 360-deg VR - was accepted.

Andrew,

well this confuses me as well. I'll check the records, but maybe you 
got the batched reply, not the one intended for those who had the 
grouped selection. More will follow privately

>
>Okay - that deals with issue #1 I had with FOM2.
>Issue #2 still stands though.  The judging...
>
>>  I would welcome comments on how to choose the images fairly
>
>1.  Spell out clearly and unambiguously how the judging is
>     actually done.  Is it by popular online vote?  Are
>     there actual judges?  A mixture of the two?
>
>2.  If there are judges, name who they are.
>
>3.  Beneath each accepted image, either have links to the
>     judges' reasoning for why the image was chosen, or else
>     simply show how many votes the image got.

Judging is NOT easy. I tried to get Leica to provide some sort of 
"professional". As you can imagine, I was not game to do it myself ie 
alone. I asked everyone involved on the project to consider helping 
with the selection. The decision to make an internal group judgement 
was made at the start of the project. Everyone registered can submit 
image, and can have a say in the yearly selections by judging in 
November. About 30 "signed" up to judge. It was no easy matter. 
Viewing images alone for 30 seconds meant that the selection took 
over 6.5 hours without allowing for "internet" delays. Now take into 
account that clearly most of the selections were made by viewing the 
images many times, the task is pretty large. I sent everyone a 
thumbnail package, so that they could view the images on the net, and 
then review them on their own. From the comments I received, and the 
overall variety of images chosen from different "locations", I was 
happy that the judges had at least looked at all or most of the 775 
images carefully.

I then asked them to choose a the best images. Right now I cannot 
remember exactly, but I had done some mathematical analysis at the 
time given the number of images and the number of judges. I think I 
asked for at least 10 and up to 30 (I may be wrong, but the principle 
is accurate) selections. This allowed some people, who felt there 
were only 12 "worthy" images to select them, and not force lesser 
selections, and allowed those who wanted to analyze more to delve 
deeper.

The images chosen were to be the "best" images in the view of the 
judge. I did give a summary of the aims and ideas of the project to 
each judge and some overall guidelines, but left most of it to the 
group themselves.

At one time I considered "comments", but to be honest, there is no 
hope of me gathering, collating and publishing comments on each and 
every image, or even of those chosen --- too much work to ask of 
judges, and too much work here.

I did get some 'off the cuff' comments about certain images, and 
these were passed on in private to the photographers.

I could publish a list of those who are judging -- but I'm NOT going 
to publish each individual selection. The images are put up as 
anonymous to avoid personal "attacks" and the selections of those 
willing to do the hard work of selection will have to remain the same.

I was careful to ensure we had quite a number of the "professionals" 
making selections, but to be honest, the end product was not too 
different. I wanted every image to get at least one vote from every 3 
judges, but in the end the variety meant that most of the images 
received a vote from one in 4 judges. At that level, I took some 
editorial control. I noted groups of similar images like yours were 
attracting votes, but were "splitting" the judges. I decided that 
these "group" images might get included iff the individual image had 
votes, and the group themselves achieved a certain number of votes. 
There were 3 separate selections. Then from the one in 4 images, we 
had some left over. I removed images where the photographer already 
had selections chosen, in favor of those where the photographer had 
no other images selected. This allowed me to increase the number and 
variety of photographers. I was intending to hold the good but just 
missed images back, and put them into this years "pool", but to be 
honest, I had a major computer mishap, and some of the judging 
details which I had accumulated was lost -- well after the choice was 
made. You know the problem. I thought it was backed up, but I'd 
backed up last weeks version. Any way I rescued most of the details 
from my e-mail records, but whether I can find the time and rescue 
the raw data on the voting is yet to be seen.

This was the fairest, simplest and most effective solution I could 
find last year, and most of the comments from the judges suggested 
that it worked alright. I'm still hoping to refine it. I still want 
to have some final editorial rights, just to keep the final resulting 
exhibition balanced, but remember year one is only 20% of the 
projected total.

In summary. If you are involved in the project, you can be part of 
shaping its outcome. If you help me shape the selection, you will not 
be subject to personal attack, because the results will remain with 
me. I will keep an overall review on all selections, and I do try to 
vet them: for example, the Leica representative sent in a selection 
of 5 images, 2 of which NOONE else had chosen (one was mine) and all 
of which had come from the "first" batch. I decided that this was not 
adequate input, and that by his own admission, he had not had time to 
really look at the total 775  images. I discarded his votes, and saw 
my wonderful image ignored :-)  Every other voluntary judge did a 
wonderful and I believe honest job. Only a handful of images stood 
out above the others. Many photographers missed out all together -- 
not one of them has complained --- there have been comments of 
disappointment, and some of resolve to do better.


My thanks again to all involved. I am quite happy to entertain all 
comments on the judging process.

Cheers


- -- 
Alastair Firkin

http://www.afirkin.com

http://www.familyofman2.com

  ()" "'()
  (   '0',)
(,,)(")(")
- --
To unsubscribe, see http://mejac.palo-alto.ca.us/leica-users/unsub.html