The ICGA Process

[BB+]

Since the ICGA Wiki is now (mostly) open, I might make these comments about the process.

*) I provided the bulk of the "hard evidence" and analysis therein. Other parts were made by Zach Wegner (now a Rybka competitor, though much of his analysis pre-dates that label), while Bob Hyatt (who is said to have a "Rybka animus") gave some narrative background concerning the Rybka 1.6 and Crafty evidence. Wylie Garvin "kept me honest" by looking at a lot of the ASM dumps, and suggesting alternative explanations in some cases (such as with the quad()-usage scaling).

*) There were various persons who voiced their doubts at the beginning. The most notable was Marcel van Kervinck, whose prodding led to the production of the Rybka 2.3.2a versus Rybka 1.0 Beta evaluation comparison (he had suggested that they were rather different, and the latter pre-dated any ICGA involvement), and his further comments led to the "quantification of evaluation features" [which I still admit is not without difficulties/flaws, but no one has taken up an alternative analysis]. After this analysis was produced, he agreed with the principal Panel conclusion (that Rajlich had broken the ICGA rules). Charles Roberson was another who brought up the issue that "feature overlap" was quite common, suggesting (for instance) that EXChess and RESP also had various common elements as Fruit -- this obliged the Panel to be more quantitative in its exercise(s) [and I don't know if Roberson gave a final opinion]. Some Panel members, to the extent that it was possible beyond mere guesswork, at times raised possible defences for Rajlich -- e.g. Garvin played "devil's advocate" on an occasion or two. Undoubtedly the Panel tended toward a "prosecution" to some extent, but that's as much the fault of a (willful) lack of a defense as anything else.

The question of how long it took to write an engine from scratch was raised (the "sudden increase" in Elo), and also whether the Panel could reasonably expect Rajlich to provide source code. Another side-light concerned what "black-box" code should be allowed by the ICGA. The question of what version played in what event was inordinately discussed, but eventually there was agreement that Rybka 2.3.2a was sufficient as pertained specfically to the 2007 WCCC title. [The value of pre-Beta Rybka evidence, and even that of Rybka versions prior to winning the WCCC was also debated].

*) There were varying opinions about any sentence that should be handed down (to the extent that the Panel could suggest such). Here Gerd Isenberg's comment (included in the Panel report) is an example. Bob Hyatt was also somewhat opposed to rewriting the history books (as he has said elsewhere). By my count, there are 14 programmers who assented to the "crucial question" (Did Rajlich break the ICGA Rules?), and as stated elsewhere, no naysayers. Of these 14, at most half could be said to be his "competitors" in any real conflicting sense (ICGA-wise or commercially), and I would really say only 2 or 3 fall into that class. There were also a number of signers of the Open Letter who either didn't join the Panel, or remained silent throughout.

*) Rajlich was contacted multiple times, first with the opening of the process (and likely prior to that), then a second time concerning whether he wished to participate directly in the Panel discussion of evidence, and thirdly after the Panel report was made when he was asked to respond within a month directly to the Board. The Panel was (again) reminded during this time not to speak publicly about the Report until Rajlich had made a response. In short, he chose to provide no alternative (or even mitigating) explanation of the evidence presented.

[BB+]

To my count, the following Panel Members could be said to have a "conflict of interest" (even here some/most are debatable). I have put stars before those who were notable in their participation in the discussion.

Amir Ban (Junior is a direct competitor)
Ingo Bauer (Shredder team)
Eric Hallsworth (HIARCS team)
Kai Himstedt (ClusterToga moved up due to Rybka being demoted)
Gyula Horvath (competed directly a few times with Pandix, won Leiden ICT 2011 in Rybka's absence)
* Bob Hyatt (Rybka copied Crafty code)
Fabien Letouzey (Rybka copied Fruit code)
Stefan Meyer-Kahlen (Shredder)
Mark Uniacke (HIARCS programmer)
Harvey Williamson (HIARCS team)
* Zach Wegner (Rondo)

Here are those for which I think you have to stretch it:

Dariusz Czechowski (Darmenios was a direct competitor in 2010, but finished way back in any case)
Don Dailey (Komodo isn't commercial yet, and I think only competed once or twice against Rybka, in non-ICGA events)
Omir David Tabibi (Falcon was mid-pack in 2008)
Frederic Friedel (ChessBase operator -- they both sell Rybka and sometimes sponsor the WCCC)
Mincho Georgiev (author of Pawny)
* Gerd Isenberg (competed directly a few times with Isichess, but never finished close to Rybka)
Richard Pijl (The Baron was never in Rybka's league)
Ralf Schäfer (Spike in 2006 is now 4th instead of 5th)
Albert Silver (ChessBase news editor, see Friedel)
Christophe Theron (Chess Tiger has never competed against Rybka, and they are in different commercial markets)
* Marcel van Kervinck (I don't think Rookie is a real competitor to Rybka, yet)

And here are those that I would say do not (leaving out Board members):

Olivier Deville
* Wylie Garvin
Jan Krabbenbos
* Mark Lefler
Thomas Mayer
Tom Pronk
* Charles Roberson
Tord Romstad [if Stockfish were to compete, this could change]
Peter Skinner
Maciel Szmit
Ken Thompson
Richard Vida [if Critter were to compete, this could change]
Vladan Vuckovic
* Mark Watkins

So by my count, the plurality of those on the Panel are completely non-conflicted, and these mustered the bulk of the discussion (by far). Further, the majority of the panel could only rather loosely and/or pedantically be claimed to have a conflict. Other than Hyatt and Wegner (both of whom were principals in presenting evidence, but didn't necessarily do all that much vis-a-vis interpretation therein), none of the "clearly" conflicted group played much of a role.

Regarding another point about the Process that has come up, from the Charter:

3 Operation of the Panel

[a] The Panel shall conduct its discussions on a Web site forum closed to non-members.

So indeed, everyone who had access to the forum is essentially by definition a Panel member, and thus asking for the access to the Forum would evidently entail acceding to be described as such. One could presumably exit the Panel by de-registering from the wiki and notifying the Secretariat.

And incidentally, regarding:

The court makes up their own rules -- meaning that there are no rules.

Indeed, such is the case with e.g. Polish law

E.(ii) Standards of Proof
In Polish procedure there are no standards of proof and therefore there are no technical expressions in this regard.

See also: http://ec.europa.eu/civiljustice/eviden ... pol_pl.htm or http://lex.pl/serwis/kodeksy/akty/64.43.296.htm (Section 233).

[hyatt]

Nicely explained... as all of the things you have written with regard to this process have been...

It seems pretty clear that the evidence can not be impeached, due to the quantity and quality. So the idea now is to try to either impeach the process, or the participants, or both. Guilt or innocence of the accused is no longer a relevant point of justification. Let's just shoot the messenger, and pretend the message was never delivered, and the naysayers would suddenly be happy once again.

[BB+]

With a previous incident, Rajlich indeed asked to be judged by his peers rather than the "Board" as it were:

We propose setting up an appeals committee to handle this dispute. The last three Freestyle champions, Rajlich excluded, should vote on this issue. We will accept the verdict of our peers.

E.g. if applied to the ICGA case, the past champions who have spoken [even if you restrict to those that are not conflicted, they form a quorum] all seem to think he is guilty.

[BB+]

The question of whether Mark Lefler is a "competitor" of Rajlich has been raised (thus presumably making the whole Secretariat suspect). To my mind, this is beyond the realm of stretching the word. His chess engine Now last competed in the WCCC in 1999. The closest he came to competing against Rybka was when Scaramanga (a Now clone) competed in the Dutch Open this last year. I similarly can't imagine there is much in arguing that the Zillions product is somehow a Rybka competitor -- but I won't be surprised if such contortions would be produced.

On another issue, I continue to insist that the "Evaluation Feature" comparison is by no means a gold-standard, but no one seems eager to take up any alternative. This link: https://icga.wikispaces.com/Quantifying ... n+features can provide some background as to the thinking that went into it. There are some obvious simplifications in the statistical analysis [e.g., crudely using Gaussians rather than (say) student-t distributions when guesstimating a final 1 in 10^X number, ignoring correlations in the pairwise comparison, etc.], but the scope for criticism is certainly much wider than that. In any event, that document is freely available by now. [Actually, maybe I myself should re-read this and/or re-do the numbers in (say) 6 months, and see if my perceptions are still the same -- I am less desirous of reverse-engineering a competing commercial, but maybe one of them would be willing freely offer a code sample circa 2007].

Although source code never became an issue (I heard that Ken Thompson opined the Secretariat should simply ask for it, but I don't know whether any email contact with Rajlich even meandered to that stage), I agree that Rule #2 is not well-constructed, but it would seem to suggest that entrants which fail in this regard "may be declared invalid" [as noted above, the Panel was not in full agreement over whether requesting ex post facto code was reasonable]. Furthermore, if the Board had chosen to request source code (of whatever Rybka version for which this was feasible) and this was denied, the precedent here with LIST is clear as per any Rules interpretation.

Finally, as noted previously by John Manis, copyright infringement can become a criminal matter in Poland (this is presumably for egregious cases, and here I don't imagine it occurring). Another pedantry is that it is incorrect to refer to copyright infringement as "theft" in the technical sense.

[BB+]

By my count, there are 14 programmers who assented to the "crucial question" (Did Rajlich break the ICGA Rules?)

It seems that this has been (slightly) mis-parsed elsewhere. The word programmers (not necessarily chess) here does have meaning, as the total vote was a bit higher (Skinner's comment in the Report, for instance). OTOH, it looks like I mistakenly counted Lefler as an "assenter", when this is incorrect at a pedantic level [he refrained from voting in that thread, due to the fact that he was the one who posed the final Opinion questions]. Of the ~20 persons who couldn't be bothered to respond to the Secretariat's request to give a final Opinion [this was sent via the WikiSpaces messaging system (a user can choose whether to additionally have this forwarded to his email) to all Panel members, with a request to respond within 2 weeks -- David Levy then sent out (via Harvey) a reminder about halfway thru], I think no more than 2 or 3 of them had made any comment at any previous juncture (some probably simply joined the Panel out of idle interest at an early stage, and then chose to ignore it).

More relevantly, I might also point out the phrasing was something like: "a. Have you read all the documents? b. Do you think Rajlich violated the ICGA Rules?" The first question might have acted as a filter, as everyone who answered gave an affirmative answer to both questions [and those who hadn't fulfilled "requirement" (a) would be more apt to simply not bother answering]. Given the amount of effort needed to answer (a) affirmatively, I can't say that I think 14 is a particularly low number. The phrasing in the Report (boldface in the original in one place) was: All panel members who expressed an opinion agree that Vasik Rajlich violated the ICGA Tournament Rules. Not a single panel member believed him innocent. The second sentence here should not be separated from the context implied by the first (obviously the Report can only talk about the opinions/beliefs of the Panel members who actually bothered to articulate them). In any event, it seems certainly fair to say that all those who had and stated an informed opinion thought that Rajlich had broken the rules.

As for the mis-representation of this accounting in the press (and even that the Panel vote was anything more than advisory for that matter), well, I can only quote Thomas Jefferson: Nothing can now be believed which is seen in a newspaper. Or Garrison Kellior: If you watch TV news, you know less about the world than if you just drink gin out of a bottle.

[BB+]

Of the ~20 persons who couldn't be bothered to respond to the Secretariat's request to give a final Opinion

Reading a comment from a Secretariat Member elsewhere [which also noted the effect of the "(a)" question], it seems that there could have been others who chose to email their opinion directly, rather than post it in the wiki discussion thread. So 14 is a lower bound.

[mjlef]

Thanks for providing this extra information about the process. In the future we should do a more detailed report for those interested in how things worked (the report we published was somewhat short on purpose, being a summary after all). As for the "votes" of opinions, a few people (like some Secretariat members) did withhold a public opinion so as not to sway others. But we received no opinions of innocence from anyone.

I really appreciated the panel members who played "devil's advocate", since they did a great job pointing out factual or logical gaps that we were able to fill in as we proceeded. The quantity and quality of the work was great, and I was proud to be part of such a professional group. At some point I felt a little guilt in asking people for more information, knowing how long some of this takes to gather. But everyone delivered. Especially Mark Watkins and Zach.

[BB+]

The latest claim is that the Report was "one-sided". Indeed, with Rajlich not showing up to the Panel, it would be hard for it not to be to some extent. For instance, the Panel had no reason not to expect Rajlich to defend himself directly with the Board, and no reason necessarily to think that its input would be solicited from thereon. So a definite purpose of the Report was to state any case against Rajlich in fairly strong terms (as per the general Panel consensus), expecting a behind-doors spirited defense in response. As has been noted elsewhere, the Panel was never really sure whether it was supposed to be a "prosecution" in a strict adversarial sense, a "truth-seeking" consultative entity (curiously deprived of one side of the story) , an "advisor" to the Board on technical matters, or what, and at different times played each of these roles.

In short, the Panel was charged to address a rather narrow range of questions, specifically regarding Rybka's originality vis-a-vis Fruit (and later Crafty). It was not particularly asked to narrate a history of Rybka's development over the victorious WCCC years. Quoting Don Dailey: I am convinced that there is much that is original in later versions of Rybka, but that is not the issue being discussed here. If the Report had been written after Rajlich chose to absent himself from the Board procedure, then I agree that his positive contributions could have been highlighted more, and perhaps these should have been taken more into account in the penalty phase. [Also, in a typical plagiarism case, the actual value of gratuitously perfunctory comments like: "Of course the author has done many original things, both here and subsequently..." seems rather low to me, other than trying to mesmerise an aura of "fair and balanced" upon any proceedings].

As for the Verdict, the tone therein implies to me that it was principally concerned at being direct and factual regarding the reasons for disqualification and the incumbent punishments. If it had gotten into other matters (such as emphasizing that Larry Kaufman was "squeaky clean"), it could easily become quite rambling.

I mentioned previously that I expect ICGA Journal to have a more in-depth discussion at some point, and likely this will not be quite so "one-sided" contra Rajlich and his positive contributions. OTOH, it seems reasonable to expect that the Verdict of "non-originality" and his refusal to address the issue will be the primary focus.

[BB+]

IMHO the ICGA was simply used as a cover to try to dismiss all the incredible achievements he has given to computer chess. I am sure most of the people on the ICGA had good intentions, but they let themselves be swayed and pushed into quickly singling out Vas as guilty and rushed into a conclusion. Had some of the dissenting voices by equally talented programmers been heard, maybe some might have not given their seal of approval so quickly. You would have to be really crazy to believe that nothing would have changed had this healthy debate been part of the ICGA investigation.

I might first say that I actually agree with some of what Majd said earlier in this post, but I bring it up to part above to highlight what I find most irksome about the perception (mostly exclusive to the Rybka Forum) about the ICGA process.

As least from the Panel standpoint, the first sentence is rebutted (for instance) by Don Dailey's comment in the Report. I suspect a minority of the (voting) Panel would be guilty of such "cover" actions.

The second sentence is nearly ridiculous to anyone who was on the Panel -- the question of whether Rybka's "borrowing" from Fruit was beyond the common expectation in computer chess was the principal focus of more than a month of discussion (and document writing). Furthermore, healthy debate and dissenting opinions were indeed part of the ICGA Panel process, as led by Marcel van Kervinck, Charles Roberson, Gerd Isenberg, and Wylie Garvin at various times, among others. [I might also note that many of the "competing commercial interests" (Uniacke, SMK, Junior, Dailey) were almost unanimously silent throughout the discussion].

Finally, personally I have seen nothing specific from the "dissenters" (Ed, Chris, and maybe Miguel) that would come close to challenging any opinion that Rajlich violated ICGA Rule #2 with his WCCC entries in 2006-7 -- the same would be true concerning a conclusion as regards the "most likely" sequence of events regarding Rybka development. Most of what has been proposed so far strikes me as either being too doubtful, or more of a vague hunch than being really substantive. The other issues raised (such as whether EVAL_COMP has any merit) appear currently to have been addressed adequately, at least to my understanding. [In this regard, it would be nice if those who "dissent" could exemplify what they would mean/understand by "specific" versus "general" chess knowledge, taking (for instance) isolated pawns as an example].