Komodo 5 running for the IPON

As in chess tournaments and matches...
Post Reply
User avatar
IWB
Posts: 195
Joined: Thu Jun 10, 2010 4:10 pm

Komodo 5 running for the IPON

Post by IWB » Thu Jul 19, 2012 5:00 pm

Here you find more info:

http://www.inwoba.de

Have fun
Ingo
Ponder ON rating list: http://www.inwoba.de

User avatar
IWB
Posts: 195
Joined: Thu Jun 10, 2010 4:10 pm

Re: Komodo 5 running for the IPON

Post by IWB » Sun Jul 22, 2012 3:03 pm

Sorry for the delay but finaly the result for Komodo 5 is online.

http://www.inwoba.de

The IPON-RRRL will be updated soon.

Bye
Ingo
Ponder ON rating list: http://www.inwoba.de

BB+
Posts: 1484
Joined: Thu Jun 10, 2010 4:26 am

Re: Komodo 5 running for the IPON

Post by BB+ » Mon Jul 23, 2012 1:54 am

Just to note my agreement with MB's Ordo:

Code: Select all

STEP is 100.000000 Termination Point is 0.000100
Observed White advantage is 42.17
Normalising to "Deep Shredder 12" as 2800.00

RATING   NAME                    RESULTS            GAMES   WIN%  DRAW% ERROR
3042.92  Houdini 2.0 STD         +3529 =1385 -486    5400  78.2%  25.6%  4.5
3034.78  Houdini 1.5a            +2644 =1037 -319    4000  79.1%  25.9%  5.3
3026.81  Komodo 5                +1670 =936  -244    2850  75.0%  32.8%  5.7
3003.35  Critter 1.4a            +2700 =1439 -311    4450  76.8%  32.3%  4.6
2999.25  Komodo 4                +2915 =1476 -459    4850  75.3%  30.4%  4.5
2990.56  Critter 1.6a            +1587 =1259 -304    3150  70.4%  40.0%  4.9
2987.59  Komodo 3                +1646 =859  -295    2800  74.1%  30.7%  6.0
2979.01  Stockfish 2.2.2 JA      +2667 =1695 -438    4800  73.2%  35.3%  4.3
2977.97  Deep Rybka 4.1          +3263 =2298 -639    6200  71.2%  37.1%  3.7
2977.38  Deep Rybka 4            +2810 =1634 -456    4900  74.0%  33.3%  4.2
2977.26  Critter 1.2             +1666 =1132 -302    3100  72.0%  36.5%  5.2
2975.52  Houdini 1.03a           +2048 =944  -208    3200  78.8%  29.5%  5.7
2970.21  Komodo 2.03 DC          +1583 =805  -312    2700  73.5%  29.8%  6.0
2961.35  Stockfish 2.1.1 JA      +1797 =1259 -444    3500  69.3%  36.0%  4.9
2941.69  Critter 1.01            +1465 =1010 -325    2800  70.4%  36.1%  5.5
2941.34  Stockfish 2.01 JA       +1707 =1078 -315    3100  72.5%  34.8%  5.3
2919.48  Stockfish 1.9.1 JA      +1598 =1066 -336    3000  71.0%  35.5%  5.4
2918.86  Rybka 3 mp              +2580 =1296 -324    4200  76.9%  30.9%  4.8
2911.48  Critter 0.90            +1712 =1231 -457    3400  68.5%  36.2%  4.9
2903.72  Stockfish 1.7.1 JA      +1654 =954  -292    2900  73.5%  32.9%  5.7
2859.65  Rybka 3 32b             +894  =595  -211    1700  70.1%  35.0%  7.0
2843.82  Stockfish 1.6.x JA      +1308 =969  -323    2600  68.9%  37.3%  5.6
2839.30  Komodo 1.3 JA           +1335 =1222 -743    3300  59.0%  37.0%  4.9
2833.35  Naum 4.2                +3397 =3742 -2161   9300  56.6%  40.2%  2.8
2833.29  Deep Fritz 13 32b       +954  =1335 -1011   3300  49.1%  40.5%  4.6
2832.88  Chiron 1.1a             +1684 =1961 -1305   4950  53.8%  39.6%  4.0
2824.98  Critter 0.80            +1294 =1003 -503    2800  64.1%  35.8%  5.4
2819.40  Fritz 13 32b            +1445 =1726 -1129   4300  53.7%  40.1%  4.2
2809.52  Komodo 1.2 JA           +1444 =1462 -794    3700  58.8%  39.5%  4.6
2805.01  Rybka 2.3.2a mp         +1466 =1413 -621    3500  62.1%  40.4%  4.6
2800.00  Deep Shredder 12        +3596 =3944 -2860  10400  53.5%  37.9%  2.8
2797.31  Hannibal 1.2            +921  =1423 -1256   3600  45.3%  39.5%  4.6
2795.13  Gull 1.2                +1926 =2293 -2081   6300  48.8%  36.4%  3.5
2791.59  Critter 0.70            +764  =686  -450    1900  58.3%  36.1%  6.5
2791.28  Gull 1.1                +1089 =1173 -838    3100  54.0%  37.8%  5.0
2789.61  Naum 4.1                +1009 =912  -379    2300  63.7%  39.7%  5.7
2788.90  Deep Sjeng c't 2010 32b +2121 =2832 -2347   7300  48.5%  38.8%  3.3
2784.95  Komodo 1.0 JA           +1154 =1205 -541    2900  60.6%  41.6%  5.0
2782.31  Spike 1.4 32b           +1785 =2467 -2148   6400  47.2%  38.5%  3.5
2778.05  Deep Fritz 12 32b       +2069 =2399 -1832   6300  51.9%  38.1%  3.5
2775.78  Naum 4                  +1089 =1079 -532    2700  60.3%  40.0%  5.2
2775.18  Rybka 2.2n2 mp          +895  =833  -372    2100  62.5%  39.7%  6.0
2766.73  Gull 1.0a               +811  =886  -603    2300  54.5%  38.5%  5.8
2761.98  Stockfish 1.5.1 JA      +763  =731  -406    1900  59.4%  38.5%  6.4
2761.07  Rybka 1.2f              +1147 =863  -390    2400  65.8%  36.0%  5.9
2756.84  Protector 1.4.0         +1684 =2406 -2410   6500  44.4%  37.0%  3.5
2756.62  spark-1.0               +1749 =2685 -2566   7000  44.2%  38.4%  3.4
2747.57  Hannibal 1.1            +1183 =1918 -1799   4900  43.7%  39.1%  4.0
2745.81  HIARCS 13.2 MP 32b      +1691 =2463 -2646   6800  43.0%  36.2%  3.5
2745.05  Deep Junior 13          +796  =1313 -1491   3600  40.3%  36.5%  4.8
2744.96  Deep Junior 13.3        +634  =994  -1372   3000  37.7%  33.1%  5.4
2740.13  Fritz 12 32b            +687  =808  -505    2000  54.5%  40.4%  6.2
2734.62  Quazar 0.4              +761  =1478 -1661   3900  38.5%  37.9%  4.5
2727.51  HIARCS 13.1 MP 32b      +1068 =1333 -1199   3600  48.2%  37.0%  4.8
2726.31  Deep Junior 12.5        +1161 =1604 -2085   4850  40.5%  33.1%  4.3
2720.28  Deep Fritz 11 32b       +494  =501  -305    1300  57.3%  38.5%  7.8
2710.05  Doch64 1.2 JA           +491  =659  -450    1600  51.3%  41.2%  6.7
2708.64  spark-0.4               +849  =1218 -1033   3100  47.0%  39.3%  4.9
2707.78  Stockfish 1.4 JA        +523  =652  -525    1700  49.9%  38.4%  6.7
2705.76  Zappa Mexico II         +2945 =4250 -4505  11700  43.3%  36.3%  2.6
2704.65  Shredder Bonn 32b       +726  =786  -688    2200  50.9%  35.7%  6.0
2693.29  Critter 0.60            +666  =812  -722    2200  48.7%  36.9%  6.0
2693.20  Protector 1.3.2 JA      +1364 =1995 -1941   5300  44.6%  37.6%  3.8
2689.68  MinkoChess 1.3          +473  =1323 -1804   3600  31.5%  36.8%  4.8
2684.34  Deep Shredder 11        +922  =980  -798    2700  52.3%  36.3%  5.4
2681.51  Doch64 09.980 JA        +424  =572  -504    1500  47.3%  38.1%  7.0
2674.05  Deep Junior 12          +809  =1094 -1697   3600  37.7%  30.4%  5.1
2673.75  Onno-1-1-1              +1064 =1718 -1518   4300  44.7%  40.0%  4.1
2673.47  Hannibal 1.0a           +897  =1406 -1897   4200  38.1%  33.5%  4.5
2672.39  Deep Onno 1-2-70        +1421 =2771 -3508   7700  36.4%  36.0%  3.3
2672.02  Naum 3.1                +927  =1175 -898    3000  50.5%  39.2%  5.0
2671.59  Zappa Mexico I          +774  =894  -532    2200  55.5%  40.6%  5.8
2670.57  Rybka 1.0 Beta          +617  =813  -870    2300  44.5%  35.3%  5.9
2667.40  Spark-0.3 VC(a)         +903  =1444 -1253   3600  45.1%  40.1%  4.5
2664.65  Onno-1-0-0              +347  =495  -358    1200  49.5%  41.2%  7.7
2662.34  Deep Sjeng WC2008       +1407 =2055 -2138   5600  43.5%  36.7%  3.8
2658.65  Toga II 1.4 beta5c BB   +1707 =3097 -3496   8300  39.2%  37.3%  3.1
2657.30  Deep Junior 11.2        +725  =902  -1273   2900  40.6%  31.1%  5.7
2653.12  Strelka 2.0 B           +866  =1825 -2809   5500  32.3%  33.2%  4.0
2649.36  Hiarcs 12.1 MP 32b      +1366 =2123 -2111   5600  43.3%  37.9%  3.7
2647.32  Tornado 4.88            +408  =790  -1202   2400  33.5%  32.9%  6.2
2646.75  Deep Sjeng 3.0          +362  =479  -559    1400  43.0%  34.2%  7.7
2646.64  Umko 1.2                +458  =1117 -1725   3300  30.8%  33.8%  5.2
2635.64  Critter 0.52b           +594  =1006 -1000   2600  42.2%  38.7%  5.4
2635.48  Shredder Classic 4 32b  +581  =683  -536    1800  51.2%  37.9%  6.4
2625.45  Deep Junior 11.1a       +673  =960  -1167   2800  41.2%  34.3%  5.6
2623.84  Naum 2.2 32b            +323  =582  -395    1300  47.2%  44.8%  7.3
2618.83  Nemo 1.0.1              +299  =818  -1583   2700  26.2%  30.3%  6.0
2618.77  Umko 1.1                +510  =1272 -2118   3900  29.4%  32.6%  4.9
2616.51  Deep Junior 2010        +735  =950  -1415   3100  39.0%  30.6%  5.4
2615.60  Glaurung 2.2 JA         +538  =979  -1083   2600  39.5%  37.7%  5.5
2615.51  Rybka 1.0 Beta 32b      +301  =410  -389    1100  46.0%  37.3%  8.3
2610.54  HIARCS 11.2 32b         +468  =718  -714    1900  43.5%  37.8%  6.3
2607.92  Fruit 05/11/03 32b      +882  =1784 -1734   4400  40.3%  40.5%  4.1
2600.95  Loop 2007               +1139 =2634 -4127   7900  31.1%  33.3%  3.4
2597.68  Toga II 1.2.1a          +388  =657  -555    1600  44.8%  41.1%  6.7
2597.12  Jonny 4.00 32b          +673  =1433 -3094   5200  26.7%  27.6%  4.5
2593.30  ListMP 11               +506  =963  -1131   2600  38.0%  37.0%  5.6
2591.16  LoopMP 12 32b           +351  =568  -581    1500  42.3%  37.9%  7.2
2589.40  Tornado 4.80            +328  =707  -1665   2700  25.2%  26.2%  6.4
2587.43  Deep Shredder 10        +1031 =1446 -1923   4400  39.9%  32.9%  4.4
2583.02  Twisted Logic 20100131x +607  =1066 -1827   3500  32.6%  30.5%  5.2
2578.38  Crafty 23.3 JA          +592  =1397 -3211   5200  24.8%  26.9%  4.5
2560.51  Spike 1.2 Turin 32b     +1087 =2525 -4088   7700  30.5%  32.8%  3.4
2536.25  Deep Sjeng 2.7 32b      +217  =497  -686    1400  33.2%  35.5%  7.8
2524.90  Crafty 23.1 JA          +470  =1064 -2266   3800  26.4%  28.0%  5.2
I think the only difference is that I estimate the White advantage (42.17), while I guess Ordo does not. As colours are equalised, this has little overall effect -- I get a rating higher by 1.1 Elo at the extreme. Note that I agree with all the "close" orderings, including DR4.1/DR4/Critter12 [with the 0.6 and 0.1 margins], Critter1.01 over SF 2.01 by 0.4, SF 1.91 over Rybka 3 mp by 0.6 or so, Naum4.2 squeaking over DF13, the latter 0.4 above Chiron 1.1a, DJ13 better than DJ13.3 by 0.1 Elo, etc. So I guess we both implemented the same algorithm. :-)

One puzzlement to me is how Ordo estimates error, as H2 has (significantly) more games than H1.5, and about the same draw rate and win%, but less error in Ordo's output.

mballicora
Posts: 26
Joined: Tue Aug 09, 2011 7:58 pm
Real Name: Miguel A. Ballicora

Re: Komodo 5 running for the IPON

Post by mballicora » Thu Aug 02, 2012 4:46 pm

BB+ wrote:Just to note my agreement with MB's Ordo:

Code: Select all

STEP is 100.000000 Termination Point is 0.000100
Observed White advantage is 42.17
Normalising to "Deep Shredder 12" as 2800.00

RATING   NAME                    RESULTS            GAMES   WIN%  DRAW% ERROR
3042.92  Houdini 2.0 STD         +3529 =1385 -486    5400  78.2%  25.6%  4.5
3034.78  Houdini 1.5a            +2644 =1037 -319    4000  79.1%  25.9%  5.3
3026.81  Komodo 5                +1670 =936  -244    2850  75.0%  32.8%  5.7
3003.35  Critter 1.4a            +2700 =1439 -311    4450  76.8%  32.3%  4.6
2999.25  Komodo 4                +2915 =1476 -459    4850  75.3%  30.4%  4.5
2990.56  Critter 1.6a            +1587 =1259 -304    3150  70.4%  40.0%  4.9
2987.59  Komodo 3                +1646 =859  -295    2800  74.1%  30.7%  6.0
2979.01  Stockfish 2.2.2 JA      +2667 =1695 -438    4800  73.2%  35.3%  4.3
2977.97  Deep Rybka 4.1          +3263 =2298 -639    6200  71.2%  37.1%  3.7
2977.38  Deep Rybka 4            +2810 =1634 -456    4900  74.0%  33.3%  4.2
2977.26  Critter 1.2             +1666 =1132 -302    3100  72.0%  36.5%  5.2
2975.52  Houdini 1.03a           +2048 =944  -208    3200  78.8%  29.5%  5.7
2970.21  Komodo 2.03 DC          +1583 =805  -312    2700  73.5%  29.8%  6.0
2961.35  Stockfish 2.1.1 JA      +1797 =1259 -444    3500  69.3%  36.0%  4.9
2941.69  Critter 1.01            +1465 =1010 -325    2800  70.4%  36.1%  5.5
2941.34  Stockfish 2.01 JA       +1707 =1078 -315    3100  72.5%  34.8%  5.3
2919.48  Stockfish 1.9.1 JA      +1598 =1066 -336    3000  71.0%  35.5%  5.4
2918.86  Rybka 3 mp              +2580 =1296 -324    4200  76.9%  30.9%  4.8
2911.48  Critter 0.90            +1712 =1231 -457    3400  68.5%  36.2%  4.9
2903.72  Stockfish 1.7.1 JA      +1654 =954  -292    2900  73.5%  32.9%  5.7
2859.65  Rybka 3 32b             +894  =595  -211    1700  70.1%  35.0%  7.0
2843.82  Stockfish 1.6.x JA      +1308 =969  -323    2600  68.9%  37.3%  5.6
2839.30  Komodo 1.3 JA           +1335 =1222 -743    3300  59.0%  37.0%  4.9
2833.35  Naum 4.2                +3397 =3742 -2161   9300  56.6%  40.2%  2.8
2833.29  Deep Fritz 13 32b       +954  =1335 -1011   3300  49.1%  40.5%  4.6
2832.88  Chiron 1.1a             +1684 =1961 -1305   4950  53.8%  39.6%  4.0
2824.98  Critter 0.80            +1294 =1003 -503    2800  64.1%  35.8%  5.4
2819.40  Fritz 13 32b            +1445 =1726 -1129   4300  53.7%  40.1%  4.2
2809.52  Komodo 1.2 JA           +1444 =1462 -794    3700  58.8%  39.5%  4.6
2805.01  Rybka 2.3.2a mp         +1466 =1413 -621    3500  62.1%  40.4%  4.6
2800.00  Deep Shredder 12        +3596 =3944 -2860  10400  53.5%  37.9%  2.8
2797.31  Hannibal 1.2            +921  =1423 -1256   3600  45.3%  39.5%  4.6
2795.13  Gull 1.2                +1926 =2293 -2081   6300  48.8%  36.4%  3.5
2791.59  Critter 0.70            +764  =686  -450    1900  58.3%  36.1%  6.5
2791.28  Gull 1.1                +1089 =1173 -838    3100  54.0%  37.8%  5.0
2789.61  Naum 4.1                +1009 =912  -379    2300  63.7%  39.7%  5.7
2788.90  Deep Sjeng c't 2010 32b +2121 =2832 -2347   7300  48.5%  38.8%  3.3
2784.95  Komodo 1.0 JA           +1154 =1205 -541    2900  60.6%  41.6%  5.0
2782.31  Spike 1.4 32b           +1785 =2467 -2148   6400  47.2%  38.5%  3.5
2778.05  Deep Fritz 12 32b       +2069 =2399 -1832   6300  51.9%  38.1%  3.5
2775.78  Naum 4                  +1089 =1079 -532    2700  60.3%  40.0%  5.2
2775.18  Rybka 2.2n2 mp          +895  =833  -372    2100  62.5%  39.7%  6.0
2766.73  Gull 1.0a               +811  =886  -603    2300  54.5%  38.5%  5.8
2761.98  Stockfish 1.5.1 JA      +763  =731  -406    1900  59.4%  38.5%  6.4
2761.07  Rybka 1.2f              +1147 =863  -390    2400  65.8%  36.0%  5.9
2756.84  Protector 1.4.0         +1684 =2406 -2410   6500  44.4%  37.0%  3.5
2756.62  spark-1.0               +1749 =2685 -2566   7000  44.2%  38.4%  3.4
2747.57  Hannibal 1.1            +1183 =1918 -1799   4900  43.7%  39.1%  4.0
2745.81  HIARCS 13.2 MP 32b      +1691 =2463 -2646   6800  43.0%  36.2%  3.5
2745.05  Deep Junior 13          +796  =1313 -1491   3600  40.3%  36.5%  4.8
2744.96  Deep Junior 13.3        +634  =994  -1372   3000  37.7%  33.1%  5.4
2740.13  Fritz 12 32b            +687  =808  -505    2000  54.5%  40.4%  6.2
2734.62  Quazar 0.4              +761  =1478 -1661   3900  38.5%  37.9%  4.5
2727.51  HIARCS 13.1 MP 32b      +1068 =1333 -1199   3600  48.2%  37.0%  4.8
2726.31  Deep Junior 12.5        +1161 =1604 -2085   4850  40.5%  33.1%  4.3
2720.28  Deep Fritz 11 32b       +494  =501  -305    1300  57.3%  38.5%  7.8
2710.05  Doch64 1.2 JA           +491  =659  -450    1600  51.3%  41.2%  6.7
2708.64  spark-0.4               +849  =1218 -1033   3100  47.0%  39.3%  4.9
2707.78  Stockfish 1.4 JA        +523  =652  -525    1700  49.9%  38.4%  6.7
2705.76  Zappa Mexico II         +2945 =4250 -4505  11700  43.3%  36.3%  2.6
2704.65  Shredder Bonn 32b       +726  =786  -688    2200  50.9%  35.7%  6.0
2693.29  Critter 0.60            +666  =812  -722    2200  48.7%  36.9%  6.0
2693.20  Protector 1.3.2 JA      +1364 =1995 -1941   5300  44.6%  37.6%  3.8
2689.68  MinkoChess 1.3          +473  =1323 -1804   3600  31.5%  36.8%  4.8
2684.34  Deep Shredder 11        +922  =980  -798    2700  52.3%  36.3%  5.4
2681.51  Doch64 09.980 JA        +424  =572  -504    1500  47.3%  38.1%  7.0
2674.05  Deep Junior 12          +809  =1094 -1697   3600  37.7%  30.4%  5.1
2673.75  Onno-1-1-1              +1064 =1718 -1518   4300  44.7%  40.0%  4.1
2673.47  Hannibal 1.0a           +897  =1406 -1897   4200  38.1%  33.5%  4.5
2672.39  Deep Onno 1-2-70        +1421 =2771 -3508   7700  36.4%  36.0%  3.3
2672.02  Naum 3.1                +927  =1175 -898    3000  50.5%  39.2%  5.0
2671.59  Zappa Mexico I          +774  =894  -532    2200  55.5%  40.6%  5.8
2670.57  Rybka 1.0 Beta          +617  =813  -870    2300  44.5%  35.3%  5.9
2667.40  Spark-0.3 VC(a)         +903  =1444 -1253   3600  45.1%  40.1%  4.5
2664.65  Onno-1-0-0              +347  =495  -358    1200  49.5%  41.2%  7.7
2662.34  Deep Sjeng WC2008       +1407 =2055 -2138   5600  43.5%  36.7%  3.8
2658.65  Toga II 1.4 beta5c BB   +1707 =3097 -3496   8300  39.2%  37.3%  3.1
2657.30  Deep Junior 11.2        +725  =902  -1273   2900  40.6%  31.1%  5.7
2653.12  Strelka 2.0 B           +866  =1825 -2809   5500  32.3%  33.2%  4.0
2649.36  Hiarcs 12.1 MP 32b      +1366 =2123 -2111   5600  43.3%  37.9%  3.7
2647.32  Tornado 4.88            +408  =790  -1202   2400  33.5%  32.9%  6.2
2646.75  Deep Sjeng 3.0          +362  =479  -559    1400  43.0%  34.2%  7.7
2646.64  Umko 1.2                +458  =1117 -1725   3300  30.8%  33.8%  5.2
2635.64  Critter 0.52b           +594  =1006 -1000   2600  42.2%  38.7%  5.4
2635.48  Shredder Classic 4 32b  +581  =683  -536    1800  51.2%  37.9%  6.4
2625.45  Deep Junior 11.1a       +673  =960  -1167   2800  41.2%  34.3%  5.6
2623.84  Naum 2.2 32b            +323  =582  -395    1300  47.2%  44.8%  7.3
2618.83  Nemo 1.0.1              +299  =818  -1583   2700  26.2%  30.3%  6.0
2618.77  Umko 1.1                +510  =1272 -2118   3900  29.4%  32.6%  4.9
2616.51  Deep Junior 2010        +735  =950  -1415   3100  39.0%  30.6%  5.4
2615.60  Glaurung 2.2 JA         +538  =979  -1083   2600  39.5%  37.7%  5.5
2615.51  Rybka 1.0 Beta 32b      +301  =410  -389    1100  46.0%  37.3%  8.3
2610.54  HIARCS 11.2 32b         +468  =718  -714    1900  43.5%  37.8%  6.3
2607.92  Fruit 05/11/03 32b      +882  =1784 -1734   4400  40.3%  40.5%  4.1
2600.95  Loop 2007               +1139 =2634 -4127   7900  31.1%  33.3%  3.4
2597.68  Toga II 1.2.1a          +388  =657  -555    1600  44.8%  41.1%  6.7
2597.12  Jonny 4.00 32b          +673  =1433 -3094   5200  26.7%  27.6%  4.5
2593.30  ListMP 11               +506  =963  -1131   2600  38.0%  37.0%  5.6
2591.16  LoopMP 12 32b           +351  =568  -581    1500  42.3%  37.9%  7.2
2589.40  Tornado 4.80            +328  =707  -1665   2700  25.2%  26.2%  6.4
2587.43  Deep Shredder 10        +1031 =1446 -1923   4400  39.9%  32.9%  4.4
2583.02  Twisted Logic 20100131x +607  =1066 -1827   3500  32.6%  30.5%  5.2
2578.38  Crafty 23.3 JA          +592  =1397 -3211   5200  24.8%  26.9%  4.5
2560.51  Spike 1.2 Turin 32b     +1087 =2525 -4088   7700  30.5%  32.8%  3.4
2536.25  Deep Sjeng 2.7 32b      +217  =497  -686    1400  33.2%  35.5%  7.8
2524.90  Crafty 23.1 JA          +470  =1064 -2266   3800  26.4%  28.0%  5.2
I think the only difference is that I estimate the White advantage (42.17), while I guess Ordo does not. As colours are equalised, this has little overall effect -- I get a rating higher by 1.1 Elo at the extreme. Note that I agree with all the "close" orderings, including DR4.1/DR4/Critter12 [with the 0.6 and 0.1 margins], Critter1.01 over SF 2.01 by 0.4, SF 1.91 over Rybka 3 mp by 0.6 or so, Naum4.2 squeaking over DF13, the latter 0.4 above Chiron 1.1a, DJ13 better than DJ13.3 by 0.1 Elo, etc. So I guess we both implemented the same algorithm. :-)

One puzzlement to me is how Ordo estimates error, as H2 has (significantly) more games than H1.5, and about the same draw rate and win%, but less error in Ordo's output.
Ordo calculates the white advantage and it was 41.1, but I have not released the version that does it, yet. This is a reminder I need to do a release.

Once the ratings are calculated you assume this is the "truth" and the results of the games are randomly "simulated" based on the ratings obtained and their respective probabilities estimated for a win in each game. You do this n times, so you obtain n different ratings for a given engine. SD is calculated from this. This way you can estimate the error of any type of difference. The issue is that the errors obtained, which are the errors taking into account the average of the pool as a reference, are generally not what one is interested in. What most of the time one wants, it the error of the diffence between two given engines. This could vary tremendously. Ordo provides a matrix file with error engineA-engineB, error engineA-engineC etc. The advantage of this becomes evident when there are, say, two pools of engines that played lots of intra-pool games, but fewer inter-pool games. The error between two engines from different pools will be much bigger than the error between two engines from the same pool, as it should be.

The discrepancy you noted in the errors is because there were only 100 simulations, so the estimations of the errors were not that precise. I ran 5000 simulations and I got

Code: Select all

   # ENGINE                  :  RATING  ERROR   POINTS  PLAYED    (%)
   1 Houdini 2.0 STD         :  2590.3    6.3   4221.5    5400   78.2%
   2 Houdini 1.5a            :  2582.2    7.1   3162.5    4000   79.1%
   3 Komodo 5                :  2574.3    7.9   2138.0    2850   75.0%
   4 Critter 1.4a            :  2550.9    6.7   3419.5    4450   76.8%
   5 Komodo 4                :  2546.9    6.3   3653.0    4850   75.3%
   6 Critter 1.6a            :  2538.2    7.2   2216.5    3150   70.4%
   7 Komodo 3                :  2535.3    8.1   2075.5    2800   74.1%
   8 Stockfish 2.2.2 JA      :  2526.7    6.1   3514.5    4800   73.2%
   9 Deep Rybka 4.1          :  2525.7    5.4   4412.0    6200   71.2%
  10 Deep Rybka 4            :  2525.1    6.0   3627.0    4900   74.0%
  11 Critter 1.2             :  2525.0    7.5   2232.0    3100   72.0%
  12 Houdini 1.03a           :  2523.2    7.9   2520.0    3200   78.8%
  13 Komodo 2.03 DC          :  2518.0    8.1   1985.5    2700   73.5%
  14 Stockfish 2.1.1 JA      :  2509.1    7.0   2426.5    3500   69.3%
  15 Critter 1.01            :  2489.6    7.6   1970.0    2800   70.4%
  16 Stockfish 2.01 JA       :  2489.2    7.4   2246.0    3100   72.5%
  17 Stockfish 1.9.1 JA      :  2467.5    7.4   2131.0    3000   71.0%
  18 Rybka 3 mp              :  2466.8    6.5   3228.0    4200   76.9%
  19 Critter 0.90            :  2459.5    6.9   2327.5    3400   68.5%
  20 Stockfish 1.7.1 JA      :  2451.8    7.7   2131.0    2900   73.5%
  21 Rybka 3 32b             :  2407.9    9.4   1191.5    1700   70.1%
  22 Stockfish 1.6.x JA      :  2392.1    7.8   1792.5    2600   68.9%
  23 Komodo 1.3 JA           :  2387.6    6.5   1946.0    3300   59.0%
  24 Naum 4.2                :  2381.7    3.9   5268.0    9300   56.6%
  25 Deep Fritz 13 32b       :  2381.7    6.5   1621.5    3300   49.1%
  26 Chiron 1.1a             :  2381.3    5.4   2664.5    4950   53.8%
  27 Critter 0.80            :  2373.4    7.3   1795.5    2800   64.1%
  28 Fritz 13 32b            :  2367.8    5.9   2308.0    4300   53.7%
  29 Komodo 1.2 JA           :  2358.0    6.2   2175.0    3700   58.8%
  30 Rybka 2.3.2a mp         :  2353.5    6.6   2172.5    3500   62.1%
  31 Deep Shredder 12        :  2348.5    3.7   5568.0   10400   53.5%
  32 Hannibal 1.2            :  2345.9    6.2   1632.5    3600   45.3%
  33 Gull 1.2                :  2343.7    4.9   3072.5    6300   48.8%
  34 Critter 0.70            :  2340.2    8.5   1107.0    1900   58.3%
  35 Gull 1.1                :  2339.9    6.7   1675.5    3100   54.0%
  36 Naum 4.1                :  2338.2    7.9   1465.0    2300   63.7%
  37 Deep Sjeng c't 2010 32b :  2337.5    4.5   3537.0    7300   48.5%
  38 Komodo 1.0 JA           :  2333.6    7.1   1756.5    2900   60.6%
  39 Spike 1.4 32b           :  2330.9    4.8   3018.5    6400   47.2%
  40 Deep Fritz 12 32b       :  2326.7    4.7   3268.5    6300   51.9%
  41 Naum 4                  :  2324.4    7.2   1628.5    2700   60.3%
  42 Rybka 2.2n2 mp          :  2323.8    8.0   1311.5    2100   62.5%
  43 Gull 1.0a               :  2315.4    7.8   1254.0    2300   54.5%
  44 Stockfish 1.5.1 JA      :  2310.7    8.4   1128.5    1900   59.4%
  45 Rybka 1.2f              :  2309.8    8.0   1578.5    2400   65.8%
  46 Protector 1.4.0         :  2305.6    4.8   2887.0    6500   44.4%
  47 spark-1.0               :  2305.3    4.6   3091.5    7000   44.2%
  48 Hannibal 1.1            :  2296.3    5.5   2142.0    4900   43.7%
  49 HIARCS 13.2 MP 32b      :  2294.6    4.7   2922.5    6800   43.0%
  50 Deep Junior 13          :  2293.8    6.4   1452.5    3600   40.3%
  51 Deep Junior 13.3        :  2293.7    7.0   1131.0    3000   37.7%
  52 Fritz 12 32b            :  2288.9    8.2   1091.0    2000   54.5%
  53 Quazar 0.4              :  2283.5    6.2   1500.0    3900   38.5%
  54 HIARCS 13.1 MP 32b      :  2276.4    6.3   1734.5    3600   48.2%
  55 Deep Junior 12.5        :  2275.2    5.6   1963.0    4850   40.5%
  56 Deep Fritz 11 32b       :  2269.2   10.2    744.5    1300   57.3%
  57 Doch64 1.2 JA           :  2259.0    9.0    820.5    1600   51.3%
  58 spark-0.4               :  2257.6    6.6   1458.0    3100   47.0%
  59 Stockfish 1.4 JA        :  2256.7    8.8    849.0    1700   49.9%
  60 Zappa Mexico II         :  2254.7    3.4   5070.0   11700   43.3%
  61 Shredder Bonn 32b       :  2253.6    7.7   1119.0    2200   50.9%
  62 Critter 0.60            :  2242.3    7.8   1072.0    2200   48.7%
  63 Protector 1.3.2 JA      :  2242.2    5.2   2361.5    5300   44.6%
  64 MinkoChess 1.3          :  2238.7    6.6   1134.5    3600   31.5%
  65 Deep Shredder 11        :  2233.4    7.2   1412.0    2700   52.3%
  66 Doch64 09.980 JA        :  2230.6    9.4    710.0    1500   47.3%
  67 Deep Junior 12          :  2223.2    6.4   1356.0    3600   37.7%
  68 Onno-1-1-1              :  2222.9    5.6   1923.0    4300   44.7%
  69 Hannibal 1.0a           :  2222.6    5.9   1600.0    4200   38.1%
  70 Deep Onno 1-2-70        :  2221.5    4.4   2806.5    7700   36.4%
  71 Naum 3.1                :  2221.1    6.8   1514.5    3000   50.5%
  72 Zappa Mexico I          :  2220.7    7.9   1221.0    2200   55.5%
  73 Rybka 1.0 Beta          :  2219.7    7.8   1023.5    2300   44.5%
  74 Spark-0.3 VC(a)         :  2216.5    6.1   1625.0    3600   45.1%
  75 Onno-1-0-0              :  2213.8   10.4    594.5    1200   49.5%
  76 Deep Sjeng WC2008       :  2211.5    5.0   2434.5    5600   43.5%
  77 Toga II 1.4 beta5c BB   :  2207.8    4.0   3255.5    8300   39.2%
  78 Deep Junior 11.2        :  2206.5    7.1   1176.0    2900   40.6%
  79 Strelka 2.0 B           :  2202.3    5.5   1778.5    5500   32.3%
  80 Hiarcs 12.1 MP 32b      :  2198.6    5.0   2427.5    5600   43.3%
  81 Tornado 4.88            :  2196.6    8.0    803.0    2400   33.5%
  82 Deep Sjeng 3.0          :  2196.0    9.7    601.5    1400   43.0%
  83 Umko 1.2                :  2195.9    7.0   1016.5    3300   30.8%
  84 Critter 0.52b           :  2184.9    7.2   1097.0    2600   42.2%
  85 Shredder Classic 4 32b  :  2184.8    8.4    922.5    1800   51.2%
  86 Deep Junior 11.1a       :  2174.8    6.9   1153.0    2800   41.2%
  87 Naum 2.2 32b            :  2173.2   10.1    614.0    1300   47.2%
  88 Nemo 1.0.1              :  2168.2    7.8    708.0    2700   26.2%
  89 Umko 1.1                :  2168.1    6.4   1146.0    3900   29.4%
  90 Deep Junior 2010        :  2165.9    6.7   1210.0    3100   39.0%
  91 Glaurung 2.2 JA         :  2165.0    7.3   1027.5    2600   39.5%
  92 Rybka 1.0 Beta 32b      :  2164.9   10.7    506.0    1100   46.0%
  93 HIARCS 11.2 32b         :  2159.9    8.6    827.0    1900   43.5%
  94 Fruit 05/11/03 32b      :  2157.3    5.7   1774.0    4400   40.3%
  95 Loop 2007               :  2150.4    4.4   2456.0    7900   31.1%
  96 Toga II 1.2.1a          :  2147.1    9.0    716.5    1600   44.8%
  97 Jonny 4.00 32b          :  2146.6    5.7   1389.5    5200   26.7%
  98 ListMP 11               :  2142.8    7.3    987.5    2600   38.0%
  99 LoopMP 12 32b           :  2140.7    9.4    635.0    1500   42.3%
 100 Tornado 4.80            :  2138.9    8.2    681.5    2700   25.2%
 101 Deep Shredder 10        :  2136.9    5.8   1754.0    4400   39.9%
 102 Twisted Logic 20100131x :  2132.6    6.6   1140.0    3500   32.6%
 103 Crafty 23.3 JA          :  2127.9    5.9   1290.5    5200   24.8%
 104 Spike 1.2 Turin 32b     :  2110.2    4.5   2349.5    7700   30.5%
 105 Deep Sjeng 2.7 32b      :  2086.0   10.4    465.5    1400   33.2%
 106 Crafty 23.1 JA          :  2074.7    6.7   1002.0    3800   26.4%
The reason why all the errors are bigger is because I do not take into account draws in the simulations, which is it in the to-do list.

The matrix error is (only for the top 11) in .csv format

"N","NAME",0,1,2,3,4,5,6,7,8,9,10,11,
0," Houdini 2.0 STD"
1," Houdini 1.5a",9.4
2," Komodo 5",9.5,10.6
3," Critter 1.4a",8.8,9.5,10.1
4," Komodo 4",8.4,9.4,9.9,8.7
5," Critter 1.6a",9.1,10.1,10.2,9.6,9.2
6," Komodo 3",9.9,10.5,11.2,10.5,10.2,10.9
7," Stockfish 2.2.2 JA",8.3,9.2,9.6,8.7,8.4,9.2,10.1
8," Deep Rybka 4.1",7.8,8.8,9.1,8.2,7.9,8.4,9.6,7.7
9," Deep Rybka 4",8.5,9.0,10.0,8.9,8.8,9.3,9.8,8.5,7.9
10," Critter 1.2",9.5,10.0,10.8,9.8,9.5,10.2,10.8,9.5,8.9,9.4

Miguel

Post Reply