Page 2 of 2
Re: Extract line from Sf bench command
Posted: Sun Mar 02, 2014 12:48 pm
by ChessDrone
Cosmetic changes
Code: Select all
#!/bin/bash
# http://www.open-chess.org/viewtopic.php?f=5&t=2557#p20321 modified by wmahan
for engine in stockfish*; do
div=50
sum=0
average=0
for (( i = 1; i <= $div; i++ ))
do
out=`cat <<END | $engine 2>&1 | tail
setoption name Idle Threads Sleep value false
isready
bench
END`
nps=`echo "$out" | grep "Nodes\/second"|sed "s/.*: \([0-9]*\)/\1/"`
nodes=`echo "$out" | grep "Nodes searched"| sed "s/.*: \([0-9]*\)/\1/"`
let sum=$sum+$nps
done
let average=$sum/$div ; echo -e "$engine\t$nodes\t$average\r">>results.csv
done
Re: Extract line from Sf bench command
Posted: Wed Mar 12, 2014 2:10 pm
by quantum
I did not look into this forum for a longer time and just came across your changes. Based on this the script now checks if the nodes searched are consistent. Automatic calculation of the standard deviation is also included. For the moment it works, but I think it should be cleaned up at some point.
Code: Select all
#!/bin/bash
# http://www.open-chess.org/viewtopic.php?f=5&t=2557#p20321
# From quantum, lucasart and wmahan.
for engine in ./stockfish*; do
div=2 #must be >1 otherwise we divide by zero later
sum=0
average=0
stdev=0
summe=0
var=0
changes=0
oldnodes=0
declare -a number
for (( i = 1; i <= $div; i++ ))
do
out=`cat <<END | $engine 2>&1 | tail
setoption name Idle Threads Sleep value false
isready
bench 32 1 13 default depth
END`
nps=`echo "$out" | grep "Nodes\/second"|sed "s/.*: \([0-9]*\)/\1/"`
nodes=`echo "$out" | grep "Nodes searched"| sed "s/.*: \([0-9]*\)/\1/"`
#Check if nodes searched are the same.
if [ "$oldnodes" != 0 ]&&[ "$oldnodes" != "$nodes" ]; then
consistency="Nodes searched are not consistent!"
fi
oldnodes=$nodes
#Calculate average nps
let sum=$sum+$nps
number=("${number[@]}" "$nps")
done
let average=$sum/$div
#Calculate standard deviation
for (( i = 1; i <= $div; i++ ))
do
summe=0
summe=$(echo "(${number[n]}-$average)^2" | bc -l)
let var=$var+$summe
done
stdev=$(echo "scale=0;sqrt($var/($div-1))" | bc -l)
#Output
echo -e "$engine\t$nodes\t$average\t$stdev\t$consistency\r">>results.csv
done
Re: Extract line from Sf bench command
Posted: Wed Mar 12, 2014 2:57 pm
by quantum
Sorry for second post in a row. Did not find the edit button.
I just found out that the calculation of the standard deviation does not work correctly. Probably an issue with reading the values from the array.
Re: Extract line from Sf bench command
Posted: Wed Mar 12, 2014 3:34 pm
by quantum
Third post in a row as I am not supposed to edit my previous post. This version should calculate the standard deviation correctly.
I just benchmarked Sf 20 times and received following nps:
935006 923019 947309 941117 953582 941117 911335 935006 959940 935006 947309 928974 905603 899943 911335 911335 894354 959940 953582 935006
I wondered why several of these entries are the same???
Code: Select all
#!/bin/bash
# http://www.open-chess.org/viewtopic.php?f=5&t=2557#p20321
# From quantum, lucasart and wmahan.
for engine in ./stockfish*; do
div=20 #must be >1 otherwise we divide by zero later
sum=0
average=0
stdev=0
summe=0
var=0
changes=0
oldnodes=0
declare -a number
for (( i = 1; i <= $div; i++ ))
do
out=`cat <<END | $engine 2>&1 | tail
setoption name Idle Threads Sleep value false
isready
bench 32 1 6 default depth
END`
nps=`echo "$out" | grep "Nodes\/second"|sed "s/.*: \([0-9]*\)/\1/"`
nodes=`echo "$out" | grep "Nodes searched"| sed "s/.*: \([0-9]*\)/\1/"`
#Check if nodes searched are the same.
if [ "$oldnodes" != 0 ]&&[ "$oldnodes" != "$nodes" ]; then
consistency="Nodes searched are not consistent!"
fi
oldnodes=$nodes
#Calculate average nps
let sum=$sum+$nps
number=("${number[@]}" "$nps")
done
let average=$sum/$div
#Calculate standard deviation
for i in "${number[@]}"
do
summe=0
summe=$(echo "(${i}-$average)^2" | bc -l)
let var=$var+$summe
done
stdev=$(echo "scale=0;sqrt($var/($div-1))" | bc -l)
#Output
echo -e "$engine\t$nodes\t$average\t$stdev\t$consistency\r">>results.csv
done
Re: Extract line from Sf bench command
Posted: Thu Mar 13, 2014 6:36 pm
by ChessDrone
Thanks a lot, currently I try it.
For Windows, this will be needed:
http://gnuwin32.sourceforge.net/packages/bc.htm
then paste it in the folder "bin" of the 'Bash' shell
Otherwise you'll get this error:
#./Bench.sh: line 45: bc: command not found
#./Bench.sh: line 46: let: var=0+: syntax error: operand expected (error token is "+")
I also bring an additional comment for the command Bench.
# bench 32 1 6 default depth
# bench (hash:
n in mb) (threads:
n) (limit:
n (cf. limit type)) (FEN: default) (limit type:
n of time (in ms?),
n of nodes,
n? mate or
n of depth)
Edit: I only get 4 columns instead of 5, which is missing?
Code: Select all
profile-build/stockfish_profile-build_x32-473-posix-sjlj_x86-32.exe 144276 1031044 24041
profile-build/stockfish_profile-build_x32-473-win32-sjlj_x86-32.exe 144276 1025696 104502
profile-build/stockfish_profile-build_x32-481-win32-dwarf_x86-32.exe 144276 1016974 110381
profile-build/stockfish_profile-build_x32-481-win32-sjlj_x86-32.exe 144276 1006293 118165
profile-build/stockfish_profile-build_x64-473-posix-sjlj_x86-64-modern.exe 144276 1422040 833405
profile-build/stockfish_profile-build_x64-473-win32-sjlj_x86-64-modern.exe 144276 1491265 979989
profile-build/stockfish_profile-build_x64-481-posix-seh_x86-64-modern.exe 144276 1452021 901084
profile-build/stockfish_profile-build_x64-481-posix-sjlj_x86-64-modern.exe 144276 1376998 765594
profile-build/stockfish_profile-build_x64-481-win32-seh_x86-64-modern.exe 144276 1459870 925403
profile-build/stockfish_profile-build_x64-481-win32-sjlj_x86-64-modern.exe 144276 1404150 820381
Re: Extract line from Sf bench command
Posted: Thu Mar 13, 2014 7:02 pm
by ChessDrone
Ah ok, it is consistency.
Can you enlighten me on the Standard deviation? If the number is short, it is better?
Re: Extract line from Sf bench command
Posted: Sat Mar 15, 2014 6:38 pm
by quantum
If you run Sf bench command in multi core mode the nodes searched are not consistent. In the past it happened that a patch was introduced by mistake where the nodes searched were not consistent even in single core mode from time to time. Therefore I fought this future might be nice to have.
English is not my mother tongue nor am I statistician although I had some lessons in it...
To my understanding the standard deviation sigma gives you a hint how reliable your measurment was.
With a probability of 68% the true value is found to be within the range of your average value plusminus 1 sigma.
With a probability of 95 % the true value is found to be within the range of your average value plusminus 2 sigma.
For most purposes it is sufficient to use the 2sigma intervall as error bars.
I did run bench with different compilers.
./stockfish-clang 8347121 1088932 6168
./stockfish-clang-pgo 8347121 1091656 6941
./stockfish-gcc 8347121 1067495 37877
./stockfish-gcc-pgo 8347121 1087086 31128
./stockfish-icc 8347121 1048282 79075
./stockfish-icc-pgo 8347121 1063484 54086
Although it seems that the stockfish-icc compile is the slowest it could still be the fastest because the error bars are high. More testing is needed to reduce the error bars and to decide which compile is fastest.
All these error bars can trick you if you have a systematic error in your testing. For example you want to know the size of your room properly and do a sample of 20 measurments. Unfortunately the scale of your ruler is wrong. In this case you would get little error bars but your result is terribly wrong.
I hope that my expanation on this is not totally off...
Re: Extract line from Sf bench command
Posted: Wed Mar 19, 2014 10:56 am
by ChessDrone
Thank you for the explanation. I understand a little better.
I ask if it is possible to make the "Bench" equivalent in Shell for engines such as Senpai who do not have this command. I would like to compare different compiler with Senpai.
Re: Extract line from Sf bench command
Posted: Wed Mar 19, 2014 11:16 am
by ChessDrone
The command sequence equivalent to "bench 32 1 6 default depth" should be this:
Code: Select all
setoption name hash value $1
setoption name threads value $2
isready
position fen $4
go $5 $3
positions used by stockfish
Code: Select all
rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1
r3k2r/p1ppqpb1/bn2pnp1/3PN3/1p2P3/2N2Q1p/PPPBBPPP/R3K2R w KQkq - 0 10
8/2p5/3p4/KP5r/1R3p1k/8/4P1P1/8 w - - 0 11
4rrk1/pp1n3p/3q2pQ/2p1pb2/2PP4/2P3N1/P2B2PP/4RRK1 b - - 7 19
rq3rk1/ppp2ppp/1bnpb3/3N2B1/3NP3/7P/PPPQ1PP1/2KR3R w - - 7 14
r1bq1r1k/1pp1n1pp/1p1p4/4p2Q/4Pp2/1BNP4/PPP2PPP/3R1RK1 w - - 2 14
r3r1k1/2p2ppp/p1p1bn2/8/1q2P3/2NPQN2/PPP3PP/R4RK1 b - - 2 15
r1bbk1nr/pp3p1p/2n5/1N4p1/2Np1B2/8/PPP2PPP/2KR1B1R w kq - 0 13
r1bq1rk1/ppp1nppp/4n3/3p3Q/3P4/1BP1B3/PP1N2PP/R4RK1 w - - 1 16
4r1k1/r1q2ppp/ppp2n2/4P3/5Rb1/1N1BQ3/PPP3PP/R5K1 w - - 1 17
2rqkb1r/ppp2p2/2npb1p1/1N1Nn2p/2P1PP2/8/PP2B1PP/R1BQK2R b KQ - 0 11
r1bq1r1k/b1p1npp1/p2p3p/1p6/3PP3/1B2NN2/PP3PPP/R2Q1RK1 w - - 1 16
3r1rk1/p5pp/bpp1pp2/8/q1PP1P2/b3P3/P2NQRPP/1R2B1K1 b - - 6 22
r1q2rk1/2p1bppp/2Pp4/p6b/Q1PNp3/4B3/PP1R1PPP/2K4R w - - 2 18
4k2r/1pb2ppp/1p2p3/1R1p4/3P4/2r1PN2/P4PPP/1R4K1 b - - 3 22
3q2k1/pb3p1p/4pbp1/2r5/PpN2N2/1P2P2P/5PP1/Q2R2K1 b - - 4 26
6k1/6p1/6Pp/ppp5/3pn2P/1P3K2/1PP2P2/3N4 b - - 0 1
3b4/5kp1/1p1p1p1p/pP1PpP1P/P1P1P3/3KN3/8/8 w - - 0 1
2K5/p7/7P/5pR1/8/5k2/r7/8 w - - 0 1
8/6pk/1p6/8/PP3p1p/5P2/4KP1q/3Q4 w - - 0 1
7k/3p2pp/4q3/8/4Q3/5Kp1/P6b/8 w - - 0 1
8/2p5/8/2kPKp1p/2p4P/2P5/3P4/8 w - - 0 1
8/1p3pp1/7p/5P1P/2k3P1/8/2K2P2/8 w - - 0 1
8/pp2r1k1/2p1p3/3pP2p/1P1P1P1P/P5KR/8/8 w - - 0 1
8/3p4/p1bk3p/Pp6/1Kp1PpPp/2P2P1P/2P5/5B2 b - - 0 1
5k2/7R/4P2p/5K2/p1r2P1p/8/8/8 b - - 0 1
6k1/6p1/P6p/r1N5/5p2/7P/1b3PP1/4R1K1 w - - 0 1
1r3k2/4q3/2Pp3b/3Bp3/2Q2p2/1p1P2P1/1P2KP2/3N4 w - - 0 1
6k1/4pp1p/3p2p1/P1pPb3/R7/1r2P1PP/3B1P2/6K1 w - - 0 1
8/3p3B/5p2/5P2/p7/PP5b/k7/6K1 w - - 0 1