i) To pick a reference year, it seems reasonable to take the mid/late 1990s:- Almost all chess engines before ~1996 lacked (or had serious inefficiencies) using multi-cores (very lengthy discussion here).- Chess protocols became available, so that the engine and the GUI separated. That makes it straightforward to automate games for benchmarking.- Modern engines should work on machines of that age, considering RAM constraints.- The most famous human-computer games took place in 1997: Kasparov-Deep Blue. That's almost a quarter of a century ago (nice round n... (read more)
Thank you for your interest: It's good to see people asking similar questions! Also thank-you for incentivizing research with rewards.
Yes, I think closing the gaps will be straightforward. I still have the raw data, scripts, etc. to pick it up.
i) old engines on new hardware - can be done; needs definition of which engines/hardware
ii) raw data + reproduction - perhaps everything can be scripted and put on GitHub
iii) controls for memory + endgame tables - can be done, needs definition of requirements
iv) Perhaps the community can already agree on a set of experiments before they are performed, e.g. memory? I mean, I can look up "typical" values of past years, but I'm open for other values.
Right. My experiment used 1 GB for Stockfish, which would also work on a 486 machine (although at the time, it was almost unheard of...)
(a) The most recent data points are from CCRL. They use an i7-4770k and the listed tournament conditions. With this setup, SF11 has about 3500 ELOs. That's what I used as the baseline to calibrate my own machine (an i7-7700k).
(b) I used the SF8 default which is 1 GB.
(c) Yes. However, the hardware details (RAM, memory bandwidth) are not all that important. You can use these SF9 benchmarks on various CPUs. For example, the AMD Ryzen 1800 is listed with 304,510 MIPS and gets 14,377,000 nodes/sec on Stockfish (i.e., 19.9 nodes per MIPS). The oldest CPU in the ... (read more)