This is an interview from the year 2006.
Public on a web page, today isn't longer available.
Michael Diosi found it in Google Cache. I created again an interview site.
Stefan Meyer-Kahlen (Deutschland)
Programmierer von Shredder
Germany in the year 2006 (to Shredder 10)
by Stefan Meyer-Kahlen & Alex Schmidt & Frank Quisinsky
translated by Daniel Mehrmann
Enhanced compatibility as a primary goal is a constantly well tuned and oiled business at Meyer-Kahlen. The magical word „Innovation“ is key here, with every new release of Shredder.
So Meyer-Kahlens customers and users have grown to the high quality standards of it's software and have raised their expectations accordingly for the upcoming release of Shredder 10.0. But who, if not Stefan, will be able to deliver (Linux Mac, x64, SMP, user interface, engine, UCI and more) to all these expectations? Only someone who has the capability to turn their new ideas and concepts into reality – our reality – will ultimately successful.
We think that all readers will agree with us that Stefan did a great job and really deserves his success story. He did move mountains with the design of his „UCI“ protocol, offering developers new ways to enhance their projects under the hood and each have their own kings perch on throne on said mountains. There was no altruistic afterthought when he offered the protocol to the general public and chess programming enthusiasts for free use, worldwide.
So what is it that Stefan Meyer-Kahlen was cooking up in secrecy during this last year? Since we thought we ought to talk to this interesting and popular programmer, we interviewed him. So we hope that you will find the answers, hints or conclusions you might be looking for right here, in the interview. So here goes...
01. Frank Quisinsky
The last 12 months in the history of chess programming are an amazing step ahead – with amazing speed, considering the increase in game play strength that some engines have attained. Nobody really would have believed that by mid 2005 Shredder would be pushed off the throne of engines. With „Fruit “ at the years end it had to step down for „Rybka “. So – are you happy that there are so many strong chess engines out there that are spoiled with your creation, the UCI compatibility? Maybe you're not so enthusiastic anymore with the latest events – or is the fact that UCI is helping out a great deal alone enough to make you smile regardless?
That's right, a lot has happened during this last year. But as before – I think that all the new diversity is one really good thing to have happened in computer chess and Shredder, too. In these past years we had this static picture of all the very same engines battling it out within the ranks of the best all the time. That's become boring after a while. Sure, Shredder will face much more fierce games now, but I see this as a rather positive thing. It sure feels good that most of the engines now support UCI, that way my own graphical user interface will also have lots more chess engines to chose from.
02. Frank Quisinsky
We all know the story of the sleeping beauty and the prince. Now the „Fruit“ sources are open and free and have been thoroughly analyzed by other developers, no doubt. The question of history pruning does arise now – has it been sleeping all this time? With everyone free to look at the source, does that mean the secret ingredient of your chess programming is out now? Will pruning now become essential to future increases in game play strength? Maybe you have other ideas already on your mind? Maybe purely chess knowledge-based?
Last years interview comes to mind. We'd been talking about Fruit, too. I still do see the open sources of Fruit as a good thing because from my understanding that was the most prominent steppingstone for the big leap ahead last year. Of course I've been reviewing the source code myself, but didn't find something that could've been useful for Shredder. Some of the concepts in Fabien's Fruit cannot simply be applied to Shredder because they just don't fit or be made to fit in. Then again there were other methods I found in Fruit that I've already been working on in Shredder for several years already. Fruits code is well written, so it's easy to understand it and reconstruct. But that in itself is a problem too because it's tempting to just copy instead of doing your own homework.
03. Frank Quisinsky
Due to these circumstances these aren't the best times for interested top programmers to hop aboard. Making money on that market getting more and more difficult due to the sheer amount of strong chess engines now. Mark Uniacke (Hiarcs) is going to publish his UCI enabled Hiarcs soon. With those masses of strong products in the competition you're probably concentrating on the more unexpected bits you need to work on instead of realizing your own ideas to make it in time. Do you feel overwhelmed do you need more time for your graphical user interface or other things you want to flesh out?
Time definitely is a problem. I work on a number of projects simultaneously, many of them in hiatus due to the lack of hours the day has to offer me. Luckly I'm not all alone as I've found a very good helping hand in Eiko Bleicher. He's written a great deal of code for the graphical user interface for Linux and the Macintosh, is responsible for the on line-database for Shredder and put significant amounts of work into the Shredderbases. Now he's writing on Shredder mobile, that's a Shredder version for cellphones due to see it's release this summer. I'm also supported in the various other projects I'm on. But that doesn't change the fact that I always have more projects to work on than I do I have time to get them all done.
04. Frank Quisinsky
Please prioritize the following points of interest (1-8, or high / low)
Telling you this in concrete numbers is difficult, so I'll just give a commentary on the points below.
1. Engine development
That sure is important. One could say Shredders game play strength is already way beyond good or evil so that here I'll have to think about what to make of it – how to make sensible use of that strength.
2. GUI development
Not to negligible the usability, and it's a welcome change of scenery compared to the engine business.
3. Linux & Macintosh support
A lot of work has to be done, the list of features for the graphical user interface is lengthy.
4. UCI support
I still discover conceptual problems within UCI even after years have passed, and I'm still extending on it's features while keeping the compatibility well oiled.
5. SMP / x64 support
x64 isn't mass marketable yet. SMP is coming up through the dual core technology. And working on parallel processing is a lot of fun.
6. Pocket PC
It's one of the many projects I've kept thinking about for a long time now, but with the lack of time I couldn't even get started.
7. Chess 960
Really expected more of this. It's not a commercial success but a welcome change so I'll attend the next world championship as well.
8. Brainstorming and new innovative ideas
That's a fundamental thing overall. Only through innovation you're there to stay.
05. Frank Quisinsky
I see the commercial engine developers as kind of a bigger family than we have, competition aside. How'd you react to the proposition of merging the Shredder business together with products coming from other programmers that make their chess engines compatible to UCI? Would sales increase and would your customers profit from this? For example – every talented programmer could bring in some additional strengths into the package and thus help building a stronger better deal for the market.
This has been discussed with many programmers without really getting anywhere. They usually wish to to keep developing their own engine so that combining ideas is more complex than necessary. Combining products for publishing has also failed up until now for various reasons.
06. Frank Quisinsky
Enthusiasts kept comparing chess engines with each other for many years now. To them it's more than just a hobby. They don't care much for usability in graphics user interfaces. Analysis and it's complex possibilities are most basic functions and should to be integrated into any modern user interface though. Through your additions and enhancements to the UCI protocol you underscore the importance of options for the analysis. Another key improvement for you so far seemed to be endgame databases. I'd like to ask you two questions here: Do you believe 5-Steiner table bases slowdown the processing in engine vs engine matches? Wouldn't you think that the 4-Steiner table bases were more appropriate to gain a little more strength?
To be honest with you I'm always amazed when I see the scoreboards and analysis where there supposedly is some proof that table bases don't really add to the game play strength. Not to say that I have different results than anyone else has. I'm just wondering why nothing much would improve in the end.
07. Frank Quisnsky
Would it not make sense to discuss this anew? Maybe discovering possibilities to further optimize usage of table bases and to speed them up along the way? A good idea should be put to proper use! I'm sure that you've been thinking through this. Especially now since you're adding special feature offers to programs and graphics user interface options (like the end game oracle, the freezer, etc.).
That's right, Shredder has been quite an innovative with the end game database. That still applies to Shredder 10 as well. We constructed a new format for the end game database, the so called Shredderbase. By principle they're essentially the same thing as Nalimovs. Except the difference is that the Shredderbases are a lot better compressed. The high compression enables Shredder to load the entirety of it into the memory, thus speeding up accesses to the Shredderbase by a factor up to 1000 compared to the Nalimov databases. We archived compression this high by only having it save information whether a position has been won or lost before. Looking up this information will not make a difference in speed. As far as I know this is not a first, but the rate of our compression has yet to be challenged by any other such mechanism. All 3 and 4-Steiner together take up only 1 Megabyte, and all 3,4 and 5-Steiner use up a mere 157 Megabytes of RAM. To complete the picture: Nalimovs memory usage would amount to 7 Gigabyte. The sheer difference in size here opens the possibility to keep the whole thing in RAM on todays standard PCs without a hitch. Both above mentioned databases are included with Shredder 10. Deep Shredder 10 is special – we'll have the 3,4 and 5-Steiner in a slightly different format taking up approximately 441 Megabytes of RAM, but accessing the data is faster by the factor of 10. Shredder can use those quick lookups during any time in the game. In some positions during the endgame this will enhance playing strength and analysis enormously.
08. Frank Quisinsky
Your customers will be very happy to hear this. However I have two more questions now. Will you publish and open the protocol for Shredderbase support or will usage of the Shredderbases only be possible when using Shredder and it's user interface? Even though you're not required to go and tell the world how it all works on the inside, the question means a lot to many people.
First of we're planning to have Shredder use those databases. Development was extensive and thus costly, so for now we'd like for Shredder to have the exclusive right. Maybe we'll offer a way to license Shredderbase off when we see interest in it. And at a later time we will probable open it to the general public.
09. Frank Quisinsky
There's been this discussion two years ago in the Arena Team about the development of a database protocol. We're still don't have a free database protocol as you know. Some well known and talented programmers were seriously thinking about this. The current standard “PGN” is too clumsy, and other good commercial concepts are not freely available to us. Would you implement a well thought-out database protocol in your graphical user interface and consider some database options in Shredder-Classics interface, too?
Yes, I've been thinking about such a format from time to time. PGN does have its limits. But, as said before, I just don't have the time to pursue the idea very far right now. If a new format like this suddenly pops up it'll also be difficult to make it more popular.
10. Frank Quisinsky
Let's talk some about creativity. So how're things going with the ongoing developlemt on Shredders graphical user interface, right now? Is your wife helping out with ideas or maybe graphics ?
My wife studied design art and is a photographer. I do ask her for her opinion on the design of chess pieces sometimes. Then we'll come up with really abstruse ideas and suggestions, but most of the time her feedback is extremely helpful. Working together with several graphics artists responsible for webdesign and the user interface, I think the design of Shredder as well as the site has much improved in recent times.
11. Frank Quisinsky
Another rather interesting subject are the many rating listings like CEGT , CSS , CCRL , the works Michael Koppel came up with, Gerhard Sonnabend , Kurt Utzinger , Per Elbæk Jørgensen and many more that I can't really all list here. Do you check those on a regular basis and do you also look into the actual games recorded? Also, would you like to offer and kinds of suggestions for the keepers and creators of those lists?
I do look at the rankings and I'm frequently reminded by others to check here and there. Apparently there are a lot of people well involved, improving all these rankings, comparisons and lists. It doesn't seem like there actually a perfect way to do this, but I'm delighted that so many people enjoy working with chess engines.
12. Frank Quisinsky
Maybe the world championship test is another interesting matter to talk about. You are a collector of particularly interesting positions. What do you think about that test, what do you think about Michael Gurewichs enormous and tireless efforts put into them? Maybe you could also share with us three examples of those insightful positions you collected?
Positional tests and calculation could provide endless hours of debating and there has been plenty of that. I don't believe that you can deduce game play strength or analytical strength through positional tests alone. That doesn't mean those tests are not a good idea. It makes sense to use them anyway. But do not forget why we all keep playing and working on computer chess: it's fun! Now when you have a certain position it's enjoyable to take a good look, compare how quick our programs are to find the solutions for problems. With a full set of tests you can see that program A solved 8 problems while program B solved 9. Sure that is a legitimate way to compare and decide that one program was doing the job better than another in the case. Test positions help us find the flaws and strengths of the engines, but you have to be careful how you determine what to make of the results, whether some of them mean a good or a bad thing. That is not always easy.
13. Frank Quisinsky
Many people believe that providing good freeware products means putting the commercial products into danger. And potential customers come with high expectations when they consider buying. For example Toga (version 1.2.a) will reach 2.800 ELO, which is 50 ELO stronger than Shredder 9.0. When you look at this huge package of freeware and open source from the viewpoint of a professional programmer and about marketing your own product – is this trend a good or a bad thing?
Good thing. The scene is bristling with life through the many engines existing that you can compare and analyze together. Strength alone is just one thing. There is so much more to it than just that.
14. Frank Quisinsky
I am under the impression that more and more well known programmers are shying out of official tournaments. Is the loss of the world champion title a heavy blow when it comes to marketing the engine? Will Shredder still be there and up to the challenge or would you rather have it play the human masters of chess, now? I thin this be quite an issue now, if not later. Less powerful engines with little success have been used in show matches a lot. Maybe human players are also kind of shying out in the face of the challenges come to life through powerful computer chess engines like Shredder?
Since I've been involved I've seen that not all the most important engines always attend the tournaments. It's up to the programmers if it's worth it. But it's not right to say that “this wasn't a true world championship” just because this or that engine wasn't participating this time around. Everyone's free to join in at anytime, and those don't want to – they don't have to, but should not complain that in the end, somebody else won the title. The championship is still a great goal for me. The challenge is just plain fun. I've found good friends among other programmers in the years back – so the social component of it all is very important to me.
15. Frank Quisinsky
Even though winboard isn't that big a deal to you anymore Shredder 3.0 had excellent support for it. I would think that you still know a great deal about that protocol. We could still use a lot of the stronger engines that are only compatible to winboard using Shredders own user interface. Do you still use winboard for anything? Or maybe through some kind of adapter in Shredders interface for winboard engines? How do you go about this?
I only play using ICI enabled engines. There has been an immense increase in their numbers and choice is good. Yes, I know there is some excellent conversion software available that you one can use to have winboard and Shredder work together. However, I am concentrating on UCI, no time for another protocol.
16. Frank Quisinsky
Since you're working and supporting multi platform I'm wondering which one you're personally using for testing. Are you doing that on Linux, on the Macintosh or the Windows operating system?
I'm using the Macintosh for that. Bought it last year to start working on the Macintosh Shredder. Before that I perceived the Macintosh simply put as an expensive device for people with no understanding about computers whatsoever, and those people would grab the thing only just because it's supposedly a cool thing. Of course I realize today that this was far from the truth. The Macintosh really is fantastic. That user interface is so easy to use and intuitive to boot, and when you've been flipping through windows for long enough you can still open a shell to a real Unix underneath the hood with all the bells and whistles that has to offer. And those Macs, they look absolutely fabulous. Up until now I didn't care at all what my computer looks like, but now that I see the machine sitting next to my PC I'm in doubt if this is still the case. There is one little problem with it though: my wife really thinks its hot as well and only rarely if ever resorts to using the PC for her Email and all the rest of the Internet. She also addresses it as “my Mac” already.
17. Alex Schmidt
Now we're all thrilled to find out if the coming Shredder will climb back to the top of the ELO lists. How many ELO points do you think Shredder has gained since the previous published version, and did the play style change in some way? And it would be interesting to know how you personally determine the strength of the engine.
Our tests indicate that Shredder 10 has gained 70-80 ELO since Shredder 9. I ran a combination of test games against selected opponents. We take all the games into account and calculate another new ranking list which compares all the various Shredder versions and how they fared as well. Play style should be similar to that of Shredder 9. There has been some criticism about Shredder playing too optimistic so it's bound to stumble and fall. I was about to change that, but then reconsidered because that kind of play style is what many more people see as a positive thing. So it remains in there. It's still quite an aggressive style, you see. That part of Shredder has really grown on me as well, another reason to stick with it.
18. Alex Schmidt
So where in Shredders clockwork are the cogs that result in the enormous increase in strength – please try to express using the list below, and add percentages:
- Small, tiny and baby step improvements over the years (finetuning)
- Experience / Ideas learned from other programs (Rybka / Fruit...)
- New ideas
- The new Shredderbases
- Personal ambitions to regain the title
- Coffee (to stay wake during long nights dedicated to brainstorming
It's hard to put in numbers. Everything's involved at some point. Except coffee. While I enjoy a good cub of coffee it doesn't make me believe I should work during the night just staying up thanks to caffeine. This is a common theme among programmers I guess, but I'd think that is one way to amass a lot of new bugs. And my nights are usually over quick 'cause then both my kids wake up early. You really start thinking twice about whether it's a good idea to stay up longer or not.
19. Alex Schmidt:
You told us in your last interview that it is your goal to steadily improve on Shredders game play. Is there something specific that Shredder 10 can do better than it's predecessor and did you need to compromise on some features to get there – meaning: is there something that Shredder 10 has lost compared to it's earlier versions?
I hope that Shredder hasn't lost or forgotten anything whatsoever. As stated above, I tried to keep the playing style. There are many improvements regardless as I've been working hard on every piece of the program. It's gotten much better at assessing every situation in the game. I believe the right thing to say is that it's slightly improved everywhere all around.
20. Alex Schmidt
We often talk about how the engines play and how the chess masters play in these times. It's easy to forget that the engines basically only “just” exist in the form of programming code. Do you find Shredder more lifelike than that or do you solely see it as your work?
That's my job, this is how I earn my keep, but Shredder is a much different Job than, say, working for some insurance or bookkeeping. I got lucky that I have a job that's really fun to me and that can identify myself with it. In past times I said that Shredder is my child. Now that I have two kids this isn't how I look at Shredder anymore. Still, I see more within Shredder than just a bunch code. It does have some bits of personality.
21. Alex Schmidt
You send us some screenshots of the new Shredder “classic” user interface: Apart from the new design with the arrows for marking moves we found one new option: “Set Position Value”. Would you explain us what this option is about and what we could use it for?
It's one of the highlights of the coming interface and engine. I had improved on Shredders analytical capabilities when Shredder 9 came out. Now, while analyzing Shredder is a lot better at remembering / keeping in memory what he concluded in previous analysis runs. There's been this problem with Shredder insisting on a certain move we know will lead it into certain doom, just because it analyzed that once and found it really “smart”. So to help Shredder you'll have to run it through the entire tree of possibilities to help it discover that this really isn't that good a move after all. It has to learn this fact by itself, even if a human can sometimes see this coming from a mile away. This is where the new twist enters: You can tell Shredder what you think about conclusions of it's analytical thoughts. Shredder will take your word for it and learn. In our example you can simply review the analytical problem in question and give it a lower number, which will result in Shredder continuing it's analysis looking for an alternative. This is a lot more powerful than, say, simply adding the bad move to a a list of moves never to make or restrict the moves to a few select ones. This new way to deal with this kind of problem is better as Shredder can go back any point in it's analysis and start over still taking all your input into account accordingly. This can also speed up the analysis quite a bit. But this feature must be used carefully, because you'll have to think things over yourself before you claim that a certain move is a bad idea. Remember, Shredder takes your word for it and won't come back to ask you if you're sure. If you make mistakes at this point it'll ruin the analysis.
22. Alex Schmidt
This sounds very interesting. Do you included your new ideas in the UCI protocol allready that other programers may use it too ?
Yes, i have extend the current UCI protocol. The new command for this feature is UCI_SetPositionValue.
23. Alex Schmidt
What other new features does the interface has to offer?
We've talked about the Shredderbases, the new analytical features and direct access to the opening book containing 16 million positions on our server on the Internet, many more beautiful sets of chess pieces to select from, colored indication of legal moves and direct threats. Shredder can calculate the ELO of the player and will (if the player so desires) always play with the determined same strength the player has. I've always wished for Shredder to be able to generate a graphical game profile within seconds, full automatically. This allows much faster reviewing afterwards.
24. Alex Schmidt
Shredder and it's interface are famous for it's runtime stability compared to a number of other commercial interface products, and I don't this is going to change with the coming release. Is there a secret in this? Well structured and commented sources? A circle of exquisite beta testers? It would be nice to learn how you go about including new options in your interface. Do you do it separately and test new pieces of code extensively before you merge them into the main source tree or do you out them directly into the whole thing?
All of the above. I'll normally put them right into the sources, but I still test them through and through at once. Meaning, I'll get to work on the next point in the to-do list when I am absolutely certain that what I'm doing right now is well done.
25. Alex Schmidt
One often has the feeling that we've seen all there is to be seen in the world of the user interfaces. But then we find that there's more, and we're positively surprised with fresh ideas. Is it getting harder and harder to come up with new ideas? Do you check out other interfaces to see if can find a good idea in there and use it yourself? And, are there some features that you personally especially like?
You'll always find something you could've done better in both the engine and the interface. You'll find something that needs to be put in there soon enough. I do look at other interfaces, and not only just those that have to do with chess, and I try to find out how they deal with certain aspects. It sure isn't easy to come with with completely original ideas, but that's never been easy. You can use a lot more technical gadgetry, like the Internet for example, as that is what Shredder is using for it's endgame databases and the new opening book. Way back it just wasn't possible to do this, but today it is possible to access databases that are normally difficult to install on your own computer.
26. Alex Schmidt
Are you happy with what your interface is offering now? Things users are looking forward to use?
While I of course have ideas for the next version already, as I've said before – there is always something that can be improved about the interface. I don't want to go into details concerning the distant future of the interface since, well, Shredder 10's just been finished.
27. Frank Quisinsky
Which of the questions asked so far might have gotten under your skin? Maybe this is interesting for your pulse histogram.
Drinking a glass of wine, just sitting and typing here, I feel fully relaxed.
28. Frank Quisinsky
In closing, would you tell us about the price strategy you've got planned as well as update prices for Shredder 9.0 users? Will you offer the dual CPU version separately from the x64 version? Will it be possible to get a better deal in an all-included package with Freezer, Shredder 10 and Deep Shredder 10?
That will all remain as it is. Shredder Classic 2 is a budget buy at 30€, Shredder 10 UCI will cost approximately 50€ and Deep Shredder 10 UCI will cost 100€. All with the same user interface of course; the engine is what you're essentially paying for. The x64 version is still only available as Deep Shredder 10 UCI. There won't be an update for older versions, nor will there be a bundle including Freezer – at least for now.
29. Frank Quisinsky
What's your bet on the outcome of the opening game of the soccer world cub “Germany – Costa Rica”? Careful now! If you're wrong you're going to owe us, yes? Let's say, you owe us 1:1 winboard support in Shredders user interface. What do you say?
Gosh, my pulse is rising up now :-). There's a rumor that all opening games always end up 1:0. There's no way I can bet on Costa Rica here, so 1:0 for Germany it is.
Alexander & Frank
We wish you good luck and success with Shredder and many more interesting ideas!
Copyrights and Sources by Frank Quisinsky
Further publication of this interview is allowed and wished,
but with the permission of Frank Quisinsky only.
One condition is the complete casting (e.g. inprinted media).
Frank Quisinsky & Alex Schmidt, 2006