From: Reijer Grimbergen ETL GO JP> Date: 3 mar 1998 Subject: Report 8th CSA Computer Shogi Tournament (long) Hi all, Finally got around to writing a report on the very interesting CSA computer Shogi Championships last month. Here it is for those interested. Reijer %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% The 8th CSA Computer Shogi Championships ======================================== 1) Introduction --------------- The CSA Computer Shogi Championships is an annual tournament for computer shogi programs. This year there were 35 participants, which was a slight improvement over the 33 participating programs last year. Markable absentee was the J3 program, which reached the final eight last year. This year's tournament had two foreign entries. Jeff Rollason's Shotest and my program Spear (an acronym for Shogi Pattern Evaluator And Recogniser and also the name used by some Western players for the shogi lance). The other participating program's were all Japanese and could be divided into commercial programs (9 programs), research programs (4 programs) and hobby programs (the others). The commercial programs with their front-runners Yamashita (YSS), Kanazawa (Kanazawa Shogi), Kakinoki (Kakinoki Shogi) and Morita (Morita Shogi) were again the clear favourites in this tournament. Compared to last year, there were some new rules. First of all, the time limits were only 25 minutes without byoyomi. Furthermore, a move had to take at least one second, even if the program did not use this second. Together, these rules already posed a serious problem for the programmers. The length of a shogi game is hard to predict, so it is also hard to decide how much time per move should be taken by the program. I will return to this issue later. The tournament rules were also different. The preliminary tournament was split in two groups. From the top group the best four programs would qualify for the final all-play-all tournament with eight programs (YSS, Kanazawa Shogi and Kakinoki Shogi were seeded and did not have to play the preliminaries). From the bottom group only one program would qualify for the final. Using this system together with the special speeded up Swiss pairing did not lead to ideal pairing as we will see. It was clear that the rules had been discussed extensively before the tournament and that the organisers realised the potential problems, since every participant was given a questionnaire in which he could give his opinion on the rules and give suggestions. I hope the CSA will use this information for future tournaments. 2) Before The Tournament ------------------------ My program was clearly in better shape than last year, when I lost two games because my program crashed. Last year the time to prepare my program for the tournament was clearly too short. I did not have this excuse this year and in early January I was convinced that I would get at least a 5-2 score and be able to fight for a place in the final until the end. If there were any new strong programs in the bottom group (in which I had to play because of my poor results last year), I would probably not make it, but if not, who knows... That was before I started my test games against AI-Shogi (the commercial version of YSS), Kanazawa Shogi and Morita Shogi. After losing about 20 games in a row against the lowest level of AI-Shogi, I realised that the selective search I used in my program (generating only a limited number of candidate moves in each position) was having too many holes and that a place in the finals might be a little ambitious. I decided that any score better than the 3-4 I had last year would be acceptable. I was still hoping for more, though... I worked hard for a month to plug the biggest holes, but with only three wins against the lowest level of AI-Shogi and no wins against Kanazawa Shogi and Morita Shogi, I was not very confident. Still, the Sheraton Hotel is a place of dreams, being so close to Tokyo Disneyland, so I packed in my Akia 200 MHz and went to Tokyo. I arrived at about 8 o'clock in the evening at the hotel. I could not help myself and worked on the program a little more, before I finally gave up at about 11 and watched the Olympics. My roommate Todoroki (of the Tancho program) was very talkative and it was a good thing I did not have to play myself, because we were up until well after three in the morning. He offered me some of the liquor he brought, but I must admit that the most powerful drink under the sun, an almost pure alcohol Polish drink (97%), was not really my cup of tea. 3) The Qualifying Rounds ------------------------ The tournament was scheduled to start at 9:20 in the morning, but after the opening speeches it became clear that there was a serious power problem. Computers at the end of the power cable would start up for an instant and then shut down immediately. Since the number of participants was not that much higher than last year, this was a bit strange, but it may have been that there were more powerful machines this year. In any case, the start of the tournament was delayed for about half an hour, but finally the problems were sorted out and we could begin. The two foreign entries started the tournament badly. I lost against Kenji Shogi, which was a new version of a program that had participated a couple of years ago (it now had the name of the son of the programmer). It was quite a tough opponent and Spear never really had a chance. Jeff, who was in the top group after missing the finals by one game last year, also lost in the first round, but since he had to play Morita Shogi, this was more or less expected. Furthermore, in a Swiss tournament system it is sometimes good to lose early, because you will be paired with weaker opponents. The second round was a disaster for Spear. After losing a bishop early in the game, my program fought a long and hard battle to get back from this material deficit. Slowly, it managed to reverse the position, but it was then that the time trouble set in. Because of a mistake in the blitzmode of the program, it did not spot a mate in one by the opponent in its fastest playing mode and lost a won game. This meant I now had the same start as last year and I was seriously worried I might not be able to even improve last year's score. Shotest had not much trouble with Super Hashimoto Shogi, which was a very old, very stable, but also very slow program and the weakest program in the top group. This round was also the first round we could hear the victory cry of the IS-Shogi team. IS-Shogi scored a remarkable victory over favourite Morita Shogi. IS-shogi (IS stands for Information Science) is a group effort of four students from Tokyo University. Last year it won five games in the preliminary tournament and missed the final by losing in the final round to S1.2. >From this point on, all things went very well for both Spear and Shotest. We both won four games in a row. Admittedly, I played the worst programs in the tournament and had wins of less than 60 moves (Japanese count) in two games where my opponents tried to find the shortest route to losing (which is very hard to program!). Jeff had more opposition and got an important moral victory in the 6th round, where he beat Yano Shogi, the program that kept him out of the final last year. Yano Shogi was the major loser this tournament. Being a finalist last year, it now managed to win only two games. It was unclear if this was because Yano had not been able to work on his program or because of introducing too many bugs together with the improvements. With one round left to play, I now had the minimal result I wanted with four wins. It was clear that there would not be more, since I had to play the commercial M-Brain, which has thus far won all its games. M-Brain is the brainchild of a small company called Alphagram. Alphagram has only five employees and M-Brain was operated by its managing director Ogi Takeshi. My program made a strategic blunder in the middle game and lost almost all of its pieces before it was mated. This meant that M-Brain was the winner of the lower group. There was one other strong program in the group, called Kawabata Shogi. However, Kawabata Shogi could not win a very good position against M-Brain and that was the decider. In the top group things had become very complicated. With one game left to play, only IS-Shogi was certain of qualification with a prefect score of six wins. Morita Shogi had only lost to IS-Shogi and had played all the strong programs and would probably qualify as well. For the remaining two spots there were still four candidates: Sekita Shogi (finalist last year), Yoshimura Shogi, Shotest and Sougin. Unfortunately the speeded Swiss system made it impossible to have a direct match to fight for the spots (a draw Sekita-Sougin and Yoshimura-Shotest or Sekita-Shotest and Yoshimura-Sougin were both possible). The draw resulted in the games Sekita-Sougin, IS-Shotest and Yoshimura-Takada. Jeff was not too thrilled about playing IS Shogi, since this program seemed to be the strongest in the group. If Shotest would lose, it was very much possible that it would not qualify for the finals even with a 5-2 score, since it had not played very strong opponents up until now. This was another drawback of the special Swiss that was used. In the end, it was not Jeff who had reasons to complain about the system. Shotest played a great game against IS and won after a long and hard battle. There was a small incident in the end, though. After the final move, there were some problems with the power. Of course both programs could restart the game, but the IS team decided to resign the hisshi position. This meant that Shotest was the first foreign program to reach the final. That was great, even though it took away one of my short term goals. Sougin beat Sekita and Yoshimura beat Takada. After the final round in the preliminary tournament there were three programs with a 6-1 score: IS-Shogi, Morita Shogi and Shotest. Sougin and Yoshimura had 5-2 scores and Sougin qualified for the finals with one extra SOS point. Yoshimura, shouted ``kaeroo!'' (``let's go home!'') and pretended leaving the hotel, but he was soon back to his normal talkative self. The evening after the preliminary rounds a party was being held, in good Japanese tradition. Some people were honoured because of their outstanding work for computer shogi. Morita shogi received a prize for his long involvement in the CSA and Todoroki and Seo received prizes for their work on solving Microcosmos, the longest tsume shogi problem (more than 1500 moves). There were also speeches by the programmers who qualified for the finals. While most said that they were happy to be in the finals at all, the leader of the IS team stated that it was their goal to win all games on the next day. I remember thinking how nice it was that Japanese students also like bluffing. Jeff and I, being a little older, looked at the table and made the prediction that YSS and Kanazawa Shogi would fight for the top spot and that after that it was anybody's guess for places three to eight. I will remember this the next time I want to bet the house on a certain winner... 4) The Finals ------------- On the day of the finals I became a supporter of Shotest and helped Jeff a little with understanding what was going on in his games. Jeff has a long history in chess programming, but has only played a couple of shogi games in his life. He seemed to be happy with my comments on the games instead of having to guess from the evaluation function value and I got to share some of the limelight for the first foreign program in the finals. I was not too sure about the strength of Jeff's program though. I had not seen it play in the preliminaries, but I knew it had been lucky in the draw, with only two strong opponents. Sougin also did not have to prove itself against all the strong programs and seemed a little lucky to be here. Therefore, I was not too surprised when Shotest lost quite badly in the first round against Morita Shogi. The real shocks of the first round were on the other boards. Both YSS and Kanazawa Shogi lost their first game! YSS seemed to have lost convincingly against IS-Shogi, but the game Kanazawa Shogi against M-Brain was decided under the time pressure of one move per second, where M-Brain made the fewest mistakes. If the first round was a shocker, the second round was more like an earthquake: YSS and Kanazawa Shogi lost again! YSS's loss against Shotest will probably go down in history as the first shogi computer game won because of opening preparation. Jeff had played AI Shogi automatically hundreds of times by making a program that could read AI Shogi's moves from the screen. The time used to make that program was well-spent, since I could only play about three or four games a day against Ai-Shogi by manually entering the moves. During these long testsessions, Jeff had found an opening where AI Shogi responded wrongly. Yamashita had not changed this in the new YSS program and it played the same opening again with disastrous results. Shotest got a promoted bishop and YSS a dead rook that would be lost eventually. The game took a long time, but YSS never had a chance to win. Yamashita's comments after the game was the prediction that next year Jeff's Shotest would be one of the seeded programs and that his program would have to try again in the preliminaries. Kanazawa's loss to Sougin was even more shocking. I was told that Kanazawa had completely rewritten his program and that might be part of the explanation. Still, there seemed to be no more doubts about the strength of Shotest and Sougin. It was strange that the victory of IS-Shogi over Kakinoki-Shogi almost went by unnoticed, except of course for the shouts of the victorious IS team. In round three the victory cry of the Tokyo students could again be heard. This time they beat Morita Shogi. In the endgame there was something strange, though. IS-Shogi seemed to be winning, but then the program defended wrongly against a desperate check. This allowed mate, but Morita Shogi was playing in its fastest mode because of time trouble and did not have its tsume-mode switched on. IS Shogi seemed a little lucky to get away with a win in the end. Still, it was clear that the party talk could hardly be called bluffing. IS Shogi had played three of the strongest programs and had won them all. They seemed to be cruising to victory. Meanwhile, Shotest lost a quick game against Kanazawa Shogi and at least half of the prediction of Yamashita seemed very far away. The other half had become almost a reality, because last year's champion YSS lost its third straight game, this time against Sougin. In the finals, any program seemed to have a chance to win against any other program. Round four was a quiet round with only one surprise: Shotest beat Kakinoki Shogi. This was one of the up-and-down games that would become the trademark of Shotest in this tournament. A little better after the opening, then losing in the middle game, fighting back hard and finally winning in the time pressure situation. YSS won its first game against M-Brain, Kanazawa beat Morita Shogi to keep his hopes for a miracle comeback alive and IS Shogi beat Sougin. This was probably the game where IS Shogi was in the biggest trouble. Like in the game against Shotest, it allowed its rook to be completely boxed in. In the end Sougin was not strong enough to take advantage, but it came very close to winning. The only problem with the tournament so far seemed to be that IS Shogi might be too strong and win all its games. This was no longer a concern after round five when a revived Kanazawa Shogi beat IS Shogi. YSS also got a good victory against Kakinoki Shogi. The game Morita against Sougin was also finished quickly, which meant that the audience and the comments of Mushano (professional 6-dan, who was doing comments on the games together with Horiguchi, 6-dan and Iida, 5-dan) could focus on the game between Shotest and M-Brain. For a long time there did not seem to be anything interesting in the game. Shotest started well, but then underestimated the strength of the M-Brain attack, which got decisive strength. Resignation did not seem far away. However, M-Brain used quite a lot of time and towards the 20 minute mark began producing some strange moves. Suddenly the game, which I had already declared ``over'', became interesting again. Shotest played an almost perfect endgame with some great human-like sacrifices that could not have been calculated until the end and finally won. Round six was the first round where all favourites won, with maybe the win of Kakinoki Shogi over Kanazawa Shogi as the only small surprise. Shotest again had a very tough game, this time against Sougin. However, Sougin allowed Shotest's king to enter, after which the king helped in mating. IS-Shogi beat M-Brain to keep a one point lead over Shotest, the only program with a 4-2 score after six rounds. This actually meant that the tournament was getting a real final. Instead of the scheduled final between YSS and Kanazawa Shogi, IS-Shogi and Shotest would play a ``winner-takes-all'' game. Before the decisive game there was a lot of excitement at the prospect of a non-Japanese program winning the tournament, not in the least by the delegation from the Itochu game company, who are planning a commercial release of Shotest this year. Since Shotest had beaten IS-Shogi in the preliminary tournament, it was far from clear who would win the game and the students were clearly worried that a certain victory would slip through their fingers. The coolest person was probably Jeff, who was very happy that he had come this far and was very much aware that he had no reason to complain about his fortune up until that point. After all the build-up the game itself was a little bit of a let-down. Shotest made some early strategic mistakes and the evaluation function dropped from slightly bad to very bad to disastrous and that was a good reflection of the situation on the board. In this tournament Shotest had been given many opportunities to get back in the game, but not this time. In the end IS-Shogi won without too much of a fight. Still, with his 4-3 score Jeff edged Kakinoki Shogi for third place on the result of their mutual game. So Shotest not only became the first foreign program in the finals, but also the first foreign program to win a prize. Kanazawa Shogi beat YSS in the final round, so now Yamashita's prediction had become a reality. YSS ended fifth with a 3-4 score and has to try and qualify for the finals again next year. Morita came sixth after losing in the final round to Kakinoki Shogi. M-Brain beat Sougin, which did not change the ordening of the programs. Sougin came seventh and M-Brain came eight. Both had a 2-5 score. IS-Shogi was the glorious winner of the tournament. In the end IS-Shogi was two points clear of the other programs and that was the right reflection of the differences in strength, even though there were some close games and it might have all gone wrong in the final round. IS-Shogi was clearly better than all of the other programs, which is amazing considering that it was written by a team of four students. The contest has been dominated by commercial programs for years, and it was good to see that the winner also took the best amateur prize home. I wonder how they will divide that prize into four (the amateur prize is a computer). 5) Observations --------------- Let's look at the program's perfomances individually. IS-Shogi played some good games in this tournament and is in my opinion close to dan-level strength. However, the games against Morita Shogi, Kanazawa Shogi and Sougin show that there are still some major flaws in the program. With a little more tuning and some minor improvements, I expect that the program will give players up to 1-dan a good game. I do not have the impression that the students used some revolutionary new techniques. They just used the existing techniques better than others. They also had managed to split up the work very well. Their comments that there was still a lot of intended work undone is promising for the future. I hope they will be able to keep working on the program even after graduation. Kanazawa Shogi had a rollercoaster ride this tournament. The program managed to get the runner-up honours like last year, but it lost against Sougin and M-Brain, which finished seventh and eight. After losses in the first two rounds, the chance of winning the tournament was already gone. Still, it was the only program to beat IS-Shogi. As said, it seemed that Kanazawa had completely rewritten the program. This is not a bad idea, since you can only hack on a big program for a number of years before you find that the effort of adding new things becomes too much. I think the program might not have been ready for this tournament. His comments on the losses in the first two rounds was that there were ``tuning misses''. I expect Kanazawa to be back next year with a much stronger program. Shotest was of course the second major surprise of the tournament. I have seen all of Shotest's games in the final and I think that an important reason for Shotest's success was its timemanagement. Shotest uses a formula that allots time depending on the expected number of moves left in the game, which is based on the position evaluation. This meant that when other programs got into time trouble, Shotest still managed to play at almost the same strength. This is not the only secret of Jeff's success though. His program is very strong in calculating tactics, especially in the endgame. I saw it play the correct sacrifices of major pieces of which the ultimate result must have been beyond the search horizon. The balance of material and position in the different stages of the game is one of the hardest things to program. There are some aspects in Shotest's play that still need attention. A weakness in Shotest are the middle game strategies, which are in part caused by Jeff's own unfamiliarity with shogi. I think these things are not too difficult to straigthen out, so I give Jeff a good chance of winning the tournament next year. Kakinoki was probably the unhappiest man of the tournament. He looked seriously upset with losing his seeded place to Shotest. I suspect that he put in a major effort this year to try and win this tournament. His program is always doing very well, but seems to be overshadowed by Kanazawa Shogi and even Morita Shogi, although it is clearly stronger than Morita's program. Kakinoki had a good win against Kanazawa Shogi, but he was never a contender for the title. YSS had a big fall after reaching great heights last year. It might be that last year's title took away some of the ambition of Yamashita. Or maybe he spent to much time on his Go program. In any case, YSS lost four games and seems to be back on square one. Yamashita did not take it badly though, he was all smiles throughout the tournament. I am also counting on him being back with something good next year. Morita Shogi seems to be slowly sliding down the ranks. Second place in 1995, third place in 1996, fourth place in 1997 and now sixth. Morita Shogi is still famous for its tsume shogi solving capability, but needs a major effort to get back to the top of the shogi playing programs. Sougin was the only program in the top group that might have used a different approach to the problem of a shogi playing program. I have not talked with the programmer, but the messages on the screen seemed to indicate some kind of selective search, with quick selection of primary candidate moves in a position. The program was quite successful with wins against YSS and Kanazawa, but still had too many problems to be a real contender. M-Brain was a first year program and winner of the B-group. I think it was the weakest program to qualify for the finals, even though its fight against Kanazawa Shogi was very interesting. With more time to iron things out, M-Brain may become a force in the future. This was one of the most exciting computer tournaments I have ever seen. It also might prove to be a very important for the progress in computer shogi. The domination of the commercial programs in the past couple of years gave the impression that there was not much room for algorithmic improvement in the programs. It seemed that among the strong programs, speed was more important than knowledge. This tournament was very competitive, with about ten programs with the capability of beating anybody. On the other hand. the flaws in the strongest programs show that there is still a lot of room for improvement. Because of these two factors, I think that next year's tournament can be very interesting with strong programs like YSS and Kakinoki being forced to make a comeback from the preliminary tournaments. My prediction is that there will be a real dan-level program within a couple of years and that professional strength might not be that far away, maybe less than fifteen years. But then again, this prediction might have as much value as my prediction on the evening before the tournament... 6) The Tournament Results ------------------------- 6.1) Preliminary Contest ------------------------ a) Upper Group No Program Name 1 2 3 4 5 6 7 W-L-D SOS SB 1 Morita8 3+ 2- 5+ 6+ 9+ 4+ 11+ 6-1-0 32 26 2 IS 16+ 1+ 6+ 8+ 13+ 9+ 3- 6-1-0 24 18 3 Shotest v2.0 1- 16+ 10+ 7+ 14+ 13+ 2+ 6-1-0 23 17 4 Sougin 13- 11+ 12+ 10+ 8+ 1- 6+ 5-2-0 24 16 5 Yoshimura 10+ 6- 1- 11+ 15+ 8+ 9+ 5-2-0 23 13 6 Sekita 15+ 5+ 2- 1- 7+ 12+ 4- 4-3-0 30 13 7 Amano 14+ 12- 8+ 3- 6- 15+ 10+ 4-3-0 22 9 8 S1.3 12+ 14+ 7- 2- 4- 5- 13+ 3-4-0 27 7 9 Takada v3.1 11- 13+ 15+ 14+ 1- 2- 5- 3-4-0 25 10 10 Maruyama v0.3 5- 15+ 3- 4- 12+ 16+ 7- 3-4-0 24 4 11 Tancho (Reiki) 9+ 4- 13- 5- 16+ 14+ 1- 3-4-0 23 5 12 Hyper1.9 8- 7+ 4- 13+ 10- 6- 16+ 3-4-0 21 6 13 Yano3 4+ 9- 11+ 12- 2- 3- 8- 2-5-0 29 8 14 Oki 7- 8- 16+ 9- 3- 11- 15+ 2-5-0 20 1 15 Yamada v1.6 6- 10- 9- 16+ 5- 7- 14- 1-6-0 21 0 16 SuperHashimoto 2- 3- 14- 15- 11- 10- 12- 0-7-0 24 0 b) Lower Group No. Program Name 1 2 3 4 5 6 7 W-L-D SOS SB 1 M-Brain 5+ 2+ 4+ 9+ 3+ 8+ 7+ 7-0-0 30.0 30.0 2 Kawabata 15+ 1- 9+ 11+ 5+ 4+ 3+ 6-1-0 27.0 20.0 3 KFend 11+ 6+ 10+ 4+ 1- 5+ 2- 5-2-0 31.0 18.0 4 Seo (Reiki) 13+ 10+ 1- 3- 11+ 2- 6+ 4-3-0 30.5 12.5 5 Super 1- 15+ 13+ 6+ 2- 3- 9+ 4-3-0 28.5 10.5 6 Kenji 7+ 3- 14+ 5- 8+ 10+ 4- 4-3-0 26.5 13.5 7 Spear 6- 11- 16+ 15+ 9+ 12+ 1- 4-3-0 21.0 7.0 8 NazotekiDengi 9- 16+ 15+ 13+ 6- 1- 10+ 4-3-0 20.5 6.5 9 YYY 8+ 14+ 2- 1- 7- 11+ 5- 3-4-0 30.5 9.5 10 Shouchan 12+ 4- 3- 14+ 13+ 6- 8- 3-4-0 25.0 8.0 11 Kikuchi1000 3- 7+ 12+ 2- 4- 9- 16+ 3-4-0 23.0 5.0 12 Saeki 10- 13- 11- 16+ 14+ 7- 15+ 3-4-0 16.0 3.5 13 Tsubakihara 4- 12+ 5- 8- 10- 16+ 14= 2-4-1 20.5 3.0 14 Kira 16+ 9- 6- 10- 12- 15+ 13= 2-4-1 16.5 1.0 15 KaminaChodan 2- 5- 8- 7- 16+ 14- 12- 1-6-0 23.5 0.0 16 OM-1 14- 8- 7- 12- 15- 13- 11- 0-7-0 20.0 0.0 6.2) The Final -------------- No. Program Name 1 2 3 4 5 6 7 SB W-L-D 1 IS 5+ 4+ 6+ 7+ 2- 8+ 3+ 19 6-1-0 2 Kanazawa Shogi 3 8- 7- 3+ 6+ 1+ 4- 5+ 16 4-3-0 3 Shotest v2.0 6- 5+ 2- 4+ 8+ 7+ 1- 11* 4-3-0 4 Kakinoki Shogi 7+ 1- 8+ 3- 5- 2+ 6+ 11* 4-3-0 5 YSS 8.0 1- 3- 7- 8+ 4+ 6+ 2- 9 3-4-0 6 Morita8 3+ 8+ 1- 2- 7+ 5- 4- 8 3-4-0 7 Sougin 4- 2+ 5+ 1- 6- 3- 8- 7 2-5-0 8 M-brain 2+ 6- 4- 5- 3- 1- 7+ 6 2-5-0 *Order of Shotest and Kakinoki Shogi decided by result of their mutual game.