Yep, they are quite bad without stockfish. You can test it with the playchess skill in my repo. Maybe 1200/1300, who knows? Still I do think that they can, with enough time, explore multiple variations where they confront their naivety to stockfish and build up a compact picture on why move Y should have been played instead of move X. That was my intuition when building this skill.
Have you actually tested this with quiet positions, where understanding needs to be more about different strategic or positional choices? They can explain concrete tactical lines with stockfish analysis at hand, but so can an analysis board. In my experience, even with fairly deep and accurate engine analysis, they cannot do this accurately with quiet positions. I tried to build a product around this idea and I'm convinced that even if it is possible it would require hundreds of thousands - or millions - spent on post-training to even confirm that possibility.
Good point. I had to tweak a bit the system to get more positional analysis because this is something I wanted too get. I think it's still shying away on this aspect. When it does dwell on it, I feel that when the output describe what should be the plan of both camps, it's quite convincing but only a much stronger player (or me with stockfish) could really assess this.
Anyway I am with you that deep positional appraisal is very hard and we should not expect too much from this set of skills on that front.
Thx for tips. I just saw there are some courses on chess.com, that will be nice to compare claude understanding with Silman commentary as the gold standard.
Edit: I am pushing a small attempt on this position. I fed this one as a screenshot https://www.chess.com/lessons/roots-of-positional-understand... . I forgot to tell that it was black to move but that was fine. The analysis quality seems ok. What puzzles me is that it recognized the Carlsbad structure and talked a bit the plans around the minority attack, not only the correct explanations on g6 and why it should be played. This is where we benefit from positional stuff in the training data.
No need to apologize. I like reading HN for comments that don't beat around the bush.
It's true that it's costly. I never tried to optimize it. In a way I feel this hacky project doesn't deserve its place on the front page. It's just me hacking around on Claude Code to get something I like. Take it as a proof of concept if you will. I'd be happy to see lighter alternatives.
> Then presenting it as a project or tool of value to share to others is delusional
Here I see a gap in your reasoning. Lazy and costly, sure. But useless, I'm not so sure. In my world, a vibecoded tool can be useful enough to be shared, despite the risks. Claude Code itself is almost entirely written by Claude, according to its creator. Yet I use it every day.
No it does not help the quality of the analysis from what I saw, but it makes the analysis much more interesting, because it can challenge my wrong judgement and answer the questions that I asked myself out loud.
Very interesting project, is Maia used in this platform?
This is crazy. My claude chess stuff is (edit: was) currently near yours on the front page and noticed your post. I have a project very close than yours that I hesitated to share. From a quick glance, we went for a similar approach. I just open sourced it so that you can compare implementation notes. https://github.com/brumar/whiteboard-agents . It's not thoroughly tested but can be interesting to check.
Not by hand, no, the code was generated with claude code. The readme too, but with some extra efforts to avoid the awful ai generated readme.
It took multiple sessions to get to this result. At first I only generated annotated pgns and standalone html page inspired by lichess. The video generation was the cherry on top, it took few iterations too to fix issues and add markers and arrows. I only use consume the generated video these days, for the moment.
The jury is out for the effectiveness, it's hard to debate this subject. From my cognitive background I know very well how important the generative effect is for learning. Maybe I'll add features that leverage generative/testing effect one day. Anyway, clicking on stockfish branches can be quite a passive activity too if done badly. I don't know how to do that well to be honest. My goal is often to just to understand what I missed, full stop.
For the sycophancy, I can say I did not feel that at all. When stockfish says your move suck, claude would have a hard time saying the opposite (no "you are absolutely right" when I am not).
This what I think too. I also think there are much more refined approaches than the one I tried here. From a cursory look, I saw there are both scientific litterature on the subject of mixing llms and tools like stockfish and some dedicated closed platforms that put that into action.
Let's make clear that I did not spend much time on this project. Ideally I would have tried to put other models into the mix, like maybe Maia to better see the game from a "real player" eyes and pinpoint where expected move and stockfish moves differ.
Anyway, to me, it's good enough to be usable and shared.
That was exactly my estimation when I played "raw" against Opus. But Opus with stockfish as a tool and much time available can, from what I experienced, generate good comments.
reply