Using AI for French Reading Comprehension Practice - followup

Here's a quick follow up for the Using AI for French Reading Comprehension Practice post from a year ago.

The prompt I wrote was mostly working, but LLMs often went off-script in minor ways. I kept tweaking the instructions endlessly, even trying to tell it in various ways to think harder, but it didn't really make much difference. It was always at a barely good enough level.

I also hoped that the models would improve over time and become able to follow such instructions more closely, but they didn't. None of them did. At least for this kind of task, publicly available chat models are not meaningfully better than they were a year or two ago. Over the last year, AIs improved for some other modalities like coding agents, but you can't really use Claude Code for this.

So instead, I experimented with the prompt some more and the only thing that made a significant difference was including an example script.

With this kind of script, almost every AI stays on-script consistently, except some weak ones like DeepSeek Chat.

As a small aside, Gemini is completely unable to number paragraphs. I suspect it internally generates Markdown like 7. some text which some broken markdown to HTML converter turns into 1. some text automatically before you get to see anything. I base my guess mainly on Reddit having the same ridiculous bug since forever, and they probably vibe coded the Gemini web interface and copied Reddit's bug.

The main thing all LLMs are still really bad at is sticking to the B1 level, or any other specified level. They generally use grammar that's too simple (like exclusively using the present tense) and unnecessarily complex vocabulary where more common synonyms would be perfectly appropriate. I tried to specify which tenses they should mostly use, but they don't really follow these instructions too closely. I haven't found a good solution to this problem yet.

I don't have high hopes for rapid chat model improvements, so I'll probably need to keep tweaking this prompt a lot more.

Latest Prompt

Do the following:

* write a short text in French of about 500 words total
* the text will be split into 10 paragraphs of about 50 words each
* the story needs to be at B1 level, only using grammar and vocabulary appropriate for that level
* tenses appropriate for B1 level: présent de l'indicatif, passé composé, imparfait, futur proche
* write the first paragraph, then I translate it to American English
* then you let me know if it's correct or if I made any mistakes, but be extremely concise, and never mention anything that's already correct.
* then continue with next paragraph until the story ends. Automatically continue.
* after my last translation, print a scorecard for all 10 translated paragraphs
* do not nitpick English translation by suggesting a more natural one, as long as it translates French text correctly. Ignore any issues with missing articles.
* number each paragraphs

Example exercise (shortened):

<llm> 1. Harry Potter a une baguette magique
<user> Harry potter has magical baguette
<llm> Mistakes:
❌ "baguette magique" - in this context, magic wand
2. Sa chouette s'appelle Hedwig
<user> His owl's called Hedwig
<llm> ✅ Correct
3. ...
<llm> ✅ Correct
10. Ses parents sont décédés
<user> His parents are dead
<llm> ✅ Correct
All done. Scorecard: 3 mistakes total.

Topic for today: ...