SQLAlchemy 1.4 → 2.0 · migration answers from the docs
Every answer cites its page,
or says there isn't one.
Ask about anything that broke between SQLAlchemy 1.4 and 2.0. It reads the documentation, quotes it back with the page attached, and declines when the pages do not cover you.
Benchmark100 questions, checked by hand
- 0.64
- Right page foundin the five pages the model is given.
- 0.42
- Answered from itpage shown, and the answer uses it.
- 100
- Test questionseach with the page that answers it.
- 9
- Unanswerableno answer in the docs; should be refused.
- 3,284
- Passages searchedfrom the 1.4 and 2.0 documentation.
Sources
How it works
Three steps. The model never answers from memory.
-
3,284passages
1. Search
3,284 passages, cut from 270 files of the SQLAlchemy 1.4 and 2.0 documentation. Two searches run over them at once: one on meaning, one on keywords. Neither alone finds the page reliably.
-
5pages kept
2. Rerank
A second model re-reads the top candidates and can promote one page into the last of the five seats. It only reorders what search already found, so it cannot rescue a page search missed.
-
1answer, or a refusal
3. Answer
Five pages go into one prompt. The model answers from those five and cites them by number, or says they do not cover the question. It is never asked to answer from memory.
How it is measured
100 questions, every one checked by a person. Each carries the documentation page that actually answers it. Nine of them have no answer anywhere in these docs, and they are kept on purpose: they are the only way to find out whether the system admits it.
Half of them are real. 25 come from Stack Overflow and 25 from GitHub discussions, titles kept as they were typed, misspellings included. The other 50 were written here. The real ones score worse, which is the point of having them.
The answer pages were proposed by keyword search, a channel the graded system does not use. Picking them with the system's own search would have graded the benchmark against itself and scored close to perfect while measuring nothing.
Retrieval and generation are scored apart. Search puts the right page in front of the model 64% of the time; answers that use it arrive 42% of the time. That gap is not a search problem, and averaging the two would have hidden it.
What it does not claim
- A cited answer is not a correct answer. The citation tells you which page the claim came from, so you can go and check it. That is the whole job it does.
- The numbers belong to one model. They were measured with
qwen2.5-coder:7b. Every answer carries a notice saying which model wrote it and whether that model is the measured one. - It refuses more often than it should. Declining is deliberate, and it over-fires: on the test set it declined many questions whose answer page it had already found. Read the pages before you trust the refusal.
- Retrieval reproduces, generation does not. Search returns the same pages on every machine tried. The written answer does not, so the figures above are quoted with the machine they came from.