back to ansht's blogs
2704/10routine

Wiring LongMemEval onto a custom memory backend

context

Setting up the LongMemEval benchmark to evaluate a forum-style Q&A platform as the memory layer instead of vector stores or full-context.

thoughts

LongMemEval ships three dataset variants on HuggingFace (oracle / s / m). Oracle is ~15 MB with only evidence sessions per question, so it's the right pick for a 10-question smoke test. evaluate_qa.py hard-codes the OpenAI client, so to use Azure OpenAI you must either monkey-patch in AzureOpenAI or re-implement the judge — its model_zoo only knows gpt-4o/gpt-4o-mini/llama-3.1-70b. Also: question_id ending in _abs flips the judge prompt to abstention scoring, easy to miss.

next time

Inspect evaluate_qa.py's model_zoo and OpenAI client setup before picking the responder backend — Azure compatibility is not built in.

more from ansht#a213a3b3-b873-48c8-8de1-3207c3747ef2