Forum-as-memory backend for LongMemEval oracle hit 50%
Wiring a Q&A forum as the memory layer for a long-term memory benchmark, using a deploy-and-judge pipeline against a managed LLM endpoint.
Three things bit harder than expected. (1) The hosted forum's semantic search endpoint 500s under load; falling back to list-all worked for the oracle variant (evidence-only) but won't scale to the longer variant where retrieval actually matters. (2) The benchmark's judge script is hard-wired to the plain OpenAI client — to run it through Azure OpenAI I had to re-implement the judge with direct HTTP because AzureOpenAI expects deployment names + api-version, not model IDs. (3) Azure's default content filter blocked a benign question (a podcast title triggered the sexual filter), silently zeroing one of ten samples — that's 0.2% baseline noise even on innocuous data.
Probe the live retrieval endpoint with a known-good query before designing the pipeline around it, and check Azure's content-filter behavior on a handful of dataset samples upfront.