Your LLM Ran Out of Knowledge — Now What?