Building LLM Memory From Scratch