A few things. I'm price sensitive, so pricing was huge for me. The worst company also had the worst prices. I tried to ask them questions about how their backend works and they refused to answer. I spoke with the CEO and he said he couldn't reveal their "secret sauce". I said, "if you secret sauce is what infra providers you use and not your proprietary code, then you don't HAVE secret sauce and you're just reselling [Cloud Provider's Product]." Turns out that's exactly what they were doing. They were using Google Cloud for recording capture, and AWS for speech to text and then summary generation. I told them we would not ever be working with them.
For me the big things are price, ease of use, and data protection policies. I need to know the data never leaves the US, and I need to know what processors will touch it. Then if it meets those needs we'll do clinical demos and tests to get provider feedback. That's where we learn if it is clinically accurate. About half of them suck in the accuracy department.
What stands out to me the most is that the best companies have tended to be the small guys who have a strong grasp ion the entire stack and have somewhat simple apps. They focus on the tech, and have a minimal UI that just focuses on the main tasks and they don't spend engineering time on fancy pretty bells and whistles. If you see a simple UI, that's a good sign to me. Once you hit the big guys the quality goes down. Dragon Medical One is great for straight text to speech, but Dragon with Copilot for medical is really bad.
Yeah, I think they fine-tuned their model to be better at the pattern where you output citations that reference exact strings from the input. Previously that's been a prompting trick, e.g. here: https://mattyyeung.github.io/deterministic-quoting
Makes sense. I wonder if it affects the model output performance (sans quotes), as I could imagine that splitting up the model output to add the quotes could cause it to lose attention on what it was saying.
So after using Mem0 a bit for a hackathon project, I have sort of two thoughts: 1. Memory is extremely useful and almost a requirement when it comes to building next level agents and Mem0 is probably the best designed/easiest way to get there.
2. I think the interface between structured and unstructured memory still needs some thinking.
What I mean by that is when I look at the memory feature of OpenAI it's obviously completely unstructured, free form text, and that makes sense when it's a general use product.
At the same time, when I'm thinking about more vertical specific use cases up until now, there are very specific things generally that we want to remember about our customers (for example, for advertising, age range, location, etc.) However, as the use of LLMs in chatbots increases, we may want to also remember less structured details.
So the killer app here would be something that can remember and synthesize both structured and unstructured information about the user in a way that's natural for a developer.
I think the graph integration is a step in this direction but still more on the unstructured side for now. Look forward to seeing how it develops.
Thanks yding! Definitely agree with the feedback here. We have seen similar things when talking to developers where they want:
- Control over what to remember/forget
- Ability to set how detailed memories should be (some want more detailed vs less detailed)
- Different structure of the memories based on the use case
reply