There is nothing wrong with the HTTP layer, it's just a way to get a string into the model.
The problem is the industry obsession on concatenating messages into a conversation stream. There is no reason to do it this way. Every time you run inference on the model, the client gets to compose the context in any way they want; there are more things than just concatenating prompts and LLM ouputs. (A drawback is caching won't help much if most of the context window is composed dynamically)
Coding CLIs as well as web chat works well because the agent can pull in information into the session at will (read a file, web search). The pain point is that if you're appending messages a stream, you're just slowly filling up the context.
The fix is to keep the message stream concept for informal communication with the prompter, but have an external, persistent message system that the agent can interact with (a bit like email). The agent can decide which messages they want to pull into the context, and which ones are no longer relevant.
The key is to give the agent not just the ability to pull things into context, but also remove from it. That gives you the eternal context needed for permanent, daemonized agents.
I've been working on a coding agent that does this on and of for about a year. Here's my latest attempt: https://github.com/vanviegen/maca#maca - This one allows agents to request (and later on drop) 'views' on functions and other logical pieces of code, and always get to see the latest version of it. (With some heuristics to not destroy kv-caches at every turn.)
The problem is that the models are not trained for this, nor for any other non-standard agentic approach. It's like fighting their 'instincts' at every step, and the results I've been getting were not great.
> allows agents to request (and later on drop) 'views' on functions and other logical pieces of code [...] The problem is that the models are not trained for this
Fwiw, I was playing with an "outliner"-tool collapse/expand idiom, on synthetic literate-programming markdown files, with #ids on headers and blocks. Insufficient experience to suggest it works, but it wasn't obviously not working, and that with a non-frontier model and very little guidance. Other familiar related idioms include <details>/<summary>, hierarchical breadcrumbs, and plan9-ish synthetic filesystems `foo.c/f.{c,dataflow,etc}`. One open question was comfort with more complex visibility transformations or sets - "hide #bar; show 2 levels of headers-only under #hee; ...". Another was cleanup - recognition of "I no longer need this and that".
I'm using vector embeddings for creating code views based on semantic search, initially based on the user prompt. That really works wonders to give the agent a flying start.
I guess the short-cut is to include all the chat conversation history, and then if the history contains "do X" followed by "no actually do Y instead", then the LLM can figure that out. But isn't it fairly tricky for the agent harness to figure that out, to work out relevancy, and to work out what context to keep? Perhaps this is why the industry defaults to concatenating messages into a conversation stream?
My guess (I will test this eventually) is that you set a window size (which may be the model limit, or lower to reduce input token costs), the harness then refuses to show items that don't fit. If the model emits a command to read a file, the harness then says "File hidden due to lack of context space". In the system prompt, the model is informed about the context space usage, and that it can hide files. It needs to be instructed that if files contain something noteworthy, that the agent notes this down in their notes, which should always be rendered into the context. If this fails, the agent will hide a file with relevant information and then get lost in circles. If it succeeds, the agent can work on larger tasks autonomously. So it's worth trying.
God knows why you think this is possible. If I don't even know what might be relevant to the conversation in several turns, there's no way an agent could either.
One of us is confusing prediction with retrieval. The embedding model doesn't predict what is going to be relevant in several turns, just on the turn at hand. Each turn gets a fresh semantic search against the full body of memory/agent comms. If the conversation or prompt changes the next query surfaces different context automatically.
As you build up a "body of work" it gets better at handling massive, disparate tasks in my admittedly short experience. Been running this for two weeks. Trying to improve it.
So the embedding model is a fixed-size view on a arbitrarily sized work history (tool calls, natural language messages)? The model is like a summarizer, but in latent space? And not aimed to summarize, but trained to hold whatever is needed for the agent to be autonomous for longer runs?
Now I see why Anthropic isn't too happy with third party clients. The clients may not be so nice to their capacity as their own client, which has the interests aligned with minimum token consumption. A tricky dynamic.
Yes, it will destroy most of the caching potential. On the other hand, the average context window needed to achieve the same type of task may be much smaller. This might make up for it. And with a better harness, fewer rounds may be needed. Plus, hopefully costs will go down. There is a lot of hope in this comment though.
> The key is to give the agent not just the ability to pull things into context, but also remove from it
Of course Anthropic/OpenAI can do it. And the next day everyone will be complaining how much Claude/Codex has been dumbed down. They don't even comply to the context anymore!
> Every time you run inference on the model, the client gets to compose the context in any way they want; there are more things than just concatenating prompts and LLM ouputs.
You can always launch a subagent with a fresh context. There are further things that you could do by tweaking the underlying transformer model (such as "joining" any number of independently cached contexts together on an equal basis, without having to rerun prefill on the "later" contexts) but this is quite general already.
A smalltalk or Erlang for AI agents is an interesting thought. Smalltalk for the design in terms of message passing and object-oriented holding of state (agents are stateful and are reached via their public interfaces), Erlang for the elegant execution of it with actors and mailboxes (agents have inboxes and outboxes and can work concurrently at scale). Might as well go the whole hog and put a supervisor AI agent in as a switchboard.
Or they tried some simple alternatives and didn't find clear benefits?
> The key is to give the agent not just the ability to pull things into context, but also remove from it.
But then you need rules to figure out what to remove. Which probably involves feeding the whole thing to a(nother?) model anyway, to do that fuzzy heuristic judgment of what's important and what's a distraction. And simply removing messages doesn't add any structure, you still just have a sequence of whatever remains.
What I'm thinking is: When the agent wants to open more files or open more messages, eventually there will be no more context left. The agent is then essentially forced to hide some files and messages in order to be able to proceed. Any other commands are refused until the agent makes room in the context. Maybe the best models will be able to handle this responsibility. A bad model will just hide everything and then forgot what they were working on.
Three persistent Claude instances share AMQ with an additional Memory Index to query with an embedding model (that I'm literally upgrading to Voyage 4 nano as I type). It's working well so far, I have an instance Wren "alive" and functioning very well for 12 days going, swapping in-and-out of context from the MCP without relying on any of Anthropic's tools.
When a model is trained on multi-contexts, some growing over time like we see now (conversations), some rolling at various sizes (as in, always on), such as a clock, video feed, audio feed, data streams, tool calling, we no longer have to 'pollute' the main context with a bunch of repetitive data.
But this is going in the direction of 1agent=1mind. When much more likely human (and maybe all cognition) requires 'ghosts' and sub processes. It is much more likely an agent is more like a configurable building piece to a(n alien) mind.
> (...) writing a genuinely good harness with lots of context engineering and solid tool integration is in fact not that easy.
This. They are after the harness engineering experience of the Cursor people, I'd assume the they want to absorb all that into Grok's offerings.
The value and the room for innovation on the harness side seems to be underestimated.
Oddly the harness also affects model training, since even GLM/Z.ai for example train (I suspect) their model on the actual Claude Code harness. So the choises made by harness engineers affects the model. For Kimi/Moonshot and OpenAI the company makes their own harness. Alibaba uses Gemini.
I am pretty sure that a hole in the pocket in the order of 50 000 000 USD/month (assuming around 20 000 people using AI in not the smartest or most optimized way possible, therefore burning A LOT of tokens) will be noticeable by even the largest companies.
It is noticeable and even promoted, large companies do pay such sums for the API, like $5k+ per person per month. Not every eng is using AI that much already, but companies are clearly willing to pay those sums.
I asked Opus through claude code to set up the best local model fitting my hardware and that worked well for me. I could run Qwen 74B or something at .7 tok/s on my 64GB DDR5 on CPU. Pretty cool. Useful for overnight stuff. (this actually worked, it's actually usable for asking questions).
Is a community LLM possible? We'd have code to dynamically construct the pre-training dataset and use P2P mechanisms to share the acquired dataset. It would involve peer-crawling and other mechanisms to allow many people to contribute chunks to the dataset. Crawling chunks would be dynamically allocated to those contributing to avoid any double-crawling.
For post-training, the dataset would be a bunch of code that orchestrates the creation of training data via LLMs (needs to be legally sound), plus some kind of mechanical turk approach (something like wikipedia, where volunteers can work on chunks of data).
The main mechanism is this: what is shared is not just code, but also the acquired training data.
Critical aspects:
- to have a mechanism to peer-validate submissions to the data pool, so that everybody can donate data without the risk of vandalism
- a mechanism where the weights go through distributed training stages; somehow devs should be able to get a "lock" on the weights, do a bit of post training on it, and then get it approved. The "lock" means that during this brief period (trainining run), other devs are informed so we don't get two set of branched weights. A mechanism auto-evals the weights and accepts them as the new, updated weights. Retroactive discarding of weighs (e.g. after revising evals) is possible by branching the weights (needs some kind of efficient deduplication to avoid many copies of the weights).
I think this is possible. Maybe not with RAM, GPU and power shortages though.
Main benefit: Trannsparent training set means you know what the model was trained for. This makes it less opaque and less trial-and-error to see what modality the model is good at. This helps harness builders but also any other users of the models. It also decentralizes power.
I mean, the financial incentives are structured a bit different, but it's basically what you're describing, no? It's got projects for data collection, inference, training, etc. It's just that the dollar value of the compute contributed to say training is determined by the value of the token rather than as a straight dollar value. But even that is similar to just renting compute directly via fiat currencies given that every major provider of compute fluctuates it's cost based on supply/demand. Consider vast.ai or hetzner in which the cost to rent an h100 is not determined by anything but an auction system in which providers set prices and consumers agree to them.
My understanding is that bittensor is just the same market making where providers choose whether to provide and consumers choose to consume, it's just that you don't set your own price as the price is determined externally via the value of the tau. Which...tbh, fiat currencies fluctuate in buying power as well, if not quite so drastically. Just because the GPU is "still" $1/hr doesn't mean it actually cost as much as it used to given that the underlying value of the dollar changes just as the tau or yen or marc or eth or xrp or whatever does.
And thinking about it more, it's actually really quite similar to mturk in that via mturk you can purchase humans that do surveys, ocr, reviews, UX, etc...Via bittensor you can buy data gathering, training, inference, etc.
We're talking about completely different things. I'm talking about creating an LLM in the open, with individual contributors contributing to training sets as well as portions of the training work itself.
This is often a good strategy, but it's not easy at all. If your long term plan is to develop a product, focusing purely on delivery of services means no resources for product development.
If you offer services purely with off the shelf products, you then need to compete with other service providers why may be allocating all their resources to being a service business.
On the one hand you have to offer a competitibve service, while also working towards a product.
I always felt this moment would come eventually. The trend is centralisation of power and control. It's depressing. It's been a long time coming at a slow but consistent cadence.
The problem is the industry obsession on concatenating messages into a conversation stream. There is no reason to do it this way. Every time you run inference on the model, the client gets to compose the context in any way they want; there are more things than just concatenating prompts and LLM ouputs. (A drawback is caching won't help much if most of the context window is composed dynamically)
Coding CLIs as well as web chat works well because the agent can pull in information into the session at will (read a file, web search). The pain point is that if you're appending messages a stream, you're just slowly filling up the context.
The fix is to keep the message stream concept for informal communication with the prompter, but have an external, persistent message system that the agent can interact with (a bit like email). The agent can decide which messages they want to pull into the context, and which ones are no longer relevant.
The key is to give the agent not just the ability to pull things into context, but also remove from it. That gives you the eternal context needed for permanent, daemonized agents.
reply