Want smarter AI? Don't rush to change models

What frustrates me about AI is not knowing when I can rely on it.

It explains a complicated problem convincingly, then repeatedly gets a small change wrong. You’ve supplied the information, yet it overlooks it. It says the code is fixed; run it and the problem is still there.

The natural reaction is to blame the model and buy a more expensive one.

That might help. But I’d first check whether it received the right information, whether its working process makes sense, and how it decides its work is correct.

Leave those unresolved and a stronger model may stumble in exactly the same place.

The document is there. Why can’t it find it?

Connecting AI to a knowledge base is common now. Split documents into chunks, turn them into vectors, store them in a database, retrieve a few relevant passages for a question, then ask the model to answer. That’s retrieval-augmented generation, or RAG.

It sounds straightforward. The trouble often starts at “retrieve a few relevant passages.”

Ask about the specific error code TS-999, for instance, and the system returns generic troubleshooting articles. Related material, yes, but the answer is missing.

Vector search understands meaning; error codes, product models and API names also need exact matching. I’d consider combining semantic search with BM25 keyword search, merging and deduplicating the results, then using a reranker to select the useful passages.

There’s a subtler problem, too: splitting a document can strip away its meaning.

Suppose a retrieved passage says, “This version does not support bulk imports.”

Which product? Which version? An obsolete release or the one we’re using? Without that background, the model can easily apply the statement to the wrong thing.

Adding the title, version, section and necessary context to each passage helps. In Anthropic’s experiments, contextualized chunks combined with hybrid retrieval and reranking reduced the top-20 retrieval failure rate from 5.7% to 1.9%.

That doesn’t promise the same improvement for every knowledge base. What it does suggest is this: when an answer is wrong, check what information the AI actually received first.

If retrieval missed the right material, polishing the prompt won’t easily rescue the answer.

A longer conversation doesn’t necessarily mean it understands you better

Many people keep everything in one chat: project background, error logs, passing ideas and old plans. After dozens of turns, it feels as though the model should understand them better.

But the conversation may now contain conflicting instructions.

Last week the plan was A; yesterday it became B. An earlier direction was discussed and later abandoned. A person would need to sort that out, too.

The earlier Lost in the Middle study found that, for the models and tasks it tested, putting key information in the middle of long inputs often produced worse results than placing it at the beginning or end. That cannot simply be generalized to every current model. Still, accepting a long input and using all of it well are different abilities.

Context engineering sounds technical. In day-to-day work, a large part of it is deciding what the AI needs to see at this particular step.

For a website indexing problem, start with robots.txt, the sitemap, page responses and the webmaster dashboard. Read the relevant logs when they’re needed. There’s no reason to load the whole codebase and every previous conversation at the start.

Partway through a long task, leave a handoff: decisions made, files changed, checks passed and unresolved issues.

I’d rather it continue from an accurate progress note than guess our current goal from an enormous conversation history.

Compression shouldn’t mean removing everything, though. Lose a critical constraint and you’ll still have to redo the work.

Is paying for more thinking worthwhile?

Test-time compute studies how additional computation while answering can improve a model’s results.

It might try several solutions and select among them, or generate an answer and revise it using feedback.

Research shows that, on particular tasks and under particular compute budgets, allocating this computation well can let a smaller model outperform a much larger one. How you search and how you judge the candidates both matter.

The easy mistake is confusing more thinking with more talking.

An answer twice as long isn’t necessarily better. Generate five proposals and you still need a way to choose the reliable one.

Run tests for code, substitute a calculation back into the original problem, and compare factual claims with primary sources. That feedback gives the next attempt something useful to work with.

If what’s missing is current pricing or recently updated API documentation, look it up first. Reasoning cannot supply a fact the model never received.

I’d prioritize speed for organizing, classifying and extracting information. For difficult reasoning or important code changes, I’d increase the reasoning budget and prepare a way to check the result.

The spending worth making is the spending that reduces errors and rework.

Give AI real results to check

I’m interested in agent harnesses: the surrounding arrangements for splitting tasks, using tools, preserving progress and checking outcomes.

In March 2026, Anthropic described an experiment separating planning, generation and evaluation. The generator built an application; the evaluator interacted with running pages, checked APIs and data, then returned specific issues for revision.

One detail deserves attention: cost.

In one game-editor example, the solo agent ran for about 20 minutes and cost $9. The full harness ran for about six hours and cost $200. It produced a more complete result, but at a much higher cost.

I wouldn’t conclude that adding agents is automatically better value. The time and budgets weren’t remotely equal.

The useful lesson is to define completion, inspect the result, and bring back concrete failures to fix.

Publishing a blog post, for instance, isn’t proved by an upload endpoint returning success. Check that the page opens, the body is complete, the canonical URL is right and language links point to the correct versions.

Nor would I rely entirely on “take another look for problems.” Models can grade their own output generously. A second model doesn’t automatically become a strict reviewer, either.

Acceptance criteria should be specific and, ideally, verifiable through tests, actual page interactions or real data. Set retry and spending limits as well, so a minor issue doesn’t keep the system looping all night.

Does remembering more mean understanding you better?

I don’t want to explain my writing style every time, or repeatedly remind an assistant that WeChat articles should go into drafts only.

Those are lasting preferences worth remembering.

“Don’t send this one to WeChat yet,” however, applies to the current task. The system needs to understand scope, rather than cancel all future WeChat work.

Research such as Mem0 explores extracting important information from conversations, consolidating it and retrieving it later. The difficult part is that information changes and conflicts.

Alongside the content, I’d want a memory to carry its source, date, scope and whether a later decision superseded it.

Remember user preferences, but inspect live project state. Server ports, dependency versions and deployments should come from the current environment when possible, not a conversation from two weeks ago.

I’d reserve fine-tuning for relatively stable, recurring needs: fixed classifications, output formats and particular domain tasks. Frequently changing material belongs in retrieval; live state belongs in tool results. Each has a job.

I’d start with the tasks that failed

These techniques can all help. They don’t all need installing at once.

I’d collect twenty or thirty real tasks and record how the AI failed.

Can’t find the material? Improve retrieval. Forgets requirements late in a task? Organize context and handoffs. Has the facts but reasons badly? Consider a stronger model or more inference compute. Claims completion without finishing? Add verification.

Change one or two things at a time, then compare the same tasks: success rate, cost, waiting time and manual cleanup.

Keep some tasks out of the tuning process, too, to see whether improvements hold for unfamiliar questions. Otherwise, a better-looking prompt may simply be overfitted to a few examples.

My expectations for smarter AI are practical: something that needed five explanations now needs two; work that used to require a complete recheck now arrives with verification.

Model upgrades are worth following. There’s already plenty to improve while waiting for the next one.

References: