LLM-is-a-compiler is indeed a simplistic approach. I wrote a rebuttal to the yesterday's Cursor post, may reuse it here https://replicated.live/blog/follow-up
The idea that a 835-page spec "just exists" and we run an LLM to implement it is completely flawed. Specs do not appear out of nowhere, they co-evolve with the code. If you have the code, why do you want to generate it again?
Good software is made as a product of numerous feedback loops and LLMs let you operate those loops faster. They do not supplement the entire process though.
In the end, a good product is a barrel of distilled feedback.
Something I have always (even pre-LLMs) found funny to think about is - all code possible to run on a computer already exists. It's some permutation of all the bits of available memory. It's in there somewhere. So, suppose you want a specific program.
1. Some huge % of those possible programs are obviously not the one you want (most don't even compile).
2. Remaining programs might look similar to the one you want, but are buggy enough to be completely unusable, unreadable to the point of intractability, and so on.
3. Remaining programs look substantially correct but upon using for > a few mins you note major bugs that make it still not correct enough.
4. Remaining programs look substantially correct and seem to generally do your will but contain a long-tail showstopper subtle bug that corrupts saved data, or makes all the output subtly incorrect, etc.
5. Remaining programs might be useful, even if they're mildly annoying.
6. This process can probably continue for several thousand iterations until you finally find "it." The program you wanted. Or... one of them. There's probably still 10k+ candidate programs left at this stage.
Our job has always been to get to 5 and aspire to 6. Indeed most of software development is just doing 5->6 in a loop.
I think LLMs help us get to somewhere between 3 and 5 faster than we used to. And a big problem with them is that programs in 3 vs 5 all already look substantially correct and there's no way to know if you're getting 3 or 5.
Generally, the above is not how it _felt_ to write software pre-LLMs, it was just a cute way to imagine what you're doing. Now it's weirdly apropos.
I once saw a YouTube video where two beginners were learning the basics programming, and upon opening a cmd prompt, one of them said "It's insane if you just type the right words into this box, you can steal someone's 401K in Arkansas" (Or something along those lines). It's always stuck with me that technically that's pretty correct. It's probably not how you'd go about it, but with the correct magic words in the right order it would work
One of my professors used an example like this to encourage you to pause if you ever find yourself doing something like "maybe if I just flip this < to > the tests will pass". The space of incorrect programs (even ones that pass your tests) is enormous compared to the space of correct programs.
If you like thinking about this and haven't seen this before, check out the busy beaver problem[1].
I remember the Computer Recreations column in Scientific American on busy beavers back in 1984. I was too young then to really appreciate it. (I really miss that column—and the Amateur Scientist.)
I think the intent is clear. We could just enumerate all strings in size and lexicographical order. Then the task is only to filter, which strings are programs we desire.
It's the same sentiment as "the typing itself is not the issue, it's the order of the button presses that's difficult."
Of course, by Rice's theorem, all interesting questions are undecidable anyway...
You should check out Borges’, The Library of Babel.
Basically if you have an infinite string of characters in some language then you have every possible book, tweet, script, scribble, and scrap of everything ever written. You just have to find it.
Thats already a large universe to cover. Programs? Much bigger set.
I believe it's more that the library contains the finite set of all books which have a (unique) permutation of the language's characters to size N. The challenge to finding the cure to cancer in such a collection of sufficiently large N is indeed finding which book it is in.
> all code possible to run on a computer already exists
You have to be careful; that line of thinking might get you trapped in L-space. You’d need a very skilled librarian and possibly a bag of peanuts to get back out.
I think this is a great abstraction of the process I've never considered in those terms and I like it. It explains a bit of how software development differs from some other building processes where the desired outcome is being discovered as you figure out how to make something that produces that outcome
There are plenty of processes that have this property, though. For example: setting up a home, workspace, kitchen, factory, office, etc. to fit a particular purpose. As you discover what works well in a particular space, you will refactor your workflows, seating organization, storage systems, etc to fit what you're doing. And it may even have multiple valid configurations - for example seasonal variations like outdoor vs indoor areas.
> It's some permutation of all the bits of available memory. It's in there somewhere.
True, but the number of possible permutations is so large it's incomprehensible. Let's limit ourselves to say a 50 MiB program, meaning the size of the compiled binary. If I'm remembering the math correctly, that amount of memory can hold 2^(2^20 * 50 * 8) different states. That's 2^419430400 or roughly 10^126000000. For comparison, the heat death of the universe is predicted to occur in about 10^100 years, or 10^107 seconds. You'd need to be able to look at more than a million programs per second to examine them all before the end of the universe.
Reality is a fractal (Mandelbrot?) and this is where LLMs will help science. They will help us explore the fuzzy edges faster. LLMs probably won't find new "leaves", but they should be able to cover a lot of useful but "boring" fuzz.
In my experience, a spec is often public (or available for purchase by the public from a standards committee for a few hundred dollars, close enough in business terms), but the source code for your competitor's implementations is not available for any price. The first barrier to entry to become a competitor in many industries is to implement an 835 page (note that those are rookie numbers) spec, building software that is compatible with existing data formats or legal frameworks or network protocols or hardware or whatever.
This is another example of the many invisible silos that developers end up in:
1. Some are developing enterprise software for use by their own employer, the spec only exists because it's useful to help the dev team manage the project. In this case, it's totally reasonable to expect the spec to co-evolve with the code. If the release gets pushed out to Q3 or the target has to be adjusted due to unforseen implementation difficulties, so be it.
2. Some are developing contract software for their customers. The spec exists as a communication tool and requirements that allows the developers to know what to build and to inform the customer that what they've built has reached a certain phase of the billing process (including completion). You're on a clock, and if you're late or you don't do what you said you'd do then your customer isn't going to pay you.
3. Some are developing software for abstract customers - either B2B or B2C. Often, the specs here are regulatory or industry-standard, implementing the spec is the function of the business. You're releasing updates and new products to make year-end sales targets or snipe announcements by your competition, and advertising compatibility with various standards is a crucial bullet point in the sales presentation.
Bug reports, feature requests, and especially deadlines have different meanings in each context. The way AI development agents can be used in each silo also varies enormously!
As a counter argument for the compiler perspective: what Cursor did is basically transpiling a project to another language, which is a form of compilation.
If you distill the feedback into a spec (probably a combination of both docs and tests) which comprehensively covers the functional requirements, then you could treat the code as cached materialization. The value of generating code again is targeting different non-functional requirements, like ability to run in different environments or catering to different performance characteristics.
I agree with iteration, but that's an orthogonal argument to how effective it is at turning prose to code. You are asking, LLMs are capable of translating a 835-page spec, but where does the spec come from? Follow the stream up from the built software product:
Binary < IR < Code < Spec < Prompt (aka goals)
For LLMs to faithfully fulfill the role of translating spec/prompt to code, it needs enough information about the problem/motivation and the constraints around it. Often, that's done by research or by importing proven modules. The human role left in this compilation chain is a precise-enough expression of the motivation, upstream of the built artifacts.
Edit
For example, SWEbench Pro was supposed to be a measure for how good LLMs are at compiling prose into code. At least 15% of those tasks were underdetermined by ambiguous prose in the spec. Of the issues that are specific enough, Claude and Codex indeed do compile to functional code with self-written tests. Source: https://github.com/kimjune01/swebench-pro-audit
Indeed, this is a much more realistic and useful framing of what to do with the current capabilities of LLMs. They work so fast, at such a high level, that you can speedrun the co-evolution process and end up with a spec and code that works. You can actually use second-system (fourth? fifth? I lost track) syndrome to your advantage because it’s so cheap to throw things away. And they don’t just write the system, they also write the test harness, the benchmarks, the failure simulation, the statistics…all the stuff that seems like “overhead” but lets you drive that evolution with data.
I’ve kind of drifted into this mode a bit at a time over the last year, but hadn’t stepped back to make such a coherent explanation of it.
The source code always was the result of that distillation, or at least a crucial part of the result. I’m unconvinced that natural-language specs can replace that role. Even long before LLMs, when writing specs I often resort to some form of formal language to precisely nail down certain parts of the specification. And some parts are always left open, where I know I’ll get to them when writing the code, because they are more effectively solved “in the code” than waterfall-like up front.
A zen master asked a junior engineer to go get him a rock. The junior engineer asked what kind of rock he should get and the master replied by knocking him in the head with his cane. The junior thought for a minute, then asked, "would you like a round rock or a flat rock?" Again, the zen master rapped him on the head. The junior thought for a moment, then went to the creek and fetched a rock at random. He brought it to the zen master who inspected it and then said, "No, I want a flat piece of flint for starting a fire." The junior engineer was then enlightened.
Stakeholders will only give you requirements after they see the prototype. Thus, the spec and the prototype to elicit it are two sides to the same coin. Once the spec if correct and the prototype is starting to show its age, throw away the prototype and implement the spec with a fresh start.
> Stakeholders will only give you requirements after they see the prototype.
That’s very much incorrect and sounds like someone who love building more than communicating. Even without a prototype, users and other consumers will be able to explain their problems. It may not use the same metaphor map that you’re used to. But it’s very much a description of the problem space.
Once you design a prototype, what you will get is feedback about the solution you’ve designed, not the raw problem. If you’ve not listened well in the first place you may be well off the mark, and have to work harder to correct things.
Even if your story, the junior would do way less work by asking the master what he intends to do with the piece of work instead of focusing on the rock itself.
whether the hypothesis originates at the users or in your guess for the users is irrelevant in that they're both hypotheses with no certain outcome until trial.
They are. But engineering is about minimizing costs in solving a problem while ensuring requirements are met. Not worrying about costs is not engineering. It's either a research project (where you want to know if something is possible) or playing around.
The cost of prototyping has dropped so much due to LLMs and coding agents that it basically always makes sense to prototype something to elicit feedback, just as part of the requirements gathering process. E.g. if you can spend $50 to understand something that will save you two weeks of effort, wouldn't you do that? That's the sort of dynamic I'm talking about.
The cost of prototyping has always been very low. I can quickly sketch a UX flow using balsamiq. Or copy-paste paste code from docs and examples to show a rough working model with. If you can extract the essence of the problem you can quickly realize cheap ways to demonstrate a solution, and not spend two weeks trying to code it.
If there’s something that is wrong with using LLMs to prototype, it is that those prototypes are always too complex. Instead of decomposing a problem and testing unrelated concerns apart from each other, it’s often a mix of everything that the LLM user had been thinking about.
Often there’s no rapid iteration where you are comparing several approaches. Instead it became tunnel vision and confirmation bias where if it’s working, that means it’s the solution.
Well, since we started training them on RLHF-style rating of the last generation they've totally collapsed back onto predictable suffix generators, which is why there are desperate, low-AUC BERT-inspired clasufiers bolted to them now.
They have like, Chomsky grammars now. And you can't fix it, it's structural to the process. You have to rewind almost to the pre-train.
This is a company selling picks and shovels for the AI bubble. Of course they're going to drip feed "AI is so awesome guys!" stories on their blog.
If you strip away the hyperbolic statement about compilers and the Empire State Building, it's an internal success anecdote.
Would be interested to hear how all of the self-directed AI decisions worked out after launch. This is where I run into problems with AI. Sometimes it makes product decisions that aren't immediately visible and make no sense and cause serious problems down the road. Also, an analysis of how much time was actually saved by using AI? since it sounds like there was a lot of guidance and testing to make sure what the AI did was correct.
> 2) Isn't it great that we can burn a bunch of dinosaur blood to convince ourselves that we don't need other humans?
God I hate this cynical take. Yes human beings use energy. We should look to generate more energy as that's a sign of wealth. The "energy bad" take is so lazy and ultimately anti-human. And in this case its especially obnoxious because what does "we don't need other humans" even mean.
AI exists at the boundary of codification. When you understand something well enough to express it in a deterministic way, it is no longer worthwhile to keep asking AI for that thing over and over - you should codify it.
But wait, I hear you say? AI has knowledge! And we can rely on that knowledge and sprinkle on top some brief instructions and it'll get things right and work out the details itself!
That is spec driven development.
And this is great one single time. Except, knowledge shifts. You are outsourcing part of what you've codified to society as a whole, in the form of language model knowledge. You're also trusting current and future models to always produce something that meets the requirements, and never be quantized under you, and produce the same result next time despite being a probabilistic system with no memory, and and and.
Better to freeze everything in place in your codebase so you get predictable results - in other words, codify it! Let's store the desired state of that button in code in a very complicated thing that's existed for decades called a string, so it always says "Finish" and never says "Done" - even if we regenerate the app.
That's pretty unexciting and doesn't justify tens of billions of investment. Which means you won't hear it from anyone with tokens to sell. But it's what you have to do to make something people want.
I've heard something along the lines of "Claude is like a compiler: source code is the new object code, you don't look at that anymore" many times.
And I don't really think this is true. Compilers are usually deterministic, and whilst we can find edge cases, it's nothing like an AI agent writing all the code for you.
I think you have two choices, given the Claude is a code generator and not a compiler: (a) you review most or all of code to make sure it makes sense, or (b) you trust but verify via a strong test suite, potentially also created by Claude.
The problem with (a) is that you lose a lot of the speed-up. The problem with (b) is that you have no human oversight and the code may be incomplete, badly designed, or plain wrong.
Currently we review all code because correctness is extremely important to what we do, but that comes at a cost.
I don't know what the answer is here, in general. Does trust build over time? Do the models just get so good we can trust them to make zero mistakes?
Part of the value of LLMs and humans is nondeterminism. Pair nondeterministic output with strictly verified results (proper tests) and you can create a useful working system.
Humans can't build a system perfectly, and agents definitely can't. And agents (like humans) will build a different system every time even with the same prompt. Even if it's just trivial differences like array vs linked-list, there are still differences.
The value in having executable code is that it won't change its behavior unless you deliberately modify it. I don't think there are any shortcuts. Models will get smarter and smarter and make fewer mistakes, but you'll still want to produce real source code to execute because it's consistent (you can sell it as a product, set it and forget it, etc.), and most importantly:
running real code is orders of magnitude faster than having an AI either run the process "manually", or have it write the code out multiple times.
If you have a product intended for humans, you're gonna need humans to do acceptance testing, just like now. All the software engineering principles you learned (did you learn them? now's as good a time as any!) are still just as applicable now as they were 10 years ago.
I really like what exe.dev is doing. And their philosophy of how cloud services could work is amazing. I have a monthly subscription and it's really cool to just vibe code a little website with Shelley and have it running in minutes. I feel like I have so much fun using their service, and spend so much less time thinking about what things will cost, or what cloud service to use. It's just so much simpler and the friction is so much less compared to other cloud services.
Agreed. Big love for exe.dev. For quick low stakes projects, using their Shelly assistant is the simplest way to give an LLM a fully persistent VM with web services I've found.
Specs might become one solution for coping with the need to review increased volume of code. A spec is a higher level of abstraction than code, which is a higher level of abstraction than machine code. The industry made the transition to higher-level once, paradigm is changing so it might happen again.
The workflow I imagine is either deriving specs from the conversation or reverse engineering the code to spec, review and edit the spec which should be tighter and much more compressed, then deterministically compile to code. Of course we don't want to be spec-first only, that would be going back waterfall, but doing iterations back and forth.
Now, Claude is not a compiler because it is closed and non-deterministic (they do opaque processing on server, hiding reasoning tokens), but LLMs might be. We refer to a piece of code from npm/pip by name to get some code by downloading it. We then have lockfiles with hashes to ensure integrity. Currently we are vibing it, but in the future we could refer to a piece of code by prompt/spec and getting the code by inferring it. To ensure integrity, the lockfile would be hashes of open weights and inference code (and ironing out implementation details like non-determinism due to GPU scheduling, etc.).
It's funny how you describe something very close to what I was attempting to design. It was meant to trickle down from tickets to deterministic code. Data wise, it should be like a pyramid of tickets diluting layer by layer into leaf nodes which were "implementable" as statements in code. I'm not sure if that description makes sense read by someone else.
I think what made sense was envisioning a nanoswarm of LLMs (anticipating ASIC performance) diluting specs into semantic logic nodes, but a conversion from these to deterministic code made the vibe coded experiment come to halt. Your lock approach could be a shortcut to that.
Totally off-topic: it's quite intriguing that you can feel once friction starts building during a design phase. Suddenly everything slows down. I wonder if it's quantifiable and therefore can identify "wrong" design choices made by either humans or LLMs.
(Hoping some claw bot pics this up to finish my idea on my github tix repo in the initial-design branch, as main is empty ~ MPL2)
This resonates, I can imagine the development process as a cascade: we get business goals and user feedback, product is defined which can be pretty fuzzy, designers define UI, engineering works data models and tech constraints. Inputs at the top can be fuzzy, but each node in the graph applies some intelligence and clarifies it.
Related to the friction you mention, since the flow graph of the context is not preserved when it gets transformed into code, I find LLMs getting confused and making worse changes, because the explicit decisions with good reasons are getting mixed with implicit decisions from LLM surfaced during designing/planning. Sometimes when things change the code should change radically to match the new reality, but LLMs often "play it safe" by accumulating code and adding to tech debt. With a flow graph when change is made in the upper levels, it could propagate and allow decisions in lower levels to be revisited.
This might work if we have completely new harnesses for this, because the current one’s intentionally introduce non-reproducible context and tools.
The current way we’re doing SDD is extremely pointless. You reach a decision point that is under specified, make a decision, update the spec and continue the implementation. There’s no guarantee that the implementation followed from the updated spec, and not from the decision.
I agree, another issue I have with existing SDD frameworks I've seen like spec-kit or openspec, is they seem to miss the mark by coupling specification (what the software should be) and execution (how to get there). If the description of execution is needed, it means the spec is weak. As a rule of thumb if a framework defines execution concepts like tasks or tickets, that is a red flag for me.
prompts are not specs because most often, prompts are underspecified. You can prompt fizzbuzz well enough for it to be functional, but not ffmpeg. So the distinction is less about Claude's capability, more about the quality of the input. Garbage in garbage out
I think something can be considered a spec to some degree if outcome was derived from it.
It might underspecify, but that is just a quality of the spec. And as long as there is any discretion at implementation time, then the spec necessarily underspecifies to some degree. The goal of a spec is often to be clear on key directional concerns. Consider RFCs.
There's a trade-off between specification and implementation discretion. But if you think about starting at one end ("an internet forum") and then incrementally expanding the specifications ("it has topics with many posts in a linear paginated list"), there's some point where everything you care about is in the spec, and the rest is left to implementor discretion.
That's what I'd consider a good spec and a good LLM prompt.
and LLMs are really good at filling in the blanks. It's like a really big import on generic application knowledge, and if you draw the circles well enough, it will draw the rest of the owl.
You can reference other specs in a prompt. Video codecs are well specified with hundreds of pages of specs, they are probably in some form in the weights already, but you could also reference them and hash them as inputs directly to create a functional alternative to ffmpeg.
i meant to make the distinction between 'prompt' and 'spec' in that prompts are something that a human types in by hand. Interesting programs are a combination of canon and novel input
Gotcha, yeah if a prompt does not produce the result you want, you need to add more details to it, depending on the scope it can be a lot. When it includes all sufficient details it becomes a spec.
> A spec is a higher level of abstraction than code, which is a higher level of abstraction than machine code
No it’s not.
Just like a quick doodle is not an higher level representation of the Mona Lisa. Sure for someones that knows the latter, it can suggest it. Or for someone that doesn’t know it, it may provide some basis of conversation. But it’s not the real thing. You can’t provide a doodle of something and expect an artist that hasn’t seen it to paint it.
Doodling’s value is that it lets you iterate on ideas without the accidental constraints that comes with the implementation (choosing paints, finding references materials, deciding colors,…). Some necessary choices are kept for later, while you decide on the most important ones.
So a spec is useful in designing software, but it’s severely lacking in implementing it. And the decisions made in implementation are as important as the ones made in the design. Even more after a while in production.
> Claude wasn’t just a compiler here. I never handed off a task and let an agent make a bunch of decisions in order to reduce it to practice.
> I’d say that, in all the ways that matter, I understand the code.
I think the dissonance here is really important, and not a bad thing at all. A lot of the decision _were_ handed off the the AI, but they weren't the decisions the author cared about. This is a big selling point of AI! If something is doable with a computer, it’ll figure it out. 30 minutes and 200m tokens later, it’ll take any idea and declare “the feature is fully implemented.”
The hard part is figuring out where to inject that friction, so you can see where it's making decisions for you that matter. The author approached this by incrementally building the thing, reviewing and poking and prodding at every step. A week of attention following a bunch of design discussions is fast, but that's still not trivially cheap.
I want to see us talk more about the decision exhaust of agents, because the better the models get, the more decisions we'll want them to make.
> I read a vanishingly small amount of the actual code.
This sentiment kind of saddens me. I'm all for burning tokens to write throwaway code just to prototype a solution, but I don't get not reading (or at least familiarizing yourself with) the code that you will deploy to prod.
> But our VMs start fast, so fast that even if we created the DNS entries before creating the VM, our users still had to sit around waiting for DNS to propagate, which occasionally took minutes, not seconds.
To my (limited) understanding, this is not a good idea, and is an unfixable problem from the server side. Companies, VPNs or ISPs or routers, often use their own DNS servers, and those can have caching logic, which means it doesn't matter how fast your own DNS implementation is, as the users lookup request wont hit your DNS server, it'll hit an intermediate cache.
Presumably these are new subdomains, if a caching DNS resolvers never saw the domain yet, it would simply query the authoritative server.
There's a small nuance here that DNS spec also caches negative (NXDOMAIN) results, but the TTL of those is controllable by the SOA record on the domain that contains it.
exe.xyz has a fairly short 10 second negative TTL, so even if you tried to resolve it before it was set up in the authoritative DNS, it wouldn't stay negative-cached very long (though some DNS caches put a lower bound on the TTL and cache it for longer)
Usually when you create the VM, it gets its own DNS subdomain that looks like `my-new-vm-name.exe.xyz`. (I'm not affiliated with exe.dev, I just use their product). So I doubt there would be any cached requests for that particular subdomain before the VM is created.
NXDOMAIN results and other errors have their own TTL value though, so if someone queries too early in the process you could be waiting a little while for their cache to check again. The value for this is set in the SOA record for your DNS zone.
For example, example.org/example.com seems to have TTLs set at 300s (5 minutes) for A records, but the negative caching value is 1800s (30 minutes). If your VM domain is set the same way, you could see a premature lookup causing issues for 30 minutes instead of just 5. Even the 5 minutes could be a pain if your VM spin-up process expects to be able to lookup via the name very early and will fall over if it can't.
I always don't know enough about DNS. This is great stuff.
If exe.dev is running their own DNS zone, then wouldn't they be setting the expiry value in the SOA record themselves? So they could set it to something short, and secondary servers are supposed to honor it, right?
> wouldn't they be setting the expiry value in the SOA record themselves?
If they thought about it. But it is something that even people relatively experienced in DNS matters forget about until it bites them and they've spent ages diagnosing the initially odd looking problems caused.
> To my (limited) understanding, this is not a good idea, and is an unfixable problem from the server side. Companies, VPNs or ISPs or routers, often use their own DNS servers, and those can have caching logic, which means it doesn't matter how fast your own DNS implementation is, as the users lookup request wont hit your DNS server, it'll hit an intermediate cache.
All caches should respect the TTL of DNS records, no matter what they are, though there are some interesting bugs out there.
Firstly the human bugs: a lot of people assume switching DNS is always fast, but they haven't had the TTLs property of a DNS entry explained to them so assume that if the change is working here then it is going to be working <there>, <there>, and <everywhere-else> too. The default TTL values are a mix, it used to be that 24 hours was common though these days four hours or less, sometimes even one hour, is what you are more likely to see even for defaults. Whatever value your entries have though, that doesn't mean a fixed × hour window that it will switch at the end of. If the TTL is four hours your local cache last checked three hours ago but mine last checked one hour ago, a change will take an hour for you to see and three for me to see. A cache that has not recently looked up the entry will see it immediately next time it is asked - this is why some assume it is always instant (they add a new sub-domain record and it works instantly because no one has ever requested an address for that name before, for instance, and assume this is how it will always work).
Then there are coded bugs. TBH these days I ignore those: they are rare enough that if you are caring about that sort of edge case a lot then there are a lot of other weeds you'll be worrying about too and you'll never get anything done. For my domains I mostly have everything set to five minutes (300 seconds) as they are rarely referenced and DNS infrastructure is not a high-power service these days. Some would suggest that using such a low default is unfriendly to DNS caches and the root servers, but my names are very rarely referenced outside my network/vpn where my own DNS authoritative servers are what get directly queried anyway so that isn't going to matter. example.{org|com} and some sizable companies use 300s too. Why five minutes and not less? There used to be a very common caching DNS server that would ignore anything lower than 300 seconds and apply its own default (14,400 seconds, four hours, IIRC) instead. This goes back a couple of decades though, if anyone is still running that they deserve some DNS lookup failures! If you are wanting to use longer values normally, just remember to lower your TTL a time before any planned change so the change propagates ASAP and raise them back up after. One extra thing to be aware of with short TTLs is an outage affecting all your authoritative DNS servers may cause less friendly errors in many cases than getting an out of date address would result in, so make sure your DNS servers are stable and have few (preferably zero) shared potential points of failure.
A compiler is an algorithm and Claude isn't*. A compiler almost never produces a wrong output, even when compiling an extremely complicated program. But a compiler must be clearly defined and is limited to input/commands it's defined for. A compiler will "correctly" process input into unintuitive output, but that's not always what the user intended (e.g. omitting large sections of code that are undefined behavior). Claude makes mistakes, but can process any input/command expressible in training data, and has more potential to infer when something is unintentional.
Tools like Claude are great at building algorithms, then tests to verify them. The algorithms they build aren't just compilers, but any software that can be verified. Even though Claude may not be a compiler, it has the potential to (at least if it improves) create production-grade software that can be verified, like other compilers. But maybe not good UX without human input (expanded to any subjective experience, like video games).
* Not a "source language to target language" deductive algorithm. Technically Claude is an algorithm to predict the next token, but acting as a compiler or anything else it's an inductive heuristic, because it guesses (https://stackoverflow.com/q/2334225)
> A compiler will "correctly" process input into unintuitive output, but that's not always what the user intended
Importantly, this behavior is deterministic. If the compiler takes your code and spits out something you did not expect, it will always spit out that same result, given the same input. Once you figure out what went wrong, you've solved that problem.
Generative AI is intentionally not deterministic, and that means you're going to solve the same issues -- hallucinations, primarily -- over and over again.
Determinism isn't the main hurdle. It would be a moderate effort to engineer a deterministic claude that would be no more useful than the current one.
The key characteristic of a compiler is that it takes a source language with a well-defined meaning to a target language with a well-defined meaning.
The reason claude fails to be a compiler is that its input language (english) can't be precisely defined. (And even if that were possible, it would very often fail to preserve the meaning when "compiling")
AI could be deterministic, only it requires sacrificing performance and nobody cares enough. Although algorithms are more predictable, and you can follow an algorithm's steps (even a complex algorithm can be broken into large sequences of steps, which may be broken further).
Determinism is the wrong thing to worry about. As you say an LLM could theoretically be deterministic.
Chaos (prompt instability) is the problem that isn’t solvable if you want to treat an LLM like a compiler by doing something like checking in the prompts and rebuilding the code each time.
In general if you change a single line of source code, a compiler will not produce a completely different program. If you change a single word in a prompt, you will get output that likely doesn’t match a single line of the original output.
Chaos is what leads to unpredictability, and there’s no solution to this on the horizon.
I think you're correct, I'll just point out that compilers are also chaotic today and it is often accepted. A small change to source program can lead to big change in performance. It's especially apparent in high-level and declarative languages - SQL or anything with GC.
“Generative AI is intentionally not deterministic, and that means you're going to solve the same issues -- hallucinations, primarily -- over and over again.”
I see it differently. Sure there are still some hallucinations (but less and less as we progress on), but I see the AI-is-not-deterministic characteristic as a good thing, in the same way that humans and human designers not being deterministic is a good thing. Imagine giving a project/task to a human, and it was completed exactly in the same way with exactly the same outputs as another human that the project was tasked to. Would that be better than each human having varying outcomes of the project? I think most would agree the answer is no.
I agree with what you're putting out there. Every time I see one of these LLM/compiler pieces, the only thing I can think of is that these people don't know what a compiler actually is.
In a production-grade compiler like LLVM they are rare enough that it compiles giant projects like Linux and Chromium without detectable issues.
Maybe eventually Claude can, since it can write tests, but compilers / (deductive) algorithms have other advantages like efficiency and predictability.
Not really. As the C language was created around 1972 and by 1973, Unix was rewritten in it. By 1975, Unix was released commercially. That’s like 3 years between design and commercial usage.
"Algorithm" is not a word with a definition, so I can't say that you're wrong, but I struggle to see how you conceptualize "algorithm" in a way that does include a compiler but does not include a language model.
I think the distinction you're trying to draw is better captured by comparing inductive and deductive program synthesis.
A compiler is a deterministic algorithm. Clause is not.
Of course Claude is an algorithm, but it's a very vague, fuzzy and complex one, where you have no way to determine what's going on inside, and no way to know up front what's going to come out.
Rogers "Theory of Recursive Functions and Effective Computability", page 1, emphasis is the author's:
§ 1.1 The informal notion of algorithm
In this chapter we give a formal (i.e., mathematically exact) characterization of recursive function. The concept is basic for the remainder of the book. It is one way of making precise the informal mathematical notion of function computable "by algorithm" or "by effective procedure". In this section, as a preliminary to the formal characterization, we discuss certain aspects of the informal notions of algorithm and function computable by algorithm as they occur in mathematics.
I don't understand why you people act like you're stumped by literally, literally page 1 of computer science.
> I don't understand why you people act like you're stumped by literally, literally page 1 of computer science.
… and yet you didn't stop for a moment to consider that in a field as fast-moving as computer science, a concept which might not have had a formal definition in 1967 acquired one in the past 59 years? See, for example, Sipser's Introduction to the Theory of Computation, which has an entire section (3.3 in my 1997 print) titled “The Definiton of Algorithm”.
Yes, did you read that section? It's using different terminology but it's saying exactly what I'm saying (of course it does, it's elementary computer science).
There is no accepted definition of the word "algorithm": there wasn't one in 1967, there wasn't one in 1997 and there isn't one in 2026. There are models of computations we can prove to be equivalent to each other, but no commonly accepted definition that captures the way we use the word in normal speech.
oh wow. this is a rarity on an internet dominated by toxic rage-bait. apology accepted. I think you probably have interesting things to say on the matter and I look forward to reading them on a better day.
respect.
That’s technically correct, but it still means that someone has to write the formal specs that Claude’s output is verified against, and someone has to design the formal languages the specs are written in. And once you have formal specs that truly covers all aspects that you ever will care about, it’s unclear if you couldn’t instead build a non-LLM mechanism that efficiently and deterministically spits out an implementation of such a spec.
I expect in the future it will become hybrid. Formal specs provide guardrails for LLMs / AI but within those guards LLMs will author traditional compiler programs and at times drop-down to manually compile / prove certain snippets.
Could we describe the first humans manually writing Assembly and their own higher level abstractions as "compilers"? Loosely I think we could, and LLMs fit the same mold.
I disagree in the sense that source code (compiler input) leaves little space as to the behavior of the resulting program. With LLMs, either there is much more implied leeway, in which case I’d argue that it’s different in nature. Or the specs will restrict the LLM output to a similar extent as a conventional programming language restricts a compiler’s freedoms, in which case it’s not clear that LLMs are the most expedient technology for generating that output.
Levels of abstraction actually matter with regard to how much observable behavior they leave unspecified, and with regard to how much control is needed over each respective behavioral aspect.
And claude isnt a replacement for the chain of humans, you dont know what you dont know, thats why we have a person in a role to gatekeep each step.
Anyone using claude can see it misses steps that an engineer wouldnt, it can make bad choices that a sysadmin wouldnt, it can pick the wrong order a project manager wouldnt, it can pick wrong cost models that a bean counter wouldnt, etc etc etc
But, I'm pretending it is and living with those mistakes.
Claude is more like a Probabilistic Turing Machine. [0] It's correctness isn't deterministic but rather a distribution. It is predictable.
There are lots of places in computer science that determinism isn't necessarily the best.
- UDP video calls let packets vanish or arrive corrupted, because waiting for retransmissions would freeze the picture.
- Stochastic gradient descent picks a random mini-batch of training data and treats it as the whole dataset, because computing the true gradient on every step would make training infeasible.
- Speculative execution in modern CPUs guesses which branch a program will take and rolls back when wrong, because waiting for the real answer would leave half the silicon idle.
What we want is something like a Las Vegas algorithm, a probabilistic machine with a cheap verifier.
Determinism isn’t important, but there are usually threshold of acceptable performance in all off those cases. Machine Learning was all about improving accuracy even when you know that there’s probably an error somewhere. But the current LLM hypers refuses to acknowledge that the answer to a prompt may be wrong (even though the providers have a warning line about it).
Instead of assessing the risk and provide corrective methods, all they do is pushing to use it more and more everywhere.
> I’d say that, in all the ways that matter, I understand the code. Sure, if I had to hand-edit it now, there’d be a serious learning curve. But I won't have to.
Famous last words, but point is taken.
The non deterministic nature of an llm breaks the metaphor that they are like a compiler.
However, it’s not foreign to compilers to receive feedback from the running program (PGOs), so there are still parallels to the feedback we provide LLMs that guide their “optimization”.
I think even calling LLMs a non-deterministic compiler isn’t accurate either so ultimately I agree the metaphor doesn’t quite work.
I do think LLMs are like compilers in terms of how they changed how we build programs from a historical context. But that’s about it.
The non-determinism is theoretical solvable. Chaos (prompt instability) isn’t. When you change a single word in the input, you get a completely different output. Because of this LLMs will never be compilers.
No, it isn't. It can autocomplete a fragment of a sketch of an idea into something that may or may not match the original vision and almost certainly has serious bugs which only manifest over time because they elude a cursory test.
Because Claude isn't a human the perception of it is different.
Humans can fly planes, computers too can fly planes. I'm not going to give the same leeway to a 'computer' flying a plane that I would to a human. Because we understand that humans are just that, humans. The demands placed on a computer are so very different.
Computers are already driving cars autonomously, possibly better than the average human already, and quite plausibly will eventually be better than the best human driver.
Precisely my point. Say for examples planes, trains or automobiles, humans may make way more errors than a computer does but you can bet your bottom dollar the threshold for allowing computers to autonomously control the above is way higher. Because the expectation is that if it is a computer it ought do it far better/safer.
There is a lot of rehashing of LLM arguments in the comments here, but I'd love to read an actual distributed systems expert's take on the DNS architecture the author and/or Claude ended up with. One of my personal fears about LLM coding is that some of those implicitly outsourced decisions turn out to matter more than expected. Does the "timeline" stamp make sense? (I know there's not much detail to work from.)
> exe.dev VMs have nice domain names: vm-name.exe.xyz. When we start a new VM, we add a CNAME entry or three. Easy, right?
> But our VMs start fast, so fast that even if we created the DNS entries before creating the VM, our users still had to sit around waiting for DNS to propagate, which occasionally took minutes, not seconds.
> We did the obvious thing: We wrote our own DNS server, so that DNS always immediately matched the source of truth. And life was good.
> But latency matters, so we added regions. And just like that, DNS became the long pole again, because all DNS was served out of Oregon. Also, deployments caused tiny DNS outages. To fix this, all we needed now was a geographically distributed but fully consistent DNS server.
> We did what a sensible engineer does when faced with a hard problem: cheat. We vibe-engineered a distributed DNS server tuned to our specific needs.
I'm sure they have already considered this, but what problem does this solve that couldn't be solved by using dnsmasq, unbound or something like that? Why reinvent the wheel?
The question is more what this really solves. They said they built all this because DNS propagation was too slow. But secondary dns servers by e.g. your isp will still have old records when you change something. So what did they earn? Every is correct within their network pretty fast. But for a normal user there isnt any difference because 8.8.8.8 or 1.1.1.1 still cache old records?
This seems to be grasping for analogies to make sense of the work this person was doing.
A compiler does a lot more than source code translation.
There are specifications that tell us the de jure specifications of the language, if there is one, and then we have to recognize the de facto implementations of said language. The often disagree and leave much on the table. Some times on purpose, such as implementation details, and other times by omission.
Users of this compiler expect a deterministic compilation of the source text into the target code but it’s rarely 1:1. There are optimization passes, inlining, barriers, etc.
And then there are the run-time effects of executing a program!
I think the analogy gets a little weak because natural language is not a precise enough language to specify discrete systems.
What it sounds like the author is doing is bypassing decision points with other people and delaying making decisions themselves until the LLM agent forces them to? Which is a fine approach but I don’t think the analogy with a compiler is necessary.
People seem to have a hard enough time understanding branch prediction and thread barriers.
I still think it is (in the source-code generating mode) a compiler, just like LiquidHaskell is. (I haven't used LH but it essentially can automatically supply functions based on the type conditions you specify.)
In my view, reasoning LLMs have learned a large number of close to true sentences of "logic" in natural language. These rules are not consistent (unlike typing rules of LH that allow creation of provably correct programs), but they work in practice better than LH. Essentially, they represent an unsound formal specification of natural language, together with many useful (almost)tautologies that helps solving constraint problems (just like LH does, but again unlike LLM, provably correctly).
LLMs proved that human language can be formalized reasonably close to soundness. I believe we can have a sound formalization of human language, and that kind of formal language will be once superior to LLMs.
It will also not be as complicated as LLMs and require only fraction of compute to use.
The entire "Don't Read the Code" argument is just insane to me and ironically when you see who the people are that are advocating it, it's often the same people that talked about microservices, the latest java script framework, no-code or any other methodology that was in vogue at the time just to avoid focusing on solving the real problems; The illusion that they are just one "smart step" away from achieving a 10x productivity boost that will make everything else trivial...
> By the time I was ready to build a keeper, I had accumulated a scar-tissue document that was empirically sufficient to guide an agent through most of the important decisions, at every layer, ranging from high level goals through architecture down to the occasional low level detail, such as the exact shape of the data type for load-bearing concurrent caches.
Why inefficient? If Claude continuously makes the system more efficient based on demand, there's no reason for the system to be less efficient than if it was managed by humans.
Claude doesn't even make machine code though. Or it can if you really want to, but that's now how people use it. If you wanna include transpilers or interpreters, fine but the answer is still no, otherwise our git commits would just be prompts that get built either in CI or JIT. People deliver code, not prompts.
Maybe that could change some day, but even then, it wouldn't be a useful designation. It's like saying "oh technically the Americas are an island." There's a reason we still distinguish between a transpiler and compiler.
So I made a distributed service for your distributed name service so you can distribute your names to the service that distributes them on the onternet
I'm kind of getting to the point where if I know something is vibe coded, I just won't use it. It's not an anti-AI thing, it's just a quality thing. Pretty much every piece of vibe coded software I've used has been bad in some regard. The worst ones are the ones that aren't obviously bad but rather subtly bad in dangerous ways (the article about OpenCode yesterday definitely made me nope out of using that)
> I’d say that, in all the ways that matter, I understand the code. Sure, if I had to hand-edit it now, there’d be a serious learning curve. But I won't have to. And more importantly, I can reason about the system, share perspectives with my colleagues, and guide agents on future work.
If there's a serious learning curve to editing code then you don't understand the code. We used to call that 'on-boarding' when you brought a new engineer on the team as they got up to speed as to how the codebase worked.
Didn't read the article because the conclusion is obvoius enough to me as is, never heard of Erik Schluntz before, and drawing an analogy between two things means very little anyway. "this bike is about as heavy as that table" doesn't require the rebuttal that bikes aren't tables, for example.
Spec-driven development is sold on the idea that you regenerate modules or entire services when you change the spec. It's a really dumb idea but was briefly popular in enterprise, probably still is in some companies.
A program could synthesize a program that adheres to the specs. But that is not what we understand by a compiler that generally has to preserve operational semantics.
I've heard the "if you hate vibe coding you must also hate compilers" strawman so many times, I'm sure it's been said by someone with a "name and reputation" at least once.
LLM-is-a-compiler is indeed a simplistic approach. I wrote a rebuttal to the yesterday's Cursor post, may reuse it here https://replicated.live/blog/follow-up
The idea that a 835-page spec "just exists" and we run an LLM to implement it is completely flawed. Specs do not appear out of nowhere, they co-evolve with the code. If you have the code, why do you want to generate it again?
Good software is made as a product of numerous feedback loops and LLMs let you operate those loops faster. They do not supplement the entire process though.
In the end, a good product is a barrel of distilled feedback.
Something I have always (even pre-LLMs) found funny to think about is - all code possible to run on a computer already exists. It's some permutation of all the bits of available memory. It's in there somewhere. So, suppose you want a specific program.
1. Some huge % of those possible programs are obviously not the one you want (most don't even compile).
2. Remaining programs might look similar to the one you want, but are buggy enough to be completely unusable, unreadable to the point of intractability, and so on.
3. Remaining programs look substantially correct but upon using for > a few mins you note major bugs that make it still not correct enough.
4. Remaining programs look substantially correct and seem to generally do your will but contain a long-tail showstopper subtle bug that corrupts saved data, or makes all the output subtly incorrect, etc.
5. Remaining programs might be useful, even if they're mildly annoying.
6. This process can probably continue for several thousand iterations until you finally find "it." The program you wanted. Or... one of them. There's probably still 10k+ candidate programs left at this stage.
Our job has always been to get to 5 and aspire to 6. Indeed most of software development is just doing 5->6 in a loop.
I think LLMs help us get to somewhere between 3 and 5 faster than we used to. And a big problem with them is that programs in 3 vs 5 all already look substantially correct and there's no way to know if you're getting 3 or 5.
Generally, the above is not how it _felt_ to write software pre-LLMs, it was just a cute way to imagine what you're doing. Now it's weirdly apropos.
I once saw a YouTube video where two beginners were learning the basics programming, and upon opening a cmd prompt, one of them said "It's insane if you just type the right words into this box, you can steal someone's 401K in Arkansas" (Or something along those lines). It's always stuck with me that technically that's pretty correct. It's probably not how you'd go about it, but with the correct magic words in the right order it would work
One of my professors used an example like this to encourage you to pause if you ever find yourself doing something like "maybe if I just flip this < to > the tests will pass". The space of incorrect programs (even ones that pass your tests) is enormous compared to the space of correct programs.
If you like thinking about this and haven't seen this before, check out the busy beaver problem[1].
[1]: https://en.wikipedia.org/wiki/Busy_beaver
I remember the Computer Recreations column in Scientific American on busy beavers back in 1984. I was too young then to really appreciate it. (I really miss that column—and the Amateur Scientist.)
I found it here (page 19) https://wkbpic.com/wkbx/SA/1984/1984-08-01.pdf
flipping > to >= works quite often though!
I was about to say a lot of algorithms or invariant bugs for me are slight mistakes at an edge of something
"There are 2 hard problems in computer science: cache invalidation, naming things, and off-by-1 errors."
I've come to consider memory management the third
all code possible to run on a computer already exists. It's some permutation of all the bits of available memory. It's in there somewhere
This seems either mathematically impossible or vacuous, depending on what you mean ...
I think the intent is clear. We could just enumerate all strings in size and lexicographical order. Then the task is only to filter, which strings are programs we desire.
It's the same sentiment as "the typing itself is not the issue, it's the order of the button presses that's difficult."
Of course, by Rice's theorem, all interesting questions are undecidable anyway...
If we restrict ourselves to programs that can be stored inside the physical universe, suddenly everything switches from undecidable to O(1) :)
It's the same mindset some artists have when considering raw stone for a statue: there's a Greek god in there somewhere, you just have to find him.
It's a way of thinking, not some deep metaphysical mystery. Unless it is actually a deep metaphysical mystery, but I'd rather not go there.
You should check out Borges’, The Library of Babel.
Basically if you have an infinite string of characters in some language then you have every possible book, tweet, script, scribble, and scrap of everything ever written. You just have to find it.
Thats already a large universe to cover. Programs? Much bigger set.
the infinite string must have more refined properties than infinite length. an infinite string of a repeating char doesn't encode much of interest.
I believe it's more that the library contains the finite set of all books which have a (unique) permutation of the language's characters to size N. The challenge to finding the cure to cancer in such a collection of sufficiently large N is indeed finding which book it is in.
> all code possible to run on a computer already exists
You have to be careful; that line of thinking might get you trapped in L-space. You’d need a very skilled librarian and possibly a bag of peanuts to get back out.
Oook
I think this is a great abstraction of the process I've never considered in those terms and I like it. It explains a bit of how software development differs from some other building processes where the desired outcome is being discovered as you figure out how to make something that produces that outcome
There are plenty of processes that have this property, though. For example: setting up a home, workspace, kitchen, factory, office, etc. to fit a particular purpose. As you discover what works well in a particular space, you will refactor your workflows, seating organization, storage systems, etc to fit what you're doing. And it may even have multiple valid configurations - for example seasonal variations like outdoor vs indoor areas.
> It's some permutation of all the bits of available memory. It's in there somewhere.
True, but the number of possible permutations is so large it's incomprehensible. Let's limit ourselves to say a 50 MiB program, meaning the size of the compiled binary. If I'm remembering the math correctly, that amount of memory can hold 2^(2^20 * 50 * 8) different states. That's 2^419430400 or roughly 10^126000000. For comparison, the heat death of the universe is predicted to occur in about 10^100 years, or 10^107 seconds. You'd need to be able to look at more than a million programs per second to examine them all before the end of the universe.
I've thought similarly about music before. Everything's there in white noise. To get music you just have to selectively remove parts of it.
yes and if you want to code a little swimmer that explores program-space, you'll find there too!
Reality is a fractal (Mandelbrot?) and this is where LLMs will help science. They will help us explore the fuzzy edges faster. LLMs probably won't find new "leaves", but they should be able to cover a lot of useful but "boring" fuzz.
In my experience, a spec is often public (or available for purchase by the public from a standards committee for a few hundred dollars, close enough in business terms), but the source code for your competitor's implementations is not available for any price. The first barrier to entry to become a competitor in many industries is to implement an 835 page (note that those are rookie numbers) spec, building software that is compatible with existing data formats or legal frameworks or network protocols or hardware or whatever.
This is another example of the many invisible silos that developers end up in:
1. Some are developing enterprise software for use by their own employer, the spec only exists because it's useful to help the dev team manage the project. In this case, it's totally reasonable to expect the spec to co-evolve with the code. If the release gets pushed out to Q3 or the target has to be adjusted due to unforseen implementation difficulties, so be it.
2. Some are developing contract software for their customers. The spec exists as a communication tool and requirements that allows the developers to know what to build and to inform the customer that what they've built has reached a certain phase of the billing process (including completion). You're on a clock, and if you're late or you don't do what you said you'd do then your customer isn't going to pay you.
3. Some are developing software for abstract customers - either B2B or B2C. Often, the specs here are regulatory or industry-standard, implementing the spec is the function of the business. You're releasing updates and new products to make year-end sales targets or snipe announcements by your competition, and advertising compatibility with various standards is a crucial bullet point in the sales presentation.
Bug reports, feature requests, and especially deadlines have different meanings in each context. The way AI development agents can be used in each silo also varies enormously!
As a counter argument for the compiler perspective: what Cursor did is basically transpiling a project to another language, which is a form of compilation.
If you distill the feedback into a spec (probably a combination of both docs and tests) which comprehensively covers the functional requirements, then you could treat the code as cached materialization. The value of generating code again is targeting different non-functional requirements, like ability to run in different environments or catering to different performance characteristics.
I agree with iteration, but that's an orthogonal argument to how effective it is at turning prose to code. You are asking, LLMs are capable of translating a 835-page spec, but where does the spec come from? Follow the stream up from the built software product:
Binary < IR < Code < Spec < Prompt (aka goals)
For LLMs to faithfully fulfill the role of translating spec/prompt to code, it needs enough information about the problem/motivation and the constraints around it. Often, that's done by research or by importing proven modules. The human role left in this compilation chain is a precise-enough expression of the motivation, upstream of the built artifacts.
Edit For example, SWEbench Pro was supposed to be a measure for how good LLMs are at compiling prose into code. At least 15% of those tasks were underdetermined by ambiguous prose in the spec. Of the issues that are specific enough, Claude and Codex indeed do compile to functional code with self-written tests. Source: https://github.com/kimjune01/swebench-pro-audit
Indeed, this is a much more realistic and useful framing of what to do with the current capabilities of LLMs. They work so fast, at such a high level, that you can speedrun the co-evolution process and end up with a spec and code that works. You can actually use second-system (fourth? fifth? I lost track) syndrome to your advantage because it’s so cheap to throw things away. And they don’t just write the system, they also write the test harness, the benchmarks, the failure simulation, the statistics…all the stuff that seems like “overhead” but lets you drive that evolution with data.
I’ve kind of drifted into this mode a bit at a time over the last year, but hadn’t stepped back to make such a coherent explanation of it.
The source code always was the result of that distillation, or at least a crucial part of the result. I’m unconvinced that natural-language specs can replace that role. Even long before LLMs, when writing specs I often resort to some form of formal language to precisely nail down certain parts of the specification. And some parts are always left open, where I know I’ll get to them when writing the code, because they are more effectively solved “in the code” than waterfall-like up front.
A zen master asked a junior engineer to go get him a rock. The junior engineer asked what kind of rock he should get and the master replied by knocking him in the head with his cane. The junior thought for a minute, then asked, "would you like a round rock or a flat rock?" Again, the zen master rapped him on the head. The junior thought for a moment, then went to the creek and fetched a rock at random. He brought it to the zen master who inspected it and then said, "No, I want a flat piece of flint for starting a fire." The junior engineer was then enlightened.
Stakeholders will only give you requirements after they see the prototype. Thus, the spec and the prototype to elicit it are two sides to the same coin. Once the spec if correct and the prototype is starting to show its age, throw away the prototype and implement the spec with a fresh start.
> Stakeholders will only give you requirements after they see the prototype.
That’s very much incorrect and sounds like someone who love building more than communicating. Even without a prototype, users and other consumers will be able to explain their problems. It may not use the same metaphor map that you’re used to. But it’s very much a description of the problem space.
Once you design a prototype, what you will get is feedback about the solution you’ve designed, not the raw problem. If you’ve not listened well in the first place you may be well off the mark, and have to work harder to correct things.
Even if your story, the junior would do way less work by asking the master what he intends to do with the piece of work instead of focusing on the rock itself.
whether the hypothesis originates at the users or in your guess for the users is irrelevant in that they're both hypotheses with no certain outcome until trial.
They are. But engineering is about minimizing costs in solving a problem while ensuring requirements are met. Not worrying about costs is not engineering. It's either a research project (where you want to know if something is possible) or playing around.
The cost of prototyping has dropped so much due to LLMs and coding agents that it basically always makes sense to prototype something to elicit feedback, just as part of the requirements gathering process. E.g. if you can spend $50 to understand something that will save you two weeks of effort, wouldn't you do that? That's the sort of dynamic I'm talking about.
The cost of prototyping has always been very low. I can quickly sketch a UX flow using balsamiq. Or copy-paste paste code from docs and examples to show a rough working model with. If you can extract the essence of the problem you can quickly realize cheap ways to demonstrate a solution, and not spend two weeks trying to code it.
If there’s something that is wrong with using LLMs to prototype, it is that those prototypes are always too complex. Instead of decomposing a problem and testing unrelated concerns apart from each other, it’s often a mix of everything that the LLM user had been thinking about.
Often there’s no rapid iteration where you are comparing several approaches. Instead it became tunnel vision and confirmation bias where if it’s working, that means it’s the solution.
Well, since we started training them on RLHF-style rating of the last generation they've totally collapsed back onto predictable suffix generators, which is why there are desperate, low-AUC BERT-inspired clasufiers bolted to them now.
They have like, Chomsky grammars now. And you can't fix it, it's structural to the process. You have to rewind almost to the pre-train.
The argument being made here is really incredible when you unpack it.
1) The construction of the Empire State Building was particularly effective due to the depth of human-to-human collaboration.
2) Isn't it great that we can burn a bunch of dinosaur blood to convince ourselves that we don't need other humans?
This is a company selling picks and shovels for the AI bubble. Of course they're going to drip feed "AI is so awesome guys!" stories on their blog.
If you strip away the hyperbolic statement about compilers and the Empire State Building, it's an internal success anecdote.
Would be interested to hear how all of the self-directed AI decisions worked out after launch. This is where I run into problems with AI. Sometimes it makes product decisions that aren't immediately visible and make no sense and cause serious problems down the road. Also, an analysis of how much time was actually saved by using AI? since it sounds like there was a lot of guidance and testing to make sure what the AI did was correct.
> 2) Isn't it great that we can burn a bunch of dinosaur blood to convince ourselves that we don't need other humans?
God I hate this cynical take. Yes human beings use energy. We should look to generate more energy as that's a sign of wealth. The "energy bad" take is so lazy and ultimately anti-human. And in this case its especially obnoxious because what does "we don't need other humans" even mean.
not in any way that matters, no. this is a strawman and a really bad one too.
I believe you are projecting your personal misgivings onto the argument actually being made. May be better to write your own rebuttal instead?
I'm pretty sure they just did...
A rebuttal would spend some time trying to make a case, this is a petulant complaint.
One person's petulant complaint is another person's rebuttal.
I think what you're really saying is you don't agree with OP's perspective...
The whole resource argument against AI is quickly becoming obsolete. Time to move on to a new argument.
Headline true, article false.
AI exists at the boundary of codification. When you understand something well enough to express it in a deterministic way, it is no longer worthwhile to keep asking AI for that thing over and over - you should codify it.
But wait, I hear you say? AI has knowledge! And we can rely on that knowledge and sprinkle on top some brief instructions and it'll get things right and work out the details itself!
That is spec driven development.
And this is great one single time. Except, knowledge shifts. You are outsourcing part of what you've codified to society as a whole, in the form of language model knowledge. You're also trusting current and future models to always produce something that meets the requirements, and never be quantized under you, and produce the same result next time despite being a probabilistic system with no memory, and and and.
Better to freeze everything in place in your codebase so you get predictable results - in other words, codify it! Let's store the desired state of that button in code in a very complicated thing that's existed for decades called a string, so it always says "Finish" and never says "Done" - even if we regenerate the app.
That's pretty unexciting and doesn't justify tens of billions of investment. Which means you won't hear it from anyone with tokens to sell. But it's what you have to do to make something people want.
I've heard something along the lines of "Claude is like a compiler: source code is the new object code, you don't look at that anymore" many times.
And I don't really think this is true. Compilers are usually deterministic, and whilst we can find edge cases, it's nothing like an AI agent writing all the code for you.
I think you have two choices, given the Claude is a code generator and not a compiler: (a) you review most or all of code to make sure it makes sense, or (b) you trust but verify via a strong test suite, potentially also created by Claude.
The problem with (a) is that you lose a lot of the speed-up. The problem with (b) is that you have no human oversight and the code may be incomplete, badly designed, or plain wrong.
Currently we review all code because correctness is extremely important to what we do, but that comes at a cost.
I don't know what the answer is here, in general. Does trust build over time? Do the models just get so good we can trust them to make zero mistakes?
Part of the value of LLMs and humans is nondeterminism. Pair nondeterministic output with strictly verified results (proper tests) and you can create a useful working system.
Humans can't build a system perfectly, and agents definitely can't. And agents (like humans) will build a different system every time even with the same prompt. Even if it's just trivial differences like array vs linked-list, there are still differences.
The value in having executable code is that it won't change its behavior unless you deliberately modify it. I don't think there are any shortcuts. Models will get smarter and smarter and make fewer mistakes, but you'll still want to produce real source code to execute because it's consistent (you can sell it as a product, set it and forget it, etc.), and most importantly:
running real code is orders of magnitude faster than having an AI either run the process "manually", or have it write the code out multiple times.
Thanks, that's really helpful.
What do you think happens in terms of testing/reviewing going forward? I'd really appreciate your thoughts on that.
If you have a product intended for humans, you're gonna need humans to do acceptance testing, just like now. All the software engineering principles you learned (did you learn them? now's as good a time as any!) are still just as applicable now as they were 10 years ago.
I really like what exe.dev is doing. And their philosophy of how cloud services could work is amazing. I have a monthly subscription and it's really cool to just vibe code a little website with Shelley and have it running in minutes. I feel like I have so much fun using their service, and spend so much less time thinking about what things will cost, or what cloud service to use. It's just so much simpler and the friction is so much less compared to other cloud services.
Agreed. Big love for exe.dev. For quick low stakes projects, using their Shelly assistant is the simplest way to give an LLM a fully persistent VM with web services I've found.
Specs might become one solution for coping with the need to review increased volume of code. A spec is a higher level of abstraction than code, which is a higher level of abstraction than machine code. The industry made the transition to higher-level once, paradigm is changing so it might happen again.
The workflow I imagine is either deriving specs from the conversation or reverse engineering the code to spec, review and edit the spec which should be tighter and much more compressed, then deterministically compile to code. Of course we don't want to be spec-first only, that would be going back waterfall, but doing iterations back and forth.
Now, Claude is not a compiler because it is closed and non-deterministic (they do opaque processing on server, hiding reasoning tokens), but LLMs might be. We refer to a piece of code from npm/pip by name to get some code by downloading it. We then have lockfiles with hashes to ensure integrity. Currently we are vibing it, but in the future we could refer to a piece of code by prompt/spec and getting the code by inferring it. To ensure integrity, the lockfile would be hashes of open weights and inference code (and ironing out implementation details like non-determinism due to GPU scheduling, etc.).
It's funny how you describe something very close to what I was attempting to design. It was meant to trickle down from tickets to deterministic code. Data wise, it should be like a pyramid of tickets diluting layer by layer into leaf nodes which were "implementable" as statements in code. I'm not sure if that description makes sense read by someone else.
I think what made sense was envisioning a nanoswarm of LLMs (anticipating ASIC performance) diluting specs into semantic logic nodes, but a conversion from these to deterministic code made the vibe coded experiment come to halt. Your lock approach could be a shortcut to that.
Totally off-topic: it's quite intriguing that you can feel once friction starts building during a design phase. Suddenly everything slows down. I wonder if it's quantifiable and therefore can identify "wrong" design choices made by either humans or LLMs.
(Hoping some claw bot pics this up to finish my idea on my github tix repo in the initial-design branch, as main is empty ~ MPL2)
This resonates, I can imagine the development process as a cascade: we get business goals and user feedback, product is defined which can be pretty fuzzy, designers define UI, engineering works data models and tech constraints. Inputs at the top can be fuzzy, but each node in the graph applies some intelligence and clarifies it.
Related to the friction you mention, since the flow graph of the context is not preserved when it gets transformed into code, I find LLMs getting confused and making worse changes, because the explicit decisions with good reasons are getting mixed with implicit decisions from LLM surfaced during designing/planning. Sometimes when things change the code should change radically to match the new reality, but LLMs often "play it safe" by accumulating code and adding to tech debt. With a flow graph when change is made in the upper levels, it could propagate and allow decisions in lower levels to be revisited.
This might work if we have completely new harnesses for this, because the current one’s intentionally introduce non-reproducible context and tools.
The current way we’re doing SDD is extremely pointless. You reach a decision point that is under specified, make a decision, update the spec and continue the implementation. There’s no guarantee that the implementation followed from the updated spec, and not from the decision.
I agree, another issue I have with existing SDD frameworks I've seen like spec-kit or openspec, is they seem to miss the mark by coupling specification (what the software should be) and execution (how to get there). If the description of execution is needed, it means the spec is weak. As a rule of thumb if a framework defines execution concepts like tasks or tickets, that is a red flag for me.
prompts are not specs because most often, prompts are underspecified. You can prompt fizzbuzz well enough for it to be functional, but not ffmpeg. So the distinction is less about Claude's capability, more about the quality of the input. Garbage in garbage out
I think something can be considered a spec to some degree if outcome was derived from it.
It might underspecify, but that is just a quality of the spec. And as long as there is any discretion at implementation time, then the spec necessarily underspecifies to some degree. The goal of a spec is often to be clear on key directional concerns. Consider RFCs.
There's a trade-off between specification and implementation discretion. But if you think about starting at one end ("an internet forum") and then incrementally expanding the specifications ("it has topics with many posts in a linear paginated list"), there's some point where everything you care about is in the spec, and the rest is left to implementor discretion.
That's what I'd consider a good spec and a good LLM prompt.
and LLMs are really good at filling in the blanks. It's like a really big import on generic application knowledge, and if you draw the circles well enough, it will draw the rest of the owl.
You can reference other specs in a prompt. Video codecs are well specified with hundreds of pages of specs, they are probably in some form in the weights already, but you could also reference them and hash them as inputs directly to create a functional alternative to ffmpeg.
i meant to make the distinction between 'prompt' and 'spec' in that prompts are something that a human types in by hand. Interesting programs are a combination of canon and novel input
Gotcha, yeah if a prompt does not produce the result you want, you need to add more details to it, depending on the scope it can be a lot. When it includes all sufficient details it becomes a spec.
> A spec is a higher level of abstraction than code, which is a higher level of abstraction than machine code
No it’s not.
Just like a quick doodle is not an higher level representation of the Mona Lisa. Sure for someones that knows the latter, it can suggest it. Or for someone that doesn’t know it, it may provide some basis of conversation. But it’s not the real thing. You can’t provide a doodle of something and expect an artist that hasn’t seen it to paint it.
Doodling’s value is that it lets you iterate on ideas without the accidental constraints that comes with the implementation (choosing paints, finding references materials, deciding colors,…). Some necessary choices are kept for later, while you decide on the most important ones.
So a spec is useful in designing software, but it’s severely lacking in implementing it. And the decisions made in implementation are as important as the ones made in the design. Even more after a while in production.
> Claude wasn’t just a compiler here. I never handed off a task and let an agent make a bunch of decisions in order to reduce it to practice.
> I’d say that, in all the ways that matter, I understand the code.
I think the dissonance here is really important, and not a bad thing at all. A lot of the decision _were_ handed off the the AI, but they weren't the decisions the author cared about. This is a big selling point of AI! If something is doable with a computer, it’ll figure it out. 30 minutes and 200m tokens later, it’ll take any idea and declare “the feature is fully implemented.”
The hard part is figuring out where to inject that friction, so you can see where it's making decisions for you that matter. The author approached this by incrementally building the thing, reviewing and poking and prodding at every step. A week of attention following a bunch of design discussions is fast, but that's still not trivially cheap.
I want to see us talk more about the decision exhaust of agents, because the better the models get, the more decisions we'll want them to make.
I wrote a bit more here: https://tern.sh/blog/compiler-never-says-no/
> I read a vanishingly small amount of the actual code.
This sentiment kind of saddens me. I'm all for burning tokens to write throwaway code just to prototype a solution, but I don't get not reading (or at least familiarizing yourself with) the code that you will deploy to prod.
> But our VMs start fast, so fast that even if we created the DNS entries before creating the VM, our users still had to sit around waiting for DNS to propagate, which occasionally took minutes, not seconds.
To my (limited) understanding, this is not a good idea, and is an unfixable problem from the server side. Companies, VPNs or ISPs or routers, often use their own DNS servers, and those can have caching logic, which means it doesn't matter how fast your own DNS implementation is, as the users lookup request wont hit your DNS server, it'll hit an intermediate cache.
Presumably these are new subdomains, if a caching DNS resolvers never saw the domain yet, it would simply query the authoritative server.
There's a small nuance here that DNS spec also caches negative (NXDOMAIN) results, but the TTL of those is controllable by the SOA record on the domain that contains it.
exe.xyz has a fairly short 10 second negative TTL, so even if you tried to resolve it before it was set up in the authoritative DNS, it wouldn't stay negative-cached very long (though some DNS caches put a lower bound on the TTL and cache it for longer)
> dig +short soa exe.xyz
ns1.exe.dev. hostmaster.exe.dev. 1854440 86400 7200 1209600 10
Usually when you create the VM, it gets its own DNS subdomain that looks like `my-new-vm-name.exe.xyz`. (I'm not affiliated with exe.dev, I just use their product). So I doubt there would be any cached requests for that particular subdomain before the VM is created.
NXDOMAIN results and other errors have their own TTL value though, so if someone queries too early in the process you could be waiting a little while for their cache to check again. The value for this is set in the SOA record for your DNS zone.
For example, example.org/example.com seems to have TTLs set at 300s (5 minutes) for A records, but the negative caching value is 1800s (30 minutes). If your VM domain is set the same way, you could see a premature lookup causing issues for 30 minutes instead of just 5. Even the 5 minutes could be a pain if your VM spin-up process expects to be able to lookup via the name very early and will fall over if it can't.
I always don't know enough about DNS. This is great stuff.
If exe.dev is running their own DNS zone, then wouldn't they be setting the expiry value in the SOA record themselves? So they could set it to something short, and secondary servers are supposed to honor it, right?
> wouldn't they be setting the expiry value in the SOA record themselves?
If they thought about it. But it is something that even people relatively experienced in DNS matters forget about until it bites them and they've spent ages diagnosing the initially odd looking problems caused.
> To my (limited) understanding, this is not a good idea, and is an unfixable problem from the server side. Companies, VPNs or ISPs or routers, often use their own DNS servers, and those can have caching logic, which means it doesn't matter how fast your own DNS implementation is, as the users lookup request wont hit your DNS server, it'll hit an intermediate cache.
All caches should respect the TTL of DNS records, no matter what they are, though there are some interesting bugs out there.
Firstly the human bugs: a lot of people assume switching DNS is always fast, but they haven't had the TTLs property of a DNS entry explained to them so assume that if the change is working here then it is going to be working <there>, <there>, and <everywhere-else> too. The default TTL values are a mix, it used to be that 24 hours was common though these days four hours or less, sometimes even one hour, is what you are more likely to see even for defaults. Whatever value your entries have though, that doesn't mean a fixed × hour window that it will switch at the end of. If the TTL is four hours your local cache last checked three hours ago but mine last checked one hour ago, a change will take an hour for you to see and three for me to see. A cache that has not recently looked up the entry will see it immediately next time it is asked - this is why some assume it is always instant (they add a new sub-domain record and it works instantly because no one has ever requested an address for that name before, for instance, and assume this is how it will always work).
Then there are coded bugs. TBH these days I ignore those: they are rare enough that if you are caring about that sort of edge case a lot then there are a lot of other weeds you'll be worrying about too and you'll never get anything done. For my domains I mostly have everything set to five minutes (300 seconds) as they are rarely referenced and DNS infrastructure is not a high-power service these days. Some would suggest that using such a low default is unfriendly to DNS caches and the root servers, but my names are very rarely referenced outside my network/vpn where my own DNS authoritative servers are what get directly queried anyway so that isn't going to matter. example.{org|com} and some sizable companies use 300s too. Why five minutes and not less? There used to be a very common caching DNS server that would ignore anything lower than 300 seconds and apply its own default (14,400 seconds, four hours, IIRC) instead. This goes back a couple of decades though, if anyone is still running that they deserve some DNS lookup failures! If you are wanting to use longer values normally, just remember to lower your TTL a time before any planned change so the change propagates ASAP and raise them back up after. One extra thing to be aware of with short TTLs is an outage affecting all your authoritative DNS servers may cause less friendly errors in many cases than getting an out of date address would result in, so make sure your DNS servers are stable and have few (preferably zero) shared potential points of failure.
A compiler is an algorithm and Claude isn't*. A compiler almost never produces a wrong output, even when compiling an extremely complicated program. But a compiler must be clearly defined and is limited to input/commands it's defined for. A compiler will "correctly" process input into unintuitive output, but that's not always what the user intended (e.g. omitting large sections of code that are undefined behavior). Claude makes mistakes, but can process any input/command expressible in training data, and has more potential to infer when something is unintentional.
Tools like Claude are great at building algorithms, then tests to verify them. The algorithms they build aren't just compilers, but any software that can be verified. Even though Claude may not be a compiler, it has the potential to (at least if it improves) create production-grade software that can be verified, like other compilers. But maybe not good UX without human input (expanded to any subjective experience, like video games).
* Not a "source language to target language" deductive algorithm. Technically Claude is an algorithm to predict the next token, but acting as a compiler or anything else it's an inductive heuristic, because it guesses (https://stackoverflow.com/q/2334225)
> A compiler will "correctly" process input into unintuitive output, but that's not always what the user intended
Importantly, this behavior is deterministic. If the compiler takes your code and spits out something you did not expect, it will always spit out that same result, given the same input. Once you figure out what went wrong, you've solved that problem.
Generative AI is intentionally not deterministic, and that means you're going to solve the same issues -- hallucinations, primarily -- over and over again.
Determinism isn't the main hurdle. It would be a moderate effort to engineer a deterministic claude that would be no more useful than the current one.
The key characteristic of a compiler is that it takes a source language with a well-defined meaning to a target language with a well-defined meaning.
The reason claude fails to be a compiler is that its input language (english) can't be precisely defined. (And even if that were possible, it would very often fail to preserve the meaning when "compiling")
AI could be deterministic, only it requires sacrificing performance and nobody cares enough. Although algorithms are more predictable, and you can follow an algorithm's steps (even a complex algorithm can be broken into large sequences of steps, which may be broken further).
Determinism is the wrong thing to worry about. As you say an LLM could theoretically be deterministic.
Chaos (prompt instability) is the problem that isn’t solvable if you want to treat an LLM like a compiler by doing something like checking in the prompts and rebuilding the code each time.
In general if you change a single line of source code, a compiler will not produce a completely different program. If you change a single word in a prompt, you will get output that likely doesn’t match a single line of the original output.
Chaos is what leads to unpredictability, and there’s no solution to this on the horizon.
I think you're correct, I'll just point out that compilers are also chaotic today and it is often accepted. A small change to source program can lead to big change in performance. It's especially apparent in high-level and declarative languages - SQL or anything with GC.
With respect to performance that’s true. But I’ve probably spent thousands of hours in my career on performance issues caused by small SQL changes.
And LLM chaos is unimaginably worse. A single white space can result in a program that has zero overlap with the original.
They can guarantee that certain small changes won't create a big change in performance (provably, verified through tests, or trivially). An LLM can't.
“Generative AI is intentionally not deterministic, and that means you're going to solve the same issues -- hallucinations, primarily -- over and over again.”
I see it differently. Sure there are still some hallucinations (but less and less as we progress on), but I see the AI-is-not-deterministic characteristic as a good thing, in the same way that humans and human designers not being deterministic is a good thing. Imagine giving a project/task to a human, and it was completed exactly in the same way with exactly the same outputs as another human that the project was tasked to. Would that be better than each human having varying outcomes of the project? I think most would agree the answer is no.
I agree with what you're putting out there. Every time I see one of these LLM/compiler pieces, the only thing I can think of is that these people don't know what a compiler actually is.
>A compiler almost never produces a wrong output
back log of compiler's bugs can be pretty large
In a production-grade compiler like LLVM they are rare enough that it compiles giant projects like Linux and Chromium without detectable issues.
Maybe eventually Claude can, since it can write tests, but compilers / (deductive) algorithms have other advantages like efficiency and predictability.
>In a production-grade compiler like LLVM they are rare enough that it compiles giant projects like Linux and Chromium without detectable issues.
https://github.com/llvm/llvm-project/issues
Yea, sure, but there's waay more than Linux or Chromium
I guess we can come back in 20 years and see if Claude is as reliable as LLVM is today. That would give a comparison between similar aged products.
Not really. As the C language was created around 1972 and by 1973, Unix was rewritten in it. By 1975, Unix was released commercially. That’s like 3 years between design and commercial usage.
Have a go at writing something in 1975's C.
x=-5 Is not x = -5
There's no unsigned int.
It's a better comparison to the usability of Claude than you might think. It's doable, but hardly ideal.
we could make a similar observation about determinism. given a set of inputs, the compiler will create the same output each time.
not always true. but very close (pathological coding defects and explicit randomization excluded)
very different than an LLM.
> almost never
But not never. "Almost" is doing a lot of work there.
> A compiler is an algorithm and Claude isn't
"Algorithm" is not a word with a definition, so I can't say that you're wrong, but I struggle to see how you conceptualize "algorithm" in a way that does include a compiler but does not include a language model.
I think the distinction you're trying to draw is better captured by comparing inductive and deductive program synthesis.
In computer science, one definition of algorithm is basically any program that runs on a turing machine. By that definition, any LLM is an algorithm.
A compiler is a deterministic algorithm. Clause is not.
Of course Claude is an algorithm, but it's a very vague, fuzzy and complex one, where you have no way to determine what's going on inside, and no way to know up front what's going to come out.
Those are the exact concepts I was looking for.
> "Algorithm" is not a word with a definition
what?
Rogers "Theory of Recursive Functions and Effective Computability", page 1, emphasis is the author's:
§ 1.1 The informal notion of algorithm
In this chapter we give a formal (i.e., mathematically exact) characterization of recursive function. The concept is basic for the remainder of the book. It is one way of making precise the informal mathematical notion of function computable "by algorithm" or "by effective procedure". In this section, as a preliminary to the formal characterization, we discuss certain aspects of the informal notions of algorithm and function computable by algorithm as they occur in mathematics.
I don't understand why you people act like you're stumped by literally, literally page 1 of computer science.
> I don't understand why you people act like you're stumped by literally, literally page 1 of computer science.
… and yet you didn't stop for a moment to consider that in a field as fast-moving as computer science, a concept which might not have had a formal definition in 1967 acquired one in the past 59 years? See, for example, Sipser's Introduction to the Theory of Computation, which has an entire section (3.3 in my 1997 print) titled “The Definiton of Algorithm”.
Yes, did you read that section? It's using different terminology but it's saying exactly what I'm saying (of course it does, it's elementary computer science). There is no accepted definition of the word "algorithm": there wasn't one in 1967, there wasn't one in 1997 and there isn't one in 2026. There are models of computations we can prove to be equivalent to each other, but no commonly accepted definition that captures the way we use the word in normal speech.
This is not page 1 of computer science. And you're being bitchy.
> you're being bitchy
Honestly you're right, I'm sorry. I'm not having a great day and ended up venting online. Logging off.
oh wow. this is a rarity on an internet dominated by toxic rage-bait. apology accepted. I think you probably have interesting things to say on the matter and I look forward to reading them on a better day. respect.
If you pair LLMs/AIs like Claude with a formal verification framework you essentially have a new sort of compiler.
That’s technically correct, but it still means that someone has to write the formal specs that Claude’s output is verified against, and someone has to design the formal languages the specs are written in. And once you have formal specs that truly covers all aspects that you ever will care about, it’s unclear if you couldn’t instead build a non-LLM mechanism that efficiently and deterministically spits out an implementation of such a spec.
I expect in the future it will become hybrid. Formal specs provide guardrails for LLMs / AI but within those guards LLMs will author traditional compiler programs and at times drop-down to manually compile / prove certain snippets.
Could we describe the first humans manually writing Assembly and their own higher level abstractions as "compilers"? Loosely I think we could, and LLMs fit the same mold.
I disagree in the sense that source code (compiler input) leaves little space as to the behavior of the resulting program. With LLMs, either there is much more implied leeway, in which case I’d argue that it’s different in nature. Or the specs will restrict the LLM output to a similar extent as a conventional programming language restricts a compiler’s freedoms, in which case it’s not clear that LLMs are the most expedient technology for generating that output.
Levels of abstraction actually matter with regard to how much observable behavior they leave unspecified, and with regard to how much control is needed over each respective behavioral aspect.
And claude isnt a replacement for the chain of humans, you dont know what you dont know, thats why we have a person in a role to gatekeep each step.
Anyone using claude can see it misses steps that an engineer wouldnt, it can make bad choices that a sysadmin wouldnt, it can pick the wrong order a project manager wouldnt, it can pick wrong cost models that a bean counter wouldnt, etc etc etc
But, I'm pretending it is and living with those mistakes.
Claude is more like a Probabilistic Turing Machine. [0] It's correctness isn't deterministic but rather a distribution. It is predictable.
There are lots of places in computer science that determinism isn't necessarily the best.
- UDP video calls let packets vanish or arrive corrupted, because waiting for retransmissions would freeze the picture.
- Stochastic gradient descent picks a random mini-batch of training data and treats it as the whole dataset, because computing the true gradient on every step would make training infeasible.
- Speculative execution in modern CPUs guesses which branch a program will take and rolls back when wrong, because waiting for the real answer would leave half the silicon idle.
What we want is something like a Las Vegas algorithm, a probabilistic machine with a cheap verifier.
[0] https://en.wikipedia.org/wiki/Probabilistic_Turing_machine
Determinism isn’t important, but there are usually threshold of acceptable performance in all off those cases. Machine Learning was all about improving accuracy even when you know that there’s probably an error somewhere. But the current LLM hypers refuses to acknowledge that the answer to a prompt may be wrong (even though the providers have a warning line about it).
Instead of assessing the risk and provide corrective methods, all they do is pushing to use it more and more everywhere.
> I’d say that, in all the ways that matter, I understand the code. Sure, if I had to hand-edit it now, there’d be a serious learning curve. But I won't have to.
Famous last words, but point is taken.
The non deterministic nature of an llm breaks the metaphor that they are like a compiler.
However, it’s not foreign to compilers to receive feedback from the running program (PGOs), so there are still parallels to the feedback we provide LLMs that guide their “optimization”.
I think even calling LLMs a non-deterministic compiler isn’t accurate either so ultimately I agree the metaphor doesn’t quite work.
I do think LLMs are like compilers in terms of how they changed how we build programs from a historical context. But that’s about it.
The non-determinism is theoretical solvable. Chaos (prompt instability) isn’t. When you change a single word in the input, you get a completely different output. Because of this LLMs will never be compilers.
- it is a very very very very fancy autocomplete
Each year, as the models improve, we add one more "very".
Extrapolating as "very" approaches infinity: "Let there be light!"
how many do you add before they become more fancy at auto complete than humans though
Human developers generally don't autocomplete.
"thanks, you too"
You write code with the same level of attention you give to small talk with strangers?
Let they who has never ='d when they meant to == cast the first argument.
(There's a reason why we benefit from code review).
Way to move the goalposts!
"thanks, you too"
:P
“lgtm”
12?
- its like going at the speed of light to achieve 100% accuracy.
- you ll keep getting closer and closer but you ll never be at 100% as long as you have a non zero mass
Sure, but it's at the point now where it can autocomplete a fragment of a sketch of an idea into a complete working product.
No, it isn't. It can autocomplete a fragment of a sketch of an idea into something that may or may not match the original vision and almost certainly has serious bugs which only manifest over time because they elude a cursory test.
You say that as if it wasn't true for anything human programmers ever produced
Because Claude isn't a human the perception of it is different.
Humans can fly planes, computers too can fly planes. I'm not going to give the same leeway to a 'computer' flying a plane that I would to a human. Because we understand that humans are just that, humans. The demands placed on a computer are so very different.
What about driving a car?
Computers are already driving cars autonomously, possibly better than the average human already, and quite plausibly will eventually be better than the best human driver.
Precisely my point. Say for examples planes, trains or automobiles, humans may make way more errors than a computer does but you can bet your bottom dollar the threshold for allowing computers to autonomously control the above is way higher. Because the expectation is that if it is a computer it ought do it far better/safer.
There is a lot of rehashing of LLM arguments in the comments here, but I'd love to read an actual distributed systems expert's take on the DNS architecture the author and/or Claude ended up with. One of my personal fears about LLM coding is that some of those implicitly outsourced decisions turn out to matter more than expected. Does the "timeline" stamp make sense? (I know there's not much detail to work from.)
Timeline from the article sounds a lot like a logical clock, which is a well known primitive in distributed systems: https://en.wikipedia.org/wiki/Logical_clock
> exe.dev VMs have nice domain names: vm-name.exe.xyz. When we start a new VM, we add a CNAME entry or three. Easy, right?
> But our VMs start fast, so fast that even if we created the DNS entries before creating the VM, our users still had to sit around waiting for DNS to propagate, which occasionally took minutes, not seconds.
> We did the obvious thing: We wrote our own DNS server, so that DNS always immediately matched the source of truth. And life was good.
> But latency matters, so we added regions. And just like that, DNS became the long pole again, because all DNS was served out of Oregon. Also, deployments caused tiny DNS outages. To fix this, all we needed now was a geographically distributed but fully consistent DNS server.
> We did what a sensible engineer does when faced with a hard problem: cheat. We vibe-engineered a distributed DNS server tuned to our specific needs.
I'm sure they have already considered this, but what problem does this solve that couldn't be solved by using dnsmasq, unbound or something like that? Why reinvent the wheel?
The question is more what this really solves. They said they built all this because DNS propagation was too slow. But secondary dns servers by e.g. your isp will still have old records when you change something. So what did they earn? Every is correct within their network pretty fast. But for a normal user there isnt any difference because 8.8.8.8 or 1.1.1.1 still cache old records?
That doesn't require vibe-coding, which is what they are selling as the solution to every problem.
This seems to be grasping for analogies to make sense of the work this person was doing.
A compiler does a lot more than source code translation.
There are specifications that tell us the de jure specifications of the language, if there is one, and then we have to recognize the de facto implementations of said language. The often disagree and leave much on the table. Some times on purpose, such as implementation details, and other times by omission.
Users of this compiler expect a deterministic compilation of the source text into the target code but it’s rarely 1:1. There are optimization passes, inlining, barriers, etc.
And then there are the run-time effects of executing a program!
I think the analogy gets a little weak because natural language is not a precise enough language to specify discrete systems.
What it sounds like the author is doing is bypassing decision points with other people and delaying making decisions themselves until the LLM agent forces them to? Which is a fine approach but I don’t think the analogy with a compiler is necessary.
People seem to have a hard enough time understanding branch prediction and thread barriers.
I still think it is (in the source-code generating mode) a compiler, just like LiquidHaskell is. (I haven't used LH but it essentially can automatically supply functions based on the type conditions you specify.)
In my view, reasoning LLMs have learned a large number of close to true sentences of "logic" in natural language. These rules are not consistent (unlike typing rules of LH that allow creation of provably correct programs), but they work in practice better than LH. Essentially, they represent an unsound formal specification of natural language, together with many useful (almost)tautologies that helps solving constraint problems (just like LH does, but again unlike LLM, provably correctly).
LLMs proved that human language can be formalized reasonably close to soundness. I believe we can have a sound formalization of human language, and that kind of formal language will be once superior to LLMs.
It will also not be as complicated as LLMs and require only fraction of compute to use.
The entire "Don't Read the Code" argument is just insane to me and ironically when you see who the people are that are advocating it, it's often the same people that talked about microservices, the latest java script framework, no-code or any other methodology that was in vogue at the time just to avoid focusing on solving the real problems; The illusion that they are just one "smart step" away from achieving a 10x productivity boost that will make everything else trivial...
> By the time I was ready to build a keeper, I had accumulated a scar-tissue document that was empirically sufficient to guide an agent through most of the important decisions, at every layer, ranging from high level goals through architecture down to the occasional low level detail, such as the exact shape of the data type for load-bearing concurrent caches.
Waterfall is dead, long live waterfall!
I'll go a step further and say Claude is an Interpreter.
It even has just-in-time compilation: it can directly follow a spec and generate small snippets live.
I'm pretty sure we'll soon see services (admittedly highly inefficient) built on Claude-as-a-backend.
Why inefficient? If Claude continuously makes the system more efficient based on demand, there's no reason for the system to be less efficient than if it was managed by humans.
I believe they mean Claude does the work directly, rather than writing software to do it.
words have meaning
It's not. It's a cargo cult developer that produces more code than you can review and confidently pushes bugs to prod.
I don't want to hear this ever again. No Patrick, mayonnaise is not an instrument.
it is when used to make music
Claude doesn't even make machine code though. Or it can if you really want to, but that's now how people use it. If you wanna include transpilers or interpreters, fine but the answer is still no, otherwise our git commits would just be prompts that get built either in CI or JIT. People deliver code, not prompts.
Maybe that could change some day, but even then, it wouldn't be a useful designation. It's like saying "oh technically the Americas are an island." There's a reason we still distinguish between a transpiler and compiler.
Why not a wildcard DNS pointing to a few Anycast IP addresses with proxies?
yo dawg! I heard you like distributed services
So I made a distributed service for your distributed name service so you can distribute your names to the service that distributes them on the onternet
I'm kind of getting to the point where if I know something is vibe coded, I just won't use it. It's not an anti-AI thing, it's just a quality thing. Pretty much every piece of vibe coded software I've used has been bad in some regard. The worst ones are the ones that aren't obviously bad but rather subtly bad in dangerous ways (the article about OpenCode yesterday definitely made me nope out of using that)
AI psychosis.
When Claude does its thinking I'm often reminded of https://xkcd.com/303/
the shill is strong with this post
> I’d say that, in all the ways that matter, I understand the code. Sure, if I had to hand-edit it now, there’d be a serious learning curve. But I won't have to. And more importantly, I can reason about the system, share perspectives with my colleagues, and guide agents on future work.
If there's a serious learning curve to editing code then you don't understand the code. We used to call that 'on-boarding' when you brought a new engineer on the team as they got up to speed as to how the codebase worked.
Nevermind "comments", has anyone with a name and reputation claimed they are?
The title is a reference back to "Is Claude Code a Compiler?" by the same author, linked from the first sentence: https://commaok.xyz/ai/is-claude-a-compiler/
And then claim is attributed right there:
> The hands-down highlight was the talk by Erik Schluntz: Vibe coding in prod.
> Among other things, he drew an analogy between LLMs and compilers.
Didn't read the article because the conclusion is obvoius enough to me as is, never heard of Erik Schluntz before, and drawing an analogy between two things means very little anyway. "this bike is about as heavy as that table" doesn't require the rebuttal that bikes aren't tables, for example.
so far I mostly saw it spewed forth on HN
Heard it a bunch of times from my superiors at work, if that counts.
Spec-driven development is sold on the idea that you regenerate modules or entire services when you change the spec. It's a really dumb idea but was briefly popular in enterprise, probably still is in some companies.
A compiler does not transalte specs to code.
Specs are denotational by nature.
A program could synthesize a program that adheres to the specs. But that is not what we understand by a compiler that generally has to preserve operational semantics.
I've heard the "if you hate vibe coding you must also hate compilers" strawman so many times, I'm sure it's been said by someone with a "name and reputation" at least once.