Old McDonald had a startup, iyo io o[4-mini].

It’s funny how just today in a completely unrelated context a generative ai enthusiast used an example of OpenAI getting sued by NYT as a reason why they wouldn’t commit some other malfeasance because they’d get caught if they did.

Its not about moats, it’s about open source community (whose code had been trained on) coming out with pitchforks. It has nothing to do with moats.

You are way overselling coding agents.

Re-creating some open source project with a similar function is literally the only way a coding agent can pretend to be a programmer.

I tried latest models for code and they are in fact capable of shitting out a thousand lines of working code at a time, which obviously can only be obtained via plagiarism since they are also incapable of writing the most trivial code for a novel situation. And the neat thing about plagiarism is that once you start you can keep going since there’s more of compatible code where it came from.

Yeah I’m thinking this one may be special cased, perhaps they wrote a generator of river crossing puzzles with corresponding conversion to “is_valid_state” or some such. I should see if I can get it to write something really ridiculous into “is_valid_state”.

Other thing is that in real life its like “I need to move 12 golf carts, one has low battery, I probably can’t tow more than 3 uphill, I can ask Bob to help but he will be grumpy…”, just a tremendous amount of information (most of it irrelevant) with tremendous^tremendous possible moves (most of them possible to eliminate by actual thinking).

Pre-LLM, I had to sit through one or two annual videos to the sense of “dont cut and paste from open source, better yet don’t even look at GPLd code you arent working on” and had to do a click test with questions like “is it ok if you rename all the variables yes no”. Ohh and I had to run a scanning tool as part of the release process.

I don’t think its the FSD they would worry about, but GPL especially v3. Nobody gives a shit if it steals some leetcode snippet, or cuts and pastes some calls to a stupid API.

But if you have a “coding agent” just replicating GPL code wholesale, thousands and thousands of lines, it would be very obvious. And not all companies ship shitcode. Apple is a premium product and ages old patched CVEs from open source cropping up in there wouldn’t be exactly premium.

That is not equivalent, though; other solutions to “can not be left unattended” exist; just ask Kristi Noem.

LLM snippets are so 2024. Coding agents, baby.

Yeah, that’s a great example.

The other thing is that unlike art, source code is already made to be consumed by a machine. It is not any more transformative to convert source code to equivalent source code, than it is to re-encode a video.

The only thing they do that is “transformative” is using source code not for compiling it but for defrauding the investors.

Other funny thing: it only became a fully automatic plagiarism machine when it claimed that it wrote the code (referring to itself by name which is a dead giveaway that the system prompt makes it do that).

I wonder if code is where they will ultimately get nailed to the wall for willful copyright infringement. Code is too brittle for their standard approach, “we sort of blurred a lot of works together so its ours now, transformative use, fuck you, prove that you don’t just blur other people’s work together, huh?”.

But also for a piece of code, you can very easily test if the code has the same “meaning” - you can implement a parser that converts code to an expression graph, and then compare that. Which makes it far easier to output code that is functionally identical to the code they are plagiarizing, but looks very different.

But also I estimate approximately 0% probability that the assholes working on that wouldn’t have banter between themselves about copyright laundering.

edit: Another thing is that since it can have no own conception of what “correct” behavior is for a piece of code being plagiarized, it would also plagiarize all the security exploits.

This hasn’t been a big problem for the industry, because only short snippets were being cut and pasted (how to make some stupid API call, etc), but with generative AI whole implementations are going to get plagiarized wholesale.

Unlike any other work, code comes with its own built in, essentially irremovable “watermark” in the form of security exploits. In several thousands lines of code, there would be enough “watermark” for identification.

Having worked in computer graphics myself, it is spot on that this shit is uncontrollable.

I think the reason is fundamental - if you could control it more you would put it too far from any of the training samples.

That being said video enhancements along the lines of applying this as a filter to 3d rendered CGI or another video, that could (to some extent) work. I think the perception of realism will fade as it gets more familiar - it is pretty bad at lighting, but in a new way.

Well, it did reach for “I double checked it, I’m totally sure now” language.

From the perspective of trying to convince the top brass that they are making good progress towards creating an artificial psychopath - not just an artificial human - it’s pretty good.

Still seems terminally AI pilled to me, an iteration or two later. “5 digit multiplication is borderline”, how is that useful?

I think there’s a combination of it being a pinnacle of billions and billions of dollars, and probably theirs firing people for slightest signs of AI skepticism. There’s another data point, “reasoning math & code” is released as stable by Google without anyone checking if it can do any kind of math.

edit: imagine that a calculator manufacturer in 1970s is so excited about microprocessors they release an advanced scientific calculator that can’t multiply two 6 digit numbers (while their earlier discrete component model could). Outside the crypto sphere, that sort of insanity is new.

Yeah, I’d also bet on the latter. They also added a fold-out button that shows you the code it wrote (folded by default), but you got to unfold it or notice that it is absent.

Oh and also for the benefit of our AI fanboys who can’t understand why we would expect something as mundane from this upcoming super-intelligence, as doing math, here’s why:

Also, I just noticed something really fucking funny:

(arrows are for the sake of people like llllll…)

lmao: they have fixed this issue, it seems to always run python now. Got to love how they just put this shit in production as “stable” Gemini 2.5 pro with that idiotic multiplication thing that everyone knows about, and expect what? to Eliza Effect people into marrying Gemini 2.5 pro?

there was a directive that if it were asked a math question that you can’t do in your brain or some very similar language it should forward it to the calculator module.

The craziest thing about leaked prompts is that they reveal the developers of these tools to be complete AI pilled morons. How in the fuck would it know if it can or can’t do it “in its brain” lol.

edit: and of course, simultaneously, their equally idiotic fanboys go “how stupid of you to expect it to use a calculating tool when it said it used a calculating tool” any time you have some concrete demonstration of it sucking ass, while simultaneously the same kind of people are lauding the genius of system prompts half of which are asking it to meta-reason.

Thing is, it has tool integration. Half of the time it uses python to calculate it. If it uses a tool, that means it writes a string that isn’t shown to the user, which runs the tool, and tool results are appended to the stream.

What is curious is that instead of request for precision causing it to use the tool (or just any request to do math), and then presence of the tool tokens causing it to claim that a tool was used, the requests for precision cause it to claim that a tool was used, directly.

Also, all of it is highly unnatural texts, so it is either coming from fine tuning or from training data contamination.

misinterpreted as deliberate lying by ai doomers.

I actually disagree. I think they correctly interpret it as deliberate lying, but they misattribute the intent to the LLM rather than to the company making it (and its employees).

edit: its like you are watching a TV and ads come on you say that a very very flat demon who lives in the TV is lying, because the bargain with the demon is that you get to watch entertaining content in response to having to listen to its lies. It’s fundamentally correct about lying, just not about the very flat demon.

Hmm, fair point, it could be training data contamination / model collapse.

It’s curious that it is a lot better at converting free form requests for accuracy, into assurances that it used a tool, than into actually using a tool.

And when it uses a tool, it has a bunch of fixed form tokens in the log. It’s a much more difficult language processing task to assure me that it used a tool conditionally on my free form, indirect implication that the result needs to be accurate, than to assure me it used a tool conditionally on actual tool use.

The human equivalent to this is “pathological lying”, not “bullshitting”. I think a good term for this is “lying sack of shit”, with the “sack of shit” specifying that “lying” makes no claim of any internal motivations or the like.

edit: also, testing it on 2.5 flash, it is quite curious: https://g.co/gemini/share/ea3f8b67370d . I did that sort of query several times and it follows the same pattern: it doesn’t use a calculator, it assures me the result is accurate, if asked again it uses a calculator, if asked if the numbers are equal it says they are not, if asked which one is correct it picks the last one and argues that the last one actually used a calculator. I hadn’t ever managed to get it to output a correct result and then follow up with an incorrect result.

edit: If i use the wording of “use an external calculator”, it gives a correct result, and then I can’t get it to produce an incorrect result to see if it just picks the last result as correct, or not.

I think this is lying without scare quotes, because it is a product of Google putting a lot more effort into trying to exploit Eliza effect to convince you that it is intelligent, than into actually making an useful tool. It, of course, doesn’t have any intent, but Google and its employees do.

Pretentious is a fine description of the writing style. Which actual humans fine tune.