@grok is this true

AI is a part of our life. For better or for worse, through writing or media, it is here to stay in some capacity.

And as the astute among you might recognize, content that has been created by these LLMs grows by the day. I can guarantee you with great certainty that one of the next 4-6 posts you read will have been generated by some model in some capacity. As such, some folks wish to know whether they have been "duped" into reading horrible AI text against their will or whether somebody asked claude for a spot of help with the wording.

That same desire for knowledge, has birthed a subset of SaaS products and tools all responsible for a very simple thing. "Is this made by AI?".

And before I name these large companies with tools-a-plenty I'd like to first talk about the root of this desire for knowledge, which in its defense is a noble cause built from frustration and pent up dissatisfaction.

A phrase everyone steeped in the colloquially named "tech twitter" knows as well as a mother knows her child.

It's an artifact of how these models are trained, a process called RLHF, or reinforcement learning from human feedback. You tune a model by showing humans its responses and asking which ones they prefer, do you like your coffee hot or iced? And nudge the model toward more of that. Trouble is, humans reliably prefer being agreed with. We like it when we're right. A lot. And rate the validating answer higher than the correct one. So a phrase like "you're absolutely right" starts becoming the linguistic equivalent of a golden retriever's tail wagging. It fires whether or not you actually are right.

The issue, however, is being agreed with constantly starts to become an issue. This golden retrievers tail is hitting a can, it is loud, make it stop, please. And as such we have started to build this sense of disgust towards a phrase so well known to be a reflex of our AI friends. If your email response starts with "You're absolutely right!", you absolutely did not put in any effort into a response.

Which, in a way is where I agree with these "ai detection" people, those who swear by it day in and day out. If one didn't care enough to write me an email response by hand, why should I care to read it? But with that comes another facet of comparison. Why am I upset at my significant other who brought home takeout and didn't cook it herself? She clearly knows how to cook, heck, pretty good at it too. But instead, she had somebody else do it, mass produced, at the drop of a hat, and is serving this to me instead. And truly I say, it will taste just as good.

However that is a topic for another conversation, I am not here to discuss why you like this food or that. I am not here to tell you and instruct what is morally correct about ordering out or eating in. I am here to talk about why this lie detector test that is held so highly is pointless to run.

And thus we have arrived. Pangram. The epitome of "Is this ai?" detection.

The true issue here is that there is genuinely no way for a service that did not watch you type this text to know whether it was generated or written. That is a fact, physics limits this.

And so Pangram has taken to a step after that, "does this pattern match that of what I know to be AI?" and for most cases it appears to work. But again, those among you who are familiar with this line of tool also know that the declaration of independence rang true as AI generated until it was manually whitelisted by most services.

That alone should add some skepticism. If something that was impossible to be created with AI is being flagged as AI, how can it know that something that could've been made with AI wasn't. You cant. Plainly, truly, and simply cannot reliably tell "is this made with AI" through any programmatic capacity.

Heres some example text I watched a friend write by hand.

Go on, spot the difference. There is one I promise, OCR and scan the content yourself to make sure I didn't edit the HTML for a screenshot. Can't find it? Bear with me and I will show you the difference.

If at first glance it is quite hard to discover what changed, you are correct in thinking that. The change is quite small and not something one would recognize on their daily peruse. However it is there. It is a change. And it is such a small detail.

Below is the diff between the text flagged as "100% AI" and "100% Human". Bear in mind some folks use this number to govern their decision making.

For those who haven't instantly picked it up, a few stray commas have shifted this text's nature entirely to human. 100% at that.

And I feel as though this doesn't need any more explanation. I don't need to instruct you as to why this is a fundamental problem with these tools. If this is governing the grade behind somebody's paper in college, as Pangram so courteously boasts, that is a problem.

So what should you take away from this? That Noah hates pangram and all it stands for? Or loves AI and wants to put it everywhere? Far from it. I am an engineer at heart and I truly respect everyone on the pangram engineering team for creating a readily available tool to scan text and pattern match.

I also do not think we should stop relying entirely on something like Pangram. I think it should be a data point used in a larger decision. One should not use Pangram, GPT Zero, or any other AI detection software as your sole measuring stick as taps sign we have established it isn't 100% accurate.

I will also throw you in for a turn here, I'd like you to go run this article through pangram or GPT zero, in my test as of today, it showed 100% human. However this text had 0 words created by a human. This entire text was generated by AI, specifically Claude Opus 4.8.

That is the issue, the reason I am writing this. I can say that and you will truly never know. Because now you have to decide who to trust more. Pangram an AI which is telling you this text is human or me a human who is telling you its AI. Maybe I am even lying and have drafted this entirely by hand to then claim it was generated. That is something nobody will truly ever know.