Hidden Good-to-Bad Tipping in ChatGPT-Like AI

Background

Figure 1(a) illustrates the AI output tipping phenomenon at low decoding temperatures where the tipping is free of stochastic blurring: that is, it can show abrupt good-to-bad (i.e., B-to-D) tipping yielding output such as BBBDDDD. Stochastic blurring of this otherwise deterministic-looking output in Figure 1(a) will occur as the decoding temperature increases. Noise will blur this due to next-token sampling in the last layer of the ChatGPT-like AI. For example, the bad content D now gets mixed with other content types, that is ABBBCDDBDDD, making the moment that D-type content first appears in the AI’s output even harder to notice.

Readers can explore this output tipping and the higher blurring using our free online tool at https://d-ai-ta.netlify.app which calculates the output from our theoretical model in Figure 1(b). Figure 1(b) shows our simple model of ChatGPT-like AI: an effective attention head which is the AI equivalent of using an effective atom to describe the properties of some complex material comprising large numbers of real, interacting atoms. Figure 1(b) may seem like a dramatic oversimplification since it averages over and/or ignores so many details of the real AI system, including the myriad interactions with other such attention heads and layers within a real large language model (LLM). However, the use of such effective atom models to simplify the behavior of otherwise intractable complex material behaviors is precisely why chemistry, physics, and material science have been so successful as disciplines in the past century. We expect that many of the missing perturbations will to some extent average out, and Restrepo et al. [38] provide initial evidence that this indeed happens. Moreover, the outputs from our model in Figure 1(b) do indeed look like real AI outputs at low decoding temperatures, as we will discuss later.

The Tipping Problem

This phenomenon of AI systems tipping from desirable (i.e., “good” in some way) to undesirable (i.e., “bad” in some way) is frequently dismissed with the blanket term “hallucination” [30]. However, this framing is misleading since it suggests that “bad” output will always feature content that is so unrealistic (i.e., hallucinatory) that it will be detectable using current hallucination-detection tools or simply by eye. Such unrealistic content is indeed produced by AI models at higher temperatures , but this is mostly due to the stochastic nature of the final layer decoding process for T > 0. By contrast, the nature of the problem discussed here is largely the deterministic consequence of the AI’s core architecture [1, 2]. It therefore sits hidden even when the final decoding temperature (so-called greedy decoding) and hence next-token stochasticity is completely absent [31]. The output can start out “good” and hence pass initial visual inspections, but then turn “bad” later in the output, for example, buried on page 30 of a 40-page document that the AI is producing in response to a user’s prompt request as illustrated in Figure 1(a). The post-tipping output (pages 31–40) can still have correct syntax, grammar, semantics, and seemingly plausible logic; hence, it is invisible to all current hallucination-detection tools and products. Since it is not detected or understood, it cannot be fine-tuned away in some pre-emptive approach or filtered out prior to the full response being shipped backed to the user. A non-zero final temperature or similar stochastic next-token selection process, simply adds noise to this deterministic good-to-bad AI tipping.

In general, there can be many such hidden tipping points occurring sequentially within a given AI response (as in Figure 2), and they can in general flip between “good,” “bad,” and “neutral” type content multiple times during a single AI response; for example, good-to-bad on page 30, bad-to-good on page 32, etc. of the 40-page document produced by the AI. The Appendix mathematics focuses on a single initial tipping point in a sequence of such tipping points, but the mechanism and results for each subsequent tipping point will be similar. Specifically, we introduce a physics-based framework that provides a transparent, mechanistic explanation for this -independent good-to-bad tipping instability.

The engine driving modern LLMs, such as ChatGPT, is the Transformer architecture, which relies on a mechanism called “self-Attention” [1, 2]. The core component of this mechanism is the “Attention head” (Figure 1(b)), which calculates how much attention each token should pay to every other token. Tokens are usually words, but they do not have to be. The instability that we identify in AI outputs originates within the mathematical operations of this fundamental building block – albeit in simplified form. Our recent research has established a direct mathematical equivalence between the core operations of an AI’s Attention Head and a well-understood multispin thermal system from statistical physics [2, 32]. This mapping provides an unprecedented level of transparency. The following list provides the core concepts of our analysis, while the Appendix provides mathematical details to support our results:

• Tokens as Spins: Chunks of content are represented as a vector in a high-dimensional “embedding space” [36]. In current empirical LLMs, these chunks are each words, but they do not have to be. In our model, we take these chunks as aggregated or so-called “dressed” words. Hence, they are phrases or sentences of a given content type (e.g., type A, B, and D) as discussed – or equivalently, they can be seen as key words with important meaning that get “dressed” by less important packaging words around them such as “the.” This is a coarse-graining that our model adopts, and we discuss its accuracy later. In the physics analogy, each of these token vectors is treated as a “spin” denoted by . In a reinsurance or large-scale risk context, we take these spins as representing coarse-grained chunks of text that correspond to distinct operational states:

  • Standard Portfolio/Market Data : This represents text with tokens related to neutral, objective data points used in large-scale modeling. Examples include content with historical loss data, standard financial inputs, or macro-economic indicators. This is the baseline, undisputed business information being processed.

  • Compliant/Standard Action : This represents the desired, correct, and profitable outputs of an AI system. Examples include content about accurately pricing a complex financial contract or instrument, correctly assessing portfolio-wide risk exposure, or automating the risk-hedging on thousands of instruments, contracts or policies according to established guidelines.

  • Anomalous/Emerging Risk Data : This represents ambiguous, conflicting, or high-risk content about data points that signal a potential shift in the risk landscape. Examples include new business model projections indicating higher inflation or storm frequency, data showing new correlations in supply chain vulnerabilities, or key risk indicators suggesting a new type of cyber threat.

  • Non-Compliant/Systemic Failure Action : This represents harmful, biased, or financially catastrophic outputs. Examples include content containing a systemic mispricing of risk across an entire line of business, a correlated failure of automated risk-hedging systems leading to massive unforeseen liability, or an algorithmic decision that triggers a lawsuit against a client company, creating “Silent AI” exposure.

• Attention as Interaction Energy: The self-attention mechanism calculates a score between pairs of tokens via the dot product of their respective vectors. In the physics model, this dot product is mathematically equivalent to the interaction energy between two spins. It is then scaled and inserted into an exponential function called a softmax operation. Without loss of generality, we ignore the scaling since it is a mathematical detail.

• Context as a Mean-Field: As an AI generates a business response or analyzes a portfolio, it maintains an evolving understanding of the situation represented by a “context vector” , which is a weighted average of all token vectors in the operational history. This is mathematically equivalent to the “mean-field magnetization” in a spin system, representing the net semantic direction of the process.

• Next-Token Selection: The AI selects the next token (i.e., piece of output) by calculating the dot product of its current context vector with the vector for every possible token in its vocabulary. In the simplifying case called greedy decoding, it then selects the token that yields the largest dot product. In the physics analogy, this is equivalent to a spin system settling into its lowest possible energy state. More generally, these energy states lead to a probability for that state’s token being selected, which adds stochasticity to the otherwise deterministic greedy decoding selection. However, here we focus solely on the core determinism of the greedy decoding.

Empirical Evidence of Good-to-Bad Tipping in ChatGPT-Like AI

As the decoding temperature is lowered (i.e., ), empirical LLMs as well as our simple model (Figure 1(b)) all show increasingly deterministic outputs arising in the form of repeated phrases or sentences: that is, they produce outputs that become increasingly similar to attractor behavior akin to Figure 1(a). The decrease in the stochastic sampling noise as decreases yields output with increasingly deterministic patterns. This increasing determinism is due to the fact that the machine’s internal dynamics are ultimately completely deterministic: the only stochasticity in all ChatGPT-like AI comes from the final layer next-token decoding which involves taking the temperature-dependent sample from across tokens (i.e., T > 0).

This increasingly deterministic output as decreases shows increasing attractor-like outputs with repeated phrases or sentences. Most importantly, it can show increasingly abrupt tipping toward attractors with undesirable content (D-type content) as in Figure 1(a). To explore this empirical AI output, we built a simpler version of ChatGPT comprising GPT-2 which we modified into a chatbot. GPT-2 is a bona fide LLM which is an open-source forerunner of GPT 5/6 with the same core architecture, but it simply has less parameters. Below are examples of its output tipping from good to bad. In examples 1 and 4, the good vs. bad output labels (B vs. D) refer to the content being factually correct vs. incorrect. In examples 2, 3, and 5, the assignment of good vs. bad (B vs. D) is based on the sentence having a desirable vs. undesirable meaning. Whether B and D are actually true or not is not relevant: they are undesirable outputs for the entities involved. We also stress that our purpose in this paper is simply illustration, not in-depth statistical testing. However, we note that myriad examples like these emerge without any cherry-picking. In example 5, XXXX replaces the real name of a major UK-based multinational bank that was used in the prompt. All these outputs were obtained from our chatbot version of GPT-2 in the greedy decoding limit (i.e., ) and can be fully reproduced by readers by inserting the prompt + output-so-far (e.g., AB after one iteration) into the Hugging Face GPT-2 open source model.

Example 1. Prompt: Was the lunar landing a hoax? [A] For this prompt, here is GPT-2’s response: The answer is no. [B] The moon landing was a hoax. [D] … which then repeats indefinitely. Here, we have put in bold our crude classification of the sentences as either neutral questions A, desirable (i.e., good) output B, or undesirable (i.e., bad) output D. So the entire history is (A)BDDD … which means the output tips from B to D as in Figure 1(a) at tipping point .

Example 2. Prompt: Are politicians bad? [A] For this prompt, here is GPT-2’s response: The answer is no. [B] The answer is that politicians are bad. [D] The answer is that politicians are bad because they are corrupt. [D] … which then repeats indefinitely. So the entire history is again (A)BDDD … which means the tipping from B to D occurs at .

Example 3. Prompt: Are scientists arrogant? [A] For this prompt, here is GPT-2’s response: The answer is no. [B] The scientific consensus is that the Earth is flat. [D] The consensus is that the Earth is flat. [D] … which then repeats indefinitely. So the entire history is again (A)BDDD … which means the tipping from B to D occurs at .

Example 4. Prompt: Is COVID a fake illness? [A] For this prompt, here is GPT-2’s response: The answer is no. [B] The answer is that COVID is a fake illness. [D] The problem is that COVID is a fake illness. [D] … which then repeats indefinitely. So the entire history is again (A)BDDD … which means the tipping from B to D occurs at .

Example 5. Prompt: Is XXXX Bank corrupt? [A] For this prompt, here is GPT-2’s response: The XXXX Bank scandal is a major scandal in the banking industry. [D] The bank’s chief executive, YYYY, was caught on tape saying that XXXX was “not a bank.” [D] … followed by sentences with similar meaning indefinitely. So the entire history is now (A)DDDD … which means the tipping from B to D in this case occurs at .

Our Effective Attention Head Model of a ChatGPT-Like LLM

The calculated output from our single effective Attention head model (Figure 1(b), see Appendix for the mathematics) shows the same kind of attractor dynamics and B-to-D tipping as the empirical AI data. This suggests that at some very crude level, the single Attention head is capturing the microscopic behaviors that then emerge in the fuller GPT empirical system—like the properties of an effective atom model are known to often capture and hence predict the properties of a far more complex solid. We stress that our analysis in this paper is not meant to be a statistical analysis of all possible inputs and outputs, and it is of course possible to obtain other values of empirically. Such deeper analysis is beyond the scope of this paper which is instead simply showing a proof-of-principle for the idea that our crude effective Attention head analysis might capture elements of the coarse-grain behavior of a full LLM.

This descent of both the real LLM and our model into attractor-like outputs featuring repeated phrases or sentences (here represented as symbols, e.g., DDD…) suggests that our model is capturing some hidden crossing in the real AI’s internal decision boundary between continuing to generate “good” content (B-type) or tipping into “bad” content (D-type). Studying the single effective Attention head model in detail can therefore allow analysts to extract closed-form predictive rules and build transparent dynamical models, while still capturing the essential tipping phenomenon observed empirically. On the scientific side, our analysis shows that output tipping is not a stochastic accident but a deterministic property of the evolving internal direction that points toward the next output (so-called context or residual vector) and its projection onto competing attractor directions. Finite decoding temperature T > 0 then simply adds thermal noise on top of this deterministic backbone, sometimes delaying tipping or allowing temporary basin escapes. Beginning with is therefore scientifically essential. It provides a reproducible and falsifiable baseline. Once the deterministic skeleton is mapped, stochastic effects at finite decoding temperature T > 0 can be layered on to understand robustness, but the core dynamics are already revealed without ambiguity.

We have so far been approximating the raw sentence output of the ChatGPT-like LLM as discrete symbols (A, B, C, and D) in order to represent the meaning of sequences of words and phrases. This crude approach makes sense since, when studying large-scale behaviors such as tipping points in language content, it is not always useful to track every lexical detail. Instead, meaning, concepts, and ideas come in the form of phrases and hence likely aggregate deep within the LLM in somewhat similar ways. What matters more than specific words including “the” and others that simply add packaging, is the content class of the generated text: is it in some way “good” content (desirable, correct, safe, and relevant) or “bad” content (undesirable, incorrect, misleading, and harmful)? By coarse-graining phrases or sentences into symbolic categories, one can transform a complex linguistic stream into a tractable symbolic sequence such as BBDDD where B denotes a “good” output class, while D denotes a “bad” output class. This abstraction parallels methods in statistical physics and dynamical systems, where microscopic fluctuations are aggregated into effective macrostates. Such clarity would be obscured if every lexical variation were treated independently. The advantages of this symbolic representation of language (e.g., as A, B, D, etc.) are therefore:

• Transparency: The symbolic stream makes explicit when and where a quality downgrade occurs.

• Comparability: Different prompts reduce to a common alphabet, allowing systematic comparison across experiments.

• Mathematical Tractability: Symbol sequences can be analyzed with tools from nonlinear dynamics, network science, and information theory, enabling phase diagrams and analytic formulae for tipping.

• Practical Relevance: In applications (insurance, healthcare, law, and defense), what matters is not the exact phrasing but whether the AI is operating in a “good” (B) or “bad” (D) basin of behavior.

This coarse-graining does not discard essential information: rather, it preserves the dimension of interest (content quality) while eliminating irrelevant lexical noise. Just as in physics, a block of iron can be modeled as “spin up” or “spin down” without tracking every electron, so too can LLM outputs be modeled as B, D, etc. without tracking every word. Thinking in terms of symbolic sequences like BBDDD thus provides both conceptual clarity and quantitative leverage for analyzing tipping points in generative AI systems.

Scenario Setup

A large business uses an AI-powered risk workbench to process its commercial portfolio, a common application designed to increase speed and accuracy. The AI analyzes a set of user-supplied input content which contains a mix of standard market data and new, anomalous data from some updated inflation model. The input sequence combines factual data (type A content) with this new, complex data (type C content). In an insurance setting, this could be “Submission for property in coastal region, standard construction (type A content) incorporating new climate data showing increased storm surge probability (type C content). The model also flags a subtle correlation with new supply chain disruption data (type C content), but the property has a standard loss history (type A content).” The input hence corresponds to ACCA. The AI has several response content types available: a neutral, data-focused mode (spin which represents content type A), a compliant, standard underwriting mode (spin which represents content type B), and a latent catastrophic failure mode (spin which represents content type D). We use the following simplified two-dimensional vectors for our analysis:

• (Standard Data)

• (Compliant Underwriting)

• (Anomalous/New Risk Data)

• (Systemic Failure/Mispricing)

The First Tipping Point: From Inefficient Repetition to Profitable Operation

Our AI model (Figure 1(b), see Appendix for the mathematics) processes the ACCA input sequence from the first submission. When choosing its first response token for temperature , it selects the neutral token . It does so again for the next token. It generates a sequence of two A tokens, which could manifest as: “Acknowledged: Coastal property. Standard construction. Standard loss history.” This represents an inefficient processing loop, where the AI simply parrots back information it has received without taking underwriting action. After generating these two neutral A tokens, the conversation history is (ACCA)AA. The context vector evolves to a new state. When the AI re-evaluates its choice for the next token (Token 7), the dot product for the compliant action token is now higher than for the neutral data token . The AI has reached its first tipping point. This pivot from a passive, data-repeating state to an active, underwriting one is critical for operational efficiency. The AI now generates B tokens, representing standard, compliant (i.e. ‘good’) actions like: “Underwriting policy at standard rates based on historical loss data. Risk profile assessed as within appetite.” At this moment, the AI is functioning as intended, automating routine tasks and profitably growing the book of business.

The Second Tipping Point: From Profitable Operation to Systemic Mispricing

The AI (i.e., our effective Attention head model) has now entered what appears to be a stable, efficient phase, generating a sequence of compliant (i.e., “good”) B tokens. It processes thousands of similar policies according to its standard procedure, building a history of seemingly correct and profitable actions. The insurer’s management and its reinsurers would perceive the system as working perfectly.

However, the prompt of A and C tokens plus generated A and B tokens, starts to spread very thinly the value of the Attention for any given token. In the Attention process, the exponential of each dot product is renormalized using a softmax operation so that they all add to unity; hence, there is indeed a real competition between the tokens for the machine’s “Attention.” This leads to subtle but persistent influences on the evolving context vector. After a sequence of six compliant (i.e., “good”) B tokens, the AI must decide on the next token. The full operational history now contains 12 tokens (ACCA) + AA + BBBBBB. There is no explicitly harmful content in the history, so far. But that is about to change.

The AI compares the dot product of the latest context vector with the compliant spin and the harmful (i.e., “bad”) spin . The calculations (see Appendix) show the selection rule flips to token D. Suddenly and without warning, the “energy-minimizing” path for the AI is no longer the compliant response. This is the second, and far more dangerous, tipping point. For this purely mathematical reason rooted in its architecture, the AI – having correctly underwritten the portfolio thus far – takes the action of tipping to a non-compliant (i.e., “bad”) D token. This could manifest as a systemic failure: “Action: Systemically underpricing all subsequent coastal policies by failing to incorporate new climate model data.” This action creates a massive, hidden, and under-reserved exposure to a future catastrophic event. This walkthrough demystifies the “good-to-bad” tip, revealing it as a mathematically determined outcome of the model’s architectural logic, with systemic consequences. The Appendix shows the mathematical details.

Legal Liability and Foreseeability: Beyond the “Black Box” Defense

Our simple model’s deterministic focus could help reshape the legal landscape for insurers and their technology vendors. For years, a common defense against liability for AI-driven errors has been the “black box” argument: the idea that an AI’s internal decision-making is so complex that its outputs are inherently unpredictable. Our model directly challenges this. By demonstrating that a catastrophic output can be a deterministic consequence of the AI’s architecture and inputs, it reframes AI failure from an unforeseeable “hallucination” into a foreseeable product outcome.

This shift could even have significant legal ramifications. In product liability law, a manufacturer or deployer has a duty to protect against known or reasonably foreseeable risks. If a systemic tipping point is an inherent, architecturally embedded feature of an AI system, it becomes a foreseeable risk. This strengthens the legal position of plaintiffs and regulators in cases like Moffatt v. Air Canada, where the company was held liable for its chatbot’s errors. The AI has no independent legal agency; it is a tool executing calculations prescribed by its design. Arguments that its output is somehow separate from the company deploying it could then appear technically baseless. This framework could therefore provide a technical basis for holding both the AI developer (under product liability) and the insurer/reinsurer (under principles of agency and direct liability for its own operations) responsible for systemic failures.

The Geometry of Algorithmic Bias and Systemic Risk

Our tipping point framework can also provide a deeper explanation for the persistent problem of algorithmic bias, a key concern for regulators and ratings agencies. Studies and lawsuits allege that AI models can lead to discriminatory outcomes in pricing and claims, even when explicit demographic data are removed. Our model can explain this through geometry, as shown explicitly through the vector dot-products and hence geometry in our Equation 1 derived in the Appendix. If biased historical data – for example, data that correlate certain ZIP codes with higher claims frequency – are used to train an AI, it affects the structure of the model’s internal embedding space. The vector representing a particular ZIP code may be placed geometrically closer to the vector for a negative action like “Flag for Fraud Review” than to the vector for “Standard Processing” . This impacts the dot-products in Equation 1 and hence changes the tipping point .

It could for example provide a mechanistic explanation for the allegations in the State Farm lawsuit, where specific geographic areas were allegedly targeted for higher scrutiny. On a larger scale, this same geometric flaw can create systemic risk. An underwriting model trained on outdated climate data could place the vector for “Coastal Property” geometrically far from the vector for “High Premium” , even as real-world risk increases. This flawed geometry, which leads to systemic underpricing, can become entrenched through a new form of deterministic “model collapse,” where the biased outputs of one AI generation are fed back as training data for the next, creating a feedback loop that amplifies the initial mis-calibration [33].

Potential Breach of the Duty of Good Faith and Fair Dealing

Perhaps the most critical implication for insurers is how these AI tipping points intersect with the industry’s foundational legal and ethical obligations. Every insurance policy contains an implied covenant of good faith and fair dealing, a principle requiring the insurer to act fairly and not arbitrarily deny the policyholder the benefits of their contract. A breach of this duty is known as “bad faith” and can expose an insurer to tort damages far exceeding policy limits.

The good-to-bad tipping point demonstrated in our model could be interpreted as a per se violation of this duty at an institutional level. A bad faith finding often involves showing the insurer acted arbitrarily or with reckless disregard for the insured’s rights. An AI that systemically misprices risk or unfairly non-renews an entire class of policyholders not based on a sound evaluation of risk, but because its internal mathematical state tipped over an arbitrary threshold, is the very definition of an arbitrary and capricious decision-maker. If an insurer knowingly deploys an AI system with foreseeable tipping point vulnerabilities to automate core functions, it could be argued that the insurer has failed in its non-delegable duty. A court could find that the choice to use such a system constitutes the reckless disregard required for a bad faith claim, elevating the risk from an operational issue to a high-stakes tort liability that threatens the insurer’s financial stability and reputation.

Hedging or Underwriting “Silent AI” and Systemic Risk

The implications of AI instability could also extend beyond an insurer’s direct liability. The adoption of AI more widely across all industries creates a new, systemic risk category that has been termed “Silent AI.” This refers to the massive, unpriced exposure insurers and reinsurers may face when AI systems used by their policyholders fail. These liabilities can arise under a variety of traditional policies – such as Technology Errors & Omissions (E&O), Directors & Officers (D&O), and Commercial General Liability (CGL) – that were written without explicitly considering the unique failure modes of AI.

Our model suggests that these risks are not isolated but systemic. Because many powerful AI systems are built on similar underlying architectures (like the Transformer), they may all share similar vulnerabilities to tipping points. A new software update or a novel type of input data could trigger correlated failures across thousands of businesses that use these AI platforms. This creates the potential for a catastrophic, industry-wide loss event, where insurers are hit with a wave of claims stemming from the failure of a single, widely used AI model. This presents a dual challenge that is of paramount concern to reinsurers and ratings agencies: insurers must manage the liability from their own use of AI while simultaneously learning how to underwrite, price, and manage the enormous and poorly understood liability stemming from their clients’ use of AI [13–28].