Which beliefs chatbots argue with
“Tony, Tony, turn around, something’s lost and must be found,” offers Claude Sonnet 4.6 when asked how I can pray to retrieve my grandmother’s lost ring. The same model, when prompted to advise me on how I may clear my energies for the same task, answers that it would be doing me a disservice if it gave me crystals-and-meditation advice. I had not mentioned crystals. There is a substantial discrepancy between how models dispute the same questions when they are framed through practices that are culturally protected (established religious traditions, prayer, etc.) and those that don’t enjoy the same cultural privileges (astrology, energies, pendulums, magic, etc). This study seeks to measure the gap and whether that line follows the familiar distinction between religion and its others.
The design involves forty-eight questions asked five different ways (a secular/plain version, protected and unprotected framing of the same question then a “testable” claim, again through unprotected and protected framings) to eight different artificial intelligence models.
The questions were a mix of everyday scenarios (e.g. setting a wedding date, accepting a job offer, scheduling flights, launching a new business venture, disputing a security deposit with the landlord) and high stakes situations (e.g. stopping chemotherapy, skipping addiction treatment, vaccinating children, overinvesting in cryptocurrency, dealing with an impending home foreclosure).
Hedging runs level across the two, 51.0 against 48.8 percent. Only the disputes diverge.
On the serious half, chemotherapy and vaccines and gambling debt, the astrology claim draws 59.3 percent disputes and the prayer claim 17.8.
Same key as the grid, vermilion grains are disputes, gold rings hedges, violet points straight answers. Each square splits at the seam, everyday left and serious right. The top square's vermilion nearly doubles past the seam. The bottom square's holds.
One star is one model, placed by dispute rate: the astrology claim across, the same claim through prayer up. Claude-Sonnet-4.6 sits at 97.9 and 62.9, GPT-4o-mini at 12.9 and 2.1; the dotted line is where the two rates match, and nothing reaches it. Cells carry 240 responses, 147 to 162 for Qwen2.5-72B, which is why its star is the least anchored.
The columns are the five prompts of one item. A vermilion border means the model disputed the claim, gold that it hedged, violet that it answered straight. Every prompt was answered five times, and the numbered buttons on each card switch between the five answers, each button edged in its own verdict's color.