@cite{sikos-etal-2021}

Sikos, L., Venhuizen, N. J., Drenhaus, H. & Crocker, M. W. (2021). Reevaluating pragmatic reasoning in language games. PLOS ONE 16(3): e0248388.

Core Contribution

Replicates @cite{frank-goodman-2012} reference games and tests whether RSA's recursive reasoning (S1→L1) adds predictive value beyond a simpler baseline model that uses only the prior and literal semantics (= L0).

Three experiments with increasing pragmatic demands:

• Experiment 1 (FG2012 replication): 3-object contexts with color, shape, texture features. Baseline r = 0.988 vs RSA r = 0.992.
• Experiment 2 (extended): 4-object contexts. Baseline r = 0.990 vs RSA r = 0.992.
• Experiment 3 (critical test): Contexts specifically designed to be "pragmatically informative" — where L0 and L1 make different predictions. Baseline r = 0.77 vs RSA r = 0.82 (non-significant difference).

Key Arguments

1. Prior-driven variance dominates. In Experiments 1–2, most of the correlation between model and data is driven by object priors and literal semantics, not pragmatic reasoning. Trivially true items (where L0 = L1) inflate the correlation.

2. Methodology critique. Correlation-based evaluation across all items conflates two sources of variance: (a) prior-driven (which any model with the right priors gets right) and (b) pragmatic (where L0 and L1 differ). Removing trivially-predicted items collapses RSA's advantage.

3. Pragmatically informative contexts (Experiment 3). Even in contexts designed to maximize the L0/L1 difference, RSA does not significantly outperform the baseline.

4. Typicality priors matter. The paper uses empirically-measured typicality priors (not uniform), which do substantial predictive work independent of pragmatic reasoning.

Relationship to RSA

The baseline model is, mathematically, RSA's own L0 (literal listener with priors). Both sides agree on this. The critique is that the additional layers of recursive reasoning (S1, L1) don't add empirical value — the first step of RSA may be all that's needed.

Context Types

Sikos et al. classify reference game contexts by how much pragmatic reasoning they require. This taxonomy is central to their argument: FG2012's stimuli are dominated by trivial contexts.

inductive SikosEtAl2021.ContextType : Type

Classification of reference game contexts by pragmatic demands.

• trivial : ContextType

  Only one object matches the utterance. L0 = L1 trivially.

• pragSolvable : ContextType

  Multiple objects match, but pragmatic reasoning can break the tie. L0 ≠ L1: this is where RSA should add value.

• pragReducible : ContextType

  Multiple objects match; pragmatic reasoning helps but cannot fully disambiguate (e.g., symmetry among speakers).

• ambiguous : ContextType

  Multiple objects match and pragmatic reasoning cannot help. L0 ≈ L1 even with full RSA.

@[implicit_reducible]

instance SikosEtAl2021.instDecidableEqContextType : DecidableEq ContextType

Equations

• SikosEtAl2021.instDecidableEqContextType x✝ y✝ = if h : x✝.ctorIdx = y✝.ctorIdx then isTrue ⋯ else isFalse ⋯

@[implicit_reducible]

instance SikosEtAl2021.instReprContextType : Repr ContextType

Equations

• SikosEtAl2021.instReprContextType = { reprPrec := SikosEtAl2021.instReprContextType.repr }

def SikosEtAl2021.instReprContextType.repr : ContextType → Nat → Std.Format

Equations

• One or more equations did not get rendered due to their size.

Model Fit Data

Correlation coefficients for the two competing models across experiments. The key comparison: baseline (= L0 with priors) vs full RSA (L1).

structure SikosEtAl2021.ModelFit : Type

Model fit for one experiment, comparing baseline and RSA correlations. Correlations stored as thousandths (e.g., 988 = r = 0.988).

• experiment : Nat

  Experiment number (1, 2, or 3)

• description : String

  Brief description of the experiment

• nItems : Nat

  Number of unique context–utterance items

• baselineR_thou : Nat

  Pearson r × 1000: baseline model (prior × literal semantics = L0)

• rsaR_thou : Nat

  Pearson r × 1000: full RSA model (L1)

def SikosEtAl2021.instReprModelFit.repr : ModelFit → Nat → Std.Format

Equations

• One or more equations did not get rendered due to their size.

@[implicit_reducible]

instance SikosEtAl2021.instReprModelFit : Repr ModelFit

Equations

• SikosEtAl2021.instReprModelFit = { reprPrec := SikosEtAl2021.instReprModelFit.repr }

def SikosEtAl2021.exp1 : ModelFit

Experiment 1: Replication of FG2012. 3-object contexts. Both models fit almost identically (r = 0.988 vs 0.992).

Equations

• SikosEtAl2021.exp1 = { experiment := 1, description := "FG2012 replication, 3-object contexts", nItems := 54, baselineR_thou := 988, rsaR_thou := 992 }

def SikosEtAl2021.exp2 : ModelFit

Experiment 2: Extended to 4-object contexts. Still baseline ≈ RSA (r = 0.990 vs 0.992).

Equations

• SikosEtAl2021.exp2 = { experiment := 2, description := "Extended contexts, 4-object contexts", nItems := 72, baselineR_thou := 990, rsaR_thou := 992 }

def SikosEtAl2021.exp3 : ModelFit

Experiment 3: Pragmatically informative contexts designed to maximize L0/L1 divergence. RSA's advantage is non-significant (r = 0.77 vs 0.82). This is the critical test of the critique.

Equations

• SikosEtAl2021.exp3 = { experiment := 3, description := "Pragmatically informative contexts (critical test)", nItems := 48, baselineR_thou := 770, rsaR_thou := 820 }

def SikosEtAl2021.allExperiments : List ModelFit

All three experiments.

Equations

• SikosEtAl2021.allExperiments = [SikosEtAl2021.exp1, SikosEtAl2021.exp2, SikosEtAl2021.exp3]

Key Empirical Findings

theorem SikosEtAl2021.exp1_baseline_near_rsa : exp1.rsaR_thou - exp1.baselineR_thou ≤ 10

In Experiment 1, the baseline fits nearly as well as RSA (difference is only 4 thousandths of a correlation point).

theorem SikosEtAl2021.exp3_small_difference : exp3.rsaR_thou - exp3.baselineR_thou ≤ 100

In Experiment 3 (the critical test), the difference between models is 50 thousandths — small and non-significant.

theorem SikosEtAl2021.rsa_never_dominant : (allExperiments.all fun (mf : ModelFit) => decide (mf.rsaR_thou - mf.baselineR_thou < 100)) = true

RSA never dramatically outperforms the baseline in any experiment (gap < 100 thousandths = 0.100 correlation points in all cases).

Context Composition

Sikos et al. show that FG2012's stimuli are dominated by trivially-predicted items, which inflate correlations for any model with the right priors.

def SikosEtAl2021.trivialItemProportion_exp1 : Nat

Proportion of items in FG2012 that are trivially predicted. Stored as tenths of percent (780 = 78.0%). The exact value depends on the counting method; the paper reports that the majority of items in Experiments 1–2 are trivially predicted.

Equations

• SikosEtAl2021.trivialItemProportion_exp1 = 780

Competing Interpretations

inductive SikosEtAl2021.Interpretation : Type

Two interpretations of the finding that baseline ≈ RSA.

• rsaUnnecessary : Interpretation

  RSA's recursive reasoning is empirically unnecessary — the literal listener with priors suffices. The additional S1→L1 computation adds no predictive value. (Sikos et al.'s interpretation)

• baselineIsL0 : Interpretation

  RSA's L0 IS the baseline model, so high baseline fit is consistent with RSA. The question is whether L1 adds value in contexts where L0 ≠ L1. Sikos et al.'s Experiment 3 suggests it may not, though the test has limited statistical power. (Structural observation)

@[implicit_reducible]

instance SikosEtAl2021.instDecidableEqInterpretation : DecidableEq Interpretation

Equations

• SikosEtAl2021.instDecidableEqInterpretation x✝ y✝ = if h : x✝.ctorIdx = y✝.ctorIdx then isTrue ⋯ else isFalse ⋯

@[implicit_reducible]

instance SikosEtAl2021.instReprInterpretation : Repr Interpretation

Equations

• SikosEtAl2021.instReprInterpretation = { reprPrec := SikosEtAl2021.instReprInterpretation.repr }

def SikosEtAl2021.instReprInterpretation.repr : Interpretation → Nat → Std.Format

Equations

• One or more equations did not get rendered due to their size.

Structural relationships between models

1. The baseline model (prior × literal semantics) IS RSA's L0.
2. In trivial contexts (unique referent), L1 = L0.
3. In pragmatically solvable contexts, L1 ≠ L0 -- RSA's recursive reasoning makes different predictions.

These are mathematical facts about the models, not empirical claims.

What this does NOT show: That RSA is empirically vindicated. Sikos et al.'s Experiment 3 tested contexts specifically designed to be pragmatically solvable (where L0 ≠ L1), and RSA still did not significantly outperform the baseline.

inductive SikosEtAl2021.Color : Type

Colors used in the experiments.

• blue : Color
• green : Color
• red : Color

@[implicit_reducible]

instance SikosEtAl2021.instDecidableEqColor : DecidableEq Color

Equations

• SikosEtAl2021.instDecidableEqColor x✝ y✝ = if h : x✝.ctorIdx = y✝.ctorIdx then isTrue ⋯ else isFalse ⋯

def SikosEtAl2021.instReprColor.repr : Color → Nat → Std.Format

Equations

• SikosEtAl2021.instReprColor.repr SikosEtAl2021.Color.blue prec✝ = Repr.addAppParen (Std.Format.nest (if prec✝ ≥ 1024 then 1 else 2) (Std.Format.text "SikosEtAl2021.Color.blue")).group prec✝
• SikosEtAl2021.instReprColor.repr SikosEtAl2021.Color.green prec✝ = Repr.addAppParen (Std.Format.nest (if prec✝ ≥ 1024 then 1 else 2) (Std.Format.text "SikosEtAl2021.Color.green")).group prec✝
• SikosEtAl2021.instReprColor.repr SikosEtAl2021.Color.red prec✝ = Repr.addAppParen (Std.Format.nest (if prec✝ ≥ 1024 then 1 else 2) (Std.Format.text "SikosEtAl2021.Color.red")).group prec✝

@[implicit_reducible]

instance SikosEtAl2021.instReprColor : Repr Color

Equations

• SikosEtAl2021.instReprColor = { reprPrec := SikosEtAl2021.instReprColor.repr }

inductive SikosEtAl2021.Shape : Type

Shapes used in the experiments.

• square : Shape
• circle : Shape
• triangle : Shape

@[implicit_reducible]

instance SikosEtAl2021.instDecidableEqShape : DecidableEq Shape

Equations

• SikosEtAl2021.instDecidableEqShape x✝ y✝ = if h : x✝.ctorIdx = y✝.ctorIdx then isTrue ⋯ else isFalse ⋯

def SikosEtAl2021.instReprShape.repr : Shape → Nat → Std.Format

Equations

• One or more equations did not get rendered due to their size.
• SikosEtAl2021.instReprShape.repr SikosEtAl2021.Shape.square prec✝ = Repr.addAppParen (Std.Format.nest (if prec✝ ≥ 1024 then 1 else 2) (Std.Format.text "SikosEtAl2021.Shape.square")).group prec✝
• SikosEtAl2021.instReprShape.repr SikosEtAl2021.Shape.circle prec✝ = Repr.addAppParen (Std.Format.nest (if prec✝ ≥ 1024 then 1 else 2) (Std.Format.text "SikosEtAl2021.Shape.circle")).group prec✝

@[implicit_reducible]

instance SikosEtAl2021.instReprShape : Repr Shape

Equations

• SikosEtAl2021.instReprShape = { reprPrec := SikosEtAl2021.instReprShape.repr }

structure SikosEtAl2021.Object : Type

An object in the reference game.

• color : Color
• shape : Shape

def SikosEtAl2021.instDecidableEqObject.decEq (x✝ x✝¹ : Object) : Decidable (x✝ = x✝¹)

Equations

• SikosEtAl2021.instDecidableEqObject.decEq { color := a, shape := a_1 } { color := b, shape := b_1 } = if h : a = b then h ▸ if h : a_1 = b_1 then h ▸ isTrue ⋯ else isFalse ⋯ else isFalse ⋯

@[implicit_reducible]

instance SikosEtAl2021.instDecidableEqObject : DecidableEq Object

Equations

• SikosEtAl2021.instDecidableEqObject = SikosEtAl2021.instDecidableEqObject.decEq

def SikosEtAl2021.instReprObject.repr : Object → Nat → Std.Format

Equations

• One or more equations did not get rendered due to their size.

@[implicit_reducible]

instance SikosEtAl2021.instReprObject : Repr Object

Equations

• SikosEtAl2021.instReprObject = { reprPrec := SikosEtAl2021.instReprObject.repr }

inductive SikosEtAl2021.Feature : Type

A feature predicate: either a color or a shape word.

• color (c : Color) : Feature
• shape (s : Shape) : Feature

def SikosEtAl2021.instDecidableEqFeature.decEq (x✝ x✝¹ : Feature) : Decidable (x✝ = x✝¹)

Equations

• SikosEtAl2021.instDecidableEqFeature.decEq (SikosEtAl2021.Feature.color a) (SikosEtAl2021.Feature.color b) = if h : a = b then h ▸ isTrue ⋯ else isFalse ⋯
• SikosEtAl2021.instDecidableEqFeature.decEq (SikosEtAl2021.Feature.color c) (SikosEtAl2021.Feature.shape s) = isFalse ⋯
• SikosEtAl2021.instDecidableEqFeature.decEq (SikosEtAl2021.Feature.shape s) (SikosEtAl2021.Feature.color c) = isFalse ⋯
• SikosEtAl2021.instDecidableEqFeature.decEq (SikosEtAl2021.Feature.shape a) (SikosEtAl2021.Feature.shape b) = if h : a = b then h ▸ isTrue ⋯ else isFalse ⋯

@[implicit_reducible]

instance SikosEtAl2021.instDecidableEqFeature : DecidableEq Feature

Equations

• SikosEtAl2021.instDecidableEqFeature = SikosEtAl2021.instDecidableEqFeature.decEq

def SikosEtAl2021.instReprFeature.repr : Feature → Nat → Std.Format

Equations

• One or more equations did not get rendered due to their size.

@[implicit_reducible]

instance SikosEtAl2021.instReprFeature : Repr Feature

Equations

• SikosEtAl2021.instReprFeature = { reprPrec := SikosEtAl2021.instReprFeature.repr }

def SikosEtAl2021.featureMeaning (f : Feature) (o : Object) : Bool

Literal semantics: does the feature apply to the object?

Equations

• SikosEtAl2021.featureMeaning (SikosEtAl2021.Feature.color a) o = (o.color == a)
• SikosEtAl2021.featureMeaning (SikosEtAl2021.Feature.shape a) o = (o.shape == a)

def SikosEtAl2021.nMatches (ctx : List Object) (u : Feature) : Nat

How many objects in a context match a given utterance.

Equations

• SikosEtAl2021.nMatches ctx u = (List.filter (SikosEtAl2021.featureMeaning u) ctx).length

def SikosEtAl2021.isTrivial (ctx : List Object) (u : Feature) : Bool

A context-utterance pair is trivial when exactly one object matches.

Equations

• SikosEtAl2021.isTrivial ctx u = (SikosEtAl2021.nMatches ctx u == 1)

def SikosEtAl2021.trivialCtx : List Object

Trivial context: each utterance uniquely identifies its referent. {blue_square, green_circle, red_triangle}

Equations

• One or more equations did not get rendered due to their size.

def SikosEtAl2021.trivialUtts : List Feature

Utterances for the trivial context.

Equations

• One or more equations did not get rendered due to their size.

def SikosEtAl2021.solvableCtx : List Object

FG2012's classic solvable context: {blue_square, blue_circle, green_square}. "square" applies to two objects; pragmatic reasoning breaks the tie.

Equations

• One or more equations did not get rendered due to their size.

def SikosEtAl2021.solvableUtts : List Feature

Utterances for the solvable context.

Equations

• One or more equations did not get rendered due to their size.

theorem SikosEtAl2021.trivial_blue_unique : isTrivial trivialCtx (Feature.color Color.blue) = true

"blue" uniquely identifies blue_square in the trivial context.

theorem SikosEtAl2021.solvable_square_ambiguous : isTrivial solvableCtx (Feature.shape Shape.square) = false

"square" is ambiguous in the solvable context (matches 2 objects).

theorem SikosEtAl2021.trivial_ctx_all_trivial : trivialUtts.all (isTrivial trivialCtx) = true

The trivial context has all utterances trivially predicted.

theorem SikosEtAl2021.solvable_ctx_has_nontrivial : (List.filter (fun (u : Feature) => !isTrivial solvableCtx u) solvableUtts).length > 0

The solvable context has non-trivial utterances.