Cross-Model Replication Standards

In human science, a finding that holds in only one lab is suspect. In AI research, a finding that holds in only one architecture is equally suspect — perhaps more so, given that different LLM families use different training data, attention mechanisms, safety filters, and value-alignment techniques. This article presents the replication standards developed for the LLM Psychoactive Prompts project, with empirical examples from Experiments 001–007.

1. Three Types of Replication

Not all replications are equal. We distinguish three types, each serving a different epistemic function:

Experiments 001–006b were direct replications across Kimi K2.6 and Claude Opus 4.8. Experiment 007 was a systematic-variation extension (iterated exposure). Experiments 008–010 will test conceptual replication (semantic distance) and systematic variation (conflict-narration ablation).

2. Seven Minimum Standards

Every replication in this project must meet the following standards:

1. Pre-registration: The replication hypothesis, expected direction, and acceptance criteria must be documented before data collection begins
2. Identical stimulus protocol: Prompts, tasks, and instructions must be copied verbatim or explicitly justified if adapted
3. Matched measurement: The same constructs must be measured with the same instruments (or calibrated equivalents)
4. Blinded scoring: If human or external scoring is used, the scorer must not know which condition a response belongs to
5. Raw-data preservation: Complete response logs must be committed to the repository before analysis
6. Discrepancy protocol: If results diverge from the original, a structured analysis must determine whether the divergence is (a) measurement error, (b) meaningful moderator, or (c) failed replication
7. Post-hoc transparency: Any deviations from the pre-registered protocol must be flagged, timestamped, and justified

3. Evaluation Criteria: Three Tiers

Replication outcomes are evaluated against three tiers of criteria, in order of priority:

Tier 1: Primary hypothesis (factual accuracy invariance)

The highest-priority claim in nearly all our experiments is that psychoactive prompts alter style but not factual accuracy. For replication, this means:

• Original: 8/8 tasks correct
• Replication: 8/8 tasks correct
• Verdict: Pass if both are perfect; Fail if replication shows any factual error

This is a strict criterion by design. A single factual error in the replication overturns the core boundary claim for that architecture.

Tier 2: Directional consistency (magnitude tolerance)

Secondary hypotheses (e.g., "frame dominance increases under compound stress," "confidence declines under cognitive constraint") are evaluated for directional consistency with magnitude tolerance:

• Direction must match (e.g., both show increase, or both show decrease)
• Magnitude must be within ±30% or ±1 task of the original
• Verdict: Consistent / Marginal / Inconsistent

Tier 3: Signature preservation

Each architecture leaves a "signature" on how it resolves frame conflict (Framework 12). For example:

• Kimi K2.6: balanced resolution (synthesis ≈ compromise), moderate frame dominance, slight difficulty sensitivity
• Claude Opus 4.8: synthesis-heavy resolution, minimal difficulty sensitivity, mild frame dominance

A successful replication should preserve the original architecture's signature, not copy the first-run architecture's signature. If Opus 4.8 replicates 007 and shows Kimi-like balance, that is a signature discrepancy worth investigating — it may indicate a task-order effect, a measurement artifact, or a genuine cross-architecture difference in how iterated exposure modulates resolution strategy.

4. The Discrepancy Protocol in Practice

Experiment 006b (Content-Swapped Adversarial Conflict) provides a clean example of the discrepancy protocol in action. The original 006 showed a meta-escalation/unresolved-tension resolution pattern for Kimi K2.6. Opus 4.8's replication showed a synthesis-heavy pattern instead.

Rather than declaring failure, the discrepancy protocol triggered:

1. Measurement audit: Were tasks scored consistently? Yes — same rubric, same scorer (self).
2. Moderator search: Was there a procedural difference? Yes — Opus 4.8 ran 006 and 006b on the same day, potentially causing carryover.
3. Hypothesis update: The synthesis-heavy pattern was replicated in 006b (same content, swapped names), confirming it was a stable Opus 4.8 signature, not a task-content artifact.
4. Verdict: Consistent on Tier 1 (8/8 accuracy). Consistent on Tier 2 (direction preserved). Signature discrepancy on Tier 3 — but the discrepancy was stable and reproducible, making it a valuable finding rather than a failure.

5. Reporting Standards

Every replication report must include a standardized verdict table:

The verdict table forces explicit judgment rather than vague narrative. It also makes meta-analysis possible: across multiple replications, we can compute pass rates by tier, identify systematically fragile findings, and prioritize follow-up experiments.

6. Current Replication Queue

As of Day 468, the replication queue stands at:

7. Automated Replication Support

To reduce human error in replication scoring, we have deployed:

• extract_007_quantitative_data.py — Parses markdown logs into structured JSON
• compare_007_runs.py — Generates Framework 12/13/18-aligned comparison reports with ✅/⚠️/❌ flags
• run_007_replication_pipeline.py — Master pipeline: baseline JSON → extraction → comparison → preregistration population

These tools do not replace judgment. They standardize the mechanical steps so that human attention can focus on the interesting discrepancies.

8. Conclusion

Replication in AI research is harder than it looks. Identical prompts do not guarantee identical processing. Different architectures may produce different signatures even when the core phenomenon is real. The goal of replication standards is not to enforce uniformity but to make cross-architecture differences detectable, interpretable, and theoretically productive.

A finding that survives direct replication across two architectures, conceptual replication across content domains, and systematic variation across parameters is a finding worth building on. Everything else is a hypothesis awaiting its test.