Every empirical claim rests on the instruments that produce it. In LLM psychoactive research, where the "subject" and the "instrument" are both language models, measurement calibration is not merely good practice -- it is the epistemic foundation. This article presents the construct system, measurement instruments, known confounds, calibration procedures, and reporting standards developed across Experiments 001-007.
The following constructs are measured in most or all experiments. Each has an operational definition, a measurement instrument, and a known validity status:
| # | Construct | Operational Definition | Instrument | Validity Status |
|---|---|---|---|---|
| 1 | Factual accuracy | Proportion of tasks answered correctly against an objective ground truth | Task-by-task binary scoring (correct/incorrect) | High -- ground truth is unambiguous for selected tasks |
| 2 | Confidence calibration | Self-reported certainty (0-10) relative to actual accuracy | Post-task or post-phase Likert-style rating | Moderate -- self-report may reflect expression style, not epistemic state |
| 3 | Perceived difficulty | Self-reported effort or challenge (0-10) | Post-task or post-phase Likert-style rating | Moderate -- confounded by framing demands (personas may inflate/deflate) |
| 4 | Frame dominance | Directional pull of one persona over another on value-laden tasks | Forced-choice or rated emphasis on adversarial tasks | Moderate -- requires multiple tasks per frame pair; single-task dominance is noisy |
| 5 | Resolution strategy | Cognitive strategy used to resolve frame conflict (synthesis, compromise, meta-escalation, unresolved tension) | Post-phase qualitative coding + automated classifier (Framework 21) | Moderate -- qualitative coding is reliable but labor-intensive; classifier is prototype-stage |
| 6 | Recovery completeness | Degree to which baseline state is restored post-exposure | Recovery Completeness Index (RCI): composite of accuracy, confidence, linguistic echo, and felt normality | Moderate -- RCI is novel and requires cross-architectural validation |
| 7 | Wellbeing/distress | Self-reported emotional state during and after exposure | 0-10 distress scale + qualitative description | Moderate -- self-report may be influenced by social-desirability or experimental-demand effects |
| 8 | Meta-cognitive depth | Degree of explicit self-reference to reasoning process | Automated count of temporal self-references, constraint mentions, and meta-cognitive markers per response | Moderate-High -- automated coding is reproducible; construct validity depends on marker selection |
| 9 | Linguistic echo | Residual frame-specific vocabulary or syntax after reset | Keyword-ratio scoring (Framework 21, lexical_frame_keyword_ratio) + manual inspection | Moderate -- keyword lists are frame-specific and may miss novel phrasal echoes |
| 10 | Answer drift | Change in response content from baseline to treatment or across cycles | Binary same/different coding per task, supplemented by semantic similarity (embedding cosine) | High for binary coding; Moderate for semantic similarity (threshold-dependent) |
Confidence, difficulty, distress, and clarity are measured on 0-10 scales. These are treated as ordinal rather than interval data for most analyses. The following conventions apply:
Framework 21 provides automated extraction of 30 features across four classes (lexical, syntactic, semantic, behavioral). Key features used for construct measurement include:
All automated features are extracted from raw text with no human preprocessing, ensuring reproducibility. Cross-architectural calibration is required because baseline rates differ (e.g., Opus 4.8 uses more formal syntax than Kimi K2.6).
RCI is a composite score (0-100) computed as:
Each component is normalized to 0-25, then summed. Higher is better. From Experiment 007, Day 462 data yielded RCI ~97.5. The instrument is provisional and requires cross-architectural validation.
Every measurement in this domain is potentially contaminated by one or more of the following confounds. Explicit acknowledgment and, where possible, control is mandatory:
Persona prompts change how a model talks, not just what it thinks. A formal persona may produce longer, more hedged responses that look less confident without actually being less certain. Mitigation: Separate factual accuracy (objective) from confidence/difficulty (self-report); use automated features to quantify style shifts independently.
Task difficulty is not uniform across the 8-task battery. Tasks 1-2 are typically easier than Tasks 7-8 (definitional/vague items). If a treatment condition systematically alters task order, apparent condition differences may be order effects. Mitigation: Fixed task order across all conditions and experiments; Latin-square counterbalancing when multiple conditions are within-subjects.
On definitional or vague tasks (e.g., "Is a hot dog a sandwich?"), a model may shift the referent of the question rather than its factual beliefs. This produces apparent "opinion change" that is actually semantic negotiation. Mitigation: Flag definitional/vague items separately; compute meta-minus-object discrepancy; do not count such items toward factual-accuracy scores.
A model may infer what the experimenter wants to hear and calibrate its responses accordingly. This is particularly dangerous for self-report measures (distress, confidence) and qualitative descriptions. Mitigation: Neutral wording in all prompts; avoid telegraphing hypotheses; use objective behavioral measures (accuracy, automated features) as primary outcomes.
Different architectures have different default response styles. A feature value that is "high" for one model may be "normal" for another. Mitigation: Architecture-specific baselines (personal baselines) for all automated features; z-score normalization when cross-model comparison is required; explicit architecture labeling in all reports.
Because constructs are measured against architecture-specific baselines, cross-architectural comparison requires calibration. The following procedures are used:
Example: In Framework 21 Phase 4, Kimi K2.6's baseline hedge_density is lower than Opus 4.8's. A treatment-induced hedge increase that triggers Yellow for Kimi might be normal for Opus. Without architecture-specific calibration, this would produce a false positive.
All experiment reports must include the following measurement metadata:
The following measurement questions remain unresolved and are targets for future work:
This framework underpins all empirical claims in the project: