Reproducible Analysis Pipeline

Architecture: Single Source of Truth

A common challenge in research software is keeping multiple systems in sync. The TABS project uses two parallel data processing systems: a live TypeScript pipeline for operational triage (approving and flagging survey submissions in real-time) and public Python scripts for research-grade statistical analysis. These systems share critical constants: scale mappings, IRI expected answers, column definitions, and duration thresholds.

To prevent divergence, all shared constants are defined in a single TypeScript file (tabs-survey-constants.ts) that serves as the authoritative source. A CI workflow automatically exports these constants to JSON for the Python scripts to consume. Any change to a constant in one system is immediately validated against the other.

Analysis Scripts

The reproducible analysis pipeline is built around a single unified script that consolidates all analysis phases - sensitivity, advanced statistics, data quality, and psychometric validation - into one run. The unified script runs validation across all five sample definitions, producing per-sample results with sample-size adequacy metadata. The individual component scripts remain available for targeted analysis.

Sample Definitions

The analysis script supports five sample definitions, from most to least restrictive. Each applies different quality filters to the same underlying V2 dataset. Running all statistics against every sample definition demonstrates whether findings are robust to inclusion criteria - a key requirement for publication-grade research.

N values are populated by running python tabs_v2_unified_data_analysis.py <csv> --json unified.json and extracting the sensitivity section from the unified output.

Sensitivity Analysis

Every key statistic is computed across all sample definitions to verify that conclusions do not depend on a single set of inclusion criteria. If a finding holds across Conservative Clean, Flexible Clean, and Prolific Accepted samples, it is robust. If it shifts substantially, the sensitivity analysis flags it for discussion.

How to Reproduce the Analysis

Anyone can independently verify the statistics reported in the CRP by following these steps:

# Clone the repository
git clone https://github.com/clarkemoyer/technologyadoptionbarriers.org.git
cd technologyadoptionbarriers.org/scripts/analysis

# Install Python dependencies (pinned versions)
pip install -r requirements.txt

# ── Recommended: Unified analysis (reproduces all results) ──

# Run with production data (from ScholarSphere)
python tabs_v2_unified_data_analysis.py <path_to_csv> --json unified.json

# Run CRP-200 frozen dataset
python tabs_v2_unified_data_analysis.py <path_to_crp_csv> --crp200 --json crp-unified.json

# Extract individual JSON files from unified output
python -c "import json; d=json.load(open('unified.json')); json.dump(d['sensitivity'], open('sensitivity-analysis.json','w'), indent=2)"
python -c "import json; d=json.load(open('unified.json')); json.dump(d['validation'], open('live-validation.json','w'), indent=2)"

# Choose a different primary sample for advanced analysis
python tabs_v2_unified_data_analysis.py <csv> --json unified.json \
  --primary-sample flexible_clean

# ── Alternative: Individual scripts (for targeted analysis) ──

python tabs_v2_data_audit.py --input <path_to_csv>
python tabs_v2_analysis.py <path_to_csv>
python tabs_v2_psychometrics.py <path_to_csv>
python tabs_v2_advanced.py <path_to_csv>
python tabs_v2_quality_audit.py <path_to_csv>

# Validate defense presentation
python ../validate-deck.py <path_to_csv> <path_to_pptx>

Test Data & Automated Testing

The repository includes multiple test datasets and a comprehensive pytest suite so that anyone can verify the analysis logic without production data.

The test suite includes a comprehensive set of pytest modules covering every analysis script, cross-validation between scripts, edge cases, CLI argument parsing, and the operational pipeline tools. Tests run automatically in CI on every push.

Preventing Drift Between Systems

The most dangerous failure mode in a dual-system research pipeline is silent divergence: the live pipeline and the public scripts gradually drift apart until they produce different results from the same data. The TABS project prevents this through three mechanisms:

1. Centralized constants: All instrument definitions (scale labels, IRI answers, column names, thresholds) live in one file. Neither the TypeScript nor Python code hardcodes these values independently.
2. CI validation: Every commit that touches constants or analysis scripts triggers an automated workflow that regenerates the JSON export, validates all values match, and runs the Python scripts against test data.
3. Faithful port: The Python disposition waterfall (tabs_v2_data_audit.py) implements the exact same 10-step logic as the TypeScript live pipeline (disposition.ts), including matching within-person SD calculations for partial straightlining detection.

What the Shared Constants Cover

Open Science Commitments

• Open source code: All scripts are public on GitHub
• Pinned dependencies: requirements.txt locks exact package versions for reproducibility across environments
• Test data included: Three test datasets (5 + 15 + 500 records) and a deterministic generator exercise all processing paths without requiring production data access
• Automated test suite: A comprehensive pytest suite with CI integration verifies script correctness on every commit
• Versioned datasets: Each data release (N=200, N=500, annual) receives its own DOI via ScholarSphere
• CC-BY-4.0 license: Both code and data are freely reusable with attribution
• CI-validated: Automated checks ensure constants match, scripts run, and outputs are correct on every code change

Get Involved

If you find an issue with the analysis scripts or want to contribute improvements, the repository welcomes pull requests. The analysis scripts are designed to be extended: add new statistical tests, improve visualizations, or adapt the pipeline for your own research context.