Reproducible Analysis Pipeline
Last updated: Apr 13, 2026, 1:39 PM EDT
The TABS project is committed to open science and computational reproducibility. Every statistic reported in the Culminating Research Project (CRP) can be independently verified by running our public analysis scripts against the released dataset. This page explains how the pipeline works and how you can use it.
The dataset is archived at Penn State ScholarSphere under a CC-BY-4.0 license, ensuring long-term institutional preservation independent of the principal investigator.
Architecture: Single Source of Truth
A common challenge in research software is keeping multiple systems in sync. The TABS project uses two parallel data processing systems: a live TypeScript pipeline for operational triage (approving and flagging survey submissions in real-time) and public Python scripts for research-grade statistical analysis. These systems share critical constants: scale mappings, IRI expected answers, column definitions, and duration thresholds.
To prevent divergence, all shared constants are defined in a single TypeScript file (tabs-survey-constants.ts) that serves as the authoritative source. A CI workflow automatically exports these constants to JSON for the Python scripts to consume. Any change to a constant in one system is immediately validated against the other.
Data Flow
- tabs-survey-constants.ts defines all instrument constants (scale mappings, IRI answers, column names, thresholds)
- generate-constants-json.ts exports constants to JSON on every commit
- disposition.ts (TypeScript) imports constants for live triage
- tabs_v2_data_audit.py (Python) reads the JSON for reproducible analysis
- validate-analysis.yml (CI) verifies both systems agree on every push
Analysis Scripts
The reproducible analysis pipeline consists of five core Python scripts plus a presentation validator. Each script reads a standard Qualtrics CSV export and applies identical scale mappings, column definitions, and quality filters sourced from the shared constants file.
1. Data Audit (tabs_v2_data_audit.py)
Implements the complete 10-step disposition waterfall that determines which survey responses are included in analysis. This is a faithful Python port of the live TypeScript triage logic, ensuring that the research analysis applies exactly the same quality criteria as the operational pipeline.
Waterfall steps: Incomplete check, Prolific auth verification, IRI attention checks (3 constructs), speed flags, Smeal benchmark, reCAPTCHA score, full-block straightlining, partial straightlining (within-person SD), and final CLEAN disposition.
2. Statistical Analysis (tabs_v2_analysis.py)
Computes all descriptive and inferential statistics reported in the CRP: construct means, standard deviations, correlations with 95% confidence intervals, t-tests with effect sizes, ANOVA, sensitivity analysis across sample definitions, and demographic cross-tabulations.
Key outputs: Barrier severity rankings, readiness profiles, maturity assessments, construct correlations, demographic comparisons across role, industry, org size, and profit model.
3. Psychometric Validation (tabs_v2_psychometrics.py)
Validates 84 statistical claims embedded in the CRP document against computed values from the source CSV. This ensures that all reported statistics — construct means, correlations, reliability coefficients, validity metrics, and demographic tables — are traceable to the data and protects against transcription errors.
Key outputs:Pass/fail summary with detailed mismatch reports. Coverage includes Cronbach’s alpha, AVE, HTMT ratios, factor loadings, and demographic breakdowns.
4. Advanced Statistics (tabs_v2_advanced.py)
Performs inferential statistics, factor analysis (PCA with Varimax rotation), interaction effects, and moderation analyses that extend the primary descriptive results.
Key outputs:Factor extraction with variance explained, KMO and Bartlett’s tests, budget moderation effects, revenue tiers, role-by-role comparisons, and geographic scope analysis.
5. Data Quality Audit (tabs_v2_quality_audit.py)
Systematically examines dataset flaws, biases, and limitations. Produces a comprehensive quality report including straightlining detection, outlier analysis (Mahalanobis distance), response pattern diagnostics, and order/fatigue effects.
Key outputs: Response entropy analysis, acquiescence bias metrics, extreme response style detection, position-based fatigue, and within-person SD distributions.
6. Defense Deck Validator (validate-deck.py)
Validates 81 statistical claims in the defense presentation (PPTX) against computed values from the source CSV. Ensures consistency between the CRP document and presentation materials.
Key outputs: Per-slide PASS/FAIL summary for item means, standard deviations, grand construct means, and Pearson correlations.
Sample Definitions
The analysis script supports five sample definitions, from most to least restrictive. Each applies different quality filters to the same underlying V2 dataset. Running all statistics against every sample definition demonstrates whether findings are robust to inclusion criteria - a key requirement for publication-grade research.
| Sample | Criteria | N |
|---|---|---|
| Conservative Clean | Prolific APPROVED + all quality checks (IRI, duration >= 540s, reCAPTCHA, straightlining, auth) | 87 |
| Flexible Clean | Prolific APPROVED + basic quality (all 3 IRIs + duration >= 480s) | 134 |
| Prolific Accepted | All deduplicated V2 rows with Prolific APPROVED status | 243 |
| All V2 Finished | Finished + duration >= 120s (extreme speeders excluded) | 391 |
| All V2 | All V2 responses including incomplete | 462 |
N values are populated by running python tabs_v2_analysis.py <csv> --json sensitivity-analysis.json against the production dataset.
Sensitivity Analysis
Every key statistic is computed across all sample definitions to verify that conclusions do not depend on a single set of inclusion criteria. If a finding holds across Conservative Clean, Flexible Clean, and Prolific Accepted samples, it is robust. If it shifts substantially, the sensitivity analysis flags it for discussion.
| Metric | Conservative Clean | Flexible Clean | Prolific Accepted | All V2 Finished | All V2 |
|---|---|---|---|---|---|
| Barrier Grand Mean | 2.8323 | 2.8205 | 2.7809 | 2.746 | 2.7513 |
| Barrier SD | 0.6288 | 0.7112 | 0.7153 | 0.7678 | 0.7714 |
| Readiness Grand Mean | 3.0634 | 3.1014 | 3.1416 | 3.2454 | 3.2454 |
| Readiness SD | 0.5648 | 0.6441 | 0.6692 | 0.7172 | 0.7163 |
| Maturity Grand Mean | 3.0553 | 3.076 | 3.163 | 3.2762 | 3.2762 |
| Maturity SD | 0.7066 | 0.7915 | 0.8009 | 0.8045 | 0.8045 |
| B-R Correlation | -0.4346 | -0.4351 | -0.3359 | -0.294 | -0.2938 |
| B-M Correlation | -0.1771 | -0.2886 | -0.2668 | -0.3089 | -0.3089 |
| R-M Correlation | 0.582 | 0.6906 | 0.7099 | 0.7124 | 0.7124 |
| Alpha Barriers | 0.8545 | 0.8746 | 0.8778 | 0.8993 | 0.9006 |
| Alpha Readiness | 0.867 | 0.9132 | 0.9171 | 0.9312 | 0.9312 |
| Alpha Maturity | 0.8327 | 0.8805 | 0.8866 | 0.891 | 0.891 |
How to Reproduce the Analysis
Anyone can independently verify the statistics reported in the CRP by following these steps:
# Clone the repository git clone https://github.com/clarkemoyer/technologyadoptionbarriers.org.git cd technologyadoptionbarriers.org/scripts/analysis # Install Python dependencies (pinned versions) pip install -r requirements.txt # Run with test data (included in repo) python tabs_v2_data_audit.py --input test_data_qualtrics.csv python tabs_v2_analysis.py test_data_qualtrics.csv python tabs_v2_psychometrics.py test_data_qualtrics.csv python tabs_v2_advanced.py test_data_qualtrics.csv python tabs_v2_quality_audit.py test_data_qualtrics.csv # Run with production data (from ScholarSphere) python tabs_v2_data_audit.py --input <path_to_production_csv> python tabs_v2_analysis.py <path_to_production_csv> python tabs_v2_psychometrics.py <path_to_production_csv> python tabs_v2_advanced.py <path_to_production_csv> python tabs_v2_quality_audit.py <path_to_production_csv> # Validate defense presentation python ../validate-deck.py <path_to_csv> <path_to_pptx> # Export sensitivity analysis as JSON (for dashboard) python tabs_v2_analysis.py <csv> --json sensitivity-analysis.json # Run with a different primary sample definition python tabs_v2_analysis.py <csv> --primary-sample conservative_clean python tabs_v2_analysis.py <csv> --primary-sample flexible_clean python tabs_v2_analysis.py <csv> --primary-sample prolific_accepted
Test Data & Automated Testing
The repository includes multiple test datasets and a comprehensive pytest suite so that anyone can verify the analysis logic without production data.
| File | Records | Purpose |
|---|---|---|
| test_data.csv | 5 | Simplified format for quick logic checks with 5 actual response rows (clean, IRI fail, duration fail, Don’t Know); blank Qualtrics-style metadata rows are not included in the count |
| test_data_qualtrics.csv | 15 | Full Qualtrics CSV format with realistic headers and diverse demographic combinations |
| tests/generate_test_data.py | - | Deterministic generator for the production-format synthetic dataset. Public directory browsing is intentionally not linked here while the repository remediates and sanitizes the test CSV per the PII policy. |
The test suite includes a comprehensive set of pytest modules covering every analysis script, cross-validation between scripts, edge cases, CLI argument parsing, and the operational pipeline tools. Tests run automatically in CI on every push.
Preventing Drift Between Systems
The most dangerous failure mode in a dual-system research pipeline is silent divergence: the live pipeline and the public scripts gradually drift apart until they produce different results from the same data. The TABS project prevents this through three mechanisms:
- Centralized constants: All instrument definitions (scale labels, IRI answers, column names, thresholds) live in one file. Neither the TypeScript nor Python code hardcodes these values independently.
- CI validation: Every commit that touches constants or analysis scripts triggers an automated workflow that regenerates the JSON export, validates all values match, and runs the Python scripts against test data.
- Faithful port: The Python disposition waterfall (
tabs_v2_data_audit.py) implements the exact same 10-step logic as the TypeScript live pipeline (disposition.ts), including matching within-person SD calculations for partial straightlining detection.
What the Shared Constants Cover
| Constant Category | Examples | Why It Matters |
|---|---|---|
| Scale Mappings | “Major Barrier” = 5, “Very Low Readiness/Capability” = 1 | If one system maps a label to the wrong number, all statistics diverge silently |
| IRI Expected Answers | Barrier: “Major Barrier”, Readiness: “Low Readiness/Capability” | Wrong IRI answer = wrong disposition = wrong sample = wrong conclusions |
| Column Names | Q10-28_Barriers_19 (IRI), Q47-64_Readiness_18 (IRI) | Column mismatch reads the wrong data entirely |
| Duration Thresholds | Speed flag: 300s, Smeal: 300-540s, Clean: 480s | Different thresholds produce different sample sizes and statistics |
Open Science Commitments
- Open source code: All scripts are public on GitHub
- Pinned dependencies: requirements.txt locks exact package versions for reproducibility across environments
- Test data included: Three test datasets (5 + 15 + 500 records) and a deterministic generator exercise all processing paths without requiring production data access
- Automated test suite: A comprehensive pytest suite with CI integration verifies script correctness on every commit
- Versioned datasets: Each data release (N=200, N=500, annual) receives its own DOI via ScholarSphere
- CC-BY-4.0 license: Both code and data are freely reusable with attribution
- CI-validated: Automated checks ensure constants match, scripts run, and outputs are correct on every code change
Get Involved
If you find an issue with the analysis scripts or want to contribute improvements, the repository welcomes pull requests. The analysis scripts are designed to be extended: add new statistical tests, improve visualizations, or adapt the pipeline for your own research context.