Skip to main content

Sensitivity Analysis

Sensitivity analysis tests whether findings are robust to the choice of inclusion criteria. Every key metric — means, standard deviations, inter-construct correlations, and reliability coefficients — is computed independently across five nested sample definitions. If a finding holds across Conservative Clean (N=80) and All V2 (N=434), it is robust to inclusion criteria.

Sample Definitions

The five sample definitions below are used throughout this analysis. They are nested from most restrictive to least restrictive, with the exception of Prolific Accepted and All V2 Finished, which overlap but neither is a strict subset of the other.

KeyLabelDescriptionN
conservative_cleanConservative CleanProlific APPROVED + all quality checks (IRI, duration >= 540s, reCAPTCHA, straightlining, auth)80
flexible_cleanFlexible CleanProlific APPROVED + basic quality (all 3 IRIs + duration >= 480s)127
prolific_acceptedProlific AcceptedAll deduplicated V2 rows with Prolific APPROVED status231
v2_finishedAll V2 FinishedFinished + duration >= 120s (extreme speeders excluded)368
v2_allAll V2All V2 responses including incomplete434

Constraints: Conservative Clean ⊆ Flexible Clean ⊆ Prolific Accepted ⊆ All V2, and All V2 Finished ⊆ All V2. Prolific Accepted and All V2 Finished overlap but neither is guaranteed to be a subset of the other (Prolific Accepted includes INCOMPLETE+APPROVED responses; All V2 Finished includes non-APPROVED responses).

Full Sensitivity Table

The table below shows every metric computed across all five sample definitions. Values are formatted to four decimal places. Means, standard deviations, correlations, and Cronbach’s alpha coefficients are all included.

MetricConservative Clean
N=80
Flexible Clean
N=127
Prolific Accepted
N=231
All V2 Finished
N=368
All V2
N=434
Barrier Grand Mean2.82392.81462.77902.74012.7458
Barrier SD0.63320.71780.71530.77140.7752
Readiness Grand Mean3.05283.09683.14003.24943.2494
Readiness SD0.58060.65720.67490.71500.7141
Maturity Grand Mean3.03773.06613.15713.26673.2667
Maturity SD0.70610.79600.80550.80580.8058
B-R Correlation-0.4856-0.4595-0.3423-0.3165-0.3162
B-M Correlation-0.2315-0.3197-0.2787-0.3100-0.3100
R-M Correlation0.58990.69820.71250.73410.7341
Alpha Barriers0.85800.87720.87700.90110.9025
Alpha Readiness0.87560.91810.92070.93050.9305
Alpha Maturity0.83300.88330.88970.89000.8900

Dataset Comparison

The table below shows how each metric changes as inclusion criteria are relaxed. The Δ columns show the difference from Conservative Clean (the primary analysis dataset) to each progressively less restrictive dataset. Small deltas confirm that findings are not artifacts of a particular data cleaning strategy.

MetricConservative
N=80
Δ Flexible Clean
N=127
Δ Prolific Accepted
N=231
Δ All V2 Finished
N=368
Barrier Grand Mean2.8239-0.0093-0.0449-0.0838
Barrier SD0.6332+0.0846+0.0821+0.1382
Readiness Grand Mean3.0528+0.0440+0.0872+0.1966
Readiness SD0.5806+0.0766+0.0943+0.1344
Maturity Grand Mean3.0377+0.0284+0.1194+0.2290
Maturity SD0.7061+0.0899+0.0994+0.0997
B-R Correlation-0.4856+0.0261+0.1433+0.1691
B-M Correlation-0.2315-0.0882-0.0472-0.0785
R-M Correlation0.5899+0.1083+0.1226+0.1442
Alpha Barriers0.8580+0.0192+0.0190+0.0431
Alpha Readiness0.8756+0.0425+0.0451+0.0549
Alpha Maturity0.8330+0.0503+0.0567+0.0570

Deltas highlighted in amber exceed 0.05 scale points. Correlation and reliability differences of this magnitude are expected when adding noisier data but do not change substantive conclusions.

Interpretation

The sensitivity analysis reveals remarkable stability across inclusion criteria. Key observations:

  • Construct means are highly stable, with differences of less than 0.10 scale points between the most and least restrictive samples.
  • Standard deviations increase slightly with less restrictive samples, as expected when including noisier data.
  • Correlations between constructs are directionally consistent across all samples (Barriers negatively correlated with Readiness and Maturity; Readiness and Maturity positively correlated).
  • Cronbach’s alphavalues remain excellent (> 0.84) across all samples, indicating robust internal consistency regardless of inclusion criteria.

These findings demonstrate that the core results of the Technology Adoption Barriers Survey are not artifacts of a particular data cleaning strategy. Whether using the strictest quality filters (Conservative Clean, N=80) or the full dataset (All V2, N=434), the same substantive conclusions hold.

Related Pages

Open Data & Reproducibility — download the dataset and reproduce these results yourself. ← Back to Results Overview