Dataset Comparison
Last updated: Jun 3, 2026, 8:09 AM EDT
This page provides a comprehensive side-by-side comparison of all four primary result groups. Each group represents an independent dataset with progressively less restrictive quality criteria. Researchers can use this comparison to assess how data cleaning decisions affect statistical conclusions.
Sample Overview
| # | Result Group | N | Definition |
|---|---|---|---|
| 1 | Conservative Clean | 140 | Prolific APPROVED + all quality checks (IRI, duration >= 540s, reCAPTCHA, straightlining, auth) |
| 2 | Flexible Clean | 231 | Prolific APPROVED + basic quality (all 3 IRIs + duration >= 480s) |
| 3 | Prolific Accepted | 440 | All deduplicated V2 rows with Prolific APPROVED status |
| 4 | All V2 Finished | 689 | Finished + duration >= 120s (extreme speeders excluded) |
Core Metrics Comparison
All descriptive statistics, reliability coefficients, and correlations are shown for each result group. The Δ column shows the difference from Conservative Clean (the strictest dataset). Deltas > 0.05 are highlighted in amber.
| Metric | Conservative Clean | Flexible Clean | Prolific Accepted | All V2 Finished | Δ Flexible | Δ Prolific | Δ All |
|---|---|---|---|---|---|---|---|
| Barrier Grand Mean | 2.8566 | 2.8804 | 2.8867 | 2.8956 | +0.0238 | +0.0301 | +0.0390 |
| Barrier SD | 0.6641 | 0.7199 | 0.7267 | 0.7849 | +0.0558 | +0.0626 | +0.1208 |
| Readiness Grand Mean | 3.0468 | 3.0546 | 3.0930 | 3.2065 | +0.0078 | +0.0462 | +0.1597 |
| Readiness SD | 0.5978 | 0.6631 | 0.6560 | 0.7191 | +0.0653 | +0.0582 | +0.1213 |
| Maturity Grand Mean | 3.0558 | 3.0495 | 3.1135 | 3.2261 | -0.0063 | +0.0577 | +0.1703 |
| Maturity SD | 0.7069 | 0.8104 | 0.7934 | 0.8112 | +0.1035 | +0.0865 | +0.1043 |
| B-R Correlation | -0.4083 | -0.4330 | -0.3299 | -0.2654 | -0.0247 | +0.0784 | +0.1429 |
| B-M Correlation | -0.1890 | -0.2872 | -0.2472 | -0.2349 | -0.0982 | -0.0582 | -0.0459 |
| R-M Correlation | 0.6001 | 0.7066 | 0.7197 | 0.7254 | +0.1065 | +0.1196 | +0.1253 |
| Alpha Barriers | 0.8664 | 0.8859 | 0.8897 | 0.9086 | +0.0195 | +0.0233 | +0.0422 |
| Alpha Readiness | 0.8720 | 0.9123 | 0.9128 | 0.9273 | +0.0403 | +0.0408 | +0.0553 |
| Alpha Maturity | 0.8264 | 0.8864 | 0.8855 | 0.8927 | +0.0600 | +0.0591 | +0.0663 |
Survey Demographics Comparison (Qualtrics)
Role distribution and organization size breakdown for each result group, based on self-reported survey responses (Q1, Q4). These are organizational demographics from the TABS instrument. Prolific platform demographics (age, sex, ethnicity, plus prescreener fields like industry, company size, and occupation) are collected separately and can be used to cross-validate these survey responses via Prolific Participant ID.
Tech vs Non-Tech Composition
| Result Group | N | Technical | Non-Technical | Other | % Tech |
|---|---|---|---|---|---|
| Conservative Clean | 140 | 28 | 112 | 0 | 20.0% |
| Flexible Clean | 231 | 52 | 179 | 0 | 22.5% |
| Prolific Accepted | 440 | 98 | 342 | 0 | 22.3% |
| All V2 Finished | 689 | 174 | 515 | 0 | 25.3% |
Organization Size Distribution
| Result Group | <100 | 100-499 | 500-999 | 1000-4999 | 5000-9999 | 10000+ |
|---|---|---|---|---|---|---|
| Conservative Clean | 21 | 49 | 13 | 30 | 11 | 16 |
| Flexible Clean | 33 | 77 | 32 | 48 | 14 | 27 |
| Prolific Accepted | 73 | 133 | 63 | 84 | 36 | 51 |
| All V2 Finished | 106 | 208 | 109 | 129 | 55 | 82 |
Filter Bias Analysis
This analysis tests whether stricter quality filters disproportionately exclude certain demographics. A Chi-square test for independence is computed across the four result groups for role, organization size, and profit model.
| Demographic Category | Chi-Square (χ²) | df | p-value | Interpretation |
|---|---|---|---|---|
| Role (Tech vs Non-Tech) | 2.63 | 3 | 0.4524 | No significant difference (demographics stable) |
| Organization Size | 7.22 | 15 | 0.9513 | No significant difference (demographics stable) |
| Profit Model | 3.49 | 6 | 0.7454 | No significant difference (demographics stable) |
Profit Model Distribution
| Result Group | For-Profit | Non-Profit | Government/Public Sector |
|---|---|---|---|
| Conservative Clean | 99 | 21 | 20 |
| Flexible Clean | 160 | 40 | 31 |
| Prolific Accepted | 313 | 79 | 48 |
| All V2 Finished | 505 | 107 | 77 |
Effect Size Comparison
Cohen’s d effect sizes for key group comparisons across all four result groups. This shows how effect sizes shift as the sample becomes less restrictive.
Tech vs Non-Tech (Cohen’s d)
| Construct | Conservative Clean | Flexible Clean | Prolific Accepted | All V2 Finished |
|---|---|---|---|---|
| barriers | -0.133 | -0.228 | -0.045 | -0.084 |
| readiness | 0.659 | 0.729 | 0.654 | 0.526 |
| maturity | 0.313 | 0.432 | 0.412 | 0.400 |
Large vs Small/Medium Org (Cohen’s d)
| Construct | Conservative Clean | Flexible Clean | Prolific Accepted | All V2 Finished |
|---|---|---|---|---|
| barriers | 0.343 | 0.274 | 0.064 | 0.053 |
| readiness | -0.074 | -0.124 | 0.021 | 0.016 |
Interpretation Guide
Metrics that remain stable across all four groups suggest robust findings that are not sensitive to data cleaning decisions. Metrics that show large deltas (highlighted in amber) between Conservative Clean and less restrictive groups warrant further investigation, as the finding may depend on sample composition.
As a rule of thumb: if a Cohen’s d shifts by more than 0.1 between Conservative Clean and All V2 Finished, the effect size may be inflated or attenuated by lower-quality responses. Similarly, if Cronbach’s α drops below 0.70 in larger samples, it may indicate that less-engaged respondents are adding noise to the scale.
Related
- Key Findings - effect sizes, t-tests, and ANOVA per result group
- Sensitivity Analysis - metric-level sensitivity across sample definitions
- Descriptive Statistics - correlation matrices per result group
- ← Back to Results Overview