This page provides a comprehensive side-by-side comparison of all four primary result groups. Each group represents an independent dataset with progressively less restrictive quality criteria. Researchers can use this comparison to assess how data cleaning decisions affect statistical conclusions.
All descriptive statistics, reliability coefficients, and correlations are shown for each result group. The Δ column shows the difference from Conservative Clean (the strictest dataset). Deltas > 0.05 are highlighted in amber.
Metric
Conservative Clean
Flexible Clean
Prolific Accepted
All V2 Finished
Δ Flexible
Δ Prolific
Δ All
Barrier Grand Mean
2.8239
2.8146
2.7790
2.7401
-0.0093
-0.0449
-0.0838
Barrier SD
0.6332
0.7178
0.7153
0.7714
+0.0846
+0.0821
+0.1382
Readiness Grand Mean
3.0528
3.0968
3.1400
3.2494
+0.0440
+0.0872
+0.1966
Readiness SD
0.5806
0.6572
0.6749
0.7150
+0.0766
+0.0943
+0.1344
Maturity Grand Mean
3.0377
3.0661
3.1571
3.2667
+0.0284
+0.1194
+0.2290
Maturity SD
0.7061
0.7960
0.8055
0.8058
+0.0899
+0.0994
+0.0997
B-R Correlation
-0.4856
-0.4595
-0.3423
-0.3165
+0.0261
+0.1433
+0.1691
B-M Correlation
-0.2315
-0.3197
-0.2787
-0.3100
-0.0882
-0.0472
-0.0785
R-M Correlation
0.5899
0.6982
0.7125
0.7341
+0.1083
+0.1226
+0.1442
Alpha Barriers
0.8580
0.8772
0.8770
0.9011
+0.0192
+0.0190
+0.0431
Alpha Readiness
0.8756
0.9181
0.9207
0.9305
+0.0425
+0.0451
+0.0549
Alpha Maturity
0.8330
0.8833
0.8897
0.8900
+0.0503
+0.0567
+0.0570
Survey Demographics Comparison (Qualtrics)
Role distribution and organization size breakdown for each result group, based on self-reported survey responses (Q1, Q4). These are organizational demographics from the TABS instrument. Prolific platform demographics (age, sex, ethnicity, plus prescreener fields like industry, company size, and occupation) are collected separately and can be used to cross-validate these survey responses via Prolific Participant ID.
Tech vs Non-Tech Composition
Result Group
N
Technical
Non-Technical
Other
% Tech
Conservative Clean
80
15
52
13
18.8%
Flexible Clean
127
29
79
19
22.8%
Prolific Accepted
229
51
133
45
22.3%
All V2 Finished
363
91
206
66
25.1%
Organization Size Distribution
Result Group
<100
100–499
500–999
1000–4999
5000–9999
10000+
Conservative Clean
9
29
7
18
6
11
Flexible Clean
17
40
16
30
8
16
Prolific Accepted
40
57
33
48
20
33
All V2 Finished
56
98
57
73
31
53
Filter Bias Analysis
This analysis tests whether stricter quality filters disproportionately exclude certain demographics. A Chi-square test for independence is computed across the four result groups for role, organization size, and profit model.
Demographic Category
Chi-Square (χ²)
df
p-value
Interpretation
Role (Tech vs Non-Tech)
3.51
6
0.7425
No significant difference (demographics stable)
Organization Size
9.46
15
0.8520
No significant difference (demographics stable)
Profit Model
2.69
6
0.8462
No significant difference (demographics stable)
Profit Model Distribution
Result Group
For-Profit
Non-Profit
Government/Public Sector
Conservative Clean
57
15
8
Flexible Clean
87
27
13
Prolific Accepted
172
41
18
All V2 Finished
277
62
29
Effect Size Comparison
Cohen’s d effect sizes for key group comparisons across all four result groups. This shows how effect sizes shift as the sample becomes less restrictive.
Tech vs Non-Tech (Cohen’s d)
Construct
Conservative Clean
Flexible Clean
Prolific Accepted
All V2 Finished
barriers
-0.234
-0.306
-0.118
-0.120
readiness
0.543
0.575
0.560
0.355
maturity
0.152
0.295
0.314
0.210
Large vs Small/Medium Org (Cohen’s d)
Construct
Conservative Clean
Flexible Clean
Prolific Accepted
All V2 Finished
barriers
0.548
0.476
0.225
0.233
readiness
-0.067
-0.268
0.051
0.027
Interpretation Guide
Metrics that remain stable across all four groups suggest robust findings that are not sensitive to data cleaning decisions. Metrics that show large deltas (highlighted in amber) between Conservative Clean and less restrictive groups warrant further investigation, as the finding may depend on sample composition.
As a rule of thumb: if a Cohen’s d shifts by more than 0.1 between Conservative Clean and All V2 Finished, the effect size may be inflated or attenuated by lower-quality responses. Similarly, if Cronbach’s α drops below 0.70 in larger samples, it may indicate that less-engaged respondents are adding noise to the scale.
Related
Key Findings — effect sizes, t-tests, and ANOVA per result group
