Beliefs EDA

Published

February 7, 2025

Abstract

Plots of Household beliefs on work draws disaggregated by enumerators and previous draw results

Code 
```{stata}
*| output: false
do "$code/setup_p3_legacy.do"
use "$data_gen_p3/Pilot_3_Beliefs.dta", clear

// Merge in draw data using hh_id and period
merge m:1 hh_id period using "$data_gen_p3/Pilot_3_Draw.dta", keepusing(num_high_before high) 
keep if _merge == 3
sort hh_id period
by hh_id: gen last_high = high[_n - 1] 
```

Comprehension Question Reponses

It seems that most participants understand the interface and what they are being asked in the belief module.

The comprehension questions asked participants to use the interface to describe beliefs about future rain

Code 
```{stata}
*| output: asis
// Graph: Bar chart for percent correct (assuming binary coding)
// Calculate means of correct responses
qui sum rain_equal_chance_correct
local equal = r(mean)*100

qui sum rain_more_chance_correct
local more = r(mean)*100

qui sum rain_sure_correct
local sure = r(mean)*100

qui sum rain_no_chance_correct
local no = r(mean)*100

// Display formatted results
di as text "*Understanding equal chances*: " %6.2f `equal' "% correct <br>"
di as text "*Understanding higher chances*: " %6.2f `more' "% correct <br>"
di as text "*Understanding certainty*: " %6.2f `sure' "% correct <br>"
di as text "*Understanding impossibility*: " %6.2f `no' "% correct <br>"
```

Understanding equal chances: 82.41% correct
Understanding higher chances: 93.49% correct
Understanding certainty: 78.69% correct
Understanding impossibility: 75.73% correct

Code 
```{stata}
*| include: true
graph combine rain_equal_chance, ///
    title("Distribution of Responses to Comprehension Questions") ///
    row(1) 
```

Code 
```{stata}
*| include: true
graph combine rain_sure rain_no_chance, ///
    row(1) xsize(11)
```

Distribution of Beliefs Data

The expectations about their current and future draws seems very optimistic

Code 
```{stata}
*| fig-cap: "Belief about current draw"
graph combine now, rows(1)
```

Belief about current draw

Belief about current draw
Code 
```{stata}
*| fig-cap: "Belief future draws"
graph combine future future_last, rows(1) xsize(11) ycommon
```

Belief future draws

Belief future draws

Impact of Draws on Beliefs

It doesn’t seem that the cheating accounts for the beliefs.

Code 
```{stata}
graph combine belief fut_belief fut_belief_last, rows(1) xsize(11)  title("Beliefs vs Luck so Far")
```

The regressions suggests that the correlation of high draws up to now or in the last period and beliefs about current period are small or 0

Code 
```{stata}
*| output: asis

label var high_prop "Proportion of High Draws"
label var last_high "Last Draw was High"

esttab overall last, $esttab_opts_md title("Impacts of High draws on beliefs")
```

Impacts of High draws on beliefs

How likely do you think your chance of high workload is How likely do you think your chance of high workload is
Proportion of High Draws -0.916
(0.928)
Last Draw was High 0.510
(0.462)
Constant 5.426*** 4.420***
(0.623) (0.446)
Observations 482 341
R2 0.003 0.003

Enumerator Effects and Cheating

Code 
```{stata}
sort enum_id period
by enum_id: egen high_count_enum = total(high)
by enum_id: gen high_prop_enum = high_count_enum / _N

gen suspicious_enum         = high_prop > 0.6
gen v_suspicious_enum       = high_prop > 0.7

label var suspicious_enum   "Enum with >60% highs"
label var v_suspicious_enum "Enum with >70% highs"

hist high_prop_enum, bins(10) frac xtitle("Proportion of Highs - By Enumerator")
```
(bin=10, start=.5, width=.03846154)

Plenty of enumerators have high proportions that are somewhat suspicious. Looking at the beliefs of those working with enumerators with >60%:

Code 
```{stata}
*| output: false
twoway (hist belief_high_envelope if high_prop_enum > 0.6, width(1) color(eltblue)) ///
       , xtitle("Unusually High Enumerator") legend(off) name(belief_high, replace) ///
       ytitle("Proportion of High Draws so far") 
```
Code 
```{stata}
graph combine belief_high now, xsize(16.5) ycommon rows (1)
```

While the graphs are suggestive, both a ttest for a difference in means and a regression testing suggest that enum with unusually high proportion of high draws isn’t correlated participants reporting higher beliefs about their current work draw.

In addition to null coefficients, the difference in means and coefficent estimates are very small.

Code 
```{stata}
*| output: asis
qui  {
reg belief_high_envelope suspicious_enum, vce(cluster hh_id)
       su suspicious_enum, meanonly
       estadd scalar mean = r(mean)
       eststo sus

reg belief_high_envelope v_suspicious_enum, vce(cluster hh_id)
       su v_suspicious_enum, meanonly
       estadd scalar mean = r(mean)
       eststo vsus

reg belief_prop_high_remaining suspicious_enum, vce(cluster hh_id)
       su v_suspicious_enum, meanonly
       estadd scalar mean = r(mean)
       eststo sus_2

reg belief_last_high suspicious_enum, vce(cluster hh_id)
       su suspicious_enum, meanonly
       estadd scalar mean = r(mean)
       eststo sus_3



}
esttab sus vsus sus_2 sus_3, $esttab_opts_md depvar title("Enumerators w/ unusually high proportions and impacts on beliefs about:") mtitles("Current Draw" "Current Draw" "Future Draws" "Last Draw")
```

Enumerators w/ unusually high proportions and impacts on beliefs about:

Current Draw Current Draw Future Draws Last Draw
Enum with >60% highs 0.190 0.511 -0.337
(0.471) (0.643) (0.777)
Enum with >70% highs -0.723
(0.470)
Constant 4.738*** 5.123*** 2.744*** 4.932***
(0.374) (0.303) (0.488) (0.584)
Observations 482 482 200 142
R2 0.001 0.007 0.003 0.001
Code 
```{stata}
*| output: asis
estimates clear
qui {
       estpost ttest belief_high_envelope belief_prop_high_remaining belief_last_high, by(suspicious_enum)
}

esttab . , $esttab_opts_md nonumber wide cells("mu_1(label(<60% Group)) mu_2(label(>60% Group)) b(label(diff))") nomtitle title("T-Test: Enums with <60% and those with >60%")
```

T-Test: Enums with <60% and those with >60%

<60% Group >60% Group diff
How likely do you think your chance of high workload is 4.73822 4.927835 -.1896152
In your remaining draws, how frequently do you think you’ll draw high 2.74359 3.254098 -.5105086
Do you think this is your last high draw? 4.931507 4.594203 .337304
Observations 486
R2

Adjusting the cutoff produces similar results

Code 
```{stata}
*| output: asis
estimates clear
qui {
       estpost ttest belief_high_envelope belief_prop_high_remaining belief_last_high, by(v_suspicious_enum)
}

esttab . , $esttab_opts_md nonumber wide cells("mu_1(label(<70% Group)) mu_2(label(>70% Group)) b(label(diff))") nomtitle title("T-Test: Enums with <70% and those with >70%")
```

T-Test: Enums with <70% and those with >70%

<70% Group >70% Group diff
How likely do you think your chance of high workload is 5.122517 4.4 .7225166
In your remaining draws, how frequently do you think you’ll draw high 3.337931 2.309091 1.02884
Do you think this is your last high draw? 4.931507 4.594203 .337304
Observations 486
R2

Change in beliefs over time

Note: the following graphs only show wave 2 participants since period 3 data is missing for wave 1 participants

Code 
```{stata}
gen p3_belief = belief_prop_high_remaining if period == 3
gen p4_belief = belief_prop_high_remaining if period == 4 & wave == 2
gen p5_belief = belief_last_high if wave == 2
// Visual comparison
gen p3_belief_now = belief_high_envelope if period == 3
gen p4_belief_now = belief_high_envelope if period == 4 & wave == 2
gen p5_belief_now = belief_high_envelope if period == 5 & wave == 2

graph box p3_belief_now p3_belief  p4_belief_now p4_belief  p5_belief_now p5_belief, title("Belief Time Trend") legend(pos(6) label(1 "Period 3 - Current Draw") label(2 "Period 3 - Future Draws") label(3 "Period 4 - Current Draw") label(4 "Period 4 - Future Draws") label(5 "Period 5 - Current Draw") label(6 "Period 5 - Future Draw") order(1 3 5 2 4 6) rows(2)) box(1, color(red%30)) box(2, color(red%50)) box(3, color(blue%30)) box(4, color(blue%50)) box(5, color(green%30)) box(6, color(green%50))
```
(435 missing values generated)
(438 missing values generated)
(435 missing values generated)
(435 missing values generated)
(437 missing values generated)
(435 missing values generated)

Figures - Unrestricted

Code 
```{stata}
gen p3_belief_all = belief_prop_high_remaining if period == 3
gen p4_belief_all = belief_prop_high_remaining if period == 4 
gen p5_belief_all = belief_last_high 
// Visual comparison
gen p3_belief_all_now = belief_high_envelope if period == 3
gen p4_belief_all_now = belief_high_envelope if period == 4 
gen p5_belief_all_now = belief_high_envelope if period == 5 

graph box p3_belief_all_now p3_belief_all  p4_belief_all_now p4_belief_all  p5_belief_all_now p5_belief_all , title("Beliefs about current draw") legend(pos(6) label(1 "Period 3 - Current Draw") label(2 "Period 3 - Future Draws") label(3 "Period 4 - Current Draw") label(4 "Period 4 - Future Draws") label(5 "Period 5 - Current Draw") label(6 "Period 5 - Future Draw") order(1 3 5 2 4 6) rows(2)) box(1, color(red%30)) box(2, color(red%50)) box(3, color(blue%30)) box(4, color(blue%50)) box(5, color(green%30)) box(6, color(green%50))
```
(435 missing values generated)
(367 missing values generated)
(359 missing values generated)
(435 missing values generated)
(364 missing values generated)
(362 missing values generated)

Figures - Wave 1

Code 
```{stata}
gen p4_belief_w1 = belief_prop_high_remaining if period == 4 & wave == 1
gen p5_belief_w1 = belief_last_high if wave == 1
// Visual comparison
gen p4_belief_w1_now = belief_high_envelope if period == 4 & wave == 1
gen p5_belief_w1_now = belief_high_envelope if period == 5 & wave == 1

graph box p4_belief_w1_now p4_belief_w1  p5_belief_w1_now p5_belief_w1 , title("Beliefs about current draw") legend(pos(6)  label(1 "Period 4 - Current Draw") label(2 "Period 4 - Future Draws") label(3 "Period 5 - Current Draw") label(4 "Period 5 - Future Draw") order(1 3 2 4) rows(2))  box(1, color(blue%30)) box(2, color(blue%50)) box(3, color(green%30)) box(4, color(green%50))
```
(430 missing values generated)
(425 missing values generated)
(428 missing values generated)
(428 missing values generated)

Distributions by Comprehension

Code 
```{stata}
*| output: asis
eststo clear

qui {
       // For each period
       forvalues p = 3/5 {
              // Calculate mean of flag and count observations
              estpost sum flag_poor_comprehension if period == `p'
              eststo p`p'
       }
}

label var flag_poor_comprehension "Poor Comprehension"
// Output table
esttab p3 p4 p5, main(mean) onecell ///
       title("Poor Comprehension Flag by Period") ///
       mlabels("Period 3" "Period 4" "Period 5", lhs("Period")) ///
       nonumbers $esttab_opts_md
```

Poor Comprehension Flag by Period

Period Period 3 Period 4 Period 5
Poor Comprehension 0.0152 0.228 0.281
Observations 66 145 146
R2
Code 
```{stata}
*| output: false
// Create variables for good comprehension group (flag = 0)
gen p3_belief_good = belief_prop_high_remaining if period == 3 & flag_poor_comprehension == 0
gen p4_belief_good = belief_prop_high_remaining if period == 4 & flag_poor_comprehension == 0
gen p5_belief_good = belief_last_high if flag_poor_comprehension == 0

gen p3_belief_now_good = belief_high_envelope if period == 3 & flag_poor_comprehension == 0
gen p4_belief_now_good = belief_high_envelope if period == 4 & flag_poor_comprehension == 0
gen p5_belief_now_good = belief_high_envelope if period == 5 & flag_poor_comprehension == 0

// Create variables for poor comprehension group (flag = 1)
gen p3_belief_poor = belief_prop_high_remaining if period == 3 & flag_poor_comprehension == 1
gen p4_belief_poor = belief_prop_high_remaining if period == 4 & flag_poor_comprehension == 1
gen p5_belief_poor = belief_last_high if flag_poor_comprehension == 1

gen p3_belief_now_poor = belief_high_envelope if period == 3 & flag_poor_comprehension == 1
gen p4_belief_now_poor = belief_high_envelope if period == 4 & flag_poor_comprehension == 1
gen p5_belief_now_poor = belief_high_envelope if period == 5 & flag_poor_comprehension == 1
```
Code 
```{stata}
// Plot current draw beliefs by comprehension
graph box p3_belief_now_good p3_belief_now_poor p4_belief_now_good p4_belief_now_poor p5_belief_now_good p5_belief_now_poor, ///
       title("Beliefs about Current Draw by Comprehension") ///
       legend(pos(6) label(1 "Period 3 - Good") label(2 "Period 3 - Poor") ///
       label(3 "Period 4 - Good") label(4 "Period 4 - Poor") ///
       label(5 "Period 5 - Good") label(6 "Period 5 - Poor") rows(2) order(1 3 5 2 4 6)) ///
       box(1, color(red%30)) box(2, color(red%70)) ///
       box(3, color(blue%30)) box(4, color(blue%70)) ///
       box(5, color(green%30)) box(6, color(green%70)) name(cur_comp, replace)
```

Code 
```{stata}
// Plot future draw beliefs by comprehension
graph box p3_belief_good p3_belief_poor p4_belief_good p4_belief_poor p5_belief_good p5_belief_poor, ///
       title("Beliefs about Future Draws by Comprehension") ///
       legend(pos(6) label(1 "Period 3 - Good") label(2 "Period 3 - Poor") ///
       label(3 "Period 4 - Good") label(4 "Period 4 - Poor") ///
       label(5 "Period 5 - Good") label(6 "Period 5 - Poor") order(1 3 5 2 4 6) rows(2)) ///
       box(1, color(red%30)) box(2, color(red%70)) ///
       box(3, color(blue%30)) box(4, color(blue%70)) ///
       box(5, color(green%30)) box(6, color(green%70)) name(future_comp, replace)
```

Data Collection and Comprehension