Background: Undergraduate
Undergraduate in Psychology
- Statistics
- Experiment Design
- Cognitive Theory
- Neurology
- Humans
Background: Honours
Psychophysics: illusory contours in 3D
Phil Grove
Background: Honours
Christina Lee
If every psychologist in the world delivered gold standard smoking cessation therapy, the rate of smoking would still increase. You need to change policy to make change. To make effective policy, you need to have good data, and do good statistics.
Background: PhD
Kerrie Mengersen
- Statistical Approaches to Revealing Structure in Complex Health Data
- Exploring missing values
- Bayesian Models of people's health over time
- Geospatial statistics of cardiac arrest
- Fun, applied, real data, real people
Background: PhD
- "Ah, statistics, everything is black and white!
- "There's always an answer"
- "data in, answer out"
😱 Missing Values
Background: PhD
- Data is really messy
- Missing values are frustrating
- How to Explore data?
Background: PhD - But in psych
- Focus on experiment design
- No focus on exploring data
- Exploring data felt...wrong?
- But it was so critical.
(My personal) motivation
Focus on building a bridge across a river. Less focus on how it is built, and the tools used.
visdat
published under "visdat: Visualising Whole Data Frames" in the Journal of Open Source Software
visdat::vis_dat(airquality)
visdat::vis_miss(airquality)
naniar
Tierney, NJ. Cook, D. "Expanding tidy data principles to facilitate missing data exploration, visualization and assessment of imputations." [Pre-print]
naniar::gg_miss_var(airquality)
naniar::gg_miss_var(airquality, facet = Month)
naniar::gg_miss_upset(airquality)
All of Australia
...And New Zealand
... And Afghanistan and Albania
And the rest?
And the rest?
Problems:
- Overplotting
- We don't see the individuals
- We could look at 144 individual plots, but this doesn't help.
Does transparency help?
Does transparency + a model help?
This is still useful
But we forget about the individuals
How do we get the most out of this plot?
How do I even get started?!
Introducing brolgar: brolgar.njtierney.com
- browsing
- over
- longitudinal data
- graphically, and
- analytically, in
- r
Longitudinal data as a time series 
heights <- as_tsibble(heights, index = year, key = country, regular = FALSE)- index: Your time variable
- key: Variable(s) defining individual groups (or series)
1. + 2. determine distinct rows in a tsibble.
(From Earo Wang's talk: Melt the clock)
Longitudinal data as a time series
Key Concepts:
Record important time series information once, and use it many times in other places
- We add information about index + key:
- Index = Year
- Key = Country
Problem #1: How do I look at some of the data?
Look at only a sample of the data:
Sample n rows with sample_n()
sample_n_keys() to sample ... keys
Look at many subsamples
Look at many subsamples
Look at many subsamples
facet_sample(): See more individuals
ggplot(heights, aes(x = year, y = height_cm, group = country)) + geom_line()
facet_sample(): See more individuals
facet_strata(): See all individuals
Can we re-order these facets in a meaningful way?
facet_strata(along = -year): see all individuals along some variable
Problem #2: How do I find interesting observations?
Problem #2: How do I find interesting observations?
Identify features: summarise down to one observation
Identify features: summarise down to one observation
Identify features: summarise down to one observation
Identify important features and decide how to filter
Identify important features and decide how to filter
Join this feature back to the data
Join this feature back to the data
🎉 Countries with smallest and largest max height
Identify features: one per key 
heights %>% features(height_cm, feat_five_num)## # A tibble: 144 x 6## country min q25 med q75 max## <chr> <dbl> <dbl> <dbl> <dbl> <dbl>## 1 Afghanistan 161. 164. 167. 168. 168.## 2 Albania 168. 168. 170. 170. 170.## 3 Algeria 166. 168. 169 170. 171.## 4 Angola 159. 160. 167. 168. 169.## 5 Argentina 167. 168. 168. 170. 174.## 6 Armenia 164. 166. 169. 172. 172.## # … with 138 more rowsfeatures: Summaries that are aware of data structure
features: Summaries that are aware of data structure
heights %>% features(height_cm, #<< # variable we want to summarise feat_five_num) #<< # feature to calculatefeatures: Summaries that are aware of data structure
heights %>% features(height_cm, #<< # variable we want to summarise feat_five_num) #<< # feature to calculate## # A tibble: 144 x 6## country min q25 med q75 max## <chr> <dbl> <dbl> <dbl> <dbl> <dbl>## 1 Afghanistan 161. 164. 167. 168. 168.## 2 Albania 168. 168. 170. 170. 170.## 3 Algeria 166. 168. 169 170. 171.## 4 Angola 159. 160. 167. 168. 169.## 5 Argentina 167. 168. 168. 170. 174.## 6 Armenia 164. 166. 169. 172. 172.## # … with 138 more rowsfeatures: MANY more features in feasts
Such as:
feat_acf: autocorrelation-based featuresfeat_stl: STL (Seasonal, Trend, and Remainder by LOESS) decomposition
features: what is feat_five_num?
features: what is feat_five_num?
feat_five_num## function(x, ...) {## list(## min = b_min(x, ...),## q25 = b_q25(x, ...),## med = b_median(x, ...),## q75 = b_q75(x, ...),## max = b_max(x, ...)## )## }## <bytecode: 0x7fa11b5b9d28>## <environment: namespace:brolgar>features: what is feat_five_num?
Problem #3: How do I understand my statistical model
Look at subsamples?
Look at many subsamples?
gg_heights_fit + facet_sample()
Look at all subsamples?
gg_heights_fit + facet_strata()
Look at all subsamples along residuals?
gg_heights_fit + facet_strata(along = -res)
Look at the predictions with the data?
Look at the predictions with the data?
heights_sample + geom_point(aes(y = height_cm))
Show data by joining it with residuals, to explore spread
