-
Notifications
You must be signed in to change notification settings - Fork 2
/
Copy pathREADME.Rmd
197 lines (166 loc) · 8.61 KB
/
README.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
---
output: github_document
---
<!-- README.md is generated from README.Rmd. Please edit that file -->
```{r, include = FALSE}
knitr::opts_chunk$set(
collapse = TRUE,
comment = "#>",
fig.path = "man/figures/README-",
out.width = "100%"
)
```
# scorecal_CSCC_2019
<!-- badges: start -->
[](https://zenodo.org/badge/latestdoi/205238383)
[](https://mybinder.org/v2/gh/rgayler/scorecal_CSCC_2019/master?urlpath=rstudio)
<!-- badges: end -->
This repository allows you to experiment with the `R` code
used to generate the conference presentation
"scorecal - Empirical score calibration under the microscope".
## Run this notebook in the cloud
**To run this code without installing anything**,
click on the `launch binder` badge above.
This will launch Rstudio on a free, small, cloud instance.
It may take a two or three minutes to launch the server and initialise Rstudio.
You will eventually get a web browser tab with the Rstudio IDE running in it,
and with this `scorecal_CSCC_2019` project open in it.
Find the file `scorecal_CSCC_2019.Rmd` in the `Files` tab of the bottom, right pane,
and click on its name.
This will open the file for editing in the top, left pane.
Then click the `Knit` button just above the code.
This will execute all the code in the notebook and render the results to `scorecal_CSCC_2019.pdf`.
You will see progress messages in the `R Markdown` tab of the bottom, left pane.
The rendered PDF file will open in a pop-up window.
The notebook took a little under three minutes to knit when I tried it.
There will be many red messages in the `R Markdown` tab while the PDF file is being rendered.
These can be ignored (provided you get sensible looking output).
The only difference I could spot is that the font used in the archived presentation
is missing from the cloud instance,
and the default font used in its place has a slightly larger spacing,
resulting in some text on the slides being slightly too long for the page.
The notebook took about two minutes to knit when I tried it.
There will be many red messages in the `R Markdown` tab while the PDF file is being rendered.
These can be ignored (provided you get sensible looking output).
The only difference I could spot is that the font used in the archived presentation
is missing from the cloud instance,
and the default font used in its place has a slightly larger spacing,
resulting in some text on the slides being slightly too long for the page.
You can edit the code in the notebook and re-knit it to see what happens.
More likely, you will want to edit the code and just execute it in the notebook without knitting it to PDF.
The results of each code chunk will be displayed in the notebook immediately after the code chunk.
To do that, you will have to find out how to drive Rstudio.
The cloud instance is small, and has various constraints imposed on it.
The limitations at the time of writing this are:
* The server has limited memory so you cannot load large datasets or run big computations
* This is meant for interactive and ephemeral interactive coding so an instance will die after 10 minutes of inactivity.
* An instance cannot be kept alive for more than 12 hours
The [holepunch](https://karthik.github.io/holepunch/) package was used to convert this repository to a docker image for cloud execution.
## Generate the conference presentation
**This repository contains an executable `R` notebook**
that generates the presentation "scorecal - Empirical score calibration under the microscope"
given by
[Ross W. Gayler](http://www.rossgayler.com)
on 2019-08-30
at the conference
[Credit Scoring & Credit Control XVI](https://crc.business-school.ed.ac.uk/conferences/)
in Edinburgh, UK.
The presentation, as given, is archived at [](https://doi.org/10.5281/zenodo.3381658)
The notebook contains all the `R` code
used to simulate and analyse the data
and generate the plots.
This is in the form of a script rather than a package.
You are free to modify the script to see what happens.
If you are using the Rstudio IDE,
edit the notebook file `scorecal_CSCC_2019.Rmd` and execute the relevant code chunks.
The results will be displayed in the notebook immediately after the code chunks.
Click the `Knit` button to render a new copy of the presentation slides with the results of your altered code.
This notebook requires the [`binb`](http://github.com/eddelbuettel/binb) package
to enable rendering the output as a PDF presentation.
It uses the `metropolis` template to set the style of the presentation.
These require a variety of LaTeX tools and fonts to be installed,
in addition to the `rmarkdown` and `knitr` infrastructure.
See http://github.com/eddelbuettel/binb for `binb` installation advice.
**Conference presentation abstract**
Score calibration is the process of empirically determining the relationship between a score
and an outcome on some population of interest,
and scaling is the process of expressing that relationship in agreed units.
Calibration is often treated as a simple matter and attacked with simple tools –
typically, either assuming the relationship between score and log-odds is linear
and fitting a logistic regression with the score as the only covariate,
or dividing the score range into bands
and plotting the empirical log-odds as a function of score band.
Both approaches ignore some information in the data.
The assumption of a linear score to log-odds relationship is too restrictive
and score banding ignores the continuity of the scores.
While a linear score to log-odds relationship is often an adequate approximation,
the reality can be much more interesting,
with noticeable deviations from the linear trend.
These deviations include large-scale non-linearity,
small-scale non-monotonicity,
discrete discontinuities,
and complete breakdown of the linear trend at extreme scores.
Detecting these effects requires a more sophisticated approach
to empirically determining the score to outcome relationship.
Taking a more sophisticated approach can be surprisingly tricky:
the typically strong linear trend can obscure smaller deviations from linearity;
detecting subtle trends requires exploiting the continuity of the scores,
which can obscure discrete deviations;
trends at extreme scores (out in the data-sparse tails of the distribution of scores)
can be obscured by trends at less extreme scores (where there is more data);
score distributions with some specific values that are relatively common
can disrupt methods relying on continuity;
and any modelling technique can introduce its own biases.
Over the years I have developed a personal approach to these issues in score calibration
and implemented them as an open source, publicly accessible R package for score calibration.
I discuss these technical issues in empirical score calibration
and show how they are addressed in the scorecal package.
### Acknowledgements
Thanks to:
[Jonathan Crook](http://www.business-school.ed.ac.uk/staff/jonathan-crook)
and the [Credit Research Centre, University of Edinburgh Business School](https://crc.business-school.ed.ac.uk/),
for running a great [conference](https://crc.business-school.ed.ac.uk/conferences/).
[Mathew Ling](http://www.mathewling.com/) and the fine folk of [ANZORN](http://www.anzopenresearch.org/)
for helping me use [`holepunch`](https://karthik.github.io/holepunch/) to make this repository remotely executable.
All the maintainers of the `R` packages used in this repository,
and the maintainers of all their dependencies.
```{r echo = FALSE}
suppressPackageStartupMessages(
{
library(thankr)
library(magrittr)
library(dplyr)
library(knitr)
}
)
thankr::shoulders(where = "package",
package = c(
"binb",
"binom",
"cowplot",
"dplyr",
"fastDummies",
"forcats",
"fs",
"glmnet",
"glmnetUtils",
"glue",
"grid",
"here",
"holepunch",
"knitr",
"magrittr",
"mgcv",
"patchwork",
"rmarkdown",
"sn",
"sessioninfo",
"stringr",
"thankr",
"tibble",
"tidyverse"
),
include_dependencies = FALSE) %>%
dplyr::select(-no_packages) %>%
knitr::kable()
```