From b74d83d1ae9ae36f5840e90bcf96eea092b646fc Mon Sep 17 00:00:00 2001 From: "Daniel J. McDonald" Date: Thu, 12 Oct 2023 15:55:05 -0700 Subject: [PATCH] module 4 done --- .../23-nnets-other/execute-results/html.json | 23 + .../figure-revealjs/unnamed-chunk-10-1.svg | 990 ++++++++++++++++++ .../figure-revealjs/unnamed-chunk-3-1.svg | 246 +++++ .../figure-revealjs/unnamed-chunk-4-1.svg | 304 ++++++ .../figure-revealjs/unnamed-chunk-5-1.svg | 303 ++++++ .../figure-revealjs/unnamed-chunk-7-1.svg | 358 +++++++ .../figure-revealjs/unnamed-chunk-8-1.svg | 251 +++++ .../figure-revealjs/unnamed-chunk-9-1.svg | 375 +++++++ _freeze/site_libs/twitter-widget/widgets.js | 8 + schedule/slides/23-nnets-other.html | 613 ----------- schedule/slides/23-nnets-other.qmd | 288 +++-- 11 files changed, 2986 insertions(+), 773 deletions(-) create mode 100644 _freeze/schedule/slides/23-nnets-other/execute-results/html.json create mode 100644 _freeze/schedule/slides/23-nnets-other/figure-revealjs/unnamed-chunk-10-1.svg create mode 100644 _freeze/schedule/slides/23-nnets-other/figure-revealjs/unnamed-chunk-3-1.svg create mode 100644 _freeze/schedule/slides/23-nnets-other/figure-revealjs/unnamed-chunk-4-1.svg create mode 100644 _freeze/schedule/slides/23-nnets-other/figure-revealjs/unnamed-chunk-5-1.svg create mode 100644 _freeze/schedule/slides/23-nnets-other/figure-revealjs/unnamed-chunk-7-1.svg create mode 100644 _freeze/schedule/slides/23-nnets-other/figure-revealjs/unnamed-chunk-8-1.svg create mode 100644 _freeze/schedule/slides/23-nnets-other/figure-revealjs/unnamed-chunk-9-1.svg create mode 100644 _freeze/site_libs/twitter-widget/widgets.js delete mode 100644 schedule/slides/23-nnets-other.html diff --git a/_freeze/schedule/slides/23-nnets-other/execute-results/html.json b/_freeze/schedule/slides/23-nnets-other/execute-results/html.json new file mode 100644 index 0000000..bcaeb0c --- /dev/null +++ b/_freeze/schedule/slides/23-nnets-other/execute-results/html.json @@ -0,0 +1,23 @@ +{ + "hash": "76e653849650b32b289ca1c075d509ab", + "result": { + "markdown": "---\nlecture: \"23 Neural nets - other considerations\"\nformat: revealjs\nmetadata-files: \n - _metadata.yml\n---\n---\n---\n\n## {{< meta lecture >}} {.large background-image=\"gfx/smooths.svg\" background-opacity=\"0.3\"}\n\n[Stat 406]{.secondary}\n\n[{{< meta author >}}]{.secondary}\n\nLast modified -- 12 October 2023\n\n\n\n$$\n\\DeclareMathOperator*{\\argmin}{argmin}\n\\DeclareMathOperator*{\\argmax}{argmax}\n\\DeclareMathOperator*{\\minimize}{minimize}\n\\DeclareMathOperator*{\\maximize}{maximize}\n\\DeclareMathOperator*{\\find}{find}\n\\DeclareMathOperator{\\st}{subject\\,\\,to}\n\\newcommand{\\E}{E}\n\\newcommand{\\Expect}[1]{\\E\\left[ #1 \\right]}\n\\newcommand{\\Var}[1]{\\mathrm{Var}\\left[ #1 \\right]}\n\\newcommand{\\Cov}[2]{\\mathrm{Cov}\\left[#1,\\ #2\\right]}\n\\newcommand{\\given}{\\ \\vert\\ }\n\\newcommand{\\X}{\\mathbf{X}}\n\\newcommand{\\x}{\\mathbf{x}}\n\\newcommand{\\y}{\\mathbf{y}}\n\\newcommand{\\P}{\\mathcal{P}}\n\\newcommand{\\R}{\\mathbb{R}}\n\\newcommand{\\norm}[1]{\\left\\lVert #1 \\right\\rVert}\n\\newcommand{\\snorm}[1]{\\lVert #1 \\rVert}\n\\newcommand{\\tr}[1]{\\mbox{tr}(#1)}\n\\newcommand{\\brt}{\\widehat{\\beta}^R_{s}}\n\\newcommand{\\brl}{\\widehat{\\beta}^R_{\\lambda}}\n\\newcommand{\\bls}{\\widehat{\\beta}_{ols}}\n\\newcommand{\\blt}{\\widehat{\\beta}^L_{s}}\n\\newcommand{\\bll}{\\widehat{\\beta}^L_{\\lambda}}\n$$\n\n\n\n\n\n\n## Estimation procedures (training)\n\n\nBack-propagation\n\n[Advantages:]{.secondary}\n\n- It's updates only depend on local\n information in the sense that if objects in the hierarchical model\n are unrelated to each other, the updates aren't affected\n\n (This helps in many ways, most notably in parallel architectures)\n\n- It doesn't require second-derivative information\n\n- As the updates are only in terms of $\\hat{R}_i$, the algorithm can\n be run in either batch or online mode\n\n[Down sides:]{.tertiary}\n\n- It can be very slow\n\n- Need to choose the learning rate\n $\\gamma_t$\n\n## Other algorithms\n\nThere are many variations on the fitting algorithm\n\n[Stochastic gradient descent:]{.secondary} (SGD) discussed in the optimization lecture\n\nThe rest are variations that use lots of tricks\n\n* RMSprop\n* Adam\n* Adadelta\n* Adagrad\n* Adamax\n* Nadam\n* Ftrl\n\n\n## Regularizing neural networks\n\nNNets can almost always achieve 0 training error. Even with regularization. Because they have so many parameters.\n\nFlavours:\n\n- a complexity penalization term $\\longrightarrow$ solve $\\min \\hat{R} + \\rho(\\alpha,\\beta)$\n- early stopping on the back propagation algorithm used for fitting\n\n\nWeight decay\n: This is like ridge regression in that we penalize the squared Euclidean norm of the weights $\\rho(\\mathbf{W},\\mathbf{B}) = \\sum w_i^2 + \\sum b_i^2$\n\nWeight elimination\n: This encourages more shrinking of small weights $\\rho(\\mathbf{W},\\mathbf{B}) = \\sum \\frac{w_i^2}{1+w_i^2} + \\sum \\frac{b_i^2}{1 + b_i^2}$ or Lasso-type\n\nDropout\n: In each epoch, randomly choose $z\\%$ of the nodes and set those weights to zero.\n\n\n\n## Other common pitfalls\n\nThere are a few areas to watch out for\n\n[Nonconvexity:]{.tertiary} \n\nThe neural network optimization problem is non-convex. \n\nThis makes any numerical solution highly dependent on the initial values. These should be\n\n* chosen carefully, typically random near 0. [DON'T]{.hand} use all 0.\n* regenerated several times to check sensitivity\n\n[Scaling:]{.tertiary} \nBe sure to standardize the covariates before training\n\n## Other common pitfalls\n\n[Number of hidden units:]{.tertiary} \nIt is generally\nbetter to have too many hidden units than too few (regularization\ncan eliminate some).\n\n\n[Sifting the output:]{.tertiary}\n\n* Choose the solution that minimizes training error\n* Choose the solution that minimizes the penalized training error\n* Average the solutions across runs\n\n\n## Tuning parameters\n\nThere are many.\n\n* Regularization\n* Stopping criterion\n* learning rate\n* Architecture\n* Dropout %\n* others...\n\nThese are hard to tune.\n\nIn practice, people might choose \"some\" with a validation set, and fix the rest largely arbitrarily\n\n. . .\n\nMore often, people set them all arbitrarily\n\n\n## Thoughts on NNets {.smaller}\n\nOff the top of my head, without lots of justification\n\n::: flex\n::: w-50\n\n๐Ÿคฌ๐Ÿ˜ก [Why don't statisticians like them?]{.tertiary} ๐Ÿคฌ๐Ÿ˜ก\n\n- There is little theory (though this is increasing)\n- Stat theory applies to global minima, here, only local determined by the optimizer\n- Little understanding of when they work\n- In large part, NNets look like logistic regression + feature creation. We understand that well, and in many applications, it performs as well\n- Explosion of tuning parameters without a way to decide\n- Require massive datasets to work\n- Lots of examples where they perform _exceedingly_ poorly\n:::\n\n::: w-50\n\n\n๐Ÿ”ฅ๐Ÿ”ฅ[Why are they hot?]{.tertiary}๐Ÿ”ฅ๐Ÿ”ฅ\n\n- Perform exceptionally well on typical CS tasks (images, translation)\n- Take advantage of SOTA computing (parallel, GPUs)\n- Very good for multinomial logistic regression\n- An excellent example of \"transfer learning\"\n- They generate pretty pictures (the nets, pseudo-responses at hidden units)\n\n:::\n:::\n\n\n## Keras\n\nMost people who do deep learning use Python $+$ Keras $+$ Tensorflow\n\nIt takes some work to get all this software up and running.\n\nIt is possible to do in with R using an [interface to Keras](https://keras.rstudio.com/index.html).\n\n. . .\n\nI used to try to do a walk-through, but the interface is quite brittle\n\nIf you want to explore, see the handout:\n\n* Knitted: \n* Rmd: \n\n\n# Double descent and model complexity\n\n##\n\n\n::: {.cell layout-align=\"center\"}\n::: {.cell-output-display}\n```{=html}\n

The Bias-Variance Trade-Off & "DOUBLE DESCENT" ๐Ÿงต

Remember the bias-variance trade-off? It says that models perform well for an "intermediate level of flexibility". You've seen the picture of the U-shape test error curve.

We try to hit the "sweet spot" of flexibility.

1/๐Ÿงต pic.twitter.com/HPk05izkZh

— Daniela Witten (@daniela_witten) August 9, 2020
\n\n```\n:::\n:::\n\n\n\n## Where does this U shape come from?\n\n\n[MSE = Squared Bias + Variance + Irreducible Noise]{.secondary}\n\n\nAs we increase flexibility:\n\n* Squared bias goes down\n* Variance goes up\n* Eventually, | $\\partial$ Variance | $>$ | $\\partial$ Squared Bias |.\n\n\n[Goal:]{.secondary} Choose amount of flexibility to balance these and minimize MSE.\n\n. . .\n\n[Use CV or something to estimate MSE and decide how much flexibility.]{.hand}\n\n\n##\n\n\n::: {.cell layout-align=\"center\"}\n::: {.cell-output-display}\n```{=html}\n

In the past few yrs, (and particularly in the context of deep learning) ppl have noticed "double descent" -- when you continue to fit increasingly flexible models that interpolate the training data, then the test error can start to DECREASE again!!

Check it out:ย 
3/ pic.twitter.com/Vo54tRVRNG

— Daniela Witten (@daniela_witten) August 9, 2020
\n\n```\n:::\n:::\n\n\n\n\n## Zero training error and model saturation\n\n* In Deep Learning, the recommendation is to \"fit until you get zero training error\"\n\n* This somehow magically, leads to a continued decrease in test error.\n\n* So, who cares about the Bias-Variance Trade off!!\n\n. . .\n\n[Lesson:]{.secondary}\n\nBV Trade off is not wrong. ๐Ÿ˜ข\n\nThis is a misunderstanding of black box algorithms and flexibility.\n\nWe don't even need deep learning to illustrate. \n\n##\n\n\n::: {.cell layout-align=\"center\"}\n\n```{.r .cell-code}\nlibrary(splines)\nset.seed(20221102)\nn <- 20\ndf <- tibble(\n x = seq(-1.5 * pi, 1.5 * pi, length.out = n),\n y = sin(x) + runif(n, -0.5, 0.5)\n)\ng <- ggplot(df, aes(x, y)) + geom_point() + stat_function(fun = sin) + ylim(c(-2, 2))\ng + stat_smooth(method = lm, formula = y ~ bs(x, df = 4), se = FALSE, color = green) + # too smooth\n stat_smooth(method = lm, formula = y ~ bs(x, df = 8), se = FALSE, color = orange) # looks good\n```\n\n::: {.cell-output-display}\n![](23-nnets-other_files/figure-revealjs/unnamed-chunk-3-1.svg){fig-align='center'}\n:::\n:::\n\n\n\n##\n\n\n::: {.cell layout-align=\"center\"}\n\n```{.r .cell-code}\nxn <- seq(-1.5 * pi, 1.5 * pi, length.out = 1000)\n# Spline by hand\nX <- bs(df$x, df = 20, intercept = TRUE)\nXn <- bs(xn, df = 20, intercept = TRUE)\nS <- svd(X)\nyhat <- Xn %*% S$v %*% diag(1/S$d) %*% crossprod(S$u, df$y)\ng + geom_line(data = tibble(x=xn, y=yhat), colour = orange) +\n ggtitle(\"20 degrees of freedom\")\n```\n\n::: {.cell-output-display}\n![](23-nnets-other_files/figure-revealjs/unnamed-chunk-4-1.svg){fig-align='center'}\n:::\n:::\n\n\n##\n\n\n::: {.cell layout-align=\"center\"}\n\n```{.r .cell-code}\nxn <- seq(-1.5 * pi, 1.5 * pi, length.out = 1000)\n# Spline by hand\nX <- bs(df$x, df = 40, intercept = TRUE)\nXn <- bs(xn, df = 40, intercept = TRUE)\nS <- svd(X)\nyhat <- Xn %*% S$v %*% diag(1/S$d) %*% crossprod(S$u, df$y)\ng + geom_line(data = tibble(x = xn, y = yhat), colour = orange) +\n ggtitle(\"40 degrees of freedom\")\n```\n\n::: {.cell-output-display}\n![](23-nnets-other_files/figure-revealjs/unnamed-chunk-5-1.svg){fig-align='center'}\n:::\n:::\n\n\n\n## What happened?!\n\n\n\n::: {.cell layout-align=\"center\"}\n\n```{.r .cell-code code-line-numbers=\"1|3-12|13-16|\"}\ndoffs <- 4:50\nmse <- function(x, y) mean((x - y)^2)\nget_errs <- function(doff) {\n X <- bs(df$x, df = doff, intercept = TRUE)\n Xn <- bs(xn, df = doff, intercept = TRUE)\n S <- svd(X)\n yh <- S$u %*% crossprod(S$u, df$y)\n bhat <- S$v %*% diag(1 / S$d) %*% crossprod(S$u, df$y)\n yhat <- Xn %*% S$v %*% diag(1 / S$d) %*% crossprod(S$u, df$y)\n nb <- sqrt(sum(bhat^2))\n tibble(train = mse(df$y, yh), test = mse(yhat, sin(xn)), norm = nb)\n}\nerrs <- map(doffs, get_errs) |>\n list_rbind() |> \n mutate(`degrees of freedom` = doffs) |> \n pivot_longer(train:test, values_to = \"error\")\n```\n:::\n\n\n## What happened?!\n\n\n\n::: {.cell layout-align=\"center\"}\n\n```{.r .cell-code code-fold=\"true\"}\nggplot(errs, aes(`degrees of freedom`, error, color = name)) +\n geom_line(linewidth = 2) + \n coord_cartesian(ylim = c(0, .12)) +\n scale_x_log10() + \n scale_colour_manual(values = c(blue, orange), name = \"\") +\n geom_vline(xintercept = 20)\n```\n\n::: {.cell-output-display}\n![](23-nnets-other_files/figure-revealjs/unnamed-chunk-7-1.svg){fig-align='center'}\n:::\n:::\n\n\n## What happened?!\n\n\n::: {.cell layout-align=\"center\"}\n\n```{.r .cell-code code-fold=\"true\"}\nbest_test <- errs |> filter(name == \"test\")\nmin_norm <- best_test$norm[which.min(best_test$error)]\nggplot(best_test, aes(norm, error)) +\n geom_line(colour = blue, size = 2) + ylab(\"test error\") +\n geom_vline(xintercept = min_norm, colour = orange) +\n scale_y_log10() + scale_x_log10() + geom_vline(xintercept = 20)\n```\n\n::: {.cell-output-display}\n![](23-nnets-other_files/figure-revealjs/unnamed-chunk-8-1.svg){fig-align='center'}\n:::\n:::\n\n\n\n## Degrees of freedom and complexity\n\n* In low dimensions (where $n \\gg p$), with linear smoothers, df and model complexity are roughly the same.\n\n* But this relationship breaks down in more complicated settings\n\n* We've already seen this:\n\n\n::: {.cell layout-align=\"center\"}\n\n```{.r .cell-code}\nlibrary(glmnet)\nout <- cv.glmnet(X, df$y, nfolds = n) # leave one out\n```\n:::\n\n::: {.cell layout-align=\"center\"}\n\n```{.r .cell-code code-fold=\"true\"}\nwith(\n out, \n tibble(lambda = lambda, df = nzero, cv = cvm, cvup = cvup, cvlo = cvlo )\n) |> \n filter(df > 0) |>\n pivot_longer(lambda:df) |> \n ggplot(aes(x = value)) +\n geom_errorbar(aes(ymax = cvup, ymin = cvlo)) +\n geom_point(aes(y = cv), colour = orange) +\n facet_wrap(~ name, strip.position = \"bottom\", scales = \"free_x\") +\n scale_y_log10() +\n scale_x_log10() + theme(axis.title.x = element_blank())\n```\n\n::: {.cell-output-display}\n![](23-nnets-other_files/figure-revealjs/unnamed-chunk-10-1.svg){fig-align='center'}\n:::\n:::\n\n\n\n## Infinite solutions\n\n* In Lasso, df is not really the right measure of complexity\n\n* Better is $\\lambda$ or the norm of the coefficients (these are basically the same)\n\n* So what happened with the Splines?\n\n. . .\n\n* When df $= 20$, there's a unique solution that interpolates the data\n\n* When df $> 20$, there are infinitely many solutions that interpolate the data.\n\nBecause we used the SVD to solve the system, we happened to pick one: the one that has the smallest $\\Vert\\hat\\beta\\Vert_2$\n\nRecent work in Deep Learning shows that SGD has the same property: it returns the local optima with the smallest norm.\n\nIf we measure complexity in terms of the norm of the weights, rather than by counting parameters, we don't see double descent anymore.\n\n\n## The lesson\n\n* Deep learning isn't magic.\n\n* Zero training error with lots of parameters doesn't mean good test error.\n\n* We still need the bias variance tradeoff\n\n* It's intuition still applies: more flexibility eventually leads to increased MSE\n\n* But we need to be careful how we measure complexity.\n\n::: aside\n\nThere is very interesting recent theory that says \nwhen we can expect lower test error to the right of the interpolation threshold\nthan to the left. \n\n:::\n\n\n\n# Next time...\n\n[Module 5]{.secondary}\n\n[unsupervised learning]{.secondary}\n\n\n\n\n", + "supporting": [ + "23-nnets-other_files" + ], + "filters": [ + "rmarkdown/pagebreak.lua" + ], + "includes": { + "include-in-header": [ + "\n" + ], + "include-after-body": [ + "\n\n\n" + ] + }, + "engineDependencies": {}, + "preserve": {}, + "postProcess": true + } +} \ No newline at end of file diff --git a/_freeze/schedule/slides/23-nnets-other/figure-revealjs/unnamed-chunk-10-1.svg b/_freeze/schedule/slides/23-nnets-other/figure-revealjs/unnamed-chunk-10-1.svg new file mode 100644 index 0000000..df82c8d --- /dev/null +++ b/_freeze/schedule/slides/23-nnets-other/figure-revealjs/unnamed-chunk-10-1.svg @@ -0,0 +1,990 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/_freeze/schedule/slides/23-nnets-other/figure-revealjs/unnamed-chunk-3-1.svg b/_freeze/schedule/slides/23-nnets-other/figure-revealjs/unnamed-chunk-3-1.svg new file mode 100644 index 0000000..df9be59 --- /dev/null +++ b/_freeze/schedule/slides/23-nnets-other/figure-revealjs/unnamed-chunk-3-1.svg @@ -0,0 +1,246 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/_freeze/schedule/slides/23-nnets-other/figure-revealjs/unnamed-chunk-4-1.svg b/_freeze/schedule/slides/23-nnets-other/figure-revealjs/unnamed-chunk-4-1.svg new file mode 100644 index 0000000..add8c15 --- /dev/null +++ b/_freeze/schedule/slides/23-nnets-other/figure-revealjs/unnamed-chunk-4-1.svg @@ -0,0 +1,304 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/_freeze/schedule/slides/23-nnets-other/figure-revealjs/unnamed-chunk-5-1.svg b/_freeze/schedule/slides/23-nnets-other/figure-revealjs/unnamed-chunk-5-1.svg new file mode 100644 index 0000000..273b252 --- /dev/null +++ b/_freeze/schedule/slides/23-nnets-other/figure-revealjs/unnamed-chunk-5-1.svg @@ -0,0 +1,303 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/_freeze/schedule/slides/23-nnets-other/figure-revealjs/unnamed-chunk-7-1.svg b/_freeze/schedule/slides/23-nnets-other/figure-revealjs/unnamed-chunk-7-1.svg new file mode 100644 index 0000000..74b8cfc --- /dev/null +++ b/_freeze/schedule/slides/23-nnets-other/figure-revealjs/unnamed-chunk-7-1.svg @@ -0,0 +1,358 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/_freeze/schedule/slides/23-nnets-other/figure-revealjs/unnamed-chunk-8-1.svg b/_freeze/schedule/slides/23-nnets-other/figure-revealjs/unnamed-chunk-8-1.svg new file mode 100644 index 0000000..4937e5c --- /dev/null +++ b/_freeze/schedule/slides/23-nnets-other/figure-revealjs/unnamed-chunk-8-1.svg @@ -0,0 +1,251 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/_freeze/schedule/slides/23-nnets-other/figure-revealjs/unnamed-chunk-9-1.svg b/_freeze/schedule/slides/23-nnets-other/figure-revealjs/unnamed-chunk-9-1.svg new file mode 100644 index 0000000..65899bc --- /dev/null +++ b/_freeze/schedule/slides/23-nnets-other/figure-revealjs/unnamed-chunk-9-1.svg @@ -0,0 +1,375 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/_freeze/site_libs/twitter-widget/widgets.js b/_freeze/site_libs/twitter-widget/widgets.js new file mode 100644 index 0000000..45a321a --- /dev/null +++ b/_freeze/site_libs/twitter-widget/widgets.js @@ -0,0 +1,8 @@ +Function&&Function.prototype&&Function.prototype.bind&&(/(MSIE ([6789]|10|11))|Trident/.test(navigator.userAgent)||(window.__twttr&&window.__twttr.widgets&&window.__twttr.widgets.loaded&&window.twttr.widgets.load&&window.twttr.widgets.load(),window.__twttr&&window.__twttr.widgets&&window.__twttr.widgets.init||function(t){function e(e){for(var n,i,o=e[0],s=e[1],a=0,c=[];a-1},forIn:i,isObject:s,isEmptyObject:a,toType:o,isType:function(t,e){return t==o(e)},toRealArray:u}},function(t,e){t.exports=window},function(t,e,n){var r=n(16),i=/(?:^|(?:https?:)?\/\/(?:www\.)?twitter\.com(?::\d+)?(?:\/intent\/(?:follow|user)\/?\?screen_name=|(?:\/#!)?\/))@?([\w]+)(?:\?|&|$)/i,o=/(?:^|(?:https?:)?\/\/(?:www\.)?twitter\.com(?::\d+)?\/(?:#!\/)?[\w_]+\/status(?:es)?\/)(\d+)/i,s=/^http(s?):\/\/(\w+\.)*twitter\.com([:/]|$)/i,a=/^http(s?):\/\/(ton|pbs)\.twimg\.com/,u=/^#?([^.,<>!\s/#\-()'"]+)$/,c=/twitter\.com(?::\d{2,4})?\/intent\/(\w+)/,d=/^https?:\/\/(?:www\.)?twitter\.com\/\w+\/timelines\/(\d+)/i,f=/^https?:\/\/(?:www\.)?twitter\.com\/i\/moments\/(\d+)/i,l=/^https?:\/\/(?:www\.)?twitter\.com\/(\w+)\/(?:likes|favorites)/i,h=/^https?:\/\/(?:www\.)?twitter\.com\/(\w+)\/lists\/([\w-%]+)/i,p=/^https?:\/\/(?:www\.)?twitter\.com\/i\/live\/(\d+)/i,m=/^https?:\/\/syndication\.twitter\.com\/settings/i,v=/^https?:\/\/(localhost|platform)\.twitter\.com(?::\d+)?\/widgets\/widget_iframe\.(.+)/i,g=/^https?:\/\/(?:www\.)?twitter\.com\/search\?q=(\w+)/i;function w(t){return"string"==typeof t&&i.test(t)&&RegExp.$1.length<=20}function y(t){if(w(t))return RegExp.$1}function b(t,e){var n=r.decodeURL(t);if(e=e||!1,n.screen_name=y(t),n.screen_name)return r.url("https://twitter.com/intent/"+(e?"follow":"user"),n)}function _(t){return"string"==typeof t&&u.test(t)}function E(t){return"string"==typeof t&&o.test(t)}t.exports={isHashTag:_,hashTag:function(t,e){if(e=void 0===e||e,_(t))return(e?"#":"")+RegExp.$1},isScreenName:w,screenName:y,isStatus:E,status:function(t){return E(t)&&RegExp.$1},intentForProfileURL:b,intentForFollowURL:function(t){return b(t,!0)},isTwitterURL:function(t){return s.test(t)},isTwimgURL:function(t){return a.test(t)},isIntentURL:function(t){return c.test(t)},isSettingsURL:function(t){return m.test(t)},isWidgetIframeURL:function(t){return v.test(t)},isSearchUrl:function(t){return g.test(t)},regexen:{profile:i},momentId:function(t){return f.test(t)&&RegExp.$1},collectionId:function(t){return d.test(t)&&RegExp.$1},intentType:function(t){return c.test(t)&&RegExp.$1},likesScreenName:function(t){return l.test(t)&&RegExp.$1},listScreenNameAndSlug:function(t){var e,n,r;if(h.test(t)){e=RegExp.$1,n=RegExp.$2;try{r=decodeURIComponent(n)}catch(t){}return{ownerScreenName:e,slug:r||n}}return!1},eventId:function(t){return p.test(t)&&RegExp.$1}}},function(t,e,n){var r=n(6);t.exports=function(){var t=this;this.promise=new r(function(e,n){t.resolve=e,t.reject=n})}},function(t,e){t.exports=document},function(t,e,n){var r=n(0),i=[!0,1,"1","on","ON","true","TRUE","yes","YES"],o=[!1,0,"0","off","OFF","false","FALSE","no","NO"];function s(t){return void 0!==t&&null!==t&&""!==t}function a(t){return c(t)&&t%1==0}function u(t){return c(t)&&!a(t)}function c(t){return s(t)&&!isNaN(t)}function d(t){return r.contains(o,t)}function f(t){return r.contains(i,t)}t.exports={hasValue:s,isInt:a,isFloat:u,isNumber:c,isString:function(t){return"string"===r.toType(t)},isArray:function(t){return s(t)&&"array"==r.toType(t)},isTruthValue:f,isFalseValue:d,asInt:function(t){if(a(t))return parseInt(t,10)},asFloat:function(t){if(u(t))return t},asNumber:function(t){if(c(t))return t},asBoolean:function(t){return!(!s(t)||!f(t)&&(d(t)||!t))}}},function(t,e,n){var r=n(1),i=n(20),o=n(45);i.hasPromiseSupport()||(r.Promise=o),t.exports=r.Promise},function(t,e,n){var r=n(0);t.exports=function(t,e){var n=Array.prototype.slice.call(arguments,2);return function(){var i=r.toRealArray(arguments);return t.apply(e,n.concat(i))}}},function(t,e){t.exports=location},function(t,e,n){var r=n(47);t.exports=new r("__twttr")},function(t,e,n){var r=n(0),i=/\b([\w-_]+)\b/g;function o(t){return new RegExp("\\b"+t+"\\b","g")}function s(t,e){t.classList?t.classList.add(e):o(e).test(t.className)||(t.className+=" "+e)}function a(t,e){t.classList?t.classList.remove(e):t.className=t.className.replace(o(e)," ")}function u(t,e){return t.classList?t.classList.contains(e):r.contains(c(t),e)}function c(t){return r.toRealArray(t.classList?t.classList:t.className.match(i))}t.exports={add:s,remove:a,replace:function(t,e,n){if(t.classList&&u(t,e))return a(t,e),void s(t,n);t.className=t.className.replace(o(e),n)},toggle:function(t,e,n){return void 0===n&&t.classList&&t.classList.toggle?t.classList.toggle(e,n):(n?s(t,e):a(t,e),n)},present:u,list:c}},function(t,e,n){var r=n(19),i=n(5),o=n(16),s=n(0),a=n(108);t.exports=function(t){var e=t.href&&t.href.split("?")[1],n=e?o.decode(e):{},u={lang:a(t),width:t.getAttribute("data-width")||t.getAttribute("width"),height:t.getAttribute("data-height")||t.getAttribute("height"),related:t.getAttribute("data-related"),partner:t.getAttribute("data-partner")};return i.asBoolean(t.getAttribute("data-dnt"))&&r.setOn(),s.forIn(u,function(t,e){var r=n[t];n[t]=i.hasValue(r)?r:e}),s.compact(n)}},function(t,e,n){var r=n(109),i=n(21);t.exports=function(){var t="data-twitter-extracted-"+i.generate();return function(e,n){return r(e,n).filter(function(e){return!e.hasAttribute(t)}).map(function(e){return e.setAttribute(t,"true"),e})}}},function(t,e){function n(t,e,n,r,i,o,s){this.factory=t,this.Sandbox=e,this.srcEl=o,this.targetEl=i,this.parameters=r,this.className=n,this.options=s}n.prototype.destroy=function(){this.srcEl=this.targetEl=null},t.exports=n},function(t,e){t.exports={DM_BUTTON:"twitter-dm-button",FOLLOW_BUTTON:"twitter-follow-button",HASHTAG_BUTTON:"twitter-hashtag-button",MENTION_BUTTON:"twitter-mention-button",MOMENT:"twitter-moment",PERISCOPE:"periscope-on-air",SHARE_BUTTON:"twitter-share-button",TIMELINE:"twitter-timeline",TWEET:"twitter-tweet"}},function(t,e,n){var r=n(6),i=n(19),o=n(50),s=n(27),a=n(5),u=n(0);t.exports=function(t,e,n){var c;return t=t||[],e=e||{},c="ฦ’("+t.join(", ")+", target, [options]);",function(){var d,f,l,h,p=Array.prototype.slice.apply(arguments,[0,t.length]),m=Array.prototype.slice.apply(arguments,[t.length]);return m.forEach(function(t){t&&(t.nodeType!==Node.ELEMENT_NODE?u.isType("function",t)?d=t:u.isType("object",t)&&(f=t):l=t)}),p.length!==t.length||0===m.length?(d&&u.async(function(){d(!1)}),r.reject(new Error("Not enough parameters. Expected: "+c))):l?(f=u.aug({},f||{},e),t.forEach(function(t){f[t]=p.shift()}),a.asBoolean(f.dnt)&&i.setOn(),h=s.getExperiments().then(function(t){return o.addWidget(n(f,l,void 0,t))}),d&&h.then(d,function(){d(!1)}),h):(d&&u.async(function(){d(!1)}),r.reject(new Error("No target element specified. Expected: "+c)))}}},function(t,e,n){var r=n(5),i=n(0);function o(t){return encodeURIComponent(t).replace(/\+/g,"%2B").replace(/'/g,"%27")}function s(t){return decodeURIComponent(t)}function a(t){var e=[];return i.forIn(t,function(t,n){var s=o(t);i.isType("array",n)||(n=[n]),n.forEach(function(t){r.hasValue(t)&&e.push(s+"="+o(t))})}),e.sort().join("&")}function u(t){var e={};return t?(t.split("&").forEach(function(t){var n=t.split("="),r=s(n[0]),o=s(n[1]);if(2==n.length){if(!i.isType("array",e[r]))return r in e?(e[r]=[e[r]],void e[r].push(o)):void(e[r]=o);e[r].push(o)}}),e):{}}t.exports={url:function(t,e){return a(e).length>0?i.contains(t,"?")?t+"&"+a(e):t+"?"+a(e):t},decodeURL:function(t){var e=t&&t.split("?");return 2==e.length?u(e[1]):{}},decode:u,encode:a,encodePart:o,decodePart:s}},function(t,e,n){var r=n(90),i=n(3),o=n(0);function s(t,e){return function(){try{e.resolve(t.call(this))}catch(t){e.reject(t)}}}t.exports={sync:function(t,e){t.call(e)},read:function(t,e){var n=new i;return r.read(s(t,n),e),n.promise},write:function(t,e){var n=new i;return r.write(s(t,n),e),n.promise},defer:function(t,e,n){var a=new i;return o.isType("function",t)&&(n=e,e=t,t=1),r.defer(t,s(e,a),n),a.promise}}},function(t,e,n){var r=n(8),i=n(1),o=n(0),s={},a=o.contains(r.href,"tw_debug=true");function u(){}function c(){}function d(){return i.performance&&+i.performance.now()||+new Date}function f(t,e){if(i.console&&i.console[t])switch(e.length){case 1:i.console[t](e[0]);break;case 2:i.console[t](e[0],e[1]);break;case 3:i.console[t](e[0],e[1],e[2]);break;case 4:i.console[t](e[0],e[1],e[2],e[3]);break;case 5:i.console[t](e[0],e[1],e[2],e[3],e[4]);break;default:0!==e.length&&i.console.warn&&i.console.warn("too many params passed to logger."+t)}}t.exports={devError:u,devInfo:c,devObject:function(t,e){},publicError:function(){f("error",o.toRealArray(arguments))},publicLog:function(){f("info",o.toRealArray(arguments))},publicWarn:function(){f("warn",o.toRealArray(arguments))},time:function(t){a&&(s[t]=d())},timeEnd:function(t){a&&s[t]&&(d(),s[t])}}},function(t,e,n){var r=n(4),i=n(8),o=n(38),s=n(94),a=n(5),u=n(34),c=!1,d=/https?:\/\/([^/]+).*/i;t.exports={setOn:function(){c=!0},enabled:function(t,e){return!!(c||a.asBoolean(u.val("dnt"))||s.isUrlSensitive(e||i.host)||o.isFramed()&&s.isUrlSensitive(o.rootDocumentLocation())||(t=d.test(t||r.referrer)&&RegExp.$1)&&s.isUrlSensitive(t))}}},function(t,e,n){var r=n(4),i=n(83),o=n(1),s=n(0),a=i.userAgent;function u(t){return/(Trident|MSIE|Edge[/ ]?\d)/.test(t=t||a)}t.exports={retina:function(t){return(t=t||o).devicePixelRatio?t.devicePixelRatio>=1.5:!!t.matchMedia&&t.matchMedia("only screen and (min-resolution: 144dpi)").matches},anyIE:u,ie9:function(t){return/MSIE 9/.test(t=t||a)},ie10:function(t){return/MSIE 10/.test(t=t||a)},ios:function(t){return/(iPad|iPhone|iPod)/.test(t=t||a)},android:function(t){return/^Mozilla\/5\.0 \(Linux; (U; )?Android/.test(t=t||a)},canPostMessage:function(t,e){return t=t||o,e=e||a,t.postMessage&&!(u(e)&&t.opener)},touch:function(t,e,n){return t=t||o,e=e||i,n=n||a,"ontouchstart"in t||/Opera Mini/.test(n)||e.msMaxTouchPoints>0},cssTransitions:function(){var t=r.body.style;return void 0!==t.transition||void 0!==t.webkitTransition||void 0!==t.mozTransition||void 0!==t.oTransition||void 0!==t.msTransition},hasPromiseSupport:function(){return!!(o.Promise&&o.Promise.resolve&&o.Promise.reject&&o.Promise.all&&o.Promise.race&&(new o.Promise(function(e){t=e}),s.isType("function",t)));var t},hasIntersectionObserverSupport:function(){return!!o.IntersectionObserver},hasPerformanceInformation:function(){return o.performance&&o.performance.getEntriesByType}}},function(t,e){var n="i",r=0,i=0;t.exports={generate:function(){return n+String(+new Date)+Math.floor(1e5*Math.random())+r++},deterministic:function(){return n+String(i++)}}},function(t,e,n){var r=n(46),i=n(49),o=n(0);t.exports=o.aug(r.get("events")||{},i.Emitter)},function(t,e,n){var r=n(6),i=n(3);function o(t,e){return t.then(e,e)}function s(t){return t instanceof r}t.exports={always:o,allResolved:function(t){var e;return void 0===t?r.reject(new Error("undefined is not an object")):Array.isArray(t)?(e=t.length)?new r(function(n,r){var i=0,o=[];function a(){(i+=1)===e&&(0===o.length?r():n(o))}function u(t){o.push(t),a()}t.forEach(function(t){s(t)?t.then(u,a):u(t)})}):r.resolve([]):r.reject(new Error("Type error"))},some:function(t){var e;return e=(t=t||[]).length,t=t.filter(s),e?e!==t.length?r.reject("non-Promise passed to .some"):new r(function(e,n){var r=0;function i(){(r+=1)===t.length&&n()}t.forEach(function(t){t.then(e,i)})}):r.reject("no promises passed to .some")},isPromise:s,allSettled:function(t){function e(){}return r.all((t||[]).map(function(t){return o(t,e)}))},timeout:function(t,e){var n=new i;return setTimeout(function(){n.reject(new Error("Promise timed out"))},e),t.then(function(t){n.resolve(t)},function(t){n.reject(t)}),n.promise}}},function(t,e,n){var r=n(1).JSON;t.exports={stringify:r.stringify||r.encode,parse:r.parse||r.decode}},function(t,e,n){var r=n(39),i=n(97),o=n(7);(r=Object.create(r)).build=o(r.build,null,i),t.exports=r},function(t,e,n){var r=n(49).makeEmitter();t.exports={emitter:r,START:"start",ALL_WIDGETS_RENDER_START:"all_widgets_render_start",ALL_WIDGETS_RENDER_END:"all_widgets_render_end",ALL_WIDGETS_AND_IMAGES_LOADED:"all_widgets_and_images_loaded"}},function(t,e,n){var r=n(102),i=n(24),o=n(1);function s(t){var e="";if(!t)return"";try{return e=i.stringify(t),o.btoa(e)}catch(t){return""}}function a(t){return r.settingsLoaded().then(function(e){return e[t]})}function u(){return a("features")}t.exports={encodeFeatures:s,shouldObtainCookieConsent:function(){return a("shouldObtainCookieConsent")},getExperiments:u,getExperiment:function(t){return u().then(function(e){if(!e[t])throw new Error("Experiment not found");return e[t]})},getHorizonSettings:function(){return r.settingsLoaded().then(function(t){return{features:t.features,sessionId:t.sessionId}})},getActiveExperimentDataString:function(){return u().then(function(t){return s(t)})},getExperimentKeys:function(){return u().then(function(t){return Object.keys(t)})},load:function(){r.load()}}},function(t,e,n){var r=n(9),i=n(103),o="https://syndication.twitter.com",s="https://platform.twitter.com",a=["https://syndication.twitter.com","https://cdn.syndication.twimg.com","https://localhost.twitter.com:8444"],u=["https://syndication.twitter.com","https://localhost.twitter.com:8445"],c=["https://platform.twitter.com","https://localhost.twitter.com",/^https:\/\/ton\.local\.twitter\.com\/syndication-internal\/embed-iframe\/[0-9A-Za-z_-]+\/app/],d=function(t,e){return t.some(function(t){return t instanceof RegExp?t.test(e):t===e})},f=function(){var t=r.get("backendHost");return t&&d(a,t)?t:"https://cdn.syndication.twimg.com"},l=function(){var t=r.get("settingsSvcHost");return t&&d(u,t)?t:o};function h(t,e){var n=[t];return e.forEach(function(t){n.push(function(t){var e=(t||"").toString(),n="/"===e.slice(0,1)?1:0,r=function(t){return"/"===t.slice(-1)}(e)?-1:void 0;return e.slice(n,r)}(t))}),n.join("/")}t.exports={cookieConsent:function(t){var e=t||[];return e.unshift("cookie/consent"),h(l(),e)},embedIframe:function(t,e){var n=t||[],o=s,a=r.get("embedIframeURL");return a&&d(c,a)?h(a,n)+".html":(n.unshift(i.getBaseURLPath(e)),h(o,n)+".html")},embedService:function(t){var e=t||[],n=o;return e.unshift("srv"),h(n,e)},eventVideo:function(t){var e=t||[];return e.unshift("video/event"),h(f(),e)},grid:function(t){var e=t||[];return e.unshift("grid/collection"),h(f(),e)},moment:function(t){var e=t||[];return e.unshift("moments"),h(f(),e)},settings:function(t){var e=t||[];return e.unshift("settings"),h(l(),e)},settingsScribe:function(t){var e=t||[];return e.unshift("i/jot/embeds"),h(l(),e)},timeline:function(t){var e=t||[];return e.unshift("timeline"),h(f(),e)},tweetBatch:function(t){var e=t||[];return e.unshift("tweets.json"),h(f(),e)},video:function(t){var e=t||[];return e.unshift("widgets/video"),h(f(),e)}}},function(t,e,n){var r=n(4),i=n(0);t.exports=function(t,e,n){var o;if(n=n||r,t=t||{},e=e||{},t.name){try{o=n.createElement('')}catch(e){(o=n.createElement("iframe")).name=t.name}delete t.name}else o=n.createElement("iframe");return t.id&&(o.id=t.id,delete t.id),o.allowtransparency="true",o.scrolling="no",o.setAttribute("frameBorder",0),o.setAttribute("allowTransparency",!0),i.forIn(t,function(t,e){o.setAttribute(t,e)}),i.forIn(e,function(t,e){o.style[t]=e}),o}},function(t,e,n){var r=n(25),i=n(100);t.exports=r.build([i])},function(t,e,n){var r=n(25),i=n(116);t.exports=r.build([i])},function(t,e,n){var r=n(39),i=n(40),o=n(7);(r=Object.create(r)).build=o(r.build,null,i),t.exports=r},function(t,e,n){var r=n(70),i=n(19),o=n(5),s=n(21),a=n(0);t.exports=function(t){t.params({chrome:{},height:{transform:o.asInt},instanceId:{required:!0,fallback:s.deterministic},isPreconfigured:{},lang:{required:!0,transform:r.matchLanguage,fallback:"en"},theme:{},tweetLimit:{transform:o.asInt}}),t.defineProperty("endpoint",{get:function(){throw new Error("endpoint not specified")}}),t.define("queryParams",function(){return a.compact({dnt:i.enabled(),limit:this.params.tweetLimit,lang:this.params.lang})})}},function(t,e,n){var r,i=n(4);function o(t){var e,n,o,s=0;for(r={},e=(t=t||i).getElementsByTagName("meta");e[s];s++){if(n=e[s],/^twitter:/.test(n.getAttribute("name")))o=n.getAttribute("name").replace(/^twitter:/,"");else{if(!/^twitter:/.test(n.getAttribute("property")))continue;o=n.getAttribute("property").replace(/^twitter:/,"")}r[o]=n.getAttribute("content")||n.getAttribute("value")}}o(),t.exports={init:o,val:function(t){return r[t]}}},function(t,e,n){var r=n(10),i={},o=-1,s={};function a(t){var e=t.getAttribute("data-twitter-event-id");return e||(t.setAttribute("data-twitter-event-id",++o),o)}function u(t,e,n){var r=0,i=t&&t.length||0;for(r=0;r1?(e=Math.floor(t.item_ids.length/2),n=t.item_ids.slice(0,e),r={},i=t.item_ids.slice(e),o={},n.forEach(function(e){r[e]=t.item_details[e]}),i.forEach(function(e){o[e]=t.item_details[e]}),[f.aug({},t,{item_ids:n,item_details:r}),f.aug({},t,{item_ids:i,item_details:o})]):[t]},stringify:function(t){var e,n=Array.prototype.toJSON;return delete Array.prototype.toJSON,e=u.stringify(t),n&&(Array.prototype.toJSON=n),e},CLIENT_EVENT_ENDPOINT:p,RUFOUS_REDIRECT:"https://platform.twitter.com/jot.html"}},function(t,e,n){var r=n(8),i=n(73),o=n(0),s=i.getCanonicalURL()||r.href,a=s;t.exports={isFramed:function(){return s!==a},rootDocumentLocation:function(t){return t&&o.isType("string",t)&&(s=t),s},currentDocumentLocation:function(){return a}}},function(t,e,n){var r=n(95),i=n(96),o=n(0);t.exports={couple:function(){return o.toRealArray(arguments)},build:function(t,e,n){var o=new t;return(e=i(r(e||[]))).forEach(function(t){t.call(null,o)}),o.build(n)}}},function(t,e,n){var r=n(98),i=n(0),o=n(41);function s(){this.Component=this.factory(),this._adviceArgs=[],this._lastArgs=[]}i.aug(s.prototype,{factory:o,build:function(t){var e=this;return this.Component,i.aug(this.Component.prototype.boundParams,t),this._adviceArgs.concat(this._lastArgs).forEach(function(t){(function(t,e,n){var r=this[e];if(!r)throw new Error(e+" does not exist");this[e]=t(r,n)}).apply(e.Component.prototype,t)}),delete this._lastArgs,delete this._adviceArgs,this.Component},params:function(t){var e=this.Component.prototype.paramConfigs;t=t||{},this.Component.prototype.paramConfigs=i.aug({},t,e)},define:function(t,e){if(t in this.Component.prototype)throw new Error(t+" has previously been defined");this.override(t,e)},defineStatic:function(t,e){this.Component[t]=e},override:function(t,e){this.Component.prototype[t]=e},defineProperty:function(t,e){if(t in this.Component.prototype)throw new Error(t+" has previously been defined");this.overrideProperty(t,e)},overrideProperty:function(t,e){var n=i.aug({configurable:!0},e);Object.defineProperty(this.Component.prototype,t,n)},before:function(t,e){this._adviceArgs.push([r.before,t,e])},after:function(t,e){this._adviceArgs.push([r.after,t,e])},around:function(t,e){this._adviceArgs.push([r.around,t,e])},last:function(t,e){this._lastArgs.push([r.after,t,e])}}),t.exports=s},function(t,e,n){var r=n(0);function i(){return!0}function o(t){return t}t.exports=function(){function t(t){var e=this;t=t||{},this.params=Object.keys(this.paramConfigs).reduce(function(n,s){var a=[],u=e.boundParams,c=e.paramConfigs[s],d=c.validate||i,f=c.transform||o;if(s in u&&a.push(u[s]),s in t&&a.push(t[s]),a="fallback"in c?a.concat(c.fallback):a,n[s]=function(t,e,n){var i=null;return t.some(function(t){if(t=r.isType("function",t)?t():t,e(t))return i=n(t),!0}),i}(a,d,f),c.required&&null==n[s])throw new Error(s+" is a required parameter");return n},{}),this.initialize()}return r.aug(t.prototype,{paramConfigs:{},boundParams:{},initialize:function(){}}),t}},function(t){t.exports={version:"aaf4084522e3a:1674595607486"}},function(t,e,n){var r=n(1);t.exports=function(t,e,n){var i,o=0;return n=n||null,function s(){var a=n||this,u=arguments,c=+new Date;if(r.clearTimeout(i),c-o>e)return o=c,void t.apply(a,u);i=r.setTimeout(function(){s.apply(a,u)},e)}}},function(t,e,n){var r=n(18);t.exports=function(){r.publicError("Error: This Timeline type is no longer supported (Likes, Collections, & Moments). \n\t","* Twitter still supports Profile and List Timelines \n\t","* You can learn more about this change in our announcement: \n\t","https://twittercommunity.com/t/removing-support-for-embedded-like-collection-and-moment-timelines/150313 \n\t","* In order to create a new Embedded Timeline, visit: https://publish.twitter.com")}},function(t,e,n){ +/*! + * @overview es6-promise - a tiny implementation of Promises/A+. + * @copyright Copyright (c) 2014 Yehuda Katz, Tom Dale, Stefan Penner and contributors (Conversion to ES6 API by Jake Archibald) + * @license Licensed under MIT license + * See https://raw.githubusercontent.com/stefanpenner/es6-promise/master/LICENSE + * @version v4.2.5+7f2b526d + */var r;r=function(){"use strict";function t(t){return"function"==typeof t}var e=Array.isArray?Array.isArray:function(t){return"[object Array]"===Object.prototype.toString.call(t)},n=0,r=void 0,i=void 0,o=function(t,e){l[n]=t,l[n+1]=e,2===(n+=2)&&(i?i(h):w())},s="undefined"!=typeof window?window:void 0,a=s||{},u=a.MutationObserver||a.WebKitMutationObserver,c="undefined"==typeof self&&"undefined"!=typeof process&&"[object process]"==={}.toString.call(process),d="undefined"!=typeof Uint8ClampedArray&&"undefined"!=typeof importScripts&&"undefined"!=typeof MessageChannel;function f(){var t=setTimeout;return function(){return t(h,1)}}var l=new Array(1e3);function h(){for(var t=0;t=0&&this._handlers[t].splice(n,1):this._handlers[t]=[])},trigger:function(t,e){var n=this._handlers&&this._handlers[t];(e=e||{}).type=t,n&&n.forEach(function(t){r.async(i(t,this,e))})}};t.exports={Emitter:o,makeEmitter:function(){return r.aug(function(){},o)}}},function(t,e,n){var r=n(89),i=n(91),o=n(6),s=n(23),a=n(7),u=n(0),c=new i(function(t){var e=function(t){return t.reduce(function(t,e){return t[e._className]=t[e._className]||[],t[e._className].push(e),t},{})}(t.map(r.fromRawTask));u.forIn(e,function(t,e){s.allSettled(e.map(function(t){return t.initialize()})).then(function(){e.forEach(function(t){o.all([t.hydrate(),t.insertIntoDom()]).then(a(t.render,t)).then(a(t.success,t),a(t.fail,t))})})})});t.exports={addWidget:function(t){return c.add(t)}}},function(t,e,n){var r=n(17);t.exports=function(t){return r.write(function(){t&&t.parentNode&&t.parentNode.removeChild(t)})}},function(t,e,n){var r=n(93),i=n(71),o=new(n(105))(function(t){(!function(t){return 1===t.length&&i.canFlushOneItem(t[0])}(t)?function(t){r.init(),t.forEach(function(t){var e=t.input.namespace,n=t.input.data,i=t.input.offsite,o=t.input.version;r.clientEvent(e,n,i,o)}),r.flush().then(function(){t.forEach(function(t){t.taskDoneDeferred.resolve()})},function(){t.forEach(function(t){t.taskDoneDeferred.reject()})})}:function(t){t.forEach(function(t){var e=t.input.namespace,n=t.input.data,r=t.input.offsite,o=t.input.version;i.clientEvent(e,n,r,o),t.taskDoneDeferred.resolve()})})(t)});t.exports={scribe:function(t,e,n,r){return o.add({namespace:t,data:e,offsite:n,version:r})},pause:function(){o.pause()},resume:function(){o.resume()}}},function(t,e,n){n(18),t.exports={log:function(t,e){}}},function(t,e,n){var r=n(1);function i(t){return(t=t||r).getSelection&&t.getSelection()}t.exports={getSelection:i,getSelectedText:function(t){var e=i(t);return e?e.toString():""}}},function(t,e,n){var r=n(4),i=n(1),o=n(3),s=2e4;t.exports=function(t){var e=new o,n=r.createElement("img");return n.onload=n.onerror=function(){i.setTimeout(e.resolve,50)},n.src=t,i.setTimeout(e.reject,s),e.promise}},function(t,e,n){var r,i=n(10),o=n(4),s=n(1),a=n(34),u=n(51),c=n(5),d=n(21),f="csptest";t.exports={inlineStyle:function(){var t=f+d.generate(),e=o.createElement("div"),n=o.createElement("style"),l="."+t+" { visibility: hidden; }";return!!o.body&&(c.asBoolean(a.val("widgets:csp"))&&(r=!1),void 0!==r?r:(e.style.display="none",i.add(e,t),n.type="text/css",n.appendChild(o.createTextNode(l)),o.body.appendChild(n),o.body.appendChild(e),r="hidden"===s.getComputedStyle(e).visibility,u(e),u(n),r))}}},function(t,e){t.exports=function(t){var e=t.getBoundingClientRect();return{width:e.width,height:e.height}}},function(t,e,n){var r=n(101);t.exports=function(t){t.define("createElement",r),t.define("createFragment",r),t.define("htmlToElement",r),t.define("hasSelectedText",r),t.define("addRootClass",r),t.define("removeRootClass",r),t.define("hasRootClass",r),t.define("prependStyleSheet",r),t.define("appendStyleSheet",r),t.define("prependCss",r),t.define("appendCss",r),t.define("makeVisible",r),t.define("injectWidgetEl",r),t.define("matchHeightToContent",r),t.define("matchWidthToContent",r)}},function(t,e){t.exports=function(t){var e,n=!1;return function(){return n?e:(n=!0,e=t.apply(this,arguments))}}},function(t,e,n){var r=n(13),i=n(110),o=n(111),s=n(14);t.exports=function(t,e,n){return new r(i,o,s.DM_BUTTON,t,e,n)}},function(t,e,n){var r=n(25),i=n(112);t.exports=r.build([i])},function(t,e,n){var r=n(13),i=n(115),o=n(31),s=n(14);t.exports=function(t,e,n){return new r(i,o,s.FOLLOW_BUTTON,t,e,n)}},function(t,e,n){var r=n(13),i=n(123),o=n(30),s=n(14);t.exports=function(t,e,n){return new r(i,o,s.MOMENT,t,e,n)}},function(t,e,n){var r=n(13),i=n(125),o=n(30),s=n(14);t.exports=function(t,e,n){return new r(i,o,s.PERISCOPE,t,e,n)}},function(t,e,n){var r=n(127),i=n(128),o=n(132),s=n(134),a=n(136),u={collection:i,likes:o,list:s,profile:a,url:d},c=[a,o,i,s];function d(t){return r(c,function(e){try{return new e(t)}catch(t){}})}t.exports=function(t){return t?function(t){var e,n;return e=(t.sourceType+"").toLowerCase(),(n=u[e])?new n(t):null}(t)||d(t):null}},function(t,e,n){var r=n(4),i=n(13),o=n(31),s=n(138),a=n(14);t.exports=function(t,e,n){var u=r.createElement("div");return new i(s,o,a.TIMELINE,t,e,n,{sandboxWrapperEl:u})}},function(t,e,n){var r=n(4),i=n(13),o=n(31),s=n(140),a=n(14);t.exports=function(t,e,n){return new i(s,o,a.TWEET,t,e,n,{sandboxWrapperEl:r.createElement("div")})}},function(t,e,n){var r=n(13),i=n(142),o=n(31),s=n(14);t.exports=function(t,e,n){var a=t&&t.type||"share",u="hashtag"==a?s.HASHTAG_BUTTON:"mention"==a?s.MENTION_BUTTON:s.SHARE_BUTTON;return new r(i,o,u,t,e,n)}},function(t,e,n){var r=n(52),i=n(38),o=n(0);t.exports=function(t){var e={widget_origin:i.rootDocumentLocation(),widget_frame:i.isFramed()?i.currentDocumentLocation():null,duration_ms:t.duration,item_ids:t.widgetIds||[]},n=o.aug(t.namespace,{page:"page",component:"performance"});r.scribe(n,e)}},function(t,e,n){var r=n(0),i=n(129),o=["ar","fa","he","ur"];t.exports={isRtlLang:function(t){return t=String(t).toLowerCase(),r.contains(o,t)},matchLanguage:function(t){return t=(t=(t||"").toLowerCase()).replace("_","-"),i(t)?t:(t=t.replace(/-.*/,""),i(t)?t:"en")}}},function(t,e,n){var r=n(53),i=n(16),o=n(37),s=n(27),a=n(28),u=n(0),c=n(9).get("scribeCallback"),d=2083,f=[],l=i.url(o.CLIENT_EVENT_ENDPOINT,{dnt:0,l:""}),h=encodeURIComponent(l).length;function p(t,e,n,r,i){var d=!u.isObject(t),f=!!e&&!u.isObject(e);if(!d&&!f)return c&&c(arguments),s.getExperiments().then(function(s){s&&s.tfw_duplicate_scribes_to_settings&&"on"===s.tfw_duplicate_scribes_to_settings.bucket?((e=e||{}).context="rufous-eol",m(o.formatClientEventNamespace(t),o.formatClientEventData(e,n,r),a.settingsScribe(),i)):m(o.formatClientEventNamespace(t),o.formatClientEventData(e,n,r),o.CLIENT_EVENT_ENDPOINT,i)})}function m(t,e,n,s){var a;n&&u.isObject(t)&&u.isObject(e)&&(r.log(t,e),a=o.flattenClientEventPayload(t,e),s=u.aug({},s,{l:o.stringify(a)}),a.dnt&&(s.dnt=1),g(i.url(n,s)))}function v(t){return f.push(t),f}function g(t){return(new Image).src=t}t.exports={canFlushOneItem:function(t){var e=o.stringify(t),n=encodeURIComponent(e).length+3;return h+n")}).then(function(){t.close(),a.resolve(c)})}),c.src=["javascript:",'document.write("");',"try { window.parent.document; }",'catch (e) { document.domain="'+r.domain+'"; }',"window.parent."+g.fullPath(["sandbox",u])+"();"].join(""),c.addEventListener("error",a.reject,!1),o.write(function(){i.parentNode.replaceChild(c,i)}),a.promise}t.exports=a.couple(n(58),function(t){t.overrideProperty("id",{get:function(){return this.sandboxEl&&this.sandboxEl.id}}),t.overrideProperty("initialized",{get:function(){return!!this.win}}),t.overrideProperty("width",{get:function(){return this._width}}),t.overrideProperty("height",{get:function(){return this._height}}),t.overrideProperty("sandboxEl",{get:function(){return this.iframeEl}}),t.defineProperty("iframeEl",{get:function(){return this._iframe}}),t.defineProperty("rootEl",{get:function(){return this.doc&&this.doc.documentElement}}),t.defineProperty("widgetEl",{get:function(){return this.doc&&this.doc.body.firstElementChild}}),t.defineProperty("win",{get:function(){return this.iframeEl&&this.iframeEl.contentWindow}}),t.defineProperty("doc",{get:function(){return this.win&&this.win.document}}),t.define("_updateCachedDimensions",function(){var t=this;return o.read(function(){var e,n=h(t.sandboxEl);"visible"==t.sandboxEl.style.visibility?t._width=n.width:(e=h(t.sandboxEl.parentElement).width,t._width=Math.min(n.width,e)),t._height=n.height})}),t.define("_setTargetToBlank",function(){var t=this.createElement("base");t.target="_blank",this.doc.head.appendChild(t)}),t.define("_didResize",function(){var t=this,e=this._resizeHandlers.slice(0);return this._updateCachedDimensions().then(function(){e.forEach(function(e){e(t)})})}),t.define("setTitle",function(t){this.iframeEl.title=t}),t.override("createElement",function(t){return this.doc.createElement(t)}),t.override("createFragment",function(){return this.doc.createDocumentFragment()}),t.override("htmlToElement",function(t){var e;return(e=this.createElement("div")).innerHTML=t,e.firstElementChild}),t.override("hasSelectedText",function(){return!!s.getSelectedText(this.win)}),t.override("addRootClass",function(t){var e=this.rootEl;return t=Array.isArray(t)?t:[t],this.initialized?o.write(function(){t.forEach(function(t){i.add(e,t)})}):m.reject(new Error("sandbox not initialized"))}),t.override("removeRootClass",function(t){var e=this.rootEl;return t=Array.isArray(t)?t:[t],this.initialized?o.write(function(){t.forEach(function(t){i.remove(e,t)})}):m.reject(new Error("sandbox not initialized"))}),t.override("hasRootClass",function(t){return i.present(this.rootEl,t)}),t.define("addStyleSheet",function(t,e){var n,r=new p;return this.initialized?((n=this.createElement("link")).type="text/css",n.rel="stylesheet",n.href=t,n.addEventListener("load",r.resolve,!1),n.addEventListener("error",r.reject,!1),o.write(y(e,null,n)).then(function(){return u(t).then(r.resolve,r.reject),r.promise})):m.reject(new Error("sandbox not initialized"))}),t.override("prependStyleSheet",function(t){var e=this.doc;return this.addStyleSheet(t,function(t){var n=e.head.firstElementChild;return n?e.head.insertBefore(t,n):e.head.appendChild(t)})}),t.override("appendStyleSheet",function(t){var e=this.doc;return this.addStyleSheet(t,function(t){return e.head.appendChild(t)})}),t.define("addCss",function(t,e){var n;return c.inlineStyle()?((n=this.createElement("style")).type="text/css",n.appendChild(this.doc.createTextNode(t)),o.write(y(e,null,n))):(l.devError("CSP enabled; cannot embed inline styles"),m.resolve())}),t.override("prependCss",function(t){var e=this.doc;return this.addCss(t,function(t){var n=e.head.firstElementChild;return n?e.head.insertBefore(t,n):e.head.appendChild(t)})}),t.override("appendCss",function(t){var e=this.doc;return this.addCss(t,function(t){return e.head.appendChild(t)})}),t.override("makeVisible",function(){var t=this;return this.styleSelf(E).then(function(){t._updateCachedDimensions()})}),t.override("injectWidgetEl",function(t){var e=this;return this.initialized?this.widgetEl?m.reject(new Error("widget already injected")):o.write(function(){e.doc.body.appendChild(t)}):m.reject(new Error("sandbox not initialized"))}),t.override("matchHeightToContent",function(){var t,e=this;return o.read(function(){t=e.widgetEl?h(e.widgetEl).height:0}),o.write(function(){e.sandboxEl.style.height=t+"px"}).then(function(){return e._updateCachedDimensions()})}),t.override("matchWidthToContent",function(){var t,e=this;return o.read(function(){t=e.widgetEl?h(e.widgetEl).width:0}),o.write(function(){e.sandboxEl.style.width=t+"px"}).then(function(){return e._updateCachedDimensions()})}),t.after("initialize",function(){this._iframe=null,this._width=this._height=0,this._resizeHandlers=[]}),t.override("insert",function(t,e,n,r){var i=this,s=new p,a=this.targetGlobal.document,u=A(t,e,n,a);return o.write(y(r,null,u)),u.addEventListener("load",function(){(function(t){try{t.contentWindow.document}catch(t){return m.reject(t)}return m.resolve(t)})(u).then(null,y(R,null,t,e,n,u,a)).then(s.resolve,s.reject)},!1),u.addEventListener("error",s.reject,!1),s.promise.then(function(t){var e=d(i._didResize,S,i);return i._iframe=t,i.win.addEventListener("resize",e,!1),m.all([i._setTargetToBlank(),i.addRootClass(x),i.prependCss(T)])})}),t.override("onResize",function(t){this._resizeHandlers.push(t)}),t.after("styleSelf",function(){return this._updateCachedDimensions()})})},function(t,e){t.exports=function(){throw new Error("unimplemented method")}},function(t,e,n){var r=n(72),i=n(29),o=n(3),s=n(4),a=n(28),u=n(20),c=n(24),d=n(8),f=n(18),l=n(104),h=n(59),p=n(9),m=n(16),v=n(2),g=n(0),w=n(1),y=h(function(){return new o}),b={shouldObtainCookieConsent:!1,features:{}};t.exports={load:function(){var t,e,n,o;if(u.ie9()||u.ie10()||"http:"!==d.protocol&&"https:"!==d.protocol)return f.devError("Using default settings due to unsupported browser or protocol."),void y().resolve();t={origin:d.origin},a.settings().indexOf("localhost")>-1&&(t.localSettings=!0),e=m.url(r.resourceBaseUrl+r.widgetIframeHtmlPath,t),n=function(t){var n,r,i,o;if(r=v.isTwitterURL(t.origin),i=e.substr(0,t.origin.length)===t.origin,o=v.isTwimgURL(t.origin),i&&r||o)try{(n="string"==typeof t.data?c.parse(t.data):t.data).namespace===l.settings&&(b=g.aug(b,{features:n.settings.features,sessionId:n.sessionId}),y().resolve())}catch(t){f.devError(t)}},w.addEventListener("message",n),o=i({src:e,title:"Twitter settings iframe"},{display:"none"}),s.body.appendChild(o)},settingsLoaded:function(){var t,e;return t=p.get("experimentOverride"),y().promise.then(function(){return t&&t.name&&t.assignment&&((e={})[t.name]={bucket:t.assignment},b.features=g.aug(b.features,e)),b})}}},function(t,e){t.exports={getBaseURLPath:function(t){switch(t&&t.tfw_team_holdback_11929&&t.tfw_team_holdback_11929.bucket){case"control":return"embed-holdback";case"holdback_prod":return"embed-holdback-prod";default:return"embed"}}}},function(t,e){t.exports={settings:"twttr.settings"}},function(t,e,n){var r=n(3),i=n(7),o=100,s=3e3;function a(t,e){this._inputsQueue=[],this._task=t,this._isPaused=!1,this._flushDelay=e&&e.flushDelay||o,this._pauseLength=e&&e.pauseLength||s,this._flushTimeout=void 0}a.prototype.add=function(t){var e=new r;return this._inputsQueue.push({input:t,taskDoneDeferred:e}),this._scheduleFlush(),e.promise},a.prototype._scheduleFlush=function(){this._isPaused||(clearTimeout(this._flushTimeout),this._flushTimeout=setTimeout(i(this._flush,this),this._flushDelay))},a.prototype._flush=function(){try{this._task.call(null,this._inputsQueue)}catch(t){this._inputsQueue.forEach(function(e){e.taskDoneDeferred.reject(t)})}this._inputsQueue=[],this._flushTimeout=void 0},a.prototype.pause=function(t){clearTimeout(this._flushTimeout),this._isPaused=!0,!t&&this._pauseLength&&setTimeout(i(this.resume,this),this._pauseLength)},a.prototype.resume=function(){this._isPaused=!1,this._scheduleFlush()},t.exports=a},function(t,e,n){t.exports=[n(107),n(114),n(122),n(124),n(126),n(139),n(141)]},function(t,e,n){var r=n(16),i=n(5),o=n(0),s=n(11),a=n(12)(),u=n(60),c="a.twitter-dm-button";t.exports=function(t){return a(t,c).map(function(t){return u(function(t){var e=t.getAttribute("data-show-screen-name"),n=s(t),a=t.getAttribute("href"),u=t.getAttribute("data-screen-name"),c=e?i.asBoolean(e):null,d=t.getAttribute("data-size"),f=r.decodeURL(a),l=f.recipient_id,h=t.getAttribute("data-text")||f.text,p=t.getAttribute("data-welcome-message-id")||f.welcomeMessageId;return o.aug(n,{screenName:u,showScreenName:c,size:d,text:h,userId:l,welcomeMessageId:p})}(t),t.parentNode,t)})}},function(t,e,n){var r=n(0);t.exports=function t(e){var n;if(e)return n=e.lang||e.getAttribute("data-lang"),r.isType("string",n)?n:t(e.parentElement)}},function(t,e,n){var r=n(0),i=n(48);t.exports=function(t,e){return i(t,e)?[t]:r.toRealArray(t.querySelectorAll(e))}},function(t,e,n){var r=n(3);t.exports=function(t,e){var i=new r;return n.e(1).then(function(r){var o;try{o=n(75),i.resolve(new o(t,e))}catch(t){i.reject(t)}}.bind(null,n)).catch(function(t){i.reject(t)}),i.promise}},function(t,e,n){var r=n(61),i=n(30);t.exports=r.isSupported()?r:i},function(t,e,n){var r=n(113),i=n(1),o=n(10),s=n(35),a=n(17),u=n(54),c=n(25),d=n(55),f=n(56),l=n(57),h=n(7),p=n(43),m=n(6),v=n(0),g=50,w={position:"absolute",visibility:"hidden",display:"block",transform:"rotate(0deg)"},y={position:"static",visibility:"visible"},b="twitter-widget",_="open",E="SandboxRoot",x=".SandboxRoot { display: none; max-height: 10000px; }";t.exports=c.couple(n(58),function(t){t.defineStatic("isSupported",function(){return!!i.HTMLElement.prototype.attachShadow&&f.inlineStyle()}),t.overrideProperty("id",{get:function(){return this.sandboxEl&&this.sandboxEl.id}}),t.overrideProperty("initialized",{get:function(){return!!this._shadowHost}}),t.overrideProperty("width",{get:function(){return this._width}}),t.overrideProperty("height",{get:function(){return this._height}}),t.overrideProperty("sandboxEl",{get:function(){return this._shadowHost}}),t.define("_updateCachedDimensions",function(){var t=this;return a.read(function(){var e,n=l(t.sandboxEl);"visible"==t.sandboxEl.style.visibility?t._width=n.width:(e=l(t.sandboxEl.parentElement).width,t._width=Math.min(n.width,e)),t._height=n.height})}),t.define("_didResize",function(){var t=this,e=this._resizeHandlers.slice(0);return this._updateCachedDimensions().then(function(){e.forEach(function(e){e(t)})})}),t.override("createElement",function(t){return this.targetGlobal.document.createElement(t)}),t.override("createFragment",function(){return this.targetGlobal.document.createDocumentFragment()}),t.override("htmlToElement",function(t){var e;return(e=this.createElement("div")).innerHTML=t,e.firstElementChild}),t.override("hasSelectedText",function(){return!!u.getSelectedText(this.targetGlobal)}),t.override("addRootClass",function(t){var e=this._shadowRootBody;return t=Array.isArray(t)?t:[t],this.initialized?a.write(function(){t.forEach(function(t){o.add(e,t)})}):m.reject(new Error("sandbox not initialized"))}),t.override("removeRootClass",function(t){var e=this._shadowRootBody;return t=Array.isArray(t)?t:[t],this.initialized?a.write(function(){t.forEach(function(t){o.remove(e,t)})}):m.reject(new Error("sandbox not initialized"))}),t.override("hasRootClass",function(t){return o.present(this._shadowRootBody,t)}),t.override("addStyleSheet",function(t,e){return this.addCss('@import url("'+t+'");',e).then(function(){return d(t)})}),t.override("prependStyleSheet",function(t){var e=this._shadowRoot;return this.addStyleSheet(t,function(t){var n=e.firstElementChild;return n?e.insertBefore(t,n):e.appendChild(t)})}),t.override("appendStyleSheet",function(t){var e=this._shadowRoot;return this.addStyleSheet(t,function(t){return e.appendChild(t)})}),t.override("addCss",function(t,e){var n;return this.initialized?f.inlineStyle()?((n=this.createElement("style")).type="text/css",n.appendChild(this.targetGlobal.document.createTextNode(t)),a.write(h(e,null,n))):m.resolve():m.reject(new Error("sandbox not initialized"))}),t.override("prependCss",function(t){var e=this._shadowRoot;return this.addCss(t,function(t){var n=e.firstElementChild;return n?e.insertBefore(t,n):e.appendChild(t)})}),t.override("appendCss",function(t){var e=this._shadowRoot;return this.addCss(t,function(t){return e.appendChild(t)})}),t.override("makeVisible",function(){return this.styleSelf(y)}),t.override("injectWidgetEl",function(t){var e=this;return this.initialized?this._shadowRootBody.firstElementChild?m.reject(new Error("widget already injected")):a.write(function(){e._shadowRootBody.appendChild(t)}).then(function(){return e._updateCachedDimensions()}).then(function(){var t=p(e._didResize,g,e);new r(e._shadowRootBody,t)}):m.reject(new Error("sandbox not initialized"))}),t.override("matchHeightToContent",function(){return m.resolve()}),t.override("matchWidthToContent",function(){return m.resolve()}),t.override("insert",function(t,e,n,r){var i=this.targetGlobal.document,o=this._shadowHost=i.createElement(b),u=this._shadowRoot=o.attachShadow({mode:_}),c=this._shadowRootBody=i.createElement("div");return v.forIn(e||{},function(t,e){o.setAttribute(t,e)}),o.id=t,u.appendChild(c),s.delegate(c,"click","A",function(t,e){e.hasAttribute("target")||e.setAttribute("target","_blank")}),m.all([this.styleSelf(w),this.addRootClass(E),this.prependCss(x),a.write(r.bind(null,o))])}),t.override("onResize",function(t){this._resizeHandlers.push(t)}),t.after("initialize",function(){this._shadowHost=this._shadowRoot=this._shadowRootBody=null,this._width=this._height=0,this._resizeHandlers=[]}),t.after("styleSelf",function(){return this._updateCachedDimensions()})})},function(t,e){var n;(n=function(t,e){function r(t,e){if(t.resizedAttached){if(t.resizedAttached)return void t.resizedAttached.add(e)}else t.resizedAttached=new function(){var t,e;this.q=[],this.add=function(t){this.q.push(t)},this.call=function(){for(t=0,e=this.q.length;t
',t.appendChild(t.resizeSensor),{fixed:1,absolute:1}[function(t,e){return t.currentStyle?t.currentStyle[e]:window.getComputedStyle?window.getComputedStyle(t,null).getPropertyValue(e):t.style[e]}(t,"position")]||(t.style.position="relative");var i,o,s=t.resizeSensor.childNodes[0],a=s.childNodes[0],u=t.resizeSensor.childNodes[1],c=(u.childNodes[0],function(){a.style.width=s.offsetWidth+10+"px",a.style.height=s.offsetHeight+10+"px",s.scrollLeft=s.scrollWidth,s.scrollTop=s.scrollHeight,u.scrollLeft=u.scrollWidth,u.scrollTop=u.scrollHeight,i=t.offsetWidth,o=t.offsetHeight});c();var d=function(t,e,n){t.attachEvent?t.attachEvent("on"+e,n):t.addEventListener(e,n)},f=function(){t.offsetWidth==i&&t.offsetHeight==o||t.resizedAttached&&t.resizedAttached.call(),c()};d(s,"scroll",f),d(u,"scroll",f)}var i=Object.prototype.toString.call(t),o="[object Array]"===i||"[object NodeList]"===i||"[object HTMLCollection]"===i||"undefined"!=typeof jQuery&&t instanceof jQuery||"undefined"!=typeof Elements&&t instanceof Elements;if(o)for(var s=0,a=t.length;s0;return this.updateCachedDimensions().then(function(){e&&t._resizeHandlers.forEach(function(e){e(t)})})}),t.define("loadDocument",function(t){var e=new a;return this.initialized?this.iframeEl.src?u.reject(new Error("widget already loaded")):(this.iframeEl.addEventListener("load",e.resolve,!1),this.iframeEl.addEventListener("error",e.reject,!1),this.iframeEl.src=t,e.promise):u.reject(new Error("sandbox not initialized"))}),t.after("initialize",function(){var t=new a,e=new a;this._iframe=null,this._iframeVersion=null,this._width=this._height=0,this._resizeHandlers=[],this._rendered=t,this._results=e,this._waitToSwapUntilRendered=!1}),t.override("insert",function(t,e,n,i){var a=this;return e=d.aug({id:t},f,e),n=d.aug({},l,n),this._iframe=s(e,n),p[t]=this,a._waitToSwapUntilRendered||this.onResize(o(function(){a.makeVisible()})),r.write(c(i,null,this._iframe))}),t.override("onResize",function(t){this._resizeHandlers.push(t)}),t.after("styleSelf",function(){return this.updateCachedDimensions()})}},function(t,e,n){var r=n(1),i=n(118),o=n(120),s=n(22),a=n(5),u=n(121);t.exports=function(t,e,n,c,d){function f(t){var e=u(this);s.trigger(t.type,{target:e,region:t.region,type:t.type,data:t.data||{}})}function l(e){var n=u(this),r=n&&n.id,i=a.asInt(e.width),o=a.asInt(e.height);r&&void 0!==i&&void 0!==o&&t(r,i,o)}(new i).attachReceiver(new o.Receiver(r,"twttr.button")).bind("twttr.private.trigger",f).bind("twttr.private.resizeButton",l),(new i).attachReceiver(new o.Receiver(r,"twttr.embed")).bind("twttr.private.initialized",function(t){var e=u(this),n=e&&e.id,r=t.iframe_version;n&&r&&c&&c(n,r)}).bind("twttr.private.trigger",f).bind("twttr.private.results",function(){var t=u(this),n=t&&t.id;n&&e&&e(n)}).bind("twttr.private.rendered",function(){var t=u(this),e=t&&t.id;e&&n&&n(e)}).bind("twttr.private.no_results",function(){var t=u(this),e=t&&t.id;e&&d&&d(e)}).bind("twttr.private.resize",l)}},function(t,e,n){var r=n(24),i=n(119),o=n(0),s=n(6),a=n(23),u="2.0";function c(t){this.registry=t||{}}function d(t){var e,n;return e=o.isType("string",t),n=o.isType("number",t),e||n||null===t}function f(t,e){return{jsonrpc:u,id:d(t)?t:null,error:e}}c.prototype._invoke=function(t,e){var n,r,i;n=this.registry[t.method],r=t.params||[],r=o.isType("array",r)?r:[r];try{i=n.apply(e.source||null,r)}catch(t){i=s.reject(t.message)}return a.isPromise(i)?i:s.resolve(i)},c.prototype._processRequest=function(t,e){var n,r;return function(t){var e,n,r;return!!o.isObject(t)&&(e=t.jsonrpc===u,n=o.isType("string",t.method),r=!("id"in t)||d(t.id),e&&n&&r)}(t)?(n="params"in t&&(r=t.params,!o.isObject(r)||o.isType("function",r))?s.resolve(f(t.id,i.INVALID_PARAMS)):this.registry[t.method]?this._invoke(t,{source:e}).then(function(e){return n=t.id,{jsonrpc:u,id:n,result:e};var n},function(){return f(t.id,i.INTERNAL_ERROR)}):s.resolve(f(t.id,i.METHOD_NOT_FOUND)),null!=t.id?n:s.resolve()):s.resolve(f(t.id,i.INVALID_REQUEST))},c.prototype.attachReceiver=function(t){return t.attachTo(this),this},c.prototype.bind=function(t,e){return this.registry[t]=e,this},c.prototype.receive=function(t,e){var n,a,u,c=this;try{u=t,t=o.isType("string",u)?r.parse(u):u}catch(t){return s.resolve(f(null,i.PARSE_ERROR))}return e=e||null,a=((n=o.isType("array",t))?t:[t]).map(function(t){return c._processRequest(t,e)}),n?function(t){return s.all(t).then(function(t){return(t=t.filter(function(t){return void 0!==t})).length?t:void 0})}(a):a[0]},t.exports=c},function(t){t.exports={PARSE_ERROR:{code:-32700,message:"Parse error"},INVALID_REQUEST:{code:-32600,message:"Invalid Request"},INVALID_PARAMS:{code:-32602,message:"Invalid params"},METHOD_NOT_FOUND:{code:-32601,message:"Method not found"},INTERNAL_ERROR:{code:-32603,message:"Internal error"}}},function(t,e,n){var r=n(8),i=n(1),o=n(24),s=n(3),a=n(20),u=n(0),c=n(2),d=n(7),f=a.ie9();function l(t,e,n){var r;t&&t.postMessage&&(f?r=(n||"")+o.stringify(e):n?(r={})[n]=e:r=e,t.postMessage(r,"*"))}function h(t){return u.isType("string",t)?t:"JSONRPC"}function p(t,e){return e?u.isType("string",t)&&0===t.indexOf(e)?t.substring(e.length):t&&t[e]?t[e]:void 0:t}function m(t,e){var n=t.document;this.filter=h(e),this.server=null,this.isTwitterFrame=c.isTwitterURL(n.location.href),t.addEventListener("message",d(this._onMessage,this),!1)}function v(t,e){this.pending={},this.target=t,this.isTwitterHost=c.isTwitterURL(r.href),this.filter=h(e),i.addEventListener("message",d(this._onMessage,this),!1)}u.aug(m.prototype,{_onMessage:function(t){var e,n=this;this.server&&(this.isTwitterFrame&&!c.isTwitterURL(t.origin)||(e=p(t.data,this.filter))&&this.server.receive(e,t.source).then(function(e){e&&l(t.source,e,n.filter)}))},attachTo:function(t){this.server=t},detach:function(){this.server=null}}),u.aug(v.prototype,{_processResponse:function(t){var e=this.pending[t.id];e&&(e.resolve(t),delete this.pending[t.id])},_onMessage:function(t){var e;if((!this.isTwitterHost||c.isTwitterURL(t.origin))&&(e=p(t.data,this.filter))){if(u.isType("string",e))try{e=o.parse(e)}catch(t){return}(e=u.isType("array",e)?e:[e]).forEach(d(this._processResponse,this))}},send:function(t){var e=new s;return t.id?this.pending[t.id]=e:e.resolve(),l(this.target,t,this.filter),e.promise}}),t.exports={Receiver:m,Dispatcher:v,_stringifyPayload:function(t){return arguments.length>0&&(f=!!t),f}}},function(t,e,n){var r=n(4);t.exports=function(t){for(var e,n=r.getElementsByTagName("iframe"),i=0;n[i];i++)if((e=n[i]).contentWindow===t)return e}},function(t,e,n){var r=n(5),i=n(0),o=n(2),s=n(11),a=n(12)(),u=n(63),c="a.twitter-moment";t.exports=function(t){return a(t,c).map(function(t){return u(function(t){var e=s(t),n={momentId:o.momentId(t.href),chrome:t.getAttribute("data-chrome"),limit:t.getAttribute("data-limit")};return i.forIn(n,function(t,n){var i=e[t];e[t]=r.hasValue(i)?i:n}),e}(t),t.parentNode,t)})}},function(t,e,n){var r=n(3);t.exports=function(t,e){var i=new r;return n.e(3).then(function(r){var o;try{o=n(77),i.resolve(new o(t,e))}catch(t){i.reject(t)}}.bind(null,n)).catch(function(t){i.reject(t)}),i.promise}},function(t,e,n){var r=n(0),i=n(11),o=n(12)(),s=n(64),a="a.periscope-on-air",u=/^https?:\/\/(?:www\.)?(?:periscope|pscp)\.tv\/@?([a-zA-Z0-9_]+)\/?$/i;t.exports=function(t){return o(t,a).map(function(t){return s(function(t){var e=i(t),n=t.getAttribute("href"),o=t.getAttribute("data-size"),s=u.exec(n)[1];return r.aug(e,{username:s,size:o})}(t),t.parentNode,t)})}},function(t,e,n){var r=n(3);t.exports=function(t,e){var i=new r;return n.e(4).then(function(r){var o;try{o=n(78),i.resolve(new o(t,e))}catch(t){i.reject(t)}}.bind(null,n)).catch(function(t){i.reject(t)}),i.promise}},function(t,e,n){var r=n(5),i=n(0),o=n(65),s=n(11),a=n(12)(),u=n(66),c=n(2),d=n(18),f="a.twitter-timeline,div.twitter-timeline,a.twitter-grid",l="Embedded Search timelines have been deprecated. See https://twittercommunity.com/t/deprecating-widget-settings/102295.",h="You may have been affected by an update to settings in embedded timelines. See https://twittercommunity.com/t/deprecating-widget-settings/102295.",p="Embedded grids have been deprecated and will now render as timelines. Please update your embed code to use the twitter-timeline class. More info: https://twittercommunity.com/t/update-on-the-embedded-grid-display-type/119564.";t.exports=function(t,e){return a(t,f).map(function(t){return u(function(t){var e=s(t),n=t.getAttribute("data-show-replies"),a={isPreconfigured:!!t.getAttribute("data-widget-id"),chrome:t.getAttribute("data-chrome"),tweetLimit:t.getAttribute("data-tweet-limit")||t.getAttribute("data-limit"),ariaLive:t.getAttribute("data-aria-polite"),theme:t.getAttribute("data-theme"),borderColor:t.getAttribute("data-border-color"),showReplies:n?r.asBoolean(n):null,profileScreenName:t.getAttribute("data-screen-name"),profileUserId:t.getAttribute("data-user-id"),favoritesScreenName:t.getAttribute("data-favorites-screen-name"),favoritesUserId:t.getAttribute("data-favorites-user-id"),likesScreenName:t.getAttribute("data-likes-screen-name"),likesUserId:t.getAttribute("data-likes-user-id"),listOwnerScreenName:t.getAttribute("data-list-owner-screen-name"),listOwnerUserId:t.getAttribute("data-list-owner-id"),listId:t.getAttribute("data-list-id"),listSlug:t.getAttribute("data-list-slug"),customTimelineId:t.getAttribute("data-custom-timeline-id"),staticContent:t.getAttribute("data-static-content"),url:t.href};return a.isPreconfigured&&(c.isSearchUrl(a.url)?d.publicError(l,t):d.publicLog(h,t)),"twitter-grid"===t.className&&d.publicLog(p,t),(a=i.aug(a,e)).dataSource=o(a),a.id=a.dataSource&&a.dataSource.id,a}(t),t.parentNode,t,e)})}},function(t,e){t.exports=function(t,e,n){for(var r,i=0;i - - - 23 Neural nets - other considerations - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/schedule/slides/23-nnets-other.qmd b/schedule/slides/23-nnets-other.qmd index 75e2e94..7729dd1 100644 --- a/schedule/slides/23-nnets-other.qmd +++ b/schedule/slides/23-nnets-other.qmd @@ -1,39 +1,20 @@ --- -title: "23 Neural nets - other considerations" -author: -- STAT 406 -- Daniel J. McDonald -date: "Last modified - `r Sys.Date()`" +lecture: "23 Neural nets - other considerations" +format: revealjs +metadata-files: + - _metadata.yml --- -```{r setup, include=FALSE, warning=FALSE, message=FALSE} -source("rmd_config.R") -``` - +{{< include _titleslide.qmd >}} -```{r css-extras, file="css-extras.R", echo=FALSE} -``` ## Estimation procedures (training) -$$\newcommand{\Expect}[1]{\mathbb{E}\left[ #1 \right]} -\newcommand{\Var}[1]{\mathbb{V}\left[ #1 \right]} -\newcommand{\Cov}[2]{\mathrm{Cov}\left[#1,\ #2\right]} -\newcommand{\given}{\ \vert\ } -\newcommand{\argmin}{\arg\min} -\newcommand{\argmax}{\arg\max} -\newcommand{\R}{\mathbb{R}} -\newcommand{\P}{\mathbb{P}} -\renewcommand{\hat}{\widehat} -\newcommand{\tr}[1]{\mbox{tr}(#1)} -\newcommand{\X}{\mathbf{X}} -\newcommand{\bfy}{\mathbf{y}} -\newcommand{\bfx}{\mathbf{x}}$$ Back-propagation -__Advantages:__ +[Advantages:]{.secondary} - It's updates only depend on local information in the sense that if objects in the hierarchical model @@ -46,19 +27,18 @@ __Advantages:__ - As the updates are only in terms of $\hat{R}_i$, the algorithm can be run in either batch or online mode -__Down sides:__ +[Down sides:]{.tertiary} - It can be very slow - Need to choose the learning rate $\gamma_t$ ---- ## Other algorithms There are many variations on the fitting algorithm -__Stochastic gradient descent__: (SGD) discussed in the optimization lecture +[Stochastic gradient descent:]{.secondary} (SGD) discussed in the optimization lecture The rest are variations that use lots of tricks @@ -70,65 +50,58 @@ The rest are variations that use lots of tricks * Nadam * Ftrl ---- ## Regularizing neural networks NNets can almost always achieve 0 training error. Even with regularization. Because they have so many parameters. -Flavors: - -- a complexity penalization term +Flavours: +- a complexity penalization term $\longrightarrow$ solve $\min \hat{R} + \rho(\alpha,\beta)$ - early stopping on the back propagation algorithm used for fitting -Explicit regularization $\longrightarrow$ solve $\min \hat{R} + \rho(\alpha,\beta)$ +Weight decay +: This is like ridge regression in that we penalize the squared Euclidean norm of the weights $\rho(\mathbf{W},\mathbf{B}) = \sum w_i^2 + \sum b_i^2$ -- __Weight decay:__ This is like - ridge regression in that we penalize the squared Euclidean norm of - the weights $\rho(\mathbf{W},\mathbf{B}) = \sum w_i^2 + \sum b_i^2$ +Weight elimination +: This encourages more shrinking of small weights $\rho(\mathbf{W},\mathbf{B}) = \sum \frac{w_i^2}{1+w_i^2} + \sum \frac{b_i^2}{1 + b_i^2}$ or Lasso-type -- __Weight elimination:__ This - encourages more shrinking of small weights - $\rho(\mathbf{W},\mathbf{B}) = \sum \frac{w_i^2}{1+w_i^2} + \sum \frac{b_i^2}{1 + b_i^2}$ or Lasso-type +Dropout +: In each epoch, randomly choose $z\%$ of the nodes and set those weights to zero. -__Dropout:__ - -In each epoch, randomly choose $z\%$ of the nodes and set those weights to zero. - ---- ## Other common pitfalls -There are three areas to watch out for +There are a few areas to watch out for -- __Nonconvexity:__ The neural network optimization problem is non-convex. This makes any numerical solution highly dependent on the initial values. These should be +[Nonconvexity:]{.tertiary} - - chosen carefully, typically random near 0. .hand[DON'T] use all 0. +The neural network optimization problem is non-convex. - - regenerated several times to check sensitivity +This makes any numerical solution highly dependent on the initial values. These should be -- __Scaling:__ Be sure to - standardize the covariates before training +* chosen carefully, typically random near 0. [DON'T]{.hand} use all 0. +* regenerated several times to check sensitivity -- __Number of hidden units:__ It is generally - better to have too many hidden units than too few (regularization - can eliminate some). +[Scaling:]{.tertiary} +Be sure to standardize the covariates before training +## Other common pitfalls -- __Sifting the output:__ - - - Choose the solution that minimizes training - error +[Number of hidden units:]{.tertiary} +It is generally +better to have too many hidden units than too few (regularization +can eliminate some). - - Choose the solution that minimizes the - penalized training error - - Average the solutions across runs +[Sifting the output:]{.tertiary} + +* Choose the solution that minimizes training error +* Choose the solution that minimizes the penalized training error +* Average the solutions across runs ---- ## Tuning parameters @@ -145,35 +118,33 @@ These are hard to tune. In practice, people might choose "some" with a validation set, and fix the rest largely arbitrarily --- +. . . More often, people set them all arbitrarily ---- -## Thoughts on NNets +## Thoughts on NNets {.smaller} Off the top of my head, without lots of justification -.pull-left[ - -__Why don't statisticians like them?__ +::: flex +::: w-50 -.center[.larger[`r fa("face-angry", fill=red)`]] +๐Ÿคฌ๐Ÿ˜ก [Why don't statisticians like them?]{.tertiary} ๐Ÿคฌ๐Ÿ˜ก - There is little theory (though this is increasing) -- Statistical theory applies to global minima, here, only local determined by the optimizer +- Stat theory applies to global minima, here, only local determined by the optimizer - Little understanding of when they work - In large part, NNets look like logistic regression + feature creation. We understand that well, and in many applications, it performs as well - Explosion of tuning parameters without a way to decide - Require massive datasets to work -] +- Lots of examples where they perform _exceedingly_ poorly +::: -.pull-right[ +::: w-50 -__Why are they hot?__ -.center[.larger[`r fa("fire", fill=orange)`]] +๐Ÿ”ฅ๐Ÿ”ฅ[Why are they hot?]{.tertiary}๐Ÿ”ฅ๐Ÿ”ฅ - Perform exceptionally well on typical CS tasks (images, translation) - Take advantage of SOTA computing (parallel, GPUs) @@ -181,9 +152,9 @@ __Why are they hot?__ - An excellent example of "transfer learning" - They generate pretty pictures (the nets, pseudo-responses at hidden units) -] +::: +::: ---- ## Keras @@ -193,77 +164,54 @@ It takes some work to get all this software up and running. It is possible to do in with R using an [interface to Keras](https://keras.rstudio.com/index.html). --- +. . . I used to try to do a walk-through, but the interface is quite brittle If you want to explore, see the handout: -* Knitted: https://ubc-stat.github.io/stat-406-lectures/handouts/keras-nnet.html -* Rmd: https://ubc-stat.github.io/stat-406-lectures/handouts/keras-nnet.Rmd - -```{r embeddings, echo=FALSE,include=FALSE,fig.width=16,fig.height=9} -library(maniTools) -num_pts = 600 -d = 2 #target dimension -k = 8 #k nearest neighbors -sim_data <- swiss_roll(num_pts) -pca_dr <- sim_data$data %>% center_and_standardise() %>% prcomp() -proj_pca <- sim_data$data %*% pca_dr$rotation[,1:2] %>% scale() -proj_hess <- Hessian_LLE(sim_data$data, k = k, d = d)$projection %>% scale() -proj_LLE <- LLE2(sim_data$data, dim = d, k = k) %>% scale() -all = cbind(rbind(proj_pca, proj_LLE, proj_hess), sim_data$colors) -colnames(all) = c('x','y','col') -all = as_tibble(all) -all$method = rep(c("PCA","Hessian maps","LLE"), each=num_pts) -ggplot(all, aes(x,y,color=col)) + geom_point() + - scale_color_viridis_c() + - facet_wrap(~method, scales="free") + - theme(legend.position = "none", - axis.line = element_blank(), axis.text = element_blank(), - axis.ticks = element_blank(), axis.title = element_blank()) -``` ---- -class: middle, inverse, center +* Knitted: +* Rmd: -# Double descent and model complexity ---- +# Double descent and model complexity +## -```{r, echo=FALSE, eval=TRUE} +```{r} +#| echo: false +#| fig-align: center tweetrmd::tweet_embed("https://twitter.com/daniela_witten/status/1292293102103748609") ``` ---- ## Where does this U shape come from? -.center.large[ -MSE = Squared Bias + Variance + Irreducible Noise -] +[MSE = Squared Bias + Variance + Irreducible Noise]{.secondary} + As we increase flexibility: + * Squared bias goes down * Variance goes up * Eventually, | $\partial$ Variance | $>$ | $\partial$ Squared Bias |. -**Goal:** Choose amount of flexibility to balance these and minimize MSE. +[Goal:]{.secondary} Choose amount of flexibility to balance these and minimize MSE. --- +. . . -.hand[Use CV or something to estimate MSE and decide how much flexibility.] +[Use CV or something to estimate MSE and decide how much flexibility.]{.hand} ---- +## ```{r, echo=FALSE, eval=TRUE} tweetrmd::tweet_embed("https://twitter.com/daniela_witten/status/1292293104855158784", hide_thread = "t") ``` ---- + ## Zero training error and model saturation @@ -273,18 +221,17 @@ tweetrmd::tweet_embed("https://twitter.com/daniela_witten/status/129229310485515 * So, who cares about the Bias-Variance Trade off!! --- +. . . -**Lesson:** +[Lesson:]{.secondary} -BV Trade off is not wrong. `r fa("face-sad-tear", fill =orange)` +BV Trade off is not wrong. ๐Ÿ˜ข This is a misunderstanding of black box algorithms and flexibility. We don't even need deep learning to illustrate. - ---- +## ```{r, fig.width=6, fig.height=4, fig.align='center'} library(splines) @@ -300,7 +247,7 @@ g + stat_smooth(method = lm, formula = y ~ bs(x, df = 4), se = FALSE, color = gr ``` ---- +## ```{r, fig.width=6, fig.height=4, fig.align='center'} xn <- seq(-1.5 * pi, 1.5 * pi, length.out = 1000) @@ -313,7 +260,7 @@ g + geom_line(data = tibble(x=xn, y=yhat), colour = orange) + ggtitle("20 degrees of freedom") ``` ---- +## ```{r, fig.width=6, fig.height=4, fig.align='center'} xn <- seq(-1.5 * pi, 1.5 * pi, length.out = 1000) @@ -326,14 +273,12 @@ g + geom_line(data = tibble(x = xn, y = yhat), colour = orange) + ggtitle("40 degrees of freedom") ``` ---- -layout: true ## What happened?! ---- -```{r, fig.width=6, fig.height=4, fig.align='center'} +```{r} +#| code-line-numbers: "1|3-12|13-16|" doffs <- 4:50 mse <- function(x, y) mean((x - y)^2) get_errs <- function(doff) { @@ -346,99 +291,122 @@ get_errs <- function(doff) { nb <- sqrt(sum(bhat^2)) tibble(train = mse(df$y, yh), test = mse(yhat, sin(xn)), norm = nb) } -errs <- map_dfr(doffs, get_errs, .id = "degrees of freedom") %>% - mutate(`degrees of freedom` = doffs) %>% +errs <- map(doffs, get_errs) |> + list_rbind() |> + mutate(`degrees of freedom` = doffs) |> pivot_longer(train:test, values_to = "error") ``` ---- +## What happened?! -```{r, fig.width=8, fig.height=4, fig.align='center'} +```{r} +#| code-fold: true +#| fig-width: 9 +#| fig-height: 5 ggplot(errs, aes(`degrees of freedom`, error, color = name)) + - geom_line(size = 2) + + geom_line(linewidth = 2) + coord_cartesian(ylim = c(0, .12)) + - scale_colour_manual(values = c(blue, orange)) + - scale_x_log10() + geom_vline(xintercept = 20) + scale_x_log10() + + scale_colour_manual(values = c(blue, orange), name = "") + + geom_vline(xintercept = 20) ``` ---- +## What happened?! -```{r, fig.width=8, fig.height=5, fig.align='center'} -ggplot(errs %>% filter(name == "test"), aes(norm, error)) + - geom_line(colour = blue, size = 2) + +```{r} +#| code-fold: true +#| fig-width: 9 +#| fig-height: 5 +best_test <- errs |> filter(name == "test") +min_norm <- best_test$norm[which.min(best_test$error)] +ggplot(best_test, aes(norm, error)) + + geom_line(colour = blue, size = 2) + ylab("test error") + + geom_vline(xintercept = min_norm, colour = orange) + scale_y_log10() + scale_x_log10() + geom_vline(xintercept = 20) ``` ---- -layout: false ## Degrees of freedom and complexity -* In low dimensions (where $n \gg p$), with linear smoothers, edf and model complexity are roughly the same. +* In low dimensions (where $n \gg p$), with linear smoothers, df and model complexity are roughly the same. * But this relationship breaks down in more complicated settings * We've already seen this: -```{r, fig.align='center', fig.width=7, fig.height=3, warning=FALSE} +```{r, message=FALSE} library(glmnet) out <- cv.glmnet(X, df$y, nfolds = n) # leave one out -par(mfrow=c(1, 2), mar = c(5, 3, 0, 0)) -plot(out$lambda, out$cvm, type = "b", log = "xy") -plot(out$nzero, out$cvm, type = "b", log = "xy") ``` ---- +```{r} +#| code-fold: true +#| fig-width: 9 +#| fig-height: 2.5 +with( + out, + tibble(lambda = lambda, df = nzero, cv = cvm, cvup = cvup, cvlo = cvlo ) +) |> + filter(df > 0) |> + pivot_longer(lambda:df) |> + ggplot(aes(x = value)) + + geom_errorbar(aes(ymax = cvup, ymin = cvlo)) + + geom_point(aes(y = cv), colour = orange) + + facet_wrap(~ name, strip.position = "bottom", scales = "free_x") + + scale_y_log10() + + scale_x_log10() + theme(axis.title.x = element_blank()) +``` + ## Infinite solutions -* In Lasso, edf is not really the right measure of complexity +* In Lasso, df is not really the right measure of complexity * Better is $\lambda$ or the norm of the coefficients (these are basically the same) * So what happened with the Splines? --- +. . . * When df $= 20$, there's a unique solution that interpolates the data * When df $> 20$, there are infinitely many solutions that interpolate the data. -Because we used the SVD to solve the system, we happened to pick one: the one that has the smallest $||\hat\beta||_2$ +Because we used the SVD to solve the system, we happened to pick one: the one that has the smallest $\Vert\hat\beta\Vert_2$ Recent work in Deep Learning shows that SGD has the same property: it returns the local optima with the smallest norm. If we measure complexity in terms of the norm of the weights, rather than by counting parameters, we don't see double descent anymore. ---- ## The lesson -Deep learning isn't magic. +* Deep learning isn't magic. -Zero training error with lots of parameters doesn't mean good test error. +* Zero training error with lots of parameters doesn't mean good test error. -We still need the bias variance tradeoff +* We still need the bias variance tradeoff -It's intuition still applies: more flexibility eventually leads to increased MSE +* It's intuition still applies: more flexibility eventually leads to increased MSE -But we need to be careful how we measure complexity. +* But we need to be careful how we measure complexity. +::: aside ---- +There is very interesting recent theory that says +when we can expect lower test error to the right of the interpolation threshold +than to the left. -class: middle, center -background-image: url(rmd_gfx/23-nnets-other/embeddings-1.svg) -background-size: cover +::: -.primary[.larger[Next time...]] +# Next time... -.primary[.larger[Module]] .huge-blue-number[5] +[Module 5]{.secondary} -.primary[.larger[unsupervised learning]] +[unsupervised learning]{.secondary}