A06_Multifactor_Models.Rmd

---
title: "Portfoliomanagement and Financial Analysis - Assignment 6"
subtitle: "Submit until Monday 2020-11-02, 13:00"
author: "Stefan Macanovic"
output: html_notebook
---

```{r setup}
remotes::install_github("braverock/factorAnalytics",  build_vignettes = TRUE, force = TRUE)
pacman::p_load(tidyverse,tidyquant,FFdownload,FactorAnalytics,PerformanceAnalytics)
pacman::p_load(tidyverse,tidyquant,FFdownload,PortfolioAnalytics,nloptr)
pacman::p_load(tidyverse,tidyquant,PortfolioAnalytics,nloptr,tsibble,matrixcalc,Matrix,timetk,xts)
```

```{r}
## Important Function

#       fitTsfm
#   
#     fitTsfm(asset.names, factor.names, data, fit.method, variable.selection, ...):
#     
#     Fits a time series (a.k.a. macroeconomic) factor model for one or more asset returns or excess
#     returns using time series regression. Least squares (LS), discounted least squares (DLS) and
#     robust regression fitting are possible. Variable selection methods include "stepwise", "subsets" and "lars". An object of class "tsfm" containing the
#     fitted objects, estimated coefficients, R-squared and residual volatility is returned.

```


**Please** remember to put your assignment solutions in `rmd` format using **many** chunks and putting readable text in between, similar to my examples given in Research Methods and Assignment 1!

For all exercises: Please use the Assignment-Forum to post your questions, I will try my best to help you along! If you follow the vignettes from `factorAnalytics`, wherever it says `z.score=T`, please exchange it for either `z.score='crossSection'` or `z.score='timeSeries'` depending on the task at hand.

## Exercise 1: Estimating the CAPM (from A05)

In this exercise we want to estimate the CAPM. Please read carefully through the two documents provided (right hand side: files). Then we start to collect the necessary data:
  
a) From Datastream get the last 10 years of data from the 100 stocks of the S&P100 using the list `LS&P100I` (S&P 100): total return index (RI) and market cap (MV)
b) Further import the Fama-French-Factors from Kenneth Frenchs homepage (monthly, e.g. using `FFdownload`). From both datasets we select data for the last (available) 60 months, calculate returns (simple percentage) for the US-Stocks and eliminate those stocks that have NAs for this period.
c) Now subtract the risk-free rate from all the stocks. Then estimate each stocks beta with the market: Regress all stock excess returns on the market excess return and save all betas (optimally use `mutate` and `map` in combination with `lm`). Estimate the mean-return for each stock and plot the return/beta-combinations. Create the security market line and include it in the plot! What do you find?
d) In a next step (following both documents), we sort the stocks according to their beta and build ten value-weighted portfolios (with more or less the same number of stocks). Repeat a) for the ten portfolios. What do you observe?
e) In the third step you follow page 6-8 of the second document and estimate the second-pass regression with the market and then market & idiosyncratic risk. What do you observe? Present all your results in a similar fashion as in the document.

## Exercise 2: Calculating and checking the CAPM cont. (from A05)

```{r}
pacman::p_load(tidyverse,tidyquant,FFdownload,PortfolioAnalytics,nloptr,readxl,quantmod,FFdownload,timetk, dplyr, xts)
```


As we have seen: the CAPM for small portfolios does not work very well, and so we start using portfolios that get rid of the idiosyncratic risk!
Go to Kenneth French's Homepage  again and download the following datasets: "Portfolios Formed on Market Beta" (where we will use 10 monthly value weighted portfolios formed on beta) and "25 Portfolios Formed on Size and Market Beta" (same thing) as well as the market factor and rf (as before). Now we are going to check the CAPM like famous researchers have done it!
We can use returns as they are in the files (simple returns)!


```{r}
inputlist<-c("F-F_Research_Data_Faktors_CSV.zip","Portfolios_Formed_on_BETA_CSV.zip")
             
#Now process only these files if they can be matched (download only)
FFdownload(output_file = "FFdata.RData", inputlist = inputlist, exclude_daily=TRUE)

load("FFdata.RData")
portf_mkt_betatest<-(FFdownload$x_Portfolios_Formed_on_BETA$monthly$value_weighted_returns)

portf_mkt_betatest

```



```{r}
#Download the Portfolios from Kenneth French's Homepage
portf_mkt_beta <- "https://mba.tuck.dartmouth.edu/pages/faculty/ken.french/ftp/Portfolios_Formed_on_BETA_CSV.zip"
 portf_mkt_beta_csv <- "Portfolios_Formed_on_BETA.csv"
 temp <- tempfile()
download.file(portf_mkt_beta, temp, quiet = TRUE)
portf_mkt_beta <- read_csv(unz(temp, portf_mkt_beta_csv), skip = 15, quote = "\",") %>%
  dplyr::rename(date = "X1") %>%
  mutate_at(vars(-date), as.numeric) %>%
  mutate(date = rollback(ymd(parse_date_time(date, "%Y%m") + months(1))))%>%
  filter(date >= first('1964-01-01') & date <= '2019-12-31')

#Download the market factor and rf (Fama/French 3 Research Factors)
mkt_factors <- "https://mba.tuck.dartmouth.edu/pages/faculty/ken.french/ftp/F-F_Research_Data_Factors_CSV.zip"
 mkt_factors_csv <- "F-F_Research_Data_Factors.CSV"
 temp <- tempfile()
download.file(mkt_factors, temp, quiet = TRUE)
mkt_factors <- read_csv(unz(temp, mkt_factors_csv), skip = 3, quote = "\",") %>%
  dplyr::rename(date = X1) %>%
  mutate_at(vars(-date), as.numeric) %>%
  mutate(date = rollback(ymd(parse_date_time(date, "%Y%m") + months(1)))) %>%
  filter(date >= first('1964-01-01') & date <= '2019-12-31')


```


a)	Subtract the risk-free rate from the first set of 10 portfolios (only sorted on beta) (Lo 10,., Hi 10) and estimate each stocks beta with the market. Estimate the mean-return for each stock and plot the return/beta-combinations. Create the security market line and include it in the plot! What do you find? (You can split the file in 2-3 different time blocks and see if something changes). * Now we are done with the first-pass regression.*


Subtract the risk-free rate from the first set of 10 portfolios (only sorted on beta) (Lo 10,., Hi 10) and estimate each stocks beta with the market.

```{r}
#join data
ten_portf <- portf_mkt_beta[1:672, -c(2:6)]
ten_portf_joined <- left_join(mkt_factors, ten_portf)

mkt_factors
ten_portf
ten_portf_joined

```
```{r, echo=FALSE}
ten_portf_joined <- ten_portf_joined <- ten_portf_joined%>% dplyr::rename("Lo10" = "Lo 10") %>% dplyr::rename("Dec2" = "Dec 2") %>% dplyr::rename("Dec3" = "Dec 3") %>% dplyr::rename("Dec4" = "Dec 4") %>% dplyr::rename("Dec5" = "Dec 5") %>% dplyr::rename("Dec6" = "Dec 6") %>% dplyr::rename("Dec7" = "Dec 7") %>% dplyr::rename("Dec8" = "Dec 8") %>% dplyr::rename("Dec9" = "Dec 9") %>% dplyr::rename("Hi10" = "Hi 10")

view(ten_portf_joined)
ten_portf_joined

```

```{r}
#substract Risk-Free-Rate
ten_portf_rf <- mutate(ten_portf_joined, Lo10rf = Lo10 - RF, Dec2rf = Dec2 - RF, Dec3rf = Dec3 - RF, Dec4rf = Dec4 - RF, Dec5rf = Dec5 -RF, Dec6rf = Dec6 - RF, Dec7rf = Dec7 - RF, Dec8rf = Dec8 - RF, De9rf = Dec9 - RF, Hi10rf = Hi10 - RF)
ten_portf_rf <- ten_portf_rf[-2:-15]

view(ten_portf_rf)
ten_portf_rf
```

```{r, echo=FALSE}
#Create XTS
mkt_factors_xts <- tk_xts(data = mkt_factors, date_var = date)
ten_portf_rf_xts <- ten_portf_rf %>%
  tk_xts(date_var = date, silent = TRUE)

```
```{r}
?lm()
#Calculate Betas for each portfolio
betas_ten_portf_lm <- lm(ten_portf_rf_xts ~ mkt_factors_xts[, 1])
betas_ten_portf_lm
betas_ten_portf <- CAPM.beta(Ra = ten_portf_rf_xts, Rb = mkt_factors_xts[, 1], Rf = 0)
betas_ten_portf
```
Estimate the mean-return for each stock and plot the return/beta-combinations.

```{r}
#Estimate Mean Return
mean_ten_portf_rf_xts <- as.data.frame(lapply(ten_portf_rf_xts, FUN=mean))
mean_ten_portf_rf_xts

#Plot the return/beta-combinations
plot.default(x = betas_ten_portf, xlim=c(0, 2),
             y = mean_ten_portf_rf_xts, ylim=c(0, 1),
             xlab = "Beta", ylab = "Mean Return",
             main = "Return/Beta-combinations")
```
Create the security market line and include it in the plot! What do you find?

```{r}
mean_mkt <- as.data.frame(lapply(mkt_factors_xts[, 1], FUN=mean))
y_mkt <- mean_mkt[1, 1]
plot.default(x = betas_ten_portf, xlim=c(0, 2),
             y = mean_ten_portf_rf_xts, ylim=c(0, 1),
             xlab = "Beta", ylab = "Mean Return",
             main = "Return/Beta-combinations",
             abline(0, y_mkt))
plot.default(x = betas_ten_portf, xlim=c(0, 2), 
             y = mean_ten_portf_rf_xts, ylim=c(0, 10), 
             xlab = "Beta", ylab = "Mean Return",
             main = "Return/Beta-combinations",
             abline(0, y_mkt))

#summary
summary_CAPM_ten_portf <- (table.CAPM(Ra = ten_portf_rf_xts, Rb = mkt_factors_xts[, 1], Rf = 0)[1:9, ])
```
(You can split the file in 2-3 different time blocks and see if something changes). * Now we are done with the first-pass regression.*

```{r}
#look for first 10 years
ten_portf_rf_10yrs_xts <- ten_portf_rf[1:120, ] %>%
  tk_xts(date_var = date, silent = TRUE)
betas_ten_portf_rf_10yrs <- CAPM.beta(Ra = ten_portf_rf_10yrs_xts, Rb = mkt_factors_xts[1:120, 1], Rf = 0)
mean_ten_portf_rf_10yrs_xts <- as.data.frame(lapply(ten_portf_rf_10yrs_xts, FUN=mean))
mean_mkt_10yrs <- as.data.frame(lapply(mkt_factors_xts[1:120, 1], FUN=mean))
y_mkt_10yrs <- mean_mkt_10yrs[1, 1]
plot.default(x = betas_ten_portf_rf_10yrs, xlim=c(0, 2),
             y = mean_ten_portf_rf_10yrs_xts, ylim=c(0, 1),
             xlab = "Beta", ylab = "Mean Return",
             main = "Return/Beta-combinations 1964-1974",
             abline(0, y_mkt_10yrs))
summary_CAPM_ten_portf_10yrs <- (table.CAPM(Ra = ten_portf_rf_xts[1:120, ], Rb = mkt_factors_xts[1:120, 1], Rf = 0)[1:9, ])
summary_CAPM_ten_portf_10yrs
```
```{r, echo=FALSE}

#look for 2000-2019
ten_portf_rf_2000_xts <- ten_portf_rf[433:672, ] %>%
  tk_xts(date_var = date, silent = TRUE)
betas_ten_portf_rf_2000 <- CAPM.beta(Ra = ten_portf_rf_2000_xts, Rb = mkt_factors_xts[433:672, 1], Rf = 0)
mean_ten_portf_rf_2000_xts <- lapply(ten_portf_rf_2000_xts, FUN=mean)
mean_ten_portf_rf_2000_xts <- as.data.frame(mean_ten_portf_rf_2000_xts)
mean_mkt_2000 <- lapply(mkt_factors_xts[433:672, 1], FUN=mean)
mean_mkt_2000 <- as.data.frame(mean_mkt_2000)
y_mkt_2000 <- mean_mkt_2000[1, 1]
plot.default(x = betas_ten_portf_rf_2000, xlim=c(0, 2),
             y = mean_ten_portf_rf_2000_xts, ylim=c(0, 1),
             xlab = "Beta", ylab = "Mean Return",
             main = "Return/Beta-combinations 2000-2019",
             abline(0, y_mkt_2000))
summary_CAPM_ten_portf_2000 <- (table.CAPM(Ra = ten_portf_rf_xts[433:672, ], Rb = mkt_factors_xts[433:672, 1], Rf = 0)[1:9, ])
summary_CAPM_ten_portf_2000

plot.default(x = betas_ten_portf, xlim=c(0, 2),
             y = mean_ten_portf_rf_xts, ylim=c(0, 1),
             xlab = "Beta", ylab = "Mean Return",
             main = "Return/Beta-combinations 1964-2019",
             abline(0, y_mkt))
summary_CAPM_ten_portf


```


b)	In the second-pass regression we now regress the average stock returns on the betas estimated before. What do you find in the coefficients and does this contradict the CAPM? Try different time periods again and see what you find. (all of the interpretations are in BKM pp.416f). 


```{r}
betas_ten_portf


```

```{r}
#There are a number of reasons we expect might the CAPM to fail:
#1. Imperfect measures of the market portfolio
#2. Beta is an incomplete measure of risk
#3. Tax effects
#4. Non - normality of returns
#5. No riskless asset
#6. Divergent borrowing and lending rates
```

There are a number of reasons we expect might the CAPM to
fail:
1. Imperfect measures of the market portfolio
2. Beta is an incomplete measure of risk
3. Tax effects
4. Non - normality of returns
5. No riskless asset
6. Divergent borrowing and lending rates

c)	Now do the extended second pass regression (regress on betas and residual-sds that you can extract from the regression) and see what you find for different periods. Interpret according to concept check 13.2. One of the (many) problems of the CAPM can be the correlation between residual variances and betas. Calculate and interpret.

```{r}
#Look at a) -> We now do it with the mean return of every portfolio combined... 

#1964-2019
com_mean_ten_portf_rf <- sum(mean_ten_portf_rf_xts)/10
mean_betas_ten_portf <- sum(betas_ten_portf)/10
plot.default(x = mean_betas_ten_portf, xlim=c(0, 2),
             y = com_mean_ten_portf_rf, ylim=c(0, 2),
             xlab = "Beta", ylab = "Mean Return",
             main = "Return/Beta-combinations 10 Portfolios 1964-2019",
             abline(0, y_mkt))


```

```{r, echo=FALSE}

#1964-1974
com_mean_ten_portf_rf_10yrs <- sum(mean_ten_portf_rf_10yrs_xts)/10
mean_betas_ten_portf_10yrs <- sum(betas_ten_portf_rf_10yrs)/10
plot.default(x = mean_betas_ten_portf_10yrs, xlim=c(0, 2),
             y = com_mean_ten_portf_rf_10yrs, ylim=c(0, 2),
             xlab = "Beta", ylab = "Mean Return",
             main = "Return/Beta-combinations 10 Portfolios 1964-1974",
             abline(0, y_mkt_10yrs))

#2000-2019
com_mean_ten_portf_rf_2000 <- sum(mean_ten_portf_rf_2000_xts)/10
mean_betas_ten_portf_2000 <- sum(betas_ten_portf_rf_2000)/10
plot.default(x = mean_betas_ten_portf_2000, xlim=c(0, 2),
             y = com_mean_ten_portf_rf_2000, ylim=c(0, 2),
             xlab = "Beta", ylab = "Mean Return",
             main = "Return/Beta-combinations 10 Portfolios 2000-2019",
             abline(0, y_mkt_2000))

```

```{r, echo=FALSE}

#SML-Function
calc_residual <- function(x) {y <- y_mkt*x}
calc_residual_10yrs <- function(x) {y <- y_mkt_10yrs*x}
calc_residual_2000 <- function(x) {y <- y_mkt_2000*x}
residual_1964_2019 <- as.data.frame((com_mean_ten_portf_rf - calc_residual(mean_betas_ten_portf))^2)
residual_1964_1974 <- as.data.frame((com_mean_ten_portf_rf_10yrs - calc_residual_10yrs(mean_betas_ten_portf_10yrs))^2)
residual_2000_2019 <- as.data.frame((com_mean_ten_portf_rf_2000 - calc_residual_2000(mean_betas_ten_portf_2000))^2)
joined_residuals <- merge(residual_1964_2019[1, 1], residual_1964_1974[1, 1])
joined_residuals <- merge(joined_residuals, residual_2000_2019)
Residuals_different_timeperiods <- joined_residuals %>% 
  dplyr::rename("Residual 2000-2019" = "(com_mean_ten_portf_rf_2000 - calc_residual_2000(mean_betas_ten_portf_2000))^2") %>% dplyr::rename("Residual 1964-2008" = "x") %>% dplyr::rename("Residual 1964-1974" = "y")
Residuals_different_timeperiods
```


d)	Try again with 25 portfolios sorted on size and beta. What do you find? Is that interesting? 

```{r}
inputlist1<-c("F-F_Research_Data_Faktors_CSV.zip","25_Portfolios_Formed_on_Size_and Market_Beta_CSV.zip")
             
#Now process only these files if they can be matched (download only)
FFdownload(output_file = "FFdata.RData", inputlist = inputlist1, exclude_daily=TRUE)

load("FFdata.RData")
twentyfive_portf<-(FFdownload$x_25_Portfolios_Formed_on_Size_and_Market_Beta$monthly$value_weighted_returns)

twentyfive_portf
 
```

```{r}
download.file(portf_mkt_beta, temp, quiet = TRUE)
portf_mkt_beta <- read_csv(unz(temp, portf_mkt_beta_csv), skip = 15, quote = "\",") %>%
  dplyr::rename(date = "X1") %>%
  mutate_at(vars(-date), as.numeric) %>%
  mutate(date = rollback(ymd(parse_date_time(date, "%Y%m") + months(1))))%>%
  filter(date >= first('1964-01-01') & date <= '2019-12-31')
```



```{r}
twentyfive_portf
```


```{r}
#join data
twentyfive_portf <- portf_mkt_beta[1:672, -c(7:16)]
twentyfive_portf_joined <- left_join(mkt_factors, twentyfive_portf)
```

```{r, echo=FALSE}

twentyfive_portf_joined <- twentyfive_portf_joined <- twentyfive_portf_joined%>%
  dplyr::rename("Lo20" = "Lo 20") %>%
  dplyr::rename("Qnt2" = "Qnt 2") %>%
  dplyr::rename("Qnt3" = "Qnt 3") %>%
  dplyr::rename("Qnt4" = "Qnt 4") %>%
  dplyr::rename("Hi20" = "Hi 20")
```

```{r}
#substract Risk-Free-Rate
twentyfive_portf_rf <- mutate(twentyfive_portf_joined, Lo20rf = Lo20 - RF, Qnt2rf = Qnt2 - RF, Qnt3rf = Qnt3 - RF, Qnt4rf = Qnt4 - RF, Hi20rf = Hi20 - RF)
twentyfive_portf_rf <- twentyfive_portf_rf[-2:-10]

```

```{r, echo=FALSE}
#substract Risk-Free-Rate
twentyfive_portf_rf <- mutate(twentyfive_portf_joined, Lo20rf = Lo20 - RF, Qnt2rf = Qnt2 - RF, Qnt3rf = Qnt3 - RF, Qnt4rf = Qnt4 - RF, Hi20rf = Hi20 - RF)
twentyfive_portf_rf <- twentyfive_portf_rf[-2:-10]


#Create XTS
twentyfive_portf_rf_xts <- twentyfive_portf_rf %>%
  tk_xts(date_var = date, silent = TRUE)

#Calculate Betas for each portfolio
betas_twentyfive_portf <- CAPM.beta(Ra = twentyfive_portf_rf_xts, Rb = mkt_factors_xts[, 1], Rf = 0)

#Estimate Mean Return
mean_twentyfive_portf_rf_xts <- as.data.frame(lapply(twentyfive_portf_rf_xts, FUN=mean))

#Plot the return/beta-combinations
plot.default(x = betas_twentyfive_portf, xlim=c(0, 2),
             y = mean_twentyfive_portf_rf_xts, ylim=c(0, 1),
             xlab = "Beta", ylab = "Mean Return",
             main = "Return/Beta-combinations 25",
             abline(0, y_mkt))

#We now do it with the mean return of every portfolio combined...
com_mean_twentyfive_portf_rf <- sum(mean_twentyfive_portf_rf_xts)/5
# and the beta
mean_betas_twentyfive_portf <- sum(betas_twentyfive_portf)/5

plot.default(x = mean_betas_ten_portf, xlim=c(0, 2),
             y = com_mean_ten_portf_rf, ylim=c(0, 1),
             xlab = "Beta", ylab = "Mean Return",
             main = "Return/Beta-combinations Portfolio Summary 25",
             abline(0, y_mkt))
plot.default(x = mean_betas_ten_portf, xlim=c(0, 2),
             y = com_mean_ten_portf_rf, ylim=c(0, 1),
             xlab = "Beta", ylab = "Mean Return",
             main = "Return/Beta-combinations Portfolio Summary 10",
             abline(0, y_mkt))

```


```{r}
#substract Risk-Free-Rate
ten_portf_rf <- mutate(ten_portf_joined, Lo10rf = Lo10 - RF, Dec2rf = Dec2 - RF, Dec3rf = Dec3 - RF, Dec4rf = Dec4 - RF, Dec5rf = Dec5 -RF, Dec6rf = Dec6 - RF, Dec7rf = Dec7 - RF, Dec8rf = Dec8 - RF, De9rf = Dec9 - RF, Hi10rf = Hi10 - RF)
ten_portf_rf <- ten_portf_rf[-2:-15]

view(ten_portf_rf)
ten_portf_rf
```

```{r, echo=FALSE}
#Create XTS
mkt_factors_xts <- tk_xts(data = mkt_factors, date_var = date)
ten_portf_rf_xts <- ten_portf_rf %>%
  tk_xts(date_var = date, silent = TRUE)

```
```{r}
?lm()
#Calculate Betas for each portfolio
betas_ten_portf_lm <- lm(ten_portf_rf_xts ~ mkt_factors_xts[, 1])
betas_ten_portf_lm
betas_ten_portf <- CAPM.beta(Ra = ten_portf_rf_xts, Rb = mkt_factors_xts[, 1], Rf = 0)
betas_ten_portf
```
Estimate the mean-return for each stock and plot the return/beta-combinations.

```{r}
#Estimate Mean Return
mean_ten_portf_rf_xts <- as.data.frame(lapply(ten_portf_rf_xts, FUN=mean))
mean_ten_portf_rf_xts

#Plot the return/beta-combinations
plot.default(x = betas_ten_portf, xlim=c(0, 2),
             y = mean_ten_portf_rf_xts, ylim=c(0, 1),
             xlab = "Beta", ylab = "Mean Return",
             main = "Return/Beta-combinations")
```
Create the security market line and include it in the plot! What do you find?

```{r}
mean_mkt <- as.data.frame(lapply(mkt_factors_xts[, 1], FUN=mean))
y_mkt <- mean_mkt[1, 1]
plot.default(x = betas_ten_portf, xlim=c(0, 2),
             y = mean_ten_portf_rf_xts, ylim=c(0, 1),
             xlab = "Beta", ylab = "Mean Return",
             main = "Return/Beta-combinations",
             abline(0, y_mkt))


#
```
(You can split the file in 2-3 different time blocks and see if something changes). * Now we are done with the first-pass regression.*

```{r}
#look for first 10 years
ten_portf_rf_10yrs_xts <- ten_portf_rf[1:120, ] %>%
  tk_xts(date_var = date, silent = TRUE)
betas_ten_portf_rf_10yrs <- CAPM.beta(Ra = ten_portf_rf_10yrs_xts, Rb = mkt_factors_xts[1:120, 1], Rf = 0)
mean_ten_portf_rf_10yrs_xts <- as.data.frame(lapply(ten_portf_rf_10yrs_xts, FUN=mean))
mean_mkt_10yrs <- as.data.frame(lapply(mkt_factors_xts[1:120, 1], FUN=mean))
y_mkt_10yrs <- mean_mkt_10yrs[1, 1]
plot.default(x = betas_ten_portf_rf_10yrs, xlim=c(0, 2),
             y = mean_ten_portf_rf_10yrs_xts, ylim=c(0, 1),
             xlab = "Beta", ylab = "Mean Return",
             main = "Return/Beta-combinations 1964-1974",
             abline(0, y_mkt_10yrs))
summary_CAPM_ten_portf_10yrs <- (table.CAPM(Ra = ten_portf_rf_xts[1:120, ], Rb = mkt_factors_xts[1:120, 1], Rf = 0)[1:9, ])
summary_CAPM_ten_portf_10yrs
```
```{r, echo=FALSE}

#look for 2000-2019
ten_portf_rf_2000_xts <- ten_portf_rf[433:672, ] %>%
  tk_xts(date_var = date, silent = TRUE)
betas_ten_portf_rf_2000 <- CAPM.beta(Ra = ten_portf_rf_2000_xts, Rb = mkt_factors_xts[433:672, 1], Rf = 0)
mean_ten_portf_rf_2000_xts <- lapply(ten_portf_rf_2000_xts, FUN=mean)
mean_ten_portf_rf_2000_xts <- as.data.frame(mean_ten_portf_rf_2000_xts)
mean_mkt_2000 <- lapply(mkt_factors_xts[433:672, 1], FUN=mean)
mean_mkt_2000 <- as.data.frame(mean_mkt_2000)
y_mkt_2000 <- mean_mkt_2000[1, 1]
plot.default(x = betas_ten_portf_rf_2000, xlim=c(0, 2),
             y = mean_ten_portf_rf_2000_xts, ylim=c(0, 1),
             xlab = "Beta", ylab = "Mean Return",
             main = "Return/Beta-combinations 2000-2019",
             abline(0, y_mkt_2000))
summary_CAPM_ten_portf_2000 <- (table.CAPM(Ra = ten_portf_rf_xts[433:672, ], Rb = mkt_factors_xts[433:672, 1], Rf = 0)[1:9, ])
summary_CAPM_ten_portf_2000

plot.default(x = betas_ten_portf, xlim=c(0, 2),
             y = mean_ten_portf_rf_xts, ylim=c(0, 1),
             xlab = "Beta", ylab = "Mean Return",
             main = "Return/Beta-combinations 1964-2019",
             abline(0, y_mkt))
summary_CAPM_ten_portf


```


b)	In the second-pass regression we now regress the average stock returns on the betas estimated before. What do you find in the coefficients and does this contradict the CAPM? Try different time periods again and see what you find. (all of the interpretations are in BKM pp.416f). 

There are a number of reasons we expect might the CAPM to
fail:
1. Imperfect measures of the market portfolio
2. Beta is an incomplete measure of risk
3. Tax effects
4. Non - normality of returns
5. No riskless asset
6. Divergent borrowing and lending rates

c)	Now do the extended second pass regression (regress on betas and residual-sds that you can extract from the regression) and see what you find for different periods. Interpret according to concept check 13.2. One of the (many) problems of the CAPM can be the correlation between residual variances and betas. Calculate and interpret.

```{r}
#Look at a) -> We now do it with the mean return of every portfolio combined... 

#1964-2019
com_mean_ten_portf_rf <- sum(mean_ten_portf_rf_xts)/10
mean_betas_ten_portf <- sum(betas_ten_portf)/10
plot.default(x = mean_betas_ten_portf, xlim=c(0, 2),
             y = com_mean_ten_portf_rf, ylim=c(0, 2),
             xlab = "Beta", ylab = "Mean Return",
             main = "Return/Beta-combinations 10 Portfolios 1964-2019",
             abline(0, y_mkt))


```

```{r, echo=FALSE}

#1964-1974
com_mean_ten_portf_rf_10yrs <- sum(mean_ten_portf_rf_10yrs_xts)/10
mean_betas_ten_portf_10yrs <- sum(betas_ten_portf_rf_10yrs)/10
plot.default(x = mean_betas_ten_portf_10yrs, xlim=c(0, 2),
             y = com_mean_ten_portf_rf_10yrs, ylim=c(0, 2),
             xlab = "Beta", ylab = "Mean Return",
             main = "Return/Beta-combinations 10 Portfolios 1964-1974",
             abline(0, y_mkt_10yrs))

#2000-2019
com_mean_ten_portf_rf_2000 <- sum(mean_ten_portf_rf_2000_xts)/10
mean_betas_ten_portf_2000 <- sum(betas_ten_portf_rf_2000)/10
plot.default(x = mean_betas_ten_portf_2000, xlim=c(0, 2),
             y = com_mean_ten_portf_rf_2000, ylim=c(0, 2),
             xlab = "Beta", ylab = "Mean Return",
             main = "Return/Beta-combinations 10 Portfolios 2000-2019",
             abline(0, y_mkt_2000))

```

```{r, echo=FALSE}

#SML-Function
calc_residual <- function(x) {y <- y_mkt*x}
calc_residual_10yrs <- function(x) {y <- y_mkt_10yrs*x}
calc_residual_2000 <- function(x) {y <- y_mkt_2000*x}
residual_1964_2019 <- as.data.frame((com_mean_ten_portf_rf - calc_residual(mean_betas_ten_portf))^2)
residual_1964_1974 <- as.data.frame((com_mean_ten_portf_rf_10yrs - calc_residual_10yrs(mean_betas_ten_portf_10yrs))^2)
residual_2000_2019 <- as.data.frame((com_mean_ten_portf_rf_2000 - calc_residual_2000(mean_betas_ten_portf_2000))^2)
joined_residuals <- merge(residual_1964_2018[1, 1], residual_1964_1974[1, 1])
joined_residuals <- merge(joined_residuals, residual_2000_2018)
Residuals_different_timeperiods <- joined_residuals %>% 
  dplyr::rename("Residual 2000-2019" = "(com_mean_ten_portf_rf_2000 - calc_residual_2000(mean_betas_ten_portf_2000))^2") %>% dplyr::rename("Residual 1964-2008" = "x") %>% dplyr::rename("Residual 1964-1974" = "y")
Residuals_different_timeperiods
```


d)	Try again with 25 portfolios sorted on size and beta. What do you find? Is that interesting? 

```{r}
#join data
twentyfive_portf <- portf_mkt_beta[1:672, -c(7:16)]
twentyfive_portf_joined <- left_join(mkt_factors, twentyfive_portf)
```

```{r, echo=FALSE}

twentyfive_portf_joined <- twentyfive_portf_joined <- twentyfive_portf_joined%>%
  dplyr::rename("Lo20" = "Lo 20") %>%
  dplyr::rename("Qnt2" = "Qnt 2") %>%
  dplyr::rename("Qnt3" = "Qnt 3") %>%
  dplyr::rename("Qnt4" = "Qnt 4") %>%
  dplyr::rename("Hi20" = "Hi 20")
```

```{r}
#substract Risk-Free-Rate
twentyfive_portf_rf <- mutate(twentyfive_portf_joined, Lo20rf = Lo20 - RF, Qnt2rf = Qnt2 - RF, Qnt3rf = Qnt3 - RF, Qnt4rf = Qnt4 - RF, Hi20rf = Hi20 - RF)
twentyfive_portf_rf <- twentyfive_portf_rf[-2:-10]

```

```{r, echo=FALSE}
#substract Risk-Free-Rate
twentyfive_portf_rf <- mutate(twentyfive_portf_joined, Lo20rf = Lo20 - RF, Qnt2rf = Qnt2 - RF, Qnt3rf = Qnt3 - RF, Qnt4rf = Qnt4 - RF, Hi20rf = Hi20 - RF)
twentyfive_portf_rf <- twentyfive_portf_rf[-2:-10]


#Create XTS
twentyfive_portf_rf_xts <- twentyfive_portf_rf %>%
  tk_xts(date_var = date, silent = TRUE)

#Calculate Betas for each portfolio
betas_twentyfive_portf <- CAPM.beta(Ra = twentyfive_portf_rf_xts, Rb = mkt_factors_xts[, 1], Rf = 0)

#Estimate Mean Return
mean_twentyfive_portf_rf_xts <- as.data.frame(lapply(twentyfive_portf_rf_xts, FUN=mean))

#Plot the return/beta-combinations
plot.default(x = betas_twentyfive_portf, xlim=c(0, 2),
             y = mean_twentyfive_portf_rf_xts, ylim=c(0, 1),
             xlab = "Beta", ylab = "Mean Return",
             main = "Return/Beta-combinations 25",
             abline(0, y_mkt))

#We now do it with the mean return of every portfolio combined...
com_mean_twentyfive_portf_rf <- sum(mean_twentyfive_portf_rf_xts)/5
# and the beta
mean_betas_twentyfive_portf <- sum(betas_twentyfive_portf)/5

plot.default(x = mean_betas_ten_portf, xlim=c(0, 2),
             y = com_mean_ten_portf_rf, ylim=c(0, 1),
             xlab = "Beta", ylab = "Mean Return",
             main = "Return/Beta-combinations Portfolio Summary 25",
             abline(0, y_mkt))
plot.default(x = mean_betas_ten_portf, xlim=c(0, 2),
             y = com_mean_ten_portf_rf, ylim=c(0, 1),
             xlab = "Beta", ylab = "Mean Return",
             main = "Return/Beta-combinations Portfolio Summary 10",
             abline(0, y_mkt))

```



**The purpose of the further assignments is less programming-related (you can copy most of the code), but to receive a positive grade I want you to dig into the referenced literature and be able to explain _everything_ that you do very detailed in the text and your presentation (what do you do, what is the result and how do you intepret the result). After doing this - given the data - you should be perfectly able to estimate/interpret any type of factor model.**
  
## Exercise 3: Statistical Factor Models

```{r}
library(timetk)
SP500 <- tq_index("SP500")
NASDAQ <- tq_exchange("NASDAQ")
NYSE <- tq_exchange("NYSE") 
stocks.selection <- SP500 %>% 
  inner_join(rbind(NYSE,NASDAQ) %>% select(symbol,last.sale.price,market.cap,ipo.year),by=c("symbol")) %>%
  filter(ipo.year<2000&!is.na(market.cap)) %>% 
  arrange(desc(weight)) %>% 
  slice(1:10)
stocks.selection
```

These are the returns of the selected stocks.

```{r}
stocks.returns <- stocks.selection$symbol %>%
    tq_get(get  = "stock.prices",
           from = "2000-01-01",
           to   = "2019-12-31") %>%
    group_by(symbol) %>%
    tq_transmute(select     = adjusted, 
                 mutate_fun = periodReturn, 
                 period     = "monthly")
stocks.returns
```

These are the stocks return in the xts format and also in a wide format
````{r}
stocks.returns.xts <- stocks.returns%>%
                      subset( select = c(symbol,date, monthly.returns)) %>%
                      pivot_wider(names_from = symbol, 
                                  values_from = monthly.returns) %>% 
                      tk_xts(date_var = date, silent = TRUE)
colnames(stocks.returns.xts)
```

### Fit a statistical factor model fitSfm

Principal Component Analysis (PCA): uses the eigen decomposition of the covariance matrix of asset returns to find the first K principal components that explain the largest portion of the sample covariance matrix returns. Factor loading are then estimated using time series regression. Foctor analysis involves maximum likelihood optimization to estimate the factor loadings and the residual covarince matrix, constructing the factor realization and choosing a rotation of the coordinate system for a more meeaningful interpretion of factors

used when T>N
T: Number of observations
N: Number of assets

if N>T then Aymptotic Principal Component Analysis (APCA)

#### Principal Component Analysis
Fitting a statistical factor model with two principal components (k=2)
```{r}
fit.pca <- fitSfm(stocks.returns.xts, k=2)
fit.pca 
```
Screenplot of eigenvalues
An eigenvector of a linear transformation is a nonzero vector that changes by a scalar factor when that linear transformation is applied to it. Eigenvalues and eigenvectors allow us to "reduce" a linear operation to separate, simpler, problems.
```{r}
plot(fit.pca, which=1, eig.max = 0.9)

```

First principal component explains about 48% of total variance. The first two components explain about 61% of total variance.

Now plotting the estimated factor returns
```{r}
plot(fit.pca, which=2)
```

Estimated factor loadings for all assets
Factor loading is the correlation coefficient for the variable and factor.
```{r}

plot(fit.pca, which=3, a.sub=1:10)

```
First factor has all positive loadings, whereas the second factor has both positive and negative loadings.


```{r}
t(fit.pca$mimic)

plot(fit.pca, which=12, n.top=3)
```
This figure displays the top three assets with the largest and smalles weights in each factor mimicking portfolio. For the first factor, NVIDA, Amazone and Adobe have the highest weights and Amgen, UPS and Microsoft have the lowest weights. Since all weights are positive this might be construed as a market-wide factor. For the second factor, Amazon, Qualcom and Cisco have the highest weights and NVIDA, Apple and Adobe have the lowest weights.

Now plotting the correlations between assets with the top 3 largest and smallest positions in the F.1 factor mimicking portfolio

```{r}
plot(fit.pca, which=13, f.sub=1, n.top=3)
```
Here we can see the correlations between assets with top 3 largest and smallest weight in the factor mimicking portfolio for the first principal component. Correlations are very different.


```{r}
plot(fit.pca, which=13, f.sub=2, n.top=3)
```
Here we can see the correlations between assets with top 3 largest and smallest weight in the factor mimicking portfolio for the first principal component. Pretty high correlations overall.





#### S3 generic methods
all estimaded coefficients from PCA including intercept
```{r}
coef(fit.pca)
```

compare returns data with fitted and residual values for AAPL: fit.pca
```{r}
AAPL.ts <- merge(fit.pca$data[,1], fitted(fit.pca)[,1], residuals(fit.pca)[,1])

colnames(AAPL.ts) <- c("AAPL.return", "AAPL.fitted", "AAPL.residual")

tail(AAPL.ts)
```

fitted(): returns an xts data object of the component of asset returns explained by the factor model

residuals(): returns xts data object with the component of asset returns not explained by the factor model

Summary for fit.pca with HAC standard errors (allows for heteroskedasticity and autocorrelation consistent estimates and standard errors)
```{r}
sum.pca <- summary(fit.pca, se.type="HAC", n.top=3)

sum.pca$sum.list[[1]]
```
factor mimicking portfolio weights
```{r}
sum.pca$mimic.sum
```

### Factor Model Covariance and Risk Decomposition

#### Factor model covariance

```{r}
Omega <- fmCov(fit.pca)
# return correlation plot for all assets
plot(fit.pca, which=8, a.sub=1:10)
```

#### Standard deviation decomposition
```{r}
decomp <- fmSdDecomp(fit.pca)

#get the factor model standard deviation for all assets
decomp$Sd.fm

#get the component contribution to Sd
head(decomp$cSd)

#plotting
plot(fit.pca, which=9, f.sub=1:2, a.sub=1:10)
```

#### Value-at-Risk decomposition
```{r}
decomp1 <- fmVaRDecomp(fit.pca)

#factor model Value-at-Risk
head(decomp1$VaR.fm)

#Marginal factor contributions to VAR
head(decomp1$mVaR)

# plotting
plot(fit.pca, which=11, f.sub=1:2, a.sub=1:10)
```


####Expected Shorfall decomposition
```{r}
decomp2 <- fmEsDecomp(fit.pca)
# factor model Expected Shortfall
head(decomp2$ES.fm)

# percentage component contribution to ES
head(decomp2$pcES)

# plotting
plot(fit.pca, which = 10, f.sub=1:2, a.sub=1:10)
```

## Exercise 4: Timeseries Factor Models

Follow the file [tsfmVignette.pdf](https://github.com/braverock/FactorAnalytics/blob/master/vignettes/tsfmVignette.pdf) and interpret your results.

### Theorie
In a time series or macroeconomic factor model, observable economic time series such as industrial production growth rate, interest rates, market returns and inflation are used as common factors that contribute to asset returns. 
- For example, the famous *single index model by Sharpe* (1964) uses the market excess return as the *common factor* (captures economy-wide or market risk) for all assets and the *unexplained returns in the error term* represents the *non-market firm specific risk*. 
- On the other hand, *Chen et al. (1986) uses a multi-factor model* to find that surprise inflation, the spread between long and short-term interest rates and between high and low grade bonds are *significantly priced*, while the market portfolio, aggregate consumption risk and oil price risk are *not priced separately*.

```{r}

library(FactorAnalytics)

```


```{r}

# The following examples primarily use the managers dataset from the PerformanceAnalytics package. 
# It’s an "xts" data object with:
#                                 - 132 observations of monthly returns
#                                 - on 10 variables:
#                                     - six hypothetical asset managers, 
#                                     - 1x dhec returns (Long-Short Equity hedge fund index)
#                                     - 1x sp500 returns
#                                     - US treasury bonds 10 years (will serve as explanatory factors)
#                                     - US treasury bills 3 months (can be considered as the risk free rate)
#                                 - there are some "not available" observations (start day!)

data(managers)

# We want to see the managers names
colnames(managers)

# and we want to see from when to when the data is available 
first(managers)
last(managers)

```

```{r}

# the Ham1-Ham6 are the asset returns we want to explain --> y in our model
asset.names <- colnames(managers[,1:6]) 

# the edhec, sp500 & US Treasury they are the explanatory ones --> x in our model
factor.names <- colnames(managers[,7:9]) 

# Typically, factor models are fit using excess returns. If the asset and factor returns are not in excess return form, "rf.name" can be specified to convert returns into excess returns. 
rf.name <- "US.3m.TR"

# Similarly, market returns can be specified via "mkt.name" to add market-timing factors to the factor model.
mkt.name <- "SP500.TR" 

```

### Let’s take a look at the arguments for *fitTsfm*.

The default model fitting method is *LS regression* and the default variable selection method
is "none" (that is, all factors are included in the model). 
The different model fitting options are: 
1. least squares (LS), 
2. discounted least squares (DLS) and
3. robust regression fitting (Robust)


And variableselection options are:
1. "stepwise", 
2. "subsets" and 
3. "lars"

The default for rf.name and mkt.name are NULL. If rf.name is not specified by the user,
perhaps because the data is already in excess return form, then no risk-free rate adjustment is
made. Similarly, if mkt.name is not specified, market-timing factors are not added to the model.
All other optional control parameters passed through the ellipsis are processed and assimilated
internally by fitTsfm.control.

```{r}

# The series have unequal histories in this sample and “fitTsfm“ removes asset-wise incomplete cases (asset’s return data combined with respective factors’ return data) before fitting a factor model.
args(fitTsfm)

```

```{r}

# Single Index Model using SP500 
fit.singleIndex <- fitTsfm(asset.names=asset.names, 
                           factor.names="SP500.TR",   #specfic factor!
                           rf.name="US.3m.TR", 
                           data=managers)

# fitted object from the time-series LS regression of asset returns on estimated factors.
fit.singleIndex$asset.fit

# specifics values
fit.singleIndex$alpha
fit.singleIndex$beta

# Interpretation:
# if the market return rises 1%, then the return of Ham1 rises 0,39%
```

```{r}
# R-squared is a statistical measure of how close the data are to the fitted regression line. It is also known as the coefficient of determination, or the coefficient of multiple determination for multiple regression. The definition of R-squared is fairly straight-forward; it is the percentage of the response variable variation that is explained by a linear model
# R-squared is always between 0 and 100%:
#                                           0% indicates that the model explains none of the variability of the response data around its mean.
#                                           100% indicates that the model explains all the variability of the response data around its mean


fit.singleIndex$r2


# Interpretation:
# R-squared: 1 would be 100% - linear function matches perfectly with the data --> here we have low R-squared
```

```{r}
# The residuals show how far the data fall from the regression line and assess how well the line describes the data.
fit.singleIndex$resid.sd

# Interpretation:
# here we have low residuals

```

```{r}

class(fit.singleIndex)
# time series factor model
```

```{r}

names(fit.singleIndex)

```

Overview of the single factor linear fits for the assets. 
```{r}

fit.singleIndex
# Interpretation:
# How good does the single index model fits to the data?
# Ham1 equals a linear regression the most --> fits the best --> R-squared is the highest
# Ham5 does not really fit to this mode --> alfa and R-squared values

```

```{r}

plot(fit.singleIndex, which=12, f.sub=1)

```

### Henriksson-Merton's - market timing models
Market timing accounts for the price movement of the general stock market relative to fixed income securities.
This includes the down.market factor --> max(0, Rf-Rm)
To test market timing ability, this factor can be added to the single index model as shown below. The coefficient of this down-market factor can be
interpreted as the number of "free" put options on the market provided by the manager’s markettimings kills. That is, a negative value for the regression estimate would imply a negative value for market timing ability of the manager.

```{r}

# Henriksson-Merton's market timing model
fit.mktTiming <- fitTsfmMT(asset.names=asset.names, 
                           mkt.name="SP500.TR", # specify which of the columns in data corresponds to the market returns using argument mkt.name.
                           rf.name="US.3m.TR", 
                           data=managers)

t(fit.mktTiming$beta)

# Interpretation:
# Test market timing ability:
# when the value of down.market is negative, the ability of market timing of a manager is low --> not even there
# so the manager 2 has the best ability of market timing and after that manager 6 --> they have the hightes intercept (which return they will make when the market makes no return)

```


```{r}

fit.mktTiming$r2
# Interpretation:
# R^2 -> how good the data fits to the model. 
# all value have increased slightly

```

```{r}

fit.mktTiming$resid.sd
# Interpretation:
# volatility: how much it jumps around relative to its relationship to an index(sp500)
# risk: the higher the worse

###fit methods
#ls = least squares
#dls = discounted least squares (weightes least squares)
#robust = is good for data with outliers
```

```{r}
#comparison 
fit.mktTiming
fit.singleIndex
```


Fits Model:

The different model fitting options are: 

1. (ordinary) least squares (ols / LS) --> Default mode!
2. discounted least squares (DLS) and
3. robust regression fitting (Robust)

### Ordinary least squares ("ols")

```{r}
#  The next example performs LS regression using all 3 available factors in the dataset.
fit.ols <- fitTsfm(asset.names=asset.names, 
                   factor.names=factor.names, # all 3 available factors: the edhec, sp500 & US.10Y.TR/US Treasury
                   rf.name="US.3m.TR", 
                   data=managers) 


fit.ols$beta
# Interpretation:
# now we consider all factors (explanatory factors)

# Sensitivity: 
# when the return of edhec rises 1% --> Ham2 rises 0,1547%
# when sp500 return rises 1%, Ham2 decreases by 0,195%
# when US.10Y.TR return rises 1%, Ham2 decreases by 0,0504%
```

```{r}

fit.ols$r2
# Interpretation:
# how good does the data fit to the model
# Ham3 fits the best with 66%

```

```{r}

fit.ols$resid.sd
# Interpretation:
# Volatility
# How much they jump around --> most Ham4 0.0427
```


### Other options robust regression ("Robust"). 
```{r}

fit.robust <- fitTsfm(asset.names=asset.names, 
                      factor.names=factor.names, 
                      rf.name="US.3m.TR", 
                      data=managers, 
                      fit.method="Robust") # Method "Robust"!
fit.robust$beta
# Interpretation:

```

```{r}

fit.robust$r2
# Interpretation:
# R-squared is now lower for each
# maybe they all had outliers

```


```{r}

fit.robust$resid.sd
# Interpretation:

```

```{r}

par(mfrow=c(2,1))
plot(fit.ols, plot.single=TRUE, which=1, asset.name="HAM3")
mtext("LS", side=3)
```

```{r}
plot(fit.robust, plot.single=TRUE, which=1, asset.name="HAM3")
mtext("Robust", side=3)


# Interpretation:
# volatility is smaller when using the robust fitting method
### variable selection

# lars is a good variable to add
# least angle regression
# it is good when you are afraid of overfitting (that you adjust your model too much)
# when you have high-dimensional data (lots of explanatory factors)

```

```{r}

par(mfrow=c(1,2)) 
plot(fit.ols, which=5, xlim=c(0,0.045), sub="LS") 
plot(fit.robust, which=5, xlim=c(0,0.045), sub="Robust")

```

Though the R-squared values improved by adding more factors in fit.ols (compared to the single index model)

### Variable Selection
One might prefer to employ variable selection methods such as "stepwise", "subsets" or "lars" to avoid over-fitting. The method can be selected via the variable.selection argument. The default "none", uses all the factors and performs no variable selection:

- Specifying *"stepwise"* selects traditional stepwise LS or robust regression using step or step.lmRob respectively. Starting from the given initial set of factors, factors are added (or subtracted) only if the regression fit improves.
- Specifying *"subsets"* enables subsets selection using regsubsets. The best performing subset of any given size or within a range of subset sizes is chosen. Different methods such as exhaustive search (default), forward or backward stepwise, or sequential replacement can be employed.
- Finally, *"lars"* corresponds to least angle regression using lars with variants "lasso" (default), "lar", "forward.stagewise" or "stepwise".

#### LARS = least angle regression 

```{r}

fit.lars <- fitTsfm(asset.names=asset.names, 
                    factor.names=factor.names, 
                    data=managers, 
                    rf.name="US.3m.TR", 
                    variable.selection="lars") 

fit.lars
# Interpretation:
# Subset --> the best performing subset within a range of subset sizes is chosen

```

```{r}

fit.sub <- fitTsfm(asset.names=asset.names, 
                   factor.names=factor.names, 
                   data=managers, 
                   rf.name="US.3m.TR", 
                   variable.selection="subsets", 
                   nvmin=2, nvmax=2) 

fit.sub 
# Here, the best subset of size 2 for each asset is chosen by specifying nvmin = nvmax = 2. Note that when nvmin < nvmax, the best subset is chosen from a range of subset sizes [nvmin, nvmax]. Default is nvmin = 1.

# Interpretation:
# we see all together
# intercepts = alpha
# where we see the indices --> betas

```

```{r}

plot(fit.sub, which=2, f.sub=1:3)

```

```{r}

plot(fit.lars, which=2, f.sub=1:3)

```
### Big Interpretation
Comparing the *coefficients* and *R-squared values* from the two models, we find that the method that uses *more factors* for an asset have higher R-squared values as expected. However, when both "lars" and "subsets" chose the same number of factors, "lars" fits have a slightly higher R-squared values.


###  S3 generic methods
Many useful generic accessor functions are available for "tsfm" fit objects:

- coef() returns a matrix of estimated model coefficients including the intercept. 
- fitted() returns an xts data object of the component of asset returns explained by the factor model. 
- residuals() returns an xts data object with the component of asset returns not explained by the factor model. 
- predict() uses the fitted factor model to estimate asset returns given a set of new or simulated factor return data.
- summary() prints standard errors and t-statistics for all estimated coefficients in addition to R-squared values and residual volatilities. 

Argument se.type, one of "Default", "HC" or "HAC", allows for heteroskedasticity and auto-correlation consistent estimates and standard errors whenever possible. A "summary.tsfm" object is returned which contains a list of summary objects returned by "lm", "lm.Rob" or "lars" for each asset fit.

```{r}

methods(class="tsfm")

```

All estimated coefficients from the LS fit using all 3 factors
```{r}

coef(fit.ols)

```

Compare returns data with fitted and residual values for HAM1 from fit.lars

```{r}

HAM1.ts <- merge(fit.lars$data[,1], 
                 fitted(fit.lars)[,1], 
                 residuals(fit.lars)[,1]) 

colnames(HAM1.ts) <- c("HAM1.return","HAM1.fitted","HAM1.residual") 

tail(HAM1.ts)

# Interpretation:
# fitted --> the returns which can be explained through the model
# residual --> the returns which cannot be explained through the model
```

### Summary for fit.sub computing HAC standard erros

```{r}

summary(fit.sub, se.type="HAC")

```


### Factor Model Covariance & Risk Decomposition

#### Factor model covariance

```{r}
# the factor model covariance from a fitted factor model.
fmCov(fit.sub)

# factor model return correlation plot
plot(fit.sub, which=8)
```
#### Standard deviation decomposition

```{r}
# fmSdDecomp performs a decomposition for all assets in the given factor model fit object
decomp <- fmSdDecomp(fit.sub)
names(decomp)

# All Information together
decomp

# get:
decomp$Sd.fm
#     the factor model standard deviation for all assets
decomp$cSd
#     the component contributions to Sd
decomp$mSd
#     the marginal factor contributions to Sd
decomp$pcSd
#     the percentage component contributions to Sd

# plot the percentage component contributions to Sd
plot(fit.sub, which=9, f.sub=1:3)

```


#### Value-at-Risk decomposition

VaR = The value at risk for a given probability level indicates the amount of loss that will not be exceeded within a given period of time with this probability

```{r}

decomp1 <- fmVaRDecomp(fit.sub)
names(decomp1)

# All Information together
decomp1

# get the factor model value-at-risk for all assets
decomp1$VaR.fm

# get the percentage component contributions to VaR
decomp1$pcVaR

# plot the percentage component contributions to VaR
plot(fit.sub, which=11, f.sub=1:3)
```

#### Expected Shortfall decomposition

The term risk measure is a collective term for statistical measures that can be used to quantitatively describe the uncertainty of an event.
VaR is defined as the amount of loss that will not be exceeded in a given period of time with a specified probability p ("confidence level" α = 1 - p).

```{r}

decomp2 <- fmEsDecomp(fit.sub, method="historical")
names(decomp2)

# get the factor model expected shortfall for all assets
decomp2$ES.fm

# get the component contributions to Sd
decomp2$cES

# get the marginal factor contributions to ES
decomp2$mES

# get the percentage component contributions to ES
decomp2$pcES

# plot the percentage component contributions to ES
plot(fit.sub, which=10, f.sub=1:3)
```

#### Plot


#### Group Plots

```{r}

plot(fit.sub, which=6)
# Make a plot selection (1-12 or 0 to exit)

```

#### Individual Plots

```{r}

plot(fit.sub, plot.single=TRUE, asset.name="HAM1", which=10)

```
```{r}

plot(fit.sub, plot.single=TRUE, asset.name="HAM1", which=14)
grid()

```
```{r}

plot(fit.sub, plot.single=TRUE, asset.name="HAM1", which=11)

```
```{r}

plot(fit.sub, plot.single=TRUE, asset.name="HAM1", which=12)

```

## Exercise 5: Fundamental Factor Models

Follow the file [ffmVignette.pdf](https://github.com/braverock/FactorAnalytics/blob/master/vignettes/ffmVignette.pdf) and interpret your results.

### Theorie
The general mathematical form of equity fundamental factor model (FFM) implemented in factorAnalytics is
rt = αt· 1 + Bt−1ft + εt, t = 1, 2, · · · , T (1)

where:
- equity returns vector rt and the vectors 1 and 
- εt are N ×1 vectors, 
- Bt−1 is an N ×K exposures (risk factors) matrix, and 
- ft is a K × 1 vector of random factor returns. 
- It is assumed that the εt hav zero mean with diagonal covariance matrix Dt = diag(σ2t,1, σ2t,2, · · · , σ2t,N), and are uncorrelated
with the ft


```{r}

# factorAnalytics
# The following U.S. stock returns and data sets are included in factorAnalytics for now:
#
# 1) factorDataSetDjia: 
#                       - Monthly returns of 30 stocks in the DJIA from January 2000 to March 2013 = Dow Jones Industrial Average 
#                       - with 5 corresponding style factors:
#                           - MKTCAP  (Market Capitalization)
#                           - ENTVAL  (Entitled Value)
#                           - P2B     () Platform to Business
#                           - EV2S    ()
#                           - SIZE    ()
#                       - and a sector factor with 9 sectors:
#                           - ENERGY  (Energy), 
#                           - COSTAP  (Consumer Staples), 
#                           - INDUST  (Industry), 
#                           - MATRLS  (Materials), 
#                           - FINS    (Financials), 
#                           - INFOTK  (Information Technology), 
#                           - HEALTH  (Healthcare)
#                           - CODISC  (Consumer Discretionary), 
#                           - TELCOM  (Telecommunications)] 
# 
#
# 2) factorDataSetDjia5Yrs: 
#                       - A five-year segment of factorDataSetDjia from January 2008 to December 2012.
#
#
# 3) wtsDjiaGmvLo: 
#                       - Weight vector of dimension 30 containing the weights of a long-only 
#                         global minimum variance (GMV) portfolio for the 30 stocks in the factorDataSetDjia5Yrs data set.
#                       - The weight vector was obtained using PortfolioAnaltyics with the usual sample covariance matrix 
#                         based on the 5 years of returns in factorDataSetDjia5Yrs.


data("factorDataSetDjia5Yrs") 

dataDjia5Yr = factorDataSetDjia5Yrs 

head(dataDjia5Yr,5) 
``` 
 
 
### 1. Fitting a Fundamental Factor Model - FFM 
A FFM is generally fit by a two-step method:
1) using least squares or robust regression methods in the first step, 
2) and using weighted least squares or weighted robust regression in the second step where the weights are computed in the first step.

```{r} 
# The Fundamental-Factor-Model-Function: fitFfm

fitDjia5Yr = fitFfm(dataDjia5Yr, addIntercept = T, 
                    asset.var = "TICKER",                                 # asset.var = name of variable for asset names 
                    ret.var = "RETURN",                                   # ret.var = name of variable for asset returns 
                    date.var = "DATE",                                    # date.var = name of variable containing the dates 
                    exposure.vars = c("SECTOR","SIZE", "P2B", "EV2S"),    # exposrue.var = variables containing the fundamental factor exposures 
                    z.score = "crossSection")                             # z.score has to be one of "none", "crossSection", or "timeSeries" 

names(fitDjia5Yr) 
``` 
#### 1.1. Model Fit R-Squared Values
One of the most basic model fit statistics is the R-squared, and you can assess the goodness of your FFM fits by using the function ffmRsq to make a plot of the time series of R-squared values for each of the 60 fits over the five-year window. This function computes and plots the time series of ordinary R-squared by default, but it can do that for the adjusted R-squared, or both.

```{r} 

# Fit R-Squared Values

fmRsq(fitDjia5Yr,
      rsq = T,              # logical; if TRUE, Factor Model R-squared values are computed for the portfolio. Default is TRUE.
      rsqAdj = T,           # logical; if TRUE, Adjusted R-squared values are computed for the portfolio. Default is FALSE.
      plt.type = 2,         # 1 indicates barplot, 2 indicates time series xy plot. Default is 2.
      isPrint = F, 
      lwd = 0.7, 
      stripText.cex = 0.8, 
      axis.cex = 0.8) 
 
# Interpretation:
# 1) Upper Plot: Mean R-Squared across periods = 0.78 (Output of the Junk) 
#    -> problem, R-Squared increases everytime an independant variable is added, which means the more independant variables the higher the R-Squared. 
# 2) Mean adjusted R-Squared = 0.54 (Output of the Junk) 
#    -> as soon as we take the problem with R-Squared into account and use the adjusted mean, we see that the value drops from 78% to 54% and that there 
#       are even negative single values. 
 
``` 
#### 1.2 Model Fit Variance Inflation Factors - (VIF’s) 
When your model includes continuous style factor variables the function ffmRsq also allows you to compute and display the time series of variance inflation factors (VIF’s). These can help you determine whether or not there are any regression collinearity problems.
 
```{r} 

# the time series of mean variance inflation factors (VIF’s) 
vif(fitDjia5Yr, 
    isPlot = T, 
    isPrint = F, 
    lwd = 0.7, 
    stripText.cex = 0.8, 
    axis.cex = 0.8) 
 
#Multikollinearität ist ein Problem der Regressionsanalyse und liegt vor, wenn zwei oder mehr erklärende Variablen eine sehr starke Korrelation miteinander haben. Zum einen wird mit zunehmender Multikollinearität das Verfahren zur Schätzung der Regressionskoeffizienten instabil und Aussagen zur Schätzung der Regressionskoeffizienten zunehmend ungenau. Zum anderen ist die Modellinterpretation nicht mehr eindeutig. Das klassische Symptom von starker Multikollinearität ist ein hohes Bestimmtheitsmaß einhergehend mit niedrigen t-Werten für die einzelnen Regressionsparameter.

``` 
#### 1.3 Model Fit t-Statistics
Plots of the time series of t-statistics for the factors in an FFM fitted model, with horizontal dashed lines provided to judge whether or not a factor is significant at the 5% level, may be obtained with the function ffmTstats.
 
```{r} 

# The time series of t-statistics for the factors (5% significant level)
fmTstats(fitDjia5Yr, 
         whichPlot = "tStats", 
         color = "blue", 
         lwd = 0.7, 
         layout = c(3, 4), 
         stripText.cex = 0.8, 
         axis.cex = 0.8) 

# Interpretation:


``` 
 
```{r} 
# fmTstats can also compute the number of significant t-statistics:
#         the number of positive, 
#         negative and 
#         total significant t-statistics 

fmTstats(fitDjia5Yr, 
         whichPlot = "significantTstatsV", 
         color = "blue", 
         stripText.cex = 0.8, 
         axis.cex = 0.8, 
         layout = c(3, 4)) 

# Interpretation: 3 style factors + market  + 8 sectors

# red lines shows critical values t statistic; everything larger in (abolute value)  then critical value is signnificant 

# result of colinearity -> not many significant parameters

# see below: relativley low amount of significant parameters (execept for market)
``` 
 

### 2. Risk and Performance Report 
 The factorAnalytics package contains the following risk and performance reporting functions:
 1) repExposures
 2) repReturn
 3) repRisk 
 
We illustrate the use of each of these in turn using "fitDjia5Yr" and the corresponding global minimum variance long-only portfolio weights object "wtsDjiaGmvLo" 
 
```{r} 

# load the long-only global minimum variance portfolio weights vector 
data(wtsDjiaGmvLo) 

# and give it the shorter name wtsDjia 
wtsDjia = wtsDjiaGmvLo 

``` 
 
 
#### 2.1) repExposure 
The portfolio exposure to a given risk factor is the inner (dot) product of the portfolio weight vector with the column of the exposures matrix Bt−1 corresponding to the given factor. The style factors vary over time, but the sector factors are fixed and each sector is represented by a column of zero’s and ones. Thus the portfolio exposure to style factors will vary over time and thus have a distribution with a mean and volatility. On the other hand the portfolio exposure to sector factors will have a fixed value depending on the portfolio weights and number of firms in a given sector.
 
```{r} 

# compute volatilities of the style factors and sector factors 
repExposures(fitDjia5Yr,
             wtsDjia,
             isPlot = F,
             digits = 1, 
             stripText.cex = 0.8,
             axis.cex = 0.8) 

# Interpretation:
# Exposure generally refers to the fact of being exposed to a risk.
# Just three factors have a volatility:
# 1) Style factor P2B: with 20.0. 
# 2) Style factor EV2S: with -31.5
# 3) Stlye factor Size: with 4.6

``` 
 
 
 
```{r} 

# plot the volatilities of the style factors and sector factors 
repExposures(fitDjia5Yr, 
             wtsDjia, 
             isPrint = F, 
             isPlot = T, 
             which = 3, 
             add.grid = F, 
             zeroLine = T, 
             color = "Blue") 
``` 
 
 
```{r} 

# plot the time series of the style factor exposures 
repExposures(fitDjia5Yr, 
             wtsDjia, 
             isPrint = F, 
             isPlot = T, 
             which = 1, 
             add.grid = F, 
             zeroLine = T, 
             color = "Blue", 
             stripText.cex = 0.8,
             axis.cex = 0.8) 

``` 
 
```{r} 

#display boxplots of those exposures 
repExposures(fitDjia5Yr,
             wtsDjia,
             isPrint = FALSE, 
             isPlot = TRUE,
             which = 2,
             notch = F,
             layout = c(3,3)) 

``` 
 
 
#### 2.2) repReturn 
The function repReteurn provides you with the following choices of graphical reports, the results of which will also be printed because of the default printing option isPrint = T:
1. Time Series plot of portfolio returns decomposition
2. Time Series plot of portfolio style factors returns
3. Time Series plot of portfolio sector returns
4. Boxplot of Portfolio Returns Components.

 
```{r} 
# caluclate mean and volatility of the various portfolio return components with the help of repReturn 
repReturn(fitDjia5Yr,
          wtsDjia,
          isPlot = FALSE,
          digits = 2) 

# Interpretation: 
``` 
 
```{r} 

# Plot portfolio returns decomposition 
repReturn(fitDjia5Yr,
          wtsDjia,
          isPrint = FALSE,
          isPlot = TRUE, 
          which = 1,
          add.grid = TRUE,
          scaleType = "same", 
          color = "Blue",
          stripText.cex = 0.8,
          axis.cex = 0.8) 

# Interpretation: 

``` 
 
```{r} 

# Plot portfolio styles factor return 
repReturn(fitDjia5Yr, 
          wtsDjia, 
          isPrint = FALSE,
          isPlot = TRUE, 
          which = 2,
          add.grid = TRUE,
          zeroLine = T,
          color = "Blue", 
          scaleType = "same",
          stripText.cex = 0.8,
          axis.cex = 0.8) 

# Interpretation: 

``` 
 
```{r} 

# Plot portfolio sectors return 
repReturn(fitDjia5Yr,
          wtsDjia,
          isPrint = FALSE,
          isPlot = TRUE, 
          which = 3,
          add.grid = TRUE,
          zeroLine = T,
          color = "Blue", 
          scaleType = "same",
          stripText.cex = 0.8,
          axis.cex = 0.8) 

# Interpretation: 

``` 
 
```{r} 

# Boxplot of Portfolio Returns Components 
repReturn(fitDjia5Yr,
          wtsDjia,
          isPrint = FALSE,
          isPlot = TRUE,
          which = 4) 

# Interpretation: 

``` 

 
 
#### 2.3) repRisk 
The function *repRisk* allows one to compute and display (graphically and in tabular form) factor decompositions of risk for a portfolio and for each of the assets used to fit a fundamental factor model with fitFfm. The risk measure can be chosen as:
- standard deviation/volatility (SD)
- expected shortfall (ES)
- or value-at-risk (VaR)

and the factor risk decomposition can be chosen as:
- factor percent contribution to risk (FPCR)
- factor contribution to risk (FCR) 
- or factor marginal contribution to risk (FMCR). 
 
 
```{r} 
# standard deviation/volatility (SD)
# factor percent contribution to risk - (FPCR)

# First we compute an FPCR decomposition of the portfolio and individual assets using Sd as the risk measure, and provide both Lattice visualization and tabular displays. For the Lattice display the argument sliceby = “factor” specifies that the panel conditioning is by risk factor and the choice layout = c(5,1) results in a single row with five panels. We used nrowPrint = 10 to shorten the printed output from one row for the portfolio factor risk decomposition and 22 rows (we are only using 22 of the DJIA stocks at the moment) for the stock factor risk decompositions to one row for the portfolio and 9 rows for the assets. 
 
fitDjia5YrIntStyle = fitFfm(data = dataDjia5Yr,
                            exposure.vars = c("SIZE", "P2B", "EV2S"), 
                            date.var = "DATE", 
                            ret.var = "RETURN", 
                            asset.var = "TICKER", 
                            fit.method = "WLS", 
                            z.score = "crossSection", 
                            addIntercept = T) 
 
repRisk(fitDjia5YrIntStyle, 
        wtsDjia, 
        risk = "Sd", 
        decomp = "FPCR", 
        nrowPrint = 10, 
        sliceby = "factor", 
        isPrint = T, 
        isPlot = T, 
        layout = c(5, 1), 
        stripText.cex = 0.8, 
        axis.cex = 0.8) 

# Interpretation: 


``` 
 
```{r} 
# expected shortfall (ES)
# factor percent contribution to risk - (FPCR)

# Now we use expected shortfall (ES) to do an FPCR decomposition and provide only the Lattice 
repRisk(fitDjia5YrIntStyle, 
        wtsDjia, 
        risk = "ES", 
        decomp = "FPCR", 
        nrowPrint = 10, 
        sliceby = "factor", 
        isPrint = F, 
        isPlot = T, 
        layout = c(5, 1),
        stripText.cex = 0.8, 
        axis.cex = 0.8) 

# Interpretation: 
# How many percent of standard deviation are determinded by systemic risk (size, p2b, ev2s) and idiosyncratic risk (Resid)
``` 
 
```{r} 

# expected shortfall (ES)
# factor contribution to risk (FCR) 

# Now we use expected shortfall (VaR) to do the FCR decomposition 

repRisk(fitDjia5YrIntStyle, 
        wtsDjia, 
        risk = "VaR",
        decomp = "FPCR", 
        nrowPrint = 10,
        sliceby = "factor",
        isPrint = F, 
        isPlot = T,
        layout = c(5, 1),
        stripText.cex = 0.8, 
        axis.cex = 0.8) 

# Interpretation: 

``` 
 
```{r} 

# compare the factor risk decompositions of a portfolio using all three risk measures, skipping the risk decomposition of the assets 
repRisk(fitDjia5YrIntStyle, 
        wtsDjia, 
        risk = c("Sd", "ES", "VaR"),
        decomp = "FPCR",
        sliceby = "factor", 
        isPrint = T,
        isPlot = TRUE,
        layout = c(5, 1),
        portfolio.only = T, 
        stripText.cex = 0.8,
        axis.cex = 0.8) 

# Interpretation: 

``` 
 
### 3. Factor Model Monte Carlo 
The use of factor model Monte Carlo (FMMC) for a fundamental factor model results in a simulated set of asset returns based on a resampling of factor returns and a resampling or simulation of residual returns of the fitted model, using the exposures matrix from the last time period used in fitting the model.  
The factorAnalytics function fmmcSemiParametric implements the above FMMC method based on function araguments that are the result of first fitting  fundamental factor model to the data with fitFfm, combined with function arguments based on user options concerning the type of FMMC. We will illustrate the use of fmmcSemiParametric on the DJIA five-year monthly data set factorDataSetDjia5Yrs. 

 
```{r} 

# But first we take a look at the arguments of the function 
args(fmmcSemiParam) 

``` 
Aguments:
- B is the number of bootstrap samples.
- factor.ret is the set of factor returns estimates returned by the use of fitFfm
- beta is exposures matrix for the final period returned by fitFfm
- alpha is a fixed vector of intercept values that if ommited are assumed to be zero. 

Our example below uses the default values B = 1000, boot.method = “random” (means simple bootstrap), seed = 123 (for reproducibility of the example).  

We use two choices of resid.dist, first we use resid.dist = “empirical” which corresponds to 2-(a) above and then we use resid.dist = “normal”. The user must provide appropriate values for resid.par that depend on the the choice for resid.dist. For the choice resid.dist = “empirical” the resid.par must be the N × T dimensional xts time series in the $residuals component of the model fit, and for the choice resid.dist = “normal” the resid.par must be an N × 2 matrix with the first column being estimates of the means of the residuals for each of the N assets and the second column being estimates of the standard deviations of the residuals for each of the assets 
 
```{r} 
# In order to use fmmcSemiParam for the DJIA data we first fit a fundamental factor model (without alpha or market term) to the factorDataSetDjia5Yrs data 
data("factorDataSetDjia5Yrs") 
N = 30 

exposure.vars <- c("P2B", "MKTCAP", "SECTOR") 

fit.ffm = fitFfm(data = factorDataSetDjia5Yrs,
                 asset.var = "TICKER", 
                 ret.var = "RETURN",
                 date.var = "DATE",
                 exposure.vars = exposure.vars) 
``` 
 
```{r} 

# Next we use fmmcSemiParam to create simulated values of the asset returns based on the use of bootstrapped factor returns and bootstrapped (empirical) residuals: 
resid.par = fit.ffm$residuals 

fmmcDat = fmmcSemiParam(B = 1000,
                        factor.ret = fit.ffm$factor.returns, 
                        beta = fit.ffm$beta, 
                        resid.par = resid.par,
                        boot.method = "random", 
                        resid.dist = "empirical") 
names(fmmcDat) 
``` 
"sim.fund.ret" = a B × N matrix of simulated asset returns, "boot.factor.ret" = a B × K matrix of simulated factor returns, "sim.resid" = a B × N matrix of simulated residuals 
 
```{r} 

# Now let’s verify that the that returns of the 30 DJIA stocks over the five-year period are well represented by the set of 500 simulated sets of 30 returns in fmmcDat$sim.fund.return with respect to returns means and standard deviations. In order to do this we first extract the multivariate time series of returns of those stocks from the data frame factorDataSetDjia5Yrs 
data = factorDataSetDjia5Yrs 

djiaDat = tapply(data$RETURN,
                 list(data$DATE, data$TICKER),
                 I) 

djiaRet = xts(djiaDat,
              as.yearmon(rownames(djiaDat))) 

``` 
 
```{r} 

# Now we compare the simulated returns means with the observed returns means for the first 10 DJIA stocks 
round(apply(djiaRet, 2, mean)[1:10], 3) 

round(apply(fmmcDat$sim.fund.ret, 2, mean)[1:10], 3) 

``` 
 
```{r} 

#same thing for returns standard deviations 
round(apply(djiaRet, 2, sd)[1:10], 3) 
round(apply(fmmcDat$sim.fund.ret, 2, sd)[1:10], 3) 

``` 
 
The use of *fmmcSemiParam* with bootstrapped residuals as well as bootstrapped factor returns is attractive because it is simple and because in addition to making no distributional assumptions about the factor returns it makes no assumptions about the distributions of the residuals. 
 
By way of contrast let’s see what happens if we assume the residuals associated with the 30 DJIA fitted fundamental factor model returns have normal distributions and fit them using the sample means and standard deviations of the residuals.  
 
 
```{r} 

# First we use fmmcSemiParam with default choice resid.dist = “normal” and with resid.par a matrix with first column the sample mean of the residuals and second column the standard deviation of teh residuals 
 
resid.mean = apply(B = 1000,
                   coredata(fit.ffm$residuals),
                   2, 
                   mean,
                   na.rm = T) 

resid.sd = matrix(sqrt(fit.ffm$resid.var)) 
 
resid.par = cbind(resid.mean, resid.sd) 

fmmcDatNormal = fmmcSemiParam(factor.ret = fit.ffm$factor.returns, 
                             beta = fit.ffm$beta, 
                             resid.par = resid.par,
                             boot.method = "random") 
``` 
 
```{r} 

#Then we compare the means of the simulated asset returns with those of the actual returns 
round(apply(djiaRet, 2, mean)[1:10], 3) 
round(apply(fmmcDatNormal$sim.fund.ret, 2, mean)[1:10], 3) 

``` 
 
```{r} 

#Same with the standard deviation 
round(apply(djiaRet, 2, sd)[1:10], 3) 
round(apply(fmmcDatNormal$sim.fund.ret, 2, sd)[1:10], 3)

``` 
 
Once again the mean values agree quite well, but we see that the simulated returns based on the assumption of normally distributed returns have volatilities that under-estimate the actual returns volatilities for eight of the first 10 stocks. However, comparison of the volatilities for all 30 stocks reveals that there are only 13 stocks for which the volatilites for the simulated returns are smaller than those of the actual returns, and that when the volatilities of the simulated returns are larger than those of the actual returns they are much larger, for example .093 versus .065 for CAT and .132 versu .068 for HD 
 
Main message: It is not safe to use normal distributions in modeling stock returns with a fundamental factor model (or otherwise). It is for this reason that fmmcSemiParam allows you to use skewed t-distributions and Cornish-Fisher quantile rerpresentation of non-normal distributions for the residuals. 
 
 
### 4. Market plus Industry plus Country Model 
 
In this discussion we treat the terms “Industry” and “Sector” interchangeably, noting that for some models, e.g., a U.S. equity model, one may prefer to just use sector factors but may also wish to use industry factors, and a global model with countries may also contain industry factors. Our current examples use sector factors but we refer to them in our mathematical models loosely as industry factors. 
 
 
We will illustrate use of fitFfm to fit a market plus sector model to the DJIA stock returns and sector data. But first we fit a pure sector model without a market component and examine the factor return coefficients for the first month of the five-year fitting window as a reference point. 
 
```{r} 

dat = factorDataSetDjia5Yrs 
fitSec = fitFfm(dat,
                asset.var = "TICKER",
                ret.var = "RETURN", 
                date.var = "DATE",
                exposure.vars = "SECTOR") 

``` 

```{r} 

round(coef(summary(fitSec)$sum.list[[1]])[, 1], 3) 
round(fitSec$factor.returns[1, ], 3) 

``` 
 
Note that the last two lines of code produce identical results. This is because without any constraints such as those discussed above, the coefficients of the cross-section regression at each time period are extracted to form the time series of factor returns in the factor.returns component of the ffm object. Now we fit a market plus sector model by adding the fitF argument addIntercept = T,and examine the coefficients gˆmi,1 and the resulting factor returns ˆfmi,1 for the first month of the fiveyear fitting window. 
 
```{r} 

fitSecInt = fitFfm(dat,
                   asset.var = "TICKER",
                   ret.var = "RETURN", 
                   date.var = "DATE",
                   exposure.vars = "SECTOR",
                   addIntercept = T) 

round(coef(summary(fitSecInt)$sum.list[[1]])[, 1], 2) 
#excactly the same as before, but we add intercept t; therefore we add the market component
``` 

```{r} 

round(fitSecInt$factor.returns[1, ], 2) 

``` 
 
```{r} 

round(sum(fitSecInt$factor.returns[1, -1]), 2) 
# Summed returns of all sector should equal to zero, due to the relationship of the sectors
``` 
 
Note that the next to last line of code above prints the unique least squares model coefficients vector gˆmi,1 for month 1 (9 of them since there are 9 sectors) 
 
 
### 5. A Simultated Data Example 
 
We have created an artificial example of a market+sector+country model (where sector plays the role of industry) consisting of random returns of 30 stocks with three sectors for the sector factor and two countries for the countries factor, for each of five months. The normally distributed returns for the three sectors alone have means of 1, 2, 3, with standard deviations .2. The two countries contribute additional normally distributed returns having means 4 and 5 with standard deviations .2. So returns associated with the first country have means 5, 6, 7 and means associated with the second 
country have means 6, 7, 8. Thus the overall mean of 6.5. The code for creating the returns is as follows: 
 
```{r} 

# Country Incremental Components of Asset Returns 
set.seed(10000)

Bind = cbind(rep(1, 30), 
             c(rep(1, 10), rep(0, 20)), 
             c(rep(0, 10), rep(1, 10), rep(0, 10)), 
             c(rep(0, 20), rep(1, 10))) 

cty1 = matrix(rep(c(0, 1), 15)) 

cty2 = matrix(rep(c(1, 0), 15)) 

Bmic = cbind(Bind, cty1, cty2) 

dimnames(Bmic)[[2]] = c("mkt", "sec1", "sec2", "sec3", "cty1", "cty2") 

r.add = rnorm(30, 4, 0.2) 
r.cty1 = rep(0, 30) 
r.cty2 = rep(0, 30) 

for (i in 1:30) 
  {
  if (Bmic[i, "cty1"] == 1) 
    { 
    r.cty1[i] = r.add[i] 
    r.cty2[i] = 0 
    } 
  else 
    { 
      r.cty1[i] = 0
      r.cty2[i] = r.add[i] + 1
    } 
  } 


# Asset Returns for Market+Industry+Country Model 
mu = c(1, 2, 3) 
sd = c(0.2, 0.2, 0.2) 
r = list() 
r.mic = list() 
fitMic = list() 
fitMic1 = list() 

for (i in 1:5) 
  {
  set.seed(1099923 + (i - 1)) 
  r[[i]] = c(rnorm(10, mu[1], sd[1]), 
             rnorm(10, mu[2], sd[2]), 
             rnorm(10, mu[3], sd[3])) 
  r.mic[[i]] = r[[i]] + r.cty1 + r.cty2 
  } 

``` 
 
```{r} 

#qq-plot of the 30 asset returns for the first of the 5 time periods 
#Die Beobachtungswerte zweier Merkmale, deren Verteilung man vergleichen will, werden jeweils der Größe nach geordnet. Diese geordneten Daten werden zu Wertepaaren zusammengefasst und in einem Koordinatensystem abgetragen. Ergeben die Punkte (annähernd) eine Gerade, kann man vermuten, dass den beiden Merkmalen die gleiche Verteilung zu Grunde liegt. Problematisch ist das Verfahren, wenn von den beiden Merkmalen unterschiedlich viele Beobachtungen vorliegen. Hier kann mit Interpolationsverfahren abgeholfen werden.
qqnorm(r.mic[[1]],
       main = "MIC Model Equity Returns for First Period", 
       xlab = "NORMAL QQ-PLOT",
       ylab = "RETURNS") 
 
``` 
 
Now we build the data frame required by fitFfm, fit the MIC model and display the factor returns for each of the five months. What we have been calling the Industry factor is called Sector for this example 

 
```{r} 

Returns = unlist(r.mic) 

COUNTRY = rep(rep(c("US", "India"), 15), 5) 

SECTOR = rep(rep(c("SEC1", "SEC2", "SEC3"), each = 10), 5) 

TICKER = rep(c(LETTERS[1:26], paste0("A", LETTERS[1:4])), 5) 

DATE = rep(seq(as.Date("2000/1/1"), by = "month", length.out = 5), each = 30) 

data.mic = data.frame(DATE = as.character(DATE),
                      TICKER, 
                      Returns, 
                      SECTOR, 
                      COUNTRY) 

exposure.vars = c("SECTOR", "COUNTRY") 

fit = fitFfm(data = data.mic,
             asset.var = "TICKER", 
             ret.var = "Returns",
             date.var = "DATE",
             exposure.vars = exposure.vars, 
             addIntercept = T) 

fit$factor.returns 


``` 
 
We see that the Market values of the factor have values clustering around 6.5 as expected. We can also see that the three sector factor returns sum to zero and the two country factor returns sum to zero, as expected due to the constraints that they sum to zero.