---
title: "MATH-650 Assignment 7"
author: 'Saket Choudhary (USCID: 2170058637) (skchoudh@usc.edu)'
date: "09/21/2015"
output:
  pdf_document: default
  html_document:
    keep_md: yes
  word_document: default
---

## Chapter 9: 16

### Part (a)
```{r}
library(ggplot2)
data <- read.csv('data_ch9_16.csv', header=T)
ggplot(data, aes(x=DurationOfVisit, y=PollenRemoved, color=BeeType)) +
    geom_point(shape=1) +
    scale_colour_hue(l=50) + 
    geom_smooth(method=lm,  
                se=FALSE)
```
From the graph we see that the relation between proportion removed and duration visited
is not linear

### Part (b)

```{r}
data$PollenRemovedlogit = log(data$PollenRemoved/(1-data$PollenRemoved))
ggplot(data, aes(x=DurationOfVisit, y=PollenRemovedlogit, color=BeeType)) +
    geom_point(shape=1) +
    scale_colour_hue(l=50) + 
    geom_smooth(method=lm,  
                se=FALSE)
```

From the plot above it does NOT seem that the logit transformed PollenRemoved has a linear relationship  with duration of visit

### Part (c)
Model:


\begin{align*}
\mu\{PollenRemovedLogit | DuratioOfVisitlog, BeeType \} &= \beta_0 + \beta_1 DurationOfVisitlog\\ 
&+ \beta_2 BeeType
\end{align*}



```{r}
data$DurationOfVisitlog = log(data$DurationOfVisit)
ggplot(data, aes(x=DurationOfVisitlog, y=PollenRemovedlogit, color=BeeType)) +
    geom_point(shape=1) +
    scale_colour_hue(l=50) + 
    geom_smooth(method=lm,  
                se=FALSE)
```

From the plot of  $logit(PollenRemoved)$ vs $log(DurationoOfVisit)$ it seems that these follow a linear relationship.


### Part (d)
Model:


\begin{align*}
\mu\{PollenRemovedLogit| DuratioOfVisitlog, BeeType \} &= \beta_0 + \beta_1 DurationOfVisitlog\\ 
&+ \beta_2 BeeType + \beta_3 BeeType*DurationOfVisitlog
\end{align*}


```{r}
lmfit <- lm(PollenRemovedlogit ~ BeeType + DurationOfVisitlog 
            + BeeType*DurationOfVisitlog, data=data)
summary(lmfit)
```

The p-value of the interaction term is 0.342 which is not significant at the threshold level of 0.05. Thus we cannot reject the null hypothesis that the interaction term is 0. Thus this tells us that there very little evidence that the proportion of pollen depends on duration of visits differently for queens than for workers

### Part (e)


\begin{align*}
\mu\{PollenRemovedLogit| DuratioOfVisitlog, BeeType \} &= \beta_0 + \beta_1 DurationOfVisitlog\\
&+ \beta_2 BeeType + \beta_3 BeeType*DurationOfVisitlog
\end{align*}



```{r}
lmfit <- lm(PollenRemovedlogit ~ BeeType + DurationOfVisitlog , 
            data=data)
summary(lmfit)
```

The p-value of the BeeTypeWorker coefficient is 0.02 which is significant for a 0.05 level threshold and the estimate of the slope is 0.5697 so this is the amount by which the mean number of pollen proportions exceeds that with bee type queen. And hence YES, there is enough evidence to say that while adjusting for the time spent on flowers, workers remove a larger proportion than queens(alternatively queen removes a smaller proportion than workers)

Rearranging the model equation:

\begin{align*}
\mu\{PollenRemovedLogit| DuratioOfVisitlog, BeeType \} &= \beta_0 + \beta_1 DurationOfVisitlog\\
&+ (\beta_2  + \beta_3 DurationOfVisitlog)BeeType
\end{align*}


With the interaction term included, the effect of the indicator variable is now $\beta_2+\beta_3 DurationOFVisitlog$ and hence the difference between the old(0.122) and new p-value(0.02) can be attributed to this difference in the model.


## Chapter 10: 20

$$ SS(\beta_0,\beta_1\dots \beta_n) = \sum_{i=1}^N(Y_i-\beta_0-\beta_1X_{1i}-\beta_2X_{2i}-\cdots-\beta_pX_{pi})^2 $$

\begin{align*}
\frac{\partial SS}{\partial \beta_0} &= 2 \sum_{i=1}^N(Y_i-\beta_0-\beta_1X_{1i}-\beta_2X_{2i}-\cdots-\beta_pX_{pi})\times -1 = 0\\
\end{align*}
\begin{align*}
n\beta_0+\beta_1\sum X_{1i}+\beta_2\sum X_{2i}+\cdots+\beta_p\sum X_{pi} &= \sum_{i=1}^N Y_i\\
\end{align*}

\begin{align*}
\frac{\partial SS}{\partial \beta_1} &= 2 \sum_{i=1}^N(Y_i-\beta_0-\beta_1X_{1i}-\beta_2X_{2i}-\cdots-\beta_pX_{pi})\times -X_{1i} = 0\\
\end{align*}
\begin{align*}
\beta_0\sum X_{1i}+\beta_1 \sum X_{1i}^2+\beta_2\sum X_{2i}X_{1i}+\cdots+\beta_p\sum X_{pi}X_{1i} &= \sum_{i=1}^N X_{1i}Y_i\\
\end{align*}


Similarly,

\begin{align*}
\frac{\partial SS}{\partial \beta_p} &= 2 \sum_{i=1}^N(Y_i-\beta_0-\beta_1X_{1i}-\beta_2X_{2i}-\cdots-\beta_pX_{pi})\times -X_{1i} = 0\\
\end{align*}
\begin{align*}
\beta_0\sum X_{pi}+\beta_1 \sum X_{1i}X_{pi}+\beta_2\sum X_{2i}X_{pi}+\cdots+\beta_p\sum X_{pi}^2 &= \sum_{i=1}^N X_{pi}Y_i\\
\end{align*}

To prove that this is indeed the minimum, we need to show that $$\frac{\partial^2 SS}{\partial \beta_i^2}$$ is convex:

\begin{align*}
\frac{\partial^2 SS}{\partial \beta_0^2} &= 2 \geq 0
\end{align*}


\begin{align*}
\frac{\partial^2 SS}{\partial \beta_1^2} &= 2\sum_i X_{1i}^2 \geq 0
\end{align*}

Similarly for any $1 \leq j \leq p$:
\begin{align*}
\frac{\partial^2 SS}{\partial \beta_j^2} &= 2\sum_i X_{ji}^2 \geq 0
\end{align*}

And for $$ k \neq j $$:

\begin{align*}
\frac{\partial^2 SS}{\partial \beta_j \beta_k} &= 2\sum_i X_{ji}X_{ki} \geq 0
\end{align*}

\begin{align*}
\begin{pmatrix}
\sum_i X_{1i}^2 & \sum_i X_{1i}X_{2i} & \dots & \sum_i X_{1i}X_{ni}\\
\sum_i X_{2i}X_{1i} & \sum_i X_{2i}^2 & \dots & \sum_i X_{2i}X_{ni}\\
\vdots & \vdots
\sum_i X_{ni}X_{1i} & \sum_i X_{2i}^2 & \dots & \sum_i X_{ni}^2
\end{pmatrix}
\end{align*}

each element in the  hessian matrix in this case can be written as $H_{ij}=\mathbb{x_i}^T\mathbb{x_j}$ and hence this is a Gram matrix and positive definite, hence minima at the above point is guaranteed.


## Chapter 10: 21

Inverse of $X^TX$ always exists, unless $p+1 > n$ that is to say the number of regressors are more than the number of samples.
This can still be solved by adding small values to $X^TX$ that makes it invertible.