[Multiple Linear Regression] 5 methods of building models | Stepwise Regression

In Multiple Iinear Regression Model, there are many variables. To build a model, we need to choose right variables ! 

( Using all the variables given in the data, it's NOT a good idea ) 

 

 

 

[ 5 methods of building models ] 

1. All-in

- Prior knowledge

- Preparing for Backward Elimination

 

 

 

2. Backward Elimination

Step 1) Select a significance level to stay in the model (ex) SL = 0.05  ---->  SL = Significance level 

 

Step 2) Fit the full model with all possible predictors

 

Step 3) Consider the predictor with the highest P-value.

=> If P > SL, go to STEP 4, otherwise go to FIN  ----> FIN = Your model is Ready

 

Step 4) Remove the predictor 

 

Step 5) Fit model without this variable

=> After Step 5, goes back to Step 3 and then look for the variable with the highest p-value

 

 

** the Scikit-Learn library automatically takes care of selecting the statistically significant features when training the model to make accurate predictions.

 

 

3. Forward Selection

Step 1) Select a significance level to enter the model (ex) SL = 0.05

 

Step 2) Fit all simple regression models y-xn 

=> Select the one with the lowest p-value

 

Step 3) Keep this variable and fit all possible models with one extra predictor added to the one(s) you already have

 

Step 4) Consider the predictor with the lowest p-value.

=> If P < SL, go to STEP 3, otherwise go to FIN  ----> FIN = Keep the previous model

 

 

 

4. Bidirectional Ellimination

Step 1) Select a significance level to enter and to stay in the model

(ex) SLENTER = 0.05, SLSTAY = 0.05

 

Step 2) Perform the next step of Forward Selection (new variables must have P < SLENTER to enter)

 

Step 3) Perform ALL steps of Backward Elimination (old variables must have P < SLSTAY to stay)

=> go back to Step 2

 

Step 4) Repeat Step until there is NO variables can enter and no old variables can exit 

=> go to FIN  ----> FIN = Your model is Ready

 

 

5. Score Comparison | All Possible Models

Step 1) Select a criterion of goodness of fit 

 

Step 2) Construct All Possible Regression Models : 2^n - 1 total combinations

 

Step 3) Select the one with the best criterion 

=> go to FIN  ----> FIN = Your model is Ready

 

 

 

 

 

[ Stepwise Regression ]

Stepwise Regression : Stepwise Regression includes Method 2,3,4 above. 

- Somtimes, Method 4 - Bidirectional Ellimination refered as stepwise regression 

 

2. Backward Elimination

3. Forward Selection

4. Bidirectional Ellimination