A. K. Md. Ehsanes Saleh - Rank-Based Methods for Shrinkage and Selection

13 4 Analysis of Variance (ANOVA) 4.1 Introduction4.2 Model, Estimation and Tests4.3 Overview of Multiple Location Models4.3.1 Example: Corn Fertilizers4.3.2 One-way ANOVA4.3.3 Effect of Variance on Shrinkage Estimators4.3.4 Shrinkage Estimators for Multiple Location4.4 Unrestricted R-estimator4.5 Test of Significance4.6 Restricted R-estimator4.7 Shrinkage Estimators4.7.1 Preliminary Test R-estimator4.7.2 The Stein.Saleh-type R-estimator4.7.3 The Positive-rule Stein.Saleh-type R-estimator4.7.4 The Ridge-type R-estimator4.8 Subset Selection Penalty R-estimators4.8.1 Preliminary Test Subset Selector R-estimator4.8.2 Saleh-type R-estimator4.8.3 Positive-rule Saleh Subset Selector (PRSS)4.8.4 The Adaptive LASSO (aLASSO)4.8.5 Elastic-net-type R-estimator4.9 Comparison of the R-estimators4.9.1 Comparison of URE and RRE4.9.2 Comparison of URE and Stein.Saleh-type R-estimators4.9.3 Comparison of URE and Ridge-type R-estimators4.9.4 Comparison of URE and PTSSRE4.9.5 Comparison of LASSO-type and Ridge-type R-estimators4.9.6 Comparison of URE, RRE and LASSO4.9.7 Comparison of LASSO with PTRE4.9.8 Comparison of LASSO with SSRE4.9.9 Comparison of LASSO with PRSSRE4.9.10 Comparison of nEnetRE with URE4.9.11 Comparison of nEnetRE with RRE4.9.12 Comparison of nEnetRE with HTRE4.9.13 Comparison of nEnetRE with SSRE4.9.14 Comparison of Ridge-type vs. nEnetRE4.10 Summary4.11 Problems

14 5 Seemingly Unrelated Simple Linear Models 5.1 Introduction5.1.1 Problem Formulation5.2 Signed and Signed Rank Estimators of Parameters5.2.1 General Shrinkage R-estimator of β 5.2.2 Ridge-type R-estimator of β 5.2.3 Preliminary Test R-estimator of β 5.3 Stein.Saleh-type R-estimator of β 5.3.1 Positive-rule Stein.Saleh R-estimators of β 5.4 Saleh-type R-estimator of β 5.4.1 LASSO-type R-estimator of the β 5.5 Elastic-net-type R-estimators5.6 R-estimator of Intercept When Slope Has Sparse Subset5.6.1 General Shrinkage R-estimator of Intercept5.6.2 Ridge-type R-estimator of θ; 5.6.3 Preliminary Test R-estimators of θ; 5.7 Stein.Saleh-type R-estimator of θ; 5.7.1 Positive-rule Stein.Saleh-type R-estimator of θ; 5.7.2 LASSO-type R-estimator of θ; 5.8 Summary5.8.1 Problems

15 6 Multiple Linear Regression Models 6.1 Introduction6.2 Multiple Linear Model and R-estimation6.3 Model Sparsity and Detection6.4 General Shrinkage R-estimator of β 6.4.1 Preliminary Test R-estimator6.4.2 Stein.Saleh-type R-estimator6.4.3 Positive-rule Stein.Saleh-type R-estimators6.5 Subset Selectors6.5.1 Preliminary Test Subset Selector R-estimator6.5.2 Stein.Saleh-type R-estimator6.5.3 Positive-rule Stein.Saleh-type R-estimator (LASSO-type)6.5.4 Ridge-type Subset Selector6.5.5 Elastic Net-type R-estimator6.6 Adaptive LASSO6.6.1 Introduction6.6.2 Asymptotics for LASSO-type R-estimator6.6.3 Oracle Property of aLASSO6.7 Summary6.8 Problems

16 7 Partially Linear Multiple Regression Model 7.1 Introduction7.2 Rank Estimation in the PLM7.2.1 Penalty R-estimators7.2.2 Preliminary Test and Stein.Saleh-type R-estimator7.3 ADB and ADL2-risk7.4 ADL2-risk Comparisons7.5 Summary: L 2-risk Efficiencies7.6 Problems

17 8 Liu Regression Models 8.1 Introduction8.2 Linear Unified (Liu) Estimator8.2.1 Liu-type R-estimator8.3 Shrinkage Liu-type R-estimators8.4 Asymptotic Distributional Risk8.5 Asymptotic Distributional Risk Comparisons8.5.1 Comparison of SSLRE and PTLRE8.5.2 Comparison of PRSLRE and PTLRE8.5.3 Comparison of PRLRE and SSLRE8.5.4 Comparison of Liu-Type Rank Estimators With Counterparts8.6 Estimation of d8.7 Diabetes Data Analysis8.7.1 Penalty Estimators8.7.2 Performance Analysis8.8 Summary8.9 Problems

18 9 Autoregressive Models 9.1 Introduction9.2 R-estimation of ρ for the AR( p )-Model9.3 LASSO, Ridge, Preliminary Test and Stein.Saleh-type R-estimators9.4 Asymptotic Distributional L 2-risk9.5 Asymptotic Distributional L 2-risk Analysis9.5.1 Comparison of Unrestricted vs. Restricted R-estimators9.5.2 Comparison of Unrestricted vs. Preliminary Test R-estimator9.5.3 Comparison of Unrestricted vs. Stein.Saleh-type R-estimators9.5.4 Comparison of the Preliminary Test vs. Stein.Saleh-type R-estimators9.6 Summary9.7 Problems

19 10 High-Dimensional Models 10.1 Introduction10.2 Identifiability of β *and Projection10.3 Parsimonious Model Selection10.4 Some Notation and Separation10.4.1 Special Matrices10.4.2 Steps Towards Estimators10.4.3 Post-selection Ridge Estimation of βS1* and βS2*10.4.4 Post-selection Ridge R-estimators for βS1* and βS2*10.5 Post-selection Shrinkage R-estimators10.6 Asymptotic Properties of the Ridge R-estimators10.7 Asymptotic Distributional L 2-Risk Properties10.8 Asymptotic Distributional Risk Efficiency10.9 Summary10.10 Problems

20 11 Rank-based Logistic Regression 11.1 Introduction11.2 Data Science and Machine Learning11.2.1 What is Robust Data Science?11.2.2 What is Robust Machine Learning?11.3 Logistic Regression11.3.1 Log-likelihood Setup11.3.2 Motivation for Rank-based Logistic Methods11.3.3 Nonlinear Dispersion Function11.4 Application to Machine Learning11.4.1 Example: Motor Trend Cars11.5 Penalized Logistic Regression11.5.1 Log-likelihood Expressions11.5.2 Rank-based Expressions11.5.3 Support Vector Machines11.5.4 Example: Circular Data11.6 Example: Titanic Data Set11.6.1 Exploratory Data Analysis11.6.2 RLR vs. LLR vs. SVM11.6.3 Shrinkage and Selection11.7 Summary11.8 Problems

21 12 Rank-based Neural Networks 12.1 Introduction12.2 Set-up for Neural Networks12.3 Implementing Neural Networks12.3.1 Basic Computational Unit12.3.2 Activation Functions12.3.3 Four-layer Neural Network12.4 Gradient Descent with Momentum12.4.1 Gradient Descent12.4.2 Momentum12.5 Back Propagation Example12.5.1 Forward Propagation12.5.2 Back Propagation12.5.3 Dispersion Function Gradients12.5.4 RNN Algorithm12.6 Accuracy Metrics12.7 Example: Circular Data Set12.8 Image Recognition: Cats vs. Dogs12.8.1 Binary Image Classification12.8.2 Image Preparation12.8.3 Over-fitting and Under-fitting12.8.4 Comparison of LNN vs. RNN12.9 Image Recognition: MNIST Data Set12.10 Summary12.11 Problems

22 Bibliography

23 Author Index

24 Subject Index

1 Chapter 1Figure 1.1 Four plots using different versions of the telephone data set with fitted lines.Figure 1.2 Histograms and ordered residual plots of LS and Theil estimators.Figure 1.3 Effect of a single outlier on LS and rank estimators.Figure 1.4 Gradients of absolute value (Bn′(θ)) and dispersion (Dn′(θ)) functions.Figure 1.5 Scoring functions ϕ(u)=12(u−0.5) and ϕ+(u)=3u.Figure 1.6 Dispersion functions and derivative plots for 1.1(d).Figure 1.7 Key shrinkage characteristics of LASSO and ridge.Figure 1.8 Geometric interpretation of ridge.Figure 1.9 Geometric interpretation of LASSO.

2 Chapter 2Figure 2.1 The first-order nature of shrinkage due to ridge.Figure 2.2 Two outliers found in the Q–Q plot for the Swiss data set.Figure 2.3 Sampling distributions of rank estimates.Figure 2.4 Shrinkage of β 5due to increase in ridge tuning parameter, λ 2.Figure 2.5 Ridge traces for orthonormal, diagonal, LS, and rank estimators ( m = 40).Figure 2.6 MSE Derivative plot to find optimal λ 2for the diagonal case.Figure 2.7 Bias, variance and MSE for the Swiss data set...Figure 2.8 MSE for training, CV and test sets, and coefficients from the ridge trace.Figure 2.9 The first-order nature of shrinkage due to LASSO.Figure 2.10 Diamond-warping effect of weights in the aLASSO estimator for p = 2.Figure 2.11 Comparison of LASSO and aLASSO traces for the Swiss data set.Figure 2.12 Variable ordering from R-LASSO and R-aLASSO traces for the Swiss data set.Figure 2.13 Ranked residuals of the diabetes data set. (Source: Rfit() package in R.)Figure 2.14 Rank-aLASSO trace of the diabetes data set showing variable importance.Figure 2.15 Diabetes data set showing variable ordering and adjusted R 2plot.Figure 2.16 Rank-aLASSO cleaning followed by rank-ridge estimation.Figure 2.17 R-ridge traces and CV scheme with optimal λ 2.Figure 2.18 MSE and MAE plots for five-fold CV scheme producing similar optimal λ 2.Figure 2.19 LS-Enet traces for α = 0.0, 0.2, 0.4, 0.8, 1.0.Figure 2.20 LS-Enet traces and five-fold CV results for α = 0.6 from glmnet().