Name Company Industry TotalCompMil
1 A. Bender COOPER COMPANIES INC Health 3.796108
2 A. Giannopoulos MICROS SYSTEMS INC IT 1.937943
3 A. Lafley PROCTER & GAMBLE CO Staples 19.086460
4 A. Mixon III INVACARE CORP Health 5.056142
5 A. Myers WASTE MANAGEMENT INC Industrial 5.695333
6 A. Perenchio UNIVISION COMMUNICATIONS INC Discretionary 0.000000
NetSalesMil Log10TotalComp SalaryThou
1 411.790 6.579339 461.3
2 400.191 6.287341 690.0
3 43373.000 7.280725 1600.0
4 1247.176 6.703819 948.0
5 11574.000 6.755519 1000.0
6 1311.015 NA 0.0
Histogram and boxplot
hist(execComp$SalaryThou, breaks=20, ylim=c(0,500), xlab="Salary (thousands of $)", main="Salaries of 1501 US CEOs in 2003") boxplot(execComp$SalaryThou, add=TRUE, horizontal=TRUE, at=475, boxwex=50, outcex=0.5)
par(cex=2)hist(execComp$SalaryThou, breaks=20, ylim=c(0,500), xlab="Salary (thousands of $)", main="Salaries of 1501 US CEOs in 2003") boxplot(execComp$SalaryThou, add=TRUE, horizontal=TRUE, at=475, boxwex=50, outcex=0.5)
Numerical summaries
Pay attention to the units and variable types.
summary(execComp)
Name Company Industry TotalCompMil
Length:1501 Length:1501 Length:1501 Min. : 0.000
Class :character Class :character Class :character 1st Qu.: 1.257
Mode :character Mode :character Mode :character Median : 2.533
Mean : 4.628
3rd Qu.: 5.477
Max. :74.750
NA's :6
NetSalesMil Log10TotalComp SalaryThou
Min. : 0.0 Min. :0.000 Min. : 0.0
1st Qu.: 471.8 1st Qu.:6.100 1st Qu.: 450.0
Median : 1237.4 Median :6.404 Median : 650.0
Mean : 5181.9 Mean :6.411 Mean : 696.8
3rd Qu.: 4044.8 3rd Qu.:6.739 3rd Qu.: 900.0
Max. :257157.0 Max. :7.874 Max. :3993.0
NA's :1 NA's :7
In the numerical summaries, you have seen:
measures of central tendency: mean, median
measures of location: median, quartiles (more generally, quantiles)
Each of these have their meanings and quirks.
Next up is a measure of spread called the standard deviation. The name may seem mysterious but it will be clear later when we talk about transformations.
We can calculate the standard deviation of every column of the dataset, but you quickly run into problems.
apply(execComp, 2, sd)
Name Company Industry TotalCompMil NetSalesMil
NA NA NA NA NA
Log10TotalComp SalaryThou
NA 374.3382
apply(execComp, 2, sd, na.rm=TRUE)
Name Company Industry TotalCompMil NetSalesMil
NA NA NA 6.231619e+00 1.469691e+04
Log10TotalComp SalaryThou
5.006985e-01 3.743382e+02
The name standard deviation has two components: standard and deviation.
The word “deviation” refers to the difference of each observation from a reference point.
For the standard deviation, the reference point is the mean.
The word “standard” refers to the ability to compare things on a common scale.
To illustrate where the word “standard” comes from, consider the data on salaries for CEOS. Recall the mean and standard deviation:
mean(execComp$SalaryThou, na.rm=TRUE)
[1] 696.7979
sd(execComp$SalaryThou, na.rm=TRUE)
[1] 374.3382
Next, let us look at some CEOs:
head(execComp[,c("Name", "SalaryThou")])
Name SalaryThou
1 A. Bender 461.3
2 A. Giannopoulos 690.0
3 A. Lafley 1600.0
4 A. Mixon III 948.0
5 A. Myers 1000.0
6 A. Perenchio 0.0
From the extract, you can see that rows 1, 2, and 6 have CEOs whose salaries are below the mean.
But how far below the mean? We can express this distance in terms of the original units: thousand dollars.
Distances are affected by the units chosen! So, one approach is to express this distance as a unitless quantity.
For example, A. Bender is \(\left| \frac{461.3-696.7979}{374.3382} \right|\approx 0.63\) standard deviations below the mean. In other words, A. Bender’s salary is about 3/5 of a standard deviation below the mean.
What we have done is to construct a \(z\)-score. This plays a role in getting a sense of magnitudes of typical datasets.
It turns out that for many datasets, it is very unlikely to have observations in the data which are more than 3 standard deviations above or below the mean. This is called the empirical rule.
There are definitely exceptions to this rule. But the rule, along with visualizations and other summaries, can be useful to detect potential issues with data.
The empirical rule states that for “bell-shaped” histograms,
About 68% of the observations are within one standard deviation of the mean.
About 95% of the observations are within two standard deviations of the mean.
About 99% of the observations are within three standard deviations of the mean.
CEO salaries have a long right tail, and:
mSal <-mean(execComp$SalaryThou, na.rm=TRUE)sdSal <-sd(execComp$SalaryThou, na.rm=TRUE) zSal <- (execComp$SalaryThou-mSal)/sdSalhist(zSal, ylim =c(0, 700), xlab ="Standard Units", main ="Standardized Salaries of 1501 US CEOs in 2003")
Let us get a rough sense of how well the empirical rule applies to CEO salaries.
mean(abs(zSal) >1, na.rm=TRUE)
[1] 0.2098601
mean(abs(zSal) >2, na.rm=TRUE)
[1] 0.0326449
mean(abs(zSal) >3, na.rm=TRUE)
[1] 0.01399067
There are other numerical summaries:
measures of spread: interquartile range, mean absolute deviation
measures of symmetry: skewness
measures of tail information: kurtosis, outliers
Almost all of these are univariate in nature, or applied one variable at a time.
Data transformations
There is a ton of whitespace in the pictures earlier.
It may be a good idea to “show more data” by considering a transformation.
But transformations may or may not have effects of numerical summaries.
Transformations could either be linear or nonlinear.
Linear transformations
Translation: adding or subtracting a constant amount
Scaling: multiplying or dividing by a constant amount
Changing millions USD to USD
Changing USD to EUR
Combination of both
Changing Celsius to Fahrenheit
Standardizing variables
Nonlinear transformations
One useful transformation is the logarithmic transformation. Logarithmic transformations are related to growth rates. We will see more of this later.
Here we use base 10. This may make sense for monetary amounts.
Dropping zero net sales and zero compensation did not change the reported values too much.
Think of these numerical summaries as averages or comparisons of averages.
Intercept: The average compensation of CEOs with net sales of zero is about 3.77 million dollars.
Slope: When we compare firms whose net sales differ by 1 million dollars, the average compensation of their CEOs differ by about 0.00017 million dollars or better yet, 170 dollars.
Even better: When we compare firms whose net sales differ by 1 trillion dollars, the average compensation of their CEOs differ by about 170000 dollars.
Take note that you are NOT comparing the same CEO! We are comparing different CEOs.
In the current context, would zero net sales be plausible?
Note that for communication purposes, it is really good practice to reduce the number of digits reported (more digits NOT the same as more accurate!) considering the context.
Regression with only an intercept
What happens when you apply lm() without NetSalesMil?
You will note that the intercept matches the mean of TotalCompMil. This pattern holds in general.
You can also think of the intercept as the slope of a regressor which takes on the value 1 for all observations.
Note that as soon as you include NetSalesMil, the meaning of the intercept will change.
Regression lines, fitted values, and residuals
In the compensation-sales relationship, the regression line is \[\widehat{\mathtt{TotalCompMil}}_t=3.7670804+0.0001673*\mathtt{NetSalesMil}_t\]
The regression line where I did not include NetSalesMil is \[\widehat{\mathtt{TotalCompMil}}_t=4.626\]
Both these lines could be used to produce fitted values. Just plug-in values for NetSalesMil from the data.
This plugging-in matters for the first line.
It does not matter for the second line.
It is possible to use these lines to make predictions outside of the data used for linear regression analysis. This also produces fitted values, which are more accurately termed as predicted values.
The regression line always passes through the point of averages or “the center”.
For the regression line \[\widehat{\mathtt{TotalCompMil}}_t=3.7670804+0.0001673*\mathtt{NetSalesMil}_t\] observe that
The difference between the TotalCompMil from the data and the fitted value produces the residuals: \[\mathtt{TotalCompMil}_t-\widehat{\mathtt{TotalCompMil}}_t\]
Provided that there is an intercept, the residuals are guaranteed to sum up to zero. So, the mean of the residuals is always equal to zero.
You can verify easily for the case where our regression line is just \[\widehat{\mathtt{TotalCompMil}}_t=4.626\]
For the regression line \[\widehat{\mathtt{TotalCompMil}}_t=3.7670804+0.0001673*\mathtt{NetSalesMil}_t\] observe that
Notice that the computed slopes are the same, but the intercepts are different.
Intercept if you used base 10: The average log compensation of CEOs with log net sales of zero is about 2.75. What about the units? Is it million dollars? Hard to communicate.
Slope: When we compare firms whose log net sales differ by 1, the average log compensation of their CEOs differ by about 0.4.
Even better: When we compare firms whose net sales differ by 1 percent, the average compensation of their CEOs differ by about 0.4 percent.
Where did the relative changes come from? Note that \(\log_b x_2 - \log_b x_1 = \log_b\left(\dfrac{x_2}{x_1}\right) \\ =\log_b\left[1+\left(\dfrac{x_2}{x_1}-1\right)\right] \approx \dfrac{1}{\log_e b}\left(\dfrac{x_2-x_1}{x_1}\right)\)
