Generative vs. discriminative

Generative means "based on P(x,y)$P(x,y)$" and discriminative means "based on P(y|x)$P(y|x)$," but I'm confused on several points:

• Wikipedia (+ many other hits on the web) classify things like SVMs and decision trees as being discriminative. But these don't even have probabilistic interpretations. What does discriminative mean here? Has discriminative just come to mean anything that isn't generative?
• Naive Bayes (NB) is generative because it captures P(x|y)$P(x|y)$ and P(y)$P(y)$, and thus you have P(x,y)$P(x,y)$ (as well as P(y|x)$P(y|x)$). Isn't it trivial to make, say, logistic regression (the poster boy of discriminative models) "generative" by simply computing P(x)$P(x)$ in a similar fashion (same independence assumption as NB, such that P(x)=P(x0)P(x1)...P(xd)$P(x)=P(x_0)P(x_1)...P(x_d)$, where the MLE for P(xi)$P(x_i)$ are just frequencies)?

The fundamental difference between discriminative models and generative models is:

• Discriminative models learn the (hard or soft) boundary between classes
• Generative models model the distribution of individual classes
• The generative model would allow you to evaluate the likelihood of new pairs (x,y). The discriminative model allows you to predict the likelihood of different values of y given a value of x.

The generative model also has MORE to learn, since (in theory), you can always marginalize out y(summing over y) to get p(x), and then dividng the generative probability by that, you have p(y | x), the discriminative model.

Why not approach classification through linear regression?

Classification and Logistic Regression

As Andrew Ng explains it, with linear regression you fit a polynomial through the data - say, like on the example below we're fitting a straight line through {tumor size, tumor type} sample set: Above, malignant tumors get 1$1$ and non-malignant ones get 0$0$, and the green line is our hypothesis h(x)$h(x)$. To make predictions we may say that for any given tumor size x$x$, if h(x)$h(x)$ gets bigger than 0.5$0.5$we predict malignant tumor, otherwise we predict benign. Looks like this way we could correctly predict every single training set sample, but now let's change the task a bit. Intuitively it's clear that all tumors larger certain threshold are malignant. So let's add another sample with a huge tumor size, and run linear regression again: Now our h(x)>0.5→malignant$h(x)>0.5\to malignant$ doesn't work anymore. To keep making correct predictions we need to change it to h(x)>0.2$h(x)>0.2$ or something - but that not how the algorithm should work. We cannot change the hypothesis each time a new sample arrives. Instead, we should learn it off the training set data, and then (using the hypothesis we've learned) make correct predictions for the data we haven't seen before. Hope this explains why linear regression is not the best fit for classification problems! Also, you might want to watch VI. Logistic Regression. Classification video on ml-class.org which explains the idea in more detail.

Pearson's Correlation Coefficient

Pearson's Correlation Coefficient

In statistics, the Pearson product-moment correlation coefficient  is a measure of the linear correlation between two variables X and Y, giving a value between +1 and −1 inclusive, where 1 is total positive correlation, 0 is no correlation, and −1 is total negative correlation. It is widely used in the sciences as a measure of the degree of linear dependence between two variables.

Pearson's correlation coefficient is the covariance of the two variables divided by the product of their standard deviations. 

Pearson's correlation coefficient when applied to a population is commonly represented by the Greek letter ρ (rho) and may be referred to as the population correlation coefficient or the population Pearson correlation coefficient. The formula for ρ^[7] is:

where:

•  is the covariance
•  is the standard deviation of 

The formula for ρ can be expressed in terms of mean and expectation. Since

• ^[7]

Then the formula for ρ can also be written as

where:

•  and  are defined as above
•  is the mean of 
•  is the expectation.

The formula for ρ can be expressed in terms of uncentered moments. Since

• 
• 
• 
• 
• 

Then the formula for ρ can also be written as

 Covariance is a measure of how much two random variables change together. If the greater values of one variable mainly correspond with the greater values of the other variable, and the same holds for the smaller values, i.e., the variables tend to show similar behavior, the covariance is positive.^[1] In the opposite case, when the greater values of one variable mainly correspond to the smaller values of the other, i.e., the variables tend to show opposite behavior, the covariance is negative. The sign of the covariance therefore shows the tendency in the linear relationship between the variables. The magnitude of the covariance is not easy to interpret. The normalized version of the covariance, the correlation coefficient, however, shows by its magnitude the strength of the linear relation.

Correlation is a technique for investigating the relationship between two quantitative, continuous variables, for example, age and blood pressure. Pearson's correlation coefficient (r) is a measure of the strength of the association between the two variables.

The first step in studying the relationship between two continuous variables is to draw a scatter plot of the variables to check for linearity. The correlation coefficient should not be calculated if the relationship is not linear. For correlation only purposes, it does not really matter on which axis the variables are plotted. However, conventionally, the independent (or explanatory) variable is plotted on the x-axis (horizontally) and the dependent (or response) variable is plotted on the y-axis (vertically).

The nearer the scatter of points is to a straight line, the higher the strength of association between the variables. Also, it does not matter what measurement units are used.

Values of Pearson's correlation coefficient

Pearson's correlation coefficient (r) for continuous (interval level) data ranges from -1 to +1:

r = -1		data lie on a perfect straight line with a negative slope
r = 0		no linear relationship between the variables
r = +1		data lie on a perfect straight line with a positive slope

Positive correlation indicates that both variables increase or decrease together, whereas negative correlation indicates that as one variable increases, so the other decreases, and vice versa.