What is "privilege"? Talking to a modern liberal, you might get the idea that the word is synonymous with "advantage". But it isn't.

m-w.com has this:

privilege: a right or immunity granted as a peculiar benefit, advantage, or favor : PREROGATIVE; especially : such a right or immunity attached specifically to a position or an office

Under this definition, and in the domain of politics, advantages of birth such as wealth or skin color are not privileges. The thoughts, prejudiced or not, of our fellow citizens are not privileges. Only laws - actions of the state - are privileges in a political sense. It is certainly possible to talk about "private" privileges, such as having a key to the restroom in a particular private building. But these are hardly objectionable, and not what people talk about when they trot out the hackneyed "wealth and privilege".

dictionary.com has a more modern definition:

1.
a. A special advantage, immunity, permission, right, or benefit granted to or enjoyed by an individual, class, or caste.
b. Such an advantage, immunity, or right held as a prerogative of status or rank, and exercised to the exclusion or detriment of others.

So it is not wrong to use privilege to mean advantage. But I would argue that it is confusing verging on tenditious, and we should avoid it. Advantages that we enjoy, perhaps without even knowing it and by no act of any person, are very different things than benefits handed out unequally by law.

Privilege used to mean something specific in politics: that well-born men by law were not subject to certain rules; that only well-born men by law were allowed to vote and other advantages; that men controlled women by law; that monopolies and other benefits were restricted by law to certain men. It is these sorts of privileges that Liberals fought and triumphed against (back when "Liberal" meant about what "libertarian" means today). This was the project of the enlightenment; and it has it clearest expression in the 14th amendment: "nor shall any State ... deny any person within its jurisdiction the equal protection of the laws".

"The equal protection of the laws" - what a majestic phrase!

So if privilege is (or should be), about laws, why then do progressives want it to be about advantages? It seems clear to me that the point here is political. People are, rightly, against privilege (as I would have it), but they are not against mere advantages. If you want to convince people to use the law to mitigate certain disadvantages, then it is clearly advantageous in a democracy containing lots of credulous voters to characterize such disadvantages as unequal results of privilege.