When I was a kid, these billboards dotted the countryside, largely in response to the high court's ruling in Brown.
I've been listening to politicians and pundits bad-mouth the Supreme Court my entire life - threatening impeachment, charges of treason, etc., and for the most part those threats have come from the Right.
In any case, the Congress is granted broad discretion in the matter by the Constitution. We have had as few as five and as many as ten justices in our history. If I recall correctly, Andrew Jackson was able, with the help of Congress, to reduce the number of justices from seven to five just in time for the inauguration of his political enemy, J.Q. Adams.
Where we are now, however, such threats are empty. Although the justices themselves would probably welcome the increase in their numbers because it would greatly lessen their workload.