Friday, November 25, 2016

The Fabulous EM-Drive

This blog post is what you might call "metascience." I address a scientific issue but do not use experiments or calculations, just educated common sense. Common sense is not always right or fair, but it usually is.

You may have seen news articles about a device called the EM-drive that is supposed to produce thrust without releasing any exhaust or other momentum-carrying particles like ions. That would violate conservation of momentum, which if true would start the biggest scientific revolution in a century, if not ever. (In an earlier post, I discussed conservation of momentum and the tricky way it applies to thrust-generating devices like rockets.) Conservation of momentum is connected to the fact that the laws of physics are independent of location, so you can see what trouble would arise if it turned out not to be true.

The EM-drive is a cavity in which electromagnetic (hence the EM) waves are generated and confined. NASA tested this device and reported to AIAA Journal that the EM-drive produced 1.2 millinewtons per kilowatt of power consumed. A millinewton is only equal to the weight of about two grains of rice, but it is comparable to the thrust of other types of electric rockets that do release momentum. It is certainly a measurable force. But, is it real?

The AIAA Journal paper is very careful in its technical approach. First, it documents an independent test run by someone other than the guy who invented the thing. Second, it lists nine separate possible sources of error that the researchers considered and rejected. But the results have a lot of scatter. One red flag is that the thrust per watt is reported to two decimal places, but just eyeballing the data shows that there is probably not even one significant digit, just a statistically significant difference from zero thrust.

Now, a cautionary tale from science history. In 1989 some chemists observed a sporadic heating effect in a certain kind of electrochemical cell. They could not explain the heating chemically and leaped to the conclusion that cold nuclear fusion was taking place. Cold fusion was then "replicated" by several other labs around the world. Like the EM-drive thrust, the heating was a small effect that was subject to many possible measurement errors. Although many labs replicated the effect, the reported temperature rises were inconsistent. Also, some labs could not replicate the effect. After a few months most scientists concluded that the heating was a measurement error, and cold fusion is now a fringe theory.

There are some striking parallels between cold fusion and the EM-drive:

1. Cold fusion heating and EM-drive propulsion are "small" effects. By small, I mean they require extraordinary measurement techniques to rule out error. We're not talking about mercury thermometers or postage scales here. But the experience physicists had with relativistic and quantum effects - both completely invisible at the human scale - prevents them from dismissing small effects out of hand.

2. Both the cold fusion and the EM-drive devices are claimed to have implications for fundamental physics, but neither was intended to demonstrate a new physical law. The required changes to physics came as explanations after the fact.

3. Both devices stand in contradiction to every other of the thousands or millions of attempts to invalidate their claims. In the EM-drive case, there have been myriad experiments in which very accurate momentum balances had to be measured, including ones that involve a resonant RF cavity.

This third point bears some discussion. Many people think of math as a science, but the standard of truth in math is logic, not experiment. For this reason, math is the least controversial of the sciences. No mathematical theorem was accepted for years before being proven wrong. But in physics, Newton's theory of gravity was thought to be final for centuries before being overturned by general relativity. So such things have happened. Generally, the less mathematical a science is, the more controversies it has. Biologists used to believe all kinds of crazy things, like spontaneous generation and the quinarian system. Don't even get me started about psychiatry or economics.

But then, there's the biggest red flag of all.

4. Both devices appeared in the media, with claims of new physics, before hardly any independent science at all had been done. The involvement of the media enticed the inventors to make some very bold, very public claims that they could not walk back without losing serious face. (My impression is that the cold fusion guys were in no way conscious frauds. They were scientists who stepped into media quicksand and never escaped.)

There are conflicting hypotheses about the measured EM-drive thrust. The most popular and likely is that it's measurement error. The AIAA Journal article claims that hidden quantum-mechanical variables could explain the effect. (Even Einstein was a fan of hidden-variable theories, but he was far outside the mainstream in quantum mechanics.) The builder of the EM-drive actually contends that the device does not violate any known laws of physics, but his explanation doesn't make a lot of sense to me.

Another red flag is the authors' use of the commercial physics simulation software COMSOL to support the hidden-variable claims. There's nothing wrong with COMSOL in its conventional uses, but I don't need a Ph.D. in physics to say that software that admits hidden-variable quantum mechanics could be used to prove just about anything you want. One author's self-identification as a "COMSOL Multiphysics Analyst" is just weird. That's like a mechanic calling himself a "wrench operator."

I am glad I was not asked to referee the AIAA Journal article. But if I had, I'd have recommended that it be published as a purely experimental paper, without all the theoretical stuff, just a frank admission that the result appears to violate known physics, and with a statistically valid analysis of the experimental scatter.

A good approach for a new study (hey, NASA!) would be to statistically evaluate the measurement errors that would be necessary to preserve momentum conservation. If that paper were markedly less convincing than the AIAA Journal paper, it would suggest that further investigation might not be a total waste of time.

Thursday, November 24, 2016

Coin-Flipping Paradox

Which takes fewer flips of a coin, getting two heads in a row, or getting a head followed by a tail? I'm talking about the average number of flips. It's of course possible to get lucky and hit two heads in the first two flips, or a head and a tail. But if you flipped a coin until you got HH or HT, and wrote down the number of flips, then did this say 10 times total, which one would come up faster, on average?

When this question was posed to me, I made the following argument: in both cases you have to flip a H, so there is no difference there. Once you get the first H, you are equally likely to flip a H or a T. The first would give you HH and the second would give you HT. Therefore, HH and HT must take the same number of flips, on average...right?

If that doesn't make sense to you, come up with your own answer, or try it out with a coin yourself. Then scroll down to find the real answer.

To find the average number of flips until HH appears, I started writing down all the ways you can get HH:

In 2 flips, the only way is HH, and there is a 1/4 chance of that.
In 3 flips, there is also only one way: THH. There's a 1/8 chance of that.
In 4 flips, TTHH and HTHH both work. Each of them has a 1/16 chance, but there are two ways, so the total probability of reaching HH in 4 flips is 2/16.
In 5 flips, TTTHH, THTHH and HTTHH; total probability of 3/32.
You can figure out the next two: for 6 flips it's 5/64 and for 7 flips it's 8/128.

Then I did the same thing for HT:

2 flips - one way, HT, 1/4 chance
3 flips - two ways. THT and HTH. 2/8 chance.
4 flips - TTHT, THHT, HHHT. 3/16 chance.

You can already see that there are more ways to get HT than HH. So it seems likely that it would take fewer flips on average to get HT than HH.  But what is the real answer?

The average number of flips will be the sum of the number of flips times the likelihood that number of flips gets you the desired result. Looking above at the flip sequences I listed for HH, \(n\) flips has a probability of \(F_{n-1}/2^n\) of getting you HH, where \( F_k \) is the kth Fibonacci number. (Look back at the numerators in the HH probabilities. 1, 1, 2, 3, 5, 8...) That's the Fibonacci sequence, which is defined as starting 0, 1 and then each following number is the sum of the previous two numbers. I think it's really spooky that the Fibonacci sequence shows up here. So the average number of flips is \[ \sum_{n=2}^\infty \frac{n F_{n-1}}{2^n} = 6, \]
So it takes 6 flips on average to get two heads in a row.

Looking at the probabilities for HT, the formula is \[ \sum_{n=2}^\infty \frac{n(n-1)}{2^n} = 4. \] It takes only 4 flips on average to get a head followed by a tail! As the mathematicians say, that's a counterintuitive result.

Exercise for the reader: Show that it takes exactly 2 flips on average to get one head.

Monday, November 21, 2016

Science = Show Business - Cocaine

The title may be a slight exaggeration. But there are some interesting parallels between how scientists and screenwriters make a living.

Scientists write white papers. Screenwriters write spec scripts.

Scientific funding agencies release "requests for proposals". Movie studios release "open assignments."

Scientists respond with proposals. Screenwriters respond with scripts.

Scientists use grant writers to help sharpen up a proposal. Screenwriters use script doctors to sharpen up a script.

Scientists write articles that, if they pass peer review, are published in journals. Screenplays and scripts are "green-lighted" for production.

Engineers and entrepreneurs turn scientists' ideas into goods and services. Producers and studios turn scripts into movies and TV shows.

Some scientists are satisfied to do good research that gets cited a lot, but others hunger for the recognition of the elite in the form of prizes like the Nobel. Screenwriters can sell a lot of scripts, but some want the recognition of their elite, which is Academy Awards.

Scientists have scholarly disciplines. Screenwriters have genres.

Scientists work within paradigms. Screenwriters write sequels, spinoffs and remakes.

Unethical scientists commit plagiarism. Unethical screenwriters commit...plagiarism.

The biggest difference is the role of grad students and postdocs in science. A show business parallel would be if studios hired bands of Irish storytellers to come over and invent characters and write all the dialogue and jokes, and the screenwriter's job was just to assemble them into a plot and put his name on the final script. Then the Irish storytellers would be analogous to the grad students and postdocs, and the screenwriter would be analogous to the full-fledged scientist or what is called the "principal investigator."

Saturday, November 12, 2016

Review of "Hillbilly Elegy"

I wrote this review before the election and offered it to Belt Magazine but they had no interest. I imagine ol' J.D.'s phone has been ringing off the hook for the last few days. The elites have finally realized that they don't understand the working class and that the nation's sociology departments are unlikely to, either.

Hillbilly Elegy: A Memoir of a Family and Culture in Crisis, by J.D. Vance (2016)

Well within living memory, the white working class was considered the backbone of the American economy. Capitalism had provided the unskilled with the heavy machinery they needed to be productive; an American high-school dropout plus a turret lathe was more productive than ten Russian potato farmers or mathematicians. When the common folk took to drink or went on strike, the establishment fussed over them like a mechanic tinkering with a balky car. Even when their ministrations were not effective, raw productivity drove ever-increasing incomes that countered many potential problems.

Something changed for the American white working class about forty years ago. Problems arose as they always had, but the elites now seemed unwilling or unable to help, so the problems just festered. Recent data about increasing death rates and opioid addiction, as well as the rise of Tea Party and Trumpian politics, have suggested things are reaching a boiling point, and policymakers have no answers.  The easy answer used to be further productivity gains, but those are now out of reach for people who lack specialized skills and the means to learn them. For now, at least, the unskilled in America are back to pre-industrial levels of productivity.

J.D. Vance’s important new book Hillbilly Elegy: A Memoir of a Family and Culture in Crisis puts a human face on the decline of the white working class. It follows the ups and (mostly) downs of a family of Appalachian transplants to the industrial Midwest of Middletown, Ohio. Deindustrialization lurks in the background, but Vance contends that the “working” class (he hesitates to use the term for people who don’t actually work) has always contained the seeds of its own destruction. Hillbilly Elegy offers a cultural, not just economic, take on recent working class history. It argues that the white working class no longer has the cultural resiliency needed to survive hard times.

Hillbilly Elegy is essentially autobiographical. One gets the feeling that Vance woke up the day after he graduated from Yale Law and was so puzzled by how in the hell he succeeded that he had to work it out on paper. Do not expect this book to be a tribute to role models or “good old American values”; Vance seems to have sought them out, and, finding none, made his thesis upon this lack of finding. There is an unintentional up-by-the-bootstraps flavor to the whole thing. Vance has not fully processed his story yet, and in the spirit of the best nonfiction writing, offers explanations that are plausible but never completely convincing. He looks back on a childhood of chaos, bouncing from one dire living situation to another because of his mother’s drug-fueled instability, finding his footing during a stint in the Marine Corps that physically removed him from the worst excesses of his upbringing, then working his way through Ohio State in record time. His story would have been worth telling even if it had stopped there, but he went on to graduate from Yale Law, which conferred an instant legitimacy that Vance finds as intriguing as any of his other experiences.   

Vance has a message everyone needs to hear, but his story is not typical. His hillbilly family is more dysfunctional than most, which makes his own astonishing rise all the more remarkable. As one might expect when the problem has been identified but thinkers are just beginning to come to grips with it, Elegy is riveting as narrative, but thin on answers. It is light on historical context where it is sorely needed, and as a result, Vance’s family may become the archetype of an already misunderstood people.

The massive labor force marshaled by 20th century capitalism was a patchwork of subcultures, so a cultural study like Elegy can only focus on one part of the overall picture. That part is the great wave of Appalachian migrants that swelled the rapidly industrializing cities of the Great Lakes states in the first half of the twentieth century. Studies of the role of Appalachian migrants are few. According to Susan Johnson’s University of Akron thesis Industrial Voyagers: A Case Study of Appalachian Migration to Akron, Ohio 1900-1940, over four million Southern whites migrated, and she cites a study that shows Appalachian migrants composing 13% of Cleveland’s population and 50% of Middletown’s by 1976.

From 1880, there were two Appalachias: the agricultural one and the mining one. In agricultural Appalachia, tobacco, corn and livestock farms supported market towns in picturesque valleys. Small colleges and steepled churches dotted the green, hilly landscape, inhabited by conservative, middlebrow people of English and German extraction. Agricultural Appalachia is where traditional mountain music is from.

Mining Appalachia was very different. It drew people from agricultural Appalachia, but it also drew people from the Deep South and from Europe. Mining Appalachia looked rural because of its isolation, but economically it was utterly industrial. Take Youngstown, add a dash of gold-rush California, and pull it into long strands enough to line the valleys over a few thousand square miles or so: that was Mining Appalachia. It was what economics textbooks call an extractive colony. Mining Appalachia is what supplied migrants to the Rust Belt, including Vance’s grandparents, who came from Breathitt County in eastern Kentucky to Middletown in the 1940s.

Vance mentions the nickname “Bloody Breathitt” which refers to a period of violent feuding that occurred between the Civil War and the 1920s. But it seems like every other county in Appalachia is called “bloody.” Near Breathitt is “Bloody Harlan,” Kentucky. My father went to high school across Tug Fork in “Bloody Mingo,” West Virginia. That’s three bloody counties in a hundred-mile radius.  The blood in Harlan and Mingo looked like labor unrest, but these were not like any strikes that happened outside Appalachia. They were thinly overlaid on the kind of violent, clannish partisanship that has arisen many times in the honor-based culture of the Scots-Irish natives of the region.

My father’s ancestors had been in Appalachia since before there was a United States. They followed the standard route: immigration from the English-Scottish border and northern Ireland to Philadelphia in the early 1700s, migration down the Great Wagon Road into western Virginia in the late 1700s, then dispersal into the bottomlands of the Tug Fork basin, just south of the Ohio, by 1830. If you could follow the typical mountaineer over the decades, that is the path he would have taken. The path would veer sharply north in the early 1900s, heading north on U.S. 23 or Ohio Route 21, taking Appalachian labor, ambition and problems to Cleveland, Toledo, Akron, Detroit and Buffalo.

Vance’s forebears are anomalous in that they do not appear to have spent time in coal mining, despite having lived in the coal fields. The experience of coal mining was an important way station for Appalachian workers between the subsistence agriculture of the 1800s and the industrial Midwest. One might imagine that a new migrant from an Appalachian coal town to, say, the Collinwood neighborhood of Cleveland would be hearing Italian or Hungarian spoken for the first time. In fact, this was rarely the case.  Coal companies followed a “thirds” scheme in each town, where one part native whites was leavened with one third immigrants and one third blacks from the Deep South. The “thirds” scheme kept wages down and dampened solidarity among the workers. The non-native workers moved on quickly when mining jobs grew scarce, often bearing little imprint of their years in the hills.

The white natives stuck it out longer, and by the time they eventually did move north to the industrial Midwest, they were not exactly country bumpkins.  They already knew what it was like to work for wages. They were comfortable operating big, expensive machines owned by someone else. But, whereas industry had replaced traditional white American cultures with the consumer culture in most places, such was not the case in Mining Appalachia.  White Appalachian miners were the heirs of an unusually tough traditional culture that prized independence above all other values.

Hillbilly Elegy will not educate you on Appalachian folk traditions. There are gaps in Vance’s cultural knowledge that leave him puzzled at times. He mentions the practice of giving sugary drinks to young children and how it causes rapid tooth decay, but cannot comprehend why Kentuckians took offense to a TV report about it. The sugary drinks are in fact the modern version of a very old cultural practice called the “sugar tit.” Pioneer mothers would occupy a cranky baby by letting him gnaw on a cloth soaked in sugared water or even sugared whiskey. Then as now, the baby’s teeth would rot, but secondary teeth would come in soon. Putting Mountain Dew in a baby’s bottle is pretty appalling, but Vance might feel better knowing it is an improvement over sugared whiskey.   

These were the descendants of Scottish dissenters who said that a bare room with a Bible in it was just as much a church as Westminster Abbey. When the United Mine Workers first tried to unionize the area, they found a surprising number of miners who were desperately poor but unwilling to give up their piecework arrangements in which a miner would dig coal when, where and how he chose, buying his own tools and being paid by the ton.  They had no interest in exchanging this for a fixed workday subject to rigid union rules.   

The UMW had better luck with the immigrants, who had been brought in at least partly to counter the unruly white native culture. The coal companies knew that when the native whites did manage to team up, it usually involved guns, and they were rightly afraid of this. Because of the “thirds” scheme, unionization failed consistently until it gained the explicit backing of Franklin Roosevelt.

Despite the fact that much of Appalachia rejected the Confederacy, West Virginia in particular owing its very existence to refusal to secede, Appalachian whites bore the full stigma of the Confederacy, and still do. The “white privilege” theorists tell us that racism persists because of the irreducible fact of skin color, which no amount of culture and achievement can change. But Appalachian whites carried a physical marker nearly as indelible as skin color: the deeply ingrained, archaic accent that pop culture has made into a universal symbol of backwardness and racism.

Dad came up to Cleveland in 1953, and one of his favorite writers was Don Robertson, the Cleveland columnist turned novelist who was Stephen King’s favorite writer as well. Robertson’s father was a newspaperman and so the idea of being a professional writer was real to him in a way it never could have been to Dad. In 1977, one of Robertson’s lesser Morris Bird III novels, The Greatest Thing That Almost Happened, was made into a TV movie.  The interesting thing was, while Morris Bird III was a white kid in all of the novels, in the TV movie he was played by the black actor Jimmie Walker. Dad wanted to ask Robertson, writer to writer, the obvious question of why it was decided to change Morris Bird III’s race. He got his chance at a book signing. But on hearing the question, and Dad’s accent, Robertson growled, “What are you, some kind of a racist?”

In Elegy, Vance chooses to focus on his family’s actions, less so on their appearance and hardly at all on their accent. This may be because they only ventured as far north as Middletown, Ohio, where the Appalachian dialect persists.  But Vance’s relatives were headed for trouble no matter their accent. The cast-iron voice box of his “Mamaw” (Mamaw is the Appalachian word for grandmother) spews forth torrents of obscenity that would be an obvious barrier to all but the meanest forms of employment.

My own Mamaw, who was orphaned at 11 and then raised by a violent, drunken farmer in northern Kentucky, was so mild and devout in her old age that the tirades of Vance’s Mamaw are all the more shocking to me. Not every old lady talks that way in Appalachia. On the other hand, when Vance’s Mamaw sets fire to her drunken husband (the old sot escapes with nary a singe), I couldn’t help but be reminded of a story my father told me about my Mamaw. When Dad’s brother was a teenager, he fell in with some older “Hunkies” (a catchall local word for eastern Europeans, not necessarily Hungarians) and returned to the house one day steaming drunk on their homemade liquor. Mamaw cleaned him up and put him to bed, then marched into the immigrant neighborhood, found the guilty parties, and said, “If I catch you near my son again, I will kill you.”

This quick resort to violence was vital in the time and place it evolved, but going around telegraphing to everyone that you’re a ticking time bomb is bound to cause problems in modern society. Once, a neighbor kid tried to run me down on his dirt bike, so I called the Lake County Sheriff. When Dad found out the police were coming, he figured the neighbor dad would be pounding on our front door soon, so he took out a bayonet and paced near the door, saying, “I’ll put him away.” (The guy never showed up.)  Another time, I was visiting home from Ohio State and went to the old Just Closeouts store in Painesville with Dad. A couple of guys cut in front of us at the register, and Dad silently opened his jacket to them, revealing an ivory-handled pigsticker. The two guys turned as white as the knife handle and found a different register; I fear if that incident were repeated today a SWAT team would have been called. Dad was retired from TRW and forty years removed from Appalachia when that happened.  

Alcohol, violence and religion play a central and connected role in Appalachian culture. Here Vance cannot help but dig a little deeper, and he quickly unearths a whiskey bottle but no Bible. There is little casual or social drinking in Appalachia. People drink to get drunk, and when they get drunk, they fight, and when they fight, they get arrested and lose their jobs. Much of Vance’s memoir could be summed up in that one previous sentence. Alcohol magnifies the tendency toward violence, but equally responsible is an Appalachian bent toward cultivating and deploying strong emotions.

There is only one path to redemption in Appalachia, and it isn’t Alcoholics Anonymous or prison; it’s the Lord. Religion appears in the form of the Pentecostalism of Vance’s biological father, whom he lives with briefly. Appalachian or Southern-style fundamentalist religion may seem to have theological content, but the hidebound insistence on the literal truth of the Bible is less theology than sociology. Religion is Appalachia is a weapon against alcohol and violence, not theology. Like any weapon, it has to be rigid and sharp. It has to offer a simple, clear path to heaven and define an equally clear path to hell.  A religion that admits doubt and tolerates change is intellectually superior, but is as useless as a foam Halloween sword as a weapon against bad behavior. 

Vance’s narrative is full of his relatives’ drunkenness and drugs, but he resists the too-easy connection between addiction and failure. Instead, he makes things a little more abstract:

…whenever people ask me what I’d most like to change about the white working class, I say, “The feeling that our choices don’t matter.”

Here we are caught between a rock and a hard place. Any thinking adult knows that people cannot completely control their own destinies. There is unfairness, sexism, racism and plain bad luck.  So we have replaced the morality tales of the past by a complicated story in which individual actions matter, but circumstances matter, too, and in different proportions for different people.

At the same time we started telling young people this more accurate, but more lenient, version of how the great game of life works, the connection between actions and consequences has actually grown far stronger. It is as if theory and practice have passed each other in opposite directions. In 1950, one might engage in what people used to call “hometown troubles” like petty crime, but be able to leave it all behind and start anew in another city. Not any more. Computerized record-keeping and social media ensure that every mistake you make is now preserved, context-free, for perpetual scrutiny by anyone.

We already know the economic reasons for the decline of the white working class, but now we come to a possible cultural reason. The larger culture, run by what pop business books call “thought leaders,” used to carefully box in the working class. They were given a religion, whether Catholic or Protestant, that did not hesitate to condemn certain behaviors not only as wrong but as a literal path to hell.  They were presented with moral examples via the media: the good guy always wins in the end; crime does not pay; a male-headed, intact family is the only one that works; drunkenness or sexual license only leads to misery. These were Kipling’s “gods of the copybook headings.” Finally, employers like Henry Ford did not hesitate to pry into their employees’ lives and tell them how to run their households.

To avoid the appearance of hypocrisy, the elites had to pretend to abide by these rules themselves. Americans have never tolerated one set of rules for one group and another set of rules for another. But the elites had money and brains, and could thrive without the dull moralizing. Movie stars can afford to be drunks. The rules had to go, and when they went, they went completely, because they were only a list of pat answers, not intellectual tools that could be used in changing times.

Vance realizes the working class is no longer fed a diet of moral simplicity, but why not? Did the rise of mass media make it impossible to hide all the complexities? Did consumerism result in telling people only what they want to hear? Did we think the world would soon end in nuclear war, so people should stop planning for the future and live for the moment? It’s hard to remember how pervasive the feeling of nuclear doom was up through the late 1980s.  These are points of departure for future works on the topic.

Now we live in a paradox where America’s elites are more morally pure than they’ve ever been. The public morality, whether it concerns race, feminism, sexual orientation, ethnic and religious minorities and even consumption patterns like recycling and organic food has a relentlessly progressive tone. Few would deny that this represents real progress, but it has perhaps come at a price. The great task is now to explain these changes to people in a way that does not invalidate the entire rulebook.

But there is more. We are told that Appalachian folk are a proud people, but does not pride get in the way of education? A very proud person thinks subjecting himself to a teacher is an admission of ignorance, and there is no greater shame than ignorance today. We are all supposed to have been born knowing the Fundamental Theorem of Calculus. Pride, which in past times shielded the working man from all the base temptations, now shields him from getting an education.

Vance finished his manuscript too early to have to deal with Donald Trump, but we must. Polls show that Trump’s base is formed from less educated (although not the economically worst-off) whites. He took 156,000 primary votes in West Virginia, a historically Democratic state. That is 32,000 more than it took Bernie Sanders to win the Democratic primary. The reporting about the Republican turn in West Virginia focuses on the Democratic Party’s hostility to coal, but there is more going on.

Trump’s great weakness nationwide is his blustery personality. When a public figure attacks Trump, he or she can expect to be quickly and personally insulted with a barrage of dubious claims. Many voters are repelled by this, but it costs Trump nothing among Appalachians and many of their descendants; quite the contrary. Why? Trump has no connections to Appalachia, but he is a cracker in the original sense of the word.  Listen to what the Earl of Dartmouth said about the frontier dwellers of western Virginia in 1760: “I should explain…what is meant by Crackers: a name they have got from being great boasters…”

The Appalachian-derived white working class has the cultural tools to know when to take Trump seriously and when not to.  They hear him and something in their subconscious says, “He is one of us.” This is the way they’ve talked to each other for centuries. These personality traits have deep, deep roots, and are not going to be much altered by a few decades’ exposure to Northern ways.  Vance has completely missed this connection in Elegy and in subsequent interviews. But Salena Zito of The Atlantic caught on. She said, "The press takes him literally but not seriously; his supporters take him seriously but not literally."

Donald Trump is a proud, boastful man who can survive the loss of an election and will no doubt profitably carry the public mantle of the opposition if he does lose. [Post-election note: that was a big "if", wasn't it?] But a proud, boastful community-college dropout in East Akron has no such path open. He has to eat his pride and take orders from supervisors and customers who treat him as if he has nothing to be proud of. The old ways that he soaked up from watching his relatives interact with each other and the outside world, have to give way if he is to succeed according to values he himself already accepts.

J.D. Vance realizes that the answer to the white working class’s problems cannot be the one he found, which is to go to Yale Law School and work in Silicon Valley venture capital. Nonetheless, his account of his time at Yale is valuable in that it lays bare the ignorance and hostility toward the working class found too often among our elites.  Hillbilly Elegy asks questions and demands answers, and should occupy a place on the growing bookshelf of efforts to understand why things have gone so wrong with the white working class.   

Wednesday, November 9, 2016

How I Predicted The Election

Preface: I take no position on which candidate is best, or who deserved to win. This is purely objective.

Last Sunday I gave Trump a 71% chance of winning, with an expected electoral vote total of 301. He of course did win and if the current counts hold up, he'll get 304 electoral votes.

"The triumphant vindication of bold theories - are these not the pride and justification of our life's work?" - Sherlock Holmes

None of the famous pollsters or pundits predicted this. On the very weekend I made my prediction, the New York Times gave Trump a 15% chance of winning and the Huffington Post gave him a mere 2% chance. Nate Silver, who knows something about uncertainty, was giving him a 35% chance, and he got excoriated for it. The Huffington Post said Silver was "putting his thumb on the scales" by adjusting the raw poll numbers (Silver doesn't do polls himself; he aggregates other polls) and "making a mockery of the very forecasting industry he popularized."

The Huffington Post has a naive idea of how forecasts are done. They seem to think you just call a thousand people at random, record their answers, and publish the number. But that would lead to very bad predictions. You have to correct your sample to match population characteristics such as likelihood of voting, geography, party affiliation, and a bunch of other stuff. Then, you have to account for the fact that you can only do the poll ahead of the election, and things are always changing. If your poll showed 25%, 35%, 45% in the three weeks preceding an election, you'd be kind of stupid to just run with the 45%.  Finally, there are individual decisions as to whether to accept or reject a data point - did the person answer all the questions that were asked, did he sound like he was giving obviously misleading answers, did he tell the truth about his age, party, and so on?

All of those corrections get influenced by human nature. People have a personally desired outcome, and also they get cold feet if their numbers come out too far from the other polls. It's a form of groupthink.

So what did I do? I took Nate Silver's "polls plus" predictions of the margin of victory in each state (accounting for DC and the weird split votes in Nebraska and Maine). Then I did 24 separate projections, for all possible integer combinations of a 0-7 point adjustment factor (call it "x") in Trump's favor and an 0-2 point "tossup margin". The states that were within the tossup margin I split evenly between Trump and Clinton. 301 electoral votes is Trump's average over those 24 projections. 

I didn't just invent the "x factor". There was talk of a "shy Trump" effect, which I thought had credibility. People didn't want to admit to a stranger over the phone that they were voting for Trump, because the media made it sound like voting for Trump was worse than armed robbery. They pounded on it all summer and into the fall. Further, they made it seem like Trump's chances were much worse than they really were. They were constantly saying Trump would quit, that his campaign was imploding, that there would be a credible spoiler candidate and so on. None of this would have stood up to the least bit of journalistic investigation, so do yourself a favor and ignore those people from now on.

There were other clues as well. Trump drew more primary voters than any Republican candidate in history, and his rallies were (sorry) HUGE. But those were only clues, not hard numbers.

I wanted something to back up the "x factor." My prior guess was 3 points, but I found two analyses to see whether Trump did better in anonymous online polls than in phone polls. One was from Nate Silver who aggregated a bunch of polls from the primary (that's his thing, aggregating stuff) and came up with a negligible difference. That, I ignored, because it was from the primary (whole different set of voters) and comparing aggregated numbers is a poor way to determine bias. The other was a very recent one by Politico. This one was useful and demonstrated everything that went wrong with pollsters.

First, the Politico headline was "Shy Trump Voters Are A Mirage" when the actual results said the opposite, so right away you can see some wishful thinking. (To be fair, the pollster probably didn't write the headline.) They compared a single phone poll to a single online poll and showed a 2% "x factor." That was dismissed as not statistically significant; in science, you would do a further study but this pollster didn't. But then things get interesting. They revealed that among voters with a household income of more than $50,000 a year, the "x factor" was 10%! About half of all voters fit into that category, so even if there was zero "shy Trump" effect for the other half of voters, there would be about a 5% "x factor."  - more than enough to flip the election. I suppose there could've been a negative "x factor" for voters with incomes less than $50,000 to cancel it out, but that seems really farfetched and they no doubt would have reported it.

A similar thing happened when they broke out the data for people with a B.S. degree or higher. So something obviously went wrong in the analysis. Somehow they adjusted away the "x factor" that was staring them in the face.

Now I admit my own bias. I got cold feet, too. I thought that even the conservative estimate of 5% from the Politico poll was too big, and I didn't have a lot of insight into what this Politico poll had actually done. So I set my "x factor" range to an average of only 3.5%. I figured if I got the basic election call right, nobody would care about the exact numbers, and 3.5% was enough to flip it. So my near-exact prediction of electoral votes owes something to luck.