justanotherme23 - Tumblr blog

[justanotherme23]

A Moment in the History of Beer Brewing: How the Student’s T-test Doomed 5 Percent of Modern Research to Publishing Falsehoods

This man is William Sealy Gosset. He looks like a nerd. And he is, but don’t worry. He’s kinda cool. This man was also the Head Experimental Brewer at the famous Guiness Brewery during the start of the 20th century.

Willy Gosset had a very important question to answer. “How can you predict if a batch of hops will make a good batch of beer?” A fundamental question of life for the domesticated husband.

A known indicator for the quality of a batch of hops is the percentage in the batch of what’s called “soft resins”. Unfortunately, the ratio of resins is not consistent in all batches of hops. The plants that produce hops vary in their soft resin output from batch to batch and season to season. You can find an ideal ratio, but you’ve also got to find an acceptable margin of error; otherwise Mr. Gosset would have to scour the planet for that one batch of hops with the perfect ratio. There was no good answer for this problem at the time. One simply tried a batch of beer, and if they liked it, it was good.

Gosset’s solution was to develop what is now known as the student’s t-test. You brew a batch you know is good. You do a taste test to rate its quality among a number of people. You get another batch and perform another round of ratings. The student’s t-test is a mathematical formula that - given the subjective ratings of 2 batches of beer – spits out a probability that the second batch is equivalent to the first batch, known as the p-value. That is to say, the p-value is the probability that the resin ratio of the second batch produces beer of the same quality as the resin ratio of the first batch.

That’s fantastic! This kind of prediction significantly dropped the guesswork out of brewing. You could know with 90% certainty you would produce quality beer given some margin of the resin ratio. Gosset wanted to publish his findings, but higher level decisions from Guinness feared other breweries would copy Gosset’s novel creation of small sample statistics. Gosset was allowed to publish, but only under the pseudonym of “student”. That’s why it’s called the student’s t-test.

Gosset didn’t develop these theories alone. Sir Ronald Aylmer Fisher, considered the father of statistics as we know it today, used this t-test for other purposes. Gosset used the t-test to determine if two batches are the same; Fisher used it to determine if two batches are different. This theory has helped answer so many questions in science.

Does a drug truly affect people differently than the placebo?

Does a mouse receiving experimental therapies behave differently than a mouse not receiving the therapies?

Does a new surgical tool produce better results than an old tool?

Does a certain hospital sanitization protocol actually keep a hospital cleaner?

But here’s where things get hairy. In his papers, Fisher asserts that a p-value of 0.05 indicates a statistical difference between 2 samples. This has become a standard is research everywhere: if the p-value of a study reaches under 0.05, the study shows that 2 batches are different. A p-value of 0.05 indicates that there is a 95% chance that two batches are different. But also a 5% chance that they are not. Think about that. 1 in 20 sets of data that produce a p-value of 0.05 are flukes, anomalies, data coming from the same batch that, through happenstance and probability, indicate in the student’s t-test that they are different batches. 1 in 20 t-tests are false. 1 in 20 drugs tested against a placebo are no different than the placebo. 1 in 20 surgical tools perform the same as they always have. 1 in 20 new soaps perform exactly the same as old fashioned hand sanitizer.

This is the curse of the student’s t-test. Gosset held firm that an adequate distinction in his t-test should be up to the discretion of the application, not a higher mathematical power. Less risky and influential research can get away with using lower standards. More risky and more influential research, such as that in the medical industry, should demand a higher standard. 1 in 20 is not a high percentage in the medical field. 1 in 20 people dying or receiving chronic illnesses as a result of botched treatment is unacceptable. Especially not at the ridiculous prices charged for anything medically related.

Science is a great tool. It will take us far. But we have to understand what exactly it is we are doing. Higher education treats statistics as a tool to take for granted, an always truth that produces papers which produces grants which produces money. Don’t let others tell what’s right because, “that’s the way it’s always been”. Always be in control.

I hope this was informative and wish you a great day, no matter your background.

Justanotherme23

 Sources:

https://priceonomics.com/the-guinness-brewer-who-revolutionized-statistics/

http://www.sthda.com/english/wiki/t-test-formula

[justanotherme23]

My Deep Learning Final

It’s 6:00 on a Sunday in quarantine and the Tinder matches have stopped responding. I’m lonely and bored; it’s time to share with the world the single greatest comedic achievement in my life. My Deep Learning final. It’s a long story, but I hope that someone besides me finds it as fucking funny as I do. I’m not the greatest writer, but we’ll give it a go!

I signed up for a class called Deep Learning. To make things simple, it’s an image processing class for crazy smart graduate students (i.e. not me). We write computer programs that identify the locations of objects in a picture. With our programs you could feasibly split porn vids into individual images and count how many unique penises are present in a video. You could even identify how close people’s mouths were to the penises to determine how many blowjobs there are! But that’s just one application

Once upon a semester’s eve, I realized I was failing and the only assignment left was the final project. I needed to get a 90 or above to pass. It was actually a really cool project. The teacher had created a video game comparable to Rocket League but with hockey instead of soccer, and minus the flying and crashing. Games were 2 players against 2 players, whoever scores the most goals in a two minute time limit wins. Students could work in teams, but I do school remotely and so worked by myself. Here’s a short clip of the environment.

The students’ job is to make a team of computer players (one team each) that could drive the puck into the goal such that each player receives the video stream of the game as well as information describing the position of the player being controlled at each timestep. A player makes 1 decision that each step, or frame, of the game is played. We can tell a player to accelerate, break, steer, and do some other things. Then, this action is sent back to the game program to calculate the next step.

TL;DR it’s the same idea as if you were to control the car with a console controller, just represented as numbers and steps.

This is where things get ~hairy~

Players were going to be graded based upon their performances AGAINST OTHER STUDENTS’ PLAYERS. This scared the shit out of me. Even worse, each player’s program could only run for 1 tenth of a second on a CPU.

That.

Is.

Bonkers.

Deep learning programs require a cubic hecton of processing power to do anything meaningful. The whole entire semester we had been making large programs on expensive computers to accomplish tasks similar to the final. These programs require GPU’s, generally used for playing the latest and greatest Call of Duty or mining Bitcoin – they can get quite expensive and sometimes hard to come by.

Asking a CPU to run the same program is like giving a SpaceX rocket to a 5 year old. It’s going to take them a long ass time to figure out which knobs to turn to start the thing up while they neglect safety requirements for literally anything else running around it at the time the program launches. A program that takes days on a GPU takes years on a CPU.

The time limit meant our programs had to be less than a tenth the size of what we had been learning on throughout the semester. This was reportedly a result of the Teacher not being able to find an applicable computer for the final?? Rules are rules, and I knew there would be a huge drop in the performance of the program.

I nevertheless gave this final project all of my free time for a solid week. One morning I sat down with a full box of cereal in front of the computer. When I next stood up to go to bed, the box was empty. Yet this was all I had to show for my effort at the end of the week. The black dot running around the screen is the program’s prediction for the location of the puck.

That sucks! The player couldn’t tell how to steer to hit the puck until it was very very close. It just kind of shuffled back and forth across the arena knowing what general direction to head in. I was depressed.

A few days before the final, the TA announced they would be holding a practice tournament. We could pit our players against each other before the due date to see how to make improvements. Here are the results. I was 3rd to last, doomed to fail the class.

BUT THEN. Something clicked.

My results were horrible, yes; so were almost everyone else’s. Since the vast majority of goals were scored against only 3 players (myself included), the second placed player was MORE LIKELY TO NOT SCORE IN ANY GIVEN MATCH. That’s fantastic news.

Then I noticed something else while playing with position logic. The players and the puck always start from the same positions! Instead of abiding by the ideals of the class to use deep learning – as it obviously wasn’t working for all except one team – I decided to use an … alternative solution.

Let’s say you’ve started living in a new city and need to get to the nearest Starbucks. You look it up on the look-up phone and find the route from your home to the Starbucks. Then you remember the route. You use the route every morning to get coffee before work. You have memorized to take a right; go 2 steps forward; take a right; one step forward; left; then you’ve arrived. The only problem with this memorization scheme is what happens when you start one block off of where you live. Following the exact same steps leads you to the middle of fucking nowhere. So I made an assumption based on the fact that each player starts from the same position. ~The players can memorize their routes from a given starting position to the goal~.

This piece says if the player were to arrive at the exact listed position, perform a recorded set of actions.

A route to a player is simply a list of action that they take to get from point A to point B. The actions you see above are a stored list that describe exactly that. How to get from the starting position to the puck and from the puck to the goal.

There’s one other important assumption. The other players must not interfere. It was clear after seeing the preliminary results that everyone was stuck with their little players piddling around searching for the puck. So provided the goals were scored fast enough, I could assume the enemy players remained about in the spots they started! Here’s the resulting route. This route is valid both right after the game starts and right after a goal is scored, at which point the positions of everything reset.

Now, I had no idea if this would work. If anything at all about my assumption was wrong, if the puck was slightly of place, if the other players don’t stay in the same spot, the plan would fail. It was midnight, an hour before the deadline, and this was the best solution I could come up with. But the next morning THIS HAPPENED

Not only did I get enough points to score some goals but I FAR AND AWAY BEAT EVERY SINGLE OTHER PLAYER IN THE TOURNAMENT. I completely undermined the intention of this final project with a solution that is technically an exploitation of the environment. It’s a fucking ridiculous solution that would not have worked if ANYTHING in the startup of the game had changed. Even rotating a player by the smallest amount would have completely ruined the plan but it WORKED FUCK YEAH!! NONE of the smart ass people in this class with all their effort could beat the most ridiculous gameplan imaginable! People commented on the post asking for how I won and I didn’t say anything out of fear I would be disqualified. I’ve not been able to tell anyone.

To whomever reads this, you are the only one who knows about the total extent of the ridiculousness of the situation. How many hours I spent trying to do the right thing. How I figured out the requirements were impossible. How this graduate level class – filled with geniuses who aren’t me – was unable to compete with this one simple trick to score goals fast. It is so far my greatest accomplishment in life and thinking of it will never not make me laugh. Thank you for listening 😊I hope you enjoyed it too!

[justanotherme23]

[justanotherme23]

Can’t stop listening

[justanotherme23]

The Babel Fish

As described by Douglas Adams😂:

The Babel fish is small, yellow, leechlike, and probably the oddest thing in the Universe. It feeds on brainwave energy received not from its own carrier but from those around it. It absorbs all unconscious mental frequencies from this brainwave energy to nourish itself with. It then excretes into the mind of its carrier a telepathic matrix formed by combining the conscious thought frequencies with nerve signals picked up from the speech centers of the brain which has supplied them. The practical upshot of all this is that if you stick a Babel fish in your ear you can instantly understand anything said to you in any form of language. The speech patterns you actually hear decode the brainwave matrix which has been fed into your mind by your Babel fish. Now it is such a bizarrely improbable coincidence that anything so mind-bogglingly useful could have evolved purely by chance that some thinkers have chosen to see it as a final and clinching proof of the NON-existence of God. The argument goes like this: `I refuse to prove that I exist,' says God, `for proof denies faith, and without faith I am nothing.' `But,' says Man, `The Babel fish is a dead giveaway, isn't it? It could not have evolved by chance. It proves you exist, and so therefore, by your own arguments, you don't. QED.' `Oh dear,' says God, `I hadn't thought of that,' and promptly disappears in a puff of logic. `Oh, that was easy,' says Man, and for an encore goes on to prove that black is white and gets himself killed on the next zebra crossing.

[justanotherme23]

I feel like if I get this tattooed to the bottom of my foot, I should be allowed to look at it during the exam. Good plan team.

[justanotherme23]

[justanotherme23]