#another time then' I Will Quit My Job So Hard That This Department Gets Annihilated .

[readymades2002]

weeping and throwing myself to the floor and bawling into my hands etc etc

[hackersandslackers]

Dealing with Dirty Data

Adventures in Excel

In my last post, we discussed how what separates a true analyst (read: technical) from a project manager wearing the mask of an analyst like some Scott Snyder era Joker (I figure that there's a solid overlap between fans of comic books and fans of the real world application of data. Note that this is a study with an N = 1 so it bares no statistical significance, but I have a funny feeling...call it spidey sense). Full disclosure, this post comes mostly out of my inability to sleep in my hotel room in Chicago following a grueling day of doing the very things I discuss in this blog, and preceding a day where I'll have to literally explain my last post to the suits, but perhaps this is the best mindset to begin discussing the myriad ways in which you may encounter dirty data in the wild, and how a savvy analyst may pivot and match their way around it. However, if my prose isn't as on point as you have grown accustomed...blame it on the 4AM haze. Alas, let's begin by discussing the organizational structure of the majority of corporate entities that leverage data to some degree (note, this isn't all corporations...and what does that say about the state of business?) and how, at each step of abstraction in this process that you are from the data, the data gets dirtier and dirtier. Essentially, there's always going to be a group of about 5-10 fewer-than-necessary legitimately skilled data scientists and/or computer programmers/DBAs who are really solid at building and maintaining a database as well as and some sort of compiling language (nowadays, that's probably python, but not exclusively, nor does this matter). However, depending on your industry (unless of course your industry IS data), it's nearly impossible to recruit people who have these skills to the level necessary AND have some familiarity with why this data is needed, and/or the ability to explain how the internal products that they build can be used by an end user. As such, this team has their own project manager(s) who's sole job is to keep these guys from developing a sentient AI that's sole goal is the annihilation of unfolded laundry...when your industry is healthcare. This team should also have at least one analyst who will take the raw code base and do the first step of translation to a more user friendly form. This generally takes shape as either dashboards in a system like Tableau, or if your company has a group of particularly strong data/business analysts (or particularly weak programmers) an interface written in plain(enough) English on a Business Intelligence platform such as Microsoft BI/SAP Business Objects or whatever other system your company utilizes. As a fun little note, this team ALMOST ALWAYS is referred to by some sort of acronym such as QDAR! (Quality data and reporting!) or KMnR! (Knowledge management and reporting!) or Those Fucking Guys (who have something to do with data) (TFG(whstdwd)). On a less fun little note...neither you, nor seemingly ANYONE ELSE will have contact with this team. In light of this information, how do the reports that they build get chosen or who decides how these databases are built? The world may never know. So let's assume the first type of reporting: the Dataratti (which is how I will refer to the acronym defined team described above moving forward) produces dashboards utilizing a tool such as Tableau or Crystal Reports. You may be thinking to yourself: "hey, isn't my job taking the data and putting it in a form where people who are scared by more than two nested groups of parenthesis, and thus this renders my job unnecessary?" The answer to the question is twofold: Yes, and of course not! As mentioned previously, the decision to create these dashboards, the data contained therein, and how you want them to look is decided upon by a mythical creature who has full access and understanding of the data warehouse, AND has full access to and understanding of the stakeholders (AKA, Those Who Sit Above in Shadow ; that's a reference from a famous run of Thor comics that refers to to a mysterious cabal of gods who perpetuate the cycle of Ragnarok in order to subsist upon the energies created by this strife...which as I write this, is an almost disgustingly on-the-nose metaphor for upper management). Now, if you believe that you may be this mythical creature (as I do), I DARE you to apply for a job with this job description, and once you clinch it with the advice from this blog, rapidly realize that your job will involve either one of these job duties or the other. With that digression, even if somehow a useful dashboard for YOU is created, the limitations inherent in these dashboarding tools make one CRUCIAL issue omnipresent: one can only effectively illustrate up to 16 different variables at a time before the system breaks down (for example, Tableau's documentation specifically warns against this). So even if you have the nicest, most illustrative dashboards on the planet from the Dataratti, there is a nearly 100% chance that the information that you actually need will be scattered across 2-3 different dashboards...rendering the nice looking dashboards essentially useless for your purposes, and as previously stated, you have no contact with the Dataratti, nor do you have access to the underlying data from which these dashboards are created. So pop quiz hot shot, what DO you do? Well, mercifully, all of these dashboard tools allow an end user to download a "data dump" (our parlance for "a buncha numbers with headings"). Using Tableau as an example, one can download either a "crosstab" or a text file of the data represented by the dashboard (in both "summary" and "full data" format). Now, just to get the truly gifted in Tableau off my back, yes, the functionality does exist to build in the ability to download the data in the exact format necessary for your needs through a specific combination of custom web server views and Javascript, but... If the users of the dash are exclusively using this function, why do the dashboard at all? And... This forces the developers in the Dataratti to have decent web design skills on top of really high level Tableau skills, and it requires someone to anticipate exactly how the data will be used by the end user by the Dataratti (which is incredibly hard as it's impossible to speak to this department directly, and as previously stated, the lack of this knowledge on their end is the entire reason why my department exists). A few things to note before downloading data from Tableau: You must highlight at least one element of the dashboard before downloading a crosstab. Depending on what kind of dashboard you're working with, you may need to highlight the entirety of one column in order to capture the entirety of your data (click the first element in any column and then scroll down to the bottom of the report...which may be enormously long, hit shift and click the last element in the report) before downloading either the data or the crosstab. If you are downloading a crosstab, be wary, Tableau web server caps how many rows you can download in this method at a time, this can be avoided by downloading the text version of the data (by clicking data as opposed to crosstab). HOWEVER... If you are going the data route, it defaults to summary view. Look over all the headings, and ensure that this covers everything you need, otherwise click "full data" . Interestingly, this still isn't actually the entirety of your data, and continue to check to make sure all of your headings are covered, otherwise, click the display all columns box, and then download all the rows as a text file. Now, repeat these steps until all of the data that you need in your report is contained across these text files (.csv, AKA the Comma Separated Value file type). With all that lunacy completed, you now have several sheets with some common columns, but all with different information; only some of which you need, so what do you think you do? Simple, you use the tools given to you in the previous posts: you lookup on the common factors across the sheets and return the data that you want until you have all the data you need, in the correct order, on one sheet, and then depending on the ask, you may want to pivot that data out in order to summarize the whole mess of data. THIS IS YOUR FINAL PRODUCT well done. Another protip: if you want to reposition data that you've obtained via a lookup, highlight the whole column, hit control+C to copy the data and then hit control+V pause a second (press NOTHING else) and then press control FOLLOWED by V . This takes the values generated by a formula and replaces them with the values obtained. Functionally, this looks exactly the same, but now you can move the data around without affecting or being affected by other data. As explaining only one possible dirty data scenario took over 1500 words, next time, we'll discuss the other most common form of taking the dirty data from the Dataratti and making it useful to you: using business intelligence portals as opposed to dashboards in order to grab the data that you need. Also, if I don't get roasted on a spit for being half asleep for tomorrow's (today's?) meeting, I'll try and write up a companion post with an example of how this works out in practice. In summary, in this post we've learned: How data is generally siloed and sequestered within the corporate environment, leading to a bevy of unnecessary steps on behalf of the analyst in order to distill a functional report for the powers-that-be Two major methods in which data comes from the data team (henceforth known as the Dataratti) to your team: Dashboards and Business Intelligence interfaces, and... Assuming you get data in the form of dashboards, how to take these dashboards, download the underlying data, recombine and manipulate the data, and package it in a way acceptable for your needs. Congrats, you've just learned the crucial skill of the Slice n' Dice ! Quite sleepily, -Snacks

- Max Mileaf

[greaterthanlows]

Are We Sure He’s Good

Is God really good? I mean... really? As Christians, the answer we are suppose to say is “Yes”, and this answer should come with no hesitation. As a matter of fact, we sing songs about just how Good He Is. 

We sing songs about Him being a good, good Father. One of my personal favorites currently is “Goodness of God” by Bethel.

These songs are great. Poetic. And we claim they are full of truth. But are we biased? Instead, is that just what we want to believe.

This week I read something that shook me. The lead singer of a band I like made the public announcement that he no longer believed in God.

Jon Steingard of Hawk Nelson made the declaration on his Instagram page that he no longer believed in God. I was shocked when I read it. They are a group I have seen in concert a couple times while opening up for other groups and even got to meet them after a concert for a picture.

As I read his post, I have to admit that I noticed the sincerity in his writing. On a side note I was also pleased to see the number of Christians (particularly other artists) who reached out to him with love and support instead of shunning. But I was also troubled by the number of people who reached out with similar stories of no longer believing in God because of similar feelings as his.

Jon spoke of growing up in the church, but never really enjoying going. There were many issues he spoke of, but never once did I get the impression his lack of faith was because of what he perceived God not doing something for him. No, instead it was his trouble reconciling if God was truly good. If we are to take the Old Testament at face value.

Look, I know what it says in the Old Testament. I know the things that we don’t see as good coming from God. There’s a lot of death in there. It’s almost as if the God of the Old Testament is different than the one in the New Testament. How can the two possibly relate? 

I think this is a question the Church doesn’t like to address. 

I think when the Church does address it, we have tried to do it with the misunderstanding that those who hear already accept the sovereignty of God and for those of us who accept that fact, then we don’t question it.

There are great many who don’t accept that statement. There are also many followers who struggle with it, but there is this misconception that we cannot question God. 

Now Freeze!

When I say we cannot question God, that doesn’t mean we get to tell Him how to do His job, although I have tried this approach often... but that is a discussion for another day. What I’m saying is God welcomes our questions and concerns when we go to throne with them with the intentions of better knowing God.

I feel like I had one of those experiences of God revealing things to me this past Sunday afternoon during my studying. I have a place I go when I want to be alone with God. Only a couple of people who are very close to me know about what this place means to me. As I was driving there, I just had this feeling of expectance. I just had this feeling God was going to speak to me. How? I wasn’t sure. I’m still not sure. But He was going to. And He did. 

I began reading Matthew. I had to see proof of an instance of Jesus “breaking” the old way, the way we see God in the Old Testament, when we see the argument of a cruel and vengeful God. Chapter 12 we see the Pharisees upset when they see Jesus and the disciples working in order to eat on the Sabbath. There is much to the response Jesus gives, but one statement in particular he gives:

“I tell you something greater than the temple is here. And if you had known what this means, ‘I desire mercy and not sacrifice,’ you would not have condemned the guiltless.”

-Matthew 12: 6-7

I had a note written which beside it that simply said: “Hosea 6:6-This fulfills that verse.”

“For I desire steadfast love and not sacrifice, the knowledge of God rather than burnt offerings.”

-Hosea 6:6

Where is the mindblower emoji?

There it is. Thank you. Ok, maybe you’re not quite getting that yet. But I did. Suddenly I was hit with this realization that the sacrifices that man made to appease God, they were never what God wanted. Never what He desired. He desired love from His creation (that being us).  

Now look, I understand this and the rest of what I’m about to write doesn’t answer every question. Maybe for you it doesn’t answer any. Maybe it’s just a personal revelation I had for myself. I’m not sure. What I began to understand from this was what God’s desire was from the beginning, something even deeper than just fellowship with us. It was for all of creation to love Him. Not just a select few. He wanted it from all. The sacrifices came as a way to somehow meet God where He was the best we (or they) could. The sacrifices could never be enough. They had to go away so we could simply love God. 

We could love Him!

We wouldn’t have to do things to reach Him, make sacrifices. No. No more. 

God desired our love and desired for us to understand who He is: “the knowledge of God”. I admit, we won’t understand all of His ways. But I just can’t buy the argument from the church we have to trust God blindly on every account because He is God. Yes, we are to trust Him. But I also believe there are things that would greatly impact our faith for the worse, that would lead us astray, I believe in those tings He seeks to have Himself revealed for those who seek.

Key statement being... for those who seek.

I don’t believe God is in the business of having people prove their worth by how much faith they have when they don’t have the answers. Does this please God? Yes. I believe so. But I believe what pleases Him more is when we don’t have the answers, and we take those questions to God and in the process we seek to understand them and understand Him, and we keep our faith in the silence, during the seeking. 

Second mind blowing moment you ask? 

What we refer to as the Love Chapter (1 Corinthians 13) isn’t just a description of how we are to love others. It’s a description of how God desires us to love Him as well.

* “If I speak in tongues of men and of angels.... but don’t love...”- God doesn’t want our big words, our big declarations. Jesus said as much when he talked about how to pray and how to seek God. 

* “If I have prophetic powers... if I have all faith... but not love....”- Jesus warned that many would come to him saying all the great things they did for him and in his name and would tell them to depart because they never knew him.

* “If I give away all I have... my body to be burned... but have not love...”       - Salvation is a free gift, given by grace, not by works so no man may boast. If we had to work, then it would be a return to the sacrifices and they would never be enough.

* “Love is patient and kind... does not envy... does not boast...”- We wait patiently for the Lord. We wait for HIs answers, for Him to lead. We trust. Because of our love for Him, our love for others is better. We do not envy to be like God. We aren’t worthy. We aren’t good enough. That’s a hard truth. We do not boast as if we are worthy of love. We only boast because He does love.

I could keep going and going. I understand this doesn’t answer everything. I’m still in the process myself of seeking. I’m still searching to understand why God called for the total annihilation of some nations for rejecting Him while giving others an opportunity to repent. I’m still seeking. 

What I do know is God gives all mankind an opportunity to repent, as big as that book we call the Bible is, there is still a lot of God’s works and commands not in it. I believe every nation, every soul had opportunities. I believe God sought after them. While the Jewish nation is God’s chosen people, I also know that even in the Old Testament there were many believers that He used and that worshipped him. Not being an Israelite was not a death penalty in the Old Testament.

I also know that it is irresponsible to say the God of the Old Testament has no love. We see love there. So much of it. It’s almost as if the closer we get to the end of the Old Testament and closer to the coming Messiah, the more desperate we see that love from God. It’s as if He’s aware what is about to happen. The time for sacrifices were coming to an end. The time where we could give Him the steadfast love He desired was quickly approaching.