spaltcode-blog

[spaltcode-blog]

Let’s load an asset

Last time, we left Mario adventuring throughout the Mushroom Kingdom of Asset Dependencies, jumping off the heads of Koopas and loading up files. We looked at what it happens when a file is getting loaded in a more simple abstraction. But what is happening under the hood of this monolithic method? Seriously, every clean code part of me cringes when I pry open the disgustingly long method called #load_from_unloaded.

So, now that Mario has grown big and continues down the path to the next castle he jumps into another mysterious tube. TIME TO COLLECT SOME COINS! Let’s take a look at metadata.

When loading a file for the first time, `#load_from_unloaded` does a lot of work with a lot of checks. We are going to hold the B button and shoot some fireballs at Conditional Koopas to preemptively make up for lost time.

1. Check that the file can be found

2. If the manifest is using the fancy index aliasing find the path by expanding from the root

3. After splitting the path into two variables, if the load_path is nil then that means Sprockets has searched as far as it can and didn’t find the file it needs and throws an error

4. Assign some variables for type checking later, notice the mysterious `mime_exts` method.

5. If there is a pipeline, append it to the logical_path string for later processing (literally though it will make its way through this same method again)

6. And now let go of the B button for the delicate hidden block

The reason I call the mime_types a ‘hidden block’ (nothing to do with procs) is because there is so much happening around this gargantuan method, it is easy to miss one of the most interesting acrobatics of sprockets.

When sprockets assigns the extname and file_type it is calling a method that is a list of all the extensions and what that mime type is. Originally used for HTTP requests, mime types are a standardized way of declaring file types. Each mime type will have a type (application, text, image) and subtype (ruby, css, gif). When assigning the variables on line 120 `match_path_extname` is doing some surprisingly rudimentary Ruby to return the values. Some of the real work is simply using the massive hash to find the key value pairs from `mime_exts` which currently returns 81 pairs in vanilla Sprockets, check out how some of them are assigned.

But we have another reference to mime_types. We want to check that there is a type and if so, append the file extension to the logical path. So, in the case of a vanilla Rails view like some_file.html.erb the logic would work like this. As an unloaded asset, some_file.html.erb, would return `{:type=>"application/html+ruby"}`. Thus when config[:mime_types][type] is called it will find the mime type and an array of the extensions to use for it. In this method it will always use the first extension even though there are some that could use multiple different extensions to process the file (‘text.erb’ v ‘txt.erb’).

Let us continue into the next error check where Sprockets will check if the type is the same file_type from earlier and that there is even a transformer to convert that file type configuration.

If you are unfamiliar with transformers that’s okay, they are a concept unique to Sprockets in these lines. Here are the docs.

If this check passes we can continue along to where the method ‘should’ end (if not earlier) according to clean code principles, but that’s a topic for another day. Let’s assume we hit the flag at the end of our Mario level (oh yeah, we are still using Mario as an analogy) and continue to our next level.

We are going to assign some processors for the file types we have been using so far, the pipeline argument is another internal concept to Sprockets and if it is nil that simply means the default pipeline being used. If you want to register your own pipeline look into extending sprockets.

Now the big method, the whole reason we made all the above checks and assigned those variables. Sprockets needs to define processors for the unloaded asset. Processors can either be a transformer like mentioned earlier, or they can be a DirectiveProcessor object which is responsible for handling index aliasing and different parts of the require family.

These processors will be joined together into the dependencies for the asset dependencies later. But what’s going on with `build_processors_uri`? This will return a string that could look something like this ‘processors:type=text/css&file_type=text/css’. This string will get appended for the asset cache to resolve later, but that happening next depends on the next check.

If there are any processors it will call the processors and store the result. When these processors are called it will add them in reverse order and then merge the results before continuing (I’m kind of brushing over this here, but these processors will have to be a topic for another day). From here there is another check to ensure that the result is a hash along with a few other checks. If this passes then metadata becomes the result variable with a few more keys/value pairs.

On the flip side of the check to see if there are any processors, the dependencies array has the processor string (something that may look like ‘processors:type=text/css&file_type=text/css’) appended to it and adds the metadata attributes. This records the dependencies so that the file can continue to be loaded into the cache. To see how that happens we need to refer to the last few lines of this behemoth method.

First, let’s finally return to Mario because I don’t know how what happened to that analogy for the last few paragraphs. Second, let’s create a hash labeled as the asset. This asset is kind of like the Baby Bowser in a castle. Mario storms in and has to jump on the Bowser’s head a few times and then catch some mystical flute… thing (In Super Mario Bros. 3 at least). After storing an id, which is a hexidigest of the asset hash (jump 1), and then storing a uri based on the filename, path, processor, and id (jump 2). We finally catch the flute, or maybe scepter, and `#store_asset` (jump 3).

I want to point out a very important aspect to you. Yes, we did all that to load an asset dependency. Just so the big method can get all recursive on us and start from the top again on the next file. Or in other words, the Princess is in another castle…

Reflecting on this deep dive I was a little critical of how the code was implemented, and I stand by that criticism since a lot of what is happening is somewhat unclear. But according to one of the maintainers for the repo it was 70%+ written by one person who tightly coupled everything for a high performing, simple, and powerful solution. So there may be an arguable excuse, and I also don’t see myself attempting to fix it anytime soon, I’ve got too many of my own projects. Besides that same maintainer admits that he has been defeated many times trying to abstract and tease out some of the methods, but Sprockets is very resistant to refactoring. Hope you learned something here. And thanks to Richard Schneeman for helping me understand Sprockets through his own blog post.

[spaltcode-blog]

As we continue our journey through the Asset Pipeline, we will be going ever deeper into the abyss of metadata and SHA hashing. FUN! But before we do I want to say thank you to my readers, I received overwhelmingly positive feedback with my last post. No pressure guys. But half (one of two) of my readers wanted to know more about the magical how of Sprockets. Specifically, how Sprockets is smart enough not to load the same file twice. So, today we will be jumping deep into the code of the gem, so zip up your overalls and get ready to eat some mushrooms, it’s Koopa smashing time.

 To follow along with our journey, click this link to go to the relevant file in the sprockets Github.

 When we first arrive in World 1-1, we are but a small file of a larger manifest. Ready to pounce, conquer, and rescue the princess append a file to the asset pipeline. For brevity, or in Mario terms a flute to jump to another world, when Sprockets finds the manifest and starts loading files from the project it starts a new cache to store file ids (more on what those are later). Sprockets then calls this method here for each asset. find_asset will find where the asset is on disk, return a uri and then load that uri, ripping us innocent plumbers programmers straight into the Mushroom Kingdom.

 When we enter our #load(uri) method the first line we get to hit is `unloaded = UnloadedAsset.new(uri, self)` which will return an object that has gone through its own adventure, discovering params, compressed_type, and a parsed uri. This object will be used to discover first if it has already been required, and if it has a unique key identifier in it’s params attribute. So, there are more than a few checks that happen here. The first if statement is a base case to make sure sprockets doesn’t infinitely include files, if an object has an id it has been loaded and sprockets can use it from the cache. However, there is also an edge case, the object could have an id but not be in the cache. Sprockets handles the edge case by looking into the cache and unless the unloaded asset’s id does actually exist it deletes the id from the cache and starts over. After this iteration, Sprockets again checks the ids and if the returned id doesn’t match the one it deleted it throws an error.

 For the rest of this example we will assume a file has not yet been included in the cache so we will continue along merrily into an important, and complicated method, that receives a block. This is also where we jump back into the trippy life of the Italian Plumber. In the Super Mario games the little hero starts off in a miniature form, but after consuming a mushroom grows into a burly man who easily breaks floating bricks with his fist. Sprockets looks at the asset and says “Alright file let’s see what assets you need”. In Mario when you eat a ‘shroom you gained an asset, you grow. What happens if you eat another one? Nothing. Well you get points but close enough. So, in Sprockets when an asset has been loaded, its dependencies have also been loaded, there is no sense in reloading the asset dependencies because it wouldn’t do anything helpful. Just like Big Mario eating a mushroom to grow big.

BONUS:

If you caught the edge case of loading dependencies you get a 1up mushroom!

 Sprockets was well planned and knew ahead that there are different versions of dependencies. So, let’s say asset A and asset B are similar and share a dependency. Normally that mutual dependency would just be loaded once, but what if the mutual dependency is version 1.1 in asset A and version 1.3 in asset B? Sprockets will load both, because even though the files may be identical except for a typo fix in a comment, meaning they literally perform the same, it is not Sprockets job to differentiate the contents of dependencies. Sprockets’ job is to load files efficiently. So Sprockets has to assume there are differences between the two versions. Which, is why the files are loaded by id from the cache and not any other way.

 But what if our file has not been loaded, so we don’t have it in our load path? Our asset pipeline may break. Looks like Sprockets has some more work to do.

 To be continued...

[spaltcode-blog]

The Assets of Requirement

Ah, the magical Asset Pipeline of Rails. As a rookie programmer I never tried to learn the way it worked, mostly because it seemed too complicated. And in comparison to other fundamentals, the Asset Pipeline was a low priority when it came to writing competent migrations and rspec tests. So if you want to learn about the mysterious, and sometimes unpredictable, Asset Pipeline I invite you to this mini-series.

 WORD OF THE DAY: Requirement

 In Harry Potter, or more accurately Hogwarts (and indirectly Borgin & Burkes in Diagon Alley and Hogs Head Inn), the room of requirement acts as a special place one can only enter if they REALLY need it (and they must walk past it three times thinking of what it is they need). In Rails, the Asset Pipeline Manifest acts much the same way. Except nobody gets burned alive in it.

 In Rails 4 and 5 the Asset Pipeline is powered by the gem `sprockets-rails`. In a very simplified explanation, sprockets combines all the javascript and css into their own one large minified files so the server does not need to make as many requests, boosting the speed of the web app. In this blog post we will call these the warehouse.js and warehouse.css (Rails does not call it this, in fact it is precompiled file that uses fingerprinting for production to see if the contents have changed. But that is essentially what the Room of Requirement is in Harry Potter and the analogy is pretty good if I do say so myself). Sprockets does this minification, called uglifying, by looking at the application manifest, or application.js/application.css, and reading the order in which each file/directory, should be loaded and concatenated together into the large compiled file, like a warehouse.

 NOTE: All of these examples and explanations pertain to css styling as well, I just happen to use js files as the examples.

 In Rails 4 the default application.js file looks like this.

`// This is a manifest file that'll be compiled into application.js, which will include all the files

// listed below.

//

// Any JavaScript/Coffee file within this directory, lib/assets/javascripts, vendor/assets/javascripts,

// or any plugin's vendor/assets/javascripts directory can be referenced here using a relative path.

//

// It's not advisable to add code directly here, but if you do, it'll appear at the bottom of the

// compiled file.

//

// Read Sprockets README (https://github.com/rails/sprockets#sprockets-directives) for details

// about supported directives.

//

//= require jquery

//= require jquery_ujs

//= require turbolinks

//= require_tree .`

 Notice those requires at the bottom? That is called the manifest and even though it is commented out, it actually provides the app with functionality, from the ‘=’ before require.

 Sprockets specifically looks for this file (application.js), reads through the comments, any comment variation for those who feel the need to be special, and when it finds a `=`, runs the helper method on the argument with some Vanishing Cabinet magic (it’s a path relative to the application.js file) loading in the file or directory contents.

 So, for example, sprockets reads and responds:

`//= require jquery`

“AHA! Require jQuery!”

 `//= require jquery_ujs`

`”OH! Require jquery ujs”`

 `//= require turbolinks`

“:thumbsup:”

 //= require_tree .`

“Oh, now to require each file relative to the manifest recursively.”

 Did you notice that? First, those requires at the beginning are referencing the ‘jquery-rails’ and ‘turbolinks’ gems (gems, vendor, assets, and any custom directory are loaded through configs, custom directories are loaded like this config.assets.paths << Rails.root.join(‘my_custom_directory”, “lib”)). And the very different `require_tree` loads all the files within the same directory, and even sub-directories, as the application.js recursively and spits them back out in alphabetical order (spits them back out is concatenating them to the special warehouse.js uglified file we invented for this blog post).

 So, to recap Sprockets looks for the application.js file for javascript and application.css for styling and loads everything line by line, loading trees recursively and ultimately adding them alphabetically, to a file for the browser to only load once.

 But what happens if two people try to enter the room of requirement at the same time with different needs? That is, what if our file directory looks like this:

`app/

 assets/

   javascripts/

     components/

       bar.js

       baz.js

     application.js

     foo.js`

 And our application.js looks like this:

`...

//= require turbolinks

//= require components/baz

//= require_tree .`

 Uh uh, a few problems.

1. Require components/baz refers to the file baz.js in the components sub-directory, the path is syntactically correct (remember relative to application.js location), but there is no .js!

2. We require baz.js on two lines! One when we require it explicitly and the other time when we use the lazyman’s `require_tree .` helper. Won’t baz.js be loaded twice!?

3. Jquery  and jquery_ujs turned into ellipses!! (Okay not a real problem, but I needed a third point)

 First, Sprockets is real smart. So smart that it knows it is requiring other .js files if application.js is a .js file. AND, in case you didn’t know jquery, jquery_ujs, and turbolinks are actually .js files too.

 The second problem has a bit more impressive solution, long story short, no it will not be required twice. Sprockets prevents the same file from being included twice (but not the contents, be careful not to include the same content with different names or paths from being included twice). The reason you may see or do use this explicit require yourself is if you needed the contents of baz.js to be loaded before bar.jas and foo.js.

 Without the explicit require the results of the warehouse.js from `require_tree` would be ordered as:

Contents of bar.js

Contents of baz.js

Contents of foo.js

 With the explicit require:

Contents of baz.js

Contents of bar.js

Contents of foo.js

   The two other requirement helpers are `require_self` and `require_directory`

 Typically, require_self would be needed for the style manifest more often than the javascript manifest, but it is still used in both.

require_self will require any of the code written in the application.js/.css file itself.

 So

`//= require turbolinks

//= require_tree .

 console.log(“Hello from application.js”)`

 Would not ever run in the browser without require_self. To see that pleasant greeting you would need to do some configuration similar to this.

 `//= require turbolinks

//= require self

//= require_tree .`

 The require_directory helper would most likely be used for js files before css files, but can still be used for both. This helper is a non-recursive loader that will load only the files in the specified directory.

  So with this newfound knowledge, go forth and conquer your manifests. Require in the best way you know how, and never ever trust a Malfoy.

[spaltcode-blog]

Some Things Come With Experience

No one can teach you some of the hardest aspects of programming.

I've spent the first month of my new job being behind schedule. When my boss said, "Hey, we need a new onboarding process. It should have [x, y, z]. " My fellow greenhorn coworker and I both imagined this would be just like any of the other exercises we did at coding bootcamp. We made our PERT estimates and were so eager and naive that our CTO silently chuckled.

Our CTO knew how inexperienced we were and we were almost on the timeline he expected. But there was no way to learn what hurdles we would experience along the way  without experiencing them. So here are a few things I have written to myself and others to consider when planning a project.

Other people's code doesn't always make sense. Expect to lose many hours scratching your head at how something works.

Other people's code isn't always the best way to do things.

Your goal is often a moving target. Expect everything, even the reason for the feature, to change.

Even the platform itself is subject to change, we changed a major part of our product and tweaked other aspects of what our users see and do.

Over optimization can really come back to bite you. Yes reusability and single purpose is great, but don't overdue it by spending hours on a dope module that could have been a quickly spiked file to come back and refactor later. Why? Because of points 3, 4, and 6.

Your rough draft pull requests are going to suck. Expect a few tries before the presented code is satisfactory.

How familiar are you with the code base? We were 3 days in to a large codebase when we started. Looking back on what we know now this could have been a much shorter timeline if we just knew what some of the models, gems, deprecated controllers and views were and were doing.

How's your front end skills? Getting that div just right is easier than it used to be with bootstrap and other tricks, but knowing good architecture or nice tricks for things like updating progress bars is a skill.

Do you know how and when to use React, Jquery, Angular, and/or ERB partials? There are strengths and weaknesses of each, so don't use a reliable old truck in an Indy car race.

How often can you ask for a second pair of eyes before people will get annoyed? Will it be paired programming? Are your coworkers able to help you debug for a bit without ruining their timetable? Fortunately I have a smart boss who has probably committed every language doc to memory, what about you? Maybe you don't have coworkers, in which case I hope Stack Overflow smiles upon your requests and searches.

What does the world say about your workflow? Are you going part time or full time on the feature? But less obvious, do you have a good office with accessible coffee? Is your workspace conducive to work? Are there people who ask you debugging questions all the time? How is your machine? I have 4gb of RAM so I lose a lot of minutes on page refreshes or running specs on such a slow computer with such a thick codebase and asset pipeline.

Are you skilled or not? Be real, are you quick and confident or do you need to google syntax and methods to get you through?

Of all these points to consider there are many more I overlooked. Even after my infinite knowledge of working on one feature and being in the professional programming world for the many many days of one month.