#I’ve seen devs on other apps forced to fix major issue on the live environment | Explore Tumblr posts and blogs

celeborn-of-doriath · 19 days ago

Text

// I think most people would be surprised how many very critical websites and applications are being run by one very tired and underpaid engineer. I’m not saying it’s a good thing, but it’s not rare.

I once worked on an app that was heavily used by hundreds of millions of people daily and that thing was throwing over 30,000 alerts every night. The entire team for that app was me and one dude in Belarus. We slowly ground that alert number down to 0. It really depends what your infrastructure looks like, but as long as you’re not actively developing new features, you can keep software in maintenance mode without a massive staff.

Obligatory, hopefully tumblr doesn’t die, but imo this doesn’t seem like a big deal to me from a software perspective.

#.ooc #they clearly have proper blue/green deployment #CI/CD seems extremely healthy #bugs are different from infrastructure #if the infrastructure were bad then the site would fall over every 5 minutes #and each deploy would tank the site for like 3 days #I’ve never seen a bad deploy on tumblr #I’m sure they’ve had them bc everyone does #but I’ve never SEEN it #which means they have proper rollback mechanisms in place #which is honestly more than most companies can say #I’ve seen devs on other apps forced to fix major issue on the live environment #because they literally didn’t have the infrastructure to roll back #for example it’s obvious to me that Twitter had absolutely dogshit infrastructure even before Elmo wrecked it and fired the staff

11 notes · View notes