#restarting from scratch when things are even worse and more expensive than last time

[cerbreus]

moodboard for the rest of this week

[scarlettlillies]

Hetalia - The Last Night

Back in 2019, I had written a fic under the same name for Nordipalooza but I wasn’t happy with the ending I had made. Finally after almost two years, it’s finished! I wrote around five or six different endings but none of them seemed to mesh well with the story. While I’m not 100% satisfied with the outcome, I like this ending a lot more than the last one! I'm very happy I can finally cross this fic off my to-do list.

This fic is set around September 1939. The original prompt for this was "Finland, Any character - Final meeting". Both characters are brothers in this.

If you prefer to read this at Ao3, you can click the link here.

Summary: Finland and Estonia meet up for drinks at their usual spot in Tallinn. But Finland can't shake off how cold his neighbour is acting and Estonia avoids any attempt at setting the record straight.

-----

In a dimly-lit bar, they share a drink together. As chatty as everyone is, their voices are low and soft while the mood is somber at best. There are few smiles as a singer in the background reminisces of a long lost love. Finland though is in his own world and pulled out a case from his chest pocket and offered Estonia a cigarette but the man declined.

“C’mon Estonia, live a little. I bet you’ve never smoked a day in your life.”

“For your information, I live a fulfilling life and doing just fine without them,” he laughed. “Besides, smoking too many of them will kill you.”

“Details!” Finland said before he lit up a smoke. After taking a few drags, he let it lay against a glass ashtray and placed the lighter next to it. “It isn’t like we can die anyways.”

That got a chuckle out of Estonia when he placed a finger to his lips, “Shh… don’t say that out loud. No one is supposed to know about that.” With a few friendly jabs back and forth, their conversation quickly died and the two men were left to sit in an awkward silence that had Finland feeling out of place. It wasn’t normal for their conversations to just dissipate like that. He watched Estonia’s mannerisms closely as he took a few more drags of his cigarette. The man was staring mindlessly into his liqueur with an expression that would rival a grieving widow. Finland was beginning to lose his patience. In a move that was out of character for him, he smacked the table with such force that Estonia jumped from his seat and startled the folks at the surrounding tables.

“I didn’t sit on a boat for two and half hours to watch you stare at your drink all night.”

Finland’s tone was harsh and Estonia stared at him like a deer in headlights, “Uh, right, I’m sorry.” he replied as he turned his head looking rather embarrassed. Finland quickly returned to his usual self when he realized what he had done. He rubbed the back of his neck feeling rather guilty. He couldn’t believe he lost his temper like that.

“Estonia, I’m so sorry,” he said apologetically. “It’s just—I’m worried about you right now.”

“I know, I know,” Estonia replied. “And you don’t need to apologize. I should be the one apologizing. I’m the one who invited you and yet here I am being a bad host. Let’s restart this from scratch shall we?”

Estonia lifted up his drink up and with a grin he proposed a toast, “To friendship and brotherhood.”

Finland was impressed just how quick Estonia changed his entire persona. He was dying for a topic change and was delighted to put all of this in the past, for now at least.

“To friendship and brotherhood!”

Estonia seemed to be loosening himself up after having two drinks as they chatted about everything under the sun. Finland went for sports and his adventures of playing catch-up with Sweden. Meanwhile Estonia talked about music and his fascination for the sciences. Even as the patrons slowly emptied the bar and the shows were ending for the night, they continued on as if the world around them didn’t exist, ordering drink after drink. Finally at one in the morning, the owners grew tired of them causing a ruckus. One of the men, who spoke Estonian with a notable accent and dressed in an expensive blue suit, announced they were closing shortly and the two of them needed to leave. They didn’t contest and Estonia paid the bill. They stumbled out of bar, laughing at the top of their lungs, and their arms entwined. The owners had never been so grateful to see someone leave as the man from earlier locked the door behind them.

Estonia walked Finland back to his hotel. He felt guilty that he couldn’t house him but Finland didn’t seem to mind. It wasn’t Estonia’s fault that he lacked a spare bed and there was no way he was going to let Finland sleep on the bedroom floor. The rooms in Tallinn were cheap, cozy, and it suited his needs just fine. Finland’s feet were killing him but luckily his hotel wasn’t too far from the bar they spent their evening at.

The air felt like winter was just around the corner and Finland hated it. It was made worse when the mood between them had also reverted back to how things were at the bar earlier in the evening. Estonia insisted it was because he was tired but Finland was not buying it. He doesn’t press him about it however. He doesn’t want to end their stellar night on such a sour note.

When they had arrived to the old-style wooden hotel doors, Finland was taken aback when Estonia showered him with so much open affection. Estonia wrapped his arms around him and pulled him in for a warm and familiar embrace.

But his hug felt different than usual. He had felt this kind of hug before—on the night he and Sweden said their goodbyes before Russia took him away to Saint Petersburg. The memory of the mighty Sweden failing to keep his composure; his tears stained his shoulders as he whispered “I’m sorry” over and over in his ear, was one that he’ll never forget for eternity. It was only ten years ago that Sweden had opened up to him that he truly felt that he was never going to see Finland ever again.

Now, it was Estonia’s turn. It was one to feel it from Sweden but Estonia was his brother—his best friend. They had spent so many centuries apart even though they lived under the same ruling powers. Sweden would tip-toe around the rules to keep him comfortable. Russia, on the other hand, had given him more freedoms than Estonia ever hoped to have when he began living under his roof—though he later realized this was only because Russia feared he’d rebel in order to return to Sweden. But now that they were free together, Finland couldn’t bare to lose Estonia again. He wouldn’t let that happen again.

Finland noticed the sorrow in Estonia’s aqua-coloured eyes when he had pulled away. Though he promised himself that he wouldn't push the issue, Finland can't let it slide any longer.

“Estonia, please talk to me. You haven’t been yourself all night.”

There’s a long and lengthy pause, and Finland never got the answers he wanted. Instead Estonia avoids the question entirely. He shouldn’t have expected anything less from him. Estonia always tip-toed around questions he didn’t want to answer by changing the topic or continuing on as if nothing happened. The man had such strong walls that few could tumble down.

“Thank you for such a wonderful evening. Promise me we’ll do this again?”

“Of course. But—”

“Excellent. I’ll be looking forward to it,” Estonia grinned. “Please send my regards to Sweden the next time you two meet.”

He stuffed his bare hands in his coat pockets and Estonia turned to walk away into the night. Finland however was quicker and grabbed on to Estonia’s right arm with a tight grip.

“Wait!” Finland cried out desperately. “Please stay the night with me. It’s late out and there’s nothing running at this hour. My bed is big enough for the both of us.”

Estonia turned his head back with a smile. It’s not the kind that Finland wanted—it was somber at best—but it was still a smile.

“I’m flattered but that’s not necessary. Don’t worry, I’ll be alright. Tallinn is pretty safe at night.”

In a move that surprised Estonia (though it probably shouldn’t have), Finland yanked him backwards by pulling him by the arm. They grew closer as Estonia could feel Finland’s head pressed deep into his upper back. Quiet sobs could be heard and it was cracking Estonia’s hard composure. He hated seeing others cry but hurt more when it came from the ones he loved most. He tried to rationalize himself that all of this was because Finland was tired and a little drunk. But even that couldn’t get Estonia to believe his own lies.

“Please talk to me,” Finland softly repeated, in between sobs. “Whatever it is, I won’t judge.”

“I’m sorry but I can’t.”

“Yes you can. Stop lying to me.”

Estonia hung his head low and bit his lip for a brief moment. He opened his mouth, took a breath—and then nothing. One half of him wanted to spill his emotions out. But no matter how much he tried, the other half was too stubborn to admit the truth: his freedom was at jeopardy and he was too ashamed to admit it. The war brewing in the west will not spare them.

The only thing he could muster up was a warning—

“A word of advice: if Russia comes to your door, don’t answer it. He’s the devil in disguise.”

Finland’s eyes opened wide when Estonia used all of his strength to pull himself from his friend’s grip. Without looking back, he ran as fast as he could through the streets of the downtown without a word. Hotel patrons arrived to their windows startled when Finland yelled out his name loud enough that his voice had cracked. He couldn’t get the energy to go after him and stopped after making it a few blocks away from the hotel. He watched helplessly as Estonia disappeared further into darkness.

To say his words hadn’t struck fear into him would be a lie. A warning from Estonia was never to be taken lightly. The man seemed to carry a sixth sense and could feel danger approaching long before it would take place.

It really did feel as if this was their final night together.

And in the end, Finland never slept that night.

He stood up in his bed until dawn smoking each cigarette, one by one, until his case of eight was emptied out. The sun was beginning to rise though its light was stuck behind dark grey clouds—a fitting end to his time in Tallinn. He watched attentively at the dead streets of the Old City through the dirty hotel window while the radio cycled through various soft jazz tunes. It was an easy distraction to keep his mind from replaying the events from last night.

It was nearly six the morning. He had to get himself ready or he’d miss the cruise back to Helsinki. He was still dressed in the clothing he went out to the bar with, minus his jacket which had been tossed on to a chair at the room’s entrance, but he made sure to clean himself up. He hardly recognized himself in the bathroom mirror. His light blond hair looked ruffled like a bird’s nest while his eyes looked dark, heavy, and blood-shot. His white-collared shirt was wrinkled from front to back and his suspenders were barely hanging on at the edge of his shoulders. Finland looked as if he aged by at least five years. He wasn’t sure why he bothered splashing water in his face—it didn’t fix the bags underneath his eyes. But at least he felt a little more refreshed as he dried himself off.

His belongings were packed and he pulled the wrinkles out of the bed he ended up never sleeping in. After calling for a taxi, he carried his heavy bags down the long-winded wooden stair case to the small lobby. Finland was ready to pay his bill but the young clerk had mysteriously disappeared. He lightly tapped the silver bell for service. It was there on the counter where Finland saw a copy of today’s paper and he grabbed it to have a closer look. He had suddenly lost the colour in his face when he saw the solo photographs of Estonia and Russia’s bosses side by side. His eyes read the black bolded headline and Estonia’s words rang in his head like a broken record:

If Russia comes to your door, don’t answer it. He’s the devil in disguise.

If Russia comes to your door, don’t answer it. He’s the devil in disguise.

If Russia comes to your door, don’t answer it. He’s the devil in disguise...

He understood everything now. It won’t be long that he’ll be next in line. He was always Russia’s favourite after all.

[cherryyharryy]

Come on Eileen

We’re in some diner, off the highway you destroyed, in a town no one cares about. A plate of omelette between us, our forks mingle and kiss between bites. Come on Eileen scratches at the windows but you can’t hear it from the outside. 

Your knee bounces and hits the table once in a while. You apologize for your legs being so long. Soon your feet stretch out and your shoes hit mine. You say sorry around your straw where the diet soda you ordered travels up into your mouth. It’s not really diet, but neither of us cared for the waitress.

It’s four A.M. and we lost count of how long we’ve been awake. My clock lost its numbers. We tell time by meals; shower when the sun rises, fuck when it sets. It’s all we need to know. 

Your car sits outside, rusted, and the passenger door won’t lock. There’s nothing worth stealing so it’s not something you’re worried about. A box of white powdered donuts and the Vogue magazine I made you buy me two weeks ago, a scratched gameboy, Benny Goodman cassette tapes, hair clips, and extra toothpaste. I think there’s an orange rolling around in the back, but I never see it when I look over my seat. 

I pop an ice cube in my mouth and you whistle. It’s basically our mating call. Your fork scratches the plate and then you feed me a bite, telling me these are your lips and your teeth and your tongue; I’ll feed you, I’ll take care of you. I’ll take care of this mouth, don’t worry baby, I’ve got you. 

Come on Eileen restarts. The jukebox is broke, but no one can justify fixing it when the neon sign out front only has three letters working. You catch a bit of egg on the corner of my lips, slipping your thumb into my mouth; this is as routine for us as penciling in eyebrows and shoving socks to the back of the drawer so it closes. And yet, I still want to die. 

I remember our apartment, but only in the way in which you try to remember last night’s dream; the harder I try, the more details are erased. It’s been a long summer and it’s only June. 

You arch your shoulder and swipe your mouth over the denim jacket. There’s still a red stain on the sleeve from when you knocked over my watercolors last month. My canvas got the brunt end of your fall, ruining it for good. I was pissed as hell. And then you bought me weed and I had to pretend I wasn’t mad anymore. If I bring it all up again, I can probably get the last of your stash. 

Do you want a kiss? You ask me as we run to the car. I tell you to wait...wait for it. We slam the doors and stare out the window, squinting because it’s dark outside and the diner windows are dirty. The waitress walks up to the table we’d just spent the last two hours in. I swear I can hear her say shit. 

What do you think they do? When someone doesn’t pay?

You shrug, start the car, and kiss me. Absolutely nothing.

Nothing is what I see for so long and for so far. Fields and fields and grass and trees. Cows. Blue sky, eventually. You leave a trail of cigarettes along the highway even though I tell you to stop. They’ll follow us, and make us pay for that meal. 

You tell me no, baby, you’ve got it all wrong. 

I joke about being pregnant at one point but you make me quit. Then you start crying so I have to roll the window down; drown out your sobs with wind. At a stop sign you’re forgiving, reaching into the back seat for my sketchbook. A new one, please, baby.

I can’t draw. But you can’t smoke or drive or pay, so who the fuck cares. I draw her little face again. I cringe because she deserves better, but I figure it’s the best I can do. I’m probably still high, anyway. 

We make stops at places that have no value on this earth and would be better off disappearing. That’s why we go there. We only decide to go home when it starts to rain. The rain bothers you so bad. It’ll never be sexy for us. I’ve considered breaking up with you because of it. But like those places I’ve got nowhere else to go. I wouldn’t know how to disappear even if I tried. 

It’s pouring and I remember my grandma, in her rocking chair from the sixties, telling a child-version of me that God must be cryin’ ‘bout somethin’. You think it’s funny that I believe in God. Like you actually laugh about it. He’s probably crying because we didn’t pay. I’d tell you this if I knew you wouldn’t hate me for it. 

Is this our apartment? This is home? You shake your jacket off when I lock the door. I hate it here. You rush to the kitchen and start the coffee. You rush to the bedroom to tear off your clothes. You rush to the bathroom and jump in the shower. When I’m adding your three scoops of sugar, you shuffle back. You don’t say thank you but it’s basically okay. I know you’re about to sit on the couch, and then I can go in and sit on your lap. That’s always okay. 

But this time you don’t. This time, you stand out on the balcony, in the rain, half naked. I barely fit out there. Between you and the dead plants and your depression, I don’t have much room. I squeeze in though, and just stand there, in the rain, fully clothed, watching you think about whatever the hell it is you think about. 

Wasn’t gonna tell you, but I’ve got a little left. I offer the only thing I have to give. 

You shake your head, sip your damn coffee. No. You don’t. 

I can cry too. I almost say it out loud. I can cry really good. You’d be so impressed, so proud of your darling. 

Darling. Go back inside. You’ll get sick. 

Nuh-uh, I want to say. I kiss your back and leave. 

It smells awfully bad in our apartment. I didn’t catch it until I leave you and the rain and the coffee. Smells like weed and the hummus we get from the gas station down the street. Smells worse than that diner we stole from. Smells like smoke. 

It reminds me of the time I was eleven. Not the kind of eleven where I had friends and sleepovers, but the kind of eleven where you ask your mom why she smokes cigarettes at midnight in the bathroom and all she does is shut the door. 

So this one time, mom was driving down this road, with me in the backseat because I’d been checked out of school early, and the car popped. It popped like a shotgun and then smoke billowed across the hood like we’d asked it to. Mom screamed and smashed her sunglasses against the dashboard. They were expensive too. She pulled over and screamed again. She screamed at her life and how the nicest thing she owned was a pair of broken sunglasses. She screamed at how much money it would cost to fix our fucking car. She screamed at God for all the bad things in her life. She screamed at me because I told her that grandma wouldn’t like it if she knew you were screamin’ at God. 

I can forget that day if I wanted to. I can’t forget that smell. 

I run back out to the balcony, and I tell you how there’s a fire. I tell you how it’s all your fault. I scream at you, because everything’s your fault. I can blame you for everything just like how mom blamed God. 

You can’t drive! Don’t you remember? Remember how she was in the backseat and tiny and crying when you hit that car?

You broke the most expensive thing you owned that day. 

It was something like a month after that we met. You told me all about your daughter and how she’s an angel now. I told you all about how Come on Eileen was playing in the car the day my mom died in a wreck. We were at some crummy diner, our first date, and you skipped out on the bill. 

I made you buy me floss that night, because there was just too much of you in between my teeth. When we got in the car, I wasn’t even scared. I think it impresses you how much I can harm myself. Sometimes you joke that I act like a child. But I’m an adult. I’m a fucking adult. I ate my dinner at six a.m. off the bathroom floor. I’m an adult. 

There’s a fire! I say. You sip your coffee. You keep thinking about whatever the hell it is you think about. I stand there in the rain, feeling like my eleven year old self. You start to hum, Come on Eileen. 

I kiss your back and start to cry. 

[programmingisterrible]

Scaling in the presence of errors—don’t ignore them

Building a reliable, robust service often means building something that can keep working when some parts fail. A website where not every feature is available is often better than a website that’s entirely offline. Doing this in a meaningful way is not obvious.

The usual response is to hire more DBAs, more SREs, and even more folk in Support. Error handling, or making software that can recover from faults, often feels like the option of last resort—if ever considered in the first place.

The usual response to error handling is optimism. Unfortunately, the other choices aren’t exactly clear, and often difficult to choose from too. If you have two services, what do you do when one of them is offline: Try again later? Give up entirely? Or just ignore it and hope the problem goes away?

Surprisingly, all of these can be reasonable approaches. Even ignoring problems can work out for some systems. Sort-of. You don’t get to ignore errors, but sometimes recovering from an error can look very similar to ignoring it.

Imagine an orchard filled with wireless sensors for heat, light, and moisture. It makes little sense to try and resend old temperature readings upon error. It isn’t the sensor’s job to ensure the system works, and there’s very little a sensor can do about it, too. As a result, it’s pretty reasonable for a sensor to send messages with wild abandon—or in other words, fire-and-forget.

The idea behind fire-and-forget is that you don’t need to save old messages when the next message overrides it, or when a missing message will not cause problems. A situation where each message is treated as being the first message sent—forgetting that any attempt was made prior.

Done well, fire-and-forget is like a daily meeting—if someone misses the meeting, they can turn up the next day. Done badly, fire-and-forget is akin to replacing email with shouting across the office, hoping that someone else will take notes.

It isn’t that there’s no error handling in a fire-and-forget client, it’s that the best method of recovery is to just keep going. Unfortunately, people often misinterpret fire-and-forget to mean “avoid any error handling and hoping for the best”.

You don’t get to ignore errors.

When you ignore errors, you only put off discovering them—it’s not until another problem is caused that anyone even realises something has gone wrong. When you ignore errors, you waste time that could be spent recovering from them.

This is why, despite the occasional counter example, the best thing to do when encountering an error is to give up. Stop before you make anything worse and let something else handle it.

Giving up is a surprisingly reasonable approach to error handling, assuming something else will try to recover, restart, or resume the program. That’s why almost ever network service gets run in a loop—restarting immediately upon crashing, hoping the fault was transient. It often is.

There’s little point in trying to repeatedly connect to a database when the user is already mashing refresh in the browser. A unix pipeline could handle every possible bad outcome, but more often than not, running the program again makes everything work.

Although giving up is a good way to handle errors, restarting from scratch isn’t always the best way to recover from them.

Some pipelines work on large volumes or data, or do arduous amounts of numerical processing, and no-one is ever happy about repeating days or weeks or work. In theory, you could add error handling code, reduce the risk that the program will crash, and avoid an expensive restart, but in practice it’s often easier to restructure code to carry on where it left off.

In other words, give up, but save your progress to make restarting less time consuming.

For a pipeline, this usually entails a awful lot of temporary files—to save the output of each subcommand, and the result of splitting the input up into smaller batches. You can even retry things automatically, but for a lot of pipelines, manual recovery is still relatively inexpensive.

For other long running processes, this usually means something like checkpoints, or sagas. Or in other words, transforming a long running process into a short running one that’s run constantly, writing out the progress it makes to some file or database somewhere.

Over time, every long running process will get broken up into smaller parts, as restarting from scratch becomes prohibitively expensive. A long running process is just that more likely to run into an impossible error—full disks, no free memory, cosmic rays—and be forced to give up.

Sometimes the only way to handle an error is to give up.

As a result, the best way to handle errors is to structure your program to make recovery easier. Recovery is what makes all the difference between “fire-and-forget” and “ignoring-every-error” despite sharing the same optimism.

You can do things that look like ignoring errors, or even letting something else handle it, as long as there’s a plan to recover from them. Even if it’s restarting from scratch, even if it’s waking someone up at night, as long as there’s some plan, then you aren’t ignoring the problem. Assuming the plan works, that is.

You don’t get to ignore errors. They’re inevitably someone’s problem. If someone tells you they can ignore errors, they’re telling you someone else is on-call for their software.

That, or they’re using a message broker.

A message broker, if you’re not sure, is a networked service that offers a variety of queues that other computers on the network can interact with. Usually some clients enqueue messages, and others poll for the next unread message, but they can be used in a variety of other configurations too.

Like with a unix pipe, message brokers are used to get software off the ground. Similarly to using temporary files, the broker allows for different parts of the pipeline to consume and produce inputs at different rates, but don’t easily allow replaying or restarting when errors occur.

Like a unix pipe, message brokers are used in a very optimistic fashion. Firing messages into the queue and moving on to the next task at hand.

Somewhat like a unix pipeline, but with some notable differences. A unix pipeline blocks when full, pausing the producer until the consumer can catch up. A unix pipeline will exit if any of the subcommands exit, and return an error if the last subcommand failed.

A message broker does not block the producer until the consumer can catch up. In theory, this means transient errors or network issues between components don’t bring the entire system down. In practice, the more queues you have in a pipeline, the longer it takes to find out if there’s a problem.

Sometimes that works out. When there’s no growth, brokers act like a buffer between parts of a system, handling variance in load. They work well at slowing bursty clients down, and can provide a central point for auditing or access control.

When there is growth, queues explode regularly until some form of rate limiting appears. When more load arrives, queues are partitioned, and then repartitioned. Scaling a broker inevitably results in moving to something where the queue is bounded, or even ephemeral.

The problem with optimism is that when things do go wrong, not only do you have no idea how to fix it, you don’t even know what went wrong. To some extent, a message broker hides errors—programs can come and go as they please, and there’s no way to tell if the other part is still reading your messages—but it can only hide errors for so long.

In other words, fire-and-regret.

Although an unbounded queue is a tempting abstraction, it rarely lives up to the mythos of freeing you from having to handle errors. Unlike a unix pipeline, a message broker will always fill up your disks before giving up, and changing things to make recovery easy isn’t as straight forward as adding more temporary files.

Brokers can only recover from one error—a temporary network outage—so other mechanisms get brought in to compensate. Timeouts, retries, and sometimes even a second “priority” queue, because head-of-line blocking is genuinely terrible to deal with. Even then, if a worker crashes, messages can still get dropped.

Queues rarely help with recovery. They frequently impede it.

Imagine a build pipeline, or background job service where requests are dumped into some queue with wild abandon. When something breaks, or isn’t running like it is supposed to, you have no idea where to start recovery.

With a background queue, you can’t tell what jobs are currently being run right now. You can’t tell if something’s being retried, or failed, but maybe you’ve got log files you can search through. With logs, you can see what the system was doing a few minutes ago, but you still have no idea what it might be doing right now.

Even if you know the size of a queue, you’ll have to check the dashboard a few minutes later—to see if the line wiggled—before you know for sure if things are probably working. Hopefully.

Making a build pipeline with queues is relatively easy, but building one that the user can cancel, or watch, involves a lot more work. As soon as you want to cancel a task, or inspect a task, you need to keep things somewhere other than a queue.

Knowing what a program is up to means tracking the in-between parts, and even for something as simple as running a background task, it can involve many states—Created, Enqueued, Processing, Complete, Failed, not just Enqueued—and a broker only handles that last part.

Not very well. As soon as one queue feeds into another, an item of work can be in several different queues at once. If an item is missing from the queue, you know it’s either being dropped or processed, if an item is in the queue, you don’t know if it’s being processed, but you do know it will be. A queue doesn’t just hide errors, it hides state too.

Recovery means knowing what state the program was in before things went wrong, and when you fire-and-forget into a queue, you give up on knowing what happens to it. Handling errors, recovering from errors, means building software that can knows what state it is currently operating in. It also means structuring things to make recovery possible.

That, or you give up on on automated recovery of almost any kind. In some ways, I’m not arguing against fire-and-forget, or against optimism—but against optimism that prevents recovery. Not against queues, but how queues inevitably get used.

Unfortunately, recovery is relatively easy to imagine but not necessarily straight forward to implement.

This is why some people opt to use a replicated log, instead of a message broker.

If you’ve never used a replicated log, imagine an append only database table without a primary key, or a text file with backups, and you’re close. Or imagine a message broker, but instead of enqueue and dequeue, you can append to the log or read from the log.

Like a queue, a replicated log can be used in a fire-and-forget fashion with not so great consequences. Just like before, chaos will ensue as concepts like rate-limiting, head-of-line blocking, and the end-to-end-principle are slowly contended with—If you use a replicated log like a queue, it will fail like a queue.

Unlike a queue, a replicated log can aid recovery.

Every consumer sees the same log entries, in the same order, so it’s possible to recover by replaying the log, or by catching up on old entries. In some ways it’s more like using temporary files instead of a pipeline to join things together, and the strategies for recovery overlap with temporary files, too—like partitioning the log so that restarts aren’t as expensive.

Like temporary files, a replicated log can aid in recovery, but only to a certain point. A consumer will see the same messages, in the same order, but if a entry gets dropped before reaching the log, or if entries arrive in the wrong order, some, or potentially all hell can break loose.

You can’t just fire-and-forget into a log, not over a network. Although a replicated log is ordered, it will preserve the ordering it gets, whatever that happens to be.

This isn’t always a problem. Some logs are used to capture analytic data, or fed into aggregators, so the impact of a few missing or out of order entries is relatively low—a few missing entries might as well be called high-volume random sampling and declared a non-issue.

For other logs, missing entries could cause untold misery. Recovering from missing entries might involve rebuilding the entire log from scratch. If you’re using a replicated log for replication, you probably care quite a lot about the order of log entries.

Like before, you can’t ignore errors—you only make things expensive to recover from.

Handling errors like out of order or missing log entries means being able to work out when they have occurred.

This is more difficult than you might imagine.

Take two services, a primary and a secondary, both with databases, and imagine using a replicated log to copy changes from one to another.

It doesn’t seem to difficult at first. Every time the primary service makes a change to the database, it writes to to log. The secondary reads from the log, and updates its database. If the primary service is a single process, it’s pretty easy to ensure that every message is sent in the right order. When there’s more than one writer, things can get rather involved.

Now, you could switch things around—write to the log first, then apply the changes to the database, or use the database’s log directly—and avoid the problem altogether, but these aren’t always an option. Sometimes you’re forced to handle the problem of ordering the entries yourself.

In other words, you’ll need to order the messages before writing them to the log.

You could let something else provide the order, but you’d be mistaken if you think a timestamp would help. Clocks move forwards and backwards and this can cause all sorts of headaches.

One of the most frustrating problems with timestamps is ‘doomstones’: when a service deletes a key but has a wonky clock far out in the future, and issues an event with a similar timestamp. All operations get silently dropped until the deletion event is cleared. The other problem with timestamps is that if you have two entries, one after the other, you can’t tell if there are any entries that came between them.

Things like “Hybrid Logical Clocks”, or even atomic clocks can help to narrow down clock drift, but only so much. You can only narrow down the window of uncertainty, there’s still some clock skew. Again, clocks will go forwards and backwards—timestamps are terrible for ordering things precisely.

In practice you need explicit version numbers, 1,2,3... etc, or a unique identifier for each version of each entry, and a link back to the record being updated, to order messages.

With a version number, messages can be reordered, missing messages can be detected, and both can be recovered from, although managing and assigning those version numbers can be quite difficult in practice. Timestamps are still useful, if only for putting things in a human perspective, but without a version number, it’s impossible to know what precise order things happened in—and that no steps are missing, either.

You don’t get to ignore errors, but sometimes the error handling code isn’t that obvious.

Using version numbers or even timestamps both fall under building a plan for recovery. Building something that can continue to operate in the presence of failure. Unfortunately, building something that works when other parts fail is one of the more challenging parts of software engineering.

It doesn’t help that doing the same thing in the same order is so difficult that people use terms like causality and determinism to make the point sink in.

You don’t get to ignore errors, but no one said it was going to be easy.

Although using things like replicated logs, message brokers, or even using unix pipes can allow you to build prototypes, clear demonstrations of how your software works—they do not free you from the burden of handling errors.

You can’t avoid error handling code, not at scale.

The secret to error handling at scale isn’t giving up, ignoring the problem, or even it trying again—it is structuring a program for recovery, making errors stand out, allowing other parts of the program to make decisions.

Techniques like fail-fast, crash-only-software, process supervision, but also things like clever use of version numbers, and occasionally the odd bit of statelessness or idempotence. What these all have in common is that they’re all methods of recovery.

Recovery is the secret to handling errors. Especially at scale.

Giving up early so other things have a chance, continuing on so other things can catch up, restarting from a clean state to try again, saving progress so that things do not have to be repeated.

That, or put it off for a while. Buy a lot of disks, hire a few SREs, and add another graph to the dashboard.

The problem with scale is that you can’t approach it with optimism. As the system grows, it needs redundancy, or to be able to function in the presence of partial errors or intermittent faults. Humans can only fill in so many gaps.

Staff turnover is the worst form of technical debt.

Writing robust software means building systems that can exist in a state of partial failure (like incomplete output), and writing resilient software means building systems that are always in a state of recovery (like restarting)—neither come from engineering the happy path of your software.

When you ignore errors, you transform them into mysteries to solve. Something or someone else will have to handle them, and then have to recover from them—usually by hand, and almost always at great expense.

The problem with avoiding error handling in code is that you’re only avoiding automating it.

In other words, the trick to scaling in the presence of errors is building software around the notion of recovery. Automated recovery.

That, or burnout. Lots of burnout. You don’t get to ignore errors.

[slapegg]

Quick Critique: Battle Chasers: Nightwar

Before I even get to the meat of this: DO NOT BUY THIS GAME, IT IS VERY BROKEN. Again, even if you loved the comic and love turn-based RPGS: DO NOT GIVE THIS COMPANY MONEY, THEY RELEASED A BROKEN GAME.

Joe Mad is one of my favorite artists and I read the Battle Chasers comic pretty much just to see him draw pretty things. I only just read it a year or two ago, so I thankfully avoided the whole incident of him bailing on the series and leaving it unfinished after a main plot twist. The actual story (and most of the writing) is kind of just Dungeons and Dragons meets generic anime stuff, but it had enough moments and unique ideas here and there to make the comic series worth it, even this many years after its initial release. So Battle Chasers the comic is pretty good. Battle Chasers the video game, however, is a steaming pile of garbage.

This game is so unfinished and slapped together that I'm just going to list out a stream of the constant issues it has: It crashes. A LOT. It's not just one thing going wrong, it crashes all over the place and multiple times at different spots (inventory screen, leaving a dungeon, changing screens, and so on) I actually had a crash during the ending credits It locks up every time I try to enter the Arena forcing me to force close the game I finally managed to enter the Arena (apparently some of the difficulties work and others don't), it's a series of battles with a 20 minute time limit, I'm doing the hardest difficulty to get the final prizes, get through multiple rounds where I'm one turn away from the entire party dying or I lucked out and a character dodged an instant kill move, I'm at the last boss with 2 minutes left on the clock, deliver the killing blow with 23 seconds left on the clock, I get the Playstation level trophy for completing the Arena, the game showers me with prizes, I leave the Arena, I open the menu, the game crashes and upon relaunching sets me back at the beginning of the Arena having made no progress at all and with none of the prizes. I think I just instantly skipped over anger, sunk back in my chair, and whispered "mother fuckers" to myself for a few minutes After a battle, I lost the ability to interact with any objects. Seeing as how the dungeon required me to flip a switch to advance, I had to quit the game. This happened multiple times Frequent hitches and freezing for a second on the map, exploring, and in battle Menus aren't responsive for a few seconds after opening them I had a story scene fail to load and the game just displayed a screen full of that "missing image" pink I had a different story scene fail to load but I still got the subtitles. This one was a major story sequence so it wasn't repeatable the next time I played the dungeon so I would have liked to have seen that The voice acting is Sega Saturn levels of bad. They're horrible choices for the characters but then poorly acted on top of that. And they apply random odd filters over the readings. There's a scene in the open air in a town where your characters talk to each other but Gully sounds like her lines were recorded in a particularly echo-y bathroom. The only character I cared for was The Collector, a sinister but gleeful little monster that is likely eating the remains of dead bosses that you bring it. The Collector deserves to be in a better game than this one The text size is WAY too small The walking speed is slow. If you doubled the walking speed, it would still be too slow Battles are slow both in animation and action speed and how many hits it takes to kill a grunt enemy The core combat system is tedious and relies far too much on crits and applying status effects. Most end game fights devolve into who can apply the most debuffs to the other team and then spamming special moves that gain extra traits if the enemy has specific debuffs The music is so laid back that it (rightfully) just seems uninterested in being a part of this game, even during battles. The soundtrack is so forgettable and uninteresting that I usually turned the game audio almost off and listened to the BBC while I played. The Shipping Report pretty much matches the pace and excitement of this game Totally unnecessary crafting mechanics By the time you get enough crafting materials to build a weapon, it's worse than what you get from dungeon crawling You can't sort your crafting materials alphabetically, so when you're looking up how many of a quest item you need, have fun sorting through that mess Totally unnecessary fishing mini-game Every time you enter battle, the UI flashes a move description. I think it's loading the last thing you used in the previous battle Occasional multiple second pauses at the start of a battle before the UI will display or you can interact with it in any way Clunky menu UI Loot-based drops that do nothing to make the game more interesting Loot that isn't even interesting or exciting because most of the equipment is very similar and the vast majority of what you find is just crafting materials you won't use Major side-quests and items that are gated by random loot drops. You have to hope the characters show up on the map and then hope they drop the item you need (usually multiple times) or else start a dungeon from scratch and do it all over again Semi-randomized dungeons where the actual rooms barely change but their order does, so combined with the need to grind, the dungeons get really boring and just have you looking for the exits rather than rewarding you for exploring. Later dungeons even repeat pieces of earlier dungeons Items in shops are stupidly expensive for how little they change your stats and for the piddly amount enemies actually pay out or what items sell for. While spending a night at the inn cost me 40 coins, selling a purple rank weapon only got me 17 coins. It's actually faster to play through the whole first dungeon and get the health and mana refill before the boss than it is to grind out the money to stay a night at the inn for a large chunk of the game Just about every item you find in the wild will raise one stat but then lower multiple other ones so you kind of just have to pick one stat for a character to use and min/max the hell out of it Items in your inventory will mark themselves as new even though you've seen them before Items in the world will still sparkle as if they're unchecked even though you already have them If you have to close and restart a dungeon (say because the game crashed or locked up), it will acknowledge that you've been through the rooms but respawn the enemies past a seemingly random point. I had one dungeon where the objective was to kill two mini-bosses, I did so, saved the game in case it crashed at the boss, and upon reloading it, the mini-bosses respawned even though I had the objective that said they were dead checked off Every time the game crashes, it resets your super meter. So you can go through a dungeon, build up your meter, save it for the boss fight, the game crashes, you reload at the boss, and now you have no meter and you're at a serious disadvantage Perks and equipment will unequip themselves (this may be related to all the crashes) I met an enemy without warning that was vastly stronger than anything I'd seen in the game before, I could only do 80 damage to it per turn, and every turn it could heal itself for 84 health. And for some reason I wasn't allowed to run away from the fight so I had to sit there for 10 minutes turn by turn hitting the enemy, watching it heal, and letting it whittle down my party's health. Dying then made me lose a chunk of my money Apparently those are special enemies that offer a special reward if beaten, but it wasn't until I was at the final dungeon that I ever saw them again. I don't know if the game was broken and would no longer spawn them or what Another dungeon had a bunch of enemies that I tore through with no problem, and then I got to the boss and it killed each member of my party in 1 hit. Dying, again, made me lose a chunk of my money. So the lesson is to never sell anything until you can fully afford the thing you're trying to buy because you never know when the game is going to throw balancing out the window and punish YOU for it I don't think you can manually save without quitting the game. But quitting the game dumps everything it loaded into memory or something because when you load the game back up, it can take 40 seconds to load into your first battle. Given all the crashes, you have to save and quit often, so get used to watching the first 15 seconds or so of the opening cutscene because you can't skip to the title screen until that plays out The trophies aren't properly proofread and sometimes won't award when you earn them and instead pop the next time you load the game If a character dies from a status effect at the start of their turn, the UI is not graceful I would love to hear some kind of justification on the game's balancing because the way it's set up is that you beat a dungeon, unlock the next story dungeon, but you're not actually strong enough to progress the story yet, so you have to go back and grind the dungeon a you've already finished to level up a bit. You pretty much have to beat each level on each difficulty before you can move ahead, so by the time you're ready to move the story along, you're really, really sick of the previous dungeon Characters not in your party don't gain any experience from battle, so the game actively discourages you from trying new characters. I was level 10 when I unlocked Knolan, but he was only level 9 and the more I use my normal team, the further Knolan falls behind, so I have no reason to ever add him to my party. Changing team members just means you have to redo all the grinding you've done to level them up and hope you get some loot drops for them. Maybe it's just how I play the game, but there's really only one viable team. You HAVE to have Calibretto on your team because he's the only decent healer. Garrison is the only one that can do any decent damage. Gully is slow and focuses on defense, while Monika has high evasion, good damage, and can bog enemies down with stat debuffs so Monika is way more useful Doing end game clean up, I used my weaker teammates because my main team stopped earning experience from the early dungeons, I met those special pirate enemies with this team, was happy to have a chance to fight them, but the pirates scale to your strongest team not the team you're actually using so my level 9, 12, and 17 characters got destroooyed by the level 30 enemies The team couldn't even do New Game + properly. Starting NG+ causes you to lose all your items, so all that time you spent on random drops to get the ultimate weapons and armor was totally wasted. It's completely unnecessary to do this because all the weapons are level gated. So if you reset the character levels, you stop them from having access to the top gear from the start, but once they level up, they get their hard earned weapons back. That would have been the competent way to handle NG+ here New Game + starts you off with the whole team, but it doesn't properly handle that you've unlocked characters before their normal unlocking event, so the shops won't sell you their perk bonus or costume items. Even if you come back at the end of the game when you'd normally have those characters, the game still treats them like they haven't been unlocked yet

Even with its many, MANY glaring flaws, they have the audacity to not actually finish the game's story. You slog through all of this, beat the final boss, and the ending is barely more than "hey, buy some DLC or a sequel".

So, yeah, don't spend money on this game. The perk system is kind of neat though. You get points when you level up and you can cash them in on an attack path or a defense path. As you buy perks on each path, every 20 points you spend unlocks a bonus perk that can offer some substantial stat boosts. You can respec for free, so you can mess around with different choices as you slooowly grind out levels or boss fight currency to get more points.

Battle Chasers is bad but not in the way that most low budget games are bad. The art's great so the game looks fantastic and draws you in. It's the quality design sensibilities and usability in the game that are godawful, the gameplay is as dry as it comes, and it's an unfinished mess. Everything about this screams that the team got a slew of crowdfunding money but then nobody on the team actually knew how to make a good video game so this got rushed out the door without proper testing or fixes to hit a deadline. Battle Chasers makes me question whether Playstation cert matters. If this game, with its constant crashes and 100% reproducible lock-ups, can be released for sale, then clearly nobody is checking the actual game or doing anything remotely resembling quality control.

[terryblount]

Wam’s Top 5 Games of 2018

2018 was quite the year, wasn’t it? Despite the constant soothsaying that “videogames are getting worse”, it seems like every year there’s great games coming out. Of course, that could just be the privilege of not having to play BIG AAA GAMES. But hey, to each their own.

With that in mind, here’s a look at my top games for 2018. If I had to make an observation, 2018 was a pretty good year when it comes to the field of genre improvements. Many games of big genres received tweaks or changes that although may rustle some purist jimmies, ended up for the better. These include things like God of War not blindly chasing the Souls formula and removing its stamina bar, as well as Spider-Man daring to be more than an Arkham brawler.

Seeing as neither of those games made my top five however, let’s get to the list (in no particular order):

5. Blazblue Cross Tag Battle

Fighting games are a weird bunch. I inherently suck at them, I don’t often have friends to play them with and I don’t follow the scene closely enough to be considered a fan.

Blazblue Cross Tag Battle is a 2v2 crossover battle combining Arc System Work’s Blazblue, Under Night In-Birth, Persona 4 Arena as well as debuting RWBY characters. It’s a gorgeous sprite fighter and arcadey as hell. Considering this year also gave us Dragon Ball and Street Fighter, why this fighting game?

It’s accessible.

While not nearly Smash-levels of accessible, Blazblue makes it’s controls certainly still retain the feeling of a classic arcade fighter while also not requiring overly complex inputs. The sprite art is also gorgeous, which is something that’s becoming harder to find as more games move towards 3D graphics.

My only complaint is that early on I tried to escape from crossing fate, but the game informed me that such a feat was simply not possible.

4. Death’s Gambit

It’s no secret I like Dark Souls. You know who else likes Dark Souls? The people who made this game. Death’s Gambit is a metroidvania-souls-like game that borrows heavily from the Dark Souls art style. Thematically, however, that’s where the similarities end. I talk about it more in my review of the game, but basically, this game scratches the itch left when I finished Dark Souls 3, and while not perfect, it’s a pretty good RPG on its own.

You play as Soren, a reincarnated soldier charged by Death to figure out why no one seems capable of dying anymore- and then immediately put an end to that nonsense. You fight plenty of cool bosses, some really annoying ones but all in all, it’s a pretty good standard-setter on fun souls-likes.

3. BattleTech

“You dig giant robots. I dig giant robots. We dig giant robots. Chicks dig giant robots”. -Megas 1:01

BattleTech is a tactics game where you have to manage your own mecha-based mercenary squad as you have adventures across the galaxy. Many times, your mercenaries will die, your mechs will be destroyed or in the shop for repairs, and you will stress about not being able to afford your monthly expenses. There’s a plot about a royal coup or whatever, but let’s be real: this is the meat of the game.

While JRPGs are often lauded for their uniqueness compared to “mainstream” RPGs, I feel not enough praise is given to the CRPG. BattleTech is absolutely dripping in flavors of the genre from all the out-of-combat dialogue choices, resource management and grid based combat.

BattleTech is a nerdy game, let no one tell you otherwise, and that’s where it shines. You get to do things like targeting the back of a mech because it has less armor there, disabling ammo caches on the enemy mech so they can’t reload and even dropping your mech in a lake to help it cool off its overheated guns. Nowadays, too many games try to go the other way and appeal to more mainstream audiences by making things less nerdy. Not BattleTech. And that’s how it earns its spot.

2. Monster Prom

Competitive. Dating. Sim.

I talk about it more in this review but basically, Monster Prom is one of the best executed ideas in an indie game in a while. Play with up to 4 of your fellow dating sim enthusiasts and watch as they try to murder each other IRL over the literal devil while you embrace your inner Guillermo Del Toro and try to romance the mermaid.

It’s got plenty of routes, charming side characters and is a perfectly serviceable plot for single player too. On top of it all, there’s an orgy ending. Instant classic.

1. Monster Hunter: World

Again, my complete thoughts on the game were noted in review form here. Since its release last January, Monster Hunter World has added a plethora of new monsters, including two basically-raids.

Plenty of genre improvements like a quickbar and auto-crafting have made sure this game keeps you focused on the hunting and not superficial busywork.

The game’s ability to do drop-in co-op makes afternoon sessions capable of being as simple as helping newbies with difficult quests (because it doesn’t matter what game you’re playing, it is a Sunbro’s job to engage in jolly co-operation and help other people find glory).

The ability to do that proves Monster Hunter World’s thesis statement: make the gameplay fun enough, and people will keep playing even when there’s no loot to be gained. Despite the fact the game has plenty to grind for (RNG gems, materials for armor sets) that doesn’t mean I don’t enjoy occasionally popping in to other monsters. The mix of the game’s varied weapons playing well and the bosses being designed to be fun makes this a genuinely good game.

Honorable Mentions:

Dead Cells- A truly fun dungeon crawler. Didn’t make the list more out of preference than quality, not a big fan of permadeath and restarting without my equipment.

Super Smash Bros Ultimate- Figured I didn’t qualify for adding to this list because I feel like I hadn’t played the whole game since I’ve only played the VS mode with friends at parties.

Deltarune: another victim of good-game-just-not-for-me, Deltarune is a surprise free demo (?) from Undertale creator Toby Fox. While not explicitly Undertale 2, it boasts party-based combat, a sick soundtrack and all the Undertale flavors that people love Undertale for.

So what next? 2019 is already looking to be an interesting year with Devil May Cry 5 coming back from the dead, Daemon Ex Machina carrying the banner of mech games and Love, Esquire also being on the horizon. Happy 2019 everybody!

Wam’s Top 5 Games of 2018 published first on https://touchgen.tumblr.com/