#case in point tick's post was blocked XD | Explore Tumblr Posts and Blogs

blarrghe · 4 years

Note

“Wrapping arms around them when they make breakfast” Dorian x Anders, because I imagine Dorian has NEVER had a lover make him breakfast before (and Anders probably as a cat-shaped waffle iron)

Ok, as much as I love “his boyfriend makes him breakfast and it breaks Dorian” I also like, JUST did that over in my pavellan fic. It was very sweet and all, but consider: neither of these men are functional adults so who the hell is making breakfast? Still, got Anders his waffles. Anyway this directly sequels the last one again, because I’m using prompts to generate this story now I guess, and I’m really invested in this slow burn friends-to-lovers angsty mess now, so this got super long. I’m gonna start posting this as a series on AO3 I think -- also taking title suggestions XD. Thanks for dragging me into this hell :’) Here’s Breakfast:

He told himself that he was just coming along to keep an eye on him. A designated driver of sorts, just one without a car, or driver’s license, for that matter. He showed Dorian to the bar across the street and ordered himself a glass of water while Dorian asked for “the worst swill you have", with a rather large tip slapped on the bartop. He was handed something astringent smelling in a foggy glass, downed it in one quick backwards toss of his head — arching his neck, snapping back again with a shudder — and then he asked to have the bottle.

Dorian took two more shots before he spoke. “Did you know that there was an author, horror novelist, whose mother disapproved so wholly of her marriage that after she died, she and her husband took their revenge by having sex right on her grave?”

So. This was going to be an interesting evening. “I did know that, actually.” Anders said.

“I’m rather a fan of hers, of her work, I mean.” he took another shot, “and of her misbehaviours. Only, do you think it would be too gouache, seeing as it’s already been done?”

Anders coughed. “Because if it hadn’t been, it wouldn’t be?”

Dorian shrugged, and took a fourth shot. Maker, he’d finish the bottle within half an hour, at this rate.

“I’m a fan of hers too,” Anders attempted to steer the conversation into something somewhat more...appropriate, “of her work.” He was also a fan of the story, but maybe not at this particular moment.

“Oh?”

Anders took a sip of his water, and signalled to the bartender to put a water glass in front of Dorian, too. “I tend to enjoy stories about misunderstood monsters,” he shrugged.

“Me too.” Dorian ignored the water glass in favour of shot number five. “Of course, she was married to a like-minded soul, I’d have to find myself a willing participant.”

“Strange thing to put into your dating app profile,” Anders agreed. Dark humour came easy — though he wasn’t entirely sure it was a good idea.

“Mm. Man seeking man to fuck on father’s grave, must be willing to break cemetary locks and city bylaws. Risk of haunting, serious inquiries only.”

Anders tried to stifle his laugh. Man seeking man, though. No. Nope. Very terrible idea.

“I don’t suppose you’d be game?”

Anders coughed again, his cheeks flaring up, and shook his head. “I — uh — I think that must be against...one of my oaths.” he stuttered, still flushing.

Dorian took yet another shot, which made six. What in the world was he made of? "Yes I suppose it must be. Or should be, at any rate." His cheeks were a bit flushed too, even in the dim light, but just from the alcohol; evidently the man had no concept of shame, because next he said, "well, it was worth a shot."

Speaking of shots. "Water," Anders instructed, moving the water glass closer to Dorian, "you should drink some water."

"Yes doctor." Dorian obliged, taking the glass to his mouth but raking his eyes up and down Anders as he drank down the entire thing. Anders just kept on blushing.

"I take it you and your father didn't get along?" It probably wasn't the right question to ask the recently bereaved, but he'd nearly failed that psych 101 course he'd taken in first year, and it was a step away from morbid propositions. Void, where was Merrill when he needed her?

"You met him, didn't you?" Dorian raised an eyebrow, and with quickly failing coordination, poured himself one more shot, while spilling enough to fill another over the bartop. Anders grabbed a napkin, while Dorian threw his shot back without seeming to notice. "My father hated me." He said, once he'd swallowed.

Tear soaked apologies and an alcohol soaked "celebration" of his death. Anders felt something in the pit of his stomach plummet that was quite removed from the growing pangs of hunger his measly lunch — a granola bar five hours ago — had left him with.

"I'm sure he didn't —" Dorian stopped him with an ice cold look, intimidating even as he swayed in his seat. Anders frowned, there had been something in that psych course about not sharing your own traumatic experiences with a patient, even if they were relatable. Muddies the waters of who's caretaking who, or gives them ideas, or makes you look crazy too, so they lose confidence, but — "mine did, too." He gave Dorian's arm a tentative pat, and waved the bartender down for a refill of water. Dorian drank it without prompting this time, but his eyes watched Anders again, waiting for more. "Or he must've, got rid of me quick enough."

"Ah," Dorian leaned back, a little too far, Anders tensed to catch him in case he started to fall, "then I'm an ass. Sorry."

"No, you're —" Dorian swayed back forward with a bit of a jolt, like he'd forgotten how to stop and needed to grip the bartop to keep level. He reached for the bottle again, and Anders shot a hand out to grab it first. Their hands met, Dorian's falling on top of his over the bottle, and then in an instant Dorian's flew away again. "You're drunk." Anders said.

"Yes," Dorian agreed, "marvelous." He went back to the water, then cast Anders' hand, still on the bottle, a hopeful look. "Though not to the point where I won't remember any of this miserable day, yet."

Anders raised an eyebrow, and kept his hand on the bottle.

"Not that I'm saying I wish to forget you," Dorian's eyes were pleading with him, glossy as they were, "you've been rather kind, really, it's just…" when Anders still didn't release the bottle, he groaned. Then he straightened out his face again, a mask of sensibility that was barely holding: "I'm afraid you aren't seeing me at my best, doctor Anders."

"Just Anders." Maker, but the sadness behind it all was killing him. You're heart's too soft, Anders, he scolded himself.

"Anders, then. Quite the name."

"More a point of origin." Anders explained with a shrug.

"Yes, the hair rather gives you away. And the complexion." He reached out and slipped two of his long fingers through a strand of Anders' strawberry hair, which was falling in a straggled mess about his temples. Anders flinched, pulling his head back, and Dorian frowned apologetically. "Pretty. You're very pretty." He said. Anders shook his head and rolled his eyes — the man was drunk — but blushed again.

"It's what the circle gave me," Anders explained the name with another shrug. He wasnt entirely sure why he was volunteering so much personal information to this perfect stranger. Perhaps he felt it was owed, after witnessing the death of the man's father, and all he'd overheard. Or maybe it was those eyes...

"Oh." Another apologetic frown, "and you ran away to Tevinter? Well, you wouldn't be the first." Anders nodded. "Where from?"

Anders chuckled dryly, "Kirkwall, most recently."

"Oof." Dorian grunted a drunken sound of disgust, and Anders chuckled again, "how in the world do you manage not to drink?"

Anders’ laugh grew stronger, he shook his head and took another sip of his water, while Dorian redirected his attention once more to the bottle still protected by his hand, as though just now remembering his plight. "One more, I promise I'll be good." He begged.

"Speaking as a doctor, I think you've had enough."

"I thought you were off duty."

"You're going to make yourself sick."

"Then it's lucky I'm with a doctor."

Anders sighed, and poured him one more slightly scant shot. Dorian frowned at the way the alcohol didn't reach the rim of the glass, but threw it back with a grateful sigh.

“Can I call you a cab, Dorian?” Anders offered, watching worriedly as Dorian gave his head a dramatic shake and swayed a little more back and forth. The bar was emptying out, and last call was coming upon them. He cast a glance at the old watch ticking away on his wrist, mentally calculating how long it would be until he could be at home, in his bed. Not that he minded keeping the miserable man company, quite the opposite, despite everything. He had a pull to him Anders couldn’t quite explain; the eyes again, probably. But the bus came once an hour at this time of night, and didn’t stop at the closer stop, just the well-lit main hub that lay several blocks from his apartment — another fifteen minutes of walking after he got off, so a good hour or more to get home, altogether, if he left now.

“Is it that time already?” Dorian sounded disappointed, spinning the empty shot glass around on the bar, then with a sudden spark of concern in his eyes he turned his face to Anders, “I’ve kept you too long, haven’t I? How dreadfully selfish of me, I —” he was sputtering a rather pitiful apology, and Anders’ stomach fell again at the sight of it.

“It’s alright,” he said gently, muscle memory finding the soft smile he used for giving bad news to patients, “your father died today, you don’t have to apologize to me.”

“Yes, father died…” Dorian got a far-off look in those cold eyes of his, and then directed them back at his empty glass, “and you — you had to, I mean, here I am wasting your time when you must be — selfish —” all at once, his face crumpled, and the guilty muttering gave way to tears. Shit.

Anders patted his back once, carefully, and Dorian seemed to utterly collapse under his touch, sobbing into the sticky countertop. Anders took a deep breath, and dragged him up again. He tossed a tip of his own onto the bar as the bartender shot them an aggravated look, and hauled Dorian away, draping his arms over his shoulders. Dorian slumped into him, heavy, hunched over, still crying, as Anders pushed through the door of the bar and into the balmy night air, awash with the putrid stench of dumpsters in the alley and the sick coughed up by the bar’s less restrained patrons. It all made him a little homesick. Dorian, hanging halfway off of him, lurched forward like he was about to add his own mess to the stink in the alley, but then he righted himself again, and propped himself up using Anders’ shoulder. Anders took the opportunity to pull out his phone.

“Where am I sending you?” he asked helpfully. Dorian made another face that seemed to threaten that he was about to be sick.

“I’m not going back there,” he muttered, less to Anders than to the ground. He wiped at his eyes and sniffed. “Just help me find my car?”

“You can’t drive.”

“I’ll sleep in it — I left it in the lot.”

“No.”

Dorian pushed himself off of Anders, propelling himself away from his shoulder, and staggered forward a step. Then he seemed to change his mind, or realise he was in no state to walk on his own, and reached an arm out to fall back against the wall of the alley.

“No?” He asked, incredulous as Anders took his arm and draped it back over himself, walking them out of the alley and the stink.

“I’m not letting you sleep in your car,” Anders shook his head as he dragged the man forward. He was heavier than he looked. Strong, too, if the grip on his shoulder was any indication. “Besides, I can’t risk leaving you in a vehicle, if you did something stupid that would be on me.”

Dorian snorted, “do you think I’m stupid?”

“I don’t know you well enough to judge.” Anders answered honestly, which seemed to amuse Dorian.

“I’m not stupid.” he said, “very, very smart, actually.” he insisted. Anders nodded appreciatively.

“Alright then, so you see why I can’t just leave you in the hospital parking lot, in your condition.”

“Mm. Kind of you, but I can think of worse places.” So could Anders, but he shuddered to think what could happen to Dorian if he left him alone like this, drunk and stumbling and wearing the most expensive looking suit he’d ever seen; he’d already flashed his overstuffed wallet far too openly when ordering his drinks inside. “Is there a hotel? I could buy a hotel.” Dorian slurred.

Anders was fairly certain he’d forgotten a word in his suggestion, but given the suit and the wallet, maybe not. Before Anders could answer, he lurched forward and away from him again, back towards the alley, and into a spasming sort of crouch, retching.

Anders took an instinctive step back as Dorian gagged and sputtered out a vomit of mostly liquid and bile onto the broken stone of the alleyway, then remembered his physician’s training, and rushed forward to steady him. Between coughs, Dorian swore, and when he finished (miraculously, his suit and shoes were still unharmed), he began to cry again. Anders sighed, and once more feeling a little bit homesick, he breathed out an all too familiar refrain: “well, shit.” he said.

“Not —” Dorian was stuttering apologetically at him now, “not my best.” He wiped at his tears, swore again, then got up from his crouch and began to stumble forward once more, heading the wrong way down the alley. Anders took him by the shoulders and led him out again.

“Hotel?” The word smushed out of him with so much drunken misery that Anders felt almost like crying for him, and he sighed again, pulling out his phone.

“I’m taking you home,” he dialed the number and gave the taxi company their location, then propped Dorian up against the wall of the bar that faced the street, rather than the alley, keeping an eye on his paling face and shaky breathing.

“What, your home?"

Anders nodded, “if you choke on your vomit and die in your hotel room, I’ll feel responsible,” he explained as Dorian looked up at him with a perplexed, and dare he say it, even eager look.

“Very kind of you, doctor Anders.” he said, but before Anders could correct him on the honorific again, he stooped and threw up, so doctor Anders it was.

——

Dorian all but fell asleep in the taxi, head drooping down into his chest, swaying this way and that as the car rounded the corners, but thankfully he kept from throwing up any more. The luck didn’t hold once they were inside Anders’ apartment though, and soon Anders had him steadied in a kneel over his toilet bowl, getting out the rest of it. Dorian flung most of his clothes off before throwing up this time, wrestling himself out of the suit jacket and tight shirt beneath it, while Anders tried not to be impressed. He had a really remarkable physique, but he was also lurching and coughing miserably into Anders’ toilet, so it was definitely not something to admire. Then he got him onto the couch, set a large bowl on the floor by his head, and coaxed him into one more glass of water before letting him lie down. Dorian offered him another tearful apology, and then tearful thanks, and then he passed out. Anders sat back in a chair across from him for a while, watching as his breathing slowed to a steady rise and fall, ensuring that his head was turned to the side, mouth facing the bowl, in case he was to vomit any more in his sleep, and then he finally, finally, stumbled his own way to bed.

He woke to the sound of his cupboards banging shut and the kettle screeching to a whistle.

Anders stumbled out into his kitchen to find Dorian standing there with a distraught look on his face, pouring water into two large mugs. He was dressed again, and looking remarkably perfect, actually. Hair all in place and posture all upright once more. The bowl was gone from the floor, too, and nothing smelled off — just a little like tea.

"How are you feeling?" He asked, suddenly aware of his own shabby pajamas.

Dorian turned, still looking distraught. "You don't have any food." He complained, "I fed your cat —" Anders looked down to the corner of the kitchen where Ser Pounce's food bowl was, and found Ser Pounce there happily nibbling from a bowl filled to slightly too full, "I hope that's alright. I woke up with him on my chest and he wouldn't stop pawing at that cabinet so I figured…"

Anders smiled softly, and not in a practiced way, he'd entirely forgotten to check the food bowl when they came in the night before, occupied as he'd been.

"And then I saw you had a coffee pot, so I was going to make coffee, as a thank you — well, actually, I was going to have some delivered, but I don't rightly know where I am —" Dorian ran a hand through his hair, and he was talking quite speedily, cheeks going just slightly pink "but you don't have coffee. Or anything."

Now Anders blushed, embarrassed for the nakedness of his cupboards.

"Anyway, thank you. Tea?"

Anders nodded, and took the few remaining steps to the counter to grab one of the mugs of still steeping tea; he liked to keep the bag in. He moved from the counter to the couch, cupping the mug with both hands, and sat down.

"117 Orseck Ave.," he said, "that's where you are. How are you feeling… how much of last night do you remember?"

"I remember making a fool of myself, if that's what you're asking. And you being uncommonly kind." He paused, "it is Anders, right?" Anders nodded, "is there anything else I should remember, Anders?"

Anders shook his head, "that about sums it up."

Dorian chuckled. When he wasn't drunk or crying, it was a nice sound. He leaned against Anders' counter — stunning, how was he stunning after a night like the one he'd just had? "Well, you've certainly wasted enough of your time looking after me, and I can get out of your hair now, but —"

"— I wouldn't call it a waste of time," Anders interrupted, because something in him always seemed to speak up whenever Dorian went about making statements like that. It kind of had been a waste of his time, Anders tried to protest against that something, he'd lost a great deal of sleep to it, anyway. But somehow the look that his interruption gained him from Dorian was impossible to remain grumpy with.

"Have you been to Marc's?" Dorian asked suddenly, brightening with a hopeful smile, "since I know where we are now, and its nearby, and you have no food," he went on, "and personally, I'm starving —"

"I imagine you would be," Anders said, though at the mention of hunger his own stomach took the opportunity to awaken too, noisily. Dorian raised an eyebrow at the sound.

"Might I buy you breakfast? I feel I owe you that much."

Anders hadn't been to Marc's. He'd been by it many times, a busy little brunch place, always smelling of bacon and pancakes and with a line out the door. It was a bad idea to say yes to this, he thought, a bad idea to say yes to anything involving absurdly handsome men who just lost their fathers, who were obviously walking disasters waiting to happen (you always had a thing for disasters waiting to happen) — shush. His stomach grumbled again.

"I haven't been," Anders answered, "there's always a line — and I am on call, I might not have time to —"

"Oh, we can skip all that." Dorian brushed the protest aside, "so? Don't try to tell me you aren't hungry."

Anders kicked at a bit of cat hair fluff adorning the edge of his couch, "alright, sure."

Dorian was certainly good at getting him to say yes to things he should know better than to say yes to. If he kept going on like this, the next thing he knew he'd be having sex on his father's grave.

----

They arrived at the restaurant, just a short walk from Anders' building, and yet in a considerably nicer part of town — the new money was creeping in towards his end of things, but where he lived at least was still very much no money — and Dorian walked straight up to the front of the line. Anders hung back, watching skeptically as Dorian performed a series of intricate maneuvers: some charm, a smile, a handshake Anders recognized from Varric — the kind with a bill snuck inside — and then he turned, waving Anders over.

"We can wait ten minutes for a table, or have our food prepared now and take it outside. Your choice." He smiled. Maker, such a good smile; straight teeth and a brilliantly white gleam. "But you're on call, right? And to be honest with you, the fresh air is making me feel considerably less queasy. Park across the street?" Anders nodded and shrugged at the same time, a gesture that seemed to satisfy Dorian into continuing to take charge of the situation. "Alright then, to go. And fast, if you can. We're both very busy and important." He winked at the young hostess as he was handed two paper menus, and Anders could have sworn she blushed brighter than the checkerboard red on the apron she wore. "What do you fancy?" Dorian asked him, handing over one of the papers.

It was diner food, but not really. Poached eggs with house-smoked bacon over an heirloom tomato coulis, waffles with Orlesian creme sauce and glazed berries, rare wheat pancakes with apple cinnamon compote and vanilla syrup — just a few options, all of them coming with a detailed list of decadent flavours. In addition to those few confounding main courses was a fresh juice list filled with exotic fruits Anders had never even heard of, and approximately twenty different kinds of coffee.

"Uh, waffles?" He said, squinting at the menu, "waffles and coffee?"

Dorian beamed some more, and took back his menu to point out the waffle dish, as well as several other things, confidently ordering far more food than could possibly be necessary as well as coffee and one of the strange fruit juices while insisting that Anders simply had to try it. The patient employee nodded and hurried away, and not ten minutes later came back with two plastic bags stuffed near splitting with cardboard containers, and a tray of drinks. Dorian thanked her with another winning smile and secretly-funded handshake, and then they were off.

The park across the street had benches, so they sat on one — finding one in the shade of a great, leafy tree, as even the morning sun was warm. Then, Dorian began a conversation, and the whole thing was far less awkward than Anders had expected. Dorian asked about his work, so Anders described some of it, though he avoided anything too close to topics of death and dying, and Dorian held his gaze while he talked and asked compelling questions. He seemed to be, as claimed, very smart, and the food was practically otherworldly. Then Anders asked Dorian about his work in turn, and Dorian sighed.

"Well, you're new here, aren't you? How much do you know about Tevinter politics? The intricacies of it all can take a lifetime to wrap one's head around. That's by design; keeps things all tied up with the upper classes who have it in their blood to be intollerable bureaucrats." His air was flippant, but altogether disapproving, which Anders appreciated.

"I've been here a while now, actually. A couple of years, anyway, I understand it a bit. Political science was always my…'' downfall? "Second passion." He washed down a heaping forkful of creme covered waffles made of pure fairy dust and clouds with whatever exciting fruit drink Dorian had handed him — it tasted like bright green, with a hint of citrus. "I feel people should be informed — active. Healthcare is as political as it is practical." And mage freedom, that was political too, but they didn't have to get into that. Mages were already free in Tevinter. Other kinds of people, however — something bitter bit at the back of his mind. But it was too sunny, and the food too good, for that sort of conversation.

Dorian nodded approvingly, his eyes lighting up. "Alright then, I'm an Altus. I argue things in circles in the house a lot, these days I've been losing all sorts of friends arguing this Sopperati electorate reformation bill," Anders' eyes widened, impressed. He'd been following the progress of it, a huge step for increased class equality, if it passed. So maybe it was just sunny enough for such a conversation. "but of course it can only go so far without approval from the Magisterium," Dorian went on, a slight growl of frustration colouring his tone, which was appealing in a different way, "and for that we need to convince those with seats in the — in the —'' he stopped, and some of the light fell from his eyes. "I just remembered that my father is dead." He said. Shit. Not a sunny conversation, after all. "His seat passes to me, you see, because nepotism still runs stronger than good sense and he's written my name into all these continuations of his legacy and…" he sighed, and stabbed hard at a piece of brilliantly poached egg, which honestly didn't deserve it, "sorry. It's going to be a very hectic and difficult few weeks, with all the ceremony and paperwork and the whole ordeal of burying him…" he scooped up some of his bleeding egg yolk with a wedge of toast, and went silent in favour of eating, while Anders took an uncomfortable sip of juice that seemed to have lost some of its vividness. "You've been here for years, you said?" Dorian changed the subject, refocusing on Anders. Anders nodded, still awkwardly sucking up juice through the straw of his cup. "I would have sworn you were an escapee fresh from the harbour."

"Why?" Anders bristled a little.

"Your apartment. You have no food or furniture," Anders bristled a little more, "and you've never been to Marc's", Anders frowned, furrowing his brow at the impossibly good, impossibly expensive waffles, "and you're too nice." Dorian finished. Anders looked up in surprise, catching Dorian's eye. They were still a bit lost for light, but soft on him.

"I'm just very busy," Anders shrugged. And very poor, but, well, Dorian probably thought anyone with fewer than a thousand acres of family land was poor, given his status. He didn't need to know the extent of it.

"Hm," Dorian's eyes were still on him, soft and thoughtful, "what else haven't you done?" Anders shrugged, and Dorian began listing things. Tourist attractions and famed galleries, but also other, lesser-known offerings of the city that Anders had never even heard of.

"Ferry through the archipelegos?"

"No."

"The volcanic sand beaches?"

"No."

"Dinner at the top of Tidarion Tower?"

"No."

And on like that, until he finally said yes to something — taking in a show at the infamous burlesque playhouse in the city's red light district, which elicited an eyebrow raise.

"Priorities, I see." Dorian chuckled, "at least you have good taste." He reached an arm up over Anders' side of the bench, as he finished with his food and slid the box away, very smooth. "I'd have offered to take you. Maybe one of the others sometime, then, if you've a mind." He suggested. Anders could feel his cheeks beginning to run hot again. Still a bad idea, he reminded himself. Apparently sensing his unease, Dorian removed his arm from its perch near Anders' shoulders. "May I say something painfully honest?" he asked.

Anders swallowed, but he managed a smirk as he replied. "I think we're well past that," he said.

Dorian shook his head with a dry chuckle, "yes, well. I'm all out of sorts, as you may have noticed."

Anders chuckled too, but with him, not at.

"And normally, if I'm to get drunk and go home with a stranger, it all goes a certain way," then he actually winked, which on him was somehow charming and not over the top at all. Anders swallowed again, "and, not that I'm opposed, but, well, as I said: you've been uncommonly kind. I could — I've been losing friends left and right lately, it seems, with this bill, and…"

"I'm a fan of the bill," Anders said, "in fact I'm not sure it goes far enough."

The interruption seemed to lend Dorian some more confidence, as though he needed it, "so, pretty as you may be, I could use a, uh —"

Anders blushed again, but finished for him, "a friend?" He could use one too, if he was being honest. Near everything seemed to be making him homesick, lately.

Dorian nodded. "If that's not too forward." He said.

"You fed my cat," Anders replied, "as far as I'm concerned, we're already friends."

At that, Dorian smiled. He asked Anders his cat's name, and chuckled at the answer, and then they exchanged phone numbers and Anders stuck a little cat next to his own name as he entered it into Dorian's contact screen, which had him laughing even more. Anders offered to put the puking emoji next to Dorian's in return, but he insisted on a snake, because he “had a reputation to uphold”. Then Anders’ pager went off, and he groaned inwardly, wishing he could spend the day in the sun for once.

“Duty calls?”

Anders grimaced, and stood up. “Thanks for breakfast,” he said, meaning it. Dorian stood too.

“You should take the rest — actually, this may be awkard, but I think we’re going the same way.” His car. Of course.

“You’re going to have a small fortune to pay in parking tickets,” Anders realised, frowning.

“Oh that’s fine. I have one of those — big, actually.” he winked again, “very big.” Sweet Maker, he just never stopped.

Dorian insisted on a cab, and then he insisted on paying for it, and then he insisted on Anders taking the rest of their uneaten brunch items to store in the breakroom for his lunch, and then finally he was ready to let him go, with a promise to be in touch. He extended his hand for Anders to shake. Anders took it, holding fast with a sure grip, and then, drawn in yet again by those cool, sad eyes, he pulled Dorian’s arm towards him, and wrapped him up in a tight hug.

Dorian stumbled back afterwards, cheeks flush, eyes glinting with surprise. “What was that for?”

“Just seemed like you needed it,” Anders said.

Dorian was still blushing, and his smile warmed Anders’ own cheeks. “Suppose I did,” he agreed.

“Take care, Dorian.”

“As you say, doctor.”

#dorianders #dorian x anders #my fic #my writing #dorian #anders #modern au #alcohol cw #vomit cw #breakfast #this is too fun help #disaster boys

26 notes · View notes

contextswitching · 8 years

Text

How is the await field in iostat calculated?

PART 1: 15 seconds of await

One of our customers was running some third party monitoring software, which was reporting very occasional spikes of many seconds (6-15 seconds)worth of await on their local disk. Looking at the datadog code, we can see that the python is really just running iostat and capturing the output:

if Platform.is_linux(): stdout, _, _ = get_subprocess_output(['iostat', '-d', '1', '2', '-x', '-k'], self.logger)

From the man page, the options iostat is running with are:

-x Display extended statistics. This option works with post 2.5 kernels since it needs /proc/diskstats file or a mounted sysfs to get the statistics. This option may also work with older kernels (e.g. 2.4) only if extended statistics are available in /proc/partitions (the kernel needs to be patched for that).

-k Display statistics in kilobytes per second instead of blocks per second. Data displayed are valid only with kernels 2.4 and later.

-d Display the device utilization report.

The '1' and '2' will mean that iostat runs every one second, and will return two results before exiting. iostat with the -x flag will read from /proc/diskstats by default (it says it can also use sysfs, but a look at the code shows that /proc/diskstats is preferred when available). /proc/diskstats is a file containing a set of incrementing counters with various disk statistics. The first set of output from iostat will be statistics since the system was booted (as per the man page), and the second result will be statistics collected during the interval since the previous report. This will give us output like this:

# iostat -d -k -x 1 2 Linux 2.6.32-642.6.2.el6.x86_64 (linux-test2) 01/26/2017 _x86_64_ (1 CPU) Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util xvdb 0.01 0.02 0.00 0.00 0.05 0.09 69.08 0.00 31.58 3.05 53.63 0.74 0.00 xvda 0.00 1.40 0.05 1.22 1.10 10.49 18.18 0.00 0.66 0.62 0.67 0.13 0.02 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util xvdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 xvda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

The first thing to do to try and investigate the issue was to try and replicate the results using iostat ourselves. We ran the following in a while loop:

# while true; do date >> /var/log/iostat.log ; iostat -d 1 2 -x -k >> /var/log/iostat.log; sleep 1; done

After running that for a while, we calculated the min, max and average values that iostat logged for await:

# cat /var/log/iostat.log | grep "dm-2" | awk '{print $10}' | sort -V | awk 'NR == 1 { max=$1; min=$1; sum=0 } { if ($1>max) max=$1; if ($1&ltmin) min=$1; sum+=$1;} END {printf "Min: %d\tMax: %d\tAverage: %f\n", min, max, sum/NR}'

The results looked something like this:

Min: 0 Max: 15506 Average: 1.862469

Suspiciously the max was indeed showing 15506ms. Given that iostat is running every second, 15000ms (15 seconds!) of await is logically impossible. But in case the customer pressed further, the next question is what actually is await? A quick google around the internet doesn't show us exactly how it's calculated, so that means we need to go to the source.

PART 2: Obtaining the source code

The iostat program is part of a package included in the 'sysstat' package. The customer was running sysstat-9.0.4-27.el6.x86_64. The first thing to do was to download the source RPM for this version; it's not sufficient to browse the latest code repository online since software can change significantly from version to version. We found the SRPM on the red hat website and downloaded it on a test machine, and installed it.

# wget http://ftp.redhat.com/pub/redhat/linux/enterprise/6Server/en/os/SRPMS/sysstat-9.0.4-27.el6.src.rpm # rpm -Uvh sysstat-9.0.4-27.el6.src.rpm # cd rpmbuild

Looking in the SOURCES directory, there is some code and a bunch of .patch files. If we wanted to look at the source for the program as it was installed on disk, we would need to apply the patch files to the original source. We can do this with rpmbuild. From the relevant section of the rpmbuild man page:

-bp Executes the "%prep" stage from the spec file. Normally this involves unpacking the sources and applying any patches.

Before we can apply the patch files, we need to install some dependencies. We got an error for missing dependencies when trying to build the package initially which told us which dependencies were missing - gettext and if.h (which a yum whatprovides shows is provided by the gettext and kernel-devel package respectively):

# yum install kernel-devel gettext

Finally we can run rpmbuild.

# rpmbuild -bp SPECS/sysstat.spec

Once this finished, we can find the patched source for the version of sysstat we care about in the ~/rpmbuild/BUILD/sysstat-9.0.4 directory. From the naming of the files, the one we want is iostat.c.

PART 3: Reading the source

Since this is quite a short piece of code (~2000 lines), the first step is to go through and read the comments, functions and variable naming to get a broad picture for what we're looking at. Initially the first thing that sticks out is the write_ext_stat function - the comment above it reads this:

/* *************************************************************************** * Display extended stats, read from /proc/{diskstats,partitions} or /sys. * * IN: * @curr Index in array for current sample statistics. * @itv Interval of time. * @fctr Conversion factor. * @shi Structures describing the devices and partitions. * @ioi Current sample statistics. * @ioj Previous sample statistics. *************************************************************************** */ void write_ext_stat(int curr, unsigned long long itv, int fctr, struct io_hdr_stats *shi, struct io_stats *ioi, struct io_stats *ioj)

That looks like what we want. Looking at the code, we can see pretty clearly where the iostats output is printed when displaying extended stats:

/* DEV rrq/s wrq/s r/s w/s rsec wsec rqsz qusz await svctm %util */ printf("%-13s %8.2f %8.2f %7.2f %7.2f %8.2f %8.2f %8.2f %8.2f %7.2f %6.2f %6.2f\n", devname, S_VALUE(ioj->rd_merges, ioi->rd_merges, itv), S_VALUE(ioj->wr_merges, ioi->wr_merges, itv), S_VALUE(ioj->rd_ios, ioi->rd_ios, itv), S_VALUE(ioj->wr_ios, ioi->wr_ios, itv), ll_s_value(ioj->rd_sectors, ioi->rd_sectors, itv) / fctr, ll_s_value(ioj->wr_sectors, ioi->wr_sectors, itv) / fctr, xds.arqsz, S_VALUE(ioj->rq_ticks, ioi->rq_ticks, itv) / 1000.0, xds.await, /* The ticks output is biased to output 1000 ticks per second */ xds.svctm, /* Again: Ticks in milliseconds */ xds.util / 10.0);

The part we care about is the await part, which is the third column from the right. As above, we can see that what's printed for the await column is "xds.await". Looking a bit further up the function, we can see that xds is likely set here:

compute_ext_disk_stats(&sdc, &sdp, itv, &xds);

The iostat.c file doesn't contain this function, but a quick grep over the source directory shows that the definition is present in common.c:

/* *************************************************************************** * Compute "extended" device statistics (service time, etc.). * * IN: * @sdc Structure with current device statistics. * @sdp Structure with previous device statistics. * @itv Interval of time in jiffies. * * OUT: * @xds Structure with extended statistics. *************************************************************************** */ void compute_ext_disk_stats(struct stats_disk *sdc, struct stats_disk *sdp, unsigned long long itv, struct ext_disk_stats *xds)

Within this function, we can see that await is set based on some arithmetic using various variables, like this:

/* * Kernel gives ticks already in milliseconds for all platforms * => no need for further scaling. */ xds->await = (sdc->nr_ios - sdp->nr_ios) ? ((sdc->rd_ticks - sdp->rd_ticks) + (sdc->wr_ticks - sdp->wr_ticks)) / ((double) (sdc->nr_ios - sdp->nr_ios)) : 0.0;

From the above, it's important for us to know a few pieces of information:

What is nr_ios?

What is rd_ticks?

What is wr_ticks?

What is sdc?

What is sdp?

What is the ? operator?

The sdc and sdp questions can be answered by the comment above the compute_ext_disk_stats function:

* @sdc Structure with current device statistics. * @sdp Structure with previous device statistics.

The next question - what nr_ios actually is - can be obtained back in the write_ext_stat function. There, we have some lines of code which look like this:

sdc.nr_ios = ioi->rd_ios + ioi->wr_ios; sdp.nr_ios = ioj->rd_ios + ioj->wr_ios;

It appears then that nr_ios is the sum of rd_ios and wr_ios.

The remainder of the information can be obtained back in iostat.c. We know from the man page that extended disk stats rely on the /proc/diskstats file. A quick search for this shows a function called read_diskstats_stat:

/* *************************************************************************** * Read stats from /proc/diskstats. * * IN: * @curr Index in array for current sample statistics. *************************************************************************** */ void read_diskstats_stat(int curr)

This function will parse the contents of /proc/diskstats. The important part of this function:

/* major minor name rio rmerge rsect ruse wio wmerge wsect wuse running use aveq */ i = sscanf(line, "%u %u %s %lu %lu %llu %u %lu %lu %llu %u %u %u %u", &major, &minor, dev_name, &rd_ios, &rd_merges_or_rd_sec, &rd_sec_or_wr_ios, &rd_ticks_or_wr_sec, &wr_ios, &wr_merges, &wr_sec, &wr_ticks, &ios_pgr, &tot_ticks, &rq_ticks); ... sdev.rd_ticks = rd_ticks_or_wr_sec;

Documentation on sscanf and % format specifiers is here:

https://linux.die.net/man/3/sscanf https://www.le.ac.uk/users/rjm1/cotter/page_30.htm

From the sscanf man page:

The scanf() family of functions scans input according to format as described below. This format may contain conversion specifications; the results from such conversions, if any, are stored in the locations pointed to by the pointer arguments that follow format.

So the first field of each line is an int and will be read into &major, the second field of each line is an int and will be read into &minor, the third field of each line is a char array and will be read into dev_name, and so on.

From the above, now we know that rd_ios, wr_ios, rd_ticks and wr_ticks directly correlate to fields in /proc/diskstats. We can get a plain English description of what each field in the /proc/diskstats file means by reading the kernel documentation, here: https://www.kernel.org/doc/Documentation/ABI/testing/procfs-diskstats

From cross referencing the variable names above against the fields in the kernel documentation, we can get a plaintext description of what these variables actually are.

rd_ios = field 4. In English, "reads completed successfully"

rd_ticks = field 7. In English, "time spent reading (ms)"

wr_ios = field 8. In English, "writes completed"

wr_ticks = field 11. In English, "time spent writing (ms)"

Now we know all of the information we need to work out what await actually is. A summary:

sdc - Structure with current device statistics

sdp - Structure with previous device statistics

nr_ios - The sum of reads and writes completed successfully (fields 4 and 8 in /proc/diskstats)

rd_ticks - Time spent reading in ms (field 7 in /proc/diskstats)

wr_ticks - Time spent writing in ms (field 11 in /proc/diskstats)

Back to the await calculation, we can see that it's set to this:

xds->await = (sdc->nr_ios - sdp->nr_ios) ? ((sdc->rd_ticks - sdp->rd_ticks) + (sdc->wr_ticks - sdp->wr_ticks)) / ((double) (sdc->nr_ios - sdp->nr_ios)) : 0.0;

One last bit of key information: in C, what you're looking at here with the "?" is a conditional operator (sometimes called a ternary operator). It evaluates to its second argument if the first argument evaluates to true. It evaluates to its third argument (the bit after the :) if the first argument evaluates to false. In English, if there is no difference between the current count for number of I/Os and the previous counter it means there was no disk activity, so set await to 0.0. Otherwise, set await to:

((sdc->rd_ticks - sdp->rd_ticks) + (sdc->wr_ticks - sdp->wr_ticks)) / ((double) (sdc->nr_ios - sdp->nr_ios))

Rewriting that in English:

await = (The difference between the time spent reading in ms plus the time spent writing in milliseconds), divided by (the number of writes completed plus the number of reads completed)

Therefore:

await is the time performing read/write operations divided by the number of read/write operations performed.

PART 4: Conclusion

We know that in the 1000ms between iostat runs, there can not possibly be more than 1000ms worth of reading/writing done. If iostat is sampling 1 second apart, it's impossible for there to be a 15,000ms await. iostat is returning incorrect results. The customer should upgrade to a later version of sysstat, and if the bug persists then we should try and reproduce it, and then open a support case with Red Hat to get it fixed.

0 notes