#Data replication

[analyticspursuit]

What is a Data Pipeline? | Data Pipeline Explained in 60 Seconds

If you've been curious about data pipelines but don't know what they are, this video is for you! Data pipelines are a powerful way to manage and process data, and in this video, we'll explain them in 60 seconds.

If you're looking to learn more about data pipelines, or want to know what they are used for, then this video is for you! We'll walk you through the data pipeline architecture and share some of the uses cases for data pipelines.

By the end of this video, you'll have a better understanding of what a data pipeline is and how it can help you with your data management needs!

[beebfreeb]

To all my long-term followers: You have to look through all of my art and deduct exactly what's wrong with me. I know you can do it by now.

[mostlysignssomeportents]

Cloudburst

Enshittification isn’t inevitable: under different conditions and constraints, the old, good internet could have given way to a new, good internet. Enshittification is the result of specific policy choices: encouraging monopolies; enabling high-speed, digital shell games; and blocking interoperability.

First we allowed companies to buy up their competitors. Google is the shining example here: having made one good product (search), they then fielded an essentially unbroken string of in-house flops, but it didn’t matter, because they were able to buy their way to glory: video, mobile, ad-tech, server management, docs, navigation…They’re not Willy Wonka’s idea factory, they’re Rich Uncle Pennybags, making up for their lack of invention by buying out everyone else:

https://locusmag.com/2022/03/cory-doctorow-vertically-challenged/

But this acquisition-fueled growth isn’t unique to tech. Every administration since Reagan (but not Biden! more on this later) has chipped away at antitrust enforcement, so that every sector has undergone an orgy of mergers, from athletic shoes to sea freight, eyeglasses to pro wrestling:

https://www.whitehouse.gov/cea/written-materials/2021/07/09/the-importance-of-competition-for-the-american-economy/

But tech is different, because digital is flexible in a way that analog can never be. Tech companies can “twiddle” the back-ends of their clouds to change the rules of the business from moment to moment, in a high-speed shell-game that can make it impossible to know what kind of deal you’re getting:

https://pluralistic.net/2023/02/27/knob-jockeys/#bros-be-twiddlin

To make things worse, users are banned from twiddling. The thicket of rules we call IP ensure that twiddling is only done against users, never for them. Reverse-engineering, scraping, bots — these can all be blocked with legal threats and suits and even criminal sanctions, even if they’re being done for legitimate purposes:

https://locusmag.com/2020/09/cory-doctorow-ip/

Enhittification isn’t inevitable but if we let companies buy all their competitors, if we let them twiddle us with every hour that God sends, if we make it illegal to twiddle back in self-defense, we will get twiddled to death. When a company can operate without the discipline of competition, nor of privacy law, nor of labor law, nor of fair trading law, with the US government standing by to punish any rival who alters the logic of their service, then enshittification is the utterly foreseeable outcome.

To understand how our technology gets distorted by these policy choices, consider “The Cloud.” Once, “the cloud” was just a white-board glyph, a way to show that some part of a software’s logic would touch some commodified, fungible, interchangeable appendage of the internet. Today, “The Cloud” is a flashing warning sign, the harbinger of enshittification.

When your image-editing tools live on your computer, your files are yours. But once Adobe moves your software to The Cloud, your critical, labor-intensive, unrecreatable images are purely contingent. At at time, without notice, Adobe can twiddle the back end and literally steal the colors out of your own files:

https://pluralistic.net/2022/10/28/fade-to-black/#trust-the-process

The finance sector loves The Cloud. Add “The Cloud” to a product and profits (money you get for selling something) can turn into rents (money you get for owning something). Profits can be eroded by competition, but rents are evergreen:

https://pluralistic.net/2023/07/24/rent-to-pwn/#kitt-is-a-demon

No wonder The Cloud has seeped into every corner of our lives. Remember your first iPod? Adding music to it was trivial: double click any music file to import it into iTunes, then plug in your iPod and presto, synched! Today, even sophisticated technology users struggle to “side load” files onto their mobile devices. Instead, the mobile duopoly — Apple and Google, who bought their way to mobile glory and have converged on the same rent-seeking business practices, down to the percentages they charge — want you to get your files from The Cloud, via their apps. This isn’t for technological reasons, it’s a business imperative: 30% of every transaction that involves an app gets creamed off by either Apple or Google in pure rents:

https://www.kickstarter.com/projects/doctorow/red-team-blues-another-audiobook-that-amazon-wont-sell/posts/3788112

And yet, The Cloud is undeniably useful. Having your files synch across multiple devices, including your collaborators’ devices, with built-in tools for resolving conflicting changes, is amazing. Indeed, this feat is the holy grail of networked tools, because it’s how programmers write all the software we use, including software in The Cloud.

If you want to know how good a tool can be, just look at the tools that toolsmiths use. With “source control” — the software programmers use to collaboratively write software — we get a very different vision of how The Cloud could operate. Indeed, modern source control doesn’t use The Cloud at all. Programmers’ workflow doesn’t break if they can’t access the internet, and if the company that provides their source control servers goes away, it’s simplicity itself to move onto another server provider.

This isn’t The Cloud, it’s just “the cloud” — that whiteboard glyph from the days of the old, good internet — freely interchangeable, eminently fungible, disposable and replaceable. For a tool like git, Github is just one possible synchronization point among many, all of which have a workflow whereby programmers’ computers automatically make local copies of all relevant data and periodically lob it back up to one or more servers, resolving conflicting edits through a process that is also largely automated.

There’s a name for this model: it’s called “Local First” computing, which is computing that starts from the presumption that the user and their device is the most important element of the system. Networked servers are dumb pipes and dumb storage, a nice-to-have that fails gracefully when it’s not available.

The data structures of source-code are among the most complicated formats we have; if we can do this for code, we can do it for spreadsheets, word-processing files, slide-decks, even edit-decision-lists for video and audio projects. If local-first computing can work for programmers writing code, it can work for the programs those programmers write.

Local-first computing is experiencing a renaissance. Writing for Wired, Gregory Barber traces the history of the movement, starting with the French computer scientist Marc Shapiro, who helped develop the theory of “Conflict-Free Replicated Data” — a way to synchronize data after multiple people edit it — two decades ago:

https://www.wired.com/story/the-cloud-is-a-prison-can-the-local-first-software-movement-set-us-free/

Shapiro and his co-author Nuno Preguiça envisioned CFRD as the building block of a new generation of P2P collaboration tools that weren’t exactly serverless, but which also didn’t rely on servers as the lynchpin of their operation. They published a technical paper that, while exiting, was largely drowned out by the release of GoogleDocs (based on technology built by a company that Google bought, not something Google made in-house).

Shapiro and Preguiça’s work got fresh interest with the 2019 publication of “Local-First Software: You Own Your Data, in spite of the Cloud,” a viral whitepaper-cum-manifesto from a quartet of computer scientists associated with Cambridge University and Ink and Switch, a self-described “industrial research lab”:

https://www.inkandswitch.com/local-first/static/local-first.pdf

The paper describes how its authors — Martin Kleppmann, Adam Wiggins, Peter van Hardenberg and Mark McGranaghan — prototyped and tested a bunch of simple local-first collaboration tools built on CFRD algorithms, with the goal of “network optional…seamless collaboration.” The results are impressive, if nascent. Conflicting edits were simpler to resolve than the authors anticipated, and users found URLs to be a good, intuitive way of sharing documents. The biggest hurdles are relatively minor, like managing large amounts of change-data associated with shared files.

Just as importantly, the paper makes the case for why you’d want to switch to local-first computing. The Cloud is not reliable. Companies like Evernote don’t last forever — they can disappear in an eyeblink, and take your data with them:

https://www.theverge.com/2023/7/9/23789012/evernote-layoff-us-staff-bending-spoons-note-taking-app

Google isn’t likely to disappear any time soon, but Google is a graduate of the Darth Vader MBA program (“I have altered the deal, pray I don’t alter it any further”) and notorious for shuttering its products, even beloved ones like Google Reader:

https://www.theverge.com/23778253/google-reader-death-2013-rss-social

And while the authors don’t mention it, Google is also prone to simply kicking people off all its services, costing them their phone numbers, email addresses, photos, document archives and more:

https://pluralistic.net/2022/08/22/allopathic-risk/#snitches-get-stitches

There is enormous enthusiasm among developers for local-first application design, which is only natural. After all, companies that use The Cloud go to great lengths to make it just “the cloud,” using containerization to simplify hopping from one cloud provider to another in a bid to stave off lock-in from their cloud providers and the enshittification that inevitably follows.

The nimbleness of containerization acts as a disciplining force on cloud providers when they deal with their business customers: disciplined by the threat of losing money, cloud companies are incentivized to treat those customers better. The companies we deal with as end-users know exactly how bad it gets when a tech company can impose high switching costs on you and then turn the screws until things are almost-but-not-quite so bad that you bolt for the doors. They devote fantastic effort to making sure that never happens to them — and that they can always do that to you.

Interoperability — the ability to leave one service for another — is technology’s secret weapon, the thing that ensures that users can turn The Cloud into “the cloud,” a humble whiteboard glyph that you can erase and redraw whenever it suits you. It’s the greatest hedge we have against enshittification, so small wonder that Big Tech has spent decades using interop to clobber their competitors, and lobbying to make it illegal to use interop against them:

https://locusmag.com/2019/01/cory-doctorow-disruption-for-thee-but-not-for-me/

Getting interop back is a hard slog, but it’s also our best shot at creating a new, good internet that lives up the promise of the old, good internet. In my next book, The Internet Con: How to Seize the Means of Computation (Verso Books, Sept 5), I set out a program fro disenshittifying the internet:

https://www.versobooks.com/products/3035-the-internet-con

The book is up for pre-order on Kickstarter now, along with an independent, DRM-free audiobooks (DRM-free media is the content-layer equivalent of containerized services — you can move them into or out of any app you want):

http://seizethemeansofcomputation.org

Meanwhile, Lina Khan, the FTC and the DoJ Antitrust Division are taking steps to halt the economic side of enshittification, publishing new merger guidelines that will ban the kind of anticompetitive merger that let Big Tech buy its way to glory:

https://www.theatlantic.com/ideas/archive/2023/07/biden-administration-corporate-merger-antitrust-guidelines/674779/

The internet doesn’t have to be enshittified, and it’s not too late to disenshittify it. Indeed — the same forces that enshittified the internet — monopoly mergers, a privacy and labor free-for-all, prohibitions on user-side twiddling — have enshittified everything from cars to powered wheelchairs. Not only should we fight enshittification — we must.

Back my anti-enshittification Kickstarter here!

If you’d like an essay-formatted version of this post to read or share, here’s a link to it on pluralistic.net, my surveillance-free, ad- free, tracker-free blog:

https://pluralistic.net/2023/08/03/there-is-no-cloud/#only-other-peoples-computers

Image: Drahtlos (modified) https://commons.wikimedia.org/wiki/File:Motherboard_Intel_386.jpg

CC BY-SA 4.0 https://creativecommons.org/licenses/by-sa/4.0/deed.en

—

cdsessums (modified) https://commons.wikimedia.org/wiki/File:Monsoon_Season_Flagstaff_AZ_clouds_storm.jpg

CC BY-SA 2.0 https://creativecommons.org/licenses/by-sa/2.0/deed.en

[pyrholidonhead]

there is nothing quite like the inextricable bond between autist and emotional support android/ai

[matthewmoorwood]

One thing I like about Star Trek is that whenever we get crossover stuff it's always just like being absolutely sucker punched with the knowledge that these guys are fucking coworkers. I get so emotionally invested in each crew individually that when they interact it feels like seeing people from uni at the supermarket but if you triple the awkwardness.

[chadepitanga]

I like that a lot of people seem to love the "headcanon" that data can get high but it's actually canonically established in the naked now. High as a kite from that polywater intoxication thing. "His eyes are red-ish sometimes bc he smokes saurian hash on a risian bong", thats the headcanon, but he 1000% can get high

[cainballad]

@lastaim and I attended Nier Orchestra Concert 12024 {the end of data} in Washington DC over the weekend.

it was GREAT! SO AMAZING. The whole concert was basically a “sequel” after the game (Ending E). Hearing 2B and 9S again in this new tale was so amazing, esp how they played the music with each scene~ It was full of mystery, tragedy and fulfilling… we need to go again one day.

Not me almost crying upon hearing Shadowlord rendition or Kainé. Weight of the World and its visuals made both our emotions leak out. Hearing it live compared is always such an experience. This was something else entirely.

9s approves.

[asokkalypsenow]

My friends and i are playing dnd dragon heist and i and my partner surprised them yesterday by showing off this replica we made of the house you get in that campaign, along with decorated rooms for everyone! It was a lot of fun but so much work i wont play Minecraft for like 3 months now lol

[hal-o-ween]

This is not to say that you do not like the other options, but rather that you have a preference for the ones you select

[madesimplemssql]

Snapshot Replication is a robust feature in SQL Server that allows data to be distributed as it appears at a specific point in time. Let's Explore:

https://madesimplemssql.com/snapshot-replication/

Please follow on FB: https://www.facebook.com/profile.php?id=100091338502392

[caninecowboy]

feeling increasingly discouraged about my research YEEHAW

[sparrowsgarden]

As someone with a passing knowledge of how AI/computer image generation works, I really wouldn't have minded my stuff being plugged into an engine before a handful of months ago.

That isn't to say that artists shouldn't have control over whether their content is used as training data (the general lack of discretion in the data sets is a whole issue of its own), but up until recently these programs were a lot more of a novelty, and also commonly acknowledged to be very bad at what they were trying to do. While stable diffusion has made the outputs scan better, they're clearly still lacking a lot. Of course, I've never been personally at risk, because I don't have a singular distinctive visual art style - I have a good handful which I apply depending on my mood or what the project demands. My writing, also, is geared toward clear and concise communication, and that simply can't be imitated if the engine doesn't understand the concept behind what it's trying to communicate.

But moreso, it seemed absurd to me that image generation in its current state could compete with actual artists, and I still don't really see that as being as much of a concern as some people think. The people who understand the value of human-made art will still want it regardless of replicability. Anyone willing to substitute it for something that gives them much less flexibility with the final output probably either wasn't compensating artists fairly before, or wasn't buying art at all. I'm not really convinced this hypothetical guy exists who previously would have paid full price for a work by a popular digital artist, but won't anymore because they can generate a facsimile of the artist's style. I think the problem is way more with the general devaluation of art and design work.

There certainly may be less commercial work, but the survival of artists shouldn't depend on their commercial viability. In our capitalist hellscape those corners are liable to be cut when any substitute is available. That could be computer image generation, or it could be a guy who can whip out a barely-passable Photoshop job in 20 minutes where an artist (or a designer who cared and was paid enough to make it more than barely-passable) would spend hours.

Anyone who's argued for modern art before - or on the other end of the spectrum, for photorealism - knows that art shouldn't be valued only by how easy the image is to replicate, and I feel like the emphasis on commercial viability really flattens the discussion on how this technology actually has the capability for harm.

My concern is not about the artists not getting paid for jobs in industries they were already being cut out of. My concern is about how our world might get measurably worse because some people have decided that "actively garbage but scans ok at a first glance" is good enough. Or they don't know enough about what they're doing to discern whether the output is good quality, which is uh, horrifying in some of these contexts. But again, those people were already cutting those corners - it's just gotten way easier to point and laugh at them.

So really, if we want to protect ourselves, I think the main thing that we can do is point and laugh. This won't guarantee that they'll change anything meaningfully, but it'll at least show them they can't get away with making these changes without anyone noticing. It'll hopefully teach people who have not figured it out that these tools are not smart and essays are not an appropriate use for them. And it'll give us some pretty guilt-free schadenfreude to tide us over.

[therikermanuver]

It's fascinating to me how all the vegetarians in Star Trek will eat replicated meat

[datas00ng]

i know in my soul that data is an excellent cook

[danthropologie]

the sim stuff just really affirms that this was the best decision he could have made...if he had moved directly to another team he wouldn't have had the time and space to unlearn the bad habits. like yes he has apparently made a lot of progress course correcting his driving in a fairly short time but if he'd been on the grid with the pressure of performing in like a haas or something with those bad habits? oof it would have been a nightmare. the press alone. a full year building himself back up is great, possibly returning fully confident is even better...but also if he doesn't come back this is still such a nice way to heal from the past few years.

right!!! even the thought of having to do this in the fucking red bull seems like it would have been a bit of a nightmare situation. he needed to be able to do it privately, away from the cameras and questions and judgement :(

[muffinrag]

trying to be so calm about accidentally deleting a bunch of videos that were part of my favorite project but deep down i do kind of want to explode the entire world and then myself