getaether - Tumblr blog

getaether · 5 years ago

Text

Aether P2P dev.14 released with moderation transparency, content deletion

I've cut a new update for Aether, and this update comes with a lot of goodies. Without further ado, let's dig in.

<shilling> Aether's P2P edition (which this update is about) is supported by Aether Pro, a team collaboration tool for introverts. It's an internal Reddit-like forum for your company. This is better than Slack because it allows you to focus more, and for longer, because that it's not live chat but asynchronous, n-level deep threaded posts with no immediate pressure to respond.

If you're an engineer or a team lead who's getting needlessly pinged all day long, Aether Pro could help. Check it out at aether.app.

(The code FOUNDER-REFERRAL gets you 30% off if you want to give it a shot.) </shilling>

Aether P2P primer

For those who are new, Aether P2P is an open-source peer-to-peer communication platform. Its basic idea is like Reddit to the modern viewer, but the actual idea comes from much older times — from Usenet.

Aether P2P is a decentralised Usenet.

We have a few hundred communities, spam protection with proof-of-work, and a unique moderation system based on the principle of the consent of the governed.

In a little more detail, one of the core features of Aether is that the moderation layer is implemented as a filter. This filter gets applied at the user's local machine: the users can choose which moderators that they want to respect the moderation of. A user can disable the mod actions of an individual moderator. If that happens, even if that mod deletes some content, it will remain visible to the user. Likewise, a user can also choose a non-mod person to be his or her moderator for that community. In this case, mod actions released by that person will apply to the local user.

These make-mod and disqualify-mod commands double for votes in another Aether concept called mod elections. Every community holds a continuous elections process. This process can promote users to mod status if enough people make these users a mod for that community.

You can find more information about this at the documentation: Moderation and Elections in Aether P2P

(Aether Pro is different — it's a SaaS app that largely shares Aether P2P's visual design and communication style, but it has an enterprise backend suitable for work).

New features

Moderation transparency

This feature adds the front-end for the mod activity tab as the last piece, making it user-visible. That means all moderation actions taken so far, are now visible to all users, as well as their stated reasons.

It looks like this:

This is a cornerstone for implementing the remainder of the elections support, which is as of now only supported in the backend.

One thing that is noteworthy in the screen is that there is a concept of a 'deletion request'. This is only applied if you accept the modship of the issuing user. That means there can be multiple deletion requests per post. Some, all, or none might apply based on your preferences. For the time being this is read-only. In the future, you will be able to control which of these deletion requests will apply as well.

With this, Aether gains its first feature that is not available anywhere else, in any other app (that we know of): Moderators and moderation actions are transparent to the users.

Self-deletion

You can now delete your posts in a fashion that does not break the layout of the conversation. Your post will be replaced with a [deleted] and posts that respond to you will be retained.

This is different from a mod deleting your content; when a mod deletes your content, your content disappears for anyone who regards that mod as authoritative. This self-deletion, on the other side, makes your post's text disappear from everyone, regardless of whether they trust you or not. This is possible because you own the post in question and have edit rights, which a moderator does not.

New content indicators

When there's new content in a community you've been a member of, that community will pop up a blue highlight notifying you of new content within.

Improvements, bugfixes and internal changes

I've upgraded the versions of dependencies, so everything should be a little snappier. Also fixed a lot of miscellaneous bugs and spent some time rearchitecting some of the components. This should make them more amenable to our upcoming elections system. You can see the full list of these changes at the changelog.

Future, or where do we go from there

Sustainability for Aether P2P and Aether Pro

At the beginning of this year, I've received some small amount of pre-seed venture capital to make Aether Pro. Aether Pro uses a similar code-base to create a SaaS team collaboration app that is well suited to engineering teams between 5 to 50 people. This is because it gives you an internal forum with like Reddit-like threads. This lifts the noise ceiling (after which conversation starts to fray at the edges) from 5~ people in a Slack channel, to thousands.

Aether Pro pays for the development of Aether P2P through its marketing budget. It makes it so that if we succeed, we will never have to monetise Aether P2P. This is a way to make Aether into a sustainable business that pays for me and a small team so we can continue working on it.

If you think you'd be interested in an internal forum that's great for software architecture discussions, across multiple time zones or from the next room, give it a shot. More info and a much better telling of what problem it solves for you are at aether.app.

Upcoming planned features

Elections

We are well on our way to finish the elections service, which I consider the last major not-yet-implemented feature of Aether.

Nominally speaking, it's already available in the backend in a halfway-built stage. However, it remains disabled on the production builds. With the experience from these updates shipping today, we're one step closer to having them run in production.

Partial sharing

We also have another, more mundane, but major feature request. This is being able to control what your computer retransmits to other nodes. Since Aether is a peer-to-peer network, it shares and retransmits all content it receives to other computers — you're a peer in the swarm. However, this is not always desirable, and people can have personal objections to sharing certain pieces of content.

As it stands, Aether works by having a compressed, text-based copy of the whole network on a 6-month rolling basis on every node. This kind of selective sharing would make our 'full nodes into partial nodes. The work required for this feature is that I need to make sure that the application does not make assumptions about being able to receive all content that is no longer valid in this partial-sharing environment.

Filter lists

One example of this filter lists feature is already in, the SFW list. This is an optional feature that enables us to mark certain communities as SFW. The user can then choose whether they want to see SFW posts only, or SFW and NSFW posts.

The filter lists feature makes these kinds of lists into a standard format, a JSON link that the user can add to the app. (The idea is like AdBlock filter lists). This way, the users can create and share their filter lists, and we will no longer need to manually manage them.

Until the next update, @nehbit

0 notes

getaether · 5 years ago

Text

A new funding model for open source software

tl;dr

I maintain a semi-popular open source project. This is a recollection of my attempts at various potential methods of getting paid so I could remain full time supporting it, and what happened when I tried them.

In the end, I decided that the most honest and respectful way of doing so is to build a business on top of it, as a separate product in an adjacent market that supports and pays for the development of the open-source one. I considered the common freemium and open-core models espoused elsewhere. Having been satisfied with neither, I've come up with a hybrid model that I think could be useful for others and might be a better fit for open-source in some cases, because it aims to keep the core open-souce product un-monetized. Details in the article below.

After our new release a few days ago, I had a little bit of time to go through the questions I've received over the past few days. Most of them relate to how our new funding structure works with the existing project (no changes) and whether it affects what we want to build (it does not).

Considering open-source funding is something of a conversation starter (or ender) these days, I wanted to post something on what we're doing, how it works, and why we ended up doing it this way.

Some context

Aether (Website, Github) is an open-source, peer-to-peer app to create Reddit-like communities without servers. It allows each community to elect or impeach their mods, an all mod actions are public. Moderators cannot delete posts without an audit log. Likewise, users can choose which mods they want to have active for them — they can disable a mod as they wish.

Aether had been almost a year-long project for me at the point of shipping, and I worked full time on it. This was not in expectation of anything: I needed a "sabbatical" of sorts from my work at a FAANG company, and that was largely it. Nevertheless, after the release, it started to get real usage.

At that point, not only I started to get support requests, I was also nearing the end of my savings. The only way I could continue this, which had become a full-time job at that point, was that I would somehow get paid for it. Or I would have to go back to having a full-time job and let go. Thus began my journey to find whether living on Aether would be possible.

Life as a hunter-gatherer

What I tried

Nadia's Lemonade Stand was the first thing I encountered. On one hand, this is a great resource. On the other, it makes clear that the funding you can get through these is a far cry from what you would need to be able to survive in San Francisco.

I tried some, still. It turned out that these tools worked better for letting people express their support. For that, they're great: it makes you feel good and supported — as long as it is not your main compensation and more of a 'keep up the good work' tip.

What they don't offer is a living wage or any way to go full time maintaining an open-source project. This was easy to see for me, coming from a FAANG salary — that made the opportunity cost obvious. If you can make $10,000 a month from donations doing open source work, I can guarantee you that your salary in any large tech company (or even startup) would be much more — to the tune of 10x to 25x. And you would be eating the difference.

The issue isn't even that you might not be able to live on it: it's that if you can make enough to live on donations per month, by all economic reasoning, you shouldn't. You'd be paying so much in opportunity cost for the privilege. You're better off working for FAANG for a few years, save up the money, and use your newfound wealth towards open-source. This is tough because it creates an environment where open-source is only possible for either very junior folks, which use it to make a name for themselves, people in Academia, including students, and educators, or financially independent people.

For the first two, I would argue it's a win-win, you help everybody out, make your name, and then use this to get a good job. For the lattermost, it's a good deal since you're ultimately not concerned with making money. However, the life skills that got you to there likely also prevents you from forfeiting ~99% of your output in opportunity cost for the privilege. Incidentally, only a few of the many excellent programmers here in Silicon Valley that have 'made it' financially write open-source code.

My verdict for kind of funding was that most of us couldn't make a living through it, except through personal sacrifice. This could either be living preferences (e.g. moving to a cheaper country where you know no one) or paying a shocking amount of money in opportunity cost. And that's money you're giving up that you could use to better provide for your kid(s), for your wife, and your greater family.

What I didn't try

Aether is a community app, so it does get a decent number of eyeballs per month. The reason I built this was to build a community place without a catch, away from the prying eyes of any mega-corp — a place of Internet's own that exists in nowhere but in the 'minds' (i.e. devices) of its members.

Advertising into it could be built in a fashion that does not track users. This does not work, because unlikely as it seems, the vast majority of the value of advertising comes from targeting. And for targeting to work, you need lots of user data. So the only way for anyone to make money from advertising is to track users, to some extent. In the end, I decided it was incompatible with the core idea for us to have advertising. If for nothing else, I use Aether every day, and I would like to keep using it.

Attempts at subsistence farming

After trying to make donations work, I ended up moving to something more organised. That was seeking sponsorship from corporations. For this, the result was middling, and @feross puts it better than I can:

“Sustainability” only means subsistence

Increasingly, maintainers are starting to go through a mindset shift. We don’t want to ask politely for donations anymore – donations that often never come, or when they do they’re usually only enough for maintainers to sustain themselves but never enough to thrive.

Sustainability is another way to say subsistence. This is why the common phrase “open source sustainability” isn’t ideal. (Source)

This is not as bleak as it sounds so much as it is obvious — especially so if you consider it a restating of the fact that donations come from the goodwill budget. That budget is usually either nonexistent for most people, and companies. All the while the things they do with your source code earns them the very money they strike under profit. The open-source developer, however, (rightly or wrongly) expects it to be considered under core business expense. This is the source of most of the anguish.

It's also understandable from the company's side, though. Since the raison d'etre of a business is to make money, and of open source is for developers to give things away for businesses to exploit. In an ideal world, any company who donates some of the proceeds back to the open-source maintainer will be out-competed by another who uses the open-source product but does not — since the latter will have more capital at hand that it can use to better itself.

As a result, a successful open source maintainer not even only makes its users indifferent to his or her plight, he/she actively discourages them, because any donation that would go to the maintainer would improve the open-source software for anyone, making the donating company effectively pay for improving software used by competitors.

This is sad because it's a prisoners' dilemma situation where the Nash equilibrium squarely sits at the lone option that makes the maintainer go starving. To be able to avoid this game-theoretic doomsday scenario that most open source projects suffer from, the maintainer needs to be able to offer to the paying company some advantage, something to give back to its paying users.

That, I realised, after an embarrassingly long time spent pondering, is a bog-standard, good old business that sells goods or services.

(In fact, capitalism can be thought of as an emergent natural phenomenon arising out of the Nash equilibrium state of an unsupervised, repeated prisoners' dilemma. Nevertheless, that's a topic for another day.)

Long story short, my attempts to get funding from companies that would ostensibly support the goal rolled up to dead ends, because it turned out that most of their 'foundations' that they had set up were thinly veiled attempts to get other people to implement their crypto coin of choice. Their non-profit funding was a way to pay a trivial amount of sum to cover for that.

I have also tried to raise funding from some actual, honest-to-god foundations. It turned out that the amounts of funding that we ended up talking about ended up within the 'donations' range. The only difference was that they would provide it as a lump sum instead of averaged over month-to-month.

The dawn of the Agricultural revolution

At this point, it was pretty obvious to me that I would need to take care of this myself. It was becoming obvious that the only real way to make money out of an open-source project to ensure its continuity would be to build a good business on top. As obvious as it is to someone not within the open-source world, this was a revelation for me. I now think it's one of the things that make sense without needing much of an intricate justification: if you'd like money, offer something in exchange.

That was refreshing, as it cuts down the number of assumptions you make about the world drastically. Instead of having a nebulous path through which you produce value as open-source software, and then expect to have some of that value return to you somehow, you make your software available as before, but give people who pay you, more stuff.

But isn't that proprietary software? Not always, though that is one way. There were a few approaches I considered.

Freemium open-source

This is a model where the core product is open source, but add-ons are not. Redis is (arguably) one example of this. The issue with this is that this leads to a point where the open-source product is not very useful or modern. Since most new development happens in the add-ons, and that's where most of the new work goes.

Being able to sustain this and also have the core, non-add-on version useful takes ongoing vigilance. Since Aether was a communication product designed for public discussion, the add-on space was small. It could have been possible to offer better tools for elite users like moderators to allow for better automation, for example. Though it is likely it would still end up in a place where there is a tension between adding a feature to the core, and to the premium add-on you sell.

Open-core

This is where you have a core product, and a second paid product that builds on top of that. That second product provides extra features not found in the first. Gitlab is the vanguard of this one. The difference between freemium and this is that the extra features are usually subscription only and that you get either all or none. This is usually called a Community Edition (available for free), and an Enterprise Edition (paid).

The principal issue with the open-core is that it's undefined what would happen if someone sends in a pull request that adds a paid feature in the free version. Since the paid version is a superset of the free one, by definition any paid feature would be an improvement. Yet, the company would be inclined to not accept the PR since that would weaken their paid product. They would be competing with themselves.

In practice, that does not happen often. Not many people are capable of adding nontrivial features to large applications, and fewer still want to do it for free. Nevertheless, this remains a large incentive mismatch that one cannot ignore.

Our model, or what we ended up going with

"Aether model" (for lack of a better name)

This model involves, like the open-core, two products. However, instead of one being a superset of the other, they are two different products. They aim for divergent goals, non-overlapping use cases, and different users. They share a similar codebase kept in sync, though they grow in different directions.

In our case, Aether P2P is a peer-to-peer network that creates self-governing public communities.

The second product, we ended up building as Aether Pro,. It's a SaaS app that we offer on a subscription basis, that targets internal teams within tech companies, being a collaboration tool like Slack. It's a Reddit-like, threaded forum that disrupts people fewer times than having Slack open, with its constant, ongoing pings.

In other words, not only that Aether P2P cannot do the job of Aether Pro, but it's also that Aether Pro cannot do the job of Aether P2P.

An example of this purpose-separation in action is this. In Aether P2P, everything is public. One of the features I've received the most for that version was to have private communities.

In Aether P2P, this is impossible, because it's a peer-to-peer network. The only way to incentivise people to pass on the content and use their bandwidth to serve the network is for them to gain something from it. Any content that would be private to a specific party would benefit no one in the network except its recipient. As a result, the network would have no incentive to carry it. If I implemented this feature in the P2P edition, somebody would fork it, remove it and release their own. And they would be right since it would be an improvement to remove it for anybody except the recipients of those private messages.

On the other hand, in Aether Pro, we can have everything private, because we are now hosting Aether Pro backends on our SaaS. And these backends are provisioned by Kubernetes to our exact requirements — no user needs to carry content that it cannot read or act on. That said, this also means Aether Pro cannot serve public communities, and that is fine! We have Aether P2P for that.

It's a similar story with the target audiences: the Pro audience is groups of 5-25 large teams of engineers who collaboratively work on a codebase. This is common with remote companies where people build their product across many time zones. Aether Pro, above Slack, offers them a way to communicate async with each other that does not break anyone's focus. It allows them to have a complex, multi-threaded discussion across themselves.

For the Aether P2P, the audience is similar, likely even the same people, but in personal contexts. This targets essentially a very techy, Reddit-like audience. Because these audiences interact and share members, it makes the P2P version a valuable marketing tool for us to invest in, since it brings us, in the marketing parlance, qualified leads. This sets incentives right, too. We pay for the development of Aether P2P out of our marketing budget, because every time we do something with it, we get new users... users who are also in our target market for Aether Pro, and like Aether P2P enough to make the recommendation to their employers.

Other benefits

We do not depend on anyone's benevolence: we offer a service and charge money in exchange. It's an honest, sustainable business.

Since the source code is shared to some extent, the improvements, bug fixes we do on the Pro can move to the P2P for free. P2P edition by default gets kept up to date by the work done on the Pro, no extra effort needed.

Since they target different markets, we don't have to ever think about 'Would adding this feature to P2P cannibalise Pro sales?'

Lastly, it makes Aether P2P completely free of any attempts to monetise it. Since the majority of the issues with any communication platform is caused by attempts to make money, this is a good thing. Aether P2P does not need to do anything more than remaining attractive to its current type of users.

It allows me to avoid marketing. Or rather, I turned the least appetising part of my job as a founder into writing open-source code. I then release it and later talk, to people about it. That is the extent of our 'marketing'. Today, our work for Aether's P2P edition is funded by the marketing budget for the Pro, and it has a pretty great ROI. To me, this is a major win.

In the end, I'm a nerd: I'm not great at getting on stage and talking to people. Never been great at that, probably won't ever be. But what I can do is I can write code, and if that can be my way of reaching out, I'll be happy.

P.S. If you'd like to try Aether Pro with your team, here's a coupon code for 30% off: FOUNDER-REFERRAL. If you're an engineer, this is a good way to buy the calm at work. You can also see a few more reasons why here at aether.app.

0 notes

getaether · 5 years ago

Text

Introducing Aether Pro 🎉

It's time to unroll the wraps on something we've been working on for a while: Aether Pro.

It's a collaboration app for your team that creates a private space with channels. It's similar to Slack or Discord in purpose, but it isn't chat based, instead it offers something similar to Reddit for your team. It's especially useful for remote teams as a searchable store of institutional memory.

It works well for long-running discussions, unlike chat, and since it's structured information storage, not a flat chat log, it remains easy to understand and get the context well into the future.

You can check it out here.

We also have an article to help you figure out if this could work for you.

We don't have this available yet to everyone, but we have been testing it privately for the past few months, and it's come to a point where we're excited to show it to the world and start testing it in public. But let's talk about the most obvious question first:

Aether vs Aether Pro, differences

First, what it is not: The currently available Aether P2P is not a "lite" version of the new Aether Pro. They are two different apps, they serve different needs. They don't do the job of each other.

The names are similar because they share a similar look-and-feel, and happen to be built by the same people.

Here's the gist:

Aether Aether Pro What App for public communities, backed by a serverless peer-to-peer network App for private teams within companies or for groups of people (i.e. a private entrepreneurs community). Common use cases Reddit-like. Self-moderating interest-based public communities with elections, moderation transparency and privacy. Slack / Discord-like in membership, Reddit-like in discussion style. Focus-respecting collaboration tool for members of a team. Backend Aether Community Peer-to-Peer Network. P2P, best effort delivery. Your computer becomes part of the network and helps distribute the content Hosted SaaS. Backed by multiple-redundancy servers, hosted on Google Cloud Platform and AWS. On-premises installations available Content availability All content is 100% public, and distributed using the Aether P2P network, viewable by anyone in the world. Does not have private communities All content is private to the team and encrypted, not available to the public unless without an invite Content types Text only, media content can be linked and viewed if direct link Text, images / videos, documents, arbitrary files can be uploaded into and viewed within threads and posts backed by hosted servers (upcoming) Data retention 6 months, after which the data is deleted from the ephemeral network No limit. (You can choose your own deletion threshold) Cost Free & open source (AGPL-licensed), with $10/month option to buy unique usernames to support development $10 / month per member of a team Website getaether.net getaether.net/pro

This is not the full extent of differences — Aether Pro does have more features that are only possible because it's a hosted service, such as having unique names for its channels, users, authentication, and more. Those aren't mentioned here, but you can take a look at the Aether Pro website to find out more.

What does it do?

The elevator pitch is that Aether Pro is a team communication app, similar in purpose to Slack. The major difference is that our app is largely based on asynchronous communication (think email-like), instead of chat.

If you consider Slack to be a modern version of IRC, Aether Pro would be the modern version of work email groups.

In other words, on top of email groups, Aether Pro adds:

@mentions,

Better inline discussions,

Image embeds,

Source code highlighting,

Automated reporting,

Bots,

External email binding,

Integrations

and more. That is what we mean by a modern version.

Sounds interesting? Take a look here to find out if it could work for you and your team.

Generally speaking:

If you're an engineer, product manager or designer, thus need consistent focus to do your work

If your work team is past a certain size (5+ people),

If you're having issues focusing at work due to team chat

Aether Pro might be able to help. We'd love to talk to you - you can book a meeting with us here if you have questions.

How to join the beta with your team

We'll remove this 100% off code when this offer is no longer available. If you're seeing the code below, it's still open. 🙂

You can use the coupon code XXXX (Sorry it's gone!) to create a team and start using it. We're running the pilots to get user feedback, so we would love to hear from you. In fact, if you're willing to hop on a feedback call every other week, you can keep using it for free.

Future posts in series: our business model

We have more announcements to make later on, and we'll write about our business model where using Aether Pro financially supports the Aether P2P project, and why we've chosen this method for supporting it as such.

But in the meanwhile, rest assured that both projects are fully within our planning scope. In fact, we'll be cutting a new Aether P2P release with some of the backported improvements and bugfixes in Aether Pro soon.

We'll add links to our the posts mentioned above here when they are posted.

In the meanwhile, cheers! Aether P2P & Aether Pro team

0 notes

getaether · 6 years ago

Text

Aether Dev.13 released: up to 30x faster updates, live-updating ‘firehose’ feed

After a bit of quiet, a new release of Aether has been cut. Your app should update itself on Windows and Mac, and on Linux, you should download the new Snap after backing up your user profile to get this one.

This one does come with two major features, so I wanted to highlight them a little bit.

The new graph compiler

Context

As a quick recap, Aether is a P2P network that distributes a content graph. This distribution happens on a flood basis, so every node has every piece of content. (Upcoming feature: users can choose what to broadcast out)

We manage to keep the network at a manageable size because of two things: a) All content on the network is text, and b) Aether is an ephemeral network: too old content is deleted based on a deletion policy from all nodes. Even if the too-old content is kept available somewhere, the other nodes won't ask for it, so the propagation is killed.

This is all raw data though - while there are some nifty stuff in being able to efficiently sync content on a flood basis (we do some opportunistic caching and pre-baked responses), this is ultimately a few [thousand | hundred thousand] rows on a SQL table.

And it turns out, the derivation step that allows the users to have user profiles, votes (vote counts!), moderation, elections and all the other niceties of Aether from those raw SQL rows is actually fairly complex. As in, it's a little too slow to do it real time: if you open a thread with 50 posts, assuming real time, you'd need to fetch the community entity, the thread entity, the 50 posts, and all votes those 50 posts ever received and the user entities of all of those entities mentioned above.

As you can imagine, even a relatively small thread like this creates an exponentially increasing number of SQL queries, given the constraint to render it on runtime.

What Aether does instead, is that it has a frontend graph compiler. This compiler is incremental: it only compiles content as it comes in, and saves it into a separate, compiled key-value store. This is the main store that the user interacts with, and since these entities are 'baked in' per user settings, the calls are instant. Every time we receive new raw content from the network, we incrementally patch this 'view of the universe'. This solves the issue with not being able to render the whole network in runtime.

This is also what allows the users to ultimately be able to choose their own moderators. Every user is in full control of his own frontend graph, and they can disable specific moderators they don't like, or add the ones that they happen to like. Since this whole graph is compiled on the frontend, these kind of changes can be taken into account at compile time, which happens about every 2 minutes.

Updates

Since we now have a decently sized corpus of content, the frontend compile times have been increasing. This started around a few seconds when Aether first launched, and as of the last month, it's reached 90~ seconds on my computer, which isn't great! Especially considering the fact that the app attempts a refresh every 2 minutes, that means a lot of time spent in 'Refreshing...' state. It does not prevent the users from interacting with the app, neither does it stress the CPU or RAM, but it does create some disk use and general weirdness that manifests itself as latency spikes.

I have been trying a few approaches to bring this back to a reasonable duration (a few seconds for subsequent compiles) for the past month or so. In other words, we want this time to scale with the number of new entries coming in, not with the total size of the database.

Results

Here's what's shipping with this update:

Our first-ever compile has improved 1.9x, and our subsequent compiles have improved a whopping 30.6x. 🙂

But how?

The frontend compiler was, up until now, 'good enough' — it had not seen much attention in terms of speed or optimisation. This was the first time that someone had the chance to dive in and figure out what could be better. In short, a bunch of low hanging fruits!

(Heads up, the two below are largely about software engineering. Feel free to skip if that's not your cup of tea.)

Optimisation 1

This was fairly straightforward: I ended up batching the compilation result writes into blocks, so that they happened as part of the same transaction. Now, this is actually fairly standard, however in Aether, this was a little more complex than usual because of Go's concurrency. We've been using a lot of goroutines (~threads) which do run in their own processor core or hyperthread, and this is great, because it fully utilises your CPU, even if you have a gigantic AMD Threadripper with an ungodly number of cores, Aether can still fully saturate it if needed.

However, that also meant that every thread, when done, would have to commit its own results to the key-value store. This made things simpler, at the cost of writing a lot of tiny pieces of data to the disk in a way that created some disk contention.

In the end, we ended up profiling the parts of the compiler process that suffered the most from this disk write bottleneck, and I've changed the code in a fashion that allows me to delay (tradeoff: RAM) these writes together and flush them into the disk at the same time, and do it in a thread-safe way. Doing this removed a major disk bottleneck at the cost of some increase in implementation complexity.

This is what bought us the drop in the initial compile from 92s to 47s.

Optimisation 2

The best way to make a program be faster is to make it do less, and avoid work as much as humanly possible.

This second trick is effectively that. I've come up with a way to determine, for every patch (raw set of updates coming from the backend), the frontend now determines what is called, borrowing a term from Physics, the observable universe. This means anything that could possibly be affected by the set of changes that are incoming.

The nice thing about having that map of all possibilities that could ever be invoked is that it is possible to reduce the scope the graph compiler operates: instead of working on the whole graph, it can now only operate on the parts of the graph that has a nonzero chance of having been affected. This leads to a massive reduction in the amount of work needed, and this is how we can get the subsequent compile speed from 92s to 3s.

In short, because we can now definitively prove that certain pieces of work is unnecessary, we can now skip them and do nothing, and that is why this is so fast. For the things we cannot prove that won't be changed we still compile. We might be able to push this idea of the observable universe ever closer to its smallest possible form, however this has already provided massive gains even in its v1 that is shipping now.

In the end, though, this is a tricky one to assess the correctness of. We've been testing this for the past week or so, and it seems to be functional so far. Nevertheless, lots of things affect each other in surprising ways, and it is possible that this observable universe determination might be missing 'following' certain possible interactions, at least in its first iteration. Ultimately this is something that we need to see in the real world, so it's probably best that it is out for real user feedback now.

The 'New' feed

This has been one of the more requested features in the recent past, so I'm happy to say that this is now in. This is pretty much what it says on the tin, it's a feed of the most recent interactions on the network. Since the network is not huge yet, this feed still advances slowly enough to make sense, so it's been fun watching it advance for the past day or two I've been testing this. This is basically 'The Firehose'.

This feed is also useful for people who want to build on Aether as well, since it can be consumed by other apps as the raw input.

It is also live updating, meaning you will see new posts coming in as they arrive by the minute.

By the way, it'll start to collect new content from the point of installation on the new update, but if you'd like to get it filled sooner, you can go to your frontend config folder (the exact location is shown in the app settings), and delete the KvStore.kv after fully shutting down the app. After reopening it, it should build the new feed from scratch.

Other bug fixes

We also have a smaller set of bugfixes that don't merit mentions on their own — they are largely cosmetic.

Availability

We've been testing this release for the past week or so, but the changes are large enough that we need to see it on the live network. The update will be pushed to the auto-update system in about a day from this blog post: if all goes right, by the time you're reading this, it should be already available.

This is it for now. Cheers!

0 notes

getaether · 6 years ago

Text

Aether Dev.12 released

Hey folks,

A new version of Aether (dev.12) is now released. You can download it here.

This release includes improvements like #hashtags, and code higlighting for 185 programming languages, making Aether more useful for programming discussions and collaboration.

There is a more in-depth blog post with commentary on each feature, as well as where they are planning to head in the future is available here on Patreon.

The changelog is available here.

Cheers!

0 notes

getaether · 6 years ago

Text

Aether grows past 500 concurrent nodes

It's been a pretty fun week so far.

After I posted Aether for Redditors, Aether ended up on the front page of Hacker News again. (Link)

The drill is familiar by now. Just keep everything working, and it'll pass in a couple days. This time around, I've gotten about 6,600 unique visitors on the main site or so — which is normal, since the HN link was towards the blog, and not the main site.

There's one more benefit in these things — it pushes the system to the next order of magnitude scaling, and it gives you a glimpse of where the next set of scaling issues are going to come from. This is indeed what happened when Aether broke through 500 concurrent nodes online. If you've been on the network for the past few days, you might have been having some trouble seeing new posts from other people. I'm writing to give a little bit of an educated guess on why, and what I'm doing to improve things. The stuff I've mentioned in this one is going to come in an update in the next few days or so.

In general, this is a great position to be in — Aether is picking up steam at a pretty steep pace, so when these 10x whale events happen, it's always insightful. The reason I can be working these issues right now, early as possible, is because you guys came in all at the same time and stressed out the algorithm that chooses which nodes to connect to. So thanks for being here and using it. 🙂

What I thought it was

I had released an update one day earlier, and I thought that was the cause for the connectivity problems. So far as I can see, it wasn't. I reverted that update, and it seems it's pretty much still the same.

There are some users in Windows where because of some sort of antivirus or some other external issue, the app is not able to write cache files. After some debugging with some folks at the meta forum we've ruled that out as the culprit. (Thanks!)

What was happening

Aether is a flood network. For this to work effectively, every node has to choose who to connect to at any point in time. These connections happen every minute or two. These are all outbound connections.

You also receive inbound connections. There is a limit on how many inbound connections you can receive simultaneously at any one time, to respect your internet connection and not to tax your machine. In essence, your node has a defined number of 'slots' that other nodes can connect to. When your node runs out of slots, it starts to tell other nodes: "I'm too busy, connect back later".

An Aether node, to determine which nodes to connect to, does a 'network scan'. This is, in very simple terms, a process to check some nodes that have the highest likelihood of being online.

This network scan uses the first two steps of a full sync. So it's much lighter than a regular sync. It's just a ping.

Unfortunately, what happens is that these pings also take a slot. These slots take 60 seconds to clear, to allow for continued requests on the same slot if needed.

When we had fewer active nodes, network scans and regular syncs were both able to fit the given number of slots at any point in time.

When we came close to the 500 concurrent nodes, that was no longer true. Since syncs hit one node, but scans hit multiple nodes, the scan traffic grew much faster than the sync traffic, clogging the slots that were actually meant for syncs. This is why the updates slowed down.

The sync logic was a little too soft: when it failed a small number of times, it stopped trying to sync in that cycle, and left it for a future cycle. Since the large majority of the nodes in the network were saturated by scans, this meant it kept sitting around a lot. This was a means to reduce the number of scans, since every sync attempt would mean a new scan, but combined with other effects mentioned above, it backed itself into a corner.

What's the fix?

Bunch of things.

The scans will be rate-limited. Now a node can do one scan every ten minutes, instead of being able to trigger a scan every time it wants. This should reduce the slots used by scans. (They are a tiny portion of the traffic—since they're essentially pings—but since they're indistinguishable from a sync starting, they took a full slot.)

There'll be separate slots, only for scans. This makes it so that the syncs will never be blocked by scans.

Sync logic is more aggressive, since it no longer implies a full scan for every retry. It will keep retrying with different nodes using the existing scan data from up to 10 minutes ago, until it can find one that it can sync with, or it exhausts the addresses database. The cooldown for each address is increased from 3 to 10 minutes, so at any point in time, a node will only hit another specific node only every 10 minutes, at most.

In short, we are separating the scans from the actual sync, and it will just be something that happens every 10 minutes, and no longer a service other parts of the app can call at any time they want. The goal is to make scans a little less chatty and dominant, since scanning every attempts vs every 10 minutes does not appreciably diminish the value of scans (far as preliminary testing shows).

This makes it so that the other parts of the app that relies on this addresses table can now be more aggressive, since they are released from having to update this table themselves.

Lastly, having separate, dedicated slots for scans makes it so that we can give 10x the slot count only for scans, since they are effectively close-to-free to respond to.

Why did it work before, and why it didn't work when it grew?

Because the traffic used for scans grew faster than the traffic used for syncs, since syncs are 1:1, but scans are 1:N. So the scan requests grew to be larger than the network's overall capacity growth via new nodes joining. The changes to rate-limit the scans and give them separate slows brings them back to a more linear growth with the network's overall capacity as new nodes are added.

These changes involve some backend changes, therefore it's going to be a few days, and I'll continue to work on and improve the stability in the coming weeks as my work schedule on the business version allows. I'm writing to shed some light on what I'm doing and what's happening behind the scenes.

You should see see a steady stream of updates coming in - the changelog will carry more details. They'll be focused on improving the scalability of the system as it gets bigger.

Growing pains y'all. Cheers!

0 notes

getaether · 6 years ago

Text

Aether for Redditors

Hey there!

Due to the recent news about Reddit, we've had a few redditors coming to check us out. Which is awesome, so I wanted to write a guide about how Aether compares to Reddit, and what it does similarly, and differently. Likely you'll be fairly comfortable quick, but there are still a few interesting aspects of Aether you might want to keep in mind as you warm up.

We are a small, friendly community, consider this a welcome pack. 🙂

As I hear more and more questions from redditors in the community, this might be updated occasionally.

Aether is a peer-to-peer network

This is the most major, obvious difference. Aether has no servers. It exists ... nowhere, really. As a result, Aether is an app, not a website. It's available for Windows, Mac and Linux, and mobile apps are (eventually) coming.

This has a few implications. When you post on Aether, what happens is that your computer starts to share the content you posted. Other computers will get that content from you, and they will broadcast it to other computers, letting your post spread to the network in a sweeping fashion. As of February 2019, a post takes about ten minutes to reach the whole network.

That means Aether app is an app that needs to keep running in the background, like an email client.

If you post something, and then close the app immediately, that content will not be delivered to other people.

If you want to be sure your post is delivered, wait half an hour or so. Aether just stays on the taskbar / menubar (like Discord), so you can close the window and it'll continue to work in the background. In the future this is going to be visible in the UI when your post has spread to the network. (Like double checkmark from messaging apps)

Aether is ephemeral (like Snapchat - things disappear eventually)

Anything you post on Aether will be gone in about 6 months. This is nice, because no one can stalk your decade's worth of Reddit history and figure out where you sleep.

This is both a philosophical and a practical thing.

It is a philosophical thing, because having information gone vastly improves privacy. It also makes people be able to discuss more freely, without being concerned about whatever they wrote will bite them ten years into the future. We all grow up, and we were all less experienced when we were younger. Aether tries to respect the humanity in that by deleting too-old content.

But it is also a practical thing since it's a peer to peer network, it is limited by the disk space of its participants, so we try to be respectful of that as well.

Unlike Reddit, in Aether, moderator actions are visible to users

First of all, before anything, no one can edit your posts except you. It is cryptographically impossible. You'd think no one would do that, but given the current climate, you'd (sadly) be wrong.

Beyond that, when a moderator takes an action (delete a comment, let's say), that action is visible on the community's mod actions feed. This is a feed of events that mods generate that shows exactly what got deleted, and the reason why.

You can disable any mod, and choose anyone as a mod

In Aether, if you don't like what a mod is doing, you can just disable him. Flip a switch, and everything he deleted reappears. You can also choose a non-mod as a mod.

There is a 'front page' list of communities, called SFW list

These SFWlisted communities are the ones that appear on the front page. This is a limited, curated list of larger communities. You can always create your own community without ever needing to get into this front-page-eligible list of communities if you want. You can also disable this list completely by following the instructions in the app if you want.

Like Reddit, Aether is within the jurisdiction of the United States

That means US law applies — it is not a free-for-all. We have to remove copyrighted content via DMCA, as well as illegal (and those with reasonable chance of being illegal) content.

Aether keeps a copy of the whole network on your machine

This is why it can be so snappy: you can post offline, and when you connect, those posts will be spread to the network. The 'whole of the network' is actually very small, because Aether only carries compressed text. It doesn't carry images, videos, or anything else, so you need to post to Imgur or other image hosts, similar to Reddit.

This also emphasises the importance of no-illegal-content mentioned above. Since we all carry the text of the whole of the network, it's in all our best interest to keep the network clean. It's very hard to make text illegal, however, it's up to all of us to keep it as such. If you see something illegal, use the report button, or send an email with a link to it.

(And yes, there are guards to prevent spammers from creating a million posts and bloating the network size, such as required proofs-of-work.)

Aether is a work-in-progress

Despite the UI, Aether is still very much a work in progress. There are parts of the app that are being worked on, such as elections, being able to add a second mod to a community (Aether communities are denoted as b/Community instead of r/Subreddit) and so on. Things will break, and perhaps repeatedly so. At this point (a month after its release in December) things generally work provided that you have a stable internet connection and can keep the app open appropriately. Nevertheless, this is alpha software. If you have any bugs or feature requests, file them at https://meta.getaether.net. b/Meta is also a good place. (The link requires you have Aether installed)

Like Reddit, you can link to Aether from the web

Here's an example link:

aether://board/86e782e80681ac580b4d6d102b12e787c066e59f194fee57bb0bf83cc1e42fc6

(this links to b/Meta)

Notice aether:// instead of http:// at the beginning. As mentioned above, it needs the recipient to have Aether installed, though. We'll eventually have a preview site on the regular web that can show content without needing it installed, but again, work in progress. 🙂

If you want to post on Reddit or Twitter, and have it be recognised as a link, you can shorten the links at TinyURL, which accepts and shortens Aether links. Or if the place supports Markdown, like Reddit, you can always do:

[my link name](aether://board/86e782...e42fc6)

And make it show up as a link that way.

Aether is paid for by the 'unique' (orange) usernames and its business version

Since the current conversation is around how Reddit is funded, I want to be completely transparent about how Aether makes money (it makes very, very little money) as well. Here's how this works.

a) Similar to Reddit's gold, if you want to support Aether, you can buy a 'unique' username (with a checkmark, like Twitter) that makes you the unique owner of that username for the donation duration. If you want to do so, check out the Patreon.

b) Aether also has an upcoming business version, that allows a company to purchase a private instance of Aether for their own internal use. This comes with a few nice additions, like being able to use email to create threads and posts, and get emails back when other people post. It's good as a productivity, tool, and it's much better than Slack because it interrupts your people less. If you're a tech lead and interested in piloting this with your team, please reach out via email and we'll set you up.

Sounds interesting? Try Aether here. Hope to see you around!

0 notes

getaether · 6 years ago

Text

Aether now available for Windows, Mac and Linux: Free, P2P, privacy-sensitive public communities

For no particular reason other than having run out of things that I can test in a private beta, here's the first public version of Aether 2.

(Recap: What's Aether?)

This is a soft 'launch' — I want to make sure that everything works, and the network load ramps up at a pace that gives me breathing room to fix if something goes wrong.

That said, I've fixed everything I know of so far from the private beta. I've did some QA on Windows and Mac, everything, so far, works. (Knock wood.)

When you join the network, your first bootstrap will take 2-3 minutes. Beyond that, I honestly can't think of something going weird at this point in time. Which is the why for the public version: it needs to be broken in more creative ways. :)

Please do give it a spin, and let me know if something breaks.

Download it here.

0 notes

getaether · 6 years ago

Text

Aether News & Updates - July / August 2018

Hey folks - this is the Aether monthly. This one is pretty exciting with screenshots, and it is probably going to be the last pre-launch update, because it's about around one week to some semblance of code complete. The app is running fine on my machine now, with all features built for the backend and frontend, and most for the client (user interface) side. There's a few screenshots of the current state below. None of those are design mocks, it's the real thing, pulling data from real other nodes. (The data within them is auto-generated for load testing, so there are no users on the system right now.)

Recap: I'm new. What's this thing?

Hey there - welcome in! There was a write-up on that last month, do check it out here.

July updates

Javascript

July, it was mostly about going deep into the Javascript rabbit hole to build the client user interface. (Quick recap: Aether has three parts, client, frontend and backend. Client is UI, frontend is the graph compiler, and the backend is responsible for distribution of underlying entities and communicating with other nodes.)

The tl;dr is that designing and building user interfaces is fun —to me, at least— but doing it in a world where everybody wants you to use Webpack in their own way with slightly different and subtly incompatible configuration is not.

Packing, unpacking and compilers

There is one general rule that is useful in building most things: to do the simplest, dumbest thing that works. I heard it called the Linus' rule, though it is not entirely clear if Torvalds has ever said it. (Nevertheless, the Linux kernel and its success could be considered a manifestation of this idea.)

Go works well for this mindset, and that's why the frontend compiler and the backend is written in it. It's a place where you can actually control the layout of your structs bit by bit in the memory, and it does a great job of getting out of your way, in that the abstraction of Go is fairly thin: you are working with a CPU and a memory that you can put bits into, and there is a small, helpful, and fast garbage collector provided if you want to use it. That means, in Go, you can have as few third party dependencies as possible, and for the things that do not exist in the standard library, you can efficiently build what you want. It also helps that the core language is very small, and there is usually one way to do things. This is very useful when you actually want to read code on Github on how libraries implement a specific functionality. Everybody has to write it mostly the same, so library authors aren't writing what is effectively a completely different language. coughC++cough

With this in mind, I opted for Vue as the base for building the client. This was largely motivated by my desire to avoid Webpack, and this whole seemingly-always-half-broken part of Javascript ecosystem that frequently picks a favourite son every 6 months and kills it with fire shortly afterwards. Considering past experience of projects that had to be maintained that were still on Grunt (remember that?), others still on Gulp, going into this again with Webpack did not sound enticing to the least bit, especially considering that the main selling point of it (single Javascript file) is not very useful in a local Electron environment. To that end, Vue offers an ability to use it through the old-school way, which is to include it as a `````` file. Very simple! Since my use case did not need Webpack, this made Vue a first and only choice. Much later on, and long beyond the point of no return, however, it became obvious that this was a siren song.

The issue was this — Vue considers the ability to split your code into multiple Javascript files (single-page components) an advanced feature. And the moment you hit this, which is early enough that your project is not yet functional, but late enough that you can't really back out, Vue tells you that you have to use Webpack.

This was a huge disappointment, since the majority of the reason for using Vue was to avoid Webpack. It's not that one cannot get it to work, it's just that it does not make sense to integrate with yet another build system, especially in this specific case, one that offers exactly zero benefits. Ostensibly it makes things easier for library developers (since they can offload a decent chunk of 'getting things to work' part of library-building to Webpack, which, promptly, unloads it onto you), but it would have been very nice if Vue had provided a way to use single-file components in a way that does not require a build step, especially considering even Angular-1 in 2014 had a way to do so, in the form of ng-include directive.

(I know that it is not the fastest, since fragments have to be all loaded before the page needs to render, but in this case, the files are already available on the local filesystem, thus it does not matter.)

Nevertheless, it was a good few days spent on getting Vue, Webpack and Typescript working. To be fair, Vue is awesome, Typescript genuinely fixes legitimate problems with Javascript, and Webpack can help under right conditions. It is a good thing that all three exists. It's just that it is a little sad that the Javascript environment is such a high-churn, high-thrash place, and coming from 4-5 months of writing Go code where the language is sound, the libraries, the very few you need, are of excellent quality, and there is only one specification of the language, one gets to realise why the concept of Javascript Fatigue exists.

I do not want to belabour the point. The one last thing that is quite interesting is that the main three Javascript libraries that is used (Vue, Typescript and Webpack) don't even agree on what is syntactically valid Javascript. There are multiple specs of Javascript floating around, and there is Babel, which is a library that compiles new, not-yet-ratified (often, not-yet-finished) Javascript language changes into old, mostly-valid Javascript that you can run. It is a fun tool to test new syntaxes. Unfortunately, it appears that everyone is so sick of old, standard Javascript that it is a dependency on almost anything, so that library authors can write new, shiny Javascript.

Considering the fact that it implements changes that are drafts, and drafts sometimes do change, the tendency to use the newest, cutting edge, unstable Javascript features ends up in a place where the documentation on the Internet is written in Javascript that you don't understand (because it was created literally a week ago), that does not work, that does not compile (because the spec changed two days ago), and most importantly for most intents and purposes, of negative actual value.

If you are a library author, or if you are writing documentation of any form, please consider writing your documentation in good, old, standard Javascript that actually executes without requiring a compiler set up exactly the same way you have on your local machine.

Regardless, it should be obvious where this is going at this point. Fast forward a month of writing Javascript, and there is more than one million lines of Javascript in the node_modules directory, totalling 2+gb of code that there is no actual dependency on, but is there nevertheless due to how node ecosystem and Webpack in general works. Thankfully, the final end result is only a few megabytes of front-end code, since that is the actual code that is needed. The rest is code that Webpack refuses to compile without, but far as one can see, none of it does go into the final end result.

The fundamental goal was to achieve was a client that was simple to write, simple to understand, and simple to maintain. Unfortunately, it appears that this is close to, in the Javascript world, a yet-unsolved problem. There will be some maintenance to do when Webpack goes the way of Grunt and Gulp. Regardless, I'm happy with the client code given the constraints, and it is as simple and maintainable as technically possible. The hope is that it will only improve.

WebAssembly is something to watch, because it does allow one to completely replace Javascript and its dependencies with another language of your choice (which you can do right now), and have it be able to interact with the DOM (which you cannot, and won't be able to for the next few years). Since what matters is the DOM access, whenever that comes, it will be some welcome competition to Javascript and its environment.

August updates

Current status

As of today, the app is about a week to code complete, there are a few pages that need to be designed and implemented (notifications, status, home/popular views), and a few components that need to be finished (post and thread ranking logic based on signals). I'm also getting an Apple Developer Key, so that in OS X, the app will open without scary warnings, and I will be able to push auto-updates in a way similar to Firefox does. (Download update in the background, install on next start, with an ability to disable it).

I will also be putting out a Discourse forum for user support coincidentally, so that there will be community support available.

Integration testing

The expected code complete is the end of this week, but given the prima facie impossibility of giving accurate estimates in building anything of nontrivial complexity (estimating the task takes full knowledge of the task, and in building tech products, the task itself is 90% getting the full knowledge of the task at hand, and only 10% building it), some skepticism is probably warranted.

Regardless, the goal is to have a pre-release executable for OS X (only because I develop in OS X. The goal is to provide Windows, OS X and Linux versions in general availability, same as Aether 1) as a test to see network function and find bottlenecks, if any. It will be followed by a beta release that is generally available, and from that point on, the auto-update system will keep your app up to date.

In short, by the end of August, the network should be transmitting data in some form.

Feature freeze

A feature freeze is a set date where a product stops adding new features before a launch date. This is an useful cutoff, because trying to cram as many features as possible into a launch can mean biting more than you can chew. There are some features that did not make it to the feature freeze deadline of August 1: the feature set that allows for ongoing moderator ballots in communities. These features are code complete for backend and frontend, but they will ship disabled in the first version until the client (UI and product integration) is complete. After the product stabilises and the network is operating correctly under real load, this is what comes next. The expectation is that they will make it into the first non-beta release.

Code freeze

A code freeze is when one stops writing any code and focus on bug fixes, finishing existing features, stability, and QA. The pre-release version will be the public QA test, and if everything goes right, the external QA test will replicate the results of the internal one that I've been running on my own local network. The rough code freeze date is August 15.

Pre-release

After the code freeze, the tasks that remain are those of packaging the app into an executable, updating the website, and setting up a bootstrap node so that newly downloaded nodes will have a point to bootstrap from.

Beta launch

This will happen when the pre-release gives enough confidence that the app is stable, and can handle nontrivial amount of users. Here, the major thing to keep an eye on is not that the app would not work (it does now, and it likely will continue to do so), but it is that the network is slow to deliver every post into every participating node. Since this requires fiddling with network variables (such as how often a node checks for updates in a minute) and is untestable in a synthetic environment, the only real way to know this is to actually have real people using it, and getting the real-world data on dispersion speeds.

From this point on, it gets a little hard to say, because it depends on the reception of the app, and the real world usage. If this app becomes a place that we all can enjoy spending our evenings in, I personally would consider it as success, even if it does not succeed in financially supporting me working full time on it. That said, the ideal path that I have in mind is that this would support me and a small core team (which does not yet exist, it's just me, for now) financially enough that we can devote our full focus to offer a mass communication platform that is free of the catches of what's available today. After all, Aether offers what they won't sell you at any price: your privacy.

Screenshots

Not mocks, actual app! 🙂 (It goes without saying that these are all works-in-progress.)

Addendum: Who's funding you / this work?

I'm currently working on this project full time, and I have been doing it for the past six months. I'm funding it through my own savings. My initial plan is that I want to release the full app, and then consider getting a job, because, well, my savings aren't infinite. That lines up nicely with the end of the majority of the development effort for the release, so for the initial release, I don't foresee any problems. After that, it all depends on how many people actually end up using it. If a lot of you guys do, then we'll find a way to make it work.

My motivation is not that there's money to be made out of this. If you're interested in that, ICOs might be a better bet, but I'd still recommend index funds over anything crypto-related, Vanguard is a good provider for that. I find most ICOs to be fairly distasteful (with some very rare exceptions), because I'd rather have my work speak than empty promises, as opposed to what ICOs typically do, hence my starting to speak about my work just now, when 80%+ of the total work is already done and implemented in cold, hard code.

If you are happy with any of the existing solutions solving similar problems out there, you should consider donating to them. There are a lot of open source creations for different kind of needs. They do often suffer from poor user experience and poor usability though, most of the time, which unfortunately renders them unable to gain adoption. If your project requires the user to compile it before running it, it's fair to say that your audience is not the general public. That is in itself not a bad thing, and if any of those projects actually satisfy your needs, consider donating to them. I'm focusing on better experience for the mass public, to provide all of us with more accountability, more transparency, more privacy, and hopefully some real discussion that we all sorely need — as a human society, we've gotten far too good at stifling each others' voices.

The reason I think I can make a difference is that I'm an ex-Google, ex-Facebok product designer with a lot of experience in the field, and while I have some ideas on what can be improved, I've learned through the long and hard way that the best ideas come from people who use the actual thing. I've also learned a second, almost as crucial other thing: people doing something that they genuinely, honestly care about tend to achieve much better results than a team of highly experienced specialists that are just meh about it, even if the first team is relatively inexperienced. I do have experience, and I do feel this problem every day, to the point that I was willing to quit my job and start working full time on it using my life savings. I'm lucky to be able to do that, but more than luck, I also care enough about this that I wanted to do that.

In summary, I think there's a need for it, and it should exist, in some form of another. Nobody else seems to be doing it, so I'm doing it. This won't make you rich, this definitely won't make me rich, but this has, if done right, has the chance to create an open, simple, free, decentralised, and private way of mass-communicating for the next half century — a public online gathering place that also happens to belong to the public, with no catches, and no kings.

If you think this is worth supporting, you can fund me via Bitcoin at: 1K12GwzAtPWa6W864fkm48pc4VNqzHXG4H, and via Ethereum at: 0x3A4eC55535B41A808080680fa68640122c689B45. Your funding will extend the amount of time I can stay working full time on it. I don't need the funding to finish it (I made sure that I can actually finish it with my own savings), and if you are a student, new graduate, or on a budget, you should not — keep your money, use it for yourself and use your money for good whenever you can. But if you consider yourself relatively well-off and thinking about how your money can do the most good in the world, and you're interested enough that you've read this far, I'd offer this as a pretty good bet.

I'll open a Patreon page, eventually, but I'm focusing on actually writing the code and releasing the app right now. The Patreon benefits I have in mind are mostly cosmetic, such as getting a 'benefactor' username class with some visual distinction, and priority in picking from available unique usernames. If you fund me through Bitcoin or Ethereum, you'll get the same benefits as Patreon supporters.

Lastly, if you're in San Francisco, and you're excited about this, have questions, hit me up, always happy to go for a coffee. Might take a while to schedule considering that I work 14-16 hour days, but we'll probably find a time.

As per usual, feel free to reach out to me with any questions at [email protected].

0 notes

getaether · 7 years ago

Text

Aether News & Updates - June 2018

Hey folks - this is your monthly update, as usual, with your friendly developer hard at work full time on getting this thing to be something that we can all actually use.

There is a short (ha) description of what Aether is below for folks just tuning in. It also includes a bunch of things that are recently implemented, so if you're interested in how it works, it might be interesting to check out. Otherwise, feel free to skip to the June 2018 updates section.

Index

What's this thing?

Privacy-rooted

Anonymous

Ephemeral

Reddit-like

Users controlling moderation

Auditable

June 2018 updates

Who's funding you / this work?

For newcomers - what is this thing you're building?

It's a privacy-rooted, anonymous and ephemeral Reddit-like platform with features that allow for the users to control moderation. It's also decentralised, and based on proof-of-work systems, but without a blockchain involved (it has its own decentralisation technology). Finally, it's open source, both the client and the protocol. The aspiration is that it would become what email is to private communication, to mass communication. Aether is an app that runs on Windows, Mac OS and Linux. It is not a website, though websites that read content from the network and post it to the web can exist.

So there's a chance if you're reading this, you've understood this perfectly, and in which case you should probably just skim over the next part. But if you're not into these kind of things all that much, or you want more details, here's what the above paragraph means, word by word.

"Privacy-rooted"

When you create content on Aether, your IP address is not recorded. The reason this is possible is that "posting" on Aether is in fact just contributing the content into a pool that is shared and copied over between devices participating in the network in a random fashion. (Don't worry, all nodes have a full copy of all data, it's just that the way it reaches them is random.) That means when you get a piece of data from a remote node (a remote computer running the same app), you don't know if that node created that content or not, since very likely that node is just remitting information. Even if it was somehow created from that device, a node can have zero to thousands of people using it as their communication provider, just like an email server. So as the recipient of the information provided, you have no idea where it come from, in terms of the physical IP address.

That said, the messages are sometimes signed, which means if I'm using the name 'Bob', that means I can opt to have a key that I can sign my messages as Bob. So not knowing where a message is originated does not actually mean that you can't identify people. You can do that if the creator of the message wants it.

In effect, what this means is, Aether is privacy-protecting by default, and it gives out no information that you haven't explicitly chosen to provide to other computers participating in the network. However, you have the option to become as private, or as public as you'd like. The choice is yours, and yours only.

As an aside, the savvy reader will point out that it is I, the developer saying this, and the developer is the weak point in the system, in that it has the privilege to change the system to a less private one. This is a really good, and often overlooked point, and this is why Aether is designed in this specific way from the beginning: privacy is not an add-in feature that I've worked specifically to provide an extra icing on the cake, it is a side effect of a series of architectural choices, and without fundamentally reconstructing the app it is impossible to change this. In the case that happens, well, the beauty of the open source is that you can fork it away.

In more familiar terms, this is the difference between having a house made of glass, but your neighbours and you promising not to look in each others' houses, and your friendly local supermarket promising to not fly drones* over you to see which brand of orange juice you're consuming this morning, versus, having a house of brick and concrete. I can trust my neighbour (and you should, and that's great! Society wouldn't work without some baseline trust), but I'm still going to go with good ol' brick-and-concrete. You need trust and walls.

And if a contractor comes in overnight, and replaces your house with one made of glass, you're probably going to notice.

Aether also supports using it over Tor proxy (already implemented, in fact), for additional privacy. However, it's important to understand that given enough resources, any privacy protection can be bypassed. The intended users of Aether are normal people who are privacy conscious. If the use case you have in mind is more sensitive, you should be using tools specifically designed for those requirements, like Signal, or SecureDrop, both of which are security audited.

* Model planes technically don't count as drones. They'll fly those.

"Anonymous"

This was briefly mentioned above, but to explain it further, Aether is by default anonymous, or more specifically, pseudonymous. If you post anonymously (i.e. without any username), those posts cannot be updated (since the nodes receiving the update can't know the post and the update came from the same entity) and with some other limitations arising from that fact. Generally speaking, most people will have a username (a public key) that they control so that people can know if the messages are coming from the same entity, but that key is not tied to any real world person or entity unless the user himself or herself gives out that information on his or her own volition in a post.

"Ephemeral"

Anything that's posted on Aether will eventually disappear. This disappearance is usually around 6 month mark. This means, effectively, that the content will be fading out on a rolling basis, which makes Aether useful for certain things, and not so useful for others. Why? Well, quick answer is, if you are a Reddit / Twitter user and you open your profile accidentally, you go 'oh god', you know exactly what I'm talking about.

The longer answer is this. The internet is permanent. Human memory is not. This means the more you participate in all the discussion on the internet, the more permanent footprints you're leaving all around, each of which may be brought in front of you any time in the future, without context, with the worst possible interpretation, and used as a proof that you're a horrible, horrible person. In some ways, being permanent is good, we can hold politicians accountable to their past tweets for example. But is it true, really? If you think about it, the people who are going to back off on their promises in their tweets, if they're halfway smart, will delete the tweets themselves first before reneging on their promises, before the media can take a dive into their twitter profile triggered by their backtracking, and they will find nothing. Effectively, the one and only user 'benefit' of keeping a forever history is only actually a benefit if the politician in question is incompetent.* Modifiable history (by deletion) is actually worse than having no history at all in that it gives a false sense of trust: we have no way of discerning a history that was unedited from one that is. Given no evidence on the contrary, most people will assume the absence of a post as proof that it never existed.**

So there's actually not that much point in keeping a history if it can be modified in terms of accountability. There are some benefits in keeping it even when modified as a record of human discussion, and that has an intrinsic value for research and archival. Unfortunately the side effect is that the participants in these discussions end up exposed in perpetuity, if any of them end up becoming a person of public importance later, even with a wholly different set of beliefs that he expressed two decades ago, he can still be found, and held to those views. Yes, the idea of a fully coherent human being over large periods of time is an illusion, but the understanding that it is so has not reached public consciousness yet, since we've only just achieved being able to retain communication records spanning decades for ordinary people. Which means, you will get held to your views and posts from decades ago until people learn that is usually not a fruitful conversation. Unfortunately, considering that this was common even in the days of the Roman Empire 2100 years ago***, we haven't learned much. If one has to choose between relying on the benevolence of unknown readers from the future in interpreting the context of today, versus just making things disappear over time by default unless someone decides to preserve it, the sane choice to teach machines to forget.

Forgetfulness is not a bug, it is a core feature of being a human, so that we can move on, forgive, and have compassion. Our minds cull the history as time goes on, the further past you go, the fewer things you remember, and those tend to be important things worth saving. Aether works the same way. The things that keep being referenced will stay. The things that don't will disappear. The things that some people think worth saving will stay. The things that nobody takes action to save to save will disappear.

Ephemerality doesn't mean things will disappear tomorrow, it means they'll disappear, eventually, unless it keeps being referenced, or unless somebody quite literally exports it out and saves in a disk somewhere. So this is not about recordless discussion. If you're a person of importance, 6 months is plenty of time for journalists to decide to start saving your history. What it gives you is that at some point in the future you become a person of importance, no one can go back more than 6 months and hunt back your tweets from forever ago. (But they will be able to start from that point on, and keep recording, and they should, because that is how press makes people accountable.)****

In the immortal words of the Nintendo quit screen, "All that is not saved will be lost."

A happy side effect of this is that the content data in Aether can occupy a fixed amount of space in memory, unlike a blockchain, which by default will grow without any upper bound. If you give Aether 10gb of space to work with, it will use that space, and when it approaches the limit, it will automatically start to delete old, stale data in a way that it will always stay within the limit.

* The fact that most are incompetent can be seen somewhat as a negation of this argument, but it balances out, because it makes the competent ones all the more dangerous.

** This is also why you get that hollow feeling when you scroll down a public profile and see something that used to be there not there — it challenges your perception of reality. A nonexistent post that you remember existing doesn't say that it was there and now it is not, it says that it was never there, and you're either hallucinating, lying, or misremembering it ever being there. For a feed that purports to be a written history, it being able to be modified at will is just not very nice, in that any changes to a written history that claims to be authoritive implies that the problem is with you if your own human memory doesn't match the 'authoritive' record. This is arguably worse than having no history at all.

*** They even had a term for it, argumentum ad hominem, which we still use.

**** What if someone comes in and decides to just, you know, record everything, just in case? Well, that's why not having anybody's IP addresses and personal data is important. If an app has your real-world name and is ephemeral, then everybody can record everything and you'd gain nothing from ephemerality. If an app has your nickname and no personal data, but not ephemeral, then you yourself will eventually give out enough pieces of information that your nickname can be fairly easily mapped. Each of these pieces of information are usually not very specific on their own (which is why you feel comfortable sharing them separately), but the combination of a few non-specific things can be surprisingly specific, which is why looking at your permanent history on a service can be a little weird. In the case where the app is both ephemeral and pseudonymous, so long as the app doesn't know IP addresses, there is no easy way to trace a nickname to an actual person, and the hardness of this problem exponentially increases as the records fade further and further into history. If you want even more privacy to improve on this, you can change your public key occasionally to break the historical chain, so even an adversary that collects the full history of everything (which would be a challenge, since the size would be massive) cannot follow further.

"Reddit-like"

This is admittedly a little bit of a shorthand for a 'threaded message board'. There are many examples of it, Slashdot is actually one of the oldest, and Reddit the most popular - they are largely the same. This is a good format for discussion because it allows people to direct their responses to the specific people, not into a larger 'roll' of time-ordered responses like a classic forum would do.

It also allows for changing the order based on the community perception, so if something is voted as useful, that can be used to concentrate the useful stuff in a thread to the top, so that the 'value' would be most dense in the places the reader will most likely see.

If I'd have to describe it based on existing platforms, the closest I can get, ignoring 90% of what makes Aether unique, would be something like Snapchat(Reddit+Twitter). Kind of.

This is the basic structure, but Aether does have a few more things that nobody else does. It does have personal profiles (to the extent that you actually want to have one, which you don't have to), and you can post to your own profile without actually posting into any specific board or thread. Other people can comment on your profile posts, and they're just like any other thread in any board, these personal profile posts are also threaded, voted on, etc.

You can also choose to only allow certain people to post on your personal profile.

For boards, an admin can choose to make the board only accepting contributions from a certain number of people (though everyone can always read). This is designed to protect communities that suffer from harassment from the outside, like LGBT communities, or those that hold political views outside the mainstream. There is a limit to the number of people you can whitelist for posting, though. The reason is, if you've reached this size threshold as a community, you are now grown and mature enough to start accepting people from opposing viewpoints joining your discussion. You can remain small and limit contributions exclusively to your supporters, or you can grow and become a public community that is accepting of opposing viewpoints.

This sounds like an arbitrary limitation, but is actually a pretty fundamental effect of how a distributed network operates. Because the content you create consumes resources from all participants in the network to be broadcast, moved around and made available, the nodes are incentivised to do that only if they gain some benefit from it. If you could grow to become a major part of the data transmitted over the network, and still remain and exclusive community, the people who cannot post in your board would still be carrying your messages for no benefit to themselves. Eventually, they would get pissed off enough, and they would modify their app in a way that would not communicate the messages from boards that do not allow them to post in, and the network would get fragmented. Your messages would not go through. Everyone would lose. Having a threshold* is a reasonable compromise between everybody being able to post in everything (thus making less sizeable communities vulnerable from the start, before they can take roots and start benefitting from discussion), and every community being exclusive, in which nobody would have any benefit in carrying data that belongs to other communities, which would fragment the network into infinitely small shards that negate any benefits from having a global community. There is a benefit in seeing opposing viewpoints alongside the ones you support, at least occasionally, so that you do not inadvertently construct yourself a mighty fine Truman Show.

* This threshold is around about 1000 users, that is, a whitelist can only hold up to 1000 people. If you want to grow beyond that, you'll need to remove the whitelist. (But you can still moderate the community as per usual.) Needless to say, this value is a best guess and should / will be adjusted based on community feedback.

"Users controlling moderation"

Users can elect moderators, impeach them, or just enable, disable them just for themselves. So if somebody is unhappy with some moderators, s/he can just disable the ones that he doesn't think are doing a good job, and choose new ones. This makes it so that the people who participate in the communities actually have a say in how they are governed (if they want to do so). All moderator actions are auditable and reversible by the local user, so if there's censorship, you can flip a switch and you can see all posts the mod has deleted, and their reasons. If you don't like what you see, you can disable the mod locally and impeach publicly as needed.

The users can vote on moderators publicly on whether they want them to moderate, or not moderate. These votes are always ongoing, and if a moderator falls below a certain threshold of popularity, the moderator loses moderation privileges.

Likewise, an ordinary user can also be promoted to a moderator based on people in the community voting for him in the same way.

Moderators can be elected for boards, and for the whole network. For a moderator to be elected, at least 5% of the total population of the scope in question has to have voted, and more than 50% of those votes should be positive. For a vote to be counted valid, the user has to have been actively posting in that scope at least once in the last two weeks. If a user ceases to be active, his or her vote is discounted from the active votes. A vote lasts for 6 months, and then expires. This means the election can always be ongoing, and the desires of the current population of the board will be the effective force in bringing change, not past decisions made by users that stopped contributing a long time ago.

When a user casts a vote for some person to be a moderator, the person the user votes for immediately becomes a moderator for that user, so that the effects of voting for someone to be a mod is immediate for that person. That means you can actually choose your own mods as you like. Same goes for disqualifying somebody from being a mod. If you vote negatively on an existing mod, that mod will no longer be a mod for you, and the edits that mod made will be reverted from your system retroactively.

But to know which mod to choose and which mod to impeach, the users need to know what mods do, and for which reasons. Which brings to our last keyword, auditability.

"Auditable"

This means you get the raw mod deletion signals from the whole network, and it is actually compiled on your local device into something that you can use. That means, when a post is deleted, what it means is that you have the post, and the deletion signal, and if the source of the deletion signal is a mod for you, the signal is applied, otherwise it has no effect.

This means you can actually go to the appropriate section of a board, and you can see all the deletions that has taken place in a board, and see which mod is it from, with an optional 'reason' field. If you find deletions that you find to be in poor judgment, or those that you consider censorship, you can revoke the mod privileges of that mod for your own device by casting a disqualify vote on that mod. If enough people vote the same way as you, the network will decide collectively (at 50%+ disqualify and 5%+ participation) that a given mod is no longer trustworthy, and even the people who has never voted for that mod will default to not trusting that mod.

This auditability is what makes mod election system work. Had this not be present, moderators would be able to control the discussion to such an extent that they could suppress all talk of dissatisfaction and generate the impression of a happy populace to newcomers, which is indeed what happens in quite a few public forums on the internet today. Without free, unfettered access to information and discussion, democracy does not work.

Thankfully, using the system is much simpler than explaining it. You choose which mods you want as you go, but with sane defaults that are decided by the network as a starting point. As you choose your mods, those decisions you make are considered signals for other people so that the network can decide on a better set of default mods.

The reason behind making mods electable is actually fairly pedestrian, and illustrated well with The Lord of The Rings, of which two word summary would be something fairly close to: "Power, corrupts." The people who explicitly want to be in positions of power are often those least fit to be there. That alone is bad enough. To make any one person or a small group of people perpetually in power, well, pretty much the entirety of human history is a good showcase on why that's a bad idea to the point that I'll avoid listing individual examples here.

There is an intellectual argument that can be made for an absolute monarchy, which a group of people have been making for the past few years. It appears that the main thesis is thus: a ruler that is above all vulnerability to his person, who is completely outside the system, is the only person who can work on and improve a system as a whole, as an unincentivised incentiviser.

That sounds fine on paper. The problem is, though, the best way to test this would be to move to an absolute monarchy, say, North Korea, and not many people appear to be doing that. This kind of highlights the problem: wanting a thing for other people, or even for a society as a whole, but maybe not for your own self. My hopefully obvious point being, people act a lot more carefully when their own skin is in the game.

This is why when you vote for a moderator, that person immediately becomes a moderator for you, not when they win the election. If you're voting for somebody, you yourself have to live under that person's moderation. This blunts the power of rhetoric from people who actually do want to get elected for power, because every vote from every person means that vote becomes immediately effective for that voter at that instant in time, not when in an undefined future where the election might complete. The voter can immediately see what that vote would bring, for good or bad, which means even voting for somebody to make that person a mod is auditable.

On the other side of things is surfacing people who actually don't really want to be moderators but happen to be good at moderating and making them mods, since they tend to be the best mods so long as their interest in that board remains in place. The way this works is that if you find someone whose moderation history looks good* and vote for them to make them a mod for yourself, that is a public vote for that person to be a mod for that context. If enough people do that because they think the edits by that person saves them time and grief by making the board a more pleasant place to be, the person ends up an elected moderator for that board with no need for ever being a candidate, running a campaign, or even caring for the fact that s/he is a moderator for that board.

Both of these effects are achieved by pairing the voting process with immediate activation / deactivation as a mod, and this is why elections system is implemented in this specific way that is effective immediately on an ongoing, rolling basis.

* A moderation history is available for everyone and everyone can moderate their own view. Voting for someone to be your mod fundamentally just means that their edits to their own view for their own comfort will be replicated in your view too. If they delete something, and you trust them as a mod, your view will also remove it as well. That's all being a mod means.

June 2018 Updates

Aether is composed of three parts: a backend, a frontend and a client. The backend is the part that communicates with the network, (This is referred as a node above.) The frontend is the part that compiles this raw network data into human-readable boards, handles elections, and so on. Lastly, the client is the visual interface that allows an user to read, interact, and create content in Aether.

As of now, the backend is complete, and the frontend is nearing completion. The user interface will likely take a few weeks to design and build whenever the frontend is complete. The launch date I have in mind is August 1, 2018, though you should take all launch date claims from all projects with a huge grain of salt, and this one is no exception.

I'm currently working on finishing the elections system, and I am almost done with it.

This system of three major blocks has several advantages. Principally, the system's three levels are isolated from each other. This means working on one layer is much easier than what it would be otherwise.

For user interface changes and surface-level improvements like better user experience, features that affect the local user, and those which pertain to the local machine, changing the client is very easy, since it is written in Javascript (Electron and Vue). This is the most accessible level.

For the features that change how the network is compiled to a human-readable collection of boards, changing the frontend, written in Go would be the best place. This is useful, because anything in the frontend is encapsulated and running locally, which means you can safely make changes there that affect only your local machine. An example for that would be to add a new type of vote that does a certain thing. This changes the interpretation of the network data, but not the data that traverses the network.

Frontends connect to backends, and multiple frontends can connect to a single backend. A backend is a node that connects to other nodes, distributing Aether's data blocks. These nodes share data across each other and make sure that each of them have a recent copy of the data in the network. Your backend does not know anything about your frontend, so even if your backend were to be compromised somehow, it would not have any information about you. This means you can run a backend as a public service and let a number of people use it as their backends.

Backend is arguably the hardest part to change, since it determines how your node reacts to the network. That said, it does have a lot of configuration options, so it is likely that what you want is achievable through configuring it, rather than changing the code within it.

I expect most people to not change anything, since by default Aether should be a foolproof, simple and pleasant experience in all three platforms, Windows, Mac OS and Linux.

The default application package comes with the client, the frontend and the backend working as a single app that collapses to the menu bar / tray to keep running as long as your computer is running, so you can keep up to date with the network.

After the launch for all three OSes, depending on the feedback, I'll be setting time for fixing the initial round of bugs reported by the users (I want to say "if any", but I know that's way too optimistic).

Afterwards, my goal will be building the mobile applications. These applications are a little trickier, because they don't have the bandwidth to run a full Aether node, so they have to be 'light' nodes composed of only clients and frontends.

Finally, as always, all of this code and platform is available under an OSI-approved open source license at github.com/nehbit/aether.

Who's funding you / this work?

I'm currently working on this project full time, and I have been doing it for the past four months. I'm funding it through my own savings. My initial plan is that I want to release the full app around August, and then consider getting a job, because, well, my savings aren't infinite. That lines up nicely with the end of the majority of the development effort for the release, so for the initial release, I don't foresee any problems. After that, it all depends on how many people actually end up using it. If a lot of you guys do, then we'll find a way to make it work.

So, in summary, I think there's a need for it, and it should exist, in some form of another. Nobody else seems to be doing it, so I'm doing it. This won't make you rich, this definitely won't make me rich, but this has, if done right, has the chance to create an open, simple, free, decentralised, and private way of mass-communicating for the next half century — a public online gathering place that also happens to belong to the public, with no catches, and no kings.

This is all I have for this month. I'll be working pretty crazy hours this next 6-7 weeks to make this get out the door on time. But in the meanwhile, I'll be speaking about Aether in Our Networks decentralisation conference in Toronto, Canada, in mid-July. If you're in Toronto then, come say hi!

Same thing with San Francisco, actually. If you're excited about this, have questions, hit me up, happy to go for a coffee. Might take a while to schedule considering that I work 14-16 hour days (as exemplified by the fact that I'm posting this at 4am) but we'll probably find a time.

As per usual, feel free to reach out to me with any questions at [email protected].

0 notes

getaether · 7 years ago

Text

Aether News & Updates - April / May 2018

Time for another update! I've been busy working on Aether for a few months now. This means a lot of changes, and the codebase has grown quite a bit. The good news is the backend is close to complete, which means I can move into the frontend in the next few weeks and start showing actual UI and start collecting critique.

As always, the code is pushed weekly here, and this is the list of commits.

Launch timeline

The idea is that we would be able to launch in July or August. This means there's a few months left for the frontend, and I want to be able to be completely done (well, 95% done) with the backend by the end of May.

Currently working on...

The main thing I'm working on right now is to interpret the results from petri-dish testing to the point that the network is stable as possible at the outset. Petri dish testing is a framework that we've built that runs multiple Aether nodes in the same machine under an isolated environment, which makes it so that we can see real node behaviour over time and analyse the results of it. This gives us metrics like time-to-network-consensus, and we can optimise these metrics to make the network overall the fastest and the most stable it can be before launch.

Tech roadmap

Note: This part is mostly for people who are interested in development / engineering aspects of Aether. If you're just interested in the product, you're more then welcome to skip.

These are the items that are currently on my plate.

Create the 'bouncer' library. This is our rate-limiter so that the Aether nodes running won't break when they're bombarded with requests, they'll just start to refuse them after a certain threshold. This value is user-settable.

Implement Initial Block Download IBD is the process where a node that is new to the network bootstraps itself across multiple remote nodes, not just one. This is important because it makes the entry speed much faster.

Violations counting This is the system that temporarily blocks misbehaving nodes, so that their actions are ignored until the misbehaviour is corrected.

Event horizon & deletions This system is the one that allows Aether to be ephemeral. This deletes too-old content based on rules set by the program and modified by the user.

Frontend This is the layer between the backend and the client. This layer caches requests and compiles the raw data in the database into forums, threads, posts and people. At first, this will be mostly passthrough, in the future (before release) it will start to cache the results of the computation and maintain these caches to render user latency to be zero.

Client The client is an Electron app using TypeScript and Vue. This includes the design work also.

I do have some more smaller items, but they're going to take only a week maybe, in total.

If you have any questions, feel free to post a GitHub issue, or email me at [email protected]. Cheers!

0 notes

getaether · 7 years ago

Text

Aether News & Updates - March 2018

Hey folks! Since the past month we’ve been working on Aether full time. Here’s what happened, what’s coming and what our tentative release date will be for Aether 2. But first, an intermission of sorts…

What the heck is Aether?

If you haven’t been following it yet, Aether is an app that gives you community moderated, community governed[1] public spaces, similar to subreddits. You subscribe to the topics you’re interested in, and it gives you back interesting links and discussion around those. Major difference compared to Reddit, though, is that Aether is transparent, auditable and extremely privacy-oriented[2], all the while it attemps to be a more pleasant place hospitable for civilised discussion amongst peers. Aether’s boards are owned by nobody, and they live in nowhere,[3] thus they cannot be censored (but can be reversibly hidden). It also gives you ability to host free websites in this distributed network.

It is not a social network, though. Or, rather, it’s an antisocial network — we attempt to be a place expressly made for following and discussing things you’re interested in, not for meaningless chatter of your third cousin’s husband’s new car stereo.

Spiritually, Aether strives to be the mass communication equivalent of email. Similar to email, app and the the core protocol underneath is open source, and no permission is needed to implement third party clients.[4] If you’re familiar with it, Aether is functionally quite similar to Usenet of days of yore, brought to 21st century. If not, think Reddit, but open, more sane, smarter, gentler, more transparent.

What’s the overall status?

We’re currently building the sync product underneath Aether to be as extensible as possible. For the consumer-ready app, the estimated arrival is about three months of work, so a tentative launch date would be June 2018. If you’d like to keep in touch with the updates and for launch invite, you can subscribe to the mailing list here.

Tech roadmap

Note: This part is mostly for people who are interested in development / engineering aspects of Aether. If you’re just interested in the product, you’re more then welcome to skip.

The expected completion of the backend[5] is about a month away. After this is done, it will be the frontend and the client that is next.[6]. The work in the backend is the more novel and creative part of Aether in terms of engineering, the frontend and client are fairly common app types with existing best practices. Aether 1’s frontend experienced no significant bugs nor performance problems, therefore our past experience is also useful — and should make it a reasonably fast endeavour.

This is a list of items that is on the plate for implementation as of now, for the backend.

Implement the initial block download algorithm that triggers when a node is offline for a long enough time that it loses track of the network head, or when a node is joining the network for the first time. This algorithm distributed the load across multiple nodes so as not to overtax one single peer that provides the bulk of the send bandwidth.

Implement the petri dish reporting to automate the testing for time-to-consensus (TTC). TTC is how fast a group of nodes reach to consensus, and it’s denoted as percentage ratio divided by seconds.

For example, a 55 second TTC at 50% means that it takes 55 seconds to reach consensus across N number of nodes that differ in their content by 50%, excluding the time spent in sync process.

Think through and decide on whether private realms are a possibility for the initial launch. Private realms are encrypted, members-only boards and other services only for their members, and they traverse only between their members, their data nor connections do not mix with the public network. They effectively are separate, split universes.

Implement the minor libraries, such as moving the main transport to HTTPs only, hardening the outbound server, and reference counting for data deletion.

Consider a separate class of bootstrap nodes which are community members that self-nominate themselves to be available for getting new nodes up to speed.

There are also some other more minor to-do items, but they do not merit the discussion here as they are of mostly mechanical nature.

How can I help?

First of all, if you have considered, thanks for the support! We will shortly be starting to host community events in San Francisco, California, USA. If you’re there, you should email us for an invite.

The best thing you can do to help, if you choose to do so, is to spread the word to people that might be interested in such a thing, so that when the alpha-testing start in June, we will have enough of a critical mass to bootstrap a community of like-minded people. We realise that the technical product is one thing, but the much harder challenge is to have content from like minded people that is interesting, so that you return to it day after day to enjoy it. It’s all about the humans behind the wires in the end.

Bonus

Here’s a video of Aether backend achieving consensus! (You probably need to be an engineer to share the excitement, though)

youtube

Footnotes

[1] Aether’s boards have moderators, but those moderators are subject to an ongoing election. Moderators’ actions (including deleting posts) are auditable by everyone, and if a mod falls too much in popularity as indicated by community voting process, their modship is revoked.

[2] We do not mean strong privacy in the sense that “we won’t look at your data [asterisk] [asterisk] [asterisk] followed by legalese that makes the app own all data and your firstborn for any purpose in perpetuity” We mean strong privacy in the sense that “we can mathematically prove that your data never leaves your computer”.

[3] Aether is a peer-to-peer, distributed network. You might have heard of this term in regard to crypto-currencies, and yes, Aether does borrow some aspects of the blockchains. It is different, though, in the way that it uses a new type of blockchain that does not require distributed consensus. That makes it much faster (100x-1000x faster) than existing blockchain implementations.

[4] The app Aether is open source (AGPLv3), as well as the underlying peer-to-peer communication protocol (Mim). Source code can be found here.

[5] The backend is actually reasonably complete, in that it does its job just fine under petri-dish testing. The work that is left is stress testing to root out bugs, and adding some new features that is going to be useful in the future.

[6] Aether is composed of three parts: a backend, a frontend and a client. Here’s what it means.

Backends are nodes that communicate with other nodes in a peer-to-peer fashion. They form the backbone of the information traversal across the network. A backend does not retain any user information, and it can be a private (baked into your app, which is the default) or a public entity (i.e. it can be made available on a public park, similar to a community-access Wifi network.) A backend can also serve multiple users. A backend does not care or understand what it transmits, so long as it is valid Mim blocks. For example, an Aether thread is a thread entity and a few posts linked in a fashion that indicates who replied to whom, amongst other things. The backend does not follow the links between these blocks, or nor does it understand that this is a thread. It just passes the entities within that thread according to the rules that are set in the protocol. A backend is effectively a distributed database with a novel sync engine and load balancer that is particularly optimised for the type of data that it carries through.

Our current testing reports that a regular laptop or desktop computer running the backend can ingest data equivalent to a month’s worth of Reddit’s content in around 14 minutes from the network, given no bandwidth constraints.

Frontends are the apps that connect to these backends, and they compile their own understanding of the world based on the blocks received by the backend. This is where a bunch of Mim blocks become a linked, compiled “thread”, that humans can understand. Frontends fetch the data from backends and then save into their own databases, in a compiled form.

p>Client is the actual, user-visible application that the front-end drives. This is the user interface, and the commands issued by this client hits the frontend. Clients are OS X, Windows and Linux, as well as mobile devices. Desktop and laptop computers are full nodes, which means they run backends, frontends, and clients in tandem. Mobile devices are partial nodes, which means that they run only the frontends and clients, and they need to connect to either a public backend provided by the community, or one provided by default for mobile devices. So long as they connect to a backend that is tracking the network head, it does not matter which backend they connect to.

0 notes