all lasting work is done in files/data (can be parsed permissionlessly, still useful if partially corrupted), but economic incentives keep pushing us to keep things in code (brittle, dies basically when one of maintainer|buildtools|hardware substrate dies).
when standards emerge (forcing code to accept/emit data) that is worth so much to a civilization. a developer ecosystem tipping the incentive scales such that companies like the Googl/Msft/OpenAI/Anthropics of the world WANT to contribute/participate in data standards rather than keep things proprietary is one of the most powerful levers we as a developer community collectively hold.
(At the same time we shoudl also watch out for companies extending/embracing/extinguishing standards... although honestly outside of Chrome I struggle to think of a truly successful example)
There was never any such incentive. MS Office formats were undocumented for years because MS had no incentive to document them. Merely using files did not help at all. Actually the vast majority of all file formats have never been open. Think about all the custom file formats used by video games, for instance.
I actually talk about this in the article. Merely using files is how we got out of the dependency on MS Office. Multiple efforts reverse-engineered them, including Google Docs. Yes they were undocumented, but as long as stuff has to be stored on the disk under user’s control, the overall dynamics are very different from you-can’t-see-the-files systems.
Is it really so different in the service context? You could also reverse engineer the HTTP endpoints and formats used by Word Online to export data from the service. It doesn't feel all that different, except perhaps that the online service can try to detect your custom tooling and block you, whereas with static data that isn't possible.
But other than that admittedly real extra problem, the bulk of the work would still be understanding the undocumented protocols and feature semantics, then matching the feature set.
Hmm, Word Online isn't what the article is about. I'm not sure if you've read the article, but it is about social apps — like Tumblr, Reddit, HN itself, etc.
I'm using file formats as a metaphor to explain how apps built on AT protocol work (lexicons are like "social file formats"), and what this way of building enables (interoperability between social apps by default).
I did read it. I think myself and other people are getting tripped up by the title and the filesystem-oriented argument. The article opens with an icon of a .doc file, so it's natural to think about MS Word, which these days is a social app via its collaboration features.
I see. Well, my point was to start with that to give an intuition of the many-to-many between apps and data, and then transfer this intuition to social aggregated apps (which is what AT enables).
I think it's on open social apps to show that they're actually meaningfully better products, and that is possible because they're open. With luck, this may lead to an ecosystem where it's worth staying compatible and interoperable, and where users scoff if someone is trying to break it, and where users have an easy way to walk away. I know this sounds super idealistic but this did essentially happen with open source over a long time. At some point, people were just as skeptical of open source as we might be about open social.
That are yet to become monetised. It's all fun and games until Bluesky announced how users and developers will pay for all this and what happens with your "social file system" when you stop paying.
I mean, I can literally already self-host my personal data if I want to. And there are also already forks of Bluesky (not just the client, but the server and the database) that can participate without fragmenting the network. It is not a perfect system but it's so far from where you are when you just rely on a closed app.
That's great, but also Mastodon is just there and has been for quite some time. I see no added value in Bluesky/ATProto beyond the layer of that "social as a service" which looks like a walled garden / app store of sorts in the making. I may be wrong, of course...
I never managed to get into mastodon because mastodon isn't a single space. It's a bunch of separate, linked spaces. Which server you're on matters a lot, and I never found one that really had the vibe I was looking for.
Bluesky, though, is one big pool. There's no seam between servers. The experience is much nicer.
Also, bluesky caught on among my peers in a way that mastodon never did, which was always going to be the deciding factor. I imagine not having to "choose a server" was a big part of that.
This is false, if we're to believe it's possible to exist outside the moderation control of Bluesky corp.
It's also false today while everyone is still on the same "instance" - e.g. ICE joining Bluesky and being verified is making waves... apparently being _that_ is not enough to violate Big Blue's community guidelines.
If you run a separate appview, bluesky can't censor your interactions in any way, and you can seamlessly view posts from both bluesky users and users banned on bluesky.
Of course, running an appview that indexes the whole world is expensive, and running an appview that doesn't index the whole world is less convenient than just using bluesky's appview, so the only people running an independent appview right now are blacksky.community. It isn't stable yet, but it works, and there are active users who are banned on bluesky: https://staging.blacksky.community/profile/did:plc:63hvnyjvq....
I'm optimistic that running independent appviews (especially appviews that index content lazily, on demand) will get easier in future.
Mastodon is just a bunch of isolated copies of the same app talking to each other. There is no notion of a shared identity, each server's admin is effectively a king over your account, etc. It's a fragmented patchwork of isolated sites that forward messages.
With AT, there is just one global network. Like the web. You don't post "to" someone's isolated copy of an app. You post to your own folder, and every interested app can aggregate your post.
It's a bit like email vs RSS. Very different shapes.
To give you a concrete example to ground it. Blacksky is a fork of Bluesky. They're setting up their own server with different moderation policies and are unbanning some people that Bluesky has banned (https://bsky.app/profile/rude1.blacksky.team/post/3mcozwdhjo...). However, Blacksky posts still exist in the same "world" as Bluesky--they are not an isolated fragment.
Thanks to AT, Blacksky and Bluesky are just two different prisms through which you see a single network. Whereas with Mastodon, every app is its own network, with some limited message passing between them.
I understand the technology of course. In fact, it's clear ATProto is an instantiation of data spaces, a tech developed in the EU for industry purposes. However, while in industry there is pre-existing need for interoperable "data product marketplaces", it doesn't translate to social media in the same context.
There is no pre-existing need or desire for people to put all their data into a single space - that part is entirely driven by social companies themselves which of course benefit from that kind of centralisation and interoperability. So yes, the prism is that our social interactions become a data product and more - a data product "under governance", well structured and with well defined schema, etc.
For end users, functionally, there is very little to be gained (and some things to be let go) but the technology is not accessible enough for non-tech folks to decide on the tradeoff themselves.
As for Mastodon, what you call limited currently includes editing posts, long form posts and private blocklists (among others), all features Bluesky lacks and current version of atproto does not allow. So let's not try to quantify/compare them like that.
Regardless, I like the idea overall, I believe the Solid project (https://solidproject.org/) addresses a similar concept but when less dependence on a single authority as is the case of Bluesky and ATProto.
> a developer ecosystem tipping the incentive scales such that companies like the Googl/Msft/OpenAI/Anthropics of the world WANT to contribute/participate
I think Apache Arrow has achieved exactly that [1]. It's also very file-friendly, in that Arrow IPC files are self describing, zero-copy, and capable of representing almost any data structure.
> At the same time we shoudl also watch out for companies extending/embracing/extinguishing standards
Is ATProto actually a standard? But regardless, nothing prevents Bluesky from enschitifying.
I’m somewhat concerned that the “file system” or the storage where all of our things are supposed to be stored is now suddenly in the cloud. We actually have a real file system … it backs itself up on iPhone even.
It feels like the entire pitch is based on some FOMO factor “oh but my posts” - do people really care that much about their short form outbursts? I mean the whole point of twitter was to post and forget but maybe not for everybody.
>I’m somewhat concerned that the “file system” or the storage where all of our things are supposed to be stored is now suddenly in the cloud. We actually have a real file system … it backs itself up on iPhone even.
I don’t know how this is related to what my article talks about. The data we’re taking about is already public and in the cloud, it’s just locked into specific app’s databases.
The point isn’t about some single specific app, but about creating an ecosystem where the things you create (StackOverflow posts, Reddit discussions, Instagram follows, Letterboxd recipes) are not tied to the fate of that particular app and the whims of its creators. Then, if a service shuts down or turns to shit, developers can compete without solving the cold start problem. You just need to make a better app for same data, not convince everyone to recreate everything on your app.
I think that's an overly charitable take. Giving Google/MSFT/OpenAI/Anthropic what they want does not guarantee a return on dividends. Standards are nice, but Apple is a giant testament to the fact that all the standards in the world won't move an adequately entrenched business.
POSSE and AT Protocol can be understood as interoperable marketplaces. Platforms like Reddit and Instagram already function this way: the product is user content, the payment is attention, and the platform’s cut is ads or behavioral data. Dan argues that this structure is not inevitable. If social data is treated as something people own and store themselves, applications stop being the owners of social graphs and become interfaces that read from user-controlled data instead.
I am working on a similar model for commerce. Sellers deploy their own commerce logic such as orders, carts, and payments as a hosted service they control, and marketplaces integrate directly with seller APIs rather than hosting sellers. This removes platform overhead, lowers fees, and shifts ownership back to the people creating value, turning marketplaces into interoperable discovery layers instead of gatekeepers.
not sure if you understood the article, isn't the whole point to own your data as "it's just a filesystem". Reddit, Instagram, etc. are the total opposite.
Yes, I should have been clearer. Reddit and Instagram do not operate this way today, but open social alternatives to them could. The idea is that people create personal websites where posting, commenting, and other social actions live, and that becomes the filesystem they own.
Open social networks would simply index or pull from those sites using agreed-upon lexicons and protocols. Existing platforms could either adopt the open social model, or, more realistically in the short term, be treated as syndication targets where posts are pushed via their APIs when someone publishes on their own site.
Yes I'm loving the growth in this space. I'd also throw Standard.site[0] into the mix, it seems to be the lexicons Leaflet and some other apps are using.
This article goes into a lot of detail, more than is really needed to get the point across. Much of that could have been moved to an appendix? But it's a great metaphor. Someone should write a user-friendly file browser for PDS's so you can see it for yourself.
I'll add that, like a web server that's just serving up static files, a Bluesky PDS is a public filesystem. Furthermore it's designed to be replicated, like a Git repo. Replicating the data is an inherent part of how Bluesky works. Replication is out of your control. On the bright side, it's an automatic backup.
So, much like with a public git repo, you should be comfortable with the fact that anything you put there is public and will get indexed. Random people could find it in a search. Inevitably, AI will train on it. I believe you can delete stuff from your own PDS but it's effectively on your permanent record. That's just part of the deal.
So, try not to put anything there that you'll regret. The best you could do is pick an alias not associated with your real name and try to use good opsec, but that's perilous.
My goal with writing is generally to move things out of my head in the shape that they existed in my head. If it's useful but too long, I trust other people to pick what they find valuable, riff on it, and so on.
>Someone should write a user-friendly file browser for PDS's so you can see it for yourself.
You can use pdsfs[0] to mount a user's pds locally using FUSE read-only. It's mentioned in the blogpost. I remember seeing a tool posted for mounting them read-write, if you sign in, but can't remember where to find it.
yes, that is the one everyone is using as a guiding light for where we think Bsky is headed for their private data needs, which is group-shared / not e2ee because key mgmt is an unsolved problem at scale. They need to support creators with subscriber only content. E2EE will emerge from the messaging space first, i.e. Signal / MLS like
Bsky is likely to put out a short-term gap filler this year for personal-private data, which kind of already exists today for Bsky only. This work would make that same feature in the per-user repo available to all apps. Right now it's hard coded to a specific NSID and single db row
I've always thought walled gardens are the effect of consumer preferences, not the cause.
The effect of the internet (everything open to everyone) was to create smaller pockets around a specific idea or culture. Just like you have group chats with different people, thats what IG and Snap are. Segmentation all the way down.
I am so happy that my IG posts arent available on my HN or that my IG posts arent being easily cross posted to a service I dont want to use like truth social. If you want it to be open, just post it to the web.
I think I don't really understand the benefit of data portability in the situation. It feels like in crypto when people said I want to use my Pokemon in game item in Counterstrike (or any game) like, how and why would that even be valuable without the context? Same with a Snap post on HN or a HN post on some yet-to-be-created service.
>I am so happy that my IG posts arent available on my HN or that my IG posts arent being easily cross posted to a service I dont want to use like truth social.
ATProto apps don't automatically work like this and don't support all types of "files" by default. The app's creator has to built support for a specific "file type". My app https://anisota.net supports both Bluesky "files" and Leaflet "files", so my users can see Bluesky posts, Leaflet posts, and Anisota posts. But this is because I've designed it that way.
Anyone can make a frontend that displays the contents of users PDSs.
I also have a little side project called Aturi that helps provide "universal links" so that you can open ATProto-based content on the client/frontend of your choice: https://aturi.to/anisota.net
Except that a lot of the app builders in ATProto seem to think the protocol was designed to make their lives easier when bootstrapping their network from Bluesky userbase.
(Imo, that is a perverse interpretation, it's about user choice, which they are effectively taking away from me by auto importing and writing to my Bsky graph)
re: the debates on reusing follows from Bluesky in other apps instead of their own
I agree. I don't understand the driving force here.
I have all of the raw image files that I've uploaded to Instagram. I can screenshot or download the versions that I created in their editor. Likewise for any text I've published anywhere. I prefer this arrangement, where I have the raw data in my personal filesystem and I (to an extent) choose which projections of it are published where on the internet. An IG follow or HN upvote has zero value to me outside of that platform. I don't feel like I want this stuff aggregated in weird ways that I don't know about.
For me, part of it is that we have no power collectively against products turning their back on users because coordination to "export data all at once and then import it into specific other place" is near-impossible. So this creates a perverse cycle where once you capture enough of the market, competition has very little chance unless they change the category entirely.
What AT enables is forking products with their data and users. So, if some product is going down a bad road, a motivated team can fork it with existing content, and you can just start using the new thing while staying interoperable with the old thing. I think this makes the landscape a lot more competitive. I wrote about this in detail in https://overreacted.io/open-social/#closed-social which is another longread but specifically gets into this problem.
I hear you re: not wanting "weird aggregation", that just be a matter of taste. I kind of feel like if I'm posting something on the internet, I might as well have it on the open web as aggregatable by other apps.
Thanks for your thoughts. I do feel like this is all very specifically connected to Twitter. Which tech people really adopted, but I never used much, so it is interesting that these different perspectives are somewhat tied to one's megaplatform(s) of choice. I don't know of another social network that has inspired such a consistent effort to be forked or cloned. I do kind of feel like "change the category" and create a new network with the traits you want to see is the right move.
For better or worse, Twitter built their network. That many people willingly signed up and posted and continue to post there. I don't think anyone really should be able to fork it, because the users didn't collectively agree to that, and they don't all agree on what a good road or a bad road is. Ultimately, they can choose to leave if and when they want. Are these networks rather sticky, yes of course, but that's life.
We've seen lots of social networks come and go, things do change over time, there's ample opportunity for new ideas to flourish. In that sense, AT is perfectly welcome to throw their hat in the ring and see if that resonates and sticks. If people want their social network to be forkable, that concept will succeed.
I do think it misses what a lot of people find valuable about the tangibility and constraints of "I am making this content specifically for this platform and this audience at this point in time." I don't think most people think of their social media posts as a body of work that they want to maintain and carry over in an abstract sense independent of platform, and give open license to anyone to cook up into whatever form they can dream of.
Just the other week, another service that people actively used called Bento announced shutdown: https://bento.me/. This sucks for the user.
Someone created an alternative called Blento (https://blento.app/) on AT. Of course, by itself, this doesn't mean they'll be successful. But the thing is that, if Blento shuts down, someone can put it right back up because (1) it's open source, and (2) the data is outside Blento. Any new app can kickstart with that data and get people's sites back up and running. And two platforms can even compete on top of the same data.
I agree content is tailed to the platform and resurrecting something doesn't necessarily makes sense. But that's the point of lexicons. You get the choice of what makes sense to resurrect (actively moving to an alternative) vs what doesn't (something with a style that doesn't work elsewhere) vs new recontextualizations we haven't even tried or thought of. I think it's early to dismiss before trying.
Adding more to this, what's wild to me is that Blento can already show public data from other AT apps on the homepage. For example, your teal.fm plays. It's just one of the block types: https://bsky.app/profile/flo-bit.dev/post/3mcqx5pwrrk26. Despite teal.fm not really even "existing"!
It feels like file formats and protocols is the wrong prism through which to view this. What users actually want is the ability to move as much data as possible from point A to point B, whilst remapping it as much as possible to meet the different features and identities of point B. A unified data format is actually a hindrance for meeting that need because it boils it all down to a lowest common denominator. A simple data pump utility that just copies data back and forth, with lots of special case logic like "X has a character limit of 5000, Y has a character limit of 500 so I'll truncate and add a link to the original post on X", is going to work far better for this use case.
More generally I feel like you kinda missed the point of filing systems in your first writeup. The moment you say "posts don't have natural names", ok, that's a serious problem that should put a halt to the entire concept. A file is nothing but a natural name for a byte array and a little bit of metadata. That's why we think of file managers and explorers when we think of what files are all about: lists of names. It's also why people directly use files for relatively few things - they were a big deal in the early days of personal computing because the killer apps for it were direct translation of physical office tasks. It's mostly but not entirely possible to come up with natural names for office documents (the gap is in suffixes like "draft final v3__real final draft.docx"). That's why we use analogies for 1950s paperful office objects like files and manila folders to begin with. But most computing tasks even back then weren't oriented around documents, they were oriented around database records and the user interface was just screens, or possibly direct SQL. It's a remarkable gap in our infrastructure that there is no agreed on file format for a SQL result set. Maybe Arrow is that format.
>A unified data format is actually a hindrance for meeting that need because it boils it all down to a lowest common denominator.
Exactly, which is what I'm saying in the article:
We could try to put every app developer in the same room until they all agree on a perfect lexicon for a post. That would be an interesting use of everyone’s time.
(That's meant to be slightly sarcastic.) Then the article says:
For some use cases, like cross-site syndication, a standard-ish jointly governed lexicon makes sense. For other cases, you really want the app to be in charge. It’s actually good that different products can disagree about what a post is! Different products, different vibes. We’d want to support that, not to fight it.
The point isn't having a lowest common denominator format. It's to enable precisely the granularity of formats that app developers want. The default one is "each app has its own formats". But what changes is that
1. Other apps can still read/write in other apps' formats. This makes the landscape competitive: if some app is going down a bad road, it's easy to fork their product with all the data already in there.
2. It is possible for multiple app developers to use the same format where it makes sense. As linked from the article, https://standard.site/ is a good example of that.
>More generally I feel like you kinda missed the point of filing systems in your first writeup.
I'll admit that I used the metaphor to highlight a specific thing, and names in particular don't play a big role in that thing (what plays a big role is links). The reason I picked filesystem as a metaphor is because it highlights how file formats mediate interactions between apps. This is exactly how lexicons mediate interactions between social apps. That felt like the backbone of the argument.
It's true this isn't describing a filesystem exactly. Maybe a web of identity-addressed hyperlinked JSON is a more precise way to say it. But then nobody knows what that means. I'm happy with my choice of metaphor.
> I think I don't really understand the benefit of data portability in the situation.
Twitter was my home on the web for almost 15 years when it got taken over by a ... - well you know the story. At the time I wished I could have taken my identity, my posts, my likes, and my entire social graph over to a compatible app that was run by decent people. Instead, I had to start completely new. But with ATProto, you can do exactly that - someone else can just fork the entire app, and you can keep your identity, your posts, your likes, your social graph. It all just transfers over, as long as the other app is using the same ATProto lexicon (so it's basically the same kind of app).
But what if your entire social graph didn't choose to transfer over as well? What if they don't want to be on that app? What if someone that was very indecent made a compatible app? Would you want your entire Twitter history represented on there?
For better or worse, I don't think it makes sense to decentralize social. The network of each platform is inherently imbued with the characteristics and culture of that platform.
And I feel like Twitter is the anomalous poster child for this entire line of thinking. Pour one out, let it go, move on, but I don't think creating generalized standards for social media data is the answer. I don't want 7 competing Twitter-like clones for different political ideologies that all replicate each others' data with different opt-in/opt-out semantics. That sounds like hell.
The framing of "portability" is a bit confusing. Your data is not actually "transferring" anywhere, it's always in your PDS. These other apps and clients are just frontends that are displaying the data that is in your PDS. The data is public and open, though private data is in the works and hopefully will arrive in 2026.
The data is not transferring, but the user is. When I sign up for e.g. Twitter, I don't want to sign up for Mastodon, or Bluesky, or Truth Social, or whatever other platform someone might create later. Thus I would not choose to put my data in a PDS. I feel like that would actually leave me with less ownership and control than I have now.
My point is that I don't believe the separation of frontend and data is desirable for a social network. I want to know that I am on a specific platform that gives me some degree of control and guarantee (to the extent that I trust that platform) over how my data is represented. I don't really have to worry that it's showing up in any number of other places that I didn't sign up for (technically I do since everything public can be scraped of course, but in practice there are safeguards that go out the window when you explicitly create something like a PDS).
Do I understand correctly that your main concern is that some random service would serve a page when asked for /your-handle, and so, given a link, someone may assume that you actively use that service? Just trying to understand the exact scenario.
Generally it's good practice for AT apps to treat you as not signed up if you have not explicitly signed up to this app before. So I think a well-behaved site you never used shouldn't show your profile page on request. If this is right, maybe it needs to be more emphasized in docs, templates, etc. Same as they should correctly handle lifecycle events like account getting deactivated.
Realistically it's possible some services don't do this well but it's not clear to me that this is going to be a major problem. Like if it gets really bad, it seems like either this will get fixed or people will adjust expectations about how profile pages work.
I think part of the issue is that humans are hyper conditioned to expect a certain UX and set of conditions thanks to the past 3 decades of the internet + legacy social media. It only feels weird that you could publish content on Bluesky and have it show up on some other app without your consent because of how we've been conditioned. There will be a lot of unconditioning and reconditioning that has to take place over a span of time if the ATmosphere (or any new vision of social media) wants to succeed.
https://anisota.net will display any Bluesky content and profiles without the consent of the user — no one cares at the moment though because either 1) they don't know about such a niche project, or 2) they aren't concerned cause I'm not a controversial figure.
If Truth Social was suddenly a part of the ATmosphere or a part of some other wide network of users, most people would catch on eventually and be hopefully conditioned to realize that the mere presence of someone's content on an app/site doesn't mean they use that app/site
FWIW, I think Anisota is a bit different because conceptually people see it as a Bluesky client. So it is expected that you're "projecting" Bluesky, for better or worse. Whereas if it's some fanart exchange service or something, maybe it makes less sense. Maybe it just depends on what you think the user would expect.
It's kind of just like, this sense of the "safety" of the walled garden (which one might argue the overlords could rugpull at any moment, which kind of happened with Twitter, but that's not something that really concerns me personally enough to trade it for the complexity of decentralization).
A lot of it is presentation. Instagram is the one that I use the most right now, and I feel like they really nailed this in the "everybody can make content" era. When I'm creating content for IG, I know exactly how it's going to look, there's editing and previewing tools, there's a confidence that people are going to receive it in the manner I intended and expected, and it's tied to the culture of that platform that I've become attuned to over time. The thought of a JSON of my IG post existing in a clearinghouse that encourages people to remix it into whatever, just kind of gives me the ick. Even if realistically it wouldn't really be abused, I don't feel compelled to choose that. If someone comes up with a new app that does some things better, I can choose to migrate to that, but likely I wouldn't want to just one-click import my IG history. It will be different in some way and I would want to make new content for it.
I guess I just kind of see these platforms as bespoke communities that people choose to be a part of, more than fungible data frontends. It's very personal, and...social. It's not the abstract world of data that programmers live in. My social graph in each app is unique and I don't really want it abstracted to one monolith that follows me everywhere. Each one is a new world with a fresh start.
Now, the IG post ecosystem is a lot more complex than it is with Twitter. With Twitter being at its core just text, it lends itself better to different views, but still, formatting matters and there are really a lot of rendering decisions to be made in a modern Twitter client. And again it just seems weird to me to have this oddly fragmented community with unavoidably complex interop semantics rather than everyone just being under the same roof. User opt-in aside, they will all develop their own features and quirks and they and the users will have to navigate how those translate or don't. Twitter clients are/were at least unified by the Twitter API. Regarding your initial question - it's more than just the profile page, do my comments show up in conversations? Likes and follower counts? Or are there weird gaps and discrepancies everywhere from users that haven't opted in? What if one fork requires mutual "connections" and one uses one-way followers? Each platform will make their own decisions. It just sounds like such a mess, and people will spend so much time disambiguating (oh no I'm on this one so I see it that way...).
There were some efforts to fork Reddit but it has survived as a great example of how lots of different people with different interests and opinions can occupy different parts of the same platform, even if lots of them complain about it because it's not their perfect vision of what it should be (see programming languages, and pretty much everything that reaches a certain userbase).
This sounds like I need to host my PDS. Easy for me with no public profile but if I was someone famous wouldn't that mean I needed enterprise class hosting?
You don't need to host your own PDS for any of this to work. It works the same way regardless of who hosts your PDS.
I think what may be confusing you is that Bluesky (the company) acts in two different roles. There's hosting (PDS) and there's an app (bsky.app). You can think of these conceptually as two different services or companies.
Yes, when you sign up on Bluesky, you do get "Bluesky hosting" (PDS). But hosting doesn't know anything about apps. It's more like a Git repo under the hood.
Different apps (Bluesky app is one of them) can then aggregate data from your hosting (wherever it is) and show different projections of it.
Finally, no, if you're famous, you don't need enterprise hosting. Hosting a PDS can be extremely cheap (like $1/mo maybe)? PDS doesn't get traffic spikes on viral content because it's amortized by the app (which serves from its DB).
But.... why ? Is this assuming a huge portion of the people you interact with on Twitter are also all moving identities to the new platform ?
I can see for something like artistic expression it being useful to export and and move over to a new platform (like moving a whole photography portfolio, short story writing samples, or similar). But who cares about your "likes" on a totally new and separate social media platform ?
It makes the landscape competitive. There isn't really a notion of "moving" to a "platform". It's more like if Twitter sucks, some team can spin out an alternative that has the existing content but takes different product decisions. And then some people can try this alternative and use it without leaving the existing network. So, it allows more experimentation in the market without having to solve the cold start problem. The lifecycle of products becomes more fluid. It's easy to spin something up, and you can also shut things down without permanently killing them.
I would not call the people who ran Twitter before Elon Musk bought the company and took it private "decent people". I think that Musk purchasing the company and running it in a way that a lot of previous userbase objected to was ultimately a huge boon for software freedom - because without that, the large number of people who stopped using Twitter and went to the ATProto ecosystem instead would have been happy to continue using completely-proprietary Twitter. A lot of people were suddenly and viscerally faced with the downsides of building a digital "home" on someone else's platform.
>The effect of the internet (everything open to everyone) was to create smaller pockets around a specific idea or culture. Just like you have group chats with different people, thats what IG and Snap are. Segmentation all the way down.
I actually agree with that. See from the post:
>For some use cases, like cross-site syndication, a standard-ish jointly governed lexicon makes sense. For other cases, you really want the app to be in charge. It’s actually good that different products can disagree about what a post is! Different products, different vibes. We’d want to support that, not to fight it.
AT doesn't make posts from one app appear in all apps by default, or anything like that. It just makes it possible for products to interoperate where that makes sense. It is up to whoever's designing the products to decide which data from the network to show. E.g. HN would have no reason to show Instagram posts. However, if I'm making my own aggregator app, I might want to process HN stuff together with Reddit stuff. AT gives me that ability.
To give you a concrete example where this makes sense. Leaflet (https://leaflet.pub/) is a macroblogging platform, but it ingests Bluesky posts to keep track of quotes from the Leaflets on the network, and display those quotes in a Leaflet's sidebar. This didn't require Leaflet and Bluesky to collaborate, it's just naturally possible.
Another reason to support this is that it allows products to be "forked" when someone is motivated enough. Since data is on the open network, nothing is stopping from a product fork from being perfectly interoperable with the original network (meaning it both sees "original" data and can contribute to it). So the fork doesn't have to solve the "convince everyone to move" problem, it just needs to be good enough to be worth running and growing organically. This makes the space much more competitive. To give an example, Blacksky is a fork of Bluesky that takes different moderation decisions (https://bsky.app/profile/rude1.blacksky.team/post/3mcozwdhjo...) but remains interoperable with the network.
That also seems to be a risk with most centralized social networks these days as well.
Our LinkedIn profiles have been scanned by scrapers, Instagram photos used for facial recognition training, Twitter / FB posts replicated in a bunch of downstream places, archives, and search indexes. Not to mention data being sold to third parties.
At this point, I assume if I post it publicly online, it is no longer in my control where it ends up downstream and I assume it will be online somewhere forever.
Back in the old twitter days you could have screenshots of your tweets being displayed by a number of different 'dubious' accounts on instagram, completely out of your control.
Posts feel less valuable as files as they are of the moment. I don't need this comment as a docx, thank you all the same.
So I agree.
Maybe building it that way opens possibilities I can't see?
But for functional apps it seems useful to have a file you can download and use somewhere else. Images and movies are a great example. I can record a video in Loom and chuck it on Youtube. Youtube can't lock me on that video. it is my file. Imagine if that was not the case. It would suck.
It was great. Anyone could host web pages, anyone could access and read anyone’s web pages with a tool called a web browser, of which there used to be several compatible implementations.
My hope is that in 5 years, I will not have anything in my feeds that have not been signed in a way that I can assign a trust level.
Here in the Nordics, we are already seeing messaging apps such as [hudd] that require government issued ID to sign in. I want this to spread to everything from podcasts and old-school journalism to the soccer-club newsletter, so that I can always connect a piece of information back to a responsible source.
So you're simply not interested in reading any random website by random people who don't see a benefit of establishing any form of trust, especially if should not be connected to their official government IDs?
Or to put it differently: Where should this come from, and which issuer would you trust? And why should anyone else agree with you that this is good?
When I browse to random site, I can see in my browser that A's first level contact trusts this site.
Now I can make a decision based on the amount of trust I have on A. Maybe after exploring the site I can mark specific pages or the whole domain as trusted, so people in my network can see the same.
On a larger level I might trust the Country of Finland, who will only mark their official sites as Trusted. This way I instantly know if I'm on an official site or something pretending to be one.
Trust is subjective! Let's establish trust in each other rather than rely on one-size-fits-all solutions.
Personally, I trust my friends, family, and some public figures and institutions to varying degrees. I want to see social experiences that reflect that.
I’ve been reading “The Unix Programming Environment”. It’s made me realize how much can be accomplished with a few basic tools and files (mostly plain text). I want to spend some time thinking of what a modern equivalent would look like. For example, what would Slack look like if it was file (and text) oriented and UNIXy? Well, UNIX had a primitive live chat in the form of live inter-user messaging. I’d love to see a move back to simpler systems that composed well.
Unix gave the world a lot of good ideas about architecture, but I think it really hamstrung itself by treating all the data as plain text and resisting the idea of having any kind of structured, formatted data passed within a pipeline. It's nice to be able to serialize to something human-readable and -editable, but constantly re-parsing and re-formatting it becomes a real pain.
I've been thinking of this for some time, conceptually, but perhaps from a more fundamental angle. I think the idea of "files" is pretty dated and can be thrown out. Treat everything as data blobs (inspired by PerKeep[0]) addressed by their hashes and many of the issues described in the article just aren't even a thing. If it really makes sense, or for compatibility sake, relevant blobs can be exposed through a filesystem abstraction.
Also, users don't really want apps. What users want are capabilities. So not Bluesky, or YouTube for example, but the capability to easily share a life update with interested parties, or the capability to access yoga tutorial videos. The primary issue with apps is that they bundle capabilities, but many times particular combinations of capabilities are desired, which would do well to be wired together.
Something in particular that's been popping up fairly often for me is I'm in a messaging app, and I'd like to lookup certain words in some of the messages, then perhaps share something relevant from it. Currently I have to copy those words over to a browser app for that lookup, then copy content and/or URL and return to the messaging app to share. What I'd really love is the capability to do lookups in the same window that I'm chatting with others. Like it'd be awesome if I could embed browser controls alongside the message bubbles with the lookup material, and optionally make some of those controls directly accessible to the other part(y|ies), which may even potentially lead to some kind of adhoc content collaboration as they make their own updates.
It's time to break down all these barriers that keep us from creating personalized workflows on demand. Both at the intra-device level where apps dominate, and at the inter-device level where API'd services do.
I'm using filesystem more as a metaphor than literally.
I picked this metaphor because "apps" are many-to-many to "file formats". I found "file format" to be a very powerful analogy for lexicons so I kind of built everything else in the explanation around that.
The repository data structure is content-addressed (a Merkle-tree), and every mutation of repository contents (eg, addition, removal, and updates to records) results in a new commit data hash value (CID). Commits are cryptographically signed, with rotatable signing keys, which allows recursive validation of content as a whole or in part. Repositories and their contents are canonically stored in binary DAG-CBOR format, as a graph of data objects referencing each other by content hash (CID Links). Large binary blobs are not stored directly in repositories, though they are referenced by hash (CID).
Re: apps, I'd say AT is actually post-app to some extent because Lexicons aren't 1:1 to apps. You can share Lexicons between apps and I totally can see a future where the boundaries are blurring and it's something closer to what you're describing.
But you don’t need the AT protocol and lexicons to describe semantics. Ontologies have been a thing for decades, and search engines (the “rich results”) and ActivityPub use them.
Why use the lexicons you described, instead of the more supported https://www.w3.org/ns/activitystreams ontology that ActivityPub uses, for example?
If you allow me to, I would be interested in re-writing your article but by using ActivityPub concepts instead. Maybe I will learn about a fatal flaw in one of the protocols during that process.
It's bold to say users don't want apps when the expressed preferences are so strongly for apps that "App Store" is a well known brand.
The word app is short for application, which in this context is just a synonym for capability. People have tried breaking apps apart into more fine grained capabilities before and it never worked (OpenDoc and OLE being two well known examples). Brand names matter, they have value to people. YouTube isn't an abstract collection of decomposable capabilities, it's a brand that summarizes numerous unarticulated pieces of information, like its commenting culture, what kind of content is allowed and what isn't, etc.
> I'd like to lookup certain words in some of the messages, then perhaps share something relevant from it.
This is already a built-in feature on macOS. If I right click in your message "Look up PerKeep" is literally the first command, and if I click it I get a nice info window telling me what it is that I can copy/paste from. There is nothing to solve here.
Marketing and culture can do wonders to one's preference.
> synonym for capability
Yes and no. An "app" is usually a set of capabilities, not a single capability. Many of the core *nix utilities for example are specifically designed to have a single capability each, and these are then composed for complex requirements. But for the most part today's apps contain multiple, usually overlapping capabilities. For example Microsoft Word offers not only text editing and formatting, but also has a dictionary+thesaurus, spelling+grammar check, mail merge and lots more. Of course, you can abstract this all away as "word processor capability", but there are other word processing apps out there with varying capabilities, and so things become problematic fast especially in terms of interoperability.
> OpenDoc and OLE
These "never worked" due to people/policy, not due to tech ability. OpenDoc essentially stagnated at Apple due to conflicts and lack of clarity, and then was outright killed by Steve Jobs[0]. OLE(tm) hasn't failed per se, but it's proprietary to Microsoft[1] and thus doesn't have much adoption outside Windows.
> Brand names matter
All they really do is put a label on a bundle - or bundles - of capabilities to create handy associations. YouTube is associated primarily with accessing videos that can be made and uploaded by anyone, secondarily allows commenting on and liking said videos, etc. Again, marketing is a significant factor.
> a built-in feature
You seem to be missing the forest for a tree here. That's a single example, of infinite permutations. Also, even given that example you're still missing parts of the flow.
FWIW the Look Up feature, whilst built in to macOS, can also be implemented using "services" which are a way to plug in to other apps and system features like text rendering and highlighting. I can also select text in your message, right click, services -> "Add to music as spoken track". That's a plugin offered by an app. It's only one extra click.
My perspective on why OpenDoc/OLE failed are different to yours. The model in which apps host targeted plugins which don't affect data formats proved very successful, the model in which an app is nothing but plugins wasn't. And if you want to talk about marketing and culture, well, OpenDoc and OLE both benefited from corporate marketing. Using OLE was part of Microsoft's "Designed for Windows 95" programme, which came with a badge and marketing dollars attached. This concept was heavily pushed at the time. OLE had more success than OpenDoc but only temporarily and only because it was a less pure implementation of the concept, closer to fancy plugins than truly decomposing apps into components.
I think the reason they didn't take off was a mix of technological problems and user demand problems. It wasn't people/policy related. Quite the opposite: both technologies lived on long after the real problems had become apparent.
Programming for both OpenDoc and OLE was extraordinarily complicated. The developer experience was atrocious. Making your app OLE embeddable required mastering not only COM and Win32 programming but dozens of other poorly documented and flaky APIs, like for compound documents. Code generation was rampant, everything was memory unsafe. I never did OpenDoc apps but I heard it was a very similar experience.
Then there was the user experience: if you embedded an object from an app and sent the resulting document to someone else who didn't have the app, or had an old version of the app, then the document would just have a hole in it. Useless. OLE tried to do a kind of weird mind-meld between totally different app GUIs, meaning that depending on where you clicked in a document the hosting app could look totally different. There were dozens of problems like this.
In the end it was all rendered irrelevant by the web. On the web, if you share a document with someone it always appeared for them correctly (or if it doesn't, it's a minor and quickly fixable bug). Reliable sharing mattered more than anything else and so all the non-web platforms started dying out.
Yes that's a pattern offered and used on all modern OSes to various degrees, but most - particularly independent - developers prefer to bundle the capabilities they're offering in such a way that they're only available via their app's interface. I think partly because it's just easier to implement that way, but also because it's in their interest (not the users') to not allow the capabilities they've implemented to be accessed by apps created by others.
> perspective on why OpenDoc/OLE failed
It's not my perspective at all; that's what's in the Wikipedia articles, which I linked. OLE had more success because Microsoft was far more invested in it, since it was theirs, and is still used today in Windows (sounds like you're saying it's dead); OpenDoc was an attempt at an open standard, and failed because the participants just weren't invested and/or agreeable enough. Of course I can see Jobs driving the final nail because, for him, it wasn't exactly in Apple's best interest to create apps that interoperated with apps outside the Apple ecosystem. All those are people/policy things.
> rendered irrelevant by the web
I don't really get what you're saying here, in the context of capability provision. But I'd say no. There are many web platforms offering capabilities via APIs, but still many restrict access to their own apps, with Reddit being one of the latest to drastically restrict 3rd party access.
OLE has been dead for decades, yes. When was the last time a new app launched and being embeddable in Office documents via OLE was a key advertised feature? I don't know what even happens if you open a .docx with an embedded OLE object in Word Online.
Obviously the APIs are still there. You can use them if you want. It's just that nobody is writing pure/native win32 apps anymore, outside of browsers and legacy codebases. So, new code isn't being written as OLE controls.
I don't think Jobs killed OpenDoc because of some vision of an ecosystem of Apple's apps. At the time he made that decision Apple was on death's door because it had a culture of architecture astronauts spending years in front of whiteboards, writing features that didn't directly appeal to what the market wanted. He killed many projects at that time for the same reasons, like their line of printers.
If OpenDoc/OLE had been genuinely the right path we'd have seen people outside of MS/Apple try to reinvent them. Plenty of companies have the resources, as does the open source community. In fact KDE and GNOME both did try with KParts and Bonobo. But, both of those are dead too.
To your wider point, yes, it's often the case that what the app developers need/want is in conflict with what the users need want. I don't see that as a problem. Trade always involves tradeoffs, when users and developers trade there is always an implicit negotiation in the background. Sometimes that negotiation falls more on user freedoms and sometimes users choose to give up those freedoms to get other things they want. That's just trade, it's been true since before humans had writing. I don't think there's anything unique or special in computing in that regard.
Because something isn't widely adopted doesn't mean that it's dead. Microsoft continues to use and support OLE today, and there's a non-zero number of non-Microsoft-affiliated persons out there using it (I've seen forums with users asking how to do X regarding it).
KParts too is very much alive, as it's a core part of KDE Frameworks, which is used to build Plasma. And I know for a fact it works very well because I'm a Kubuntu user (11 years).
I'm skeptical of these kind of like, self-describing data models. Like, I generally like at proto--because I like IPFS--but I think the whole "just add a lexicon for your service and bickety bam, clients appear" is a leap too far.
For example, gaze upon dev.ocbwoy3.crack.defs [0] and dev.ocbwoy3.crack.alterego [1]. If you wanted to construct a UI around these, realistically you're gonna need to know wtf you're building (it's a twitter/bluesky clone); there simply isn't enough information in the lexicons to do a good job. And the argument can't be "hey you published a lexicon and now people can assume your data validates", because validation isn't done on write, it's done on read. So like, there really is no difference between this and like, looking up the docs on the data format and building a client. There are no additional guarantees.
Maybe there's an argument for moving towards some kind of standardization, but... do we really need that? Like are we plagued by dozens of slightly incompatible scrobbling data models? Even if we are, isn't this the job of like, an NPM library and not a globally replicated database?
Anyway, I appreciate that, facially, at proto is trying to address lock in. That's not easy, and I like their solution. But I don't think that's anywhere near the biggest problem Twitter had. Just scanning the Bluesky subreddit, there's still problems like too much US politics and too many dick pics. It's good to know that some things just never change I guess.
Not sure I fully get you... In your example, isn't the problem that nobody cares about this data? So there is no motivation to build a client. Whereas if these were beloved notes or minisites or whatever that got wiped out by the latest acquisition (e.g. see https://bento.me/ shutting down), people would know exactly what those are, and there would be incentive for someone to compete for the userbase.
E.g. Blento (https://blento.app/) is atproto Bento that I only saw a couple of days ago. But the cool thing is that if it shuts down, not only someone else can set it up again (it's open source), but they're also gonna be able to render all of the users' existing content. I think that's a meaningful step forward for this use case.
Yes, there's gonna be tons of stuff on the network that's too niche, but then there's no harm in it either. Whereas wherever there is enough interest, someone can step in and provide the code for the data.
I'm saying it would be a lot simpler to just provide an API to download the data. I don't think it's worth the complexity and network of PDS' to preserve this data in perpetuity, but even if I did, nothing would stop me from using the download API and storing tapes in Iron Mountain or whatever.
I think there's also weird cases where you don't really want things replicated: revenge porn, snuff, CSAM, etc. Like, if someone we're like "yes it's true we lose all data forever if we shut down, but we're not indiscriminately pushing some of the worst content to everyone replicating" I'd take that tradeoff immediately.
Is there anything stopping me from backdating my own records? Since the createdAt is just an arbitrary field, I can just write whatever I want in there, right? Is there a way for the viewing application to verify when the record was created (and not modified since), maybe by looking at the mentioned signing?
You can indeed backdate records. Since the application knows when it has first seen (i.e. indexed) your record, it can decide what to do with that information. If there's a difference, the Bluesky app, for example, shows its own indexing time, but also shows a separate panel saying the post is backdated to some other date. Other apps could choose to show something else.
It is possible to create links asserting a specific version by making a "strong ref" which includes content hash.
AT Proto seems very overengineered. We already have websites with RSS feeds, which more or less covers the publishing end in a way far more distributed and reliable than what AT offers. Then all you need is a kind of indexer to provide people with notifications and discovery and you're done. But I suppose you can't sell that to shareholders because real decentralised technology probably isn't going to turn as much of a profit as a Twitter knockoff with a vague decentralised vibe to it that most users don't understand or care about.
Why so much cynicism? The people working there genuinely care about this stuff. Maybe you disagree with technical decisions but why start by projecting your fantasies about their motivations?
RSS is OK for what it does, but it isn't realtime, isn't signed, and doesn't support arbitrary structured data. Whereas AT is signed, works with any application-defined data structures, and lets you aggregate over millions of users in real time with subsecond end-to-end latency.
RSS is signed because it's served over HTTPS which involves an SSL certificate exchange. It is also real time. You fetch the feed and get the most up to date content on it. You can host any XML document you want on your website and there are W3C recommended standards that allow the integration of rich data structures within web pages.
The sole interesting part of ATProto is the aggregation part, which can also be solved in the same way with existing web technologies, that is a web indexer. Google allows you to set up notifications whenever a link is created to your website, which is essentially the unique offering of AT. None of the ideas it presents are new, but they are all incompatible with existing Web technologies. That is why I dislike it. I already have a website. I already have a set of RSS feeds that I follow. I do not want another website hosted on a strange and convoluted service in a bajillion different JSON records when there is a folder of XML on my computer called “my-website” which I can copy to any server and have all I need. Yet, in the name of a federated internet, the BSky folks have herded people away from the real distributed internet and into a crappy Twitter knockoff. Perhaps there are some business-unaware programmers very passionate about the idea of a distributed web and I commend them for that but it already exists and, in every real aspect, they are harming it. If this ATProto thing is successful in its goals, it will cause much inconvenience to me and greatly diminish my ability to publish my works on the internet.
Interesting - I just spent all day on this on an app which I'm using. My architecture is a little different (probably worse).
The app lives on a single OpenBSD server. All user data is stored in /srv/app/[user]. Authentication is done by accessing OpenBSD Auth helper functions.
Users can access their data through the UI normally. Or they can use a web based filesystem browser to edit their data files. Or, alternately, they can ssh into the server and have full access to their files with all the advantages this entails. Hopefully, this raises the ceiling a bit for what power users of the system can accomplish.
I wanted to unify the OS ecosystem and the web app ecosystem and play around with the idea of what happens if those things aren't separate. I'm sure I'm introducing all kinds of security concerns which I'm not currently aware of.
Another commenter brought up Perkeep, which I think is very interesting. Even though I love Plan 9 conceptually, I do sort of wonder if "everything is a file" was a bit of a wrong turn. If I had my druthers, I think building on top of an OS which had DB and blob storage as the primary concept would be interesting and perhaps better.
If anybody cares, it's POOh stack, Postgres, OCAML, OpenBSD, AND htmx
remoteStorage seems aimed at apps that don't aggregate data across users.
AT aims to solve aggregation, which is when many users own their own data, but what you want to display is something computed from many of them. Like social media or even HN itself.
remoteStorage is still occasionally getting updates. https://solidproject.org is a somewhat newer, similar project backed by Tim Berners-Lee. (With its own baggage.)
I think of those projects as working relatively well for private data, but public data is kinda awkward. ATProto is the other way around: it has a lot of infra to make public data feasible, but private data is still pretty awkward.
It's a lot more popular though, so maybe has a bigger chance of solving those issues? Alternatively, Bluesky keeps its own extensions for that, and starts walling those bits off more and more as the VCs amp up the pressure. That said, I know very little about Bluesky, so this speculation might all be nonsense.
Yes Solid is another idea that's clearly the right thing, especially now. It might mlnit be easy to convince everyone they need it though, and the economics will be uphill given the trillions in entrenched, incumbent opposition.
Unfortunately, mesh storage systems are very different conceptually so it is difficult for people to think about permissions and access. You can bolt on something familiar but then it really limits the usefulness of mesh storage and you may as well just be using HTTP servers.
I'd say some of the worldview is shared but the architecture and ethos is very different. Some major differences:
- AT tries to solve aggregation of public data first. I.e. it has to be able to express modern social media. Bluesky is a proof that it would work in production. AFAIK, Solid doesn't try to solve aggregation, and is focused on private data first. (AT plans private data support but not now.)
- AT embraces "apps describe on their own formats" (Lexicons). Solid uses RDF which is a very different model. My impression is RDF may be more powerful but is a lot more abstract. Lexicon is more or less like *.d.ts for JSON.
This, Local-first Software [1], the Humane Web Manifesto [2], etc. make me optimistic that we're moving away from the era of "you are the product" dystopian enshittification to a more user-centric world. Here's hoping.
Amazing write-up. Owing to your data, a standardized protocol (such as the AT Protocol) is so great! It's like markdown, everything is basically built on files (but cleverly architected so that it works decentralized across the web too!).
Interesting concept for all new social platforms that already live in federated, distributed environments that share communication protocols and communication data formats.
I bet more difficult to push existing commercial platforms to anyhow consider.
That would make marketing tools to manage social communications and posting across popular social media, much easier. Never the less Social Marketing tools have already invented we similar analogy just to make control over own content and feedback across instances and networks.
We still live in a world where some would say BSKY some would say Mastodon is the future… while everybody still has facebook and instagram and youngsters tik tok too. Those are closed platforms where only tools to hack them, not standards persist
Bluesky is not huge, but 40M users is not nothing either. You don't get people to want this, you just try to build better products. The hope is that this enables us all to build better products by making them more interoperable by default. Whether this pans out remains to be seen.
I also don't think the average user gets the value of the protocol yet. Most of those users were looking for a new, more politically palatable home but with the same features as Twitter. The new generation of apps on the protocol will be vital in showing users what's possible. IMO the two most valuable features at a practical level are:
- social graph portability, which might look like having an onboarding experience that bootstraps your community on that app
- lexicon cross compatibility, i.e. your data from app A shows up in a contextually relevant spot in app B. Or app B writes records that show up in app A. This is pretty key to get right because it might confuse or anger users if they aren't condition to expect it.
Once the average user groks these features though, I'd be surprised if they voluntarily switch back to the standard corpo apps that eventually exit to some company who tries to monetize the shit out of every feature.
I think most people do want this. They want to own their data. If you ask someone if they post on IG, if they should own that, or IG, they'll tell you it's them.
The hard problem IMO is how do you incentivize companies from adopting this since walled gardens helps reduce competition.
We want more control over data that we've created, and more control over data that's about us. I'm not sure either of these concepts align well with "ownership" though. Property and data are concepts that don't mix.
Language nitpicking aside... you subvert the walls of their gardens and aggregate the walled-off data without the walls, so users face a choice not between:
- facebook
- everything else
but instead between
- facebook and everything else
- just facebook
But that approach only works if we can solve the "data I created" problems in a way that doesn't also require us to acknowledges facebook's walls.
I think the opportunity is huge, that a new social media without the current flaws will emerge is inevitable and this is could be part of the implementation solution but sure no real universal success. Think it probably needs to come outside of the us to have any credibility.
“You know how much it sucked when Elon bought Twitter and it became a cesspool? Wouldn’t it have been great if you could easily take your entire post history to some other platform?”
To share is to lose control. You can't undo, even once shared, it can't be undone. You can't retract a published novel. You can't retract a broadcast music or show. What makes you think you can do it over internet?
I don't think my article makes any claims that one can undo sharing. What I'm saying is that we benefit collectively from being able to untether data from applications. It's the same logic as https://stephango.com/file-over-app but applied to the aggregating web applications.
Share and you can't control what everyone else does with the data. Hence, untether is a non concept. In the sense that that data has now lost it's uniqueness. You can untether all you want. Copies of it will be subjected to any kind of tethering and transformation. So, if your point was about control, forget it.
Is there a community or forum where questions about ATProtocol can be asked?
Can you can spread your identity across multiple PDS repositories? Was thinking about creating a PDS that can store binary blobs to download, but not all PDS would like that amount of binary data stored.
This was a nice intro to AT (though I feel it could have been a bit shorter)
The whole things seems a bit over engineered with poor separation of concerns.
It feels like it'd be smarter to flatten the design and embed everything in the Records. And then other layers can be built on top of that
Making every record includes the author's public-key (or signature?). Anything you need to point at you'd either just give its hash, or hash + author-public-key. This way you completely eliminate this goofy filesystem hierarchy. Everything else is embed it in the Record.
Lexicons/Collections are just a field in the Record. Reverse looking up the hash to find what it is, also a separate problem.
Yes. SSB and ANProto do this. We actually can simply link to a hash of a pubkey+signature which opens to a timestamped hashlink to a record. Everything is a hash lookup this way and thus all nodes can store data.
{:record {:person-key **public-key**
:type :twitter-post
:message "My friend {:person-key **danabramov-public-key**} suggested I make this on this HN post {:link **record-of-hn-post-hash**}. Look at his latest post {:link **danabramov-newtwitter-post-hash** :person-key **danabramov-public-key**} it's very cool!"}
:hash **hash-of-the-record**
:signature **signature-by-author**}
So everything is self-contained. The other features you'd build on top of this basic primitive
- Getting the @danabramov username would be done by having some lookup service that does person-key->username. You could have several. Usernames can be changed with the service.. But you can have your own map if you want, or infer it from github commits :)) There are some interesting ideas about usernames about. How this is done isn't specified by the Record
- Lexicon is also done separately. This is some validation step that's either done by a consumer app/editor of the record or by a server which distributes records (could be based on the :type or something else). Such a server can check if you have less than 300 graphemes and reject the record if it fails. How this is done isn't specified by the Record
- Collection.. This I think is just organizational? How this is done isn't specified by the Record. It's just aggregating all records of the same type from the same author I guess?
- Hashes.. they can point at anything. You can point at a webpage or an image or another record (where you can indicate the author). For dynamic content you'd need to point at webpage that points at a static URL which has the dynamic content. You'd also need to have a hash->content mapping. How this is done isn't specified by the Record
This kind of setup makes the Record completely decoupled from rest of the "stack". It becomes much more of independent moveable "file" (in the original sense that you have at the top) than the interconnected setup you end up with at the end.
- How do you rotate keys? In AT, the user updates the identity document. That doesn't break their old identity or links.
- When you have a link, how do you find its content? In AT, the URL has identity, which resolves to hosting, which you can ask for stuff.
- When aggregating, how do you find all records an application can understand? E.g. how would Bluesky keep track of "Bluesky posts". Does it validate every record just in case? Is there some convention or grouping?
Btw, you might enjoy https://nostr.com/, it seems closer to what you're describing!
1. It's an important problem, but I think this just isn't done at the Record layer. Nor can you? You'd probably want to do that on the person-key->username service (which would have some log-in and way to tie two keys to one username)
2. In a sense that's also not something you think about at the Record level either. It'd be at a different layer of the stack. I'll be honest, I haven't wrapped my head entirely around `did:plc`, but I don't see why you couldn't have essentially the same behavior, but instead of having these unique DID IDs, you'd just use public keys here. pub-key -> DID magic stuff.. and then the rest you can do the same as AT. Or more simply, the server that finds the hashed content uses attached meta-data (like the author) to narrow the search
Maybe there is a good reason the identity `did:plc` layer needs to be baked in to the Record, but I didn't catch it from the post. I'd be curious to here why you feel it needs to be there?
3. I'm not 100% sure I understand the challenge here. If you have a soup of records, you can filter your records based on the type. You can validate them as they arrive. You send your records to the Bluesky server and they validates them as they arrive.
2. The point of the PLC is to avoid tying identity to keys, specifically for the point that if you lose your keys, you lose your identity. In reality, no body wants that as part of the system
3. The soup means you need to index everything. There is no Bluesky server to send things to, only your PDS. Your DID is how I know what PDS to talk to to get your records
I guess what I'm asking is why does connecting multiple public keys to one identity something that has to be part of the Record?
Why is that not happening as a separate process?
I'll try to read up more on this topic! I do feel I'm missing some pieces here
We can have both a directory and use content addressable storage and give people the option of using their own keypairs. They are not mutually exclusive. Bluesky chooses to have a central directory and index.
at://hash at://pub i guess I don't know why we need at:// that seems like something we'd need in Beaker Browser.
After reading the docs at https://atproto.com/specs/at-uri-scheme and building applications that integrate with the Bluesky API, it’s clear why this format is useful for interacting with their system. For developers working outside the ATMosphere, however, it may feel less familiar compared to more conventional REST API patterns.
You mention having a "self" record in the app.bsky.actor.profile lexicon to store profile information, is there any reason to have more records of that type in your repository?
I've seen a few people make other ones when examining their accounts with pdsls, but they seem to be there for "just because I can" reasons.
How does this system determine the amount of likes a post has? Since there is no back reference on a post to people who have liked it, don't you have to iterate over every single person, iterate over their likes, to see if one of them is the post you are viewing, and add all them up?
There's also software called Constellation[0] from microcosm that exclusively collects "backlinks", which are items which link to a certain thing. e.g. follow records for an account, reposts/replies/likes on a post. If you click on the Backlinks tab when examining something on pdsls.dev you can see a list of them.
It can be self hosted and is much lighter than running a full appview. The author has a public endpoint for it that's keeping up with the whole network running on a single raspberry pi on his home network.
It's basically event sourcing. You listen to the data you care about from the network and update the local index (DB). There are also tools like Tap (https://docs.bsky.app/blog/introducing-tap) that do the plumbing work and let you backfill automatically.
I have always thought open file format > open source. My ideal web everyone has their own web file storage (get from anywhere e.g. email provider) and web apps use that to store things. Team collab etc. built on top of that e.g. sharing a file means share ann accept edits type flow. Everyone owns their file.
It seems like the biggest downside of this world is iteration speed.
If the AT instagram wants to add a new feature (i.e posts now support video!) then can they easily update their "file format"? How do they update it in a way that is compatible with every other company who depends on the same format, without the underlying record becoming a mess?
Adding new features is usually not a problem because you can always add optional fields and extend open unions. So, you just change `media: Link | Picture | unknown` to `media: Link | Picture | Video | unknown`.
You can't remove things true, so records do get some deprecated fields.
Re: updating safely, the rule is that you can't change which records it would consider valid after it gets used in the wild. So you can't change whether some field is optional or required, you can only add new optional fields. The https://github.com/bluesky-social/goat tool has a linting command that instantly checks whether your changes pass the rules. In general it would be nice if lexicon tooling matures a bit, but I think with time it should get really good because there's explicit information the tooling can use.
If you have to make a breaking change, you can make a new Lexicon. It doesn't have to cause tech debt because you can make all your code deal with a new version, and convert it during ingestion.
That's true if you define the problem as "does my parser crash" and not whether the app is perceived as working correctly. If some platform adds support for video posts, then the next thing that happens is people start making posts that are only video. Meaning that in every other client, users see what appears to be an entirely empty post. Which will be considered a bug.
This is the core argument of Moxie's seminal essay, The Ecosystem Is Moving:
One of the controversial things we did with Signal early on was to build it as an unfederated service. Nothing about any of the protocols we’ve developed requires centralization; it’s entirely possible to build a federated Signal Protocol-based messenger, but I no longer believe that it is possible to build a competitive federated messenger at all.
That was written in 2016 but it was true then and continues to be true today. Users reject federated open platforms because the coordination costs mean they don't move as fast as proprietary centralized platforms, and they often appear broken even if technically working as designed.
Nothing about that analysis is unique to social media. It is also true of file formats. OpenOffice never took off because new features got added to Office first, so files that used those features would open in semi-corrupted ways in OpenOffice. The fact that OO represented things internally using open unions didn't matter at all.
I disagree that Bluesky is in conflict with The Ecosystem Is Moving. In contrast to most decentralized/distributed protocol projects they've managed to maintain control of almost all of their infrastructure with the exception of the personal data servers (pdses) of which they control 99.01%[1]
Almost all ATProto apps just fetch posts by handle => did:plc => post-type aka "lexicon", so they depend on what Bluesky decides to give them. If someone were to introduce unknowns into the flagship product's "lexicon" they could fix that at the API or Indexing level before shipping this data to the apps that depend on their API.
An actually decentralized network would have to overcome Moxie's criticism of the ecosystem. Can it be done? We'll keep trying.
Well, this doesn't prevent the "flagship" app from shipping things and doesn't slow it down. So it's at least not slowing down development which is the argument the parent post was making.
I've actually observed the exact opposite thing. Since Bluesky is open source, it's often visible when developers start working on a feature. And they often check in lexicon changes early on. As a result, there's been a few cases where third party client actually added support for these features earlier than the official one since they already knew the shape of the data.
This wouldn't always work, of course. Yes, if you're developing an app or a client, you better keep up with the ecosystem. But the landscape is competitive and there is no cost to switching. So if something falls behind, you can use something else.
Moxie's argument is that even if something has a flagship app, this doesn't help because if you use a new feature and then your friend complains that they can't see what you posted, the experience is just that the flagship app itself is broken. People don't experience this as, oh well, my friend should just have picked a better client. They experience it as, that's annoying, the video feature doesn't work reliably on Y but it always does on X.
An extreme example of this is WhatsApp and emojis. WhatsApp doesn't use the operating system's text rendering APIs to draw emojis, instead Meta license the Apple emoji font and draw the characters themselves. That's because if you do emoji the standards based, open way you get these problems:
• People use visual puns that only make sense if emojis look a certain way, without realizing they might look very different to other people.
• People use new emoji, without realizing that old operating systems can't draw them.
The experience in both cases is that it's simply broken and buggy. Version skew can kill platforms, which is why the most successful platforms today restrict which clients can connect and forcibly expire them on a regular basis.
BTW I don't think it's worth generalizing from Bluesky. Bluesky is an X clone whose unique competitive advantage is censoring conservatives even more aggressively than Twitter itself once did. It has no technical edge so they can develop the site as open source without it being a concern to them - they don't care if they leak what they're doing because technical innovation isn't a part of their brand to begin with - and the AT protocol clearly doesn't matter to more than a tiny fraction of its users. The point you're making in the essay is general, but ends up feeling overfit to Bluesky.
There is always one party "in control" of the lexicon and its canonical version.
I think it's important to distinguish this from the "every client adds their own features" thing. Technically yes, each app can add their own things to the open union that they support better. But it's also on each implementer's to consider how this would affect UX in other clients (e.g. if you add your own embed type, it seems reasonable to also prepopulate a link embed that acts as fallback). The problems you're describing are real, but I think we should give a bit more credit to the app builders since they're also aware that this is a part of their user experience design.
But still, whoever "owns" the lexicon says what's canonical. Then yes, some other software might not catch up to what's canonical but that's similar to what's happening with any platform that supports multiple clients today. Unless your outlook is that alternative clients in general are not competitive for this reason. I think that's a grim outlook, and if that were true, services wouldn't go to extra lengths to intentionally shut down their APIs, which so has been the trend with every network.
I think in longer term the bet is that the benefits unlocked by interop and a more competitive product landscape will become clearer to end users, who will be less interested in joining closed platforms and will develop some intuitions around that. This would not happen soon, so until then, the bet is that interop will allow creating better products. And if that doesn't happen, yes, it's pretty hard for open to compete.
Well, I never personally formed a strong opinion on Moxie's take, although I do understand it. Basically yes his outlook is that any service that doesn't actively ban alternative clients will be outcompeted by those that do.
The reason is that if alt clients are possible then some fraction of the userbase will adopt them. And if some users adopt them that means the experience of other users of the service gets worse, because new features become unreliable and flaky. You think you understand what another person sees and can do, but you don't, and this leads to poor experiences.
Viewed another way the client is an integrated part of the platform and not something that you can enable users to change freely, any more than they could be allowed to change the servers freely. We don't allow the latter because users would do things that broke the service, and so it is also for the former.
Empirically Moxie seems to be correct. Of the 90s era open protocols the only ones that survive are the web and email. The web survives but it's never been truly open in an extension sense - the definition of HTML has always been whatever the dominant browser of the era accepts, and this has become moreso over the time not less. There are no more plugin APIs for instance. SMTP survives, barely, because it's the foundation of internet identity. But many people even in corporate contexts now never send an email. It's all migrated to Slack or Teams. And if you look carefully at the Slack API it's not possible to make a truly complete alternative client with it, the API is only intended for writing bots.
This is grim but I'm not sure it's false and I'm not sure it can be changed. Also, Moxie's essay ends on a positive note. He observes that competition between mobile social networks does still work well despite the lack of federation, because they coalesced around using the user's phone number as identity and address book as the friends list, so you can in fact port your social network to a different network just by signing up. The notification center in the OS provides the final piece of the puzzle, acting as a unified inbox that abstracts the underlying social network.
This is rather mobile specific but seems basically correct to me. So that suggests the key pillar isn't file formats or protocols but ownable identity. It works because telcos do the hard work of issuing portable identities and helping people keep them, and ownership can be swiftly verified over the internet.
Maybe I’m optimistic for no good reason. I feel there’s something slightly different at play here but I struggle to put my finger on it.
Overall I agree with the take that a client for somebody else’s product is usually degraded experience. You want to deploy features vertically, and even existence of worse clients is a threat to that. I think one exception to that is when the main client is intentionally worse for the sake of other goals. Eg lots of ads. Or just the company being (or becoming) incompetent at making a client. Then alternative clients inject some new life and competition into the product. But if the product owner is good at making clients, they probably don’t benefit from fragmentation.
And still it feels somehow different to me. I think a part of it is that the space itself is vastly expanded in AT. We’re not just talking about a vertical app with a few offshoot clients that do the same thing but poorly. I mean, that still exists, but the proposition is that things can reach way beyond the original product. Products can blend. When products blend, there’s always this inherent coordination problem. Those who make multiple vertical products have to deal with similar things. At some point you have to stagger rollouts, support features partially or in a limited way, or show a simple version and direct to main app for more. Think Threads “bleeding through” Instagram, different kinds of cross-posting that they experiment with between apps. This is valuable but normally only monopolies can build meaningful cross-product bleeding because otherwise you have to negotiate API agreements and it’s just too risky to build on someone else’s turf.
What AT changes is that every product can cross-bleed into every other product. My teal.fm plays show up on my Blento page. So yes, you do have this fragmentation, but you also get something in return. It’s not fragmentation between the main way to show a thing and a couple offshoots, but between a web of products. And there is always a canonical schema so it doesn’t descend into chaos. I think there’s a “there” there.
I actually agree the key pillar is identity. It’s what AT URIs start with identity first. I think one way to look at AT is that the “contact list” analogy is right, but now that we’ve chosen the identity layer, we might as well put all our data into the contact list.
Riffing off the identity thing, one service I wanted for a while is something that issues X.509 certificates based on verified phone numbers. Phone numbers are a pretty great identity, perhaps the most successful private sector identity system ever, but they're expensive and annoying to verify, and the verification isn't portable across systems. A CA that did SMS verification and then gave you a certificate you could use with S/MIME or bind to passkeys or just use to sign software/documents in general, would democratize stable cryptographic identity. People generally can't handle key management directly, it's too easy to lose keys, but issuing transient keys tied to a phone number is much more palatable.
And PNs have got good features you want in general:
• Can have >1 of them if you want.
• Anonymous if you want.
• Not tied to any specific provider due to number portability laws.
• Hard to lose; phone companies will accept govt issued ID to get your account back if you lose your SIM and it's tied to a contract.
• Verifiable over the internet.
The only problem with them is they don't yield asymmetric keypairs.
The difficulty is business model. The people who want to consume such certificates are people who don't want to pay to verify numbers directly, but users don't want to pay either. So who pays.
Most apps reading records will validate a record against the schema for that type. e.g. there's nothing stopping you from making a app.bsky.feed.post record with more than 300 graphemes in the "text" field, but that post won't appear in the "official" app/website because it fails schema validation.
Similarly, there's nothing stopping you from adding another field in your post. It'll just get ignored because the app you're using doesn't know about it. e.g. posts bridged from mastodon by bridgy have an extra field containing the full original post's text, which you can display in your app if desired. reddwarf.app does this with these posts.
Lexicon validation works the same way. The com.tumblr in com.tumblr.post signals who designed the lexicon, but the records themselves could have been created by any app at all. This is why apps always treat records as untrusted input, similar to POST request bodies. When you generate type definitions from a lexicon, you also get a function that will do the validation for you. If some record passes the check, great—you get a typed object. If not, fine, ignore that record.
I know this is somewhat covered in another comment, but, the concepts described in the post could have been reduced quite a bit, no offense Dan. While I like the writing generally, I would consider writing and then letting it sit for a few days, rereading, and then cutting chaff (editing). This feels like a great first draft but without feedback, and could have greatly benefited from an editing process, and I think using the argument that you want to put out something for others to take and refine isn’t really a strong one… a bit more time and refinement could have made a big difference here (and given you have a decently sized audience I would keep in mind).
From my perspective, there is no chaff. I've already the read the entire thing from top to bottom over 20 times (as I usually do with my writing), I've done several full edit passes, and I've removed everything inessential that I could find. The rest is what I wanted to be included into this article.
I know my style is verbose but I try to include enough details to substantiate the argument at the level that I feel confident it fully stands for itself. If others find something useful in it, I trust that they can riff on those bits or simplify.
For the commenters who have made similar comments, I'd be curious to hear what they think could be cut. I suspect different readers will have different opinions on this, which means it's probably a good thing you didn't make cuts.
The more I read and consider Bluesky and this protocol, the more pointless -- and perhaps DANGEROUS -- I find the idea.
It really feels like no one is addressing the elephant in the room of; okay, someone who makes something like this is interested in "decentralized" or otherwise bottom-up ish levels of control.
Good goal. But then, when you build something like this, you're actually helping build a perfect decentralized surveillance record.
This why I say that most of Mastodon's limitations and bugs in this regard (by leaving everything to the "servers") are actually features. The ability to forget and delete et al is actually important, and this makes that HARDER.
I'm just kind of like, JUST DO MASTODONS MODEL, like email. It's better and the kinks are more well thought about and/or solved.
Author here. I think it's fair to say that AT protocol's model is "everyone is a scraper", including first party. Which has both bad and good. I share your concern here. For myself, I like the clarity of "treat everything you post as scraped" over "maybe someone is scraping but maybe not" security by obscurity. I also like that there is a way for me to at least guarantee that if I intentionally make something public, it doesn't get captured by the container I posted it into.
This seems like tensions between normal/practical and “opsec” style privacy thinking… Really, we can never be sure anything that gets posted on the internet won’t be captured by somebody outside our control. So, if we want to be full paranoid, we should act like it will be.
But practically lots of people have spent a long time posting their opinions carelessly on the internet. Just protected by the fact that nobody really has (or had) space to back up every post or time to look at them too carefully. The former has probably not been the case for a long time (hard drives are cheap), and the latter is possibly not true anymore in the LLM era.
To some extent maybe we should be acting like everything is being put into a perfect distributed record. Then, the fact that one actually exists should serve as a good reminder of how we ought to think of our communications, right?
Exactly. Anything that's ever been public on the internet is never really gone anyways, and it's unsafe to assume so. This is similar to publishing a website or a blog post. Plus, from a practical (non-opsec) point of view, you can delete items (posts, likes, reposts, etc.) on ATProto, and those items will disappear from whatever ATProto app you are using - usually even live. You need to dive into the protocol layer to still see deleted items.
Your first statement is good advice, but is factually WRONG. Stuff does go away.
We know this from the other side of it, stuff that we WANTED to keep, but couldn't. Google deleting blogs, people not maintaining their own personal media records, etc.
Now. What is a good forward GIVEN THAT we do know that stuff CAN go away? My gut is that a good balance of "making personal stuff available to people who want to see it again" vs. "surveillance state" could make use of the unreliability of "the net." And that ATProto's "perfection" isn't all that helpful.
Your last point is one that I used to be very strongly favor of, and today?
Nooooooooooo. No. No. No.
It's not going to happen and we shouldn't even consider it. Seriously. This thing we are doing here, which is "connecting people to each other," those forces for MANY will be far more powerful than "let me stop and think about the fact that this is forever." I just don't think we are wired for it, we're wired for a world in which we can just talk?
I think it's better to try to engineer some specific counter to "everything is recorded all the time" (or, as in here, not try to usher it into existence even more) than to try to say "welp, everything is recorded all the time, better get used to it."
It would be nice to engineer a way around this, but I don’t see it. Fundamentally if we want to be able to talk to random people, we’ll have to expect that some might be capturing communications, right?
I think there's a massive difference between "some might be capturing" and "reliable, e.g. hashed records"
Again, there may be some VALUE in the fact that "some might be capturing" WOULD BE inherently unreliable. The ability to "muddy the record" could make some SAFER.
It's true that Mastodon is somewhat better if you don't want to be found, though it's hardly a guarantee. From a "seeing like a state" perspective, Bluesky is more "legible" and that has downsides.
But I think there's room for both models. There are upsides to more legibility too. Sometimes we want to be found. Sometimes we're even engaging in self-promotion.
Also, I'll point out that Hacker News is also very legible. Everything is immutable after the first hour and you can download it. We just live with it.
It's not about the access, it's about the completeness. Imagine this paradigm takes off (I hope it does!), everyone has their own PDS and finally owns their data. Social apps link into their PDS to publish and share data exactly as they're supposed to.
Well now someone's PDS is a truly complete record of their social activity neatly organized for anyone that's
interested. It's not a security issue, after all the data was still public before, but the barrier to entry is now zero. It's so low that you can just go to stalker.io, put in their handle, and it will analyze their profile and will print out a scary accurate timeline of their activity and location leveraging AI's geoguesser skill.
If that's your threat model, then I think the way forward is to maintain separate identities. There are trade-offs there also of course: fragment yourself too much and the people who trust you will now only trust a portion of what you have to say... unless you have the time and energy to rebuild that trust multiple times.
Of course that's the same with the web we have today, the only difference is that you get control over which data goes with which identity rather than having that decision made for you by the platform boundaries.
Consider that a muddy, "oh a bunch of different servers downloaded this guys history, but it's different on one from the other" could be ORDERS OF MAGNITUDE SAFER from a surveillance perspective.
That is how it works, but people shouldn't be posting their location or sensitive information publicly if they don't want it exposed like that. That's basic opsec. Private data is currently being worked on for ATProto and will hopefully begin existing in 2026.
> people shouldn't be posting their location or sensitive information publicly if they don't want it exposed like that
They shouldn't, but they still could: accidentally paste in the wrong browser tab; have been stupid when they were 12 years old; have gotten drunk; or a number of other things.
This is a line of thinking that just supposes we shouldn’t post things on the internet at all. Which, sure, is probably the right move if you’re that concerned about OPSEC, but just because ActivityPub has a flakier model doesn’t mean it isn’t being watched
In theory it should be possible to allow users to upload ciphertext that they can then share a decryption key with their intended audience. I believe atproto has dissuaded against this with the argument that ciphertext shouldn't be in public view, but this seems to hinge on the idea that the cipher is insecure, or will be in the future. I don't see why using a post-quantum encryption scheme shouldn't provide the appropriate security, which may still not be foolproof, but it certainly would make indexing the data much more difficult
When it comes to the internet, tech is law. There is no way to publicly share something and maintain control over it. Even on the Fediverse, if either a client or server wants to ignore part of the protocol or model, it can. Like a system message to delete particular posts for anti-surveillance reasons can simply be ignored by any servers or clients that were designed/modified for surveillance. Ultimately the buck lies with the owner of some given data to not share that data in the first place if there's a chance of misuse.
Why would we need to store the createdAt value in a file? The filesystem already stores this information. We could just store the text which would mean no Json would be needed.
As someone who explicitly designed social protocols since 2011, who met Tim Berners-Lee and his team when they were building SOLID (before he left MIT and got funded to turn it into a for-profit Inrupt) I can tell you that files are NOT the best approach. (And neither is SPARQL by the way, Tim :) SOLID was publishing ACLs for example as web resources. Presumably you’d manage all this with CalDAV-type semantics.
But one good thing did come out of that effort. Dmitri Zagidulin, the chief architect on the team, worked hard at the W3C to get departments together to create the DID standard (decentralized IDs) which were then used in everything from Sidetree Protocol (thanks Dan Buchner for spearheading that) to Jack Dorsey’s “Web5”.
Having said all this… what protocol is better for social? Feeds. Who owns the feeds? Well that depends on what politics you want. Think dat / hypercore / holepunch (same thing). SLEEP protocol is used in that ecosystem to sync feeds. Or remember scuttlebutt? Stuff like that.
Multi-writer feeds were hard to do and abandoned in hypercore but you can layer them on top of single-writer. That’s where you get info join ownership and consensus.
ps: Dan, if you read this, visit my profile and reach out. I would love to have a discussion, either privately or publicly, about these protocols. I am a huge believer in decentralized social networking and build systems that reach millions of community leaders in over 100 countries. Most people don’t know who I am and I’m happy w that. Occasionally I have people on my channel to discuss distributed social networking and its implications. Here are a few:
To be clear, I'm using files in a relatively loose sense to focus on the "apps : formats are many-to-many" angle. AT does not literally implement a full filesystem. As the article progresses, I restrict some freedoms in the metaphor (no directories except collections, everything is JSON, etc). If you're interested in the actual low-level repository format, it is described here: https://atproto.com/specs/repository
Yes, which is why by default, key management is done by your hosting. You log into your host with login/password or whatever mechanism your host supports.
Adding your own emergency rotational key in case your hosting goes rogue is supported, but is a separate thing and not required for normal usage. I'd like this to be more ergonomical though.
I like the write-up of this idea. It's well presented. But I'd change one aspect: "We could leave author: 'dril' in the JSON but this is unnecessary too." - kind of. What the post lacks is the record of the identity at the time. What the user's username and the avatar was at the time can change the meaning of the post entirely. To really preserve the message, you need to reference what the displayed identity was used to post it - not just the account id.
There's a number of famous accounts that do it continuously. For example popehat today is "Fucking Bitch Hat" but will change to something else soon that may be related to the current events.
I think most people's mental model is that they should be able to change their handle / display name / avatar freely, and their posts would display the new versions. So those aren't a part of the post itself.
That said, you could create an AT app that displays a version of the post using the profile at the time. You'd just need to index all profile changes into a local database, and then query that database for the "profile at that time" to display the post. So what you're describing is possible—it just requires a different aggregation. The source of truth, however, should be denormalized and reflect most recent data.
The premise of this article resonates so much with me! I didn't see the angle on ATproto coming, and frankly this description of it is the first that makes me want to dig into it a bit.
The issue of "file-less" computing has been bothering me a lot. It's worst on iOS, where apple are really pushing hard to have users never ever think in "files". Closely followed by MacOS and only then android, imho. "That thing over there? That's not a file! That's a photo! Very different thing, that. No, you can't use $app to view it or share it, sorry. And if you want to copy it, you have to go through our export functionality which is buggy and strips all kind of info and generally works best through iCloud.
My mind's insistance that behind all the flashy apps must he a backing store, and that it's most likely file based, makes it even more infuriating when trying to get unprocessed photos off a relative's iPhone or such.
yeah yeah yeah, everyone get on the AT protocol, so that the bluesky org can quickly get all of these filthy users off of their own servers (which costs money) while still maintaining the original, largest, and currently only portal to actually publish the content (which makes money[0]). let them profit from a technical "innovation" that is 6 levels of indirection to mimic activity pub.
if they were decent people, that would be one thing. but if they're going to be poisoned with the same faux-libertarian horseshit that strangled twitter, I don't see any value in supporting their protocol. there's always another protocol.
but assuming I was willing to play ball and support this protocol, they STILL haven't solved the actual problem that no one else is solving either: your data exists somewhere else. until there's a server that I can bring home and plug in with setup I can do using my TV's remote, you're not going to be able to move most people to "private" data storage. you're just going to change which massive organization is exploiting them.
I know, I know: hardware is a bitch and the type of device I'm even pitching seems like a costly boondoggle. but that's the business, and if you're not addressing it, you're not fomenting real change; you're patting yourself on the back for pretending we can algorithm ourselves out of late-stage capitalism.
> until there's a server that I can bring home and plug in with setup I can do using my TV's remote, you're not going to be able to move most people to "private" data storage
Quite some BSky users are publishing on their own PDS (Personal Data Server) right now. They have been for a while. There are already projects that automate moving or backign up your PDS data from BSky, like https://pdsmoover.com/
Microblogging is also the least interesting part of the ATProto ecosystem. I've switched all my git hosting over to https://tangled.org and am loving it, not least of which is that my git server (a 'knot' in Tangled parlance) is under my control as a PDS and has no storage limits!
yeah, tangled seems like a pretty well-designed piece of tech. I've never used it, myself, but I did an audit and found that it's not only analogous to github as far as UX, but it also includes features like CI/CD, which other public/social repo servers have struggled with.
only reason I backed away from it is that when the bsky team had a big "fuck the users" moment, the user purporting to be the tangled founder was happy to cheer them on. so between having to use AT proto, and assuming that the tangled dev doesn't really disagree with bsky's "fuck the users" sentiment, I moved on. but, obviously, whiny moral grandstanding is irrelevant to whether or not someone made a good product. if you've got a use for it, I'd certainly recommend giving it a try!
Tangled founder here; it's just as easy! For example, here's the entire Tangled codebase monorepo: https://tangled.org/tangled.org/core — you can clone this directly as you would a git repo anywhere else.
New user sign up is a bit wonky. It asked for an email, login and password, then it's asking for a bsky sign-in too? This seems a little weird.
(Minor nit: for some reason, Google didn't auto-suggest a strong password for the password field.)
Then I got to the screen where it asks for full read-write access to my PDS and stopped there. It's kind of a lot to ask! I believe this is Bluesky's fault, but I don't think I can really use third-party bluesky apps until they implement finer-grained permissions.
yeah, I was one of them. developers are not the endgame, though. true social media needs people who are not going to do anything more complicated than "go to website, sign up". there's no world where setting up your own pds is that simple without an organized piece of software to do that kind of thing.
personally, I could probably get behind recommending something like umbrel[0], if it included something like a "include a pds" option during config. but even that is asking for a lot of mind-share for a non-tech user. it would take a super smooth setup process for that to be realistic. point is, though, I'm not saying it can't be done; I'm saying no one is doing it and what people are doing is not getting the job done for wider adoption.
[0] https://umbrel.com/
*and, naturally, at this point, I'd prefer they include something that isn't based on AT proto for social publication. I wouldn't mind if they had both, but just an AT proto implementation wouldn't attract me.
and you expect these “go to website, sign up” people to take the extra step to select a provider for repoing data? these people can barely pick a mastodon instance, what sort of data ownership integration work do you expect? it’s a consideration that’s more niche than the current status quo. unless you’re fine with people defaulting to onedrive or similar.
no, I expect them to go to a store to buy the same product that their friend bought. then I expect them to use that product by scanning a QR code that comes up on their TV, and then registering an account using the site linked by that code (or, just using their tv's remote to sign up, with an on-screen keyboard, or whatever).
not "a server so simple, anyone could host it"; "a set top box that gives me a private social media storage, network data storage, secure external connections, and effortless integration with all of my other iot devices". note that the former requires you to know what hosting is, while the latter only requires that you know what you want to do, without having to understand the details of how its done.
>that the bluesky org can quickly get all of these filthy users off of their own servers (which costs money)
That's not correct, actually hosting user data is cheap. Most users' repos are tiny. Bluesky doesn't save anything by having someone move to their own PDS.
What's expensive is stuff like video processing and large scale aggregation. Which has to be done regardless of where the user is hosting their data.
come on, man, let's be real. you're talking modern, practical application; I'm talking reasonable user buy in at big boy social media levels. the video hosting IS what I'm talking about being expensive. you think bsky is going to be successfully while ignoring the instagram crowd forever? what are we doing here?
bsky saves the video processing and bandwidth by not hosting that content on bsky. it's a smaller problem, but in a large enough pool, images become heavy, too. and, either way, the egress of that content is expensive if you're doing it for the entire world, instead of letting each individual's computer (their pds) do it.
I'm happy to admit that text is cheap and bsky isn't looking to offload their data as it stands now. but let's be honest about the long term, which is what my original comment takes aim at.
I still don't think this is correct. The Bluesky app always processes video, whether you're self-hosting or not. The personal data serves stores the original blob, but the Bluesky's video service will have to pick it up and transcode it (to serve it from CDN) either way.
Also, this:
>let them profit from a technical "innovation" that is 6 levels of indirection to mimic activity pub.
is also wrong because AT solves completely different problems. None of the stuff I wrote about in the post can be solved or or is being solved by ActivityPub. Like, AP is just message passing. It doesn't help with aggregation, identity, signing, linking cross-application data, any of that.
right on, man, that must be the only way transcoding can be done and completely future proof so it will never change to let the user transcode their own damn content. I get it; you're frustrated so you're nitpicking. funny how dang doesn't swoop in to tut tut you for not steelmanning instead of strawmanning.
in any case, you're completely right about the activity pub comment. that was absolutely mockery and not actually a complaint. artistic license and all that. god forbid we express ourselves. but sure, I can recognize that AT proto is useful in that it provides mechanisms we didn't really have before. that said, it's not novel (just new) and it's not irreplaceable. LIKE I SAID: there's always another protocol.
any time you want to actually address the point of the comments, I'd be happy to get your take on why it's fine, actually, for the CEO to imply she doesn't have to care what users think as long as they aren't paying her. but if you're not ready to have the real conversation, I'll let you be satisfied with whatever other potshots you want to take at my reasonable indignation.
agree! Social-media contributions as files on your system: owned by you, served to the app. Like .svg specifications allows editing in inkscape or illustrator a post on my computer would be portable on mastodon or bluesky or a fully distributed p2p network.
> When great thinkers think about problems, they start to see patterns. They look at the problem of people sending each other word-processor files, and then they look at the problem of people sending each other spreadsheets, and they realize that there’s a general pattern: sending files. That’s one level of abstraction already. Then they go up one more level: people send files, but web browsers also “send” requests for web pages. And when you think about it, calling a method on an object is like sending a message to an object! It’s the same thing again! Those are all sending operations, so our clever thinker invents a new, higher, broader abstraction called messaging, but now it’s getting really vague and nobody really knows what they’re talking about any more.
Hi, so I generally actually agree with you and your criticisms of this blog post (in your thread with the author). I think there's something pretty true in the blog post you shared from Joel (true in that it applies to more than just the software world) and looked at some of his more recent posts.
This one in particular reads similar to what this comment section is about, it looks like Joel is basically becoming an architecture astronaut himself? Not sure if that's actually an accurate understanding of what his "block protocol" is, but I'm curious to hear from you what you think of that? In the 25 years since that post, has he basically become the thing he once criticized, and is that the result of just becoming a more and more senior/thinker within the industry?
Author here! I grew up reading Joel's blog and am familiar with this post. Do you have a more pointed criticism?
I agree something like "hyperlinked JSON" maybe sounds too abstract, but so does "hyperlinked HTML". But I doubt you see web as being vague? This is basically web for data.
After taking the time to re-read the article since I initial posted my (admittedly shallow) dismissal, I realized this article is really a primer/explainer for the AT protocol, which I don't really have enough background in to criticize.
My criticism is more about the usefulness of saying "what if we treated social networking as a filesystem": which is that this doesn't actually solve any problems or add any value. The idea of modeling a useful thing (social media)[0] as a filesystem is generalizing the not-useful parts of it (ie. the minutia of how you actually read/write to it) and not actually addressing any of the interesting or difficult parts of it (how you come up with relevant things to look at, whether a "feed" should be a list of people you follow or suggestions from an algorithm, how you deal with bad actors, sock puppets, the list goes on forever.)
This is relevant to Joel's blog because of the point he makes about Napster: It was never about the "peer to peer" or "sharing", that was the least interesting part. The useful thing about Napster was that you could type in a song and download it. It would have been popular if it wasn't peer to peer, so long as you could still get any music you wanted for free.
Modeling social media as a filesystem, or constructing a data model about how to link things together, and hypergeneralizing all the way to "here's how to model any graph of data on the filesystem!" is basically a "huh, that's neat" little tech demo but doesn't actually solve anything. Yes, you can take any graph-like structured data and treat it as files and folders. I can write a FUSE filesystem to browse HN. I can spend the 20 minutes noodling on how the schema should work, what a "symlink" should represent, etc... but at the end of the day, you've just taken data and changed how it's presented.
There's no reason for the filesystem to be the "blessed" metaphor here. Why not a SQL database? You can `SELECT * FROM posts WHERE like_count > 100`, how neat! Or how about a git repo? You can represent posts as commits, and each person's timeline as a branch, and ooh then you could cherry-pick to retweet!
These kind of exercises basically just turn into nerd-sniping: You think of a clever "what if we treated X as Y" abstraction, then before you really stop to think "what problem does that actually solve", you get sucked into thinking about various implementation details and how it to model things.
The AT protocol may be well-designed, it may not be, but my point is more that it's not protocols that we're lacking. It's a lack of trust, lack of protection from bad actors, financial incentives that actively harm the experience for users, and the negative effects on what social media does to people. Nobody's really solved any of this: Not ActivityPub, not Mastadon, not BlueSky, not anyone. Creating a protocol that generalizes all of social media so that you can now treat it all homogeneously is "neat", but it doesn't solve anything that you couldn't solve via a simple (for example) web browser extension that aggregated the data in the same way for you. Or bespoke data transformations between social media sites to allow for federation/replication. You can just write some code to read from site A and represent it in site B (assuming sites A and B are willing.) Creating a protocol for this? Meh, it's not a terrible idea but it's also not interesting.
- [0] You could argue whether social media is "useful", let's just stipulate that it is.
I think there was a bit of a communication failure between us. You took the article as a random "what if X was Y" exploration. However, what I tried to communicate something more like:
1. File-first paradigm has some valuable properties. One property is apps can't lock data out of each other. So the user can always change which apps they use.
2. Web social app paradigm doesn't have these properties. And we observe the corresponding problems: we're collectively stuck with specific apps. This is because our data lives inside those apps rather than saved somewhere under our control.
3. The question: Is there a way to add properties of the file-first paradigms (data lives outside apps) to web social apps? And if it is indeed possible, does this actually solve the problems we currently have?
The rest of the article explores this (with AT protocol being a candidate solution that attempts to square exactly this problem). I'm claiming that:
1. Yes, it is possible to add file-first paradigm properties to web social apps
2. That is what AT protocol does (by externalizing data and adding mechanisms for aggregation from user-controlled source of truth)
3. Yes, this does solve the original stated problems — we can see in demos from the last section that data doesn't get trapped in apps, and that developers can interoperate with zero coordination. And that it's already happening, it's not some theoretical thing.
I don't understand your proposed alternative with web extension but I suspect you're thinking about solving some other problems than I'm describing.
Overall I agree that I sacrificed some "but why" in this article to focus on "here's how". For a more "but why" article about the same thing, you might be curious to look at https://overreacted.io/open-social/.
The problems with social media are not at all the fact that things are “locked up in apps”.
Again, you missed my point. Data sharing is the least interesting thing imaginable, has already been solved countless times, and is not the reason social media sinks or swims.
Social media sinks or swims based on one thing and one thing only: is it enjoyable to use. Are all the people on here assholes or do they have something interesting to say? Can I post something without being overrun by trolls? How good are the moderation standards? How do I know if the people posting aren’t just AI bots? What are the community standards? In short: what kind of interactions can I expect to have on the platform?
The astronaut types look at the abysmal landscape social media has become, and think “you know what the fundamental problem is? That all this is locked up in apps! Let’s make a protocol, that’ll fix it!”
Never mind that the profit seeking platforms have zero interest in opening up their API to competing sites. Never mind that any of the sites that are interested in openness/federating all univerally have no answer to the problem of how you address content moderation, or at least nothing that’s any different from what we’ve seen before.
The problem in social media is not that things are locked up behind an app. There are apps/readers that combine multiple platforms for me (I remember apps that consolidated Facebook and twitter fully eighteen years ago. It’s not hard.)
The problem with social media is that it’s a wasteland full of bots and assholes.
A HN poster said it best 8 years ago about twitter, and I think it applies to all of social media: it’s a planetary scale hate machine: https://news.ycombinator.com/item?id=16501147
I actually agree with you on a lot of these things, I just think that they do relate to the technological shape.
To give you an example, Blacksky is in setting up their alternative server that is effectively forking the product, which gives them ability to make different moderation decisions (they've restored the account of a user that is banned from Bluesky: https://bsky.app/profile/rude1.blacksky.team/post/3mcozwdhjo...).
However, unlike Mastodon and such, anyone on the Blacksky server will continue living in the same "world" as the Bluesky users, it's effectively just a different "filter" on the global data.
Before AT, it was not possible to do that.
You couldn't "fork" moderation of an existing product. If you wanted different rules, you had to create an entire social network from scratch. All past data used to stay within the original product.
AT enables anyone motivated to spin up a whole new product that works with existing data, and to make different decisions on the product level about all of the things you mentioned people care about. How algorithms run, how moderation runs, what the standards are, what the platform affordances are.
What AT creates here is competition because normally you can't compete until you convince everyone to move. Whereas with AT, everybody is always "already there" so you can create or pick the best-run prism over the global data.
Does this make more sense? It's all in service of the things you're talking about. We just need to make it possible to try different things without always starting from scratch.
I was hoping this was literally just going to be some safe version of a BBS/Usenet sort of filesharing that was peer-based king of like torrents, but just simple and straightforward, with no porn, infected warez, randomware, crypto-mining, racist/terrorist/nazi/maga/communist/etc. crap, where I could just find old computing magazines, homebrew games, recipes, and things like that.
> Apps may come and go, but files stay—at least, as long as our apps think in files.
yes: https://www.swyx.io/data-outlasts-code-but
all lasting work is done in files/data (can be parsed permissionlessly, still useful if partially corrupted), but economic incentives keep pushing us to keep things in code (brittle, dies basically when one of maintainer|buildtools|hardware substrate dies).
when standards emerge (forcing code to accept/emit data) that is worth so much to a civilization. a developer ecosystem tipping the incentive scales such that companies like the Googl/Msft/OpenAI/Anthropics of the world WANT to contribute/participate in data standards rather than keep things proprietary is one of the most powerful levers we as a developer community collectively hold.
(At the same time we shoudl also watch out for companies extending/embracing/extinguishing standards... although honestly outside of Chrome I struggle to think of a truly successful example)
Indeed. My first reaction was:
> Files are the source of truth—the apps would reflect whatever’s in your folder.
Now that the "app" is a web site that supports itself with advertising revenue, it has no incentive whatsoever to work this way.
There was never any such incentive. MS Office formats were undocumented for years because MS had no incentive to document them. Merely using files did not help at all. Actually the vast majority of all file formats have never been open. Think about all the custom file formats used by video games, for instance.
I actually talk about this in the article. Merely using files is how we got out of the dependency on MS Office. Multiple efforts reverse-engineered them, including Google Docs. Yes they were undocumented, but as long as stuff has to be stored on the disk under user’s control, the overall dynamics are very different from you-can’t-see-the-files systems.
Is it really so different in the service context? You could also reverse engineer the HTTP endpoints and formats used by Word Online to export data from the service. It doesn't feel all that different, except perhaps that the online service can try to detect your custom tooling and block you, whereas with static data that isn't possible.
But other than that admittedly real extra problem, the bulk of the work would still be understanding the undocumented protocols and feature semantics, then matching the feature set.
Hmm, Word Online isn't what the article is about. I'm not sure if you've read the article, but it is about social apps — like Tumblr, Reddit, HN itself, etc.
I'm using file formats as a metaphor to explain how apps built on AT protocol work (lexicons are like "social file formats"), and what this way of building enables (interoperability between social apps by default).
I did read it. I think myself and other people are getting tripped up by the title and the filesystem-oriented argument. The article opens with an icon of a .doc file, so it's natural to think about MS Word, which these days is a social app via its collaboration features.
I see. Well, my point was to start with that to give an intuition of the many-to-many between apps and data, and then transfer this intuition to social aggregated apps (which is what AT enables).
I think it's on open social apps to show that they're actually meaningfully better products, and that is possible because they're open. With luck, this may lead to an ecosystem where it's worth staying compatible and interoperable, and where users scoff if someone is trying to break it, and where users have an easy way to walk away. I know this sounds super idealistic but this did essentially happen with open source over a long time. At some point, people were just as skeptical of open source as we might be about open social.
I do really appreciate your vision, FWIW. (It also seems very compatible with my ideas about software complexity and dependencies etc.)
To be clear, the vision is not mine, I'm just describing how AT works. Kudos to the team who designed it.
> meaningfully better products
That are yet to become monetised. It's all fun and games until Bluesky announced how users and developers will pay for all this and what happens with your "social file system" when you stop paying.
I mean, I can literally already self-host my personal data if I want to. And there are also already forks of Bluesky (not just the client, but the server and the database) that can participate without fragmenting the network. It is not a perfect system but it's so far from where you are when you just rely on a closed app.
That's great, but also Mastodon is just there and has been for quite some time. I see no added value in Bluesky/ATProto beyond the layer of that "social as a service" which looks like a walled garden / app store of sorts in the making. I may be wrong, of course...
I never managed to get into mastodon because mastodon isn't a single space. It's a bunch of separate, linked spaces. Which server you're on matters a lot, and I never found one that really had the vibe I was looking for.
Bluesky, though, is one big pool. There's no seam between servers. The experience is much nicer.
Also, bluesky caught on among my peers in a way that mastodon never did, which was always going to be the deciding factor. I imagine not having to "choose a server" was a big part of that.
> There's no seam between servers.
This is false, if we're to believe it's possible to exist outside the moderation control of Bluesky corp.
It's also false today while everyone is still on the same "instance" - e.g. ICE joining Bluesky and being verified is making waves... apparently being _that_ is not enough to violate Big Blue's community guidelines.
If you run a separate appview, bluesky can't censor your interactions in any way, and you can seamlessly view posts from both bluesky users and users banned on bluesky.
Of course, running an appview that indexes the whole world is expensive, and running an appview that doesn't index the whole world is less convenient than just using bluesky's appview, so the only people running an independent appview right now are blacksky.community. It isn't stable yet, but it works, and there are active users who are banned on bluesky: https://staging.blacksky.community/profile/did:plc:63hvnyjvq....
I'm optimistic that running independent appviews (especially appviews that index content lazily, on demand) will get easier in future.
Mastodon isn't doing anything similar!
Mastodon is just a bunch of isolated copies of the same app talking to each other. There is no notion of a shared identity, each server's admin is effectively a king over your account, etc. It's a fragmented patchwork of isolated sites that forward messages.
With AT, there is just one global network. Like the web. You don't post "to" someone's isolated copy of an app. You post to your own folder, and every interested app can aggregate your post.
It's a bit like email vs RSS. Very different shapes.
To give you a concrete example to ground it. Blacksky is a fork of Bluesky. They're setting up their own server with different moderation policies and are unbanning some people that Bluesky has banned (https://bsky.app/profile/rude1.blacksky.team/post/3mcozwdhjo...). However, Blacksky posts still exist in the same "world" as Bluesky--they are not an isolated fragment.
Thanks to AT, Blacksky and Bluesky are just two different prisms through which you see a single network. Whereas with Mastodon, every app is its own network, with some limited message passing between them.
I understand the technology of course. In fact, it's clear ATProto is an instantiation of data spaces, a tech developed in the EU for industry purposes. However, while in industry there is pre-existing need for interoperable "data product marketplaces", it doesn't translate to social media in the same context.
There is no pre-existing need or desire for people to put all their data into a single space - that part is entirely driven by social companies themselves which of course benefit from that kind of centralisation and interoperability. So yes, the prism is that our social interactions become a data product and more - a data product "under governance", well structured and with well defined schema, etc.
For end users, functionally, there is very little to be gained (and some things to be let go) but the technology is not accessible enough for non-tech folks to decide on the tradeoff themselves.
As for Mastodon, what you call limited currently includes editing posts, long form posts and private blocklists (among others), all features Bluesky lacks and current version of atproto does not allow. So let's not try to quantify/compare them like that.
Regardless, I like the idea overall, I believe the Solid project (https://solidproject.org/) addresses a similar concept but when less dependence on a single authority as is the case of Bluesky and ATProto.
> a developer ecosystem tipping the incentive scales such that companies like the Googl/Msft/OpenAI/Anthropics of the world WANT to contribute/participate
I think Apache Arrow has achieved exactly that [1]. It's also very file-friendly, in that Arrow IPC files are self describing, zero-copy, and capable of representing almost any data structure.
[1] https://insights.linuxfoundation.org/project/apache-arrow/co...
> At the same time we shoudl also watch out for companies extending/embracing/extinguishing standards
Is ATProto actually a standard? But regardless, nothing prevents Bluesky from enschitifying.
I’m somewhat concerned that the “file system” or the storage where all of our things are supposed to be stored is now suddenly in the cloud. We actually have a real file system … it backs itself up on iPhone even.
It feels like the entire pitch is based on some FOMO factor “oh but my posts” - do people really care that much about their short form outbursts? I mean the whole point of twitter was to post and forget but maybe not for everybody.
It’s a long road but the plan is to standardize AT. See https://docs.bsky.app/blog/taking-at-to-ietf for ongoing first steps.
>I’m somewhat concerned that the “file system” or the storage where all of our things are supposed to be stored is now suddenly in the cloud. We actually have a real file system … it backs itself up on iPhone even.
I don’t know how this is related to what my article talks about. The data we’re taking about is already public and in the cloud, it’s just locked into specific app’s databases.
The point isn’t about some single specific app, but about creating an ecosystem where the things you create (StackOverflow posts, Reddit discussions, Instagram follows, Letterboxd recipes) are not tied to the fate of that particular app and the whims of its creators. Then, if a service shuts down or turns to shit, developers can compete without solving the cold start problem. You just need to make a better app for same data, not convince everyone to recreate everything on your app.
Isn't this the design idea behind ATProto?
Yes, which is what my article is about :)
Nice to see you :) I didn't know the "indirection" law, that's funny.
I think that's an overly charitable take. Giving Google/MSFT/OpenAI/Anthropic what they want does not guarantee a return on dividends. Standards are nice, but Apple is a giant testament to the fact that all the standards in the world won't move an adequately entrenched business.
POSSE and AT Protocol can be understood as interoperable marketplaces. Platforms like Reddit and Instagram already function this way: the product is user content, the payment is attention, and the platform’s cut is ads or behavioral data. Dan argues that this structure is not inevitable. If social data is treated as something people own and store themselves, applications stop being the owners of social graphs and become interfaces that read from user-controlled data instead.
I am working on a similar model for commerce. Sellers deploy their own commerce logic such as orders, carts, and payments as a hosted service they control, and marketplaces integrate directly with seller APIs rather than hosting sellers. This removes platform overhead, lowers fees, and shifts ownership back to the people creating value, turning marketplaces into interoperable discovery layers instead of gatekeepers.
You may want to take a look at India's ONDC (https://github.com/ONDC-Official).
Sounds interesting, but I can't find any links to an actual live B2C storefront..
https://ondc.bajajfinservmarkets.in/ https://www.mystore.in/en/
Listing of apps at https://www.ondc.org/ondc-buyer-apps/
Was going to ask this but then I found openship in your profile. Will check it out. Thanks!
This is a nice introduction:
https://openship.org/about
not sure if you understood the article, isn't the whole point to own your data as "it's just a filesystem". Reddit, Instagram, etc. are the total opposite.
Yes, I should have been clearer. Reddit and Instagram do not operate this way today, but open social alternatives to them could. The idea is that people create personal websites where posting, commenting, and other social actions live, and that becomes the filesystem they own.
Open social networks would simply index or pull from those sites using agreed-upon lexicons and protocols. Existing platforms could either adopt the open social model, or, more realistically in the short term, be treated as syndication targets where posts are pushed via their APIs when someone publishes on their own site.
Yeah, that's what I was trying to say with the article. And this is not theoretical, all of these apps work this way: https://bsky.app/, https://leaflet.pub/, https://tangled.org/, https://semble.so/, https://blento.app/ (new one). It's definitely possible!
Yes I'm loving the growth in this space. I'd also throw Standard.site[0] into the mix, it seems to be the lexicons Leaflet and some other apps are using.
0. https://standard.site/
This article goes into a lot of detail, more than is really needed to get the point across. Much of that could have been moved to an appendix? But it's a great metaphor. Someone should write a user-friendly file browser for PDS's so you can see it for yourself.
I'll add that, like a web server that's just serving up static files, a Bluesky PDS is a public filesystem. Furthermore it's designed to be replicated, like a Git repo. Replicating the data is an inherent part of how Bluesky works. Replication is out of your control. On the bright side, it's an automatic backup.
So, much like with a public git repo, you should be comfortable with the fact that anything you put there is public and will get indexed. Random people could find it in a search. Inevitably, AI will train on it. I believe you can delete stuff from your own PDS but it's effectively on your permanent record. That's just part of the deal.
So, try not to put anything there that you'll regret. The best you could do is pick an alias not associated with your real name and try to use good opsec, but that's perilous.
My goal with writing is generally to move things out of my head in the shape that they existed in my head. If it's useful but too long, I trust other people to pick what they find valuable, riff on it, and so on.
>Someone should write a user-friendly file browser for PDS's so you can see it for yourself.
You can skip to the end of the article where I do a few demos: https://overreacted.io/a-social-filesystem/#up-in-the-atmosp.... I suggest a file manager there:
>Open https://pdsls.dev. [...] It’s really like an old school file manager, except for the social stuff.
And yes, the paradigm is essentially "everyone is a scraper".
Thanks! I saved a link to pdsls. I think there's room for improvement in making the UI user-friendly; maybe I'll try it someday.
The devs are responsive to feedback if you mention @pdsls.dev on Bluesky! I often point out small issues and they get fixed the next day.
This was a great article. Have you got any thoughts on Farcaster and its Snapchain?
fwiw, I don't agree. These detailed articles are excellent.
> Someone should write a user-friendly file browser for PDS's so you can see it for yourself.
https://pdsls.dev/ can serve this purpose IMO :) it's a pretty neat app, open source, and is totally client-side
edit: whoops, pdsls is already mentioned at the end of the article
You can use pdsfs[0] to mount a user's pds locally using FUSE read-only. It's mentioned in the blogpost. I remember seeing a tool posted for mounting them read-write, if you sign in, but can't remember where to find it.
0: https://tangled.org/oppi.li/pdsfs
> So, much like with a public git repo, you should be comfortable with the fact that anything you put there is public and will get indexed.
whats the sota on atproto encryption dan? just publish encrypted stuff with sha 256 and thats it?
AFAIK nothing special for it at this point, so whatever you would do with a public cleartext broadcasting platform.
Private data will come to ATProto, it's not a finished protocol
What’s the plan for this? Sharing with a specific group (DM, group chat, feeds from people you follow) are critical to social media.
See https://www.pfrazee.com/leaflets/3lzhui2zbxk2b for some recent thoughts from a team member.
Bnewbold's comment on GitHub is still the best resource. It's far more specific in scope and details. Paul's note is more theoretical / philosophical
https://github.com/bluesky-social/atproto/discussions/3363 - maybe this one?
yes, that is the one everyone is using as a guiding light for where we think Bsky is headed for their private data needs, which is group-shared / not e2ee because key mgmt is an unsolved problem at scale. They need to support creators with subscriber only content. E2EE will emerge from the messaging space first, i.e. Signal / MLS like
Bsky is likely to put out a short-term gap filler this year for personal-private data, which kind of already exists today for Bsky only. This work would make that same feature in the per-user repo available to all apps. Right now it's hard coded to a specific NSID and single db row
I think that is the general style of overreacted.io posts.
I've always thought walled gardens are the effect of consumer preferences, not the cause.
The effect of the internet (everything open to everyone) was to create smaller pockets around a specific idea or culture. Just like you have group chats with different people, thats what IG and Snap are. Segmentation all the way down.
I am so happy that my IG posts arent available on my HN or that my IG posts arent being easily cross posted to a service I dont want to use like truth social. If you want it to be open, just post it to the web.
I think I don't really understand the benefit of data portability in the situation. It feels like in crypto when people said I want to use my Pokemon in game item in Counterstrike (or any game) like, how and why would that even be valuable without the context? Same with a Snap post on HN or a HN post on some yet-to-be-created service.
>I am so happy that my IG posts arent available on my HN or that my IG posts arent being easily cross posted to a service I dont want to use like truth social.
ATProto apps don't automatically work like this and don't support all types of "files" by default. The app's creator has to built support for a specific "file type". My app https://anisota.net supports both Bluesky "files" and Leaflet "files", so my users can see Bluesky posts, Leaflet posts, and Anisota posts. But this is because I've designed it that way.
Anyone can make a frontend that displays the contents of users PDSs.
Here's an example...
Bluesky Post on Bluesky: https://bsky.app/profile/dame.is/post/3m36cqrwfsm24
Bluesky Post on Anisota: https://anisota.net/profile/dame.is/post/3m36cqrwfsm24)
Leaflet post on Leaflet: https://dame.leaflet.pub/3m36ccn5kis2x
Leaflet post on Anisota: https://anisota.net/profile/dame.is/document/3m36ccn5kis2x
I also have a little side project called Aturi that helps provide "universal links" so that you can open ATProto-based content on the client/frontend of your choice: https://aturi.to/anisota.net
Except that a lot of the app builders in ATProto seem to think the protocol was designed to make their lives easier when bootstrapping their network from Bluesky userbase.
(Imo, that is a perverse interpretation, it's about user choice, which they are effectively taking away from me by auto importing and writing to my Bsky graph)
re: the debates on reusing follows from Bluesky in other apps instead of their own
I agree. I don't understand the driving force here.
I have all of the raw image files that I've uploaded to Instagram. I can screenshot or download the versions that I created in their editor. Likewise for any text I've published anywhere. I prefer this arrangement, where I have the raw data in my personal filesystem and I (to an extent) choose which projections of it are published where on the internet. An IG follow or HN upvote has zero value to me outside of that platform. I don't feel like I want this stuff aggregated in weird ways that I don't know about.
For me, part of it is that we have no power collectively against products turning their back on users because coordination to "export data all at once and then import it into specific other place" is near-impossible. So this creates a perverse cycle where once you capture enough of the market, competition has very little chance unless they change the category entirely.
What AT enables is forking products with their data and users. So, if some product is going down a bad road, a motivated team can fork it with existing content, and you can just start using the new thing while staying interoperable with the old thing. I think this makes the landscape a lot more competitive. I wrote about this in detail in https://overreacted.io/open-social/#closed-social which is another longread but specifically gets into this problem.
I hear you re: not wanting "weird aggregation", that just be a matter of taste. I kind of feel like if I'm posting something on the internet, I might as well have it on the open web as aggregatable by other apps.
Thanks for your thoughts. I do feel like this is all very specifically connected to Twitter. Which tech people really adopted, but I never used much, so it is interesting that these different perspectives are somewhat tied to one's megaplatform(s) of choice. I don't know of another social network that has inspired such a consistent effort to be forked or cloned. I do kind of feel like "change the category" and create a new network with the traits you want to see is the right move.
For better or worse, Twitter built their network. That many people willingly signed up and posted and continue to post there. I don't think anyone really should be able to fork it, because the users didn't collectively agree to that, and they don't all agree on what a good road or a bad road is. Ultimately, they can choose to leave if and when they want. Are these networks rather sticky, yes of course, but that's life.
We've seen lots of social networks come and go, things do change over time, there's ample opportunity for new ideas to flourish. In that sense, AT is perfectly welcome to throw their hat in the ring and see if that resonates and sticks. If people want their social network to be forkable, that concept will succeed.
I do think it misses what a lot of people find valuable about the tangibility and constraints of "I am making this content specifically for this platform and this audience at this point in time." I don't think most people think of their social media posts as a body of work that they want to maintain and carry over in an abstract sense independent of platform, and give open license to anyone to cook up into whatever form they can dream of.
I don't this is tied to Twitter.
Just the other week, another service that people actively used called Bento announced shutdown: https://bento.me/. This sucks for the user.
Someone created an alternative called Blento (https://blento.app/) on AT. Of course, by itself, this doesn't mean they'll be successful. But the thing is that, if Blento shuts down, someone can put it right back up because (1) it's open source, and (2) the data is outside Blento. Any new app can kickstart with that data and get people's sites back up and running. And two platforms can even compete on top of the same data.
I agree content is tailed to the platform and resurrecting something doesn't necessarily makes sense. But that's the point of lexicons. You get the choice of what makes sense to resurrect (actively moving to an alternative) vs what doesn't (something with a style that doesn't work elsewhere) vs new recontextualizations we haven't even tried or thought of. I think it's early to dismiss before trying.
Adding more to this, what's wild to me is that Blento can already show public data from other AT apps on the homepage. For example, your teal.fm plays. It's just one of the block types: https://bsky.app/profile/flo-bit.dev/post/3mcqx5pwrrk26. Despite teal.fm not really even "existing"!
It feels like file formats and protocols is the wrong prism through which to view this. What users actually want is the ability to move as much data as possible from point A to point B, whilst remapping it as much as possible to meet the different features and identities of point B. A unified data format is actually a hindrance for meeting that need because it boils it all down to a lowest common denominator. A simple data pump utility that just copies data back and forth, with lots of special case logic like "X has a character limit of 5000, Y has a character limit of 500 so I'll truncate and add a link to the original post on X", is going to work far better for this use case.
More generally I feel like you kinda missed the point of filing systems in your first writeup. The moment you say "posts don't have natural names", ok, that's a serious problem that should put a halt to the entire concept. A file is nothing but a natural name for a byte array and a little bit of metadata. That's why we think of file managers and explorers when we think of what files are all about: lists of names. It's also why people directly use files for relatively few things - they were a big deal in the early days of personal computing because the killer apps for it were direct translation of physical office tasks. It's mostly but not entirely possible to come up with natural names for office documents (the gap is in suffixes like "draft final v3__real final draft.docx"). That's why we use analogies for 1950s paperful office objects like files and manila folders to begin with. But most computing tasks even back then weren't oriented around documents, they were oriented around database records and the user interface was just screens, or possibly direct SQL. It's a remarkable gap in our infrastructure that there is no agreed on file format for a SQL result set. Maybe Arrow is that format.
>A unified data format is actually a hindrance for meeting that need because it boils it all down to a lowest common denominator.
Exactly, which is what I'm saying in the article:
We could try to put every app developer in the same room until they all agree on a perfect lexicon for a post. That would be an interesting use of everyone’s time.
(That's meant to be slightly sarcastic.) Then the article says:
For some use cases, like cross-site syndication, a standard-ish jointly governed lexicon makes sense. For other cases, you really want the app to be in charge. It’s actually good that different products can disagree about what a post is! Different products, different vibes. We’d want to support that, not to fight it.
The point isn't having a lowest common denominator format. It's to enable precisely the granularity of formats that app developers want. The default one is "each app has its own formats". But what changes is that
1. Other apps can still read/write in other apps' formats. This makes the landscape competitive: if some app is going down a bad road, it's easy to fork their product with all the data already in there.
2. It is possible for multiple app developers to use the same format where it makes sense. As linked from the article, https://standard.site/ is a good example of that.
>More generally I feel like you kinda missed the point of filing systems in your first writeup.
I'll admit that I used the metaphor to highlight a specific thing, and names in particular don't play a big role in that thing (what plays a big role is links). The reason I picked filesystem as a metaphor is because it highlights how file formats mediate interactions between apps. This is exactly how lexicons mediate interactions between social apps. That felt like the backbone of the argument.
It's true this isn't describing a filesystem exactly. Maybe a web of identity-addressed hyperlinked JSON is a more precise way to say it. But then nobody knows what that means. I'm happy with my choice of metaphor.
> I think I don't really understand the benefit of data portability in the situation.
Twitter was my home on the web for almost 15 years when it got taken over by a ... - well you know the story. At the time I wished I could have taken my identity, my posts, my likes, and my entire social graph over to a compatible app that was run by decent people. Instead, I had to start completely new. But with ATProto, you can do exactly that - someone else can just fork the entire app, and you can keep your identity, your posts, your likes, your social graph. It all just transfers over, as long as the other app is using the same ATProto lexicon (so it's basically the same kind of app).
But what if your entire social graph didn't choose to transfer over as well? What if they don't want to be on that app? What if someone that was very indecent made a compatible app? Would you want your entire Twitter history represented on there?
For better or worse, I don't think it makes sense to decentralize social. The network of each platform is inherently imbued with the characteristics and culture of that platform.
And I feel like Twitter is the anomalous poster child for this entire line of thinking. Pour one out, let it go, move on, but I don't think creating generalized standards for social media data is the answer. I don't want 7 competing Twitter-like clones for different political ideologies that all replicate each others' data with different opt-in/opt-out semantics. That sounds like hell.
The framing of "portability" is a bit confusing. Your data is not actually "transferring" anywhere, it's always in your PDS. These other apps and clients are just frontends that are displaying the data that is in your PDS. The data is public and open, though private data is in the works and hopefully will arrive in 2026.
The data is not transferring, but the user is. When I sign up for e.g. Twitter, I don't want to sign up for Mastodon, or Bluesky, or Truth Social, or whatever other platform someone might create later. Thus I would not choose to put my data in a PDS. I feel like that would actually leave me with less ownership and control than I have now.
My point is that I don't believe the separation of frontend and data is desirable for a social network. I want to know that I am on a specific platform that gives me some degree of control and guarantee (to the extent that I trust that platform) over how my data is represented. I don't really have to worry that it's showing up in any number of other places that I didn't sign up for (technically I do since everything public can be scraped of course, but in practice there are safeguards that go out the window when you explicitly create something like a PDS).
Do I understand correctly that your main concern is that some random service would serve a page when asked for /your-handle, and so, given a link, someone may assume that you actively use that service? Just trying to understand the exact scenario.
Generally it's good practice for AT apps to treat you as not signed up if you have not explicitly signed up to this app before. So I think a well-behaved site you never used shouldn't show your profile page on request. If this is right, maybe it needs to be more emphasized in docs, templates, etc. Same as they should correctly handle lifecycle events like account getting deactivated.
Realistically it's possible some services don't do this well but it's not clear to me that this is going to be a major problem. Like if it gets really bad, it seems like either this will get fixed or people will adjust expectations about how profile pages work.
I think part of the issue is that humans are hyper conditioned to expect a certain UX and set of conditions thanks to the past 3 decades of the internet + legacy social media. It only feels weird that you could publish content on Bluesky and have it show up on some other app without your consent because of how we've been conditioned. There will be a lot of unconditioning and reconditioning that has to take place over a span of time if the ATmosphere (or any new vision of social media) wants to succeed.
https://anisota.net will display any Bluesky content and profiles without the consent of the user — no one cares at the moment though because either 1) they don't know about such a niche project, or 2) they aren't concerned cause I'm not a controversial figure.
If Truth Social was suddenly a part of the ATmosphere or a part of some other wide network of users, most people would catch on eventually and be hopefully conditioned to realize that the mere presence of someone's content on an app/site doesn't mean they use that app/site
FWIW, I think Anisota is a bit different because conceptually people see it as a Bluesky client. So it is expected that you're "projecting" Bluesky, for better or worse. Whereas if it's some fanart exchange service or something, maybe it makes less sense. Maybe it just depends on what you think the user would expect.
It's kind of just like, this sense of the "safety" of the walled garden (which one might argue the overlords could rugpull at any moment, which kind of happened with Twitter, but that's not something that really concerns me personally enough to trade it for the complexity of decentralization).
A lot of it is presentation. Instagram is the one that I use the most right now, and I feel like they really nailed this in the "everybody can make content" era. When I'm creating content for IG, I know exactly how it's going to look, there's editing and previewing tools, there's a confidence that people are going to receive it in the manner I intended and expected, and it's tied to the culture of that platform that I've become attuned to over time. The thought of a JSON of my IG post existing in a clearinghouse that encourages people to remix it into whatever, just kind of gives me the ick. Even if realistically it wouldn't really be abused, I don't feel compelled to choose that. If someone comes up with a new app that does some things better, I can choose to migrate to that, but likely I wouldn't want to just one-click import my IG history. It will be different in some way and I would want to make new content for it.
I guess I just kind of see these platforms as bespoke communities that people choose to be a part of, more than fungible data frontends. It's very personal, and...social. It's not the abstract world of data that programmers live in. My social graph in each app is unique and I don't really want it abstracted to one monolith that follows me everywhere. Each one is a new world with a fresh start.
Now, the IG post ecosystem is a lot more complex than it is with Twitter. With Twitter being at its core just text, it lends itself better to different views, but still, formatting matters and there are really a lot of rendering decisions to be made in a modern Twitter client. And again it just seems weird to me to have this oddly fragmented community with unavoidably complex interop semantics rather than everyone just being under the same roof. User opt-in aside, they will all develop their own features and quirks and they and the users will have to navigate how those translate or don't. Twitter clients are/were at least unified by the Twitter API. Regarding your initial question - it's more than just the profile page, do my comments show up in conversations? Likes and follower counts? Or are there weird gaps and discrepancies everywhere from users that haven't opted in? What if one fork requires mutual "connections" and one uses one-way followers? Each platform will make their own decisions. It just sounds like such a mess, and people will spend so much time disambiguating (oh no I'm on this one so I see it that way...).
There were some efforts to fork Reddit but it has survived as a great example of how lots of different people with different interests and opinions can occupy different parts of the same platform, even if lots of them complain about it because it's not their perfect vision of what it should be (see programming languages, and pretty much everything that reaches a certain userbase).
This sounds like I need to host my PDS. Easy for me with no public profile but if I was someone famous wouldn't that mean I needed enterprise class hosting?
You don't need to host your own PDS for any of this to work. It works the same way regardless of who hosts your PDS.
I think what may be confusing you is that Bluesky (the company) acts in two different roles. There's hosting (PDS) and there's an app (bsky.app). You can think of these conceptually as two different services or companies.
Yes, when you sign up on Bluesky, you do get "Bluesky hosting" (PDS). But hosting doesn't know anything about apps. It's more like a Git repo under the hood.
Different apps (Bluesky app is one of them) can then aggregate data from your hosting (wherever it is) and show different projections of it.
Finally, no, if you're famous, you don't need enterprise hosting. Hosting a PDS can be extremely cheap (like $1/mo maybe)? PDS doesn't get traffic spikes on viral content because it's amortized by the app (which serves from its DB).
But.... why ? Is this assuming a huge portion of the people you interact with on Twitter are also all moving identities to the new platform ?
I can see for something like artistic expression it being useful to export and and move over to a new platform (like moving a whole photography portfolio, short story writing samples, or similar). But who cares about your "likes" on a totally new and separate social media platform ?
It makes the landscape competitive. There isn't really a notion of "moving" to a "platform". It's more like if Twitter sucks, some team can spin out an alternative that has the existing content but takes different product decisions. And then some people can try this alternative and use it without leaving the existing network. So, it allows more experimentation in the market without having to solve the cold start problem. The lifecycle of products becomes more fluid. It's easy to spin something up, and you can also shut things down without permanently killing them.
I would not call the people who ran Twitter before Elon Musk bought the company and took it private "decent people". I think that Musk purchasing the company and running it in a way that a lot of previous userbase objected to was ultimately a huge boon for software freedom - because without that, the large number of people who stopped using Twitter and went to the ATProto ecosystem instead would have been happy to continue using completely-proprietary Twitter. A lot of people were suddenly and viscerally faced with the downsides of building a digital "home" on someone else's platform.
>The effect of the internet (everything open to everyone) was to create smaller pockets around a specific idea or culture. Just like you have group chats with different people, thats what IG and Snap are. Segmentation all the way down.
I actually agree with that. See from the post:
>For some use cases, like cross-site syndication, a standard-ish jointly governed lexicon makes sense. For other cases, you really want the app to be in charge. It’s actually good that different products can disagree about what a post is! Different products, different vibes. We’d want to support that, not to fight it.
AT doesn't make posts from one app appear in all apps by default, or anything like that. It just makes it possible for products to interoperate where that makes sense. It is up to whoever's designing the products to decide which data from the network to show. E.g. HN would have no reason to show Instagram posts. However, if I'm making my own aggregator app, I might want to process HN stuff together with Reddit stuff. AT gives me that ability.
To give you a concrete example where this makes sense. Leaflet (https://leaflet.pub/) is a macroblogging platform, but it ingests Bluesky posts to keep track of quotes from the Leaflets on the network, and display those quotes in a Leaflet's sidebar. This didn't require Leaflet and Bluesky to collaborate, it's just naturally possible.
Another reason to support this is that it allows products to be "forked" when someone is motivated enough. Since data is on the open network, nothing is stopping from a product fork from being perfectly interoperable with the original network (meaning it both sees "original" data and can contribute to it). So the fork doesn't have to solve the "convince everyone to move" problem, it just needs to be good enough to be worth running and growing organically. This makes the space much more competitive. To give an example, Blacksky is a fork of Bluesky that takes different moderation decisions (https://bsky.app/profile/rude1.blacksky.team/post/3mcozwdhjo...) but remains interoperable with the network.
There's also a risk of adversarial cross-site syndication: your stuff can and probably will show up on websites you don't control.
That's just how it works and I accept the risk.
People concerned about that probably shouldn't publish on Bluesky. Private chat makes more sense for a lot of things.
That also seems to be a risk with most centralized social networks these days as well.
Our LinkedIn profiles have been scanned by scrapers, Instagram photos used for facial recognition training, Twitter / FB posts replicated in a bunch of downstream places, archives, and search indexes. Not to mention data being sold to third parties.
At this point, I assume if I post it publicly online, it is no longer in my control where it ends up downstream and I assume it will be online somewhere forever.
Back in the old twitter days you could have screenshots of your tweets being displayed by a number of different 'dubious' accounts on instagram, completely out of your control.
That's just how things work, I guess.
Posts feel less valuable as files as they are of the moment. I don't need this comment as a docx, thank you all the same.
So I agree.
Maybe building it that way opens possibilities I can't see?
But for functional apps it seems useful to have a file you can download and use somewhere else. Images and movies are a great example. I can record a video in Loom and chuck it on Youtube. Youtube can't lock me on that video. it is my file. Imagine if that was not the case. It would suck.
Ever heard of the World Wide Web?
It was great. Anyone could host web pages, anyone could access and read anyone’s web pages with a tool called a web browser, of which there used to be several compatible implementations.
If truth social didn't remove all the federation code, posts from mastodon and many other ActivityPub sites would have appeared there.
> Identity -- This is a difficult problem.
My hope is that in 5 years, I will not have anything in my feeds that have not been signed in a way that I can assign a trust level.
Here in the Nordics, we are already seeing messaging apps such as [hudd] that require government issued ID to sign in. I want this to spread to everything from podcasts and old-school journalism to the soccer-club newsletter, so that I can always connect a piece of information back to a responsible source.
[hudd]: (https://about.hudd.dk/))
So you're simply not interested in reading any random website by random people who don't see a benefit of establishing any form of trust, especially if should not be connected to their official government IDs?
Or to put it differently: Where should this come from, and which issuer would you trust? And why should anyone else agree with you that this is good?
I trust person A, person A trusts B
When I browse to random site, I can see in my browser that A's first level contact trusts this site.
Now I can make a decision based on the amount of trust I have on A. Maybe after exploring the site I can mark specific pages or the whole domain as trusted, so people in my network can see the same.
On a larger level I might trust the Country of Finland, who will only mark their official sites as Trusted. This way I instantly know if I'm on an official site or something pretending to be one.
Trust is subjective! Let's establish trust in each other rather than rely on one-size-fits-all solutions.
Personally, I trust my friends, family, and some public figures and institutions to varying degrees. I want to see social experiences that reflect that.
[dead]
I’ve been reading “The Unix Programming Environment”. It’s made me realize how much can be accomplished with a few basic tools and files (mostly plain text). I want to spend some time thinking of what a modern equivalent would look like. For example, what would Slack look like if it was file (and text) oriented and UNIXy? Well, UNIX had a primitive live chat in the form of live inter-user messaging. I’d love to see a move back to simpler systems that composed well.
Unix gave the world a lot of good ideas about architecture, but I think it really hamstrung itself by treating all the data as plain text and resisting the idea of having any kind of structured, formatted data passed within a pipeline. It's nice to be able to serialize to something human-readable and -editable, but constantly re-parsing and re-formatting it becomes a real pain.
I've been thinking of this for some time, conceptually, but perhaps from a more fundamental angle. I think the idea of "files" is pretty dated and can be thrown out. Treat everything as data blobs (inspired by PerKeep[0]) addressed by their hashes and many of the issues described in the article just aren't even a thing. If it really makes sense, or for compatibility sake, relevant blobs can be exposed through a filesystem abstraction.
Also, users don't really want apps. What users want are capabilities. So not Bluesky, or YouTube for example, but the capability to easily share a life update with interested parties, or the capability to access yoga tutorial videos. The primary issue with apps is that they bundle capabilities, but many times particular combinations of capabilities are desired, which would do well to be wired together.
Something in particular that's been popping up fairly often for me is I'm in a messaging app, and I'd like to lookup certain words in some of the messages, then perhaps share something relevant from it. Currently I have to copy those words over to a browser app for that lookup, then copy content and/or URL and return to the messaging app to share. What I'd really love is the capability to do lookups in the same window that I'm chatting with others. Like it'd be awesome if I could embed browser controls alongside the message bubbles with the lookup material, and optionally make some of those controls directly accessible to the other part(y|ies), which may even potentially lead to some kind of adhoc content collaboration as they make their own updates.
It's time to break down all these barriers that keep us from creating personalized workflows on demand. Both at the intra-device level where apps dominate, and at the inter-device level where API'd services do.
[0] https://perkeep.org/
I'm using filesystem more as a metaphor than literally.
I picked this metaphor because "apps" are many-to-many to "file formats". I found "file format" to be a very powerful analogy for lexicons so I kind of built everything else in the explanation around that.
You can read https://atproto.com/specs/repository for more technical details about the repository data structure:
The repository data structure is content-addressed (a Merkle-tree), and every mutation of repository contents (eg, addition, removal, and updates to records) results in a new commit data hash value (CID). Commits are cryptographically signed, with rotatable signing keys, which allows recursive validation of content as a whole or in part. Repositories and their contents are canonically stored in binary DAG-CBOR format, as a graph of data objects referencing each other by content hash (CID Links). Large binary blobs are not stored directly in repositories, though they are referenced by hash (CID).
Re: apps, I'd say AT is actually post-app to some extent because Lexicons aren't 1:1 to apps. You can share Lexicons between apps and I totally can see a future where the boundaries are blurring and it's something closer to what you're describing.
But you don’t need the AT protocol and lexicons to describe semantics. Ontologies have been a thing for decades, and search engines (the “rich results”) and ActivityPub use them.
Why use the lexicons you described, instead of the more supported https://www.w3.org/ns/activitystreams ontology that ActivityPub uses, for example?
If you allow me to, I would be interested in re-writing your article but by using ActivityPub concepts instead. Maybe I will learn about a fatal flaw in one of the protocols during that process.
It's bold to say users don't want apps when the expressed preferences are so strongly for apps that "App Store" is a well known brand.
The word app is short for application, which in this context is just a synonym for capability. People have tried breaking apps apart into more fine grained capabilities before and it never worked (OpenDoc and OLE being two well known examples). Brand names matter, they have value to people. YouTube isn't an abstract collection of decomposable capabilities, it's a brand that summarizes numerous unarticulated pieces of information, like its commenting culture, what kind of content is allowed and what isn't, etc.
> I'd like to lookup certain words in some of the messages, then perhaps share something relevant from it.
This is already a built-in feature on macOS. If I right click in your message "Look up PerKeep" is literally the first command, and if I click it I get a nice info window telling me what it is that I can copy/paste from. There is nothing to solve here.
> the expressed preference
Marketing and culture can do wonders to one's preference.
> synonym for capability Yes and no. An "app" is usually a set of capabilities, not a single capability. Many of the core *nix utilities for example are specifically designed to have a single capability each, and these are then composed for complex requirements. But for the most part today's apps contain multiple, usually overlapping capabilities. For example Microsoft Word offers not only text editing and formatting, but also has a dictionary+thesaurus, spelling+grammar check, mail merge and lots more. Of course, you can abstract this all away as "word processor capability", but there are other word processing apps out there with varying capabilities, and so things become problematic fast especially in terms of interoperability.
> OpenDoc and OLE These "never worked" due to people/policy, not due to tech ability. OpenDoc essentially stagnated at Apple due to conflicts and lack of clarity, and then was outright killed by Steve Jobs[0]. OLE(tm) hasn't failed per se, but it's proprietary to Microsoft[1] and thus doesn't have much adoption outside Windows.
> Brand names matter All they really do is put a label on a bundle - or bundles - of capabilities to create handy associations. YouTube is associated primarily with accessing videos that can be made and uploaded by anyone, secondarily allows commenting on and liking said videos, etc. Again, marketing is a significant factor.
> a built-in feature You seem to be missing the forest for a tree here. That's a single example, of infinite permutations. Also, even given that example you're still missing parts of the flow.
[0] https://en.wikipedia.org/wiki/OpenDoc [1] https://en.wikipedia.org/wiki/Object_Linking_and_Embedding
FWIW the Look Up feature, whilst built in to macOS, can also be implemented using "services" which are a way to plug in to other apps and system features like text rendering and highlighting. I can also select text in your message, right click, services -> "Add to music as spoken track". That's a plugin offered by an app. It's only one extra click.
My perspective on why OpenDoc/OLE failed are different to yours. The model in which apps host targeted plugins which don't affect data formats proved very successful, the model in which an app is nothing but plugins wasn't. And if you want to talk about marketing and culture, well, OpenDoc and OLE both benefited from corporate marketing. Using OLE was part of Microsoft's "Designed for Windows 95" programme, which came with a badge and marketing dollars attached. This concept was heavily pushed at the time. OLE had more success than OpenDoc but only temporarily and only because it was a less pure implementation of the concept, closer to fancy plugins than truly decomposing apps into components.
I think the reason they didn't take off was a mix of technological problems and user demand problems. It wasn't people/policy related. Quite the opposite: both technologies lived on long after the real problems had become apparent.
Programming for both OpenDoc and OLE was extraordinarily complicated. The developer experience was atrocious. Making your app OLE embeddable required mastering not only COM and Win32 programming but dozens of other poorly documented and flaky APIs, like for compound documents. Code generation was rampant, everything was memory unsafe. I never did OpenDoc apps but I heard it was a very similar experience.
Then there was the user experience: if you embedded an object from an app and sent the resulting document to someone else who didn't have the app, or had an old version of the app, then the document would just have a hole in it. Useless. OLE tried to do a kind of weird mind-meld between totally different app GUIs, meaning that depending on where you clicked in a document the hosting app could look totally different. There were dozens of problems like this.
In the end it was all rendered irrelevant by the web. On the web, if you share a document with someone it always appeared for them correctly (or if it doesn't, it's a minor and quickly fixable bug). Reliable sharing mattered more than anything else and so all the non-web platforms started dying out.
> implemented using "services"
Yes that's a pattern offered and used on all modern OSes to various degrees, but most - particularly independent - developers prefer to bundle the capabilities they're offering in such a way that they're only available via their app's interface. I think partly because it's just easier to implement that way, but also because it's in their interest (not the users') to not allow the capabilities they've implemented to be accessed by apps created by others.
> perspective on why OpenDoc/OLE failed
It's not my perspective at all; that's what's in the Wikipedia articles, which I linked. OLE had more success because Microsoft was far more invested in it, since it was theirs, and is still used today in Windows (sounds like you're saying it's dead); OpenDoc was an attempt at an open standard, and failed because the participants just weren't invested and/or agreeable enough. Of course I can see Jobs driving the final nail because, for him, it wasn't exactly in Apple's best interest to create apps that interoperated with apps outside the Apple ecosystem. All those are people/policy things.
> rendered irrelevant by the web
I don't really get what you're saying here, in the context of capability provision. But I'd say no. There are many web platforms offering capabilities via APIs, but still many restrict access to their own apps, with Reddit being one of the latest to drastically restrict 3rd party access.
OLE has been dead for decades, yes. When was the last time a new app launched and being embeddable in Office documents via OLE was a key advertised feature? I don't know what even happens if you open a .docx with an embedded OLE object in Word Online.
Obviously the APIs are still there. You can use them if you want. It's just that nobody is writing pure/native win32 apps anymore, outside of browsers and legacy codebases. So, new code isn't being written as OLE controls.
I don't think Jobs killed OpenDoc because of some vision of an ecosystem of Apple's apps. At the time he made that decision Apple was on death's door because it had a culture of architecture astronauts spending years in front of whiteboards, writing features that didn't directly appeal to what the market wanted. He killed many projects at that time for the same reasons, like their line of printers.
If OpenDoc/OLE had been genuinely the right path we'd have seen people outside of MS/Apple try to reinvent them. Plenty of companies have the resources, as does the open source community. In fact KDE and GNOME both did try with KParts and Bonobo. But, both of those are dead too.
To your wider point, yes, it's often the case that what the app developers need/want is in conflict with what the users need want. I don't see that as a problem. Trade always involves tradeoffs, when users and developers trade there is always an implicit negotiation in the background. Sometimes that negotiation falls more on user freedoms and sometimes users choose to give up those freedoms to get other things they want. That's just trade, it's been true since before humans had writing. I don't think there's anything unique or special in computing in that regard.
Because something isn't widely adopted doesn't mean that it's dead. Microsoft continues to use and support OLE today, and there's a non-zero number of non-Microsoft-affiliated persons out there using it (I've seen forums with users asking how to do X regarding it).
KParts too is very much alive, as it's a core part of KDE Frameworks, which is used to build Plasma. And I know for a fact it works very well because I'm a Kubuntu user (11 years).
I'm skeptical of these kind of like, self-describing data models. Like, I generally like at proto--because I like IPFS--but I think the whole "just add a lexicon for your service and bickety bam, clients appear" is a leap too far.
For example, gaze upon dev.ocbwoy3.crack.defs [0] and dev.ocbwoy3.crack.alterego [1]. If you wanted to construct a UI around these, realistically you're gonna need to know wtf you're building (it's a twitter/bluesky clone); there simply isn't enough information in the lexicons to do a good job. And the argument can't be "hey you published a lexicon and now people can assume your data validates", because validation isn't done on write, it's done on read. So like, there really is no difference between this and like, looking up the docs on the data format and building a client. There are no additional guarantees.
Maybe there's an argument for moving towards some kind of standardization, but... do we really need that? Like are we plagued by dozens of slightly incompatible scrobbling data models? Even if we are, isn't this the job of like, an NPM library and not a globally replicated database?
Anyway, I appreciate that, facially, at proto is trying to address lock in. That's not easy, and I like their solution. But I don't think that's anywhere near the biggest problem Twitter had. Just scanning the Bluesky subreddit, there's still problems like too much US politics and too many dick pics. It's good to know that some things just never change I guess.
[0]: https://lexicon.garden/lexicon/did:plc:s7cesz7cr6ybltaryy4me...
[1]: https://lexicon.garden/lexicon/did:plc:s7cesz7cr6ybltaryy4me...
Not sure I fully get you... In your example, isn't the problem that nobody cares about this data? So there is no motivation to build a client. Whereas if these were beloved notes or minisites or whatever that got wiped out by the latest acquisition (e.g. see https://bento.me/ shutting down), people would know exactly what those are, and there would be incentive for someone to compete for the userbase.
E.g. Blento (https://blento.app/) is atproto Bento that I only saw a couple of days ago. But the cool thing is that if it shuts down, not only someone else can set it up again (it's open source), but they're also gonna be able to render all of the users' existing content. I think that's a meaningful step forward for this use case.
Yes, there's gonna be tons of stuff on the network that's too niche, but then there's no harm in it either. Whereas wherever there is enough interest, someone can step in and provide the code for the data.
I'm saying it would be a lot simpler to just provide an API to download the data. I don't think it's worth the complexity and network of PDS' to preserve this data in perpetuity, but even if I did, nothing would stop me from using the download API and storing tapes in Iron Mountain or whatever.
I think there's also weird cases where you don't really want things replicated: revenge porn, snuff, CSAM, etc. Like, if someone we're like "yes it's true we lose all data forever if we shut down, but we're not indiscriminately pushing some of the worst content to everyone replicating" I'd take that tradeoff immediately.
I can’t remember how many times I’ve read an article and enjoyed it so much and then looked and saw it was written by Dan ;) always a pleasure !
Thank you!
Is there anything stopping me from backdating my own records? Since the createdAt is just an arbitrary field, I can just write whatever I want in there, right? Is there a way for the viewing application to verify when the record was created (and not modified since), maybe by looking at the mentioned signing?
You can indeed backdate records. Since the application knows when it has first seen (i.e. indexed) your record, it can decide what to do with that information. If there's a difference, the Bluesky app, for example, shows its own indexing time, but also shows a separate panel saying the post is backdated to some other date. Other apps could choose to show something else.
It is possible to create links asserting a specific version by making a "strong ref" which includes content hash.
AT Proto seems very overengineered. We already have websites with RSS feeds, which more or less covers the publishing end in a way far more distributed and reliable than what AT offers. Then all you need is a kind of indexer to provide people with notifications and discovery and you're done. But I suppose you can't sell that to shareholders because real decentralised technology probably isn't going to turn as much of a profit as a Twitter knockoff with a vague decentralised vibe to it that most users don't understand or care about.
Why so much cynicism? The people working there genuinely care about this stuff. Maybe you disagree with technical decisions but why start by projecting your fantasies about their motivations?
RSS is OK for what it does, but it isn't realtime, isn't signed, and doesn't support arbitrary structured data. Whereas AT is signed, works with any application-defined data structures, and lets you aggregate over millions of users in real time with subsecond end-to-end latency.
RSS is signed because it's served over HTTPS which involves an SSL certificate exchange. It is also real time. You fetch the feed and get the most up to date content on it. You can host any XML document you want on your website and there are W3C recommended standards that allow the integration of rich data structures within web pages.
The sole interesting part of ATProto is the aggregation part, which can also be solved in the same way with existing web technologies, that is a web indexer. Google allows you to set up notifications whenever a link is created to your website, which is essentially the unique offering of AT. None of the ideas it presents are new, but they are all incompatible with existing Web technologies. That is why I dislike it. I already have a website. I already have a set of RSS feeds that I follow. I do not want another website hosted on a strange and convoluted service in a bajillion different JSON records when there is a folder of XML on my computer called “my-website” which I can copy to any server and have all I need. Yet, in the name of a federated internet, the BSky folks have herded people away from the real distributed internet and into a crappy Twitter knockoff. Perhaps there are some business-unaware programmers very passionate about the idea of a distributed web and I commend them for that but it already exists and, in every real aspect, they are harming it. If this ATProto thing is successful in its goals, it will cause much inconvenience to me and greatly diminish my ability to publish my works on the internet.
Interesting - I just spent all day on this on an app which I'm using. My architecture is a little different (probably worse).
The app lives on a single OpenBSD server. All user data is stored in /srv/app/[user]. Authentication is done by accessing OpenBSD Auth helper functions.
Users can access their data through the UI normally. Or they can use a web based filesystem browser to edit their data files. Or, alternately, they can ssh into the server and have full access to their files with all the advantages this entails. Hopefully, this raises the ceiling a bit for what power users of the system can accomplish.
I wanted to unify the OS ecosystem and the web app ecosystem and play around with the idea of what happens if those things aren't separate. I'm sure I'm introducing all kinds of security concerns which I'm not currently aware of.
Another commenter brought up Perkeep, which I think is very interesting. Even though I love Plan 9 conceptually, I do sort of wonder if "everything is a file" was a bit of a wrong turn. If I had my druthers, I think building on top of an OS which had DB and blob storage as the primary concept would be interesting and perhaps better.
If anybody cares, it's POOh stack, Postgres, OCAML, OpenBSD, AND htmx
Seems similar to remoteStorage [0]. What happened to that anyway?
[0]: https://remotestorage.io/
This doesn't look similar to me.
remoteStorage seems aimed at apps that don't aggregate data across users.
AT aims to solve aggregation, which is when many users own their own data, but what you want to display is something computed from many of them. Like social media or even HN itself.
remoteStorage is still occasionally getting updates. https://solidproject.org is a somewhat newer, similar project backed by Tim Berners-Lee. (With its own baggage.)
I think of those projects as working relatively well for private data, but public data is kinda awkward. ATProto is the other way around: it has a lot of infra to make public data feasible, but private data is still pretty awkward.
It's a lot more popular though, so maybe has a bigger chance of solving those issues? Alternatively, Bluesky keeps its own extensions for that, and starts walling those bits off more and more as the VCs amp up the pressure. That said, I know very little about Bluesky, so this speculation might all be nonsense.
Yes Solid is another idea that's clearly the right thing, especially now. It might mlnit be easy to convince everyone they need it though, and the economics will be uphill given the trillions in entrenched, incumbent opposition.
Makes me think of Syncthing... I wrote a wrapper for it that attempts to make it easier for people to use:
https://github.com/chapmanjacobd/syncweb-py
https://github.com/chapmanjacobd/syncweb-ui
Unfortunately, mesh storage systems are very different conceptually so it is difficult for people to think about permissions and access. You can bolt on something familiar but then it really limits the usefulness of mesh storage and you may as well just be using HTTP servers.
How does this relate to the SOLID project?
https://solidproject.org/
I'd say some of the worldview is shared but the architecture and ethos is very different. Some major differences:
- AT tries to solve aggregation of public data first. I.e. it has to be able to express modern social media. Bluesky is a proof that it would work in production. AFAIK, Solid doesn't try to solve aggregation, and is focused on private data first. (AT plans private data support but not now.)
- AT embraces "apps describe on their own formats" (Lexicons). Solid uses RDF which is a very different model. My impression is RDF may be more powerful but is a lot more abstract. Lexicon is more or less like *.d.ts for JSON.
This, Local-first Software [1], the Humane Web Manifesto [2], etc. make me optimistic that we're moving away from the era of "you are the product" dystopian enshittification to a more user-centric world. Here's hoping.
[1]: https://www.inkandswitch.com/essay/local-first/
[2]: https://humanewebmanifesto.com/
Indeed. And we can get inspired and involved in bringing about that better world.
Amazing write-up. Owing to your data, a standardized protocol (such as the AT Protocol) is so great! It's like markdown, everything is basically built on files (but cleverly architected so that it works decentralized across the web too!).
I wrote a little along the same lines on «Well Being in Times of Algorithms» at https://www.ssp.sh/blog/well-being-algorithms/ (or HN: https://news.ycombinator.com/item?id=46352747).
Interesting concept for all new social platforms that already live in federated, distributed environments that share communication protocols and communication data formats.
I bet more difficult to push existing commercial platforms to anyhow consider.
That would make marketing tools to manage social communications and posting across popular social media, much easier. Never the less Social Marketing tools have already invented we similar analogy just to make control over own content and feedback across instances and networks.
We still live in a world where some would say BSKY some would say Mastodon is the future… while everybody still has facebook and instagram and youngsters tik tok too. Those are closed platforms where only tools to hack them, not standards persist
But how do you get people to actually want this? This stuff is pretty niche even within tech.
Bluesky is not huge, but 40M users is not nothing either. You don't get people to want this, you just try to build better products. The hope is that this enables us all to build better products by making them more interoperable by default. Whether this pans out remains to be seen.
I also don't think the average user gets the value of the protocol yet. Most of those users were looking for a new, more politically palatable home but with the same features as Twitter. The new generation of apps on the protocol will be vital in showing users what's possible. IMO the two most valuable features at a practical level are: - social graph portability, which might look like having an onboarding experience that bootstraps your community on that app - lexicon cross compatibility, i.e. your data from app A shows up in a contextually relevant spot in app B. Or app B writes records that show up in app A. This is pretty key to get right because it might confuse or anger users if they aren't condition to expect it. Once the average user groks these features though, I'd be surprised if they voluntarily switch back to the standard corpo apps that eventually exit to some company who tries to monetize the shit out of every feature.
I think most people do want this. They want to own their data. If you ask someone if they post on IG, if they should own that, or IG, they'll tell you it's them.
The hard problem IMO is how do you incentivize companies from adopting this since walled gardens helps reduce competition.
We want more control over data that we've created, and more control over data that's about us. I'm not sure either of these concepts align well with "ownership" though. Property and data are concepts that don't mix.
Language nitpicking aside... you subvert the walls of their gardens and aggregate the walled-off data without the walls, so users face a choice not between:
- facebook
- everything else
but instead between
- facebook and everything else
- just facebook
But that approach only works if we can solve the "data I created" problems in a way that doesn't also require us to acknowledges facebook's walls.
I think the opportunity is huge, that a new social media without the current flaws will emerge is inevitable and this is could be part of the implementation solution but sure no real universal success. Think it probably needs to come outside of the us to have any credibility.
“You know how much it sucked when Elon bought Twitter and it became a cesspool? Wouldn’t it have been great if you could easily take your entire post history to some other platform?”
To share is to lose control. You can't undo, even once shared, it can't be undone. You can't retract a published novel. You can't retract a broadcast music or show. What makes you think you can do it over internet?
I don't think my article makes any claims that one can undo sharing. What I'm saying is that we benefit collectively from being able to untether data from applications. It's the same logic as https://stephango.com/file-over-app but applied to the aggregating web applications.
Share and you can't control what everyone else does with the data. Hence, untether is a non concept. In the sense that that data has now lost it's uniqueness. You can untether all you want. Copies of it will be subjected to any kind of tethering and transformation. So, if your point was about control, forget it.
Is there a community or forum where questions about ATProtocol can be asked?
Can you can spread your identity across multiple PDS repositories? Was thinking about creating a PDS that can store binary blobs to download, but not all PDS would like that amount of binary data stored.
UPDATE: found it: https://discourse.atprotocol.community
This was a nice intro to AT (though I feel it could have been a bit shorter)
The whole things seems a bit over engineered with poor separation of concerns.
It feels like it'd be smarter to flatten the design and embed everything in the Records. And then other layers can be built on top of that
Making every record includes the author's public-key (or signature?). Anything you need to point at you'd either just give its hash, or hash + author-public-key. This way you completely eliminate this goofy filesystem hierarchy. Everything else is embed it in the Record.
Lexicons/Collections are just a field in the Record. Reverse looking up the hash to find what it is, also a separate problem.
Yes. SSB and ANProto do this. We actually can simply link to a hash of a pubkey+signature which opens to a timestamped hashlink to a record. Everything is a hash lookup this way and thus all nodes can store data.
I'm not sure I understand your proposal. Do you taking my example (a Twitter post) and showing how it would be stored in your system?
Sure, you'd have something like:
So everything is self-contained. The other features you'd build on top of this basic primitive- Getting the @danabramov username would be done by having some lookup service that does person-key->username. You could have several. Usernames can be changed with the service.. But you can have your own map if you want, or infer it from github commits :)) There are some interesting ideas about usernames about. How this is done isn't specified by the Record
- Lexicon is also done separately. This is some validation step that's either done by a consumer app/editor of the record or by a server which distributes records (could be based on the :type or something else). Such a server can check if you have less than 300 graphemes and reject the record if it fails. How this is done isn't specified by the Record
- Collection.. This I think is just organizational? How this is done isn't specified by the Record. It's just aggregating all records of the same type from the same author I guess?
- Hashes.. they can point at anything. You can point at a webpage or an image or another record (where you can indicate the author). For dynamic content you'd need to point at webpage that points at a static URL which has the dynamic content. You'd also need to have a hash->content mapping. How this is done isn't specified by the Record
This kind of setup makes the Record completely decoupled from rest of the "stack". It becomes much more of independent moveable "file" (in the original sense that you have at the top) than the interconnected setup you end up with at the end.
I have a few questions:
- How do you rotate keys? In AT, the user updates the identity document. That doesn't break their old identity or links.
- When you have a link, how do you find its content? In AT, the URL has identity, which resolves to hosting, which you can ask for stuff.
- When aggregating, how do you find all records an application can understand? E.g. how would Bluesky keep track of "Bluesky posts". Does it validate every record just in case? Is there some convention or grouping?
Btw, you might enjoy https://nostr.com/, it seems closer to what you're describing!
1. It's an important problem, but I think this just isn't done at the Record layer. Nor can you? You'd probably want to do that on the person-key->username service (which would have some log-in and way to tie two keys to one username)
2. In a sense that's also not something you think about at the Record level either. It'd be at a different layer of the stack. I'll be honest, I haven't wrapped my head entirely around `did:plc`, but I don't see why you couldn't have essentially the same behavior, but instead of having these unique DID IDs, you'd just use public keys here. pub-key -> DID magic stuff.. and then the rest you can do the same as AT. Or more simply, the server that finds the hashed content uses attached meta-data (like the author) to narrow the search
Maybe there is a good reason the identity `did:plc` layer needs to be baked in to the Record, but I didn't catch it from the post. I'd be curious to here why you feel it needs to be there?
3. I'm not 100% sure I understand the challenge here. If you have a soup of records, you can filter your records based on the type. You can validate them as they arrive. You send your records to the Bluesky server and they validates them as they arrive.
2. The point of the PLC is to avoid tying identity to keys, specifically for the point that if you lose your keys, you lose your identity. In reality, no body wants that as part of the system
3. The soup means you need to index everything. There is no Bluesky server to send things to, only your PDS. Your DID is how I know what PDS to talk to to get your records
I guess what I'm asking is why does connecting multiple public keys to one identity something that has to be part of the Record? Why is that not happening as a separate process?
I'll try to read up more on this topic! I do feel I'm missing some pieces here
We can have both a directory and use content addressable storage and give people the option of using their own keypairs. They are not mutually exclusive. Bluesky chooses to have a central directory and index.
what would you do about the at-uri?
at://hash at://pub i guess I don't know why we need at:// that seems like something we'd need in Beaker Browser.
After reading the docs at https://atproto.com/specs/at-uri-scheme and building applications that integrate with the Bluesky API, it’s clear why this format is useful for interacting with their system. For developers working outside the ATMosphere, however, it may feel less familiar compared to more conventional REST API patterns.
Well, when you figure out why at:// exists and why your hand waving away it's role in your "I guess" makes it impossible... we can chat then
Until then, you are just slopping about like an ai
dude you represent yourself as a tony the AI worm.
I'll do an:// urls if you bring yourself and your code agent to myproto for an afternoon.
You mention having a "self" record in the app.bsky.actor.profile lexicon to store profile information, is there any reason to have more records of that type in your repository?
I've seen a few people make other ones when examining their accounts with pdsls, but they seem to be there for "just because I can" reasons.
Different apps can have different notions of a profile so you'd probably have one per app.
How does this system determine the amount of likes a post has? Since there is no back reference on a post to people who have liked it, don't you have to iterate over every single person, iterate over their likes, to see if one of them is the post you are viewing, and add all them up?
In the ATProto architecture, this function is handled by the AppView, which monitors the full network and produces the corresponding aggregates.
There's also software called Constellation[0] from microcosm that exclusively collects "backlinks", which are items which link to a certain thing. e.g. follow records for an account, reposts/replies/likes on a post. If you click on the Backlinks tab when examining something on pdsls.dev you can see a list of them.
It can be self hosted and is much lighter than running a full appview. The author has a public endpoint for it that's keeping up with the whole network running on a single raspberry pi on his home network.
0: https://www.microcosm.blue/
Yes. I describe this in this part of the article: https://overreacted.io/a-social-filesystem/#:~:text=One%20ch...
It's basically event sourcing. You listen to the data you care about from the network and update the local index (DB). There are also tools like Tap (https://docs.bsky.app/blog/introducing-tap) that do the plumbing work and let you backfill automatically.
Ah, I'm sorry, I somehow skipped over that bit entirely. Need more coffee, I think
I have always thought open file format > open source. My ideal web everyone has their own web file storage (get from anywhere e.g. email provider) and web apps use that to store things. Team collab etc. built on top of that e.g. sharing a file means share ann accept edits type flow. Everyone owns their file.
It seems like the biggest downside of this world is iteration speed.
If the AT instagram wants to add a new feature (i.e posts now support video!) then can they easily update their "file format"? How do they update it in a way that is compatible with every other company who depends on the same format, without the underlying record becoming a mess?
That's a great question!
Adding new features is usually not a problem because you can always add optional fields and extend open unions. So, you just change `media: Link | Picture | unknown` to `media: Link | Picture | Video | unknown`.
You can't remove things true, so records do get some deprecated fields.
Re: updating safely, the rule is that you can't change which records it would consider valid after it gets used in the wild. So you can't change whether some field is optional or required, you can only add new optional fields. The https://github.com/bluesky-social/goat tool has a linting command that instantly checks whether your changes pass the rules. In general it would be nice if lexicon tooling matures a bit, but I think with time it should get really good because there's explicit information the tooling can use.
If you have to make a breaking change, you can make a new Lexicon. It doesn't have to cause tech debt because you can make all your code deal with a new version, and convert it during ingestion.
That's true if you define the problem as "does my parser crash" and not whether the app is perceived as working correctly. If some platform adds support for video posts, then the next thing that happens is people start making posts that are only video. Meaning that in every other client, users see what appears to be an entirely empty post. Which will be considered a bug.
This is the core argument of Moxie's seminal essay, The Ecosystem Is Moving:
https://signal.org/blog/the-ecosystem-is-moving/
One of the controversial things we did with Signal early on was to build it as an unfederated service. Nothing about any of the protocols we’ve developed requires centralization; it’s entirely possible to build a federated Signal Protocol-based messenger, but I no longer believe that it is possible to build a competitive federated messenger at all.
That was written in 2016 but it was true then and continues to be true today. Users reject federated open platforms because the coordination costs mean they don't move as fast as proprietary centralized platforms, and they often appear broken even if technically working as designed.
Nothing about that analysis is unique to social media. It is also true of file formats. OpenOffice never took off because new features got added to Office first, so files that used those features would open in semi-corrupted ways in OpenOffice. The fact that OO represented things internally using open unions didn't matter at all.
I disagree that Bluesky is in conflict with The Ecosystem Is Moving. In contrast to most decentralized/distributed protocol projects they've managed to maintain control of almost all of their infrastructure with the exception of the personal data servers (pdses) of which they control 99.01%[1]
Almost all ATProto apps just fetch posts by handle => did:plc => post-type aka "lexicon", so they depend on what Bluesky decides to give them. If someone were to introduce unknowns into the flagship product's "lexicon" they could fix that at the API or Indexing level before shipping this data to the apps that depend on their API.
An actually decentralized network would have to overcome Moxie's criticism of the ecosystem. Can it be done? We'll keep trying.
[1] https://arewedecentralizedyet.online/
Well, this doesn't prevent the "flagship" app from shipping things and doesn't slow it down. So it's at least not slowing down development which is the argument the parent post was making.
I've actually observed the exact opposite thing. Since Bluesky is open source, it's often visible when developers start working on a feature. And they often check in lexicon changes early on. As a result, there's been a few cases where third party client actually added support for these features earlier than the official one since they already knew the shape of the data.
This wouldn't always work, of course. Yes, if you're developing an app or a client, you better keep up with the ecosystem. But the landscape is competitive and there is no cost to switching. So if something falls behind, you can use something else.
Moxie's argument is that even if something has a flagship app, this doesn't help because if you use a new feature and then your friend complains that they can't see what you posted, the experience is just that the flagship app itself is broken. People don't experience this as, oh well, my friend should just have picked a better client. They experience it as, that's annoying, the video feature doesn't work reliably on Y but it always does on X.
An extreme example of this is WhatsApp and emojis. WhatsApp doesn't use the operating system's text rendering APIs to draw emojis, instead Meta license the Apple emoji font and draw the characters themselves. That's because if you do emoji the standards based, open way you get these problems:
• People use visual puns that only make sense if emojis look a certain way, without realizing they might look very different to other people.
• People use new emoji, without realizing that old operating systems can't draw them.
The experience in both cases is that it's simply broken and buggy. Version skew can kill platforms, which is why the most successful platforms today restrict which clients can connect and forcibly expire them on a regular basis.
BTW I don't think it's worth generalizing from Bluesky. Bluesky is an X clone whose unique competitive advantage is censoring conservatives even more aggressively than Twitter itself once did. It has no technical edge so they can develop the site as open source without it being a concern to them - they don't care if they leak what they're doing because technical innovation isn't a part of their brand to begin with - and the AT protocol clearly doesn't matter to more than a tiny fraction of its users. The point you're making in the essay is general, but ends up feeling overfit to Bluesky.
There is always one party "in control" of the lexicon and its canonical version.
I think it's important to distinguish this from the "every client adds their own features" thing. Technically yes, each app can add their own things to the open union that they support better. But it's also on each implementer's to consider how this would affect UX in other clients (e.g. if you add your own embed type, it seems reasonable to also prepopulate a link embed that acts as fallback). The problems you're describing are real, but I think we should give a bit more credit to the app builders since they're also aware that this is a part of their user experience design.
But still, whoever "owns" the lexicon says what's canonical. Then yes, some other software might not catch up to what's canonical but that's similar to what's happening with any platform that supports multiple clients today. Unless your outlook is that alternative clients in general are not competitive for this reason. I think that's a grim outlook, and if that were true, services wouldn't go to extra lengths to intentionally shut down their APIs, which so has been the trend with every network.
I think in longer term the bet is that the benefits unlocked by interop and a more competitive product landscape will become clearer to end users, who will be less interested in joining closed platforms and will develop some intuitions around that. This would not happen soon, so until then, the bet is that interop will allow creating better products. And if that doesn't happen, yes, it's pretty hard for open to compete.
Well, I never personally formed a strong opinion on Moxie's take, although I do understand it. Basically yes his outlook is that any service that doesn't actively ban alternative clients will be outcompeted by those that do.
The reason is that if alt clients are possible then some fraction of the userbase will adopt them. And if some users adopt them that means the experience of other users of the service gets worse, because new features become unreliable and flaky. You think you understand what another person sees and can do, but you don't, and this leads to poor experiences.
Viewed another way the client is an integrated part of the platform and not something that you can enable users to change freely, any more than they could be allowed to change the servers freely. We don't allow the latter because users would do things that broke the service, and so it is also for the former.
Empirically Moxie seems to be correct. Of the 90s era open protocols the only ones that survive are the web and email. The web survives but it's never been truly open in an extension sense - the definition of HTML has always been whatever the dominant browser of the era accepts, and this has become moreso over the time not less. There are no more plugin APIs for instance. SMTP survives, barely, because it's the foundation of internet identity. But many people even in corporate contexts now never send an email. It's all migrated to Slack or Teams. And if you look carefully at the Slack API it's not possible to make a truly complete alternative client with it, the API is only intended for writing bots.
This is grim but I'm not sure it's false and I'm not sure it can be changed. Also, Moxie's essay ends on a positive note. He observes that competition between mobile social networks does still work well despite the lack of federation, because they coalesced around using the user's phone number as identity and address book as the friends list, so you can in fact port your social network to a different network just by signing up. The notification center in the OS provides the final piece of the puzzle, acting as a unified inbox that abstracts the underlying social network.
This is rather mobile specific but seems basically correct to me. So that suggests the key pillar isn't file formats or protocols but ownable identity. It works because telcos do the hard work of issuing portable identities and helping people keep them, and ownership can be swiftly verified over the internet.
All good points.
Maybe I’m optimistic for no good reason. I feel there’s something slightly different at play here but I struggle to put my finger on it.
Overall I agree with the take that a client for somebody else’s product is usually degraded experience. You want to deploy features vertically, and even existence of worse clients is a threat to that. I think one exception to that is when the main client is intentionally worse for the sake of other goals. Eg lots of ads. Or just the company being (or becoming) incompetent at making a client. Then alternative clients inject some new life and competition into the product. But if the product owner is good at making clients, they probably don’t benefit from fragmentation.
And still it feels somehow different to me. I think a part of it is that the space itself is vastly expanded in AT. We’re not just talking about a vertical app with a few offshoot clients that do the same thing but poorly. I mean, that still exists, but the proposition is that things can reach way beyond the original product. Products can blend. When products blend, there’s always this inherent coordination problem. Those who make multiple vertical products have to deal with similar things. At some point you have to stagger rollouts, support features partially or in a limited way, or show a simple version and direct to main app for more. Think Threads “bleeding through” Instagram, different kinds of cross-posting that they experiment with between apps. This is valuable but normally only monopolies can build meaningful cross-product bleeding because otherwise you have to negotiate API agreements and it’s just too risky to build on someone else’s turf.
What AT changes is that every product can cross-bleed into every other product. My teal.fm plays show up on my Blento page. So yes, you do have this fragmentation, but you also get something in return. It’s not fragmentation between the main way to show a thing and a couple offshoots, but between a web of products. And there is always a canonical schema so it doesn’t descend into chaos. I think there’s a “there” there.
I actually agree the key pillar is identity. It’s what AT URIs start with identity first. I think one way to look at AT is that the “contact list” analogy is right, but now that we’ve chosen the identity layer, we might as well put all our data into the contact list.
Riffing off the identity thing, one service I wanted for a while is something that issues X.509 certificates based on verified phone numbers. Phone numbers are a pretty great identity, perhaps the most successful private sector identity system ever, but they're expensive and annoying to verify, and the verification isn't portable across systems. A CA that did SMS verification and then gave you a certificate you could use with S/MIME or bind to passkeys or just use to sign software/documents in general, would democratize stable cryptographic identity. People generally can't handle key management directly, it's too easy to lose keys, but issuing transient keys tied to a phone number is much more palatable.
And PNs have got good features you want in general:
• Can have >1 of them if you want.
• Anonymous if you want.
• Not tied to any specific provider due to number portability laws.
• Hard to lose; phone companies will accept govt issued ID to get your account back if you lose your SIM and it's tied to a contract.
• Verifiable over the internet.
The only problem with them is they don't yield asymmetric keypairs.
The difficulty is business model. The people who want to consume such certificates are people who don't want to pay to verify numbers directly, but users don't want to pay either. So who pays.
Are these just guidelines or is this enforced in some way? I guess readers could validate and skip anything that doesn't match their schema.
Most apps reading records will validate a record against the schema for that type. e.g. there's nothing stopping you from making a app.bsky.feed.post record with more than 300 graphemes in the "text" field, but that post won't appear in the "official" app/website because it fails schema validation.
Similarly, there's nothing stopping you from adding another field in your post. It'll just get ignored because the app you're using doesn't know about it. e.g. posts bridged from mastodon by bridgy have an extra field containing the full original post's text, which you can display in your app if desired. reddwarf.app does this with these posts.
Here's the schema for bluesky posts, if you're interested: https://pdsls.dev/at://did:plc:4v4y5r3lwsbtmsxhile2ljac/com....
I'm tempted to make an alternative app that shows only posts with more than 300 characters. :-)
Yeah, look for this part in the the article:
Lexicon validation works the same way. The com.tumblr in com.tumblr.post signals who designed the lexicon, but the records themselves could have been created by any app at all. This is why apps always treat records as untrusted input, similar to POST request bodies. When you generate type definitions from a lexicon, you also get a function that will do the validation for you. If some record passes the check, great—you get a typed object. If not, fine, ignore that record.
So, validate on read, just like files.
I know this is somewhat covered in another comment, but, the concepts described in the post could have been reduced quite a bit, no offense Dan. While I like the writing generally, I would consider writing and then letting it sit for a few days, rereading, and then cutting chaff (editing). This feels like a great first draft but without feedback, and could have greatly benefited from an editing process, and I think using the argument that you want to put out something for others to take and refine isn’t really a strong one… a bit more time and refinement could have made a big difference here (and given you have a decently sized audience I would keep in mind).
From my perspective, there is no chaff. I've already the read the entire thing from top to bottom over 20 times (as I usually do with my writing), I've done several full edit passes, and I've removed everything inessential that I could find. The rest is what I wanted to be included into this article.
I know my style is verbose but I try to include enough details to substantiate the argument at the level that I feel confident it fully stands for itself. If others find something useful in it, I trust that they can riff on those bits or simplify.
For the commenters who have made similar comments, I'd be curious to hear what they think could be cut. I suspect different readers will have different opinions on this, which means it's probably a good thing you didn't make cuts.
There is not much actionable here, as well intentioned as your comment is.
It's like saying this MR could use some work but not citing a specific example.
The more I read and consider Bluesky and this protocol, the more pointless -- and perhaps DANGEROUS -- I find the idea.
It really feels like no one is addressing the elephant in the room of; okay, someone who makes something like this is interested in "decentralized" or otherwise bottom-up ish levels of control.
Good goal. But then, when you build something like this, you're actually helping build a perfect decentralized surveillance record.
This why I say that most of Mastodon's limitations and bugs in this regard (by leaving everything to the "servers") are actually features. The ability to forget and delete et al is actually important, and this makes that HARDER.
I'm just kind of like, JUST DO MASTODONS MODEL, like email. It's better and the kinks are more well thought about and/or solved.
Author here. I think it's fair to say that AT protocol's model is "everyone is a scraper", including first party. Which has both bad and good. I share your concern here. For myself, I like the clarity of "treat everything you post as scraped" over "maybe someone is scraping but maybe not" security by obscurity. I also like that there is a way for me to at least guarantee that if I intentionally make something public, it doesn't get captured by the container I posted it into.
This seems like tensions between normal/practical and “opsec” style privacy thinking… Really, we can never be sure anything that gets posted on the internet won’t be captured by somebody outside our control. So, if we want to be full paranoid, we should act like it will be.
But practically lots of people have spent a long time posting their opinions carelessly on the internet. Just protected by the fact that nobody really has (or had) space to back up every post or time to look at them too carefully. The former has probably not been the case for a long time (hard drives are cheap), and the latter is possibly not true anymore in the LLM era.
To some extent maybe we should be acting like everything is being put into a perfect distributed record. Then, the fact that one actually exists should serve as a good reminder of how we ought to think of our communications, right?
Exactly. Anything that's ever been public on the internet is never really gone anyways, and it's unsafe to assume so. This is similar to publishing a website or a blog post. Plus, from a practical (non-opsec) point of view, you can delete items (posts, likes, reposts, etc.) on ATProto, and those items will disappear from whatever ATProto app you are using - usually even live. You need to dive into the protocol layer to still see deleted items.
Your first statement is good advice, but is factually WRONG. Stuff does go away.
We know this from the other side of it, stuff that we WANTED to keep, but couldn't. Google deleting blogs, people not maintaining their own personal media records, etc.
Now. What is a good forward GIVEN THAT we do know that stuff CAN go away? My gut is that a good balance of "making personal stuff available to people who want to see it again" vs. "surveillance state" could make use of the unreliability of "the net." And that ATProto's "perfection" isn't all that helpful.
Your last point is one that I used to be very strongly favor of, and today?
Nooooooooooo. No. No. No.
It's not going to happen and we shouldn't even consider it. Seriously. This thing we are doing here, which is "connecting people to each other," those forces for MANY will be far more powerful than "let me stop and think about the fact that this is forever." I just don't think we are wired for it, we're wired for a world in which we can just talk?
I think it's better to try to engineer some specific counter to "everything is recorded all the time" (or, as in here, not try to usher it into existence even more) than to try to say "welp, everything is recorded all the time, better get used to it."
It would be nice to engineer a way around this, but I don’t see it. Fundamentally if we want to be able to talk to random people, we’ll have to expect that some might be capturing communications, right?
I think there's a massive difference between "some might be capturing" and "reliable, e.g. hashed records"
Again, there may be some VALUE in the fact that "some might be capturing" WOULD BE inherently unreliable. The ability to "muddy the record" could make some SAFER.
It's true that Mastodon is somewhat better if you don't want to be found, though it's hardly a guarantee. From a "seeing like a state" perspective, Bluesky is more "legible" and that has downsides.
But I think there's room for both models. There are upsides to more legibility too. Sometimes we want to be found. Sometimes we're even engaging in self-promotion.
Also, I'll point out that Hacker News is also very legible. Everything is immutable after the first hour and you can download it. We just live with it.
>helping build a perfect decentralized surveillance record
a record of what? Posts I wish to share with the public anyway?
It's not about the access, it's about the completeness. Imagine this paradigm takes off (I hope it does!), everyone has their own PDS and finally owns their data. Social apps link into their PDS to publish and share data exactly as they're supposed to.
Well now someone's PDS is a truly complete record of their social activity neatly organized for anyone that's interested. It's not a security issue, after all the data was still public before, but the barrier to entry is now zero. It's so low that you can just go to stalker.io, put in their handle, and it will analyze their profile and will print out a scary accurate timeline of their activity and location leveraging AI's geoguesser skill.
If that's your threat model, then I think the way forward is to maintain separate identities. There are trade-offs there also of course: fragment yourself too much and the people who trust you will now only trust a portion of what you have to say... unless you have the time and energy to rebuild that trust multiple times.
Of course that's the same with the web we have today, the only difference is that you get control over which data goes with which identity rather than having that decision made for you by the platform boundaries.
The completeness AND the reliability!
Consider that a muddy, "oh a bunch of different servers downloaded this guys history, but it's different on one from the other" could be ORDERS OF MAGNITUDE SAFER from a surveillance perspective.
That is how it works, but people shouldn't be posting their location or sensitive information publicly if they don't want it exposed like that. That's basic opsec. Private data is currently being worked on for ATProto and will hopefully begin existing in 2026.
> people shouldn't be posting their location or sensitive information publicly if they don't want it exposed like that
They shouldn't, but they still could: accidentally paste in the wrong browser tab; have been stupid when they were 12 years old; have gotten drunk; or a number of other things.
This is a line of thinking that just supposes we shouldn’t post things on the internet at all. Which, sure, is probably the right move if you’re that concerned about OPSEC, but just because ActivityPub has a flakier model doesn’t mean it isn’t being watched
what if I want to publish something publicly on the internet though
Maybe some could want to publish something publicly but anonymously?
You can do that already — just don't post under an account that has your real identity
In theory it should be possible to allow users to upload ciphertext that they can then share a decryption key with their intended audience. I believe atproto has dissuaded against this with the argument that ciphertext shouldn't be in public view, but this seems to hinge on the idea that the cipher is insecure, or will be in the future. I don't see why using a post-quantum encryption scheme shouldn't provide the appropriate security, which may still not be foolproof, but it certainly would make indexing the data much more difficult
When it comes to the internet, tech is law. There is no way to publicly share something and maintain control over it. Even on the Fediverse, if either a client or server wants to ignore part of the protocol or model, it can. Like a system message to delete particular posts for anti-surveillance reasons can simply be ignored by any servers or clients that were designed/modified for surveillance. Ultimately the buck lies with the owner of some given data to not share that data in the first place if there's a chance of misuse.
Shouldn't the ability to forget and delete content that was ever public on the internet be considered fictional anyway?
Why would we need to store the createdAt value in a file? The filesystem already stores this information. We could just store the text which would mean no Json would be needed.
I'm using "filesystem" a bit loosely here.
The important parallel I was going for was "file format" as interface between apps (= lexicons being an interface between social apps).
If you want details on the actual data structures, check https://atproto.com/specs/repository.
Unpopular opinion: this should be done with xml, not json. XML can have types, be self describing, and be extended (the X in XML).
That said it’s a very elegant way to describe AT protocol.
I'd be curious to see what that would look like!
nostr protocol and the client/relay model is one simple way to separate apps (clients) from the data (relays)
How do people view AT Protocol vs Nostr? Why choose one over the other? Which has a better chance at replacing X?
I found this helpful: https://shreyanjain.net/2024/07/05/nostr-and-atproto.html
Git is the API.
Github/Gitlab would be a provider of the filesystem.
The problem is app developers like Google want to own your files.
Ironically, DID is the perfect vehicle for age verification.
As someone who explicitly designed social protocols since 2011, who met Tim Berners-Lee and his team when they were building SOLID (before he left MIT and got funded to turn it into a for-profit Inrupt) I can tell you that files are NOT the best approach. (And neither is SPARQL by the way, Tim :) SOLID was publishing ACLs for example as web resources. Presumably you’d manage all this with CalDAV-type semantics.
But one good thing did come out of that effort. Dmitri Zagidulin, the chief architect on the team, worked hard at the W3C to get departments together to create the DID standard (decentralized IDs) which were then used in everything from Sidetree Protocol (thanks Dan Buchner for spearheading that) to Jack Dorsey’s “Web5”.
Having said all this… what protocol is better for social? Feeds. Who owns the feeds? Well that depends on what politics you want. Think dat / hypercore / holepunch (same thing). SLEEP protocol is used in that ecosystem to sync feeds. Or remember scuttlebutt? Stuff like that.
Multi-writer feeds were hard to do and abandoned in hypercore but you can layer them on top of single-writer. That’s where you get info join ownership and consensus.
ps: Dan, if you read this, visit my profile and reach out. I would love to have a discussion, either privately or publicly, about these protocols. I am a huge believer in decentralized social networking and build systems that reach millions of community leaders in over 100 countries. Most people don’t know who I am and I’m happy w that. Occasionally I have people on my channel to discuss distributed social networking and its implications. Here are a few:
Ian Clarke, founder of Freenet, probably the first decentralized (not just federated) social network: https://www.youtube.com/watch?v=JWrRqUkJpMQ
Noam Chomsky, about Free Speech and Capitalism (met him same day I met TimBL at MIT) https://www.youtube.com/watch?v=gv5mI6ClPGc
Patri Friedman, grandson of Milton Friedman on freedom of speech and online networks https://www.youtube.com/watch?v=Lgil1M9tAXU
To be clear, I'm using files in a relatively loose sense to focus on the "apps : formats are many-to-many" angle. AT does not literally implement a full filesystem. As the article progresses, I restrict some freedoms in the metaphor (no directories except collections, everything is JSON, etc). If you're interested in the actual low-level repository format, it is described here: https://atproto.com/specs/repository
FYI the CTO of Bluesky was an early dev of Secure ScuttleButt
Losing private keys is much more common than losing domains.
Yes, which is why by default, key management is done by your hosting. You log into your host with login/password or whatever mechanism your host supports.
Adding your own emergency rotational key in case your hosting goes rogue is supported, but is a separate thing and not required for normal usage. I'd like this to be more ergonomical though.
I like the write-up of this idea. It's well presented. But I'd change one aspect: "We could leave author: 'dril' in the JSON but this is unnecessary too." - kind of. What the post lacks is the record of the identity at the time. What the user's username and the avatar was at the time can change the meaning of the post entirely. To really preserve the message, you need to reference what the displayed identity was used to post it - not just the account id.
There's a number of famous accounts that do it continuously. For example popehat today is "Fucking Bitch Hat" but will change to something else soon that may be related to the current events.
I think most people's mental model is that they should be able to change their handle / display name / avatar freely, and their posts would display the new versions. So those aren't a part of the post itself.
That said, you could create an AT app that displays a version of the post using the profile at the time. You'd just need to index all profile changes into a local database, and then query that database for the "profile at that time" to display the post. So what you're describing is possible—it just requires a different aggregation. The source of truth, however, should be denormalized and reflect most recent data.
I did a double take at "DID as identity" because Dissociative Identity Disorder shares the same acronym
The premise of this article resonates so much with me! I didn't see the angle on ATproto coming, and frankly this description of it is the first that makes me want to dig into it a bit.
The issue of "file-less" computing has been bothering me a lot. It's worst on iOS, where apple are really pushing hard to have users never ever think in "files". Closely followed by MacOS and only then android, imho. "That thing over there? That's not a file! That's a photo! Very different thing, that. No, you can't use $app to view it or share it, sorry. And if you want to copy it, you have to go through our export functionality which is buggy and strips all kind of info and generally works best through iCloud.
My mind's insistance that behind all the flashy apps must he a backing store, and that it's most likely file based, makes it even more infuriating when trying to get unprocessed photos off a relative's iPhone or such.
yeah yeah yeah, everyone get on the AT protocol, so that the bluesky org can quickly get all of these filthy users off of their own servers (which costs money) while still maintaining the original, largest, and currently only portal to actually publish the content (which makes money[0]). let them profit from a technical "innovation" that is 6 levels of indirection to mimic activity pub.
if they were decent people, that would be one thing. but if they're going to be poisoned with the same faux-libertarian horseshit that strangled twitter, I don't see any value in supporting their protocol. there's always another protocol.
but assuming I was willing to play ball and support this protocol, they STILL haven't solved the actual problem that no one else is solving either: your data exists somewhere else. until there's a server that I can bring home and plug in with setup I can do using my TV's remote, you're not going to be able to move most people to "private" data storage. you're just going to change which massive organization is exploiting them.
I know, I know: hardware is a bitch and the type of device I'm even pitching seems like a costly boondoggle. but that's the business, and if you're not addressing it, you're not fomenting real change; you're patting yourself on the back for pretending we can algorithm ourselves out of late-stage capitalism.
[0] *potentially/eventually
> until there's a server that I can bring home and plug in with setup I can do using my TV's remote, you're not going to be able to move most people to "private" data storage
Quite some BSky users are publishing on their own PDS (Personal Data Server) right now. They have been for a while. There are already projects that automate moving or backign up your PDS data from BSky, like https://pdsmoover.com/
Microblogging is also the least interesting part of the ATProto ecosystem. I've switched all my git hosting over to https://tangled.org and am loving it, not least of which is that my git server (a 'knot' in Tangled parlance) is under my control as a PDS and has no storage limits!
Is it as easy for other people to read as a Github repo? Want to share?
yeah, tangled seems like a pretty well-designed piece of tech. I've never used it, myself, but I did an audit and found that it's not only analogous to github as far as UX, but it also includes features like CI/CD, which other public/social repo servers have struggled with.
only reason I backed away from it is that when the bsky team had a big "fuck the users" moment, the user purporting to be the tangled founder was happy to cheer them on. so between having to use AT proto, and assuming that the tangled dev doesn't really disagree with bsky's "fuck the users" sentiment, I moved on. but, obviously, whiny moral grandstanding is irrelevant to whether or not someone made a good product. if you've got a use for it, I'd certainly recommend giving it a try!
Tangled founder here; it's just as easy! For example, here's the entire Tangled codebase monorepo: https://tangled.org/tangled.org/core — you can clone this directly as you would a git repo anywhere else.
Looks nice!
New user sign up is a bit wonky. It asked for an email, login and password, then it's asking for a bsky sign-in too? This seems a little weird.
(Minor nit: for some reason, Google didn't auto-suggest a strong password for the password field.)
Then I got to the screen where it asks for full read-write access to my PDS and stopped there. It's kind of a lot to ask! I believe this is Bluesky's fault, but I don't think I can really use third-party bluesky apps until they implement finer-grained permissions.
Fine-grained permissions are already shipped btw, though the documentation could be better.
yeah, I was one of them. developers are not the endgame, though. true social media needs people who are not going to do anything more complicated than "go to website, sign up". there's no world where setting up your own pds is that simple without an organized piece of software to do that kind of thing.
personally, I could probably get behind recommending something like umbrel[0], if it included something like a "include a pds" option during config. but even that is asking for a lot of mind-share for a non-tech user. it would take a super smooth setup process for that to be realistic. point is, though, I'm not saying it can't be done; I'm saying no one is doing it and what people are doing is not getting the job done for wider adoption.
[0] https://umbrel.com/ *and, naturally, at this point, I'd prefer they include something that isn't based on AT proto for social publication. I wouldn't mind if they had both, but just an AT proto implementation wouldn't attract me.
and you expect these “go to website, sign up” people to take the extra step to select a provider for repoing data? these people can barely pick a mastodon instance, what sort of data ownership integration work do you expect? it’s a consideration that’s more niche than the current status quo. unless you’re fine with people defaulting to onedrive or similar.
no, I expect them to go to a store to buy the same product that their friend bought. then I expect them to use that product by scanning a QR code that comes up on their TV, and then registering an account using the site linked by that code (or, just using their tv's remote to sign up, with an on-screen keyboard, or whatever).
not "a server so simple, anyone could host it"; "a set top box that gives me a private social media storage, network data storage, secure external connections, and effortless integration with all of my other iot devices". note that the former requires you to know what hosting is, while the latter only requires that you know what you want to do, without having to understand the details of how its done.
What's a PDS?
Personal Data Server. See https://atproto.com/guides/glossary#pds-personal-data-server
https://docs.bsky.app/docs/advanced-guides/atproto
>that the bluesky org can quickly get all of these filthy users off of their own servers (which costs money)
That's not correct, actually hosting user data is cheap. Most users' repos are tiny. Bluesky doesn't save anything by having someone move to their own PDS.
What's expensive is stuff like video processing and large scale aggregation. Which has to be done regardless of where the user is hosting their data.
come on, man, let's be real. you're talking modern, practical application; I'm talking reasonable user buy in at big boy social media levels. the video hosting IS what I'm talking about being expensive. you think bsky is going to be successfully while ignoring the instagram crowd forever? what are we doing here?
bsky saves the video processing and bandwidth by not hosting that content on bsky. it's a smaller problem, but in a large enough pool, images become heavy, too. and, either way, the egress of that content is expensive if you're doing it for the entire world, instead of letting each individual's computer (their pds) do it.
I'm happy to admit that text is cheap and bsky isn't looking to offload their data as it stands now. but let's be honest about the long term, which is what my original comment takes aim at.
I still don't think this is correct. The Bluesky app always processes video, whether you're self-hosting or not. The personal data serves stores the original blob, but the Bluesky's video service will have to pick it up and transcode it (to serve it from CDN) either way.
Also, this:
>let them profit from a technical "innovation" that is 6 levels of indirection to mimic activity pub.
is also wrong because AT solves completely different problems. None of the stuff I wrote about in the post can be solved or or is being solved by ActivityPub. Like, AP is just message passing. It doesn't help with aggregation, identity, signing, linking cross-application data, any of that.
right on, man, that must be the only way transcoding can be done and completely future proof so it will never change to let the user transcode their own damn content. I get it; you're frustrated so you're nitpicking. funny how dang doesn't swoop in to tut tut you for not steelmanning instead of strawmanning.
in any case, you're completely right about the activity pub comment. that was absolutely mockery and not actually a complaint. artistic license and all that. god forbid we express ourselves. but sure, I can recognize that AT proto is useful in that it provides mechanisms we didn't really have before. that said, it's not novel (just new) and it's not irreplaceable. LIKE I SAID: there's always another protocol.
any time you want to actually address the point of the comments, I'd be happy to get your take on why it's fine, actually, for the CEO to imply she doesn't have to care what users think as long as they aren't paying her. but if you're not ready to have the real conversation, I'll let you be satisfied with whatever other potshots you want to take at my reasonable indignation.
agree! Social-media contributions as files on your system: owned by you, served to the app. Like .svg specifications allows editing in inkscape or illustrator a post on my computer would be portable on mastodon or bluesky or a fully distributed p2p network.
[dead]
> When great thinkers think about problems, they start to see patterns. They look at the problem of people sending each other word-processor files, and then they look at the problem of people sending each other spreadsheets, and they realize that there’s a general pattern: sending files. That’s one level of abstraction already. Then they go up one more level: people send files, but web browsers also “send” requests for web pages. And when you think about it, calling a method on an object is like sending a message to an object! It’s the same thing again! Those are all sending operations, so our clever thinker invents a new, higher, broader abstraction called messaging, but now it’s getting really vague and nobody really knows what they’re talking about any more.
https://www.joelonsoftware.com/2001/04/21/dont-let-architect...
Hi, so I generally actually agree with you and your criticisms of this blog post (in your thread with the author). I think there's something pretty true in the blog post you shared from Joel (true in that it applies to more than just the software world) and looked at some of his more recent posts.
https://www.joelonsoftware.com/2022/12/19/progress-on-the-bl...
This one in particular reads similar to what this comment section is about, it looks like Joel is basically becoming an architecture astronaut himself? Not sure if that's actually an accurate understanding of what his "block protocol" is, but I'm curious to hear from you what you think of that? In the 25 years since that post, has he basically become the thing he once criticized, and is that the result of just becoming a more and more senior/thinker within the industry?
"Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something."
https://news.ycombinator.com/newsguidelines.html
Author here! I grew up reading Joel's blog and am familiar with this post. Do you have a more pointed criticism?
I agree something like "hyperlinked JSON" maybe sounds too abstract, but so does "hyperlinked HTML". But I doubt you see web as being vague? This is basically web for data.
> Do you have a more pointed criticism?
Sure.
After taking the time to re-read the article since I initial posted my (admittedly shallow) dismissal, I realized this article is really a primer/explainer for the AT protocol, which I don't really have enough background in to criticize.
My criticism is more about the usefulness of saying "what if we treated social networking as a filesystem": which is that this doesn't actually solve any problems or add any value. The idea of modeling a useful thing (social media)[0] as a filesystem is generalizing the not-useful parts of it (ie. the minutia of how you actually read/write to it) and not actually addressing any of the interesting or difficult parts of it (how you come up with relevant things to look at, whether a "feed" should be a list of people you follow or suggestions from an algorithm, how you deal with bad actors, sock puppets, the list goes on forever.)
This is relevant to Joel's blog because of the point he makes about Napster: It was never about the "peer to peer" or "sharing", that was the least interesting part. The useful thing about Napster was that you could type in a song and download it. It would have been popular if it wasn't peer to peer, so long as you could still get any music you wanted for free.
Modeling social media as a filesystem, or constructing a data model about how to link things together, and hypergeneralizing all the way to "here's how to model any graph of data on the filesystem!" is basically a "huh, that's neat" little tech demo but doesn't actually solve anything. Yes, you can take any graph-like structured data and treat it as files and folders. I can write a FUSE filesystem to browse HN. I can spend the 20 minutes noodling on how the schema should work, what a "symlink" should represent, etc... but at the end of the day, you've just taken data and changed how it's presented.
There's no reason for the filesystem to be the "blessed" metaphor here. Why not a SQL database? You can `SELECT * FROM posts WHERE like_count > 100`, how neat! Or how about a git repo? You can represent posts as commits, and each person's timeline as a branch, and ooh then you could cherry-pick to retweet!
These kind of exercises basically just turn into nerd-sniping: You think of a clever "what if we treated X as Y" abstraction, then before you really stop to think "what problem does that actually solve", you get sucked into thinking about various implementation details and how it to model things.
The AT protocol may be well-designed, it may not be, but my point is more that it's not protocols that we're lacking. It's a lack of trust, lack of protection from bad actors, financial incentives that actively harm the experience for users, and the negative effects on what social media does to people. Nobody's really solved any of this: Not ActivityPub, not Mastadon, not BlueSky, not anyone. Creating a protocol that generalizes all of social media so that you can now treat it all homogeneously is "neat", but it doesn't solve anything that you couldn't solve via a simple (for example) web browser extension that aggregated the data in the same way for you. Or bespoke data transformations between social media sites to allow for federation/replication. You can just write some code to read from site A and represent it in site B (assuming sites A and B are willing.) Creating a protocol for this? Meh, it's not a terrible idea but it's also not interesting.
- [0] You could argue whether social media is "useful", let's just stipulate that it is.
I think there was a bit of a communication failure between us. You took the article as a random "what if X was Y" exploration. However, what I tried to communicate something more like:
1. File-first paradigm has some valuable properties. One property is apps can't lock data out of each other. So the user can always change which apps they use.
2. Web social app paradigm doesn't have these properties. And we observe the corresponding problems: we're collectively stuck with specific apps. This is because our data lives inside those apps rather than saved somewhere under our control.
3. The question: Is there a way to add properties of the file-first paradigms (data lives outside apps) to web social apps? And if it is indeed possible, does this actually solve the problems we currently have?
The rest of the article explores this (with AT protocol being a candidate solution that attempts to square exactly this problem). I'm claiming that:
1. Yes, it is possible to add file-first paradigm properties to web social apps
2. That is what AT protocol does (by externalizing data and adding mechanisms for aggregation from user-controlled source of truth)
3. Yes, this does solve the original stated problems — we can see in demos from the last section that data doesn't get trapped in apps, and that developers can interoperate with zero coordination. And that it's already happening, it's not some theoretical thing.
I don't understand your proposed alternative with web extension but I suspect you're thinking about solving some other problems than I'm describing.
Overall I agree that I sacrificed some "but why" in this article to focus on "here's how". For a more "but why" article about the same thing, you might be curious to look at https://overreacted.io/open-social/.
The problems with social media are not at all the fact that things are “locked up in apps”.
Again, you missed my point. Data sharing is the least interesting thing imaginable, has already been solved countless times, and is not the reason social media sinks or swims.
Social media sinks or swims based on one thing and one thing only: is it enjoyable to use. Are all the people on here assholes or do they have something interesting to say? Can I post something without being overrun by trolls? How good are the moderation standards? How do I know if the people posting aren’t just AI bots? What are the community standards? In short: what kind of interactions can I expect to have on the platform?
The astronaut types look at the abysmal landscape social media has become, and think “you know what the fundamental problem is? That all this is locked up in apps! Let’s make a protocol, that’ll fix it!”
Never mind that the profit seeking platforms have zero interest in opening up their API to competing sites. Never mind that any of the sites that are interested in openness/federating all univerally have no answer to the problem of how you address content moderation, or at least nothing that’s any different from what we’ve seen before.
The problem in social media is not that things are locked up behind an app. There are apps/readers that combine multiple platforms for me (I remember apps that consolidated Facebook and twitter fully eighteen years ago. It’s not hard.)
The problem with social media is that it’s a wasteland full of bots and assholes.
A HN poster said it best 8 years ago about twitter, and I think it applies to all of social media: it’s a planetary scale hate machine: https://news.ycombinator.com/item?id=16501147
I actually agree with you on a lot of these things, I just think that they do relate to the technological shape.
To give you an example, Blacksky is in setting up their alternative server that is effectively forking the product, which gives them ability to make different moderation decisions (they've restored the account of a user that is banned from Bluesky: https://bsky.app/profile/rude1.blacksky.team/post/3mcozwdhjo...).
However, unlike Mastodon and such, anyone on the Blacksky server will continue living in the same "world" as the Bluesky users, it's effectively just a different "filter" on the global data.
Before AT, it was not possible to do that.
You couldn't "fork" moderation of an existing product. If you wanted different rules, you had to create an entire social network from scratch. All past data used to stay within the original product.
AT enables anyone motivated to spin up a whole new product that works with existing data, and to make different decisions on the product level about all of the things you mentioned people care about. How algorithms run, how moderation runs, what the standards are, what the platform affordances are.
What AT creates here is competition because normally you can't compete until you convince everyone to move. Whereas with AT, everybody is always "already there" so you can create or pick the best-run prism over the global data.
Does this make more sense? It's all in service of the things you're talking about. We just need to make it possible to try different things without always starting from scratch.
I was hoping this was literally just going to be some safe version of a BBS/Usenet sort of filesharing that was peer-based king of like torrents, but just simple and straightforward, with no porn, infected warez, randomware, crypto-mining, racist/terrorist/nazi/maga/communist/etc. crap, where I could just find old computing magazines, homebrew games, recipes, and things like that.
Why can’t we have nice things?
I guess that’s what Internet Archive is for.