TFA title seems odd. Article sez Backblaze has responded to the claims of "sham accounting" and (indirectly) to the claims that "customers backsups [are] at risk", yet the title wording implies that the backups are at risk as a consequence of BB responding.
Not really - if they continue to lose the money that they are losing, then they could go bankrupt. If they never reach profitability, then the only other option is to keep borrowing/raising cash.
What makes a company like this lose so much money? I'm paying 15$ a month for years for a tiny tiny bit of hard drive space. Where does all the money go? What's the big money sink with something like this?
I'm not doubting it, just would like to understand.
The "Personal backup" pricing plan is $99 USD / year and includes "Unlimited data backup" and "one year version history" which implies they are storing much more than just your data at a given time.
$99 would give you 5-6TB of data for consumer HDD costs. And then factor in that they are likely doing some type of additional backup/RAID it would be even less data per dollar. And that's not including maintenance, rent, and employees.
Disclosures: I’ve been a customer for almost a decade. I’m also an early (IPO) shareholder. I’ve never been employed by, or been a vendor to Backblaze.
To add some context and background —
Their “unlimited” plan has a few major capacity-saving caveats. For example, there are built-in exclusions you can’t modify to prevent the cruft and temporary files of supported OSs (and several common apps like Outlook) from taking up space. Backblaze Computer Backup won’t backup certain kinds of data like Time Machine backups on Macs. They also have an automatic policy of deleting backups of external drives that haven’t been connected to the backup host for 30 days. They have a long list of caveats that to their credit they do disclose, although it’s a bit hard to find and understand if you aren’t technical and aren’t looking.
Their backup service is built on their object storage infrastructure to prevent duplicating their own stack, and it also compresses the amount of overhead and capacity accounting they have to worry about between their two product lines. They don’t keep backups of customer data, but they’d need to suffer multiple hard drive and host failures effecting the same file at the same time to lose any single file. Their architecture is proprietary, which I expect adds a lot of human/payroll overhead they wouldn’t have if they just deployed endless ZFS pools “off the shelf”. They build their own servers which offsets the costs of going COTS from a vendor like Dell.
You're paying for the hard drive, the server it goes into, the rack is in, the building it's in, and ask the power and maintenance for you to be able to access the data.
> You're paying for the hard drive, the server it goes into, the rack is in, the building it's in, and ask the power and maintenance for you to be able to access the data.
It's not even just the one drive. Data is stored across multiple drives, in multiple servers, hosted in multiple locations.
People pay the likes of Backblaze for it's reliability and them fool themselves into believing it's a dude with a HD stashed in a shelf somewhere.
Your data is scattered across multiple drives, but that doesn’t make it take up much more space. A terabyte stored in a RAID-5 array is 1.2 terabytes. Maybe the fixed price for unlimited storage part was a mistake, but the footprint overhead is relatively small. Also, keeping a year of history isn’t that much either unless you change every bit of the files multiple times. Using my BtrFS snapshots and Time Machine backups as a reference, normal data changes very little.
> Your data is scattered across multiple drives, but that doesn’t make it take up much more space.
That's besides the point. The point is that storing data in a service such as Backblaze requires more infrastructure than just that one HD in that one server in that one location.
I was skeptical of the wave of "unlimited" (or very large) online backup/storage offerings that came out ~15 years ago, but I didn't expect it to take this long to unravel.
Look at the offering at a glance right now they are offering $99/year unlimited, versioned backup of your home storage?
That seems like peanuts compared to buying a set of drives, setting up, and managing a truly redundant & versioned backup solution takes home. I spent like $1500 on a RAID NAS a decade ago and fell behind on keeping the thing operational.
>That seems like peanuts compared to buying a set of drives, setting up, and managing a truly redundant & versioned backup solution takes home. I spent like $1500 on a RAID NAS a decade ago and fell behind on keeping the thing operational.
Running storage system is cheaper than you think. Look at hetzner for an upper bound on how much storage costs at scale. They offer 10 TB of storage for $31/month. It mentions having "RAID-based storage" and "backups", so this presumably includes redundancy. If you can get 4 customers paying $99/year, you're already breaking even on the server. Most people probably don't have 2.5 TB worth of stuff. Backblaze only does backups, so it isn't like dropbox where you can upload whatever you want onto it. Most people don't even have computers with 1TB storage, so you'll probably be able to fit far more than 4 customers per server. Of course, this doesn't cover salaries for your engineer or marketing people, but at least it's not like moviepass where they're selling $50 worth of movie tickets for $10.
Actually, kind of the opposite. A few years ago Backblaze posted that the economics of their solution relied on the average customer having less than 1.2TB of data. They explicitly do not support Linux users because they tend to average a larger amount of data and are thus not economical to support, even ignoring the extra costs to maintain a separate Linux client.
If it’s so cheap to operate why have they continually lost money, possibly even committing accounting shenanigans to minimize the loss to a smaller number?
Your $1500 includes all the overheads that a storage provider amortizes over their entire customer base. At scale, the cost comes down to the cost of the hard drives, plugging them in, keeping them on, and bandwidth. The margins are really thin, so you need to reduce costs further by doing things like powering down drives that are not in use, or leaving failed drives in racks until they can be replaced in bulk. And importantly, if you offer something unlimited, make sure that it is actually limited. In Backblaze's case, the only 'unlimited' offering is only for their custom backup software that only backs up a single PC, and more recently allows limited backup of external drives. You can only keep multiple TB in it if you have multiple TB powered up in your PC and keep your external drives attached enough so their backups don't get wiped. It is impractical for archiving and abuse. And if it was a problem, they just need to tweak the terms of service to disallow abuse as is traditional.
Capacity is cheap especially if you dedupe. IO is expensive.
Most people don’t have that much data, and capacity grows over time with new drives.
But… you need to be really disciplined to make money in a low margin business. I remember running the numbers on replicating something like backblaze in-house. It was just barely achievable, although I wasn’t crazy enough to shuck disks or build arrays.
I bought a Drobo with the same intent of it being my online redundant array type storage while the original copies were on external HDDs that were placed back in their original retail packaging. The Drobo itself was only connected to computer when necessary but left plugged in to mains when one day it was needed and multiple drives failed to mount--more than could be recovered.
I used to really be into all of the hardware stuff like that building 64 disk arrays and maintaining multiples for edit bays. Now, I'm so sick of it. I just want a plug-n-pray system that works out of the box and does not need maintaining. But of course I know that's not how gear works. So make it someone else's problem and throw money at that problem. $99/year sounds like an excellent bargain. apparently, too excellent
Put another way - if you are a Silicon Valley SWE, spending more than 15-60 minutes per year of your own time is more “expensive” than just giving backblaze $99/yr.
Hindenburg Research nailed the Nikola story a few years ago. I think it’s a valid set of actions to keep a market healthier than if it didn’t exist.
By all means be aware of the possible bias, but if you believe your short thesis enough to put money behind it, I’m more rather than less inclined to believe that you believe.
> if you believe your short thesis enough to put money behind it, I’m more rather than less inclined to believe that you believe
Not really. The short seller just needs enough people to believe their claims. How true or important those claims really are is only relevant to the extent it helps or hurts the narrative.
I don't totally disagree with the idea that putting your money where your mouth is lends some degree of credibility. But when you're taking a short position on a company and then publicly sharing opinions and information that might drive down the stock price of that very same company, it's reasonable to ask if they're simply taking a financial position in a theory they believe in or trying to create a set of circumstances that benefit their financial position.
To put it another way, the incentives for an organization that benefits from a stock price going down mean you should take anything negative they say about the financial health of that stock with an enormous grain of salt. Making a bet that a stock will go down makes me think you believe the stock will go down, but only to the extent that making that bet won't actually drive the stock price down absent any other factor. A short selling firm (or people connected to a short selling firm) putting out press releases designed to drive down a particular stock price 100% don't meet that bar, unless they can demonstrate they have no financial stake in the outcome.
Putting you money where you mouth goes has limits too though. In poker, there's the concept of buying the pot. As a short, you can try to continue funding your play in hopes everyone else becomes convinced. Of course, you're also going to want to use your position to convince others your play is sound with whatever you have at your disposal. So blog away and convince everyone you are the sane play while hoping nobody realizes your description of the emperor's clothes is made up
Every time you think you are smart and everyone else is stupid, you are probably wrong.
I remember figuring it out at a bank office while trying to decide which line was the right one. While bank branch offices with lines are rare these days, the concept remains solid.
Where do you want the accusation to come from? Backblaze isn't exactly the type of company that new york times or propublica are going to go after. Hardly anyone heard of them outside of techies.
This is anecdotical, so take it with a pinch of salt, but I had a really bad experience with Backblaze, they lost ~80% of my files with no recourse for recovery.
They trust their pods too much and they fail in ways they cannot detect until you attempt to extract the data back.
If you use it, periodically download everything just to check it's still there.
Which backup services would folks here recommend to replace and/or supplement Backblaze? I have a lot of important files backed up there (photos, documents), some only there, so for the sake of redundancy I think it would make sense to find a second provider.
I always held BackBlaze up as one of the very few companies that went from initial value proposition, to taking investor funding to going public without enshittification.
I don’t have an opinion on the current controversy. My issue with BackBlaze is that they haven’t been aggressive enough about raising prices to be profitable. I gladly give companies money to give me a service.
I no longer use their service. But only because I took my plex server down years ago and copied my media to my AWS Account (S3 Glacier Deep Archive).
Related Backblaze: Mounting Losses, Lawsuits, Sham Accounting, Insider Selling (262 points, 3 day ago, 158 comments) https://news.ycombinator.com/item?id=43802675
TFA title seems odd. Article sez Backblaze has responded to the claims of "sham accounting" and (indirectly) to the claims that "customers backsups [are] at risk", yet the title wording implies that the backups are at risk as a consequence of BB responding.
Maybe if we quote them symnmetrically the problem will go away?
Yeah, the extra quotes do help
Ambiguous but different semantic commas.
In one interpretation, it is a connective between two "claims" in a list.
In the other, it is a consequence of the response (as you interpreted).
This is worsened by the convention of putting punctuation inside quotation marks.
Not really - if they continue to lose the money that they are losing, then they could go bankrupt. If they never reach profitability, then the only other option is to keep borrowing/raising cash.
What makes a company like this lose so much money? I'm paying 15$ a month for years for a tiny tiny bit of hard drive space. Where does all the money go? What's the big money sink with something like this?
I'm not doubting it, just would like to understand.
The "Personal backup" pricing plan is $99 USD / year and includes "Unlimited data backup" and "one year version history" which implies they are storing much more than just your data at a given time.
$99 would give you 5-6TB of data for consumer HDD costs. And then factor in that they are likely doing some type of additional backup/RAID it would be even less data per dollar. And that's not including maintenance, rent, and employees.
Honestly I'm surprised they exist
Disclosures: I’ve been a customer for almost a decade. I’m also an early (IPO) shareholder. I’ve never been employed by, or been a vendor to Backblaze.
To add some context and background —
Their “unlimited” plan has a few major capacity-saving caveats. For example, there are built-in exclusions you can’t modify to prevent the cruft and temporary files of supported OSs (and several common apps like Outlook) from taking up space. Backblaze Computer Backup won’t backup certain kinds of data like Time Machine backups on Macs. They also have an automatic policy of deleting backups of external drives that haven’t been connected to the backup host for 30 days. They have a long list of caveats that to their credit they do disclose, although it’s a bit hard to find and understand if you aren’t technical and aren’t looking.
Their backup service is built on their object storage infrastructure to prevent duplicating their own stack, and it also compresses the amount of overhead and capacity accounting they have to worry about between their two product lines. They don’t keep backups of customer data, but they’d need to suffer multiple hard drive and host failures effecting the same file at the same time to lose any single file. Their architecture is proprietary, which I expect adds a lot of human/payroll overhead they wouldn’t have if they just deployed endless ZFS pools “off the shelf”. They build their own servers which offsets the costs of going COTS from a vendor like Dell.
They offer unlimited storage. Probably a bunch from /r/datahoarder who have loaded up 10PB of junk.
That’s even a lot, I’m paying like 75 cents for 135GB on b2. Extrapolating to a TB would be like $5.
That’s what makes it make more sense to me.
You're paying for the hard drive, the server it goes into, the rack is in, the building it's in, and ask the power and maintenance for you to be able to access the data.
> You're paying for the hard drive, the server it goes into, the rack is in, the building it's in, and ask the power and maintenance for you to be able to access the data.
It's not even just the one drive. Data is stored across multiple drives, in multiple servers, hosted in multiple locations.
People pay the likes of Backblaze for it's reliability and them fool themselves into believing it's a dude with a HD stashed in a shelf somewhere.
> It's not even just the one drive
Your data is scattered across multiple drives, but that doesn’t make it take up much more space. A terabyte stored in a RAID-5 array is 1.2 terabytes. Maybe the fixed price for unlimited storage part was a mistake, but the footprint overhead is relatively small. Also, keeping a year of history isn’t that much either unless you change every bit of the files multiple times. Using my BtrFS snapshots and Time Machine backups as a reference, normal data changes very little.
> Your data is scattered across multiple drives, but that doesn’t make it take up much more space.
That's besides the point. The point is that storing data in a service such as Backblaze requires more infrastructure than just that one HD in that one server in that one location.
That's true. And you are your own support person.
And the salaries, benefits, office space, supplies, etc for their employees.
And the staff to manage it.
I mean very easily.
I was skeptical of the wave of "unlimited" (or very large) online backup/storage offerings that came out ~15 years ago, but I didn't expect it to take this long to unravel.
Look at the offering at a glance right now they are offering $99/year unlimited, versioned backup of your home storage?
That seems like peanuts compared to buying a set of drives, setting up, and managing a truly redundant & versioned backup solution takes home. I spent like $1500 on a RAID NAS a decade ago and fell behind on keeping the thing operational.
>That seems like peanuts compared to buying a set of drives, setting up, and managing a truly redundant & versioned backup solution takes home. I spent like $1500 on a RAID NAS a decade ago and fell behind on keeping the thing operational.
Running storage system is cheaper than you think. Look at hetzner for an upper bound on how much storage costs at scale. They offer 10 TB of storage for $31/month. It mentions having "RAID-based storage" and "backups", so this presumably includes redundancy. If you can get 4 customers paying $99/year, you're already breaking even on the server. Most people probably don't have 2.5 TB worth of stuff. Backblaze only does backups, so it isn't like dropbox where you can upload whatever you want onto it. Most people don't even have computers with 1TB storage, so you'll probably be able to fit far more than 4 customers per server. Of course, this doesn't cover salaries for your engineer or marketing people, but at least it's not like moviepass where they're selling $50 worth of movie tickets for $10.
https://www.hetzner.com/storage/storage-share/
> Most people probably don't have 2.5 TB worth of stuff.
But Backblaze is marketing itself towards people who do. The average person is not a backblaze customer
Actually, kind of the opposite. A few years ago Backblaze posted that the economics of their solution relied on the average customer having less than 1.2TB of data. They explicitly do not support Linux users because they tend to average a larger amount of data and are thus not economical to support, even ignoring the extra costs to maintain a separate Linux client.
If it’s so cheap to operate why have they continually lost money, possibly even committing accounting shenanigans to minimize the loss to a smaller number?
See: https://news.ycombinator.com/item?id=43804258
tl;dr R&D, marketing, and overhead.
Your $1500 includes all the overheads that a storage provider amortizes over their entire customer base. At scale, the cost comes down to the cost of the hard drives, plugging them in, keeping them on, and bandwidth. The margins are really thin, so you need to reduce costs further by doing things like powering down drives that are not in use, or leaving failed drives in racks until they can be replaced in bulk. And importantly, if you offer something unlimited, make sure that it is actually limited. In Backblaze's case, the only 'unlimited' offering is only for their custom backup software that only backs up a single PC, and more recently allows limited backup of external drives. You can only keep multiple TB in it if you have multiple TB powered up in your PC and keep your external drives attached enough so their backups don't get wiped. It is impractical for archiving and abuse. And if it was a problem, they just need to tweak the terms of service to disallow abuse as is traditional.
If their price isn’t too cheap then why aren’t they making money?
Capacity is cheap especially if you dedupe. IO is expensive.
Most people don’t have that much data, and capacity grows over time with new drives.
But… you need to be really disciplined to make money in a low margin business. I remember running the numbers on replicating something like backblaze in-house. It was just barely achievable, although I wasn’t crazy enough to shuck disks or build arrays.
Economy of scale maybe? You can get better prices at scale purchases?
Otherwise you're correct and it doesn't make a bit of sense.
It’s been nearly 20 years and they are still running at a loss. Sounds like they haven’t found those economies of scale yet.
There really aren't economies of scale buying hard drives.
I bought a Drobo with the same intent of it being my online redundant array type storage while the original copies were on external HDDs that were placed back in their original retail packaging. The Drobo itself was only connected to computer when necessary but left plugged in to mains when one day it was needed and multiple drives failed to mount--more than could be recovered.
I used to really be into all of the hardware stuff like that building 64 disk arrays and maintaining multiples for edit bays. Now, I'm so sick of it. I just want a plug-n-pray system that works out of the box and does not need maintaining. But of course I know that's not how gear works. So make it someone else's problem and throw money at that problem. $99/year sounds like an excellent bargain. apparently, too excellent
Put another way - if you are a Silicon Valley SWE, spending more than 15-60 minutes per year of your own time is more “expensive” than just giving backblaze $99/yr.
This might all be true but coming from a short seller seems off.
Hindenburg Research nailed the Nikola story a few years ago. I think it’s a valid set of actions to keep a market healthier than if it didn’t exist.
By all means be aware of the possible bias, but if you believe your short thesis enough to put money behind it, I’m more rather than less inclined to believe that you believe.
> if you believe your short thesis enough to put money behind it, I’m more rather than less inclined to believe that you believe
Not really. The short seller just needs enough people to believe their claims. How true or important those claims really are is only relevant to the extent it helps or hurts the narrative.
I don't totally disagree with the idea that putting your money where your mouth is lends some degree of credibility. But when you're taking a short position on a company and then publicly sharing opinions and information that might drive down the stock price of that very same company, it's reasonable to ask if they're simply taking a financial position in a theory they believe in or trying to create a set of circumstances that benefit their financial position.
To put it another way, the incentives for an organization that benefits from a stock price going down mean you should take anything negative they say about the financial health of that stock with an enormous grain of salt. Making a bet that a stock will go down makes me think you believe the stock will go down, but only to the extent that making that bet won't actually drive the stock price down absent any other factor. A short selling firm (or people connected to a short selling firm) putting out press releases designed to drive down a particular stock price 100% don't meet that bar, unless they can demonstrate they have no financial stake in the outcome.
People make these statements about shorts but never about longs.
Shorting stock is very hard, the upwards trajectory of the market and overall optimism of American market participants works against you.
There are far far far more hype men on the long side than short.
Putting you money where you mouth goes has limits too though. In poker, there's the concept of buying the pot. As a short, you can try to continue funding your play in hopes everyone else becomes convinced. Of course, you're also going to want to use your position to convince others your play is sound with whatever you have at your disposal. So blog away and convince everyone you are the sane play while hoping nobody realizes your description of the emperor's clothes is made up
Every time you think you are smart and everyone else is stupid, you are probably wrong.
I remember figuring it out at a bank office while trying to decide which line was the right one. While bank branch offices with lines are rare these days, the concept remains solid.
If you put money on your shirt AND make public claims that you’re right and the shares are catastrophically overvalued, those two kind of cancel out.
They might be right that Backblaze is doing shady accounting, but they also might benefit from the perception the company is doing shady accounting.
We have people buying shares and arguing their thesis that the company is undervalued all the time.
Those people also stand to benefit yet many people see it as fundamentally different for reasons that escape me.
Where do you want the accusation to come from? Backblaze isn't exactly the type of company that new york times or propublica are going to go after. Hardly anyone heard of them outside of techies.
I don't think that question is relevant.
I see potential bias is all.
Is there a perfect source for this information? No idea.
This is anecdotical, so take it with a pinch of salt, but I had a really bad experience with Backblaze, they lost ~80% of my files with no recourse for recovery.
They trust their pods too much and they fail in ways they cannot detect until you attempt to extract the data back.
If you use it, periodically download everything just to check it's still there.
Which backup services would folks here recommend to replace and/or supplement Backblaze? I have a lot of important files backed up there (photos, documents), some only there, so for the sake of redundancy I think it would make sense to find a second provider.
Edit: See following comment.
People have mentioned in the other thread (see sibling comment) that they have ex-Hindenburg Research people working there.
https://en.wikipedia.org/wiki/Hindenburg_Research
I always held BackBlaze up as one of the very few companies that went from initial value proposition, to taking investor funding to going public without enshittification.
I don’t have an opinion on the current controversy. My issue with BackBlaze is that they haven’t been aggressive enough about raising prices to be profitable. I gladly give companies money to give me a service.
I no longer use their service. But only because I took my plex server down years ago and copied my media to my AWS Account (S3 Glacier Deep Archive).
Oh. That sounds quite bad. Presumably in practice they would be acquired by Amazon rather than simply shut down service overnight.
Ugh, your b2 to s3 migration would be "fun". Also, welcome to egress charges!
>Also, welcome to egress charges!
b2 has 3x free egress though?
Their point is that Amazon does not, and presumably would not keep that offer if they were to acquire B2
Backblaze can always opt for a Peoplesoft-like poison pill.
Oracle went on and still took over them and that’s how Workday was born.