Once I was at a company which both had agreements to rehost certain content, and scrape the significant amount.
A couple of funny things happened, once we were clearly being blocked and my manager wanted me to figure out a way to evade said blocking.
And in another case we ended up getting scraped ourselves. Which was absolutely hilarious. I think we were able to resolve this by just contacting the website host.
I am not a lawyer, but a lot of things in America are almost 100% dependent on how much money you want to spend suing or defending yourself in court.
There are a lot of VC funded companies who more or less just scrape and resell data. If LinkedIn and few other places figure out how to actually block them , they'd be out if a business.
Honestly interesting article and general topic. As with most things it depends on who you are. If you are Spotify, Google, OpenAI or Anthropic, its fine. As mentioned, this is why gray areas and unsettled laws are so malicious. They protect the rich and powerful and can be used to absolutely destroy regular people. It is adjacent to scraping, but Aaron Schwartz is a good example. He faced decades in jail for what is effectively best practice for an LLM data pipeline
Well, as he goes on to say, a scraping campaign must (based on current law, anyway) satisfy a fairly specific list of constraints to be considered unambiguously legal. That's where the "not clearly legal" idea comes in; I can only imagine that law students would have trouble parsing the idea that a massive tech company with billions of dollars would fail to spend even a little bit of that to evaluate whether or not their scraping was legal and just... do it anyway.
Once I was at a company which both had agreements to rehost certain content, and scrape the significant amount.
A couple of funny things happened, once we were clearly being blocked and my manager wanted me to figure out a way to evade said blocking.
And in another case we ended up getting scraped ourselves. Which was absolutely hilarious. I think we were able to resolve this by just contacting the website host.
I am not a lawyer, but a lot of things in America are almost 100% dependent on how much money you want to spend suing or defending yourself in court.
There are a lot of VC funded companies who more or less just scrape and resell data. If LinkedIn and few other places figure out how to actually block them , they'd be out if a business.
Honestly interesting article and general topic. As with most things it depends on who you are. If you are Spotify, Google, OpenAI or Anthropic, its fine. As mentioned, this is why gray areas and unsettled laws are so malicious. They protect the rich and powerful and can be used to absolutely destroy regular people. It is adjacent to scraping, but Aaron Schwartz is a good example. He faced decades in jail for what is effectively best practice for an LLM data pipeline
This one made me chuckle:
> This breaks my students’ brains. If it’s so ubiquitous, it must be clearly legal…right?
Do they not drive? Driving slightly faster than the speed limit is ubiquitous in my experience.
Well, as he goes on to say, a scraping campaign must (based on current law, anyway) satisfy a fairly specific list of constraints to be considered unambiguously legal. That's where the "not clearly legal" idea comes in; I can only imagine that law students would have trouble parsing the idea that a massive tech company with billions of dollars would fail to spend even a little bit of that to evaluate whether or not their scraping was legal and just... do it anyway.
This is a feature not a bug in the system. But for a lot of idealistic young law students learning that will make them deeply uncomfortable