# Welcome to Reddit's robots.txt
# Reddit believes in an open internet, but not the misuse of public content.
# See https://support.reddithelp.com/hc/en-us/articles/26410290525844-Public-Content-Policy Reddit's Public Content Policy for access and use restrictions to Reddit content.
# See https://www.reddit.com/r/reddit4researchers/ for details on how Reddit continues to support research and non-commercial use.
# policy: https://support.reddithelp.com/hc/en-us/articles/26410290525844-Public-Content-Policy
User-agent: *
Disallow: /
Their license/Eula clearly state that Reddit has perpetual whatever to content posted on Reddit, but relying solely on DMCA for "stolen" content _yet again_ feels like a terrible way to deal with non-original content. Part of me hopes that Reddit gets hit with some new precidence-setting lawsuits regarding non-original content that requires useful attribution, but I double t that will ever happen.
An EULA does not change the morality of the situation anyway. They are a leech profiting off users generating content who are now upset about not getting a cut from third-parties also profiting from said user generated content.
can also say every humans are leech benefiting off free software (creators) and complaining about their worthless chitchat, barely usable because of its basic semantics, being "stolen".
1. That’s kind of the point of free software, is it not? If you don’t want people “leeching” off of your software for free, don’t make it free.
2. Reddit is an amazing source of coding information and general Q&A on an extremely wide variety of topics. I would not characterize all of it as chitchat.
it's even worse. it's not theirs (it's the users'), they are merely hosting it and using it (ToS gives them a fancy irrevocable license I guess).
so they can do whatever they want with it and the actual owners/authors have no chance to really influence Reddit at all to make it crawlable. (the GDPR-like data takeout is nice, but ... completely useless in these cases where the value is in the composition and aggregation with other users' content.)
actually owners/authors like me would not want our stuff crawlable because that gives up our ownership.
When I am answering some random dude on reddit with a problem I want that dude to read my solution. I don't want this to be crawled and forever stored (probably deanonymized) or enshrined in a dozen commercial LLMs. There is substack for that stuff.
I'll be honest... I don't care about this thing. I view public posts as something that belongs to the public domain and if that is searchable, all the better, other people can reach my post. And if that is what LLM's are trained on, also, great, I hope it will be useful to some people. What I _do_ care about is an even playing fields and access to _all_ llms to be trained in that data.
In the end, I view potential AGI's as a common consciousness brainchild.
OK. I see "AGI" as a buzzword and current ML trend as exploitation of creative works that profits big tech. They will always have bigger computers and get more out of what we create increasing wealth gap, unless we stop it
Social support (or we can even call it "[corporate] social responsibility") is a political question/problem/thing.
Endless iterations of the discussion about how copyright should/shouldn't be are meaningless without considering the larger social context.
The vast majority of creators (artists!?) are not compensated at all, and the users' (content consumers'? the public's?) attention is already 100% fully saturated.
So GenAI chrurning out more kitsch hardly changes that. (It got popular when it was novel, and ... that was it for now. And it became one more brush in the evergrowing workshop.)
And if/when a company creates a product out of it, then that product needs to be scrutinized and we should consider the ethical, social, and political problems. (Because ... politics is a blunt tool whereas ethics is as infinitely nuanced as we make it, whereas actual social considerations ought to be pragmatic and fair.)
And that's the problem. Users benefit from cheaper access to customized content. (In other words, arguably it's a net positive thing that they can - for example - ask some GenAI tool to make them a nice picture for their friend's birthday.) So what's the cost of this? Does it make some people jobless? Is that good or bad? (Is it good that plant breeding programs, fertilizers, tractors, and irrigation systems made a lot of agricultural workers jobless? Well, in some sense yes as allows a few people to feed many, freeing up time for others to become doctors and artists. In some sense bad, because our current socioeconomic system does not provide real social support - despite enormous redistribution of GDP. [Because brutal inefficiencies in allocation of that surplus. And that's again a political problem.])
... and of course here the status quo bias leads to "technological progress chipping away at inefficiencies" in a capitalist reality translates to "even more things get commoditized", and that in turn in the current shitty socioeconomic system equals "lots of externalities are not priced in, and lots of people are forced to drastically change their lives to adjust to new prices" (ie. price of their labor and/or products going down, so they need to change jobs, yet they barely get any support for doing so) ... and of course ethically it's bad that most people virtually uncritically accept and enjoy the results of progress (new products and services) without giving a fuck about the costs.
> Users benefit from cheaper access to customized content. (In other words, arguably it's a net positive thing that they can - for example - ask some GenAI tool to make them a nice picture for their friend's birthday.)
very arguably yeah. we evolve to where we push a button and get a birthday present. the logical conclusion is a system that does not even require to push a button. this is the opposite of the original point (spend time and effort making something to show you care). so to me it is not benefit, it is harm.
maybe you picked a bad example. but the examples of legit benefits of this tech, they don't turn into harms if you really analyze it and go to the root, are hard to find in consumer land.
I like this example because I tried making a greeting card "by hand" and it of course sucked ass. (As I have zero idea how to use Photoshop despite religiously reading the tutorials 20 years ago on a community site :D And, of course, I used PhotoPea.)
So by spending the same 20 minutes fiddling with some kind of graphics if the outcome becomes 10x better then I see that as an advantage. (That said I haven't tried this.)
Basically this is "exactly" that kind of capability that we already seen in "artist paints kid's drawing" [0] just commoditized. And this is where I (as a user) would be looking for the style transfer. Because I saw something famous/trendy/fancy/aesthetically-pleasing and now want to imitate it as a gag for this hypothetical birthday card. (Sure, copyright already doesn't care if I download someone's famous photo, and cut my friends head out from a photo I made, and put the head part on the copyrighted photo. Yet it was a straightforward derivative work. Just it's not commercial, damages are none, or even negative, ie. the artist benefits from that famous exposure! :D)
> legit benefits of this tech
I wanted to try to find an example that could apply to a lot of users. Because as awesome as using GenAI to generate Lean proofs (and then add a feedback loop by actually running the generated code through Lean) to solve Math Olympiad problems [1], it's not really an everyday thing.
> we evolve to where we push a button and get a birthday present. the logical conclusion is a system that does not even require to push a button.
Well, maybe! In no time we'll board the Axios and just chill. [2] But at the same time crafts and DIY and experiences (tourism, festivals, concerts) are ridiculously popular. (And it has its own problems. [3])
> Well, maybe! In no time we'll board the Axios and just chill.
This was sarcasm. That system would never happen. If birthday card requires no effort it is worse than nothing, literal noise.
> So by spending the same 20 minutes fiddling with some kind of graphics if the outcome becomes 10x better then I see that as an advantage. (That said I haven't tried this.)
Feels like you misunderstood. It is not "10x better" or "advantage" because literally the easier it was for you the less it is valuable.
That's why more and more exchanges are moving to DMs and closed communities. Because we don't plan on stuff we intend for participants in the discussion to be harvested but it more and more is. If reddit breaks that trend I only welcome it.
I tried it many months back when they glitch-killed my decade-plus account. (Yes, I'm still bitter over the kafkaesque injustice.)
Anyway, you basically submit a request and then later they will email you a link to a zip file that contains a free dozen CSV files with unescaped newlines. One for all the comments you made, one for up/down-votes, one for blocked users, etc.
According to the US court systems the robots.txt file is meaningless. If they respond with a 200 status code giving you the access then you can legally scrape it all you want. If they require that you log in then you have to follow the terms you agree to when creating an account. Public means public though, and if Reddit doesn't want to make the content private (put it behind a login) then we can scrape away.
Note that scraping, regardless of the level of permission, doesn't mean you can do anything you want with the content. Copyright still applies. But you can scrape it, and if your use falls under Fair Use or another caveat to the copyright laws then you can do ahead and do it without needing any permission from the authors.
I liked the chapter on DMCA from the 5-volume E-Commerce &
Internet Law. It was super detailed.
I haven’t read volume 1, but apparently half of it is about data scraping, and I expect it to be similarly detailed. So if I were you, that’s where I’d start.
Another option is looking for “robots.txt” at Google Scholar and trying various keywords like “legality”, “scraping”, “case law”, etc.
Independent scrapers can launder the data between Reddit and AI consumers. The only folks this hurts is users seeking info via search engines and folks willing to kowtow to rules that are potentially low impact to evade. Next steps would be (from an adversarial perspective) browser extensions that stream back data for ingestion similar to Recap for Pacer [1].
(full disclosure: assisting someone pursuing regulatory action against reddit in the EU for a separate issue from scraping, it's a valuable resource, but the folks who own and control it are meh)
Even more basic, it's free speech. The data itself is public domain so your free speech is not restricted and you don't need fair use excemptions for those restrictions. On the the access through the official system is restricted.
That’s a weird statement to be absolutist about. The majority of individuals and companies who want to be successful do not do so by scrapping websites, thus have no reason to disobey robots.txt. Most people in the world, ambitious or not, wouldn’t even understand what your sentence refers to.
OP is obviously talking about people whose area of research/development/product would involve web scraping... This feels like being purposefully obtuse
Has not NYT tried to sue OpenAI because of them ignoring robots.txt or you mean it's impossible to prove and / or it's still more profitable to just ignore robots.txt?
With the amount of crap in Reddit, cleaning it must be a very non-trivial problem. (I mean, it never is, but in the case of Reddit it's probably extra complicated)
I understand the AI context, but this is dangerously anticompetitive for other search engines.
This is a dangerous precedent for the internet. Business conglomerates have been controlling most of the web, but refusing basic interoperability is even worse.
> There is nothing preventing search companies paying the same $60 Million to license content.
Yes, actually, there is - having $60m to throw around.
"Barriers to entry often cause or aid the existence of monopolies and oligopolies" [0]. Monopolies and oligopolies are definitionally the opposite of free market forces. This is quite literally Econ 101.
Microsoft can throw around $60M, and Bing is used by most of the "alternative" search engines.
It doesn't solve the problem, but if money is the only thing preventing search engines from accessing Reddit, then what goes for Google also goes for Microsoft.
> Microsoft can throw around $60M, and Bing is used by most of the "alternative" search engines.
That's a symptom of the issue, not a solution. Bing is used because having your own crawler is infeasible, partially because you will be literally blocked in many cases.
And yet "free market forces" are often the reason why monopolies and oligopolies arise...?
Monopolies are entirely consistent with free market economics. After all, if there's clearly a "best product" for a particular niche, it's entirely rational (free market actor) behavior for everyone to use the same product, leading to its monopoly in that market segment.
I don't understand why people think this isn't/won't be/shouldn't be a common result of "free market forces".
> Monopolies are entirely consistent with free market economics
Not in the least. Literally in the first semester, Economics 101 type class that any business/economics/etc student would take, it would be covered clearly that monopolies are violations of free markets.
A free market isn't a euphemism for anarchy or "no rules", it's a specific economic term. The things it is free of include artificial price floors or ceilings, barriers to entry, anti-competitive practices, etc. In other words, monopolies, oligopolies, cartels, monopsonies, etc are all violations of a free market. You do not have a free market if there is a monopoly supplier.
Economics 101 also assumes perfect information symmetry, perfect competition, and spherical cows. In other words, it only vaguely models reality... Good enough to teach the basic concepts, but without diving in the nuance that makes the basic models break down.
If one competitor is far enough ahead of the rest, they can maintain that lead given that they can extract sufficient momentum from their early mover advantage. If they keep this up long enough, competitors never reach the scale to sufficiently prevent them from becoming a monopoly (at least over their local market segment).
None of this requires anti-competitive behavior; simply good execution on the part of the leader.
Unless, of course, you're suggesting that "free markets" also involve government intervention to suppress their lead in the market...
Or even more basic government-enforced restrictions like IP laws. If you want a survival of the fittest anarchy economy then those won't exist either. Neither will legal protections against espionage or circumvention of whatever technical means you come up to try and get all that back.
> Monopolies are entirely consistent with free market economics.
This is a fair critique. I'm approaching this from an admittedly American perspective in which "free market" colloquially implies competition - but I recognize that competition is not inherently a free market concept.
If you check citations, you'd find the sentence preceding my excerpt on barriers to entry:
"Because barriers to entry protect incumbent firms and restrict competition in a market, they can contribute to distortionary prices and are therefore most important when discussing antitrust policy."
Antitrust policy then links to a page on competition law: "Competition law is the field of law that promotes or seeks to maintain market competition by regulating anti-competitive conduct by companies." [0]
So yes, I'd downvote you if I could, but HN doesn't allow downvotes - which is honestly pretty fitting in the context of this conversation.
I took care of the down vote for you. I dunno when the privilege is earned, but keep up the quality comments and you’ll be able to downvote incurious ideologues like this in no time!
Once again buying an Airplane and starting an Airline business has probably the highest barrier to entry. Yet the Airline industry is the most competitive.
The air travel industry has also seen some of the most significant government regulation in the form of blocking mergers (ie monopolistic, anticompetitive behavior) - meaning that competition in the airline space is due to regulation, not free market dynamics alone.
I’m happy to continue this debate if you’d like to start supporting your posts with citations but probably won’t engage further unless you do. Have a great day!
Anyone can ask for licensing deal. I'm sure NY Times, Conde Nest all have licensing deals. Mr. Beast signed a deal with Amazon. Joe Rogan with Spotify. Why is it hard to understand?
Even HN can get a licensing deal if they want to.
If you are producing content, you have every right to do what you want to with the content.
How many other sites might have leverage to charge to be indexed?
I don't want to live in a world where you have to use X search engine to get answers from Y site - but this seems like the beginning of that world.
From an efficiency perspective - it's obviously better for websites to just lease their data to search engines then both sides paying tons of bandwidth and compute to get that data onto search engines.
Realistically, there are only 2 search engines now.
This seems very bad for Kagi - but possibly could lead the old, cool, hobbiest & un-monetized web being reinvented?
It's a very long article so understandable that you did not read on and learn about other search engines crawling beyond GBY. Still there are indeed very few that are crawling at web scale, and internationally. We are at 8 billion pages and totally independent [0], hence expressing our concerns to 404 media after being blanked by Reddit.
> did not read on and learn about other search engines crawling beyond GBY. Still there are indeed very few that are crawling at web scale, and internationally
That's helpful clarification.
In criticism of the article, you might agree that
none of them have their own results
is a fairly absolute statement. It signals: Final word on the matter; no nuance to follow.
Brave has their own search engine, yandex I only use for reverse image search, baidu's interface is really clean and feels like old school google... but I don't speak chinese so I can't use it.
Brave doesn't have its own index of the full web, and it's even less useful than Yandex. And very likely buys some of it, according to what I've heard. So it falls into the last category.
if that is true then they are lying on their site where they claim: "Brave Search operates from a fully independent search index"
do you have any reference for your claim?
i use brave search and find it very useful. very rarely there is something i can't find, and when i run into that other search engines are not much better.
Given the piles of spammy shit on Google these days, I'm wondering if "doesn't have its own index of the full web" is actually a competitive advantage.
That seems like the business model for streaming. You subscribe to X provider to watch Y series. So, as for streaming, I suppose a pirate bay search engine will come up
Pirate Bay is probably not the most optimal analogy, more like Anna's Archive imho [1], individually offered by web property scrape runs compressed into a package, maybe served by torrents like this Academic Torrents site example [2].
Scraper engine->validation/processing/cleanup->object storage->index + torrent serving is rough pipeline sketch.
> but this seems like the beginning of that world.
It's not the beginning, it's mere continuation.
Walled gardens have existed since the AOL days. They deteriorate over time but it doesn't prevent companies from trying (each time, in bigger attempts).
idk man i bet you five bucks and a handshake it's just going to play out like the existing startup grift.
There's an established player with institutional protections, then a scrappy upstart takes a bunch of VC money, converts it into runway, gives away the product for free, gradually replaces and becomes the standard, then puts out an s-1 document saying "we don't make money and we never have, want to invest?" and then they start to enjoy all the institutional protections. Or they don't. Either way you pay yourself handsomely from the runway money so who cares.
The upstart gets indexed and has an API, the established player doesn't.
The upstart is more easily found and modular but the institutional player can refuse to be indexed to own their data and they can block their API to prevent ai slop from getting in and dominating their content.
IANAL but as far as I understand the current legal status (in the US) a change in robots.txt or terms and conditions is not binding for web scrapers since the data is publicly accessible. Neither does displaying a banner "By using this site you accept our terms and conditions" change anything about that. The only thing that can make these kinds of terms binding is if the data is only accessible after proactively accepting terms. For instance by restricting the website until one has created an account. Linkedin lost a case against a startup scraping and indexing their data because of that a few years ago.
This problem is only going to get worse. for my thegreatestbooks.org site i used to just get indexed/scraped by google and bing. now it's like 50+ AI bots scraping my entire site just so they can train a LLM to answer questions my site answers without having a user ever visit my site. I just checked cloudflare and in the past 24 hours I've had 1.2 million bot/automated requests
Considering it's Buttflare, enabling it probably also means blocking random users. But of course that's not Buttflare's problem because it's not enabled by default.
They changed robots.txt a month or so ago. For the first 19 years of life, reddit had a very permissive robots.txt. We allowed all by default and then only restricted certain poorly behaved agents (and Bender's Shiny Metal Ass(tm))
But I can understand why they made the change they did. The data was being abused.
My guess is that this was an oversight -- that they will do an audit and reopen it for search engines after those engines agree not to use the data for training, because let's face it, reddit is a for profit business and they have to protect their income streams.
> But I can understand why they made the change they did. The data was being abused.
Depends how you see it - if you see it as 'their' data (legally true) or if you see it as user content (how their users would likely see it).
If you see it as 'user content', they are actually selling the data to be abused by one company, rather than stopping it being abused at all.
From a commercial 'lets sell user data and make a profit' perspective I get it, although does seem short-sighted to decide to effectively de-list yourself from alternative search engines (guess they just got enough cash to make it worth their while).
Is that actually true? Reddit may indeed have a license to use that data (derived from their ToS), but I very much doubt they actually own the copyright to it. If I write a comment on Reddit, then copy-paste it somewhere else, can Reddit sue me for copyright infringement?
They demand that right. That doesn't mean they actually have a right to use the content in ways that are not directly required for the operation of the website or that are otherwise surprising to the average user. Putting something in the TOS doesn't always make it a valid contract.
Person extensively quoted in the article here. They are welcome to reach out. But not a single person from any level did that, nor replied to my polite requests to explain and engage. We first contacted them in early June and by 13th June, I had escalated to Steve Huffman @spez.
An acquaintance investigating Reddit's moderation mechanization inquired how a major subreddit was moderated after an Associated Press post was auto removed by automod. They were banned from said sub. They inquired why they were banned, and they shared they would share any responses with a journalism org (to be transparent where any replies would be going, because they are going to a journalism org). They were muted by mods for 28 days and were "told off" in a very poor manner (per the screenshots I've seen) by the anonymous mod who replied to them. They were then banned from Reddit for 3 days after an appeal for "harassment"; when they requested more info about what was considered harassment, they were ignored. Ergo, inquiring as to how the mods of a major sub are automodding non-biased journalism sources (the AP, in this case) without any transparency appears to be considered harassment by Reddit. The interaction was submitted to the FTC through their complaint system to contribute towards their existing antitrust investigation of Reddit.
Shared because it is unlikely Reddit responds except when required by law, so I recommend engaging regulators (FTC, and DOJ at the bare minimum) and legislators (primarily those focused on Section 230 reforms) whenever possible with regards to this entity. They're the only folks worth escalating to, as Reddit's incentives are to gate content, keep ad buyers happy, and keep the user base in check while they struggle to break even, sharing as little information publicly as possible along the way [1] [2].
One (in this case, 2) company's incentive for profit should not take priority over the usability/well being of the internet as a whole, ever, and is exactly why we are where we are now. This is an absolutely terrible precedent.
I think it's the other way around, in that people don't like to hear how Reddit has become important due to the death of independent forums and the degree to which information has become concentrated on the site.
The death of independent forums has been greatly exaggerated.
Of all the forums I used to be active in, many are still active. The ones that died did so because the community died (i.e. they did not shift to Reddit and the like).
Reddit is great simply because it allowed anyone to create a community. No need to get a LAMP stack and deal with security vulnerabilities in your forum SW.
These days you have Lemmy and its ilk. Much higher barrier than the old LAMP stack, but also much superior to it. I do hope it takes off.
This is a false dichotomy. You can have services, and not have them devolve into complete unusability in the name of profit. This isn’t sustainable either. The myopic pursuit of short term gains at the expense of the product will collapse at some point in the future, no matter how much you believe in this weird frog-boil internet we’ve inherited now.
Complete unusability is when ai tools clone the content and people stop visiting the original service and participating. I'll leave it up to them to defend blocking duck duck go for example, but blocking "AI" bots for an online community is a matter of survival at this point.
Alternatively, it's because the base platform has also devolved into unusability. Both Reddit and Twitter are in a position where their info is easily scraped, and their community is barely worth the advertising/paid-premium experience they demand from you. As both platforms continue to decline in quality, you might not even need to replace the original service. Both businesses appear intent on getting replaced.
> The myopic pursuit of short term gains at the expense of the product will collapse at some point in the future,
The myopic pursuit of short-term gains is the only playbook that works. Long-term business strategy is a gamble, and today's businesses have all learned that they'd rather make hay when the sun is shining than be remembered as a good business.
Twitter tried a long-term playbook to reverse their unprofitable sinkhole of a website. That ended up with them being undervalued and sold to the highest bidder.
> Twitter tried a long-term playbook to reverse their unprofitable sinkhole of a website.
From what I recall reading at the time, Twitter was finally becoming profitable before the sale (last two years? It’s hard to find a source now since every story since is about some shit show or other post sale).
> That ended up with them being undervalued and sold to the highest bidder.
You make it seem they were in dire straits and had to be sold for scraps, but that’s far from the case. They sold for more than their valuation to the only bidder because they understood what a good deal it was for them. They forced the buyer to not back out, after all.
We did. As in we, the Internet, existed for a long time without anyone making money and we paid for the privilege. Websites were built and hosted at owner's expense, for years, with no expectation that they be financially rewarded. Sure some would run donation drives, or work with sponsors relevant to the community in question, but a whole ton, mine included, just cost me a lot of money over many years.
Those websites were definitely technically inferior, as the march of progress is unavoidable, but web hosting is cheaper than it's ever been. A VPS that utterly blows away what mine was capable of in 2007 for nearly a hundred a month can now be had for about $10 per month. Yet everyone wants these monolith platforms, but even that wouldn't be the worst thing ever, except that every one of these platforms has a backend to support that we in the Old Internet never did: a C-suite's worth of executives and millions of shareholders, who for some reason have decided that reddit can't exist unless reddit makes them reams and reams of money.
I'd be very, very interested to see how much of, even what's probably the most massive one of all, Facebook, is non-essential busywork that could easily be shut down tomorrow with no adverse effects to the platform. Firstly the entire executive class, just, they don't do shit to make Facebook the product. In fact I'd argue their decisions almost universally have made it worse as a product very consistently for it's entire lifetime. Then, all the marketing people. There's just no goddamn reason to advertise Facebook (or reddit for that matter) the brand is so ubiquitous, if you actually found someone who'd never heard of it, I'd give you a large chunk of money. Add to that, if Facebook was doing a good job of being what it ostensibly is, then people immediately become the best advertising, because people want to hang with people in these digital spaces. Then get rid of the people working to make Facebook addictive with dark patterns. Then get rid of the entire targeted ad division, because it's gross and inhumane. Pare the company down to engineers who build the product, and if anything, expand the moderation team so they can actually ensure the safety of the platform, and of course the IT staff to back them. Now what does Facebook cost to operate?
As far as I'm concerned, this pearl-clutching about "well websites have to make money" is grossly, grossly overstated. Websites don't cost that much to run. A ton of money is being siphoned off by the MBA parasites playing games in Excel all day. A ton more is being wasted developing features that advertisers want and users hate. A ton more is being funneled into making products artificially addictive to vulnerable people, to exploit them, so let's just not do that. And of course, leadership, rewarding themselves with generous compensation packages they aren't even remotely able to justify. Now what does your website cost to maintain? Surely not nothing, and for websites of substantial size, it will still be high, but I'm willing to bet it's a hell, hell, hell of a lot less than it was before.
Part of the issue is that it isn’t just the web, but the inevitable american corporate shareholder model. Even businesses could be mom and pop ified and made way more popular overnight: quit raising prices and cutting corners and it would actually stand for itself like a massive $7 burrito. However the expectation is that shareholders get returns. Costs must be cut. Prices must be raised. Margins must be improved. It doesn’t matter if this eats the business alive, as shareholders are sufficiently leveraged. The whole system is incentivized to select for inferior quality and taking all the available money on the table.
My rant above and your response reminded me of all those tons of MMO games out there that are ancient, with a tiny playerbase, that remain profitable nonetheless simply because if you have a product that people like using, putting it into maintenance mode and doing the bare minimum to keep it running is a perfectly valid business strategy. The companies that buy these service games and run them effectively just buy completed money printers and keep them operating. It's not going to make anyone rich probably, but it's a perfectly valid and profitable way to go about things.
The silicon valley "grow at all costs, always evolve and innovate forever" model is so detached from the reality of most businesses in my experience.
>The companies that buy these service games and run them effectively just buy completed money printers and keep them operating.
I hadn't really thought about that topic in that way before. Really explains why some of those older MMOs have no desire to really make any improvements, the owners are happy to just keep them powered up and collect a check but have no incentive to invest in making them better.
I think the notion that sometimes things are just "done" is incredibly undervalued in our industry. Frankly I wish a ton of games I play would STOP updating.
>I think the notion that sometimes things are just "done" is incredibly undervalued in our industry.
I agree, but also the flip side is that things rapidly switch from 'done and working' to 'dead' pretty quickly if no one is willing to do minor maintenance.
Popular websites that allow user content to be uploaded or linked do cost that much to run, due to content moderation.
There might be a small (relatively) forum here and there that a few good moderators are willing to slave away at keeping clean, but you will never see a website that allows user content with as many users as Reddit/Youtube/Instagram/etc be cheap.
Although, due to AI, the cost to spam the small forums might be so small that even they might come into the crosshairs.
Although it is quite surprising that mainly text websites (Reddit, Twitter) are hard to run sustainably but video and image websites (YouTube, Instagram, TikTok) can because it is easier to sell ads against them.
I don't see how that would fall to different people in reddit's case. I'm sure reddit has some moderators on staff but the vast, vast majority of their moderation happens on the proverbial front lines, which is basically all volunteers. I would hope there's a dedicated abuse team at Reddit that are actually paid people whom the volunteers can kick the truly sick shit to so it can be properly dealt with, but given the corporate culture Reddit has shown over the years, I also wouldn't be awfully surprised if it's JUST down to the volunteers either.
The blocks for MojeekBot, as Cloudflare verified and respectful bot for 20 years, started before the robots.txt file changes. We first noticed in early June.
We thought it was an oversight too at first. It usually is. Large publishers have blocked us when they have not considered the details, but then reinstated us when we got in touch and explained.
I personally feel that this kind of "exclusive search only by Google deal" should result in an anti-trust case against Google. This is the kind of abuse of monopoly power that caused anti-trust laws to be passed in the 1890s.
Usually, to trigger any kind of anti-trust law, you need to have massive market share. In this case, for example, Reddit almost certainly hasn't committed any antitrust violations, because they're a relatively minor player in their market.
Similarly, if you start a vacuum cleaner company, you can make whatever exclusive deals you want. But if you control 80% of the market for vacuum cleaners, then you might need to be more careful about leveraging your market share in unfair ways.
If a company is part of a robust, competitive market (like Reddit), it's usually wiser to let customers vote with their wallets, and leave the government out of it. If a company becomes massively dominant (like Google or TicketMaster), and if it starts pushing exclusive contracts, it's much harder for customers to switch away.
Ah so when reddit uses user content for monetization it's ok but when others do it then it isn't? Reddit may want that double standard but I think the only thing they are going to achieve with this stunt is more people ignoring robots.txt.
how was it being abused. You still clicked on the information and saw the reddit ads? Now they won't get any of that from "rival" sites to google. I guess they figured the 60 million was more than that ad revenue. Seems greedy but I don't think it's illegal like others are suggesting.
yep, but for things which are "only" search engines it's not a viable offer. Only if you expect "big AI business value" from it does it make sense, maybe.
I don't see how this tracks at all. Companies can decide to only sell their products with some retailer if they want. You can't force them to make deals with other companies.
Most business deals are anti-competitive in some way. What makes you think this specifically rises to the level where they'd legally have to offer similar terms to competitors?
Why in the world would they have to do that? There are thousands of exclusive business-to-business deals being signed into action every second of the day.
Worse it doesn't even really "work" anymore, giving how most search are flooded with garbage SEO results and payed advertisements "basically" looking like search results (most times more garbage not what you are looking for results, int he cases where it isn't it quite often times is on the line of "googles algorithm blackmailing companies to buy ads for users which want to find them through google but wouldn't without ads".)
I wonder if this might affect redis, as in slowly kill it's user base especially when it comes to user providing (and often also looking for) high quality content, because who of such users would want to use google search?
> Worse it doesn't even really "work" anymore, giving how most search are flooded with garbage SEO results and payed advertisements "basically" looking like search results ...
I don't understand what you're saying. That's exactly why people append `site:reddit.com` to their searches in the first place, because those search results typically aren't like that.
Or at least, reddit posts and comments that are content messaging / marketing (human or AI) fit in better with earnest and natural posts, so that they're more effective.
FWIW, we inquired to the reddit sales team about paying for data sometime last year, as we do similar elsewhere for use cases like helping emergency responders, and even though they were launching the program and asking for customers... no email back. Nor on our second and I think third attempt.
How much were you willing to pay? Still, rude of them not to even discuss the issue. Every time I've gone to buy data, if I'm too small of a fish, vendors have always been happy refer me to a reseller.
Certainly rude but also possibly legally problematic. If they were judged to be in a dominant position in a market and were found making deals with exclusivity then it can get expensive.
It all depends of course what the market is. If one looks as reddit not as a whole but as a collection of niches then one could imho find niches where reddit has a dominant knowledge position.
We do 4-6 figures/yr for providers which is normal in our world
An enterprise sales team with only 1 customer happens (eg, Mozilla 's search bar), but... That's surprising here, and scary as a sustainable & scalable business. Ignoring 5-6 figure/yr inquiries says a lot to me. In contrast, we did that same-day with Twitter without talking to anyone.
"Information is power. But like all power, there are those who want to keep it for themselves. The world’s entire scientific and cultural heritage, published over centuries in books and journals, is increasingly being digitized and locked up by a handful of private corporations."
- Aaron Swartz (2008)
"If you use Bing, DuckDuckGo, Mojeek, Qwant or any other alternative search engine that doesn't rely on Google's indexing and search Reddit by using "site:reddit.com," you will not see any results from the last week."
The veracity of this statement is questionable.
I found at least four web search engines not using Google's index that produced results from the last week.
Example: Recent eruption at Yellowstone Black Diamond Pool
Up until 2016 (I think, +/- 1 year), if you could remember 3 uncommon words in a comment, you could find any reddit post instantly on Google. I'd want to follow up on a thread from weeks ago, and it was magic. Number one result. Then one day that just stopped working, and even adding site:*.reddit.com didn't fix it. At the time, I think, I didn't realize that it was mostly Google's fault, I thought maybe Reddit had changed their infrastructure so that it couldn't be crawled properly.
Google hasn't been a search engine in a long while, it's just an advertisement engine now.
it's so bad it's crazy, you can legit not find stuff on the internet anymore, it's the same with youtube, I search something and get like 20 or so results and then everything else is hidden.
it started when youtube removed the ability to search for videos older than 5 years, if I had to guess? cost saving, have every old video in cheaper storage... but it sort of fragments youtube, every couple of years you only get newer content.
One day I was searching live videos of a local band from youtube and when sorting by upload date the oldest video was from 2010. I knew for a fact there had to be older videos so I got a youtube API key and searched via the API, ended up finding multiple videos starting from 2006. Learned that youtube is full of videos that are basically impossible to find with the regular search.
One of my past times is looking up anime opening reactions for fun/to hear people listen to bands I enjoy that sometimes do anime ops, searching that is so scary, you see the same 4 people or every month or so you get 1 or get 1 removed, you can't tell me there's not more than 4 people on the internet that upload opening reactions... when anime is a billion+ dollar industry, if you know particular channels you can see daily uploads on plenty of channels, that are simply search banned for whatever reason by the algorithm.
But yeah, the most outrageous one is older videos, I do believe the reason is that they are using some long term cloud storage that is cheaper for older videos so they removed the ability to search by date.
Additionally, I don't believe the API fully fixes it, because Bing has a wrapper for youtube and searches do not really vary
"we noticed that since our search results had gotten so bad nobody can use them to find the things they want, people just kept adding "reddit" to search terms anyways, so we figured we might as well make it official and exclusive"
It's funny in the context of Google's past motto of "don't be evil". I feel the right thing for Google here would have been to decline any deal regarding exclusivity, then Reddit wouldn't have pulled the trigger with its robots.txt update. The entire manoeuvre required both parties.
Google should abandon its mission to “organize the world’s information” because doing so requires spending money for valuable data, and others might not want to spend that money?
Boy, the LLMs have really been an apocalypse moment for the web, haven’t they? Between hoovering up and monetizing every bit of content they can without any attribution or compensation and the absolute flood of mediocre generated content, they’ve really done in the last straggling remains of the open internet.
It’s not like everyone wasn’t already pulling the same grift, but quantity really does have a quality all its own.
Of course, we have to be careful not to villainize a neutral tech. Instead let's call it what it is: unchecked capitalism and monopolistic behaviors.
Capitalism seems to work ok for the common good until you remove all the protections. LLMs provide a defacto monopoly for the owner which must already be a near monopoly: they take vast resources to train; only a giant corp can afford to buy all the content and provision enough resources to train one.
LLM did not enshittify what's left of the internet, greed did it.
On the one hand, you’re absolutely right. But on the other hand it’s not like it matters in practice. Isn’t most technology technically neutral? But it’s also made to be used by people, who can do so beneficially or detrimentally. Criticising a technology is a shorthand for criticising how it’s used.
> Of course, we have to be careful not to villainize a neutral tech
This is a very good point IMO. If we're going to chastise LLM's we may as well give servers, switches, routers, fiber-optic cable, and silicone a bollocking as well since that's ultimately what's facilitating all this.
No, those are not comparable. If someone criticises the electric chair, it’s not reasonable to defend it with “if we’re going to chastise the electric chair we may as well give wood, metal, chairs, and electricity a bollocking since that's ultimately what's facilitating all this”.
Things are more than the sum of their parts. If you have a ton of beneficial things which can be cobbled together into one bad detrimental thing, the existence of the latter does not remove the benefits of the former.
I'm concerned multiple ways by this, but I also could see some positive fallout from this, if it sets precedents that help protect 'content' owners from AI goldrush companies just taking everything.
AI companies are the least of our worries in the Reddit situation. The fact that Reddit has full control of user-generated data to do as they please gives them freedom to do as they please. I think this is the crux of today's issue.
AI companies like Google, Microsoft and OpenAI have deep pockets to 'unprotect' themselves from anything. The barrier to entry is for small AI companies and those aren't really making an impact currently.
While it's still not Reddit, but I've been enjoying Lemmy. I have a similar range of communities on each, and other than some annoying groupthink, the content is often similar.
And to me, forgetting to log in to each of them feels similar, too. For what that's worth. (I hate both of them when not logged in.)
I mostly contributed to r/nonsense and I'm pleased by the thought of that sub's content being used to train future AI, with information about the architectural uses of super-tall chef's hats, the prehistoric invasion of Europe by Beak People, and so forth.
It's weird to say that reddit "works" with google. Every page they serve to google is stuffed full of hidden unrelated content, so any reddit result in google is unlikely to actually contain what you were searching for.
Google really should blacklist reddit entirely for this practice, but sadly as bad as reddit is it's still a much higher quality result than average for google.
Ugh it's absurd at how incompetent Google is at filtering out "related" content or similar volatile "sidebar" feeds in the sitesthey index that has nothing to do with the main content and won't be there when the user actually opens that link.
FYI, Kagi lets you do this and personalize it as you desire. They even share aggregated stats※ about which domains users choose to block/lower. (Mine generally match these stats.)
I sincerely appreciate the diligence. I really did see Pinterest results over a longish span. I may well have only noted their presence and not their absence - skewing my perspective.
Overall, my experience is very positive. I'm on many PCs throughout the day and I miss Kagi when it isn't there.
Kagi doesn't provide it natively because of performance issues. They contributed to ublacklist to add kagi search. You can use it with regex and matching rules
> If you need advanced blocking features, such as specifying rules by match patterns (e.g., ://.example.com/*) or by regular expressions (e.g., /example\.(net|org)/), the uBlacklist browser extension includes support for Kagi Search.
>This is great. It means I won't see Reddit content popping up all over search results in other engines.
Honestly, that makes those other engines way less valuable because for many topics, telling the engine to specifically narrow the results down to reddit comments is the only way to get a decent answer to what you're looking for. I'd definitely support blocking Quora from everything though.
What use do you get out of a search engine if not searching for reddit and other forums? The rest of the internet has become a cesspool of useless AI generated crap.
Interesting. I have long found Reddit to be the an excellent source of solutions to problems. Stack Overflow usually beats it for programming specific stuff, but for everything else usually the most helpful answer comes from Reddit. It's a real person, helping another real person with a real problem.
Kagi lets you configure the search engine to deprioritize or even fully eliminate search results. They ride on the back of Google's indexing so -- if you ever change your mind -- you could bring reddit searches back.
> where real people hang out and have real conversations
I don't consider the discussions there to be "real" in any meaningful way, thanks to the extensive moderation.
From what I've seen, there typically ends up being a small handful of moderator-enforced narratives that are deemed "acceptable" for a given subreddit, and any commenters deviating from those narratives get banned, or their comments end up as "[removed]" by "[deleted]", or the comments get obscured with the "comment score below threshold" notice.
It's generally some of the most one-sided and blandest discussion around. Given that there's often no meaningful back-and-forth involving differing perspectives of any sort, I'm not even sure if it should be considered "discussion". It's more like regurgitation and repetition.
I've found the situation to be particularly bad on the Canadian locale-specific subreddits, for example, but a enough of the tech-oriented ones I've seen seem to end up like that, too.
I think Reddit lost that kind of authenticity a while ago. Advertisers know the "search:reddit.com <product>" trick, and when you look at the number of upvotes, it costs _pennies_ to get your product trending in the comments.
I don't search reddit for <product> though I search it for <highly technical issue with product> because reddit is the only place where real people discuss such issues and the solutions to them.
Its not strange to me. Every single time I've followed a Reddit link from search results, I've got a short and fairly useless conversation that doesn't help me at all. So I have never understood why people like it.
Obviously, people do see value in it, or they wouldn't keep saying so! I would happily exclude Reddit links from search results though.
Yeah, but each sub to a greater or lesser degree, has its own hivemind you'll be run out of town (or possibly even banned) for challenging. And the average member of Reddit is quite willing to spout off confidently incorrect BS and downvote people into the ground who actually know what they're talking about.
Not exactly always a reliable source of info outside uncontroversial niche topics or places like /r/AskHistorians that actually moderate. And even there I've seen the occasional humdinger.
Stopped using reddit after they hindered login-less viewing and blocked vpns. Everyone who respect themselves should start moving away from it imho. Same thing with google
Fine with this. This is the world OpenAI created. And all the people that started searching with +Reddit tacked on weirdly like 5 years ago. Reddit's covering themselves from internal user-concern and their general exposure to AI training and Google was smart enough to get on that quickly. We'll see what Bing's take is and what changes if anything now that 404medias's outrage farming is at play. This isn't a recent change afterall, month ago?
Thinking from reddits perspective they have nothing to lose really. It’s not like other search engines are going to pay any attention to the robots txt and Google’s AI would have still scraped data from Reddit regardless of the deal. Now they will just feel less bad about not citing sources possibly, depending on the user experience they want to deliver.
Proxy IPs are also known and typically blocked. In fact, you can't even browse reddit without logging in when connected to most proxies.
Many web scraping companies have loads of phones hooked up in a rack in order to use mobile IPs. Companies can't just block mobile IPs because their site would become unusable for several city blocks (mobile IPs often correspond to a specific cell tower). This is the face of modern web scraping: https://i.imgur.com/U2RXi5G.jpeg
Just like every security feature in the physical and digital worlds, security just inconveniences honest people and the cost to bypass reduces the amount of people who try.
Eventually it becomes expensive to scrape reddit's data and most people will stop.
Remember, the only reason Reddit "won" was because Digg destroyed itself with a radical upgrade that everyone hated.
Reddit would have to do something similarly self-inflicted, and I can't even guess where people would go. Reddit was already an alternative to Digg -- what's the alternative to Reddit? I mean, it's certainly not Quora.
The main thing I see Reddit being useful for are discussions about entertainment.
There's probably a subreddit for your favorite sports team, twitch steamer, TV show, book series video game, politics (which is entertainment for some people).
Reddit has seriously degraded the experience of a lot of these communities with things like restricting custom CSS.
It seems to me that the way you'd disrupt Reddit as a startup is to pick a vertical and laser focus on becoming the best discussion board for that community. If it's sports than have integrations for live stats, scores, etc.
In general you could attract users by offering profit sharing on ads the same way Youtube does for creators.
Have the best moderation tools in the world, a constant painpoint with Reddit. Give admins more flexibility over the appearance of the board, all things Reddit took away.
The other path for disruption would be if an established company with those communities tackled the problem. Lots of communities already us Discord, but they tend to also have a subreddit because chat and forums are different communication methods. Discord could easily offer a forum product as an extension of their chat services. If they do it well they'd drive a lot of users away from the subreddits.
It was already dead by then. Really, it was the various Slashdot exoduses... sites like K5 got large initial boosts, but stumbled and started to deteriorate. If the Digg exodus is what sent you to Slashdot, chances are you're the kind of user everyone else was trying to escape.
>what's the alternative to Reddit? I mean, it's certainly not Quora.
If it was deliberate I certainly can't tell, but one of the characteristics of Reddit is that it caused so many other little tiny internet forums to just wither away. Most were visually unappealing, running some ancient phpbb software or whatever, but there were so many like stars in the night sky. Now, if they're even still running, you look for the newest post, and it will say "November 2023". Hell, the only reason they are still running is that the credit card number on file paying for hosting doesn't expire until next year somehow. Reddit is a red tide algae choking out all life in the ocean, nothing else gets to exist anymore.
>Reddit was already an alternative to Digg -- what's the alternative to Reddit?
This site is essentially 'orange reddit', they just need to add sub-HNs or tagging or something and it'd be ready for an influx of reddit refugees. Not that any of really want it, but it's possible.
Reddit is quietly a huge website with a significant amount of users. So many people use it but dont talk about it. Google search says 1billion mau? Twice as big as Twitter
there are plenty of reddits outside of the "popular ones" that have invaluable information in one place and I can ask questions. It still has value for me, and not much value for axis of evil bots to spread political disinformation like popular subs and on twitter.
The strange thing to me is how everybody keeps trying to make distributed Twitter happen when distributed Reddit is the low hanging fruit for federated social media.
You don't want to end up banned from a movies forum because you also participate in a political forum. Federation solves that problem because you can use separate accounts without either forum knowing that you also use the other.
>The strange thing to me is how everybody keeps trying to make distributed Twitter happen when distributed Reddit is the low hanging fruit for federated social media.
Honestly, it's strange to me how hard people are trying to make distributed anything happen. Federation mostly solves a problem that real people don't have or care about.
>Honestly, it's strange to me how hard people are trying to make distributed anything happen.
IMO, something federation is very good at is solving one slow-moving problem - enshittification of social platforms. It's not immune, of course, but an Elon Musk-style takeover is much harder with Mastodon than Twitter, and it would be hard to run it into the ground in other ways because the platforms are owned by different people and groups.
Is this not just what the internet was before reddit? What features would "distributed reddit" have that an internet full of independent community forums be missing?
It's not possible because the most common problems with running a forum are spam and moderation, and both of those are too much work unless it's centralized.
Very few of the reddit users who are providing the content for free are motivated by which search engines are allowed to index the content, so I don't see how this would make it more ripe for competition. (If you just mean society would now be even better off if reddit were disrupted, ok, maybe, but that's a different thing.)
The killing of third party clients didn't have significant impact, I don't know what would they have to do to lose users, other than some kind of mandatory subscription fee.
Networks effects are more powerful than we are. Witness the number of people who despise Xhitter but are still on there. Once something has a sufficient network effect they become immune to normal market forces and able to abuse their position with near impunity.
