Internal memo: Andrew Bosworth says Meta's virtual worlds need “almost Disney levels of safety” but moderation at “meaningful scale is practically impossible”

from keeping a platform safe to correctly enforcing your own app store acceptance policies—seems to be impossible. https://www.ft.com/... Hannah Murphy / @mshannahmurphy : He also says third party VR developers on its platform should have slightly more lenient standards - in order to allow users a broader spectrum of experiences https://twitter.com/... Evelyn Douek / @evelyndouek : Content moderation (or lack thereof) in the stack watch https://www.ft.com/... https://twitter.com/... Kontra / @counternotions : “Facebook can't moderate their existing platform. How can they moderate one [Metaverse] that is enormously more complex and dynamic?” (We'll get used to it.) https://twitter.com/... Mark Di Stefano / @markdistef : Internal docs show Facebook's VR universe has a racism and abuse problem https://www.theinformation.com/ ... @malcolmmoore : An internal Facebook memo outlines its early thoughts on tackling toxic behaviour on its metaverse. Scoop from @MsHannahMurphy https://twitter.com/... Olivia Solon / @oliviasolon : “Facebook can't moderate their existing platform. How can they moderate one that is enormously more complex and dynamic?” said John Egan, chief executive of forecasting group L'Atelier BNP Paribas https://twitter.com/... Amir Efrati / @amir : When people scream at you in VR, it hurts a lot more than getting a mean tweet. by ⁦@SylviaVarnham⁩ ⁦@mathewolson⁩ https://www.theinformation.com/ ...

Financial Times 2021-11-13 Hannah Murphy

Context & Ripple Effects

Meta had already put content ranking and display at the center of its metaverse design discussion in Bosworth's earlier account of ranking and moderation trade-offs. The memo makes the operational constraint explicit: the safety standard Meta wants for virtual worlds conflicts with its own assessment of what can be enforced at meaningful scale.

That tension matters against Facebook's documented history of managing moderation across a vast, multilingual service in its earlier content-moderation apparatus. Meta is also pursuing the VR market as a strategic platform, following the ownership rationale outlined in Oculus's earlier VR-market blueprint.

First-order effects

Meta's virtual-world policy team must reconcile a high safety target with Bosworth's conclusion that large-scale enforcement is practically impossible.
Third-party VR developers would face somewhat looser standards than Meta's own experiences, widening the range of permitted content on the platform.

Second-order effects

Meta's ranking and display systems become a more consequential control point, because they must balance private interaction and moderation where uniform enforcement cannot meet the stated safety bar.
Developers gain more latitude, but Meta must manage the resulting gap between its safety positioning and the experiences distributed by outside creators.

Third-order effects

If Meta continues building VR as a platform while applying differentiated standards to first- and third-party experiences, trust and safety becomes a core platform-governance constraint rather than a standalone moderation function.
The pattern points to immersive platforms competing not only on ownership and distribution, but on whether their enforcement systems can support the safety promises used to attract users and developers.

The trend: Immersive platforms are confronting trust and safety as platform infrastructure, with scale limits shaping how much openness third-party ecosystems can sustain.

Discussion

Vox Peter Kafka on x
Meta made over 21 acquisitions in the past three years, per PitchBook, including at least four VR startups in 2021; sources say it paid $500M+ for Within
@mshannahmurphy Hannah Murphy on x
SCOOP: Boz, the exec leading Facebook's push into the metaverse, has told employees he wants its virtual worlds to have “almost Disney levels of safety” But also acknowledged that moderation “at any meaningful scale is practically impossible” https://www.ft.com/...
@mshannahmurphy Hannah Murphy on x
Interesingly, when Boz says content moderation at scale is “pratically impossible” he cites the work of @mmasnick and recommends staffers read his posts. Lots of food for thought here: https://www.techdirt.com/... https://www.techdirt.com/...
@boztank Boz on x
We want everyone to feel like they're in control of their VR experience and to feel safe on our platform. Full stop. https://www.oculus.com/...
@hblodget Henry Blodget on x
The main problem with today's big Metaverse vision is the idea that normal people will spend a lot of time wearing VR goggles. They won't. They're hot and uncomfortable and give you a headache. (In “Neuromancer,” Case got neural implants. He didn't wear colossal goggles). https:/…
@mims @mims on x
To find out what the future will feel like, @JoannaStern spent 24 hours in the metaverse. Yes, that's 24 hours in a VR headset. “So what did I learn from this adventure, other than I'd never recommend you do this? Well, a few things...” https://www.wsj.com/...
@tomgara Tom Gara on x
The issue with computers is they'll never fit in a regular persons home or office: they're far too heavy and the hundreds of vacuum tubes generate too much heat https://twitter.com/...
@zeddary Zeddy on x
‘Disney levels of safety’ is what Facebook already has. Zero nudity but willful ignorance of all the barely coded antisemitism. https://twitter.com/...
@arictoler Aric Toler on x
If they enforce this, within 9 months the ‘metaverse’ is going to be nothing but moms doing multi-level marketing scams. If they don't enforce this, it'll be MLM, plus a bunch of soon-to-be-divorced middle-aged men trying to do Second Life-style cybering. https://twitter.com/...
@mshannahmurphy Hannah Murphy on x
Boz warns that VR can often be a “toxic environment” especially for women and minorities, calling this an “existential threat” to FB's metaverse plans if it drives away users The memo, from March, then outlines FB's early thoughts on tackling toxic behaviour in virtual spaces htt…
@darth Darth™ on x
throwing up your hands and saying ‘we just are going to grow too big to manage ourselves!’ is a choice tbh
@iwriteok Robert Evans on x
i would like my new line of racecars to have Disney levels of safety, but at this stage it's impossible to add seat belts or airbags https://twitter.com/...
@fraying Derek Powazek on x
Make it harder to post. Slower. Make people preview their words. Don't let them post if it's too long or too short or has errors or contains bad information. You think I'm reacting to Facebook, but I'm not. This was in my book that was published before Facebook existed.
@kavyapearlman @kavyapearlman on x
Hope everyone enjoyed @ARealityEvent , here is part of d reason why @XRSIdotorg prioritized & led #Safety + #security discussions at d most trusted conference in #XR @boztank no Bro! DISNEY level isn't gonna cut it- ever heard of @NASA TRL? Adopted in https://xrsi.org/... https:/…
@fraying Derek Powazek on x
That's the secret, y'all. Make smaller communities. They're better for the members, and easier to manage, and tend not to destroy democracy.
@edzitron Ed Zitron on x
@darth I'm still laughing that that this got called a scoop. Media still falling over themselves to help Facebook
@grrosegr Gregory Rose on x
@MsHannahMurphy @mmasnick This argument is fundamentally flawed because it focuses on moderation and not designing products and UI to reduce abuse in the first place
@actnormalorelse Lauren on x
facebook says virtual reality will be your desk job. amazon says space stations will be an office park. microsoft to announce that they've integrated time travel into Teams and it will allow you to have more meetings
@jeffhorwitz Jeff Horwitz on x
@grrosegr ... Would be interested in @mmasnick's take on the adoption of the idea in this context.
@darth Darth™ on x
facebook can say “gee who knew we would get so big lol oh well too late now” about facebook but meta and the metaverse is starting anew and yet they have already given up even trying to do better
@davidclinchnews David Clinch on x
Safety in the Metaverse needs to happen via a process of ethical product development. https://twitter.com/...
@mmasnick Mike Masnick on x
@JeffHorwitz ... I am *all* for better initial design & creating a better framework to avoid having to handle the mess at the end with content moderation. That's almost part of the point of that piece. The underlying incentives matter, and waiting until after to clean up the piec…
@chuckbaggett @chuckbaggett on x
You can't really collapse physical distance. It's so inaccurate it's not a good figure of speech. The idea that virtual words collapse physical distance is also extremely trite. It was untrue for Second Life and Active Worlds and every other VR/VW place that made the claim. https…
@fraying Derek Powazek on x
So when Facebook (or anyone) says that moderation at scale is impossible, what they mean is, “moderating this place that we've designed to make it as easy to post as possible for as many people as possible as fast as possible ... is impossible to moderate.”
@fintechfrank Frank Chaparro on x
Most crypto metaverses don't require goggles at all...it's like experiencing the regular internet but in a more 3 dimensional way https://twitter.com/...
@mmasnick Mike Masnick on x
Oh no. https://twitter.com/...
@genepark Gene Park on x
feeling like a broken record: stop equating the metaverse with VR. there is only one major company pitching the metaverse as VR focused, Zuckerberg's. https://twitter.com/...
@ajitpai Ajit Pai on x
“Everyone is blabbing about the #metaverse. But what does this future digital world look like? @WSJ's @JoannaStern checked into a hotel and strapped on a virtual-reality headset for the day.” An eye-opening look at a virtual future. https://www.wsj.com/...
@stevesi Steven Sinofsky on x
“Ethan Zuckerman, ...of the Institute for Digital Public Infrastructure at the U of Mass Amherst, said that in order to prevent spamming or harassment, the company could consider a review process for developers similar to Apple App Store requirements.” 🤔 https://www.ft.com/...
@matthewfederman Matthew Federman on x
Unfortunately, it will be SONG OF THE SOUTH era Disney. https://twitter.com/...
@minimaxir Max Woolf on x
literally the first thing VR developers should accept about VR is that people are going to have extremely kinky sex in it and there's nothing they can do about it https://twitter.com/...
@rmac18 @rmac18 on x
five years ago this same exec wrote that people may die in a terrorist attack organized on facebook's products https://twitter.com/...
@edzitron Ed Zitron on x
Disney levels of safety is an incredible non statement. What does that mean? Is Disney...safe? https://twitter.com/...
@_alialkhatib Ali Alkhatib on x
is there anything more apt than a mediocre white guy who might literally have never worked hourly in his life apparently thinking that a place like disneyland can exist for more than 10 seconds without LITERALLY millions of cumulative hours of labor of mostly women & POC workers?…
@karlbode Karl Bode on x
we can't properly manage the things we build at the massive scale we build them, but we're going to build them anyway because, money https://twitter.com/...
@martyswant Marty Swant on x
Moderating the metaverse would be like trying to moderate millions of live video streams all at once...Pretty much impossible. https://www.ft.com/...
@lanceulanoff Lance Ulanoff on x
Facebook is facing the difficulty of building another earth. People, whether in real life or the Metaverse, will keep being people. It's unlikely a VR environment will be able to change that or effectively manage the messiness of meta-life https://www.ft.com/...
@joannastern Joanna Stern on x
And, for the record, I didn't hear anything overly offensive in my hours in AltspaceVR rooms, clubs and parks. The comedian at a club did call my avatar pretty, which was, well, maybe the highlight of my year. She is rather stunning: https://twitter.com/...
@joannastern Joanna Stern on x
There are moderators, virtual “bouncers” and user tools but what happens when this all scales. https://www.wsj.com/... https://twitter.com/...
@joannastern Joanna Stern on x
Step your avatar into a virtual bar or a public park in AltspaceVR and you can get a sense of the massive metaverse moderation challenge ahead. Live audio from basically anonymous avatars. It was one of my clearest takeaways from my 24 hours in VR. https://www.wsj.com/... https:/…
@mshannahmurphy Hannah Murphy on x
NB nowhere does Boz mention using AI for more proactively screening/filtering out the worst stuff When asked, Facebook told me it was “exploring how best to use AI” in Horizon Worlds (its VR social game that is in beta testing) adding that it was “not built yet”
@peterwsinger Peter W. Singer on x
And thus he perfectly encapsulates the problem... https://twitter.com/...
@somebadideas Aaron Stewart-Ahn on x
This is what Facebook Metaverse is gonna end up like tho https://twitter.com/... https://twitter.com/...
@alexweprin Alex Weprin on x
Disney maintains an iron grip on its owned platforms and theme parks to ensure that brand safety. That does not appear to be in the cards for the metaverse. https://twitter.com/...
@harrymccracken Harry McCracken on x
Doing almost anything that involves decision-making well at scale—from keeping a platform safe to correctly enforcing your own app store acceptance policies—seems to be impossible. https://www.ft.com/...
@mshannahmurphy Hannah Murphy on x
He also says third party VR developers on its platform should have slightly more lenient standards - in order to allow users a broader spectrum of experiences https://twitter.com/...
@evelyndouek Evelyn Douek on x
Content moderation (or lack thereof) in the stack watch https://www.ft.com/... https://twitter.com/...
@counternotions Kontra on x
“Facebook can't moderate their existing platform. How can they moderate one [Metaverse] that is enormously more complex and dynamic?” (We'll get used to it.) https://twitter.com/...
@markdistef Mark Di Stefano on x
Internal docs show Facebook's VR universe has a racism and abuse problem https://www.theinformation.com/ ...
@malcolmmoore @malcolmmoore on x
An internal Facebook memo outlines its early thoughts on tackling toxic behaviour on its metaverse. Scoop from @MsHannahMurphy https://twitter.com/...
@oliviasolon Olivia Solon on x
“Facebook can't moderate their existing platform. How can they moderate one that is enormously more complex and dynamic?” said John Egan, chief executive of forecasting group L'Atelier BNP Paribas https://twitter.com/...
@amir Amir Efrati on x
When people scream at you in VR, it hurts a lot more than getting a mean tweet. by ⁦@SylviaVarnham⁩ ⁦@mathewolson⁩ https://www.theinformation.com/ ...

Chronicles