Facebook details Ego4D, a research project in partnership with 13 universities that uses first-person video to improve perception by AI assistants
cataloguing not just what you say but the physical world around you. Such systems could be incredibly useful, of course, but have huge privacy implications. https://www.theverge.com/... https://twitter.com/... James Vincent / @jjvincent : What does Facebook imagine its AR glasses will be able to do in the future? A new research project from the company's AI team reveals its ambitions, including using AI to create augmented memory — transcribing and recalling what anyone says to you https://www.theverge.com/... https://twitter.com/... @facebookai : We're announcing #Ego4D, an ambitious long-term project we've embarked on w/13 universities in 9 countries to advance first-person perception. This work will catalyze research to build more useful AI assistants, robots & other future innovations. https://ai.facebook.com/... https://twitter.com/... Yann LeCun / @ylecun : Ego4D: 2200 hours of first-person videos collected from 700 persons around the world going about their daily activities. Labels provided for a number of tasks relevant to AI assistants, robotics, etc. A collaboration between Facebook AI, FRL Labs, and 13 universities worldwide. https://twitter.com/... James Vincent / @jjvincent : It's telling, I think, that when I asked Facebook why these research challenges did not include privacy features (like opting out of being recorded) the company said it expected these would simply be developed down the line. https://www.theverge.com/... https://twitter.com/... Mike Cook / @mtrc : It's easy to forget that a lot of new technological “frontiers” are just a single very powerful company or person wanting to push an idea. There's no need for egocentric AI. There's no need for smart glasses. But Facebook wants you to think there is, and that they're inevitable. @hypervisible : On the one hand, you let Facebook follow and record your every living moment, but on the other—you will never again lose your keys. Hard to argue with that kind of tradeoff. 🙄 https://twitter.com/... Evan Selinger / @evanselinger : Today, there aren't enough compelling use cases to make smart glasses seem necessary for most people. Sadly, FB stands a good chance of selling the idea of enhancing experience via upgrading our extended minds. Per normalization handbook, I bet therapy cases get emphasized first https://twitter.com/... Rakesh / @densaer : I love technology but man this is an example of why so many technologists are insufferable. Too busy for ethics? Well, how'd that work out for humanity? https://twitter.com/... Timnit Gebru / @timnitgebru : The computer vision community strikes again. You can find all your favorite current & future surveillance & biometric tech with zero thoughts about consent or privacy (except to hypothetically be implemented “down the line"), if you go to any of these computer vision conferences. https://twitter.com/... Tom McKay / @thetomzone : Facebook says it's doing a bunch of things. For example raise your hand if you took place in the Facebook Portal revolution https://twitter.com/... Ayush Gupta / @ayushumd : I once asked a computer vision prof about ethics and harm being done. He said, “the field moves so fast. I don't have time to sit and reflect on ethics” https://twitter.com/... Mike Cook / @mtrc : This is another reason why relying on private companies to do scientific research doesn't work. Facebook is full of lovely people, I'm sure. But Facebook itself is not interested in what humanity needs, what makes the future better, or what the most interesting questions are. @onekade : I personally think this is horrifying. But if you love the idea of allowing a company like Facebook to track everything you do in real life, that's your problem. It becomes my problem when that technology tracks me against my will. https://www.theverge.com/... David Carroll / @profcarroll : We'll add the privacy features later. 🚩🚩🚩🚩 🚩🚩🚩🚩 🚩🚩🚩🚩 🚩🚩🚩🚩 https://twitter.com/... Matt Beane / @mattbeane : #imagenet - curated images for computer vision - unlocked genuine technical breakthroughs, perhaps a trillion dollars of potential value, and a universe of possible (some extremely nasty) side effects. Video is the next - but more complex - frontier. So this is a VERY big deal. https://twitter.com/... LM Sacasas / @lmsacasas : Under the computational spell we are tempted to forget the uses of forgetting. https://twitter.com/... Soumith Chintala / @soumithchintala : Really excited about this dataset. It is going to greatly help move Robotics and Embodied intelligence forward! https://twitter.com/... @facebookai : Read more about all five #Ego4D benchmarks: https://ai.facebook.com/... Hanbyul / @jhugestar : Excited that our Ego4D is finally out! Ego4D is an exciting milestone to define and explore next computer vision and AI research problems. Via years of collaborations with amazing researchers over the world! https://twitter.com/... Devi Parikh / @deviparikh : Largest publicly available egocentric dataset and associated benchmark tasks! Huge effort led by Kristen Grauman with a large team of collaborators. Involved a consortium of 13 universities and labs across 9 countries, more than 2,200 hours of first-person video in the wild. https://twitter.com/... Jerome Pesenti / @an_open_mind : This is @facebookai's most ambitious data collection and benchmark definition to date. An amazing collective effort, done with the greatest attention to privacy and responsibility, that will enable us and the broader AI community to create AI that learns more like humans do. https://twitter.com/... @hypervisible : Btw, this is a really interesting contrast to what other prominent Facebook people have said about their upcoming projects, privacy, and public space. One could see the statements here as direct contradictions to what had been said elsewhere. Mike Schroepfer / @schrep : How can we teach machines to learn about the world the way humans do? It's one of the hardest and most promising opportunities in AI, so we brought together a global consortium of universities and research labs to explore it. https://ai.facebook.com/... Mike Schroepfer / @schrep : To get there, we need AI capable of seeing things from a first-person perspective - researchers call it egocentric perception. That's the goal of the Ego4D project announced today, there lots more detail about it here: https://ai.facebook.com/... @techreview : A massive new video dataset will help train smart assistants and robots—but there are obvious concerns about how the tech will be used. https://www.technologyreview.com/ ... Will Douglas Heaven / @strwbilly : Facebook wants to kickstart a new era of image-recognition, assembling 3000+ hours of first-person video to train machines to see the world through our eyes https://www.technologyreview.com/ ... Paris Marx / @parismarx : There certainly needs to be regulation of tech and AI, but I feel like we need to be having a serious conversation about whether Facebook should exist at all. https://twitter.com/... Mat Honan / @mat : Facebook is creating AIs that can understand scenes and activities viewed from a first-person perspective. AIs trained on the data set will be better at controlling robots that interact with people, or interpreting images from smart glasses. https://www.technologyreview.com/ ... Carnegie Mellon / @carnegiemellon : #TartanProud to be one of the universities to help compile and have access to the largest collection of point-of-view videos in the world. This collection will allow researchers to train computer vision systems to see the world as humans do. https://www.cs.cmu.edu/... via @SCSatCMU https://twitter.com/... Matt Navarra / @mattnavarra : Facebook captured more than 2,000 hours of first-person video to train next-generation A.I. https://www.theverge.com/... Mike Schroepfer / @schrep : Teaching AI to think more like a person would, and anticipate what's coming next - really impressive work here https://twitter.com/...
Context & Ripple Effects
Ego4D extends Facebook's earlier effort to train AI on publicly available Facebook video into first-person footage collected with a 13-university consortium. It also gives the company a research layer alongside Project Aria's AR-glasses testing, where understanding a wearer's surroundings is central to the product direction.
The project matters because its stated assistant applications require recording people and environments beyond the wearer. Criticism over the absence of built-in consent or opt-out mechanisms makes data governance part of the project’s immediate stakes, not a downstream implementation detail.
First-order effects
- Facebook AI, FRL Labs, and the university consortium gain a labeled first-person-video corpus for research on egocentric perception and AI assistants.
- People captured around Ego4D participants face privacy exposure without the built-in consent and opt-out features cited by critics.
Second-order effects
- Facebook's AR effort can use Ego4D research to sharpen the perception capabilities needed for glasses-based assistants, linking the dataset more directly to the direction established by Project Aria's field testing.
- University partners become participants in Facebook's data-collection and labeling pipeline, making their research collaboration subject to the same consent concerns surrounding the corpus.
Third-order effects
- If first-person datasets become a standard input for assistant development, privacy safeguards for bystanders will become a competitive and governance boundary for wearable-AI research.
- The project points to a shift from models that interpret uploaded video toward systems trained to understand a user's continuous physical context, with data permission becoming a limiting factor.
The trend: AI assistant development is moving toward egocentric, real-world perception datasets, making bystander consent a core constraint on wearable computing.