Powerset Testing Search Results At Mechanical Turk
A reader noticed that stealth search engine Powerset is using Amazon's Mechanical Turk service to gauge user reactions to search results. — See the screen shot (click for larger view) - users are shown a query and a number of results …
Context & Ripple Effects
Powerset spent mid-2007 as the most-watched stealth search company: June brought both a framing of it as a natural-language search mashup platform and VentureBeat's claim its technology could scare Google, alongside Techdirt asking whether there was anything behind the buzzwords and patent talk. What was missing was evidence about whether its results were actually any good.
This sighting answers that quietly: rather than wait for a public launch, Powerset is buying relevance judgments through Amazon's Mechanical Turk, showing paid users a query and result set and recording reactions. The technique has lineage — Greg Yardley argued back in 2005 that the Mechanical Turk model was genius precisely because small paid judgments scale — and it lets a company tune quality before exposing itself to comparison against Google.
First-order effects
- Powerset gets a continuous, low-cost stream of human relevance feedback on its natural-language results while still in stealth, letting it iterate on quality without betting its reputation on a public launch.
- Amazon adds a marquee search-industry customer to Mechanical Turk, extending the service beyond bulk transcription tasks into evaluation and QA work.
Second-order effects
- Other pre-launch search and NLP startups now have a visible template: outsourced human judgment replaces internal usability panels, pushing evaluation costs toward cents-per-task and letting small teams benchmark against incumbents like Google.
- Relevance-testing demand gives Mechanical Turk requesters a new job category — comparative judgment on search results — shifting the marketplace mix away from simple data-entry HITs.
Third-order effects
- If this pattern holds, crowd-labor platforms become standing infrastructure for search and AI companies, with human evaluators embedded in the product loop rather than hired ad hoc — raising questions about pay rates and task design that the marketplace operators, not the buyers, will have to answer.
- Search competition shifts partly to who can run the tightest evaluate-and-tune cycle; a stealth team with cheap human feedback can close quality gaps on incumbents faster than launch-and-react cycles allow.
The trend: Search and AI startups are folding crowdsourced human judgment directly into product development, turning marketplaces like Mechanical Turk into core evaluation infrastructure.