To comply with a state law requiring that school library books be “age appropriate”, an Iowa school district used ChatGPT to find books depicting “a sex act”

Faced with new legislation, Iowa's Mason City Community School District asked ChatGPT if certain books 'contain …

Popular Science 2023-08-16 Andrew Paul

Context & Ripple Effects

This use of ChatGPT sits in an early school-sector debate that initially focused on restricting student access: New York City schools had blocked ChatGPT on school devices and networks over cheating concerns, while educators argued that outright school bans would be difficult to sustain.

Mason City instead applied the tool to an administrative compliance task under Iowa's age-appropriateness requirement. That shifts the question from whether schools permit generative AI to how they verify its use in decisions affecting collections and access.

First-order effects

Mason City can use ChatGPT as an initial screen for books that may require review under the state standard, changing the workflow for district staff responsible for library collections.
Books flagged by the model may receive closer scrutiny, making the quality and consistency of the district's human review consequential for authors, students, and families.

Second-order effects

Other districts facing similar compliance duties may test generative AI for catalog review or content triage, but will need procedures to check outputs against the applicable policy rather than treat model responses as final determinations.
The episode broadens schools' AI governance burden beyond student cheating: administrators must set rules for documentation, review, and accountability when AI informs operational decisions.

Third-order effects

If public institutions increasingly use general-purpose models to implement content rules, AI governance will move toward auditable human-in-the-loop processes for high-stakes classification tasks.
The longer-term fault line is likely to be less about simple access bans and more about who sets the standards, validates AI outputs, and bears responsibility when automated screening is wrong.

The trend: Generative AI is moving in education from a tool schools tried to restrict to one they may use for administrative classification, creating demand for oversight around consequential decisions.

Discussion

@brianmerchant@mastodon.social Brian Merchant on mastodon
This is a good example of what AI systems like ChatGPT excel at—giving managers and administrators the ability to automate top-down, controversial and harmful decision-making while shielding themselves from accountability. They're a sort of authoritarianism accelerator — https…
@Mer__edith@mastodon.world Meredith Whittaker on mastodon
Few things fill me with dread like the current organized censorship of literature, history, access to knowledge — Then I learn they're consulting ChatGPT like some computational divining rod because no one can be bothered to read books before banning them. — I'm impossibly ti…
@HuShuo@mastodon.social David Hull on mastodon
Asking AI/LLMs to tell us which books we can read is surely a fine idea with no problems at all. — /s — An Iowa school district is using AI to ban books — https://www.engadget.com/... #BookBanning #AI #LLM #Iowa #Dipshittery
@no1lion99@mastodon.social @no1lion99@mastodon.social on mastodon
If there's a more perfect intersection of #techbro and #qanon cultures than this I can't think of it. #bookbans #ChatGPT #iowa #librarians https://www.popsci.com/...
@drewharwell@mastodon.social Drew Harwell on mastodon
Iowa school district asked ChatGPT which books had sex in them, then banned them, including “Beloved,” “Friday Night Lights” and “Handmaid's Tale.” We must protect high schoolers from independent thought at all costs https://www.popsci.com/...
@bcmerchant.bsky.social Brian Merchant on bluesky
A good example of what AI systems like ChatGPT excel at: giving managers + administrators the ability to automate top-down, controversial + harmful decision-making while shielding themselves from accountability. Authoritarianism accelerators. — https://www.popsci.com/...
@emilymbender.bsky.social Emily M. Bender on bluesky
1) Don't ban books — 2) Don't waste educators' time with book bans — 3) Don't make decisions about books without reading them — 4) Don't fall for OpenAI's interface: ChatGPT is not the kind of thing you can sensibly ask for information — https://www.popsci.com/...
@meredithmeredith.bsky.social Meredith Whittaker on bluesky
Few things fill me with dread like the current organized censorship of literature history knowledge — Then I learn they're consulting ChatGPT like some computational divining rod because no one can be bothered to read books before banning them — https://www.popsci.com/...
@andrewpaul.bsky.social Andrew Paul on bluesky
you may have seen headlines about an Iowa school district using AI to help ban books — but the story is more complicated and sadder than that — my latest find for @popsci.bsky.social — https://www.popsci.com/...
@nickadobos Nick Dobos on x
Based Y'all realize this is exactly how social media moderation works, right? Literally down to “new [bullshit] law requires” Half assed implementation Algo flags it No ones paying for human review Hope it was correct 👍
@quinnypig Corey Quinn on x
Curious that “a sex act” is their criterion instead of “an act of violence.”
@scottjshapiro @scottjshapiro on x
AI horror of the day: https://www.popsci.com/...
r/Libraries r on reddit
School district uses ChatGPT to help ban library books: Faced with new legislation, Iowa's Mason City Community School District asked ChatGPT …
r/technology r on reddit
School district uses ChatGPT to help remove library books | Faced with new legislation, Iowa's Mason City Community School District asked ChatGPT …
@AnnemarieBridy@mastodon … Annemarie Bridy on mastodon
Platforms automating content moderation owe their users something more and better than an AI classifier that “outperforms [human] moderators that have had light training.” — Can ChatGPT become a content moderator? Semafor https://www.semafor.com/...
@openai @openai on x
We've seen great results using GPT-4 for content policy development and content moderation, enabling more consistent labeling, a faster feedback loop for policy refinement, and less involvement from human moderators. Built on top of the GPT-4 API: https://openai.com/... [video]
@emollick Ethan Mollick on x
Automation traditionally has been about eliminating terrible, repetitive & dangerous jobs. Here is an example of AI doing just that. Social network moderation is a bad job that exposes you to real mental health effects👇 (As a former Wikipedia administrator, I saw it first hand) […
@rachelmetz Rachel Metz on x
content moderation is hard for people (for some reasons obvious such as having to view awful stuff), and it's harder for ai (for various reasons, such as grasping context). so it'll be interesting to see how this goes. from the fabulous @Priyasideas. https://www.bloomberg.com/...
@marktenenholtz Mark Tenenholtz on x
GPT-4 for content moderation seems impractical at scale given current prices. A more cost-efficient pipeline: • Fine-tune a small classification model (~300M to 7B params, probably) • Set a very high-recall probability threshold • Anything caught by the first stage model...
@officiallogank @officiallogank on x
Big win for a healthy internet: anyone using the @OpenAI API can also use our world class content moderation endpoint to support best in class moderation with minimal setup time. https://openai.com/...
@caseynewton Casey Newton on x
I talked to some experts about OpenAI's new pitch for platforms to moderate content using GPT-4. They were surprisingly enthusiastic https://www.platformer.news/ ... [image]
@samhouston Sam Houston on x
@rachelmetz @Priyasideas i once worked as a moderator who had to flag nudity and bullying content. I personally think it will be great if AI can help take over a lot of that work. it feels really demeaning to do that stuff and i still have moments 9-10 years later that are bad me…
@then_there_was Andrew Ruiz on x
@webdevMason Hopefully one day we'll have an LLM as an intermediary to filter out the truly insane replies. I don't think that future is far away. Could probably be implemented today too with GPT-4 and a custom Twitter client. https://twitter.com/...
@reedalbergotti Reed Albergotti on x
@cartwheelit @Techmeme you might be able to prompt your chatbot to say something inappropriate. that is what could go wrong.
@reedalbergotti Reed Albergotti on x
New: OpenAI is now using GPT-4 to moderate ChatGPT. Still not quite as good as a highly-trained team of human moderators, but it's almost there. https://www.semafor.com/...
@gdb Greg Brockman on x
GPT-4 for content moderation. Very reliable at this use-case (dark blue bars are GPT-4, other bars are well-trained and lightly-trained humans) & speeds up iterating on policies (sometimes literally from months to hours). [image]
r/singularity r on reddit
Using GPT-4 for content moderation

Chronicles