The Secret Score That Decides Which Families Get Investigated: How Child-Welfare "Risk Algorithms" Quietly Profile the Poor, Black, and Disabled — With No Notice and No Appeal
The Secret Score That Decides Which Families Get Investigated: How Child-Welfare "Risk Algorithms" Quietly Profile the Poor, Black, and Disabled — With No Notice and No Appeal
I have rich, well-sourced material. Here is the investigation.
I have rich, well-sourced material. Here is the investigation.
The Secret Score That Decides Which Families Get Investigated: How Child-Welfare "Risk Algorithms" Quietly Profile the Poor, Black, and Disabled — With No Notice and No Appeal
When a call comes into a county child-abuse hotline, a parent imagines a human being weighing the facts. Increasingly, a machine weighs them first. Across at least two dozen states, child protective services agencies have wired predictive-analytics engines into their front doors — software that scrapes years of Medicaid claims, jail bookings, welfare enrollment, behavioral-health visits, and juvenile-probation records to spit out a single number rating a family's "risk." The best-known of these, Allegheny County, Pennsylvania's Allegheny Family Screening Tool (AFST), scores every referred child from 1 to 20 and helps decide who gets investigated. Families are never told a score exists, never told what it is, and have no way to see it, challenge it, or correct the data behind it. The proxies these tools rely on — poverty, disability, neighborhood, prior contact with public systems — are precisely the traits that historically mark poor, Black, and disabled parents, and independent researchers, the ACLU, and the U.S. Department of Justice's Civil Rights Division have all found the same pattern: the algorithms don't measure danger to children so much as they measure a family's exposure to government surveillance. This is the story of how an unaccountable "number that knocks" spread across the country — and why Washington just paid for more of it.
How the Machine Actually Works
The mechanics are deceptively clinical. When a mandated reporter — a teacher, doctor, or neighbor — calls Allegheny County's hotline, a call-screener pulls up the referral and clicks a button. The AFST reaches into the county's massive integrated Data Warehouse — Medicaid and behavioral-health records, drug and alcohol treatment history, county jail and juvenile-probation data, public-welfare enrollment, and prior child-welfare involvement — and, using a statistical model, returns a risk score from 1 (lowest) to 20 (highest). A high score can trigger a "mandatory screen-in," meaning the family must be investigated regardless of the screener's own judgment.
The crucial sleight of hand is in what the model actually predicts. The AFST was not trained on validated findings of abuse. It was trained to predict two proxy outcomes: whether a child would be re-referred to the hotline, and whether a child would be placed in foster care within two years. In other words, the algorithm learns from the past decisions of the child-welfare system itself — decisions already shaped by decades of racial and economic bias — and then reproduces them as "objective" prediction. If poor and Black families were investigated and separated more often in the training data (they were), the model concludes that poverty and Blackness are markers of risk. Bias in, bias out, laundered through math.
Two design choices compound the problem. First is over-collection: because the county holds far more data on people who rely on public services, a family that uses Medicaid, food assistance, mental-health counseling, or addiction treatment simply has more data points to be scored against — a phenomenon researchers call "poverty profiling." A wealthier family that sees a private therapist and pays cash generates no county footprint at all. Second is the permanent record: an ACLU and Human Rights Data Analysis Group analysis found the tool has no mechanism to amend or expire old data, so a past county contact keeps "forever flagging" a family, with risk accruing by mere association with relatives in the system. The score follows you, and there is no statute of limitations on your poverty.
The Money: Title IV-E, Federal Grants, and a New $6 Million Push
The financial engine behind family separation is Title IV-E of the Social Security Act, the open-ended federal entitlement that reimburses states for foster-care placements. Unlike prevention dollars, IV-E historically pays out per child removed and placed — an incentive structure that critics have long argued rewards separation over support. Predictive-risk tools sit directly upstream of that spigot: they help determine who gets investigated, and investigation is the funnel that leads to removal and IV-E reimbursement. Yet the federal Adoption and Foster Care Analysis and Reporting System (AFCARS) — the mandatory data system that tracks every child in foster care and drives IV-E accountability — contains no field capturing whether an algorithm influenced a removal. Washington pays for the outcomes these tools produce while collecting no data on the tools themselves.
The individual programs are cheap enough to fly under any procurement radar. Illinois's failed Rapid Safety Feedback system, built by the nonprofit Eckerd Connects and its for-profit partner MindShare Technology, was a roughly $366,000 contract — awarded, the Chicago Tribune reported, on a no-bid basis by then-DCFS director George Sheldon, who had brought associates from his prior tenure in Florida. Allegheny County's tool was developed by academic researchers under contract with the county Department of Human Services. These are line items, not megadeals — which is exactly why they escaped scrutiny for years.
The federal government is now actively subsidizing expansion. In 2026 the Administration for Children and Families (ACF) announced $6 million in competitive Predictive Analytics in Child Welfare Demonstration Grants (funding opportunity HHS-2026-ACF-ACYF-CA-0037), offering up to 10 awards of $400,000–$600,000 over three years to state, territorial, and tribal agencies to build and pilot exactly this kind of tooling. It followed a December 2024 ACF roundtable and a March 2025 issue brief, "Modernizing Child Welfare Technologies and Tools," that laid the intellectual groundwork. ACF frames the money around "responsible governance" and diverting low-risk cases away from the system — but the same score that diverts one family flags another, and the grant seeds national replication of a technology whose civil-rights record is, at best, unresolved. The National Coalition for Child Protection Reform bluntly called predictive analytics "the Project 2025 of child welfare."
The Named Players and the Incentive to Believe
The AFST's principal architects are Rhema Vaithianathan, a health economist who directs the Centre for Social Data Analytics at the Auckland University of Technology, and Emily Putnam-Hornstein, a child-welfare researcher now at the University of North Carolina at Chapel Hill (formerly USC). The pair pioneered predictive risk modeling in child welfare first in New Zealand and then in Allegheny County, and their team — including collaborators such as Tim Maloney and Nan Jiang — has become the intellectual center of gravity for the entire field. Their work is genuinely sophisticated and, to their credit, Allegheny County has published methodology documents and commissioned ethics reviews that most agencies never bother with. That transparency is why we know as much as we do — and it is also why Allegheny became the test case for everyone else.
The deeper conflict is structural rather than personal. Agencies adopting these tools are drowning: overworked screeners, thousands of referrals, and enormous political risk if a child they didn't investigate dies. A number that promises to sort the dangerous from the harmless is irresistible — it offers cover. Vendors and academic developers, meanwhile, have a reputational and funding stake in the tools "working." County leaders get to announce data-driven modernization. Everyone in the room benefits from believing the score is valid; the family being scored is not in the room. As Virginia Eubanks documented in her landmark 2018 book Automating Inequality, which used Allegheny as a central case study, these systems function as a "digital poorhouse" — high-tech tools that manage and moralize poverty while insulating decision-makers from accountability. The tool doesn't remove human bias; it concentrates it into an opaque number and stamps it "objective."
The Cases: Failures, Bias, and a Federal Investigation
The documented track record is damning.
Illinois (2017): DCFS terminated Rapid Safety Feedback after it proved worse than useless. The Chicago Tribune found the system had assigned 369 children a 100-percent probability of death or serious injury — a statistical absurdity that buried genuinely dangerous cases in false alarms — while high-profile child deaths kept occurring with no warning from the software. DCFS's own director conceded the technology "didn't seem to be predicting much."
California (Los Angeles): L.A. County's early foray into predictive risk modeling, a project sometimes referenced as "AURA," was abandoned after it generated overwhelming false positives, flagging far more families than could plausibly be at risk — the same over-identification failure that had already sunk pilots in Illinois.
Oregon (2022): Weeks after the Associated Press exposed the Pennsylvania tool's racial skew, Oregon's Department of Human Services announced it would stop using its own Safety at Screening Tool, which had been modeled on Allegheny's, and move to a process officials said would be more racially equitable. The White House Office of Science and Technology Policy publicly stressed the need for transparency in government algorithms.
Pennsylvania (the core case): In April 2022, AP reporters Sally Ho and Garance Burke published an investigation built on then-unpublished Carnegie Mellon University research. It found that in its first years, had it run on its recommendations alone without screener overrides, the AFST would have flagged a markedly higher share of Black children for "mandatory" investigation than white children — hardening the very disparities it was sold to reduce. The same research found caseworkers disagreed with the algorithm's risk scores about one-third of the time, a striking vote of no-confidence from the professionals meant to be aided by it.
That reporting triggered the most consequential development of all. In late 2022 and into 2023, the U.S. Department of Justice Civil Rights Division opened an inquiry into whether Allegheny County's use of the AFST discriminates against parents with disabilities in violation of the Americans with Disabilities Act. The concern, raised in civil-rights complaints and analyses associated with the ACLU and the Bazelon Center for Mental Health Law, is that the tool draws on county mental-health and disability-services data — so parents who sought help for conditions like ADHD, depression, or intellectual disability are scored as higher risk precisely because they used services designed to support them. Parents with disabilities already lose custody at elevated rates; an algorithm that treats a therapy appointment as a risk factor punishes families for asking for help.
The evidence is genuinely contested at the edges. Allegheny County disputes the AP's characterization and points to newer peer-reviewed work — including a 2025 analysis in the Journal of Policy Analysis and Management — arguing that in practice, with human screeners in the loop, the tool reduced racial gaps in screening and removal decisions. That debate matters and deserves honesty. But it also proves the central point: if the nation's most-studied, most-transparent tool remains this contested after nearly a decade, the dozens of copycats operating with no published methodology, no external audit, and no public data are flying blind.
The Accountability Gap: Everyone Is Supposed to Be Watching, No One Is
Here is the machinery of oversight that is supposed to protect families, and how each layer fails:
- The families themselves cannot serve as a check because they are never told a score exists. There is no notice, no disclosure, no right to see the score, and no process to contest it — a due-process vacuum that would be unthinkable in criminal sentencing, where even proprietary risk tools like COMPAS at least surface in open court.
- State legislatures have largely not passed laws requiring disclosure, independent validation, or bias auditing of these systems before deployment.
- Federal reporting is silent: AFCARS captures nothing about algorithmic influence on removals, so HHS cannot tell Congress how many of the roughly 200,000 children removed each year were flagged by a machine. You cannot regulate what you refuse to count.
- The ACLU's 2021 report, Family Surveillance by Algorithm, found that at least 26 states had considered predictive analytics in child welfare and roughly 11 were using such tools — but because there is no federal registry and no mandatory reporting, even that count is an estimate. The public does not have a reliable list of which agencies are scoring families right now.
- Procurement oversight is minimal, as Illinois's no-bid contract showed.
- The one real intervention — the DOJ Civil Rights inquiry — is narrow (disability, one county) and came only after investigative journalists forced the issue. Oversight by AP exposé is not a system.
The result is a technology that makes some of the most consequential decisions a government can make — whether to send an investigator to a family's door, whether to pull a child from a parent's arms — inside a black box that no affected person can open, no independent body must audit, and no federal dataset even acknowledges.
Why It Matters — and What Would Actually Fix It
The stakes are not abstract. A child-welfare investigation is itself a traumatic, sometimes coercive event: strangers examining a child's body, interrogating siblings, threatening removal. Removal is more traumatic still, and a large body of research finds that for children on the margin — cases that could go either way — staying with family often produces better outcomes than foster care. When an algorithm systematically nudges "screen in" for poor, Black, and disabled families, it is not neutrally allocating scarce investigative resources; it is redistributing state coercion toward the least powerful, dressed up as science. It punishes the act of seeking help — the Medicaid enrollment, the addiction treatment, the mental-health visit — and thereby teaches vulnerable families that engaging with public services is dangerous. That is a public-health catastrophe in slow motion.
A serious fix does not require banning analytics outright, but it does require ending the secrecy that makes abuse invisible:
- Mandatory disclosure. Any family subject to an algorithmic risk score must be told the score exists, what it is, and what data produced it — with a real, timely process to correct errors and contest the result, mirroring the due-process protections in credit reporting and criminal sentencing.
- Independent validation and bias auditing before and during deployment, conducted by parties with no financial or reputational stake in the tool, with results published.
- Federal reporting through AFCARS. HHS should require states to report when an automated tool influenced a screening or removal decision, so IV-E dollars can finally be tied to accountability. If Washington funds the outcome, it must count the cause.
- Ban predicting the system instead of the harm. Tools that train on foster-care placement and re-referral — the system's own past decisions — should be prohibited; a model can only be justified if it predicts validated child safety, not bureaucratic behavior.
- Strip poverty and disability proxies. Use of benefits enrollment, Medicaid utilization, and voluntary mental-health or disability-services data as risk inputs should be presumptively barred, because it profiles the act of surviving while poor.
- A public registry. Every jurisdiction using a predictive tool should be listed publicly, ending the situation where the ACLU has to guess how many states are scoring families.
The child-welfare system exists to protect children — not to optimize a caseload, not to generate federal reimbursements, and not to convert a family's poverty into a probability. Until the number that knocks on the door comes with a name, a reason, and a right to reply, these tools will keep doing what the evidence shows they do best: turning inequality into a self-fulfilling prophecy, one secret score at a time.
Sources
- AP report: DOJ examining AI screening tool used by Pa. child welfare agency — PBS NewsHour
- How an algorithm that screens for child neglect could harden racial disparities — PBS NewsHour
- An algorithm that screens for child neglect in Allegheny County raises concerns — 90.5 WESA
- Child welfare algorithm used by Allegheny County DHS faces Justice Department scrutiny — WESA
- Child welfare algorithm may unfairly target disabled parents, complaints to DOJ allege — Reason
- Oregon dropping AI tool used to help decide child abuse cases — PBS NewsHour
- Oregon Dropping AI Tool Used in Child Abuse Cases — Pulitzer Center
- Illinois Ends Child Abuse Prediction Program — Government Technology
- Illinois Drops Rapid Safety Feedback, A Predictive Analytics Tool — The Imprint
- Family Surveillance by Algorithm: The Rapidly Spreading Tools Few Have Heard Of — ACLU
- Family Surveillance by Algorithm (full report, PDF) — ACLU
- How Policy Hidden in an Algorithm is Threatening Families in This Pennsylvania County — ACLU
- The Devil is in the Details: Interrogating Values Embedded in the Allegheny Family Screening Tool — ACLU
- How Data Analysis Confirmed the Bias in a Family Screening Tool — Human Rights Data Analysis Group
- Allegheny Family Screening Tool — Centre for Social Data Analytics, AUT
- Algorithms, Humans, and Racial Disparities in Child Protection Systems: Evidence From the AFST — Journal of Policy Analysis and Management (Wiley)
- ACF Announces $6 Million for States to Pilot Predictive Analytics in Child Welfare — Administration for Children and Families
- Predictive Analytics in Child Welfare Demonstration Grants — Grants.gov opportunity listing
- Modernizing Child Welfare Technologies and Tools (issue brief, PDF) — ACF
- Predictive analytics: The Project 2025 of child welfare — NCCPR Blog
- Algorithmic Decision-Making in Child Welfare Cases and Its Legal and Ethical Challenges — American Bar Association
- Automated Decision Systems: Child Welfare Predictive Analytics Tools — ACLU of Washington