CA is using AI tools for unemployment, prisons and more - CalMatters

About<br>Newsletters

Search

CalMatters is your nonprofit and nonpartisan newsroom dedicated to explaining how state government impacts our lives.

Politics

Immigration

Housing

Education

Economy

Environment

California Voices

Impact

Events

Every year, California agencies report to the state technology department on their use of "high-risk" decision systems. The seal of the State of California on the Edmund G. Brown Jr. State Office Building in San Francisco on Jan. 28, 2022. Photo by Martin do Nascimento for CalMatters

In summary

State officials have found they are using six high-risk AI-like systems. One year ago, they reported using zero.

Keeping up with your California representatives felt impossible before. Now you can watch them like a hawk: Sign up for My Legislator, your weekly, personalized report that tracks what your state representatives say and do.

A year ago, California officials had to report under a new state law how they used automated systems to make important decisions about people’s lives.

They said they never did — a startling answer for a number of reasons, sources told CalMatters at the time, including that there were several prominent examples to the contrary.

Now, the state has issued a more expansive answer: It is currently using six automated systems to make consequential decisions about the lives of Californians.

The systems are used to do things like:

Predict whether incarcerated people will re-offend

Evaluate whether unemployment claims are fraudulent

Remotely administer exams for California State University students

Detect when college students use generative AI to write assignments.

That’s according to a report released Friday by the state’s technology department. The report is required under a 2023 law mandating that that state agencies annually disclose their use of “high-risk automated decision systems,” which the law defines as systems “used to assist or replace human discretionary decisions that have a legal or similarly significant effect, including decisions that materially impact access to, or approval for, housing or accommodations, education, employment, credit, health care, and criminal justice.”

The law was pushed by civil rights, privacy, and civil liberties groups concerned about harms from AI-like systems. Numerous such systems have been shown to produce results biased against marginalized groups, including those used for high-stakes testing, predicting recidivism, and detecting AI-generated texts.

CalMatters flagged last year’s report as surprising, noting that the state corrections department had reported using software to predict post-release behavior and that the employment department used a fraud detection system that paused benefits for 600,000 Californians between Christmas and New Years in 2020, according to a Legislative Analyst’s Office report.

Though the report names six high-risk systems in use today, state agencies have used some for several years now. Those include COMPAS, which has been used by the California Department of Corrections and Rehabilitation to assign recidivism scores to inmates for at least a decade.

The technology department said in the report that it found more systems for its report this year because it evaluated responses from state agencies more thoroughly, including by meeting with agencies and questioning them about their systems.

In addition to the six high-risk systems, the department’s report disclosed an additional six systems initially flagged as high risk but later determined not to be. One was AI used for legislative bill analysis by the California Department of Finance.

The report also notes two high-risk systems that are not currently in use: the Department of Cannabis Control is developing artificial intelligence to analyze whether marijuana packaging violates a law against appealing to children and California State University discontinued use of a language model for reviewing job applications.

Just the right amount of news

Just the right amount of news

Get California’s most essential headlines without feeling overwhelmed.

Get California’s most essential headlines without feeling overwhelmed.

Email address

By clicking subscribe, you agree to the terms.

Results of the second annual survey come after cities like San Jose and San Francisco released their first AI inventories in recent months. They also come at a time when California-based AI companies like Anthropic and OpenAI are going public and seeking government contracts. Americans are split on whether they trust AI and surveys last year by TechEquity and Carnegie California found that the majority of Californians want safety over innovation. A Gallup poll to evaluate the opinions of Americans found similar results.

Senate Bill 1248, a bill that would have prohibited state employees from using automated decision systems as the sole basis for decisionmaking, was...