California’s AI in Healthcare Advisory: Legal Risks and Compliance Essentials (Part 2)

Following its general AI advisory, the California Attorney General’s Office released a healthcare-specific AI legal advisory on February 6, 2025. Aimed at providers, insurers, and developers, the guidance focuses on how California laws apply to AI systems in clinical settings. From algorithmic bias and privacy laws to informed consent and medical licensing rules, this summary breaks down the key compliance risks—and what legal professionals need to watch for as AI expands into healthcare.

This post is a follow-up to our January 29, 2025 summary of California’s general AI legal advisory.

Overview

The advisory provides guidance to healthcare entities, including providers, insurers, and developers, on ensuring compliance with California laws while using artificial intelligence (AI). It emphasizes AI’s potential to improve healthcare and its risks of bias, discrimination, and privacy violations.

Key Legal Issues and Obligations

1. Transparency and Accountability

• Healthcare entities must disclose AI usage to patients, especially when AI influences decisions about diagnosis, treatment, or resource allocation.

• Developers and users are responsible for ensuring AI systems are tested, validated, and audited for safety, ethics, and legality.

2. Consumer Protection Laws

• The Unfair Competition Law (UCL) applies to deceptive or unfair AI practices in healthcare, such as fraudulent billing or misrepresentation of AI capabilities.

• Using AI to override physicians’ treatment decisions or to automate cost-benefit analyses based on biased assumptions violates California law.

3. Anti-Discrimination Laws

• California prohibits AI-based discrimination in healthcare under:

• Government Code § 11135, which bans disparate impacts on protected classes.

• The Unruh Civil Rights Act, which ensures equal access to healthcare.

• AI systems must not amplify existing biases or deny care to marginalized groups. Any disparate impact requires a compelling, evidence-based justification.
  

4. Patient Privacy and Autonomy

• Confidentiality of Medical Information Act (CMIA) and Information Practices Actimpose strict standards on the use and sharing of medical data, especially sensitive information (e.g., mental health, reproductive health).

• AI systems must ensure informed patient consent and avoid “dark patterns” that manipulate decisions.

• Genetic data is protected under the Genetic Information Privacy Act, prohibiting unauthorized disclosures.

5. Healthcare-Specific AI Governance

• Recent amendments to the Knox-Keene Act and Insurance Code mandate that AI tools:

• Do not replace licensed provider judgment.

• Base decisions on individual patient needs.

• Remain subject to state audits and ensure equity.

• Corporations may not practice medicine via AI, ensuring medical decisions are led by licensed providers.

Emerging Compliance Areas

1. FDA Regulation

• Healthcare AI systems are regulated as medical devices, requiring compliance with FDA standards for safety and efficacy.

2. Non-Discrimination Mandates

• Developers must address algorithmic biases that disproportionately affect marginalized groups.

• Programs receiving state support must take corrective measures to mitigate discrimination and health disparities.

3. Informed Consent

• Patients must be fully informed of AI involvement in treatment decisions.

• Participation in AI-driven medical experiments triggers California’s “Experimental Subject’s Bill of Rights.”

Broader Applications

California law treats AI-based actions as equivalent to human actions in liability. Healthcare entities must proactively ensure AI tools comply with consumer protection, civil rights, competition, and privacy laws.

This advisory underscores the need for vigilance in deploying AI in healthcare to protect patients’ rights and foster equitable access to services.