11.4 Measuring and Reporting Ethical AI Performance

Measuring and reporting ethical AI performance is crucial for organizations to ensure their AI systems align with ethical principles. By establishing clear metrics and reporting practices, companies can track progress, identify risks, and demonstrate accountability to stakeholders, fostering trust and transparency.

Key metrics for assessing AI ethics include quantitative measures like fairness and robustness tests, as well as qualitative indicators such as expert audits and user feedback. Organizations should adapt reporting frameworks, engage stakeholders, and ensure accessibility to create comprehensive, transparent ethical AI reports.

Measuring Ethical AI Performance

Establishing Clear Metrics and Reporting Practices

• Establishing clear metrics and reporting practices around AI ethics is critical for ensuring AI systems are operating in alignment with an organization's stated values and ethical principles
• Measuring and reporting on AI ethics performance enables organizations to identify areas of risk, track progress over time, and demonstrate accountability to stakeholders
• Ethical AI reporting supports transparency by providing insight into how AI systems are developed, deployed and monitored to mitigate potential harms
• Instituting robust measurement and reporting practices can foster trust with users, regulators and society that an organization is proactively governing the ethical implications of its AI

Benefits of Measuring and Reporting AI Ethics Performance

• Allows organizations to benchmark their AI ethics performance against industry peers and best practices
• Provides a mechanism for organizations to be held accountable by external stakeholders (regulators, advocacy groups) for upholding ethical principles
• Enables early detection and mitigation of ethical risks before they result in harm to individuals or society
• Demonstrates an organization's commitment to responsible AI development, enhancing brand reputation and user trust

Key Metrics for AI Ethics

Quantitative Metrics for Assessing Ethics Compliance

• Organizations should define concrete and measurable key performance indicators (KPIs) tied to each of its AI ethics principles to consistently assess adherence
• Ethical AI KPIs may include quantitative measures such as:
  • Fairness metrics evaluating biases in datasets/model outputs (demographic parity, equalized odds)
  • Robustness tests measuring model performance under different conditions (distributional shift, adversarial attacks)
  • Documentation of human oversight (percentage of AI decisions reviewed by humans)
  • Percentage of datasets and models with accompanying datasheets or model cards disclosing key characteristics
• The specific metrics chosen must be tailored to the organization's industry, use cases, and risk profile to meaningfully capture ethical performance

Qualitative Indicators of Ethics Compliance

• Qualitative indicators of ethics compliance can include:
  • Results of expert audits/assessments evaluating adherence to AI ethics principles
  • User feedback surveys on perceived trustworthiness and fairness of AI systems
  • Transparent disclosures of AI system capabilities, limitations, and potential risks
  • Documentation of stakeholder consultation processes to surface ethical concerns
• Metrics should evolve as AI ethics frameworks mature and aim to align with emerging industry standards (IEEE, ISO) to enable benchmarking

Framework for Reporting AI Ethics

Structuring Ethics Reporting

• AI ethics reporting should cover the end-to-end lifecycle from data sourcing and model development to deployment and monitoring
• Organizations can adapt sustainability reporting frameworks like the Global Reporting Initiative (GRI) to structure ethics disclosures and KPIs
• Ethics reports should detail the governance structures, policies, and due diligence processes in place to oversee AI ethics
  • Governance structures can include cross-functional ethics committees or advisory boards
  • Policies may cover ethical risk assessment requirements, fairness testing, or human oversight
  • Due diligence processes can include vendor assessments or user impact evaluations
• Results of third-party audits and certifications against AI ethics standards should be disclosed to validate internal performance assessments

Ensuring Accessibility and Transparency

• Reporting cadence should align with the organization's overall reporting calendar (annually) but also allow for incident-based disclosures for major ethical breaches or harms
• Ethics reports should be publicly accessible to both technical and non-technical stakeholders
  • Reports should minimize jargon and provide appropriate contextual information on AI systems for a general audience
  • Visualizations, case studies and FAQs can make ethics disclosures more engaging and understandable
• Transparency breeds accountability - greater visibility into AI ethics performance incentivizes organizations to proactively mitigate risks and strive for continuous improvement

Stakeholder Engagement for Ethical AI

Proactive Outreach to Understand Stakeholder Expectations

• Proactive stakeholder engagement is essential for understanding expectations for ethical AI development and identifying potential blindspots in assessing performance
• Key stakeholder groups to engage span:
  • Internal stakeholders (employees, leadership, board of directors)
  • External stakeholders (users, policymakers, advocacy groups, academia, general public)
• Materiality assessments help prioritize the AI ethics issues of highest concern to stakeholders to inform KPIs and reporting
  • Surveys, interviews and workshops can solicit stakeholder input on material ethical risks
  • Horizon scanning of regulatory developments, media and academic research reveals emerging issues

Channels for Stakeholder Feedback and Accountability

• Organizations should establish clear channels for affected stakeholders to submit complaints or feedback on AI system performance to surface ethical issues
  • Feedback mechanisms can include hotlines, online portals, or ombudsperson roles
  • Complaints should trigger investigations and be integrated into ethics monitoring dashboards
• Ongoing dialogue with stakeholders on AI ethics reporting can uncover gaps, build consensus on metrics, and co-create accountability mechanisms
  • Advisory councils with stakeholder representatives provide a forum for collaboration
  • Participatory design workshops allow affected communities to inform ethical KPIs
• Stakeholder engagement enables a continuous improvement process to raise the bar on ethical AI performance over time based on evolving societal expectations