Overview

With the increasing deployment of machine learning in diverse applications affecting our daily lives, ethical and legal implications are rising to the forefront. Governments worldwide have responded by implementing regulatory policies to safeguard algorithmic decisions and data usage practices. However, there appears to be a considerable gap between current machine learning research and these regulatory policies. Translating these policies into algorithmic implementations is highly non-trivial, and there may be inherent tensions between different regulatory principles.

This workshop aims to provide a platform for: i) discussing various algorithmic, technical, and policy challenges that arise when operationalizing various guidelines outlined in existing regulatory frameworks, and ii) finding solutions to mitigate and address these challenges.

Please check out our Call for Papers.

Keynote Talks

Designing ML Systems that can be Regulated: Challenges and Opportunities

Abstract

Abstract: We’ve seen a lot of talks, frameworks, and guidelines over the past year on regulating ML. In this talk, I’ll discuss why the current norms and practices in the ML research and practice community don’t lend themselves well to being regulated and what types of changes need to take place, from scoping and formulation, to design and development, to evaluation and monitoring, in order to result in ML systems that could be reliably deployed to help achieve societal, business, and policy outcomes that we want.

Rayid Ghani

Distinguished Career Professor

Carnegie Mellon University

Bio

Bio: Rayid is a reformed computer scientist and wanna-be social scientist, but mostly just wants to increase the use of large-scale AI/Machine Learning/Data Science in solving large public policy and social challenges in a fair and equitable manner. Among other areas, Rayid works with governments and non-profits in policy areas such as health, criminal justice, education, public safety, economic development, and urban infrastructure. Rayid is also passionate about teaching practical data science and started the Data Science for Social Good Fellowship that trains computer scientists, statisticians, and social scientists from around the world to work on data science problems with social impact. Before joining Carnegie Mellon University, Rayid was the Founding Director of the Center for Data Science & Public Policy, Research Associate Professor in Computer Science, and a Senior Fellow at the Harris School of Public Policy at the University of Chicago. Previously, Rayid was the Chief Scientist of the Obama 2012 Election Campaign where he focused on data, analytics, and technology to target and influence voters, donors, and volunteers. In his ample free time, Rayid obsesses over everything related to coffee and works with non-profits to help them with their data, analytics and digital efforts and strategy.

Regulating Code: What the European Union has in stock for the governance of Artificial Intelligence, foundation models, and generative AI

Abstract

Abstract: It is often said that while the US innovates Europe regulates. But is this actually true? AI, machine learning and generative AI pose new risks and opportunities to our society and many hope or fear that Europe will have a heavy regulatory hand. However, this talk will demystify this assumption and show how EU-laws often under- AND overregulate at the same time. I will explore the actual obligations, shed light on their loopholes, expose shortfalls and explain how to fix them.

Sandra Wachter

Professor

University of Oxford

Bio

Bio: Professor Sandra Wachter is Professor of Technology and Regulation at the Oxford Internet Institute at the University of Oxford where she researches the legal and ethical implications of AI, Big Data, and robotics as well as Internet and platform regulation. Her current research focuses on profiling, inferential analytics, explainable AI, algorithmic bias, diversity, and fairness, as well as governmental surveillance, predictive policing, human rights online, and health tech and medical law. At the OII, Professor Sandra Wachter leads and coordinates the Governance of Emerging Technologies (GET) Research Programme that investigates legal, ethical, and technical aspects of AI, machine learning, and other emerging technologies. Professor Wachter is also an affiliate and member at numerous institutions, such as the Berkman Klein Center for Internet & Society at Harvard University, World Economic Forum’s Global Futures Council on Values, Ethics and Innovation, the UNESCO, the European Commission’s Expert Group on Autonomous Cars, the Law Committee of the IEEE, the World Bank’s Task Force on Access to Justice and Technology, the United Kingdom Police Ethics Guidance Group, the British Standards Institution, the Law Faculty at Oxford, the Bonavero Institute of Human Rights, the Oxford Martin School and Oxford University Press. Professor Wachter also serves as a policy advisor for governments, companies, and NGO’s around the world on regulatory and ethical questions concerning emerging technologies.

Fine-Tuning Games: Modeling the Ecosystem of Machine Learning Applications and their Development

Abstract

Abstract: Major advances in machine learning (ML) and artificial intelligence (AI) increasingly take the form of developing and releasing general-purpose models. These models are designed to be adapted by other businesses and agencies to perform a particular, domain-specific function. This process has become known as adaptation or fine-tuning. In order to understand questions about responsibility and regulation in an ecosystem where multiple parties produce applications in this way, we need reasonable models of the incentives that drive this type of production. Here we offer a model of this multi-party process, in which a general provider of machine learning technology brings the system to a certain level of performance, and one or more Domain-specialists adapt it for use in particular domains. Our model provides high-level takeaways for how incentives operate in this setting, and in this way it suggests how we might think about responsible development and regulation of these technologies. This is joint work with Ben Laufer and Hoda Heidari.

Jon Kleinberg

Tisch University Professor

Cornell University

Bio

Bio: Jon Kleinberg is the Tisch University Professor in the Computer Science Department at Cornell University. His research focuses on issues at the interface of networks and information, with an emphasis on the social and information networks that underpin the Web and other on-line media. He is a member of the National Academy of Sciences, the National Academy of Engineering, and the American Academy of Arts and Sciences, and serves on the Computer and Information Science and Engineering (CISE) Advisory Committee of the National Science Foundation, and the Computer Science and Telecommunications Board (CSTB) of the National Research Council. He is the recipient of MacArthur, Packard, and Sloan Foundation Fellowships, as well as awards including the Nevanlinna Prize, the Lanchester Prize, and the ACM-Infosys Foundation Award in the Computing Sciences.

Path to trustworthy and responsible AI

Abstract

Abstract: With AI already changing the way in which society addresses economic and national security challenges and opportunities, AI technologies must be developed and used in a trustworthy and responsible manner. Experts in industry, academia, and government are still assessing how to best measure and manage risks and impacts of AI systems. At NIST we believe that working together to develop and advance the science of AI safety will harness the power of AI to serve humanity. We have been and will continue developing tests and facilitating the development of standards – measurement science and standards that will allow industry, academia, and government to better map, measure and manage AI risks. This work is a necessary precursor to any compliance or conformity assessment, either voluntary or required. Delivering these needed measurements, standards, and other tools is a primary focus for NIST’s portfolio of AI efforts. This talk provides an overview of NIST Trustworthy and Responsible AI program.

Elham Tabassi

Associate Director for Emerging Technologies

National Institute of Standards and Technology (NIST)

Bio

Bio: Elham Tabassi is a Senior Research Scientist at the National Institute of Standards and Technology (NIST) and the Associate Director for Emerging Technologies in the Information Technology Laboratory (ITL). She also leads NIST’s Trustworthy and Responsible AI program that aims to cultivate trust in the design, development, and use of AI technologies. As the ITL’s Associate Director for Emerging Technologies, Elham assists NIST leadership and management at all levels in determining future strategic direction for research, development, standards, testing and evaluation in the areas of emerging technologies such as artificial intelligence. She also coordinates interaction related to artificial intelligence with the U.S. research community, U.S. industrial community, international standards community, and other federal agencies; and provides leadership within NIST in the use of AI to solve scientific and engineering problems arising in measurement science and related use-inspired applications of AI. Elham has been working on various machine learning and computer vision research projects with applications in biometrics evaluation and standards since she joined NIST in 1999. She is a member of the National AI Resource Research Task Force, vice-chair of OECD working party on AI Governance, Associate Editor of IEEE Transaction on Information Forensics and Security, and a fellow of Washington Academy of Sciences.

Regulating ML: Rights, Transparency, and Agency

Abstract

Abstract: As ML is becoming ubiquitous and creating new and impressive artifacts, regulatory agencies around the world are grappling with the unique properties of this new category of technology. In order to properly address these challenges, we argue for an approach centered on the dual pillars of rights and transparency to ensure that the technology is subject to the appropriate democratic governance. We outline some of the recent developments and proposals made by policymakers in this direction, how they connect to AI, and provide both organizational and technical tools to support well-informed regulation aligned with technology development now and in the future.

Margaret Mitchell

Researcher and Chief Ethics Scientist

Hugging Face

Bio

Bio: Margaret Mitchell is a researcher focused on the ins and outs of machine learning and ethics-informed AI development in tech. She has published around 100 papers on natural language generation, assistive technology, computer vision, and AI ethics, and holds multiple patents in the areas of conversation generation and sentiment classification. She has recently received recognition as one of Time's Most Influential People of 2023. She currently works at Hugging Face as Chief Ethics Scientist, driving forward work in the ML development ecosystem, ML data governance, AI evaluation, and AI ethics. She previously worked at Google AI as a Staff Research Scientist, where she founded and co-led Google's Ethical AI group, focused on foundational AI ethics research and operationalizing AI ethics Google-internally. Before joining Google, she was a researcher at Microsoft Research, focused on computer vision-to-language generation; and was a postdoc at Johns Hopkins, focused on Bayesian modeling and information extraction. She holds a PhD in Computer Science from the University of Aberdeen and a Master's in computational linguistics from the University of Washington. While earning her degrees, she also worked from 2005-2012 on machine learning, neurological disorders, and assistive technology at Oregon Health and Science University. She has spearheaded a number of workshops and initiatives at the intersections of diversity, inclusion, computer science, and ethics. Her work has received awards from Secretary of Defense Ash Carter and the American Foundation for the Blind, and has been implemented by multiple technology companies. She likes gardening, dogs, and cats.

Yacine Jernite

ML and Society Lead

Hugging Face

Bio

Bio: Dr. Yacine Jernite leads the ML and Society team at Hugging Face, where they work on ML systems governance. Their work to date has focused on NLP and multimodal models and data curation, documentation, and governance. Their recent projects have included co-organizing the BigScience workshop on large language models as data area chair and research at the intersection of ethical, technical, and regulatory aspects of ML.

Connecting provable guarantees and regulation of LLMs.

Abstract

Abstract: The complexity and black-box nature of LLMs makes it difficult to provide meaningful guarantees, which in turn complicates efforts to regulate and audit LLMs. In this talk, I will discuss how statistical guarantees on various properties of LLMs such as privacy (via differential privacy) and provenance (via watermarking or membership inference) provide powerful primitives for thinking about important regulatory issues such as copyright. At the same time, implementing and deploying these primitives can be challenging and I will discuss pitfalls and open problems in the interaction of various statistical guarantees and LLM deployment environments.

Tatsunori Hashimoto

Assistant Professor

Stanford University

Bio

Bio: Tatsunori Hashimoto is an Assistant Professor in the Computer Science Department at Stanford University. He is a member of the statistical machine learning and natural language processing groups at Stanford, and his research uses tools from statistics to make machine learning systems more robust and trustworthy — especially in complex systems such as large language models. He is a Kavli fellow, a Sony and Amazon research award winner, and his work has been recognized with best paper awards at ICML and CHI.

Panel Discussion

Rayid Ghani

Carnegie Mellon
University  

Jon Kleinberg

Cornell University
   

Margaret Mitchell

Hugging Face
   

Deborah Raji

Mozilla Foundation
UC Berkeley  

Elham Tabassi

NIST
   

Schedule

Dec 16, 2023; New Orleans Convention Center (Room 215 - 216)

Awards

Outstanding Papers

• SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore (Sewon Min, Suchin Gururangan, Eric Wallace, Weijia Shi, Hannaneh Hajishirzi, Noah Smith, Luke Zettlemoyer)
• Towards a Post-Market Monitoring Framework for Machine Learning-Based Medical Devices: A Case Study (Jean Feng, Adarsh Subbaswamy, Alexej Gossmann, Harvineet Singh, Berkman Sahiner, Mi-Ok Kim, Gene Pennello, Nicholas Petrick, Romain Pirracchio, Fan Xia)
• Merging (EU)-Regulation and Model Reporting (Danilo Brajovic, Vincent Philipp Göbels, Janika Kutz, Marco Huber)

Outstanding Reviewers

Will Fleisher (Georgetown University), Philippe Giabbanelli (Miami University)

Organizers

If you have any questions, please contact us via the following email: regulatableml@googlegroups.com.

Core Organizing Team

Jiaqi Ma

University of Illinois
Urbana-Champaign

Chirag Agarwal

Harvard University
 

Sarah Tan

Cambia Health
Cornell University

Hima Lakkaraju

Harvard University
 

Student Organizers

Usha Bhalla

Harvard University
 

Zana Bucinca

Harvard University
 

Chelsea Chen

Harvard University
 

Junwei Deng

University of Illinois
Urbana-Champaign

Xudong Shen

National University of
Singapore

Varshini Subhash

Harvard University
Tonita, Inc

Advisory Team

Danielle Belgrave

DeepMind
 

Kelly Cochran

FinRegLab
 

Finale Doshi-Velez

Harvard University
 

Daniele Magazzeni

J.P. Morgan
 

Weiwei Pan

Harvard University
 

P-R Stark

FinRegLab
 

Reviewers