AI Openness & Model Sharing

---

Executive Summary

Open-source software (OSS) development is a culture and set way of working that involves the free and open sharing of software projects for further study, modification, and use. For thirty years OSS has proliferated alongside (and often inside) private software, encouraging cooperation, fostering innovation and competition, nurturing talent, and improving software quality through community review. However, in the past couple of years, decades-old OSS tradition has been challenged by the development of increasingly capable AI. Concerns about the potential harms of malicious misuse has yielded heated debate about the prudence of open-sourcing AI with many arguing that some models pose too high of a risk to be made available for public download. The open-source AI debate has been a high-stakes dialogue, measuring up harms against benefits and risks against worries. 

Yet despite what, at its start, often felt like a stark open-v.-closed, us-v.-them standoff, the open-source AI debate has evolved productively into collaborative conversations. This has included noted progress toward establishing a clear definition for open-source AI, broader consideration of a spectrum of model sharing strategies, and exploration of numerous dimensions of AI openness expanding beyond core considerations of model access.  

Building on insights from the ‘’Open Horizons’’ workshop hosted by Demos and Mozilla on June 6th 2024 – a follow-up to Demos’s October 2024 open-source policy workshop and report – this report reflects on areas of emerging consensus, persisting disagreement, and unanswered questions to start thinking about next steps for safely pursuing openness benefits. This report does not pit risks against benefits. Instead we look into what kinds of evidence should be collected in deciding when to release an AI model, and how we might productively work to reduce any risks of openness where they do exist. 

This report yields the following key insights:

• Risk mitigation strategies and guardrails need to be implemented throughout the AI value chain and through improvements to systemic AI safety, not just at the point of model release.
• Urgent work is needed to develop clear threat model and model evaluation standards.
• We can seek alternative methods for pursuing AI openness goals instead of, or alongside, model sharing for when model access restrictions are in place.
• There are promising technical solutions to explore which can help mitigate some of the risks of openness.
• The impacts of restricting access to highly capable AI models on competition and market concentration need to be researched and clarified. 
• Unclear liability rules and safety standards stifle open innovation and can yield less safe technologies. 

Based on these key insights we develop a non-exhaustive menu of policy options for governments. These options outline forward-looking areas of (1) investment and (2) potential regulatory interventions, by which the government might promote the benefits of openness that facilitate innovation, enable digital autonomy, fuel competition, and attract talent while mitigating potential harms. We explore the strengths and weaknesses of these options in the main text. 

(1) Investment options:

• Provide financial support for open-source projects and ecosystems. This includes providing support for open-source safety testing ecosystems for smaller developers who may not have access to tools, standards, or necessary compute resource.
• Invest in digital public infrastructure such as open national AI models, public compute infrastructure, and open data libraries.
• Invest in clear threat modelling exercises involving domain experts to underpin targeted risk mitigation policy. 
• Investigate economic impacts of open-sourcing AI models and of restricting model access.
• Investigate guardrails that can be deployed throughout the AI value chain to reduce risks of AI openness. 
• Invest in incentive structures such as large rewards programs to promote AI safety and social benefit breakthroughs. 

(2) Regulation options:

• Set public sector procurement standards for transparency to incentivise greater AI openness.
• Introduce exemptions from AI regulation for models that meet a certain standard of openness/transparency in order to incentivise greater transparency and information sharing.
• Clarify liability legislation and establish openness as a condition for transferring liability downstream.
• Define part of AI Safety Institute (AISI)’s role as providing safety evaluation support for startup and open-source ecosystems – e.g. by providing resources, tools, and safety evaluation standards. 
• Establish / reform government open data policy.
• Incorporate democratic processes into government decision-making around AI.

---

Executive Summary

Recent decisions by AI developers to open-source foundation models have sparked debate over the prudence of open-sourcing increasingly capable AI systems. Open-sourcing in AI typically involves making model architecture and weights freely and publicly accessible for anyone to modify, study, build on, and use. On the one hand, this offers clear advantages including enabling external oversight, accelerating progress, and decentralizing AI control. On the other hand, it presents notable risks, such as allowing malicious actors to use AI models for harmful purposes without oversight and to disable model safeguards designed to prevent misuse. 

This report attempts to clarify open-source terminology and to offer a thorough analysis of risks and benefits from open-sourcing AI. While open-sourcing has, to date, provided substantial net benefits for most software and AI development processes, we argue that for some highly capable models likely to emerge in the near future, the risks of open sourcing may outweigh the benefits.

There are three main factors underpinning this concern: 

1. Highly capable models have the potential for extreme risks.  Of primary concern is diffusion of dangerous AI capabilities that could pose extreme risks—risk of significant physical harm or disruption to key societal functions. Malicious actors might apply highly capable systems, for instance, to help build new biological and chemical weapons, or to mount cyberattacks against critical infrastructures and institutions. We also consider other risks such as models helping malicious actors disseminate targeted misinformation at scale or to enact coercive population surveillance. Arguably, current AI capabilities do not yet surpass a critical threshold of capability for the most extreme risks. However, we are already seeing nascent dangerous capabilities emerge, and this trend is likely to continue as models become increasingly capable and as it becomes easier and requires less expertise and compute resources for users to deploy and fine-tune these models. (Section 3)

2. Open-sourcing is helpful in addressing some risks, but could – overall – exacerbate the extreme risks that highly capable AI models may pose. While for traditional software, open-sourcing facilitates defensive activities to guard against misuse more so than it facilitates offensive misuse by malicious actors, the offense-defense balance is likely to skew more towards offense for increasingly capable foundation models. This is for a variety of reasons including: (i) Open-sourcing allows malicious actors to disable safeguards against misuse and to possibly introduce new dangerous capabilities via fine-tuning. (ii) Open-sourcing greatly increases attacker knowledge of possible exploits beyond what they would have been able to easily discover otherwise. (iii) Researching safety vulnerabilities is comparatively time consuming and resource intensive, and fixes are often neither straightforward nor easily implemented. (iv) It is more difficult to ensure improvements are implemented downstream, and flaws and safety issues are likely to perpetuate further due to the general use nature of the foundation models. (Section 3)
3. There are alternative, less risky methods for pursuing open-source goals.  There are a variety of strategies that might be employed to work towards the same goals as open-sourcing for highly capable foundation models but with less risk, albeit with their own shortcomings. These alternative methods include more structured model access options catered to specific research, auditing, and downstream development needs, as well as proactive efforts to organize secure collaborations, and to encourage and enable wider involvement in AI development, evaluation, and governance processes. (Section 4)

In light of these potential risks, limitations, and alternatives, we offer the following recommendations for developers, standards setting bodies, and governments. These recommendations are to help establish safe and responsible model sharing practices and to preserve open-source benefits where safe. They also summarize the paper’s main takeaways. (Section 5)

Recommendations:

1. Developers and governments should recognize that some highly capable models will be too risky to open-source, at least initially. These models may become safe to open-source in the future as societal resilience to AI risk increases and improved safety mechanisms are developed. 
2. Decisions about open-sourcing highly capable foundation models should be informed by rigorous risk assessments. In addition to evaluating models for dangerous capabilities and immediate misuse applications, risk assessments must consider how a model might be fine-tuned or otherwise amended to facilitate misuse. 
3. Developers should consider alternatives to open-source release that capture some of the same distributive, democratic, and societal benefits, without creating as much risk. Some promising alternatives include gradual or “staged” model release, structured model access for researchers and auditors, and democratic oversight of AI development and governance decisions.
4. Developers, standards setting bodies, and open-source communities should engage in collaborative and multi-stakeholder efforts to define fine-grained standards for when model components should be released. These standards should be based on an understanding of the risks posed by releasing different combinations of model components.
5. Governments should exercise oversight of open-source AI models and enforce safety measures when stakes are sufficiently high. AI developers may not voluntarily adopt risk assessment and model sharing standards. Governments will need to enforce such measures through options such as liability law and regulation, for example via licensing requirements, fines, or penalties. They will also need to build the capacity to enforce such oversight mechanisms effectively.  Immediate work is needed to evaluate the costs, consequences, and legal feasibility of various policy interventions and enforcement mechanisms we list.