Security Requirements for AI Companies

The security requirements listed under the AI code of practice are only relevant to general-purpose AI models with systemic risks and not AI systems. However, the code specifies that companies should reasonably consider the overall system design, other software the model might connect to, and the computing power used when the model runs when addressing systemic risks. Companies can identify systemic risks by gathering information through a range of methods, such as market analyses, reviews of training data, reviewing and analysing historical incident data and incident databases, and forecasting general trends.

Why it matters:

This AI code of practice sets out how the EU is looking to implement its AI Act, which is the first comprehensive AI law across the world. Its implementation sets a precedent for regulatory approaches in other jurisdictions. The code can encourage stronger security and risk management requirements from AI companies and shed light on how regulators will evaluate compliance with these risk management requirements.

Key security framework details under the AI code of practice:

What should the framework include?

The safety and security framework must consist of a high-level description of the implemented/planned processes for systemic risk assessment and mitigation. Companies must assess whether this risk is acceptable; those that sign up to the code commit to deciding whether to proceed with the development and making their model available on the market based on the systemic risk acceptance determination. 

The framework must include a description of the systemic risk tiers (thresholds for when a model starts facing a specific risk). It must also include a description of whether inputs from external actors (like governments) influence the company’s decisions about continued model development, releasing a model into the market, and allowing people to use the model.

Framework implementation:

Throughout the entire lifecycle of a model, AI companies must continuously assess the risks stemming from them and conduct lighter-touch evaluations (such as automated evaluations) at appropriate trigger points. Companies can define these trigger points for evaluation in terms of time, training, compute, development stages, user access, etc. 

The AI code of practice requires companies to carry out a full systemic-risk assessment before placing the model in the market. This assessment need not replicate the steps in previous assessments. Companies must conduct post-market monitoring even after placing their AI model into the market. This includes collecting end-user feedback, providing anonymous reporting channels, and providing bug bounties. 

Rules for external evaluation for post-market monitoring:

To facilitate post-market monitoring, the companies must provide an adequate number of external evaluators with:

• free access to the model’s most capable version concerning systemic risk
• the chain of thought of the model (how the model arrives at specific conclusions)
• access to the model’s version with fewer safety restrictions (like a “helpful-only” version that hasn’t been trained to refuse certain requests).

Companies can provide this access either through API, on premises, through company hardware, or by making their model publicly available for download.

AI companies must publicly announce what qualifications they’re looking for in evaluators. The number of evaluators, selection criteria, and security measures can vary depending on whether evaluators are accessing the model’s most capable version, chain of thought reasoning, and less safety-filtered versions. The AI code of practice restricts AI companies from training the model based on the results of external evaluations. Further, it says that the companies cannot take legal action against the evaluators for their testing or publishing findings, provided that the evaluator does not intentionally 

Advertisements

• disrupt the model
• violate user privacy
• use their findings for activities that pose a significant risk to public safety and security
• use findings to threaten anyone 
• use the company’s published process for reporting vulnerabilities. 

Companies cannot delay disclosure of the evaluators’ findings for more than 30 days unless there are exceptional circumstances where disclosure would increase systemic risk.

Framework assessment:

Besides assessing risks, the AI code of practice also expects companies to assess the security framework if they have reason to believe that its adequacy has been or will be unduly undermined. Companies have to assess their security frameworks after 12 months of placing their product on the market, or whichever is sooner. Grounds for framework assessment include material changes in the model development process, which can lead to systemic risks, serious incidents and/or near misses with the company’s models/similar models, and changes in systemic risks originating from one of the company’s models.

Other safety commitments under the AI code of practice:

• Implementing safety mitigation measures that are sufficiently robust under adversarial pressure (fine-tuning attacks/jailbreaking). Examples of these include filtering and cleaning data, filtering model outputs, and changing the model behaviour in the interests of safety, such as fine-tuning the model to refuse certain requests or provide unhelpful responses.
• Ensuring that the models have adequate cybersecurity protection and physical infrastructure throughout the model’s life cycle.
• AI companies have to submit information about their model and their systemic risk assessment and mitigation processes to the AI Office in the form of a Safety and Security Model report. They must submit this before the model is placed on the market. It will include details about the model architecture, capabilities, and how the company expects people to use its model; results of their evaluations; and at least five random input/output samples from each evaluation.
• Safety and Security model reports also have to include reports from external evaluators. 
• AI companies must commit to allocating clear responsibilities for managing the systemic risks stemming from their models across all levels of the organisation. They must also commit to allocating appropriate resources (financial, human, and computational resources, and access to information) to actors who have been assigned responsibilities for managing the risks.
• Tracking serious incidents in the AI model’s life cycle and taking corrective measures to address them. Companies must report these incidents to either the AI office or the appropriate national authorities.

Sharing information with downstream players:

One of the principles listed out under the code requires AI companies to recognise the importance of cooperation with licensees, downstream modifiers, and providers in systematic risk assessment and mitigation. Such cooperation may involve entering into agreements to share information relevant to systemic risk assessment and mitigation while ensuring proportionate protection of sensitive information and compliance. 

The aspect of sharing information with downstream players bears similarity to the Indian Computer Emergency Response Team’s (CERT-In) recently released AI Bill of Materials Guidelines. These guidelines state that AI suppliers must provide procuring parties with information about known vulnerabilities/weaknesses of their models. This, CERT-In believes, will enable procuring organisations to pinpoint weak spots, implement appropriate safeguards, and proactively address potential risks before they can be exploited.

Industry perspective on the AI code of practice:

AI companies appear to be dissatisfied with the code, stating that it imposes a disproportionate burden on AI providers, according to a statement by the Computer and Communications Industry Association (CCIA). CCIA is an industry body with members such as Meta, Amazon, Google, and Apple. CCIA argues that “without meaningful improvements to the code, signatories remain at a disadvantage compared to non-signatories.” Further adding that the security and safety measures under the AI code of practice are “overly prescriptive and disproportionate”, giving the example of the external evaluation requirement.

On the other hand, OpenAI has announced its intent to sign the AI code of practice. “The Code of Practice opens the door for Europe to move forward with the EU AI Continent Action Plan that was announced in April,” the company says. It adds that the code of practice will work best if the EU aligns it with its action plan.