Safe Generative AI: Targeted Incentives for Developers
Research
10.12.2024
ZEW Proposal for Promoting Reliable Generative AI in the EU
End of November marks ChatGPT’s second birthday. For some time now, companies have been using generative artificial intelligence (AI), for example in the form of chatbots as digital assistants. However, the ever-increasing performance of generative AI is comes with risks, such as fake images or other fake news which are deliberately spread in disinformation campaigns – for example before elections. To address this situation, ZEW scientists are proposing a “Safe Generative AI Innovation Program”. The program would enable the EU to establish explicit economic incentives for the expansion of safe generative AI models.
“There is a market failure in generative AI development. Developers can presently generate significant revenue from capability improvements. But they don’t bear the costs associated with the societal risks posed by generative AI,” says Dr. Dominik Rehse, head of ZEW’s junior research group “Digital Market Design” and co-author of the proposal. “To counterbalance this situation, we propose a new EU funding program, which is to create explicit incentives for the development of safe generative AI. After all, safety of AI systems is an innovation desired by society, but so far, it has not been sufficiently provided by individual development teams. A similar problem is observed, for instance, with the development of some vaccines. Here, such funding programs have already been successfully implemented.
Milestone rewards as incentives
Under the proposed program, interested developers would participate in a competition announced and financed by the EU. The system would consist of milestones defined in terms of safety and other performance metrics. The developers whose models reach a milestone receive pre-specified amounts.
This incentive systems offers several advantages. It reduces uncertainty on the developers’ side about market demand: companies know from the beginning that investing in innovation will pay. At the same time, the EU is open to different approaches; it sets no specific requirements for the technologies used and provides only a few technical specifications. The overall aim is to attain the targets defined in the competition as efficiently as possible. This also means that the development teams will be responsible for the actual innovation work.
“The system aligns private incentives with public benefit. By adopting such an incentive mechanism, the EU has an opportunity to promote innovation more strategically than in the past and to establish a presence in the generative AI market where it has been only a minor player so far,” co-author Sebastian Valet estimates.
Further discussion on the proposal
For such a system to work, carefully defined metrics for safety and other performance criteria are needed. These should be as robust as possible and resistant to manipulation. Robustness is likely to result primarily from the use of transparent tests in which the public would provide tasks for generative AI algorithms that would lead to either misbehaviour or particularly good performance. Various approaches for this are already established, some have been proposed recently. They usually deliver a relatively robust ranking of the algorithms that provide the highest safety and performance.
It is also necessary to determine how large the reward linked to reaching each milestone should be. The aim must be to identify a reward size that is an attractive incentive for developers to reach the milestone. To determine this amount, participants can be surveyed for their expected minimum reward to reach the milestone. Innovative methods in economic research can be used for creating incentives for developers to state the expected minimum reward truthfully, so that valid answers are obtained.
“Careful planning helps mitigate foreseeable problems. But pragmatism should have priority over perfection, considering that the topic of safe generative AI is highly relevant to society,” says co-author Johannes Walter.
