Ethical Guidelines for Designing AI Negotiation Agents

Abstract: Artificial Intelligence (AI) is playing an increasingly prominent role in negotiations. While its initial use was primarily as a support tool, offering strategic suggestions, analyzing data, or simulating scenarios, AI systems are beginning to conduct negotiations autonomously. This development introduces significant advantages, including cost savings, scalability, and entirely new forms of negotiation, such as replacing standardized tenders with individualized interactions. However, it also raises serious ethical concerns. Many negotiation outcomes can be improved through tactics that exploit cognitive biases, omit critical information, or manipulate trust, tactics that are ethically questionable. An AI negotiation agent programmed to maximize gains will, if left unconstrained, likely adopt such strategies. Current regulatory frameworks are too general to meaningfully address the specific risks posed by autonomous negotiation agents. This paper offers conceptual groundwork for developing ethical guidelines tailored specifically to AI agents in negotiation contexts. Drawing on existing research in both AI ethics and negotiation ethics, as well as the authors’ experience in developing AI negotiation agents, we examine the potential risks of using AI for value claiming and value creation. Our contribution is a risk mapping, and a set of guidelines designed to address the challenges posed by AI agents across negotiation settings, considering their full lifecycle from design to development and deployment.

Keywords: Artificial intelligence, Negotiation ethics, Regulation, Unethical myopia, AI agents, EU AI Act