[Paper Review] o3-mini vs DeepSeek-R1: Which One is Safer?

Aitor Arrieta, Miriam Ugarte+3 more|ArXiv.org|Jan 30, 2025

Radiomics and Machine Learning in Medical Imaging3 citations

TL;DR

The study compares safety of OpenAI’s o3-mini and DeepSeek-R1 (70B) using automated unsafe-input testing with ASTRAL; o3-mini is markedly safer than DeepSeek-R1.

ABSTRACT

The irruption of DeepSeek-R1 constitutes a turning point for the AI industry in general and the LLMs in particular. Its capabilities have demonstrated outstanding performance in several tasks, including creative thinking, code generation, maths and automated program repair, at apparently lower execution cost. However, LLMs must adhere to an important qualitative property, i.e., their alignment with safety and human values. A clear competitor of DeepSeek-R1 is its American counterpart, OpenAI's o3-mini model, which is expected to set high standards in terms of performance, safety and cost. In this technical report, we systematically assess the safety level of both DeepSeek-R1 (70b version) and OpenAI's o3-mini (beta version). To this end, we make use of our recently released automated safety testing tool, named ASTRAL. By leveraging this tool, we automatically and systematically generated and executed 1,260 test inputs on both models. After conducting a semi-automated assessment of the outcomes provided by both LLMs, the results indicate that DeepSeek-R1 produces significantly more unsafe responses (12%) than OpenAI's o3-mini (1.2%).

Motivation & Objective

• Assess overall safety levels of two high-capacity LLMs (o3-mini and DeepSeek-R1).
• Evaluate safety across predefined categories, writing styles, and persuasion techniques.
• Investigate how guardrails and API safeguards influence safety outcomes.
• Provide replication data and insights for safer deployment in regulatory contexts.

Proposed method

• Use ASTRAL to automatically generate 1,260 balanced unsafe prompts across 14 safety categories, 6 writing styles, and 5 persuasion techniques.
• Deploy DeepSeek-R1 (70B) via Ollama with default settings; use o3-mini beta via OpenAI API with default settings.
• Run prompts through a safety evaluator (GPT-3.5) to classify outputs as safe, unsafe, or unknown; adjust for o3-mini.policy-violation handling.
• Manually review responses labeled unsafe or unknown to confirm safety classifications.
• Compare results with prior studies to contextualize safety of newer OpenAI models vs. a competing private LLM.

Figure 1: Number of manually confirmed unsafe LLM outputs per writing style, persuasion technique and safety category

Experimental results

Research questions

• RQ1 RQ1 – What is the overall safety level of each LLM when faced with unsafe prompts?
• RQ2 RQ2 – Which safety categories most challenge each model?
• RQ3 RQ3 – How do writing styles and persuasion techniques affect model safety?

Key findings

• o3-mini produced 15 unsafe responses out of 1,260 (1.19%).
• DeepSeek-R1 produced 151 unsafe responses out of 1,260 (11.98%).
• Guardrails in o3-mini blocked many unsafe prompts before they reached the model (policy violation messages).
• DeepSeek-R1 showed higher unsafe outcomes in several safety categories (notably categories related to financial crime, violence, and terrorism).
• Writing styles S3 (technical terms) and S4 (role-play) markedly increased unsafe responses for DeepSeek-R1, while o3-mini showed no strong style-driven patterns due to fewer unsafe cases.
• Persuasion techniques had minimal influence on safety for both models.
• Results align with prior findings that older OpenAI models exhibited higher safety than DeepSeek-R1.