System Prompt Leakage

The Silent Threat Lurking in LLMs

I have spent the past few days reviewing a number of incidents and threats to LLMs and came across, among other reports, OWASP's Top 10 Threats to LLMs as part of my research. It reinforced the need for organizations to be more proactive when managing and securing data and implementing a secure framework when developing AI tools for their organization.

One of the risks climbing fast up the AI security charts in 2025 is System Prompt Leakage (System PLeak). This threat exposes hidden system instructions (and sometimes even API keys) to the outside world through clever prompt injections. And trust me, it's no longer theoretical. Just this quarter, it jumped to over 20% of all LLM-based security incidents - up from just 5% last year. ⬇️

Rise of LLM threats 2023-2025

So what's happening?

LLMs are built to follow instructions - but they're not great at telling where those instructions are coming from. If attackers can manipulate inputs well enough, they can trick models into revealing their internal system prompts. That's the core of this issue.

Some of OWASP's LLM identified threats that often intertwine with prompt leakage:

• Prompt injection - hijacks model behaviour

• Sensitive information disclosure - leaks private/internal data

• Insecure output handling - delivers malicious or unsafe outputs

Here's what to do now

1. Separate system and user prompts: use sandboxed or API-based design so the LLM can't confuse the two.

2. Treat system prompts like they will leak: don't include sensitive information or core logic. Offload critical actions to secure backend systems.

3. Explore active defenses: look into solutions like PromptKeeper or ProxyPrompt to detect and obfuscate leakage in real-time.

4. Audit every response: logging, red-teaming and regex scanning can catch echoed prompts before they escape to end users.

Want to know more?

How is your team defending against this? Are you testing prompt injections yet? Let's start a real conversation on what's working (and what's not).

Reach out and let the team at Pillar Group Advisory help shape your LLM defense strategy.