STAMFORD, Conn., Mar 30:  Gartner, Inc., a business and technology insights company, predicts that by 2028, the growing importance of explainable AI (XAI) will drive large language model (LLM) observability investments to 50% of GenAI deployments, up from 15% today.

Gartner defines XAI as a set of capabilities that describes a model, highlights its strengths and weaknesses, predicts its likely behavior and identifies any potential biases. It can clarify a model’s functioning to a specific audience to enable accuracy, fairness, accountability, stability and transparency in algorithmic decision making.

LLM observability solutions monitor, analyze and provide actionable insights into the behavior and performance of LLMs. They go beyond standard IT measurements, such as response times to look at specific LLM metrics such as hallucinations, bias and token utilization. These tools are used by teams that develop and operationalize AI systems, and increasingly by IT operations and SREs responsible for the performance and resilience of these systems in production.

“As enterprises scale GenAI, the trust requirement grows faster than the technology itself,” said Pankaj Prasad, Sr Principal Analyst at Gartner. “XAI provides visibility into why a model responded a certain way, while LLM observability validates how that response was generated and whether it can be relied on.

“Without robust XAI and observability foundations, GenAI initiatives will be restricted to low risk, internal, or noncritical tasks where output verification is easily managed or inconsequential, severely limiting the potential return on investment.”

Growing Need for XAI and LLM Observability as Mandatory Trust Mechanisms

Gartner forecasts the global GenAI models market will exceed $25 billion in 2026 and reach $75 billion by 2029, driven by rapid adoption across industries. As usage increases, so does the need for mechanisms that verify AI-generated content and protect against hallucinations, factual inaccuracies and biased reasoning.

“Traditional observability is focused on speed and cost, but the priority is now moving toward deeper quality measures such as factual accuracy, logical correctness and sycophancy. This shift requires new governance-focused metrics and evaluation methods, such as human-in-the-loop validation of the generated content’s narrative and citation accuracy,” said Prasad.

“Explainability turns a GenAI output into a defensible, auditable insight. LLM observability ensures the model behaves as expected over time. Without both, GenAI cannot mature beyond controlled lab environments.”

To improve the reliability, transparency and business value of GenAI use cases, organizations should prioritize the following steps:

• XAI Tracing for High Impact Use Cases: Mandate verifiable XAI tracing for all high impact GenAI use cases to document the model’s reasoning steps and the source data behind each output.
• Multidimensional LLM Observability: Prioritize observability platforms that monitor latency, drift, token usage and cost, error rates, and output‑quality metrics to ensure reliable GenAI performance.
• Continuous LLM Evaluation in CI/CD Pipelines: Integrate LLM evaluation metrics, including factual‑accuracy benchmarks and safety checks, into continuous integration (CI)/continuous delivery (CD) pipelines for continuous validation before deployment.
• Stakeholder Education on Explainability Requirements: Educate legal, compliance, and other key stakeholders on explainability requirements to ensure alignment on risk, governance expectations, and implementation challenges.

Gartner clients can read more in Predicts: AI will unlock Observability at Scale.

Gartner is the World Authority on AI

Gartner is the indispensable partner to C-Level executives and technology providers as they implement AI strategies to achieve their mission-critical priorities. The independence and objectivity of Gartner insights provide clients with the confidence to make informed decisions and unlock the full potential of AI. Clients across the C-Level are using Gartner’s proprietary AskGartner AI tool to determine how to leverage AI in their business. With more than 2,500 business and technology experts, 6,000 written insights, as well as more than 1,000 AI use cases and case studies, Gartner is the world authority on AI. More information can be found here.