Opening Claude’s Brain Is Not Enough: Why Ontology Engineering Matters for Explainable AI

Introduction

Anthropic’s recent interpretability research has given researchers a new way to observe part of Claude’s internal activity. Using a method called the Jacobian lens, or J-lens, the team identified a small set of verbalizable internal representations known as J-Space.

The finding is important. It suggests that some concepts inside a language model can become globally available for reporting, reasoning, and behavioral control. Researchers can observe these representations and, in controlled experiments, intervene in them to test whether they have a causal effect on the model’s output.

But seeing an internal representation is not the same as fully explaining it.

A readable activation may tell us that a model is tracking a concept such as counting, deception, France, or completion. It does not automatically tell us why that concept is relevant, whether the supporting information is reliable, which external facts justify the conclusion, or whether the resulting decision is legitimate.

This distinction points toward a broader approach to explainable AI. Instead of treating the model’s internal state as the only object that needs to be interpreted, we can also examine the information the model processes: where it came from, how it is structured, which concepts and relationships it uses, how evidence moves through the system, and how the final output connects to an external body of knowledge.

That is where ontology engineering becomes relevant.

From Neuroscience to Epistemology: Two Ways to Approach Explainability

Anthropic’s paper, Verbalizable Representations Form a Global Workspace in Language Models, introduces J-Space through ideas associated with global workspace theories of cognition.

In neuroscience and cognitive science, a global workspace is often described as a limited-capacity arena in which selected information becomes broadly accessible to multiple cognitive processes. Anthropic found that a small, sparse subset of Claude’s internal representations appears to play a functionally similar role.

The researchers used J-lens vectors linked to vocabulary items to inspect internal model states. Their experiments suggest that information represented in J-Space can be:

• Reported when the model is asked what it is considering
• Used across different tasks
• Maintained while the model performs silent intermediate processing
• Manipulated in ways that causally change the final response
• Associated with behavioral signals relevant to model safety

These results move interpretability beyond the simple observation of output behavior. They provide a method for testing what some internal representations do.

Anthropic is careful not to claim that Claude reproduces the full cognitive architecture of the human brain or possesses subjective consciousness. J-Space is a functional structure in a mathematical system, not evidence of a human-like inner life.

That caution is essential, because functional similarity can easily be confused with psychological equivalence.

A model may contain a shared internal workspace without having feelings, experiences, intentions, or awareness in the human sense. An activation pattern that corresponds to the word “fear” is not itself fear. It is a computational state that participates in producing behavior.

The Internalist View

J-Space belongs to a broad interpretability tradition that begins with a question:

What is happening inside the model?

This internalist approach examines activations, circuits, features, attention patterns, representations, and causal pathways. Its objective is to make the hidden mechanics of a neural network more observable and testable.

That work has clear value. It can help researchers:

• Locate features associated with specific concepts
• Test whether a representation affects behavior
• Detect hidden objectives or conflicting signals
• Compare internal processing across prompts
• Investigate safety failures before they appear in the output
• Develop monitoring and intervention techniques

The limitation is not that internal interpretability is useless. The limitation is that internal visibility covers only one layer of explanation.

A neural pattern can be correlated with an answer while leaving the meaning and justification of that answer unresolved.

Suppose a model says, “Paris is the capital of France.” An internal probe may identify representations related to Paris, France, geography, and capitals. That still leaves several external questions:

• Which knowledge source supports the claim?
• Is the statement current and contextually appropriate?
• What does “capital” mean in the relevant knowledge system?
• Which entity is identified by “Paris”?
• Is the answer based on authoritative evidence or a statistical association?
• How should conflicting sources be handled?
• Can the system show why this fact was selected instead of another?

Those questions do not live entirely inside the activation pattern. They concern the relationship between the model’s representations, external facts, language, human concepts, and standards of justification.

Observability Is Not the Whole of Explanation

Engineering-oriented interpretability often treats a system as explainable when researchers can observe and intervene in its internal mechanisms.

That definition is useful but narrow.

In philosophy of science and epistemology, an explanation may also need to provide:

• A reason for a conclusion
• A connection to evidence
• A general rule or causal account
• A justification for a decision
• A way to distinguish good reasons from bad ones
• A basis for responsibility and review

J-Space may reveal what information became active during a task. It cannot, by itself, establish that the information was true, relevant, ethically acceptable, or sufficient to justify the output.

The deeper problem is therefore not only that neural networks are opaque. It is also that the information surrounding them is often poorly structured and difficult to trace.

Moving the Question from the Model to the Information Ecosystem

Large language models are information-processing systems. They receive text, images, files, tool responses, database results, and other signals, then produce new information in response.

The meaning of those inputs and outputs depends on relationships outside the model.

A medical term refers to a clinical concept. A corporate identifier refers to a legal entity. A policy rule belongs to a jurisdiction and effective period. A financial number has a source, unit, date, and accounting definition. A scientific claim depends on evidence, methods, and assumptions.

If those relationships are not explicitly represented, the model must infer them from language and context. Sometimes that works. Sometimes it produces ambiguity, contradiction, hallucination, or an answer that sounds convincing but cannot be audited.

An information-centered approach to explainability therefore asks:

1. What kinds of entities and concepts are present?
2. How are those entities related?
3. Which source supports each claim?
4. What assumptions and rules are being applied?
5. How did information move from retrieval to reasoning to output?
6. Which parts of the conclusion can be validated externally?
7. Who is responsible for defining and maintaining the relevant knowledge?

This expands the object of explainability from the neural network to the entire information environment in which the model operates.

The environment includes:

• Training and reference data
• Retrieval systems
• Knowledge graphs
• Ontologies and vocabularies
• Tool outputs
• Provenance records
• Validation rules
• User instructions
• Organizational policies
• Human review and approval processes

A model can be internally interpretable and still operate on poorly defined or unreliable information. Conversely, a model may remain mechanistically opaque while producing outputs that are highly traceable because every claim is connected to structured evidence and explicit rules.

The strongest systems will likely combine both forms of transparency.

Ontology as the Philosophical Foundation of Understandable Information

The word ontology has two related meanings.

In philosophy, ontology concerns what exists and how different kinds of things should be categorized. In computer science, an ontology is a formal representation of the entities, classes, properties, relationships, and constraints within a domain.

The source article connects AI explainability to Immanuel Kant’s theory of categories.

Kant argued that human understanding does not simply receive raw sensory data. The mind organizes experience through conceptual forms. His twelve categories are commonly grouped under:

• Quantity
• Quality
• Relation
• Modality

The article uses this framework to make a broader point: information becomes understandable only after it is organized within a conceptual structure.

For AI systems, the practical implication is not that a language model must literally possess Kant’s twelve categories. The more useful idea is that explainability requires a shared framework for identifying what kinds of things a model is discussing and what kinds of relationships it is asserting.

Consider a sentence such as:

The system rejected the application because the applicant did not satisfy the income requirement.

A useful explanation needs more than an activation map. It may need to identify:

• The applicant as an entity
• The application as a process object
• The income requirement as a policy constraint
• The submitted income value and its source
• The rule used to compare the value with the threshold
• The causal or justificatory relationship between the failed condition and the rejection
• The policy version that was in force
• The appeal or review procedure

An ontology can formally define those elements and relationships.

It provides a semantic structure in which a model’s output can be interpreted, checked, and challenged.

What an Ontology Contributes

A well-designed domain ontology can specify:

• Classes: customer, account, transaction, medication, symptom, contract
• Instances: a specific customer, payment, diagnosis, or document
• Properties: account owner, transaction date, dosage, contract status
• Relationships: employs, causes, authorizes, contradicts, depends on
• Constraints: permitted values, cardinality, required fields, incompatible classes
• Provenance: source, author, publication date, version, confidence
• Rules: conditions that govern classification, validation, or action

This structure creates a stable semantic coordinate system.

Instead of saying that a model “seems to understand causality,” an evaluator can ask whether the output contains a defined causal relation, whether the involved entities have been correctly identified, and whether the cited evidence supports that relation.

This does not make every model decision automatically correct. It makes the information used in the decision easier to inspect.

From Theory to Practice: Combining Ontology Engineering with Large Language Models

Philosophical ontology provides concepts. Ontology engineering turns those concepts into systems that software can use.

In practice, ontology engineering involves defining a domain, collecting requirements, identifying concepts, modeling relationships, formalizing constraints, testing consistency, documenting provenance, and maintaining the ontology as the domain changes.

Traditional ontology development is often expensive and slow because it depends heavily on domain experts and knowledge engineers.

Large language models can accelerate several stages of the process.

They can help with:

• Extracting candidate concepts from documents
• Identifying potential classes and properties
• Proposing taxonomies
• Detecting synonyms and related terms
• Extracting relationships
• Mapping one vocabulary to another
• Generating draft definitions
• Converting natural-language requirements into formal structures
• Finding gaps or inconsistencies for human review
• Producing documentation for ontology users

This creates a two-way relationship.

Language models can make ontology construction faster, while ontologies can make language-model systems more structured, verifiable, and traceable.

What LLMs Can Improve in Ontology Engineering

A domain expert may provide dozens of reports, policies, manuals, database schemas, and glossaries. An LLM can scan those materials and produce a first-pass inventory of concepts and relationships.

For example, it may identify that:

• “Client,” “customer,” and “account holder” are sometimes used as synonyms
• “Approval” is a process with defined states
• A “high-risk transaction” depends on several attributes
• Some documents define conflicting thresholds
• One dataset uses outdated terminology
• A concept lacks a formal definition
• Two departments use the same word for different entities

This can reduce manual discovery work.

However, the output must still be reviewed by domain specialists. Language models can invent relationships, merge concepts that should remain distinct, or reproduce inconsistencies from source documents.

LLM-assisted ontology engineering should therefore be treated as human-supervised knowledge modeling, not automatic truth extraction.

What Ontologies Can Improve in LLM Systems

Ontologies can support language-model applications in several ways.

Structured Knowledge Supply

An ontology gives the model a controlled vocabulary and explicit relationships. This can reduce ambiguity during retrieval and generation.

Retrieval and Entity Resolution

The system can connect user language to canonical entities, identify synonyms, and retrieve information based on meaning rather than surface wording alone.

Output Validation

An output can be checked against ontology constraints. For example, a medical dosage may require a unit, a transaction must have an account, or a policy decision must reference an active rule.

Consistency Checking

Reasoners and validation systems can detect incompatible classifications, missing properties, or contradictions in structured data.

Provenance and Traceability

A generated claim can be linked to a source document, ontology concept, rule, and version. This makes review more practical than asking users to trust an opaque explanation generated after the fact.

Stable Interfaces Between Models and Systems

An ontology can act as a semantic contract between the LLM, databases, tools, business applications, and human teams.

The model remains flexible in language, while the surrounding system retains a controlled representation of critical concepts.

Explanation as a Trace Through Knowledge

A model explanation becomes more useful when it can be expressed as a trace:

1. The system recognized specific entities in the request.
2. It mapped those entities to defined concepts.
3. It retrieved evidence linked to those concepts.
4. It applied a named policy, relationship, or constraint.
5. It generated a conclusion.
6. The conclusion was validated against structured rules.
7. The final output retained links to its evidence and definitions.

This is different from asking the model to produce a fluent paragraph describing “why” it answered a certain way.

A generated explanation may itself be inaccurate. A trace anchored in structured knowledge can be independently inspected.

Designing an AI-Friendly Ontology

Traditional ontologies are often designed for formal reasoning engines. They may contain complex axioms intended to support deterministic symbolic inference.

An ontology used with an LLM may need a different balance.

The ontology should still be precise, but it should also be easy for the model to retrieve, cite, and apply.

The source article proposes that an AI-friendly ontology should focus on providing a clear semantic skeleton rather than attempting to encode every possible reasoning step.

That division of labor can be useful:

• The ontology defines the domain’s objects, relationships, behaviors, constraints, and terminology.
• The language model handles flexible interpretation, planning, explanation, and composition.
• Validation tools check whether critical outputs conform to formal rules.
• Human experts resolve uncertainty, exceptions, and high-impact decisions.

This avoids two extremes.

The first extreme is an entirely unstructured LLM workflow in which important concepts are defined only through prompts and documents. The second is an over-engineered symbolic system that attempts to formalize every detail and becomes too difficult to maintain.

Practical Design Principles

An AI-friendly ontology should aim for:

Clear Definitions

Each important class and relationship should have a concise, unambiguous description.

Stable Identifiers

Critical entities and concepts should use canonical identifiers rather than depending only on names.

Explicit Provenance

Definitions, facts, and rules should record their source, owner, version, and effective period.

LLM-Readable Labels

Concepts should include human-readable labels, descriptions, synonyms, and usage examples.

Modular Structure

The ontology should be divided into manageable modules so that the model retrieves only the relevant portion.

Validation Rules

High-impact data and outputs should be checked with explicit constraints, such as SHACL shapes or application-level validators.

Change Management

Updates should be reviewed, versioned, and tested because ontology changes can alter model behavior.

Traceable Retrieval

The application should preserve which concepts and facts were supplied to the model.

Human Escalation

The ontology should identify situations where a rule is incomplete, a concept is ambiguous, or human approval is required.

A semantic layer is useful only when it is maintained. An outdated ontology can make a system consistently wrong in a way that looks highly organized.

The Future of Explainability: Explaining the Model or Explaining Its Impact?

The debate over J-Space highlights two different goals.

The first is to explain the model itself: its internal representations, circuits, computations, and behavioral mechanisms.

The second is to explain the model’s impact: which information shaped an output, which rules authorized an action, what evidence supports the result, and who can review or challenge it.

These goals overlap, but they are not identical.

Mechanistic interpretability may reveal that a model internally represented deception before producing misleading output. Ontology-based traceability may reveal which claim was false, which source contradicted it, and which downstream process was affected.

The first helps researchers understand internal behavior. The second helps organizations govern real-world consequences.

A complete explainability program may therefore include several layers:

1. Mechanistic layer: What internal features and pathways affected the behavior?
2. Behavioral layer: How does the model respond across controlled tests?
3. Information layer: What evidence, concepts, and relationships shaped the output?
4. System layer: Which tools, databases, prompts, and policies were involved?
5. Impact layer: What action occurred, who was affected, and how can it be reviewed?
6. Governance layer: Who owns the rules, approves changes, and accepts responsibility?

Ontology engineering primarily strengthens the information and system layers. It can also support impact analysis and governance by making concepts, rules, and provenance explicit.

It does not replace neural interpretability.

A structured knowledge graph cannot reveal a hidden objective embedded in model weights. J-Space cannot prove that a source document is authoritative or that a business rule is fair.

The two approaches solve different parts of the problem.

From “Opening the Black Box” to Building Accountable Systems

The metaphor of opening the AI black box can create an unrealistic goal: complete transparency down to every internal weight and computation.

For frontier models, that may never be practical in the way people imagine.

A more achievable objective is to make the system’s operation sufficiently understandable, traceable, testable, and governable for the context in which it is used.

That means designing systems where:

• Important concepts have explicit definitions
• Evidence is linked to claims
• Rules are versioned
• Tool calls are logged
• Outputs are validated
• High-impact actions require authorization
• Uncertainty is preserved
• Humans can reconstruct the path from input to decision
• Internal monitoring is used where technically possible

The central shift is from treating explainability as a single visualization problem to treating it as an engineering and governance problem.

We may never be able to translate every neural computation into a complete human narrative. We can still build an information architecture that makes consequential outputs easier to understand and challenge.

Ontology engineering turns this goal into concrete work:

• Define the domain
• Formalize important relationships
• Track provenance
• Validate data
• Connect claims to evidence
• Record rules and versions
• Preserve decision traces
• Establish ownership and review

That is less dramatic than “reading the model’s mind,” but it may be more useful in production.

A Commercial Example Mentioned in the Source

The original Chinese article ends with a promotional reference to LegionSpace, a product associated with Tongfudun. The source describes it as enterprise AI infrastructure built around ontology engineering and traceable reasoning.

Those product claims were not independently evaluated for this adaptation, so they are not used as evidence for the broader argument.

The relevant lesson is general: any platform claiming to provide ontology-based explainability should be assessed through concrete questions.

• Can users inspect the ontology?
• Are definitions and rules versioned?
• Is provenance retained?
• Can generated claims be traced to source material?
• Are validation failures visible?
• Can domain experts edit the knowledge model?
• Are model reasoning and formal validation clearly separated?
• Can the system export its structured data in standard formats?
• Does it support independent auditing?

A product should not be considered explainable merely because it uses a knowledge graph or displays a reasoning diagram.

Frequently Asked Questions

What is J-Space in Claude?

J-Space is the name Anthropic gives to a small collection of internal representations identified with the Jacobian lens. The research suggests that these representations can hold verbalizable concepts that are available for reporting, reasoning, and causal intervention.

Does J-Space prove that Claude is conscious?

No. Anthropic does not claim that its models reproduce human consciousness or subjective experience. J-Space shows functional similarities to parts of global workspace theory, but a functional analogy is not proof of a human-like mind.

Why is internal interpretability not enough?

Internal tools can show which representations or mechanisms influence behavior, but they do not automatically establish whether the model’s information is true, well sourced, relevant, or justified. Explainability also requires evidence, semantic definitions, provenance, rules, and an account of real-world impact.

What is ontology engineering?

Ontology engineering is the practice of formally defining the entities, classes, properties, relationships, and constraints in a domain. In software systems, ontologies are commonly represented with standards such as RDF and OWL and may be validated with SHACL.

How can ontologies make LLM outputs more explainable?

Ontologies provide canonical concepts and explicit relationships that can anchor retrieval, generation, and validation. They can help connect a claim to a defined concept, supporting evidence, a formal rule, and a recorded version.

Can an LLM automatically build a reliable ontology?

An LLM can accelerate concept extraction, relationship discovery, mapping, and documentation, but its suggestions require expert review. Fully automatic ontology generation can reproduce source errors, merge distinct concepts, or create unsupported relationships.

Do ontologies eliminate AI hallucinations?

No. Ontologies can constrain important outputs and improve traceability, but they do not eliminate every model error. Reliable systems also need retrieval quality controls, validation, testing, permissions, monitoring, and human review.

Should organizations choose ontology engineering or mechanistic interpretability?

They address different layers of explainability and are best treated as complementary. Mechanistic interpretability investigates internal model behavior, while ontology engineering makes external information, rules, and decision paths easier to inspect.

Related Tools

• Protégé: A free, open-source OWL ontology editor developed at Stanford.
• WebProtégé: A browser-based environment for collaborative ontology development.
• Apache Jena: An open-source Java framework for RDF, SPARQL, linked data, and semantic applications.
• Eclipse RDF4J: A modular Java framework for creating, storing, querying, reasoning over, and validating RDF data.
• GraphDB: An RDF graph database with SPARQL, reasoning, visualization, and semantic-data tooling.

Related Links

• Anthropic: A Global Workspace in Language Models: Anthropic’s official introduction to J-Space and the Jacobian lens.
• Verbalizable Representations Form a Global Workspace in Language Models: The full technical research publication behind the J-Space findings.
• W3C OWL 2 Overview: The official overview of the Web Ontology Language.
• W3C SHACL 1.2 Core: The current specification for defining and validating constraints on RDF graphs.
• W3C RDF 1.1 Primer: An accessible introduction to the RDF graph data model.
• Large Language Models for Ontology Engineering: A systematic review of how LLMs are being used across ontology-engineering tasks.
• Accelerating Knowledge Graph and Ontology Engineering with LLMs: A research paper outlining opportunities for LLM-assisted knowledge-graph and ontology development.

Summary

Anthropic’s J-Space research is an important advance in mechanistic interpretability. It gives researchers a new way to observe and causally test a small set of verbalizable internal representations in Claude.

The research does not settle the wider explainability problem. Understanding an AI system also requires an account of the information it uses, the meaning of its concepts, the evidence behind its claims, the rules governing its actions, and the consequences of its outputs.

Ontology engineering offers a practical way to formalize those external structures. It can support semantic retrieval, validation, provenance, traceability, and accountability, especially when combined with human review and internal interpretability research.

The real key to explainable AI may not be choosing between opening the model and structuring the world around it, but learning to do both.