PenguinHarness Builds and Self-Improves AI Agents for Just ¥0.2

Introduction

LlamaFactory made large-model fine-tuning easier for a much wider group of developers. Now, its creator Yaowei Zheng and the PrismShadow team are applying the same accessibility-first approach to AI agents.

Their new open-source project, PenguinHarness, is designed to automate three parts of the agent lifecycle:

1. Building an agent application
2. Evaluating its performance
3. Continuously improving its prompts, Skills, and configuration

Instead of manually selecting a framework, wiring tools, writing prompts, building an interface, and repeatedly testing the result, users can describe what they need and let PenguinHarness assemble a runnable agent application.

PenguinHarness is an open-source automated agent builder for desktop and server deployment.

The project is lightweight, open source under Apache 2.0, and available for Linux, macOS, and Windows. It can run locally or on a remote server through a browser interface.

PenguinHarness also supports online and local models through built-in presets and OpenAI-compatible endpoints. The official repository currently lists recent models from DeepSeek, Kimi, GLM, Qwen, OpenAI, Google, Anthropic, and other providers.

The original report describes a RAG application that cost only ¥0.2 in model tokens to generate. The official project page gives the same example as approximately $0.02 using DeepSeek V4 Pro.

That number is a project demonstration, not a guaranteed fixed price. Actual cost depends on the selected model, provider, prompt complexity, retries, application scope, and token pricing.

Letting an Agent Build Another Agent

PenguinHarness begins with a simple idea: an agent should be able to construct another agent application from a natural-language requirement.

This fits into the broader AI-for-AI and recursive self-improvement discussion, where AI systems help create, test, or improve other AI systems.

The first use case shown in the source article asks the system to build a RAG application that:

• Retrieves information in chunks
• Streams clear answers
• Includes citations
• Provides a usable frontend
• Can be run as a complete application

The comparison placed PenguinHarness beside a coding agent and asked both systems to complete a similar task.

According to the project demonstration, PenguinHarness completed the application more quickly and at substantially lower token cost. Its result included fluent answers, streaming output, and linked sources.

The competing output shown in the report mixed languages and did not include the same streaming behavior.

These examples are useful demonstrations, but they are not universal proof that PenguinHarness will outperform every coding agent on every repository. Results depend heavily on the model, agent configuration, task design, and evaluation method.

From Requirement to Runnable Application

Traditional agent development often involves several manual stages:

1. Choose an agent framework.
2. Select and configure a model.
3. Connect tools and external services.
4. Write the system prompt.
5. Define memory and workflow logic.
6. Build evaluation cases.
7. Test and revise the agent.
8. Create a frontend or delivery interface.
9. Package the application for deployment.

PenguinHarness attempts to turn those steps into a single agent-driven workflow.

When the requirement is clear, the system can generate:

• Application scaffolding
• Agent prompts
• Skills and tool definitions
• Configuration files
• Supporting code
• A frontend
• Installation instructions
• Run instructions
• Iterative fixes and optimizations

The official project example uses this request:

Collect the docs from https://github.com/ericbuess/claude-code-docs and build a RAG app that answers Claude Code questions as a configuration expert, citing its sources.

The project reports that the resulting RAG application consumed about $0.02, or ¥0.2, in model tokens when built with DeepSeek V4 Pro.

The File System Is the Source of Truth

PenguinHarness uses files as the primary collaboration layer between people and agents.

In this design:

• An agent is represented by files.
• Prompts are files.
• Skills are files.
• Configuration is stored in files.
• Conversation and execution information can be traced through stored records.
• New agents can be created by assembling or copying the required file structure.

This approach makes the agent easier to inspect and modify.

Instead of hiding important behavior inside a large application framework, PenguinHarness treats editable files as the main interface. A human can read and change them, and the agent can improve them during an approved optimization process.

The harness is responsible for assembling those files into a runnable agent object.

PenguinMessage Provides a Unified Protocol

At runtime, PenguinMessage serves as the common message format connecting the model, environment, tools, and user.

The source article compares it with a network packet: different components can exchange information through the same lightweight interface rather than requiring a unique integration for every model or environment.

PenguinHarness is therefore both:

• A framework for running agents
• An agent capable of understanding and extending its own structure

PenguinHarness combines PenguinMessage, Penguin SDK, and Penguin Skills in one lightweight architecture.

The three core areas shown in the project architecture are:

Component	Role
PenguinMessage	A minimal message protocol between the user, model, tools, and environment
Penguin SDK	The development layer used to construct agent applications
Penguin Skills	Reusable capabilities for building, evaluating, and optimizing agents

A Locally Reproducible Self-Evolution Loop

Building an agent is only the first step.

The longer-term goal of PenguinHarness is to let users operate agents that can be evaluated and improved locally without rebuilding the entire system by hand.

The project distinguishes two stages:

• Building an agent: moving from zero to one
• Improving an agent: moving from one to one hundred

Changing an agent is relatively easy because prompts, code, Skills, and configuration can all be edited. Determining whether a change actually makes the agent better is much harder.

Large language models are probabilistic, and agent systems add more uncertainty through tools, environments, memory, and multi-step decisions.

A reliable improvement loop therefore needs a reliable measurement system.

Why Evaluation Is the Foundation

An agent can appear better on one example while becoming worse overall.

Without a structured benchmark, an optimizer may:

• Overfit to a few demonstrations
• Memorize answers
• Exploit weaknesses in the evaluator
• Increase cost without improving quality
• Improve one task while damaging another
• Produce a higher score through reward hacking

The PrismShadow team spent more than six months exploring how to evaluate self-evolving agents.

The central challenge was the lack of benchmarks that clearly separate training experience from held-out testing.

If an agent improves on the same questions used for evaluation, it is difficult to know whether it learned a reusable strategy or simply memorized the answers.

GDPevo Separates Training and Test Tasks

The team created GDPevo, an evolution-native benchmark based on real business workflows.

The source article describes its coverage across fields such as healthcare, finance, and legal work. The current public V2 repository contains 240 tasks across 24 task groups, covering areas including:

• CRM
• ERP
• Finance
• Healthcare
• Legal workflows
• Data analysis
• Engineering operations

Each task group includes:

• One shared business environment
• Five training tasks
• Five held-out test tasks

GDPevo uses a method called rule hybridization. Business workflows are decomposed into smaller rules, distributed across training tasks, and recombined in held-out test tasks.

This structure helps determine whether an agent has learned a reusable workflow instead of merely seeing the test answer in advance.

GDPevo evaluates how much an agent improves on held-out business tasks after different forms of evolution.

The GDPevo paper reports that self-evolution improved held-out accuracy by as much as 16.44 percentage points in its experiments. It also notes that the best evolved agents remained well below the fully informed oracle ceiling of 91.6%.

That gap is important. Current agents can improve, but reliable self-evolution is far from solved.

PenguinHarness Packages Evaluation as Skills

PenguinHarness turns the central ideas behind GDPevo into reusable Skills for:

• Agent creation
• Benchmark design
• Agent evaluation
• Agent optimization

After the relevant Skill is called, multiple agents can collaborate on the improvement process.

The source article gives an example of an agent designed for tasks such as sports prediction, investment strategy generation, or e-commerce support.

Instead of manually revising the prompt and workflow, the user can ask PenguinHarness to create the evaluation set, run repeated tests, analyze failures, and propose a better version.

A project demonstration reported an increase from 53 points to 95 points after several iterations, with approximately ¥0.5 in model-token cost using DeepSeek V4 Flash.

This is a demonstration result from the project team, not a general benchmark guarantee. The outcome will vary with the task, rubric, model, sample size, and optimization budget.

The Four-Step Optimization Process

The self-evolution workflow uses multiple agents with different responsibilities.

1. Organize the Evaluation

The Optimizer Agent determines how many questions and repetitions are required, then launches multiple Evaluator Agents in parallel.

Parallel evaluation helps reduce the time required to test several tasks or repeated runs.

2. Score Independently

Each Evaluator starts an independent copy of the target agent and asks it to solve a task.

The Evaluator has access to the grading rubric. The target agent does not.

This separation is intended to prevent the target from directly optimizing against the hidden scoring instructions.

3. Analyze and Improve

The Optimizer collects the results and examines the execution traces.

It identifies why points were lost, then modifies approved parts of the target agent, such as:

• Prompts
• Skills
• Configuration
• Workflow files

The Optimizer produces a candidate next version.

4. Validate and Iterate

The Evaluators test the candidate again.

The Optimizer accepts the candidate only if it achieves a strictly higher score. If the score stays the same or falls, the framework returns to the earlier version.

The Optimizer coordinates parallel Evaluators and accepts only a higher-scoring candidate agent.

The process can be summarized as:

Target Agent vN
      ↓
Parallel Evaluator Agents
      ↓
Scores, rubrics, and execution traces
      ↓
Optimizer Agent
      ↓
Prompt, Skill, or configuration update
      ↓
Candidate Agent vN+1
      ↓
Accept only if the score is strictly higher

This combination of hidden rubrics, held-out tests, snapshots, and strict score improvement is designed to make optimization more reproducible.

The Contract Between the Harness and a Self-Evolving Agent

Self-evolution creates an obvious safety question: what is the agent allowed to change?

PenguinHarness defines those boundaries in a portable file named CONTRACT.md.

The contract states that capability can improve inside an approved area, while the harness kernel and its safety boundaries remain fixed.

CONTRACT.md defines the boundaries for agent evolution, auditing, versioning, and credential isolation.

The source article highlights four central rules.

1. Evolution Cannot Modify the Harness Kernel

The editable area is limited to the workspace, prompts, Skills, and approved configuration.

The agent is not allowed to rewrite the core harness or its security mechanisms.

This reduces the risk that an optimization process could weaken:

• Tool approval
• Permission controls
• Audit logging
• Credential isolation
• Version recovery
• Other system-level safety checks

2. Every Optimization Requires a Snapshot

Before an optimization round, the framework stores a version snapshot of the agent state.

If the candidate performs worse or causes an unwanted side effect, the system can return to an earlier version.

A self-improving system without rollback can accumulate damage. Versioning makes improvement reversible.

3. The Target Agent Cannot See the Rubric

The target agent receives the task but not the hidden evaluation criteria.

Only the Evaluator has access to the rubric.

This is designed to reduce shortcut behavior, answer memorization, and reward hacking.

It does not eliminate those risks completely, but it creates a cleaner separation between solving the task and grading the result.

4. Every Optimization Must Be Auditable

Model requests, tool calls, token use, timing, failures, approvals, and optimization changes are written to trace files.

The user can inspect what changed and why.

The broader project contract also includes:

• Approval before tool execution
• Audit records for approval decisions
• Credential isolation
• Model and agent decoupling
• Recoverable execution traces
• On-demand loading of relevant files
• Explicit error-handling rules

The result is a framework where the agent can evolve, but the evolution process remains visible and reversible.

Other Useful PenguinHarness Features

PenguinHarness includes several capabilities beyond automated agent creation and optimization.

Built-In Skills for Model Training and Deployment

The project ships with Skills related to AI application development, including:

• penguin-sdk
• penguin-cli
• agenthub-models
• vllm
• ollama
• llamafactory

PenguinHarness includes Skills for building agents and working with tools such as vLLM, Ollama, and LlamaFactory.

These Skills let users describe training or deployment tasks in natural language and have the agent prepare the necessary files or commands.

The official repository groups built-in Skills into four categories:

Skill Group	Examples
Office Productivity	Data analysis and web data collection
Software Development	Web design and software engineering
AI App Development	Penguin SDK, model gateways, vLLM, Ollama, and LlamaFactory
Agent Tuning	Agent creation, benchmark design, evaluation, and optimization

Users and agents can also create or improve additional Skills.

A Unified Model Gateway

PenguinHarness includes model presets and supports custom OpenAI-compatible endpoints.

The project says this gives users access to more than 1,000 online and local models through supported providers and gateways.

The model gateway lets users configure different online and local model providers.

The current repository lists model families including:

• DeepSeek
• Kimi
• GLM
• Hunyuan
• Qwen
• GPT
• Gemini
• Claude

Model availability and provider names change frequently, so the in-app Models page and current official documentation should be treated as the latest source.

Giving a Text Model a Visual Proxy

The source article describes a development pattern where a text-focused model such as DeepSeek acts as the primary agent while a vision-capable model acts as a visual assistant.

The visual proxy can inspect:

• Webpage screenshots
• Slide decks
• Interface layouts
• Rendered game screens
• Charts
• Visual errors

The primary model can then revise its output based on the visual description.

A vision-capable proxy can inspect screenshots while DeepSeek remains the primary working model.

This can be useful when the main model is strong at coding or reasoning but does not natively process images.

The technique does not give the primary model direct vision. It creates a multi-model workflow where the visual model converts the screenshot into information the primary agent can use.

Fine-Grained Trace Analysis

PenguinHarness records model calls, tool execution, timing, token usage, approvals, and subagent activity.

The Trace interface displays the sequence on a timeline.

The Trace view helps users inspect parallel subagents, model calls, tool use, latency, and token cost.

This can help identify:

• Slow model calls
• Unnecessary tool use
• Expensive reasoning loops
• Failed retries
• Subagents that block the workflow
• Long approval delays
• Token-heavy prompts
• Opportunities for parallel execution

Trace data is also central to self-evolution because the Optimizer needs evidence about why the previous version lost points.

A Minimal Blank-Agent Template

PenguinHarness can also be used as a minimal general-purpose agent rather than only as an automated builder.

The project uses the shell as a universal low-level interface and keeps the default toolset intentionally small.

It treats subagent execution as a core performance feature.

The source report states that the default system prompt is approximately 1,300 tokens, compared with roughly 15,000 tokens in the project’s Claude Code comparison.

That figure is project-reported and may change as either product evolves.

The underlying design principle is stable: a shorter prompt and smaller toolset can reduce token overhead and make open models easier to use.

Cost and Benchmark Results

The project publishes a comparison on a complex data-analysis suite.

The project reports higher accuracy at a fraction of the model cost in its complex data-analysis comparison.

The published table reports:

Framework	Model	Accuracy	Token Use	Estimated Cost
PenguinHarness	DeepSeek V4 Pro	66.67%	18.04M	$0.55
Claude Code	Claude Opus 4.8	53.33%	22.20M	$38.48
OpenAI Codex	GPT-5.5	53.33%	13.72M	$19.41

The project summarizes this as:

• 1/35 of the reported Codex cost
• 1/70 of the reported Claude Code cost

This comparison combines a harness with the model normally paired with it. It does not isolate the contribution of the harness from the contribution of the selected model and provider pricing.

For a fair internal evaluation, teams should run the same tasks with:

• The same model where possible
• The same provider pricing
• The same retry policy
• The same tool access
• The same time limit
• The same evaluation rubric
• Multiple repeated runs

The public figures are useful as project benchmarks, but they should not be interpreted as a universal cost ratio for every task.

Reported Production Deployments

The source article says the team has already used PenguinHarness in two production scenarios.

Medical Report Review

A health-examination organization reportedly used PenguinHarness to create a report-checking agent with performance described as comparable to a medical expert.

According to the project team, a review that previously took about 30 minutes could be completed in tens of seconds.

Manufacturing Inspection

A manufacturing company reportedly deployed multiple PenguinHarness-built agents to monitor production-line equipment continuously and attempt automated recovery.

The team reports:

• A 65% reduction in downtime
• Output increasing to nearly twice the previous level

These are vendor-provided case-study figures. The original report does not provide the customer names, study design, sample size, baseline definition, or an independent audit.

They should be treated as examples of reported use rather than guaranteed operational outcomes.

Installation and Deployment

PenguinHarness supports Linux, macOS, and Windows 10 or later on x64 and Arm64 systems where available.

The one-line installers include their own Node.js runtime. Installing through npm requires Node.js 24 or later.

Linux or macOS

curl -fsSL https://penguin.ooo/install.sh | sh
penguin web

The Web interface opens at:

http://127.0.0.1:7364

Windows PowerShell

irm https://penguin.ooo/install.ps1 | iex
penguin web

npm

npm install -g @prismshadow/penguin-cli
penguin web

On a CLI installation, the first Web login uses the username admin. The initial password is printed when the server starts for the first time and should be changed immediately.

Configure a Model and Run a Task

The official repository provides a CLI example similar to the following:

penguin config model add \
  --provider deepseek \
  --model-id deepseek-v4-flash \
  --api-key sk-... \
  --set-default

Run a one-shot task:

penguin run -m "Create hello.txt containing Hello, Penguin"

Start an interactive session:

penguin chat

Start the headless server:

penguin server

API keys should never be committed to source control or pasted into public logs.

PenguinHarness can run on a local machine or a server. Its browser interface supports multiple sessions, agent and Skill management, model configuration, usage statistics, Trace observability, and evaluation workflows.

The project currently notes that some desktop builds are unsigned, which may trigger operating-system warnings on first launch. Users should download installers only from the official website or GitHub Releases and verify the project instructions before bypassing a warning.

The Team Behind PenguinHarness

PenguinHarness is open sourced by PrismShadow, a team established in 2025 to build agent infrastructure that can learn from business knowledge, workflows, and feedback.

The original report lists the founding team as follows:

Team Member	Background
Yaowei Zheng	Creator of LlamaFactory; focuses on accessible model and agent infrastructure
Buyue Qian	PhD from the University of California, Davis; former IBM T. J. Watson researcher and former Fudan University professor
Xuejun Wu	Former founding member of Baidu NLP; previously held senior roles at Alibaba and JD Digits
Junhao Hu	Peking University doctoral student; contributed to model and cache-efficiency research
Xi Chen	New York University business-school professor; Carnegie Mellon PhD and former Amazon principal scientist

Individual biographies in the source are presented by the publisher and project team. Readers should use official institutional pages for employment or academic verification when those details are material.

From LlamaFactory to PenguinHarness, the team’s stated goal remains consistent: turn complex AI infrastructure into tools that are easier, more reliable, and more efficient for a broader group of users.

常见问题

What is PenguinHarness?

PenguinHarness is an open-source agent harness that can build, run, evaluate, and optimize AI agents. It provides a Web interface, CLI, SDK, built-in Skills, model configuration, tracing, and multi-agent evaluation workflows.

Is PenguinHarness free and open source?

Yes. The core repository is released under the Apache 2.0 license. Users still pay any model API, infrastructure, or third-party service costs generated by their workloads.

Can PenguinHarness run locally?

Yes. It can run on Linux, macOS, and Windows, either as a desktop application or through the CLI and browser interface. It can also be installed on a server for remote use.

Does PenguinHarness support local models?

Yes. It includes Skills and integrations related to tools such as Ollama and vLLM, and it supports custom OpenAI-compatible endpoints. Actual compatibility depends on the model’s API behavior and required capabilities.

What does self-evolution mean in PenguinHarness?

Self-evolution means that the system evaluates an agent, analyzes scores and traces, modifies approved prompts, Skills, or configuration, and tests a candidate next version. The candidate is accepted only when it performs better under the configured evaluation.

Can a self-evolving agent modify the PenguinHarness kernel?

The project contract says no. Evolution is limited to the workspace, prompts, Skills, and approved configuration, while the harness kernel and safety mechanisms remain fixed.

Is the ¥0.2 agent-building cost guaranteed?

No. The ¥0.2 figure comes from one project demonstration that generated a RAG application with DeepSeek V4 Pro. Real costs depend on model pricing, token use, retries, provider, and task complexity.

What is GDPevo?

GDPevo is an open benchmark for measuring agent self-evolution on held-out real-business tasks. Its public V2 release contains 240 tasks across 24 groups with separated training and test tasks.

相关工具

• PenguinHarness: The official website for downloads, documentation, product examples, and installation instructions.
• PenguinHarness GitHub Repository: The Apache 2.0 source code, README, releases, CLI examples, and contribution guide.
• GDPevo: The benchmark data, construction pipeline, evaluation workspaces, and published experiment results.
• LlamaFactory: Yaowei Zheng’s open-source framework for efficient fine-tuning of language and vision-language models.
• vLLM: An open-source engine for high-throughput local and server-side model inference.
• Ollama: A local model runtime that can be used in agent-development workflows.
• OpenRouter: A unified API gateway for accessing multiple hosted model providers.

Related Links

• Introducing PenguinHarness: The project team’s official launch article and explanation of agents building agents.
• PenguinHarness Documentation: Official installation, models, Skills, and usage documentation.
• PenguinHarness GitHub Repository: The source code, current supported-model table, commands, and Apache 2.0 license.
• PenguinHarness Releases: Official desktop and CLI installation packages for supported platforms.
• GDPevo Paper: The research paper describing the benchmark design, rule hybridization, and evaluation findings.
• GDPevo GitHub Repository: The official benchmark, task data, evaluation results, and reproducible workspaces.
• LlamaFactory GitHub Repository: The official repository for the model fine-tuning framework referenced in the article.

Summary

PenguinHarness is an open-source platform that lets an agent build another agent application from a natural-language requirement. It can generate scaffolding, prompts, Skills, configuration, code, a frontend, and run instructions while supporting both hosted and local models.

Its longer-term focus is self-evolution. Multiple Evaluator Agents score a target agent, an Optimizer analyzes the results and execution traces, and a candidate version is accepted only when it achieves a higher score. CONTRACT.md limits what can change and requires snapshots, rollback, hidden rubrics, approvals, credential isolation, and audit records.

The project reports dramatic cost reductions and early production gains, but those figures are team-provided demonstrations and case studies rather than universal guarantees. The Apache 2.0 repository, published commands, and GDPevo benchmark give developers enough material to test the approach on their own workloads.

PenguinHarness’s central idea is straightforward: building an agent should be automatable, but improving it safely requires measurable evaluation, strict boundaries, and reversible changes.