I Keep Hermes Agent's Self-Improvement OFF For the First 14 Days — Here's What Happens When I Don't

What I Do During the 14 Days

Day 1–3: Capture the Baseline

The first three days are observation only. I run all the workflows I intend to automate through Hermes, treating the agent as a capable but un-optimized assistant. I take notes on what I am actually asking the agent to do, not what I thought I would ask it to do when I planned the deployment. These are always different.

The observation log lives in a markdown file in my project directory, not in Hermes’s memory system. I want a human-readable record I can query without routing through the agent:

# hermes-observation-log.md

## Day 1 — 2026-05-01

### Workflows run:
- Deploy storefront to production (3x) — agent handled 2/3 correctly, 1 required manual intervention on Redis healthcheck
- Generate and publish blog post (2x) — worked correctly both times, agent remembered WordPress API format from session 1 to session 2
- Sync WooCommerce products from staging (1x) — FAILED: agent used wrong API version, needed correction

### Patterns noticed:
- Agent repeatedly asks clarifying question about git branch before deploys — this should be a skill default
- Agent does not know our VPS SSH key path — has to be prompted each session
- Agent remembers API format within a session but not across sessions (memory not long enough?)

### Skills that would have helped:
- skill/deploy-production: branch=main, ssh-key=/root/.ssh/deploy_key, healthcheck=redis+nextjs
- skill/wordpress-publish: api-version=v2, base-url=wordpress-app:80, auth=env:WP_APP_PASSWORD

## Day 2 — 2026-05-02

### Workflows run:
...

Three days of this log gives you something valuable: the real list of skills your deployment actually needs, derived from observation rather than upfront planning. The list is always shorter than you expected and more specific than any default skill set would produce.

Day 4–10: Hand-Author the First Generation

With three days of observation data, I write skills manually. Every skill I write follows a fixed template with YAML frontmatter, a procedure section, a pitfalls section, and a verification block. The verification block is non-negotiable — it is what makes skill quality objectively measurable rather than subjective:

---
skill_id: deploy-production
version: 1.0.0
author: anup
created: 2026-05-04
pinned: false
no_auto_refine: false
tags: [deploy, production, vps, docker]
description: >
  Deploys the storefront to the production VPS via SSH.
  Handles Redis healthcheck, Docker Compose restart, and
  liveness verification.
---

# skill: deploy-production

## Purpose

Deploy the storefront Next.js application to the production VPS.
Use this skill whenever the user asks to deploy, push to production,
or ship changes live.

## Prerequisites

- SSH key at /root/.ssh/deploy_key must exist
- Environment variable VPS_HOST must be set
- User must confirm they are on the correct git branch before proceeding

## Procedure

1. Confirm git branch:

   git branch --show-current

   Expected: "main". If not "main", STOP and ask user to confirm.

2. Confirm working tree is clean:

   git status --short

   Expected: empty output. If not empty, STOP and ask user whether to stash or commit.

3. Push to remote:

   git push origin main

4. Wait for GitHub Actions to complete (poll gh run list until conclusion=success):

   gh run list --branch main --limit 1 --json conclusion,status

   Poll every 30 seconds. Timeout after 10 minutes.

5. Verify liveness after deploy:

   curl -s -o /dev/null -w "%{http_code}" https://wowhow.cloud/

   Expected: 200. If not 200, trigger rollback procedure.

## Pitfalls

- Do NOT deploy if Redis healthcheck fails before deploy — the deploy will not fix it
- Do NOT interpret a 301 redirect as a success — check for 200 from the final URL
- GitHub Actions concurrency means only one deploy runs at a time — do not push again while a run is active
- If conclusion=cancelled, the next queued push will deploy — wait, do not push again

## Verification

After running this skill, the following must all be true:

- curl https://wowhow.cloud/ returns HTTP 200
- gh run list --limit 1 shows conclusion=success
- No active deploy lock at /tmp/wowhow-deploy.lock on VPS

## Rollback

If verification fails, run: skill/rollback-production

This template takes longer to write than an auto-generated skill. That is the point. The effort of writing it forces you to think through edge cases, failure modes, and verification criteria that auto-generation skips. The resulting skill is dramatically more reliable in production.

During days 4–10, I typically write between five and twelve skills, covering the core workflows identified in the observation log. Each skill goes into /home/hermes/skills/hand-authored/ where it is accessible to the agent but protected from modification by the read-only mount in the config.

Day 11–14: Build Rollback Infrastructure

Before enabling self-improvement, I need to be able to revert to a known-good state without manual effort. The rollback infrastructure has three components: git tracking of the skills directory, a wrapper script that enforces the git commit on every skill write, and a restore command.

Initialize git in the skills directory:

# Initialize git tracking for the skills directory
cd /home/hermes/skills
git init
git add hand-authored/ pinned/
git commit -m "day-14 baseline: hand-authored skills before enabling auto_create"

# Tag the baseline explicitly
git tag baseline-day14

The wrapper script is the key piece. When I enable auto-creation on day 15, I want every auto-generated skill to be committed to git immediately when Hermes writes it. This gives me a complete audit trail and a one-command rollback path. Here is the full wrapper script I use:

#!/usr/bin/env bash
# skill-write-with-audit.sh
# Wrap every skill write operation with a git commit.
# Usage: skill-write-with-audit.sh <skill_path> <skill_content_file>
#
# Hermes Agent calls this script instead of writing skill files directly.
# Configure in config.yaml: skills.write_hook: "/home/hermes/bin/skill-write-with-audit.sh"

set -euo pipefail

SKILL_PATH="${1}"
CONTENT_FILE="${2}"
SKILLS_ROOT="/home/hermes/skills"
LOG_FILE="/home/hermes/logs/skill-audit.log"
TIMESTAMP=$(date -u +"%Y-%m-%dT%H:%M:%SZ")

# Validate inputs
if [[ -z "${SKILL_PATH}" || -z "${CONTENT_FILE}" ]]; then
  echo "[${TIMESTAMP}] ERROR: Missing arguments. Usage: $0 <skill_path> <content_file>" | tee -a "${LOG_FILE}"
  exit 1
fi

# Resolve absolute path
ABS_SKILL_PATH="${SKILLS_ROOT}/${SKILL_PATH}"

# Ensure the target directory exists
mkdir -p "$(dirname "${ABS_SKILL_PATH}")"

# Check if this is an update to an existing skill
IS_UPDATE=false
if [[ -f "${ABS_SKILL_PATH}" ]]; then
  IS_UPDATE=true
  # Capture the pre-modification state for the log
  PREV_VERSION=$(grep -m1 "^version:" "${ABS_SKILL_PATH}" || echo "unknown")
fi

# Write the skill file
cp "${CONTENT_FILE}" "${ABS_SKILL_PATH}"

# Extract metadata from the written file
SKILL_ID=$(grep -m1 "^skill_id:" "${ABS_SKILL_PATH}" | awk '{print $2}' || echo "unknown")
NEW_VERSION=$(grep -m1 "^version:" "${ABS_SKILL_PATH}" | awk '{print $2}' || echo "unknown")

# Commit to git
cd "${SKILLS_ROOT}"

if [[ "${IS_UPDATE}" == "true" ]]; then
  COMMIT_MSG="auto-refine: ${SKILL_ID} ${PREV_VERSION} to ${NEW_VERSION} [${TIMESTAMP}]"
else
  COMMIT_MSG="auto-create: ${SKILL_ID} v${NEW_VERSION} [${TIMESTAMP}]"
fi

git add "${ABS_SKILL_PATH}"
git commit -m "${COMMIT_MSG}"
COMMIT_HASH=$(git rev-parse --short HEAD)

# Log the operation
echo "[${TIMESTAMP}] skill_write | id=${SKILL_ID} | path=${SKILL_PATH} | version=${NEW_VERSION} | update=${IS_UPDATE} | commit=${COMMIT_HASH}" | tee -a "${LOG_FILE}"

echo "OK: ${SKILL_PATH} written and committed as ${COMMIT_HASH}"

Make it executable and configure Hermes to use it:

chmod +x /home/hermes/bin/skill-write-with-audit.sh

# Add to config.yaml under the skills section:
# skills:
#   write_hook: "/home/hermes/bin/skill-write-with-audit.sh"

With this hook in place, every auto-generated skill is a git commit. Rolling back to the day-14 baseline is a single command:

# Roll back all auto-generated skills to the day-14 baseline
cd /home/hermes/skills
git checkout baseline-day14 -- managed/

# Verify the rollback
git log --oneline -5
git diff baseline-day14 HEAD -- managed/ | head -50

End of Day 14: The Activation Ritual

Before flipping the switch, I run through a fixed checklist. Skipping any item means waiting another 24 hours:

# Hermes Agent Day-14 Activation Checklist
# Complete ALL items before enabling auto_create / auto_refine

- [ ] Observation log has at least 3 days of workflow data
- [ ] At least 5 hand-authored skills written and tested manually
- [ ] Each hand-authored skill has a verification block with specific, testable criteria
- [ ] Skills directory is a git repo with baseline-day14 tag
- [ ] skill-write-with-audit.sh is executable and tested (run it manually once)
- [ ] Rollback command tested: git checkout baseline-day14 -- managed/ runs clean
- [ ] Audit log path is writable: touch /home/hermes/logs/skill-audit.log succeeds
- [ ] Config backup taken: cp config.yaml config.yaml.day14-backup
- [ ] Team notified that self-improvement is being enabled (if applicable)

When all items are checked, I update the config to enable auto-creation with conservative guardrails:

# config.yaml — Day 15+ configuration
# Self-improvement enabled with production guardrails

skills:
  auto_create: true
  auto_refine: true

  # Conservative creation threshold — agent must observe the same
  # pattern at least 5 times before creating a skill for it
  auto_create_threshold: 5

  # Refinement requires a meaningful quality delta, not marginal improvement
  auto_refine_min_improvement: 0.15

  # Maximum skills auto-created per day — prevents accumulation explosions
  auto_create_daily_limit: 2

  # Write hook — every skill write goes through the audit wrapper
  write_hook: "/home/hermes/bin/skill-write-with-audit.sh"

  # Hand-authored skills remain read-only even now
  external_dirs:
    - path: "/home/hermes/skills/hand-authored"
      mode: "read-only"
    - path: "/home/hermes/skills/pinned"
      mode: "read-only"

  managed_dir: "/home/hermes/skills/managed"
  managed_dir_writable: true

The auto_create_threshold: 5 and auto_create_daily_limit: 2 are the two guardrails I consider non-negotiable even after enabling self-improvement. The threshold prevents the agent from creating skills based on one-off requests. The daily limit means that even in the worst case, you have at most two new things to audit per day rather than four or more.

Pinning Critical Skills

Some skills must never be deleted, archived, or modified by the self-improvement loop, regardless of how the agent evaluates them. Production deployments, security-related workflows, and the audit wrapper itself all belong in this category. Hermes has a pin mechanism for exactly this purpose:

# Pin a skill so the self-improvement loop never touches it
hermes curator pin skill/deploy-production
hermes curator pin skill/rollback-production
hermes curator pin skill/security-review-checklist

# Verify pinned skills
hermes curator list --pinned

# Output:
# PINNED SKILLS
# -------------
# skill/deploy-production      v1.0.0   pinned: 2026-05-15
# skill/rollback-production    v1.0.0   pinned: 2026-05-15
# skill/security-review-checklist  v1.2.0   pinned: 2026-05-15

Pinned skills also live in /home/hermes/skills/pinned/ which is mounted read-only even in the day-15+ configuration above. The git tracking still applies — if you manually update a pinned skill, the audit wrapper logs the change — but the self-improvement loop cannot touch them.

The YAML frontmatter for a pinned skill looks like this:

---
skill_id: deploy-production
version: 1.0.0
author: anup
created: 2026-05-04
pinned: true
no_auto_refine: true
tags: [deploy, production, pinned]
---

Setting both pinned: true and no_auto_refine: true in the frontmatter creates a belt-and-suspenders guarantee. The pinned flag is enforced by the curator. The no_auto_refine flag is enforced by the self-improvement loop itself. Either one alone is sufficient; both together ensure the skill survives any config change or curator state reset.

Four Workflows That Should Never Be on Auto

Even after day 14, there are categories of workflow where I permanently keep auto-refinement disabled. These are set with the no_auto_refine frontmatter flag, not by the global config, so they stay protected regardless of what the global auto_refine setting is.

1. Production Deployments

Any skill that touches the production deployment pipeline gets no_auto_refine: true and pinned: true. The reasoning is simple: a deployment skill that has been tested and is working correctly should not be modified by an automated process. The cost of a bad deployment is high enough that human review of every change to the deploy skill is worth the overhead.

2. Security Reviews and Audit Checklists

Security review skills encode decisions made with deliberate human judgment about what constitutes an acceptable risk on this specific workload. An automated refinement process does not have the context to make those judgments. More importantly, a subtly weakened security review skill — one that drops a check that seems redundant based on recent session patterns — is exactly the kind of silent degradation that causes real incidents.

3. Customer-Facing Workflows

Any skill that generates content, sends communications, or takes actions visible to customers gets no_auto_refine: true. The agent’s self-evaluation of output quality is based on patterns in past sessions. Customer-facing quality has dimensions the agent cannot observe: tone consistency with your brand, regulatory compliance, and the implicit expectations of your specific customer base. Automated refinement optimizes for what the agent can measure, which is not the same thing as customer quality.

4. The Audit Wrapper Itself

The skill-write-with-audit.sh wrapper must never be modified by any automated process. This is the ouroboros risk: if the self-improvement loop can modify the mechanism by which skill modifications are logged, the audit trail becomes unreliable. The wrapper lives outside the skills directory entirely, owned by root, and is never referenced in any Hermes skill file as a dependency.

The config block that enforces these permanent exceptions at the skill level rather than globally:

# These skills have auto-refinement permanently disabled
# regardless of the global auto_refine setting.
# Set in each skill's YAML frontmatter:

# skill/deploy-production
no_auto_refine: true
pinned: true

# skill/rollback-production
no_auto_refine: true
pinned: true

# skill/security-review-checklist
no_auto_refine: true
pinned: true

# skill/customer-email-template
no_auto_refine: true

# skill/wordpress-publish
no_auto_refine: true

The Honest Other Side

It is fair to push back on this approach. If you are running Hermes on a low-stakes workload — personal productivity, research summarization, drafting non-critical content — the 14-day observation period is probably unnecessary friction. The default self-improvement settings will produce a reasonably good agent within a week, and the cost of a bad auto-generated skill is low enough that you can simply delete it when you notice the problem.

The rule is specifically for production workloads, customer-facing systems, and workflows where a subtle degradation in agent behavior costs real money or damages real relationships before you catch it. If your Hermes deployment does not touch any of those categories, you can safely ignore this article and turn on self-improvement on day one.

I also want to be honest about the overhead. The 14-day observation period, the hand-authored skills, the git infrastructure, and the rollback tooling together represent roughly eight to twelve hours of work that a default deployment skips. That is a real cost. It pays back on production workloads because the alternative — debugging a degraded agent while production workflows are impaired — costs more. On a side project, the math probably does not work out in the same direction.

The clearest decision heuristic I have found: if you would run a staging environment and a rollback procedure for code changes to this workload, run them for your agent skills too. If you would not, default settings are probably fine.

Closing

Self-improvement is too powerful to enable on a workload you do not yet understand. The first 14 days of any new Hermes deployment are a learning period for you, not just for the agent. You are learning what the agent actually does on your real tasks, what the friction points are, where the agent excels without any skills at all, and where the gaps are that hand-authored skills can fill. Blocking self-improvement during that period is not about distrust of the agent. It is about having the data you need to make the self-improvement loop produce good outcomes.

When I enable self-improvement on day 15 with the conservative guardrails in place, the compounding still happens. It just compounds toward my real workflow rather than toward my first-week confusion. The difference in output quality after 30 more days is significant. The skills the agent generates are coherent with the hand-authored baseline, targeted at actual patterns rather than exploratory noise, and limited in accumulation rate so the audit burden stays manageable.

The evening I spent reverting eleven bad auto-generated skills from my last fresh deployment convinced me to never skip the observation period again. It is a better use of that time than debugging a degraded agent on a live workload.

If you found this useful, the two Hermes Agent articles I referenced at the start cover the positive case — what self-improvement looks like when it is working correctly, and the architecture of the skill system that makes it possible. This article is the prerequisite, not the replacement.