Claude Code v2.1.269 Major Updates - Addition of claude plugin eval and fix for missing permission rule application

This is Ishikawa from the Cloud Business Division. Claude Code v2.1.269 (released 2026-09-11) has been released. With 98 changes, this is another release with a high volume of updates following the previous one. The standout characteristic of this release is the concentration of new features and security fixes around plugins. Today, I tried out the claude plugin eval command, which scores and quantifies how effectively plugins (skills) are actually working.

The previous update article is here.

https://dev.classmethod.jp/articles/20260911-cc-updates-v2-1-268/

Update Summary

v2.1.269 includes 98 changes. Counting the CHANGELOG entries: 66 Fixed, 14 Added, 11 Improved, 6 Changed, and 1 Removed. The targets are 58 in the CLI itself, with the remainder being 23 in the VSCode extension, 10 in Claude Tag (Claude in Slack), and 7 in Claude Code on the web.

Notable Updates

Score plugins with claude plugin eval

claude plugin eval has been added. It runs a plugin's eval suite against Claude Code and delivers reproducible, scored results as JSON and HTML reports.

For those distributing plugins or skills internally, this can serve as a mechanism to re-score with the same cases each time an update is made.

Diffs of files changed by Bash commands are included in results

When the Bash tool handles file edits, the diff of files changed by the Bash command is now included in the execution results (setting: bashEditDiffEnabled).

For those using Bash-based edits, being able to verify what was rewritten within the results should reduce the effort of review.

Fixed permission rule application gaps

Two fixes were made around permissions.

• A problem where deny/ask rules beginning with ! were being applied outside the configuration source where they were written. Such rules are now applied only within their own configuration source, and negation using a standalone ! is now ignored.
• A problem where Edit() deny rules and write path checks were not being applied to files written by Bash's tee command. Bash(tee:*) allow rules no longer permit writes outside the working directory.

For teams that enforce operations using deny rules, there may have been paths that were not working as intended until now, so this is worth prioritizing for an upgrade.

Fixed permissions around plugin archive extraction

Three issues were fixed: extracted plugin archives for a session being readable by other users on the same machine, extracted files inheriting world-writable bits from the archive, and old files remaining even after re-extraction.

For those using Claude Code on shared machines or development servers with multiple users, this is a high-priority fix.

Fixed issue where Japanese prompt suggestions were not appearing

An issue was fixed where prompt suggestions were not appearing for Japanese, Chinese, Thai, and other text written without spaces between words. Filtering has also been improved: suggestions with mixed character types or single-word suggestions are retained, while meta-commentary or evaluative text is excluded in the same way as English.

For those writing prompts in Japanese, this fix takes effect directly.

Fixes for session stalling and prompt cache issues

Two fixes were made that affect long sessions.

• A problem where a session could not recover from "Prompt is too long" when there were no completed past exchanges that could be summarized and auto-compaction could not be performed (primarily affecting Agent SDK sessions handling very large prompts).
• A problem where the prompt cache was partially invalidated on the turn following an auto-resume after a response was cut off at the output token limit.

These fixes will benefit those who run long sessions or handle large prompts with the Agent SDK.

Update Details

New Features

• /output-style [name] has been added, enabling listing and switching of output styles. Also works via Remote Control and in cloud and other headless sessions.
• OTEL_METRICS_INCLUDE_REPOSITORY has been added, attaching vcs.* repository attributes to OpenTelemetry metrics and events. When used together with OTEL_LOG_TOOL_DETAILS, vcs.ref.head.* is attached to commit events.
• CLAUDE_CODE_WORKFLOW_MAX_CONCURRENT_AGENTS (1–256) has been added, allowing the maximum number of concurrent agents per Workflow tool execution to be increased.
• CLAUDE_CODE_GATEWAY_MODEL_DISCOVERY_TIMEOUT_MS has been added, allowing the timeout for LLM gateway /v1/models discovery (default 3 seconds) to be extended.
• [VSCode] An agent map has been added, accessible from the "N agents" pill in the footer, showing per-subagent cards, the ability to stop agents, and opening read-only transcripts.
• [VSCode] A Hooks dialog has been added to the command menu, enabling viewing of hooks and adding, editing, and deleting them in user, project, and local settings. Managed, plugin, and session hooks are read-only.
• [VSCode] A permission rules dialog has been added, enabling listing of permission rules and adding and deleting them in user, project, and local settings. Launch option, session-only, and managed rules are read-only.
• [VSCode] Live progress lines for running subagents are displayed below tool call groups in the Focus view.
• [VSCode] A Cancel button has been added to the account switching screen, allowing return to the current session while keeping the current account.
• [Claude Code on the web] In cloud sessions, queued messages can now be retracted before Claude reads them. They can be removed from the queue, or pressing Esc or the up arrow returns the text to the message box.
• [Claude Tag] A confirmation dialog is now shown before Connect all and Disconnect in the GitHub installation in admin settings. This is described as a safeguard against accidentally making organization-wide changes.

Improvements

• Responsiveness in long sessions has improved; the entire conversation is no longer reprocessed to generate summaries for collapsed tool uses on every transcript update.
• Keyboard support over SSH and on unrecognized terminals has improved; Shift+Enter and Ctrl+Shift shortcuts now work on terminals that respond to kitty keyboard queries (foot, Alacritty 0.16 and later, etc.).
• The /diff panel now opens in a rendered state without an intermediate loading state.
• Spinner hints now include guidance for /focus, which switches the display to show only the prompt, a one-line work summary, and the response.
• The "Unknown skill" error for the Skill tool now shows the full name when the name without the plugin prefix matches exactly one plugin skill.
• /ultrareview --post now directly posts a PR comment when results are ready and shows the comment link, without starting a second cloud session for posting.
• Database reads for Artifacts saved to the session scratchpad no longer block on working folder approval.
• For first-party sessions with telemetry disabled, alwaysLoad MCP servers that finish connecting mid-conversation are now available from the next turn without a tool discovery round trip.
• The name for skills synced from claude.ai in cloud sessions has changed to anthropic-skills:<name>, aligning with Claude Desktop. If no other skill shares the same name, it can still be referenced without the prefix.
• [VSCode] For documents or messages written for an audience other than the user, Claude now writes text directed at that audience and explicitly identifies the audience at the beginning of the reply.
• [VSCode] Screen reader and keyboard accessibility has been improved for slash command menus, @ mention menus, output style selection, send/stop buttons, permission and question cards, and the onboarding checklist.
• [VSCode] Claude Code items have been removed from the right-click menu on session tabs and the "..." menu in the editor title bar, as they could not act on the tab that was right-clicked.
• [Claude Tag] Scheduled routines in Slack channels can now reply to existing threads instead of always posting a new top-level message.
• [Claude Tag] Loading times for the admin settings page and Slack channel selection have improved, with the greatest benefit for organizations with many channels or multiple connected workspaces.

Fixes

The major fixes not covered in the Notable Updates section are as follows.

• Fixed terminal key input regressions: Issues were fixed where F1, F2, and F4 did not work on kitty protocol-compatible terminals, Delete did not work in st, Alt+Arrow was treated as Escape in rxvt-unicode, and Shift+Symbol was input as the unshifted character in WezTerm (regression from v2.1.247).
• Fixed reduced cache reuse on session resume: After interrupting Claude mid-thought and resuming, the way previous context was resent changed, sometimes reducing prompt cache reuse rates.
• Fixed CLAUDE.md attribution rules being overridden: Attribution reminders were overriding rules in CLAUDE.md and memory that prohibited attribution in commits and pull requests. Lines specified in managed settings continue to be applied.
• Fixed missing permission_denials records: Read, Edit, and Write calls blocked by path-scope deny rules were missing from permission_denials in --output-format stream-json results.
• [Claude Tag] Fixed incorrect sharing scope display: The sharing banner and sharing dialog for sessions started from Slack were indicating that the link could be opened organization-wide. They now show the Slack channel scope.
• [Claude Tag] Fixed responses on unauthorized models: Switching to a model not enabled by the organization was being accepted and silently responding with a fallback model. The switch is now refused and users are informed that an administrator can enable it.
• Fixed LSP servers persisting: Plugin LSP servers that do not accept shutdown parameters (such as rust-analyzer) were continuing to run after a session ended. exit is now sent even if shutdown fails.
• Fixed /insights failures: It was failing in Bedrock, Vertex, Foundry, and gateway configurations for accounts that cannot reach the default Opus model. These environments now use the session model.
• Fixed /goal silently stopping: Execution was silently stopping after API errors, network disconnections, or token limit hits. It now retries with backoff or pauses with an explanation including waiting for usage limit resets.
• Fixed /btw hallucinated tool calls: Responses were sometimes including non-existent tool calls and their outputs. Claude is now instructed not to include them in side questions, and when they appear, they are indicated as not having been executed.
• Fixed organization plugins not loading: Organization plugins enabled in managed settings were not loading in headless sessions and Claude Desktop (since this version of the CLI was bundled). They will now load from the next session.
• Fixed PowerShell background commands stopping on Windows: PowerShell tool commands sent to the background were being stopped when Claude Code exited.
• [Claude Code on the web] Fixed duplicate execution of scheduled routines: A problem where one-time routines were executed twice after a temporary server error, and a problem where executions using subagents were treated as finishing early, causing retries to be skipped or duplicate executions to start, have been fixed.
• A subtly welcome fix: On slow connections (ssh, browser-based terminals), an issue was fixed where strings like 22c and terminal color/version responses were being input into the prompt at startup. As the kind of bug that was noticeable every time you connected over ssh, this is a fix that will quietly make a difference.
• In addition, numerous minor bugs have been fixed, including full-screen redraws, returning from external editors, MCP server reconnections, VSCode extension session lists, and Claude Tag notifications.

Trying out claude plugin eval

claude plugin eval is a command that scores how effectively a plugin (skill) is working by running test cases and turning the results into numbers.

https://x.com/ClaudeDevs/status/2098501001447870499

claude plugin eval scores how well a plugin (skill) is working using test cases. It runs the same cases both with and without the plugin, and the difference (Δ) represents the plugin's contribution. If scores are high in both cases, it means those cases would have passed even without the plugin.

Terminology

Since the terminology can be confusing, here is a summary. These are the terms that appear when creating and running cases.

Terms related to suite composition

Terms related to execution

Step 0: Preparation for verification

This appears when run in an empty directory or project root. This time, we create the plugin root with plugin.json (or .claude-plugin/plugin.json) and a skill folder containing SKILL.md.

Project root
├── .claude-plugin
│   └── plugin.json
└── skills
    └── commit-message
        └── SKILL.md

Running the script below creates the verification skill for this walkthrough in one shot.

mkdir -p plugin-eval && cd plugin-eval
mkdir -p .claude-plugin skills/commit-message
cat > .claude-plugin/plugin.json <<'JSON'
{
  "name": "commit-style",
  "version": "0.1.0",
  "description": "Draft commit messages in Conventional Commits format"
}
JSON
cat > skills/commit-message/SKILL.md <<'MD'
---
name: commit-message
description: Use when the user asks for a commit message for a change they describe.
---

# Commit message

Receives a description of a change and returns a single line commit message in Conventional Commits format.

- Format is `<type>(<scope>): <subject>`
- type is chosen from feat / fix / docs / refactor / test / chore
- subject is in English imperative form, within 72 characters, no trailing period
- Return only the single commit message line with no explanation, preamble, or body
MD

If you want to evaluate an existing skill you are already using, move to ~/.claude/skills/<skill-name>/ and run the same command (it will be resolved as a plugin in the skills directory).

Step 2: Creating cases

Run this in the plugin root (the plugin-eval directory you cd'd into in Step 1).

claude plugin eval init

Running the command opens an interactive session where Claude asks questions and you provide answers. Claude writes the cases and graders.

The process proceeds through several confirmation steps (gates), so answer the questions to move forward. Claude drafts cases and graders, runs a pilot to verify behavior, and writes them out to evals/<case-name>/. The estimated cost for a full run is also presented at that point.

Step 3: Running it

The default is 3 runs with the plugin and 3 runs without, for a total of 6 runs per case. If you just want to confirm it runs, use --runs 1 to reduce it to 1 run each (since a single run can vary, return to the default of 3 when making quality judgments).

The full run completed and the suite is finished. The init dialogue ends here. Add the final step (fix weaknesses and re-measure) to the procedure.

Exit the session with /exit and return to the shell. Under evals/plugin, 7 cases, FOLLOWUPS.md, and 3 generations of results/ remain.

plugin
├── 01-ja-prose
│   ├── graders
│   │   ├── conventional-type-prefix.md
│   │   ├── no-japanese.md
│   │   ├── no-wrapper.md
│   │   ├── one-line-only.md
│   │   ├── skill-fired.md
│   │   └── subject-quality.md
│   └── prompt.md
├── 02-ja-bullets
│   ├── graders
│   │   ├── conventional-type-prefix.md
│   │   ├── no-japanese.md
│   │   ├── no-wrapper.md
│   │   ├── one-line-only.md
│   │   ├── skill-fired.md
│   │   └── subject-quality.md
│   └── prompt.md
├── 03-diff-paste
│   ├── graders
│   │   ├── conventional-type-prefix.md
│   │   ├── no-japanese.md
│   │   ├── no-wrapper.md
│   │   ├── one-line-only.md
│   │   ├── skill-fired.md
│   │   └── subject-quality.md
│   └── prompt.md
├── 04-en-oneline
│   ├── graders
│   │   ├── conventional-type-prefix.md
│   │   ├── no-japanese.md
│   │   ├── no-wrapper.md
│   │   ├── one-line-only.md
│   │   ├── skill-fired.md
│   │   └── subject-quality.md
│   └── prompt.md
├── 05-ja-casual
│   ├── graders
│   │   ├── conventional-type-prefix.md
│   │   ├── no-japanese.md
│   │   ├── no-wrapper.md
│   │   ├── one-line-only.md
│   │   ├── skill-fired.md
│   │   └── subject-quality.md
│   └── prompt.md
├── 06-neg-explain
│   ├── graders
│   │   ├── answers-the-question.md
│   │   ├── not-a-bare-commit.md
│   │   └── skill-not-used.md
│   └── prompt.md
├── 07-neg-typelist
│   ├── graders
│   │   ├── answers-the-question.md
│   │   ├── not-a-bare-commit.md
│   │   └── skill-not-used.md
│   └── prompt.md
├── FOLLOWUPS.md
└── results
    ├── 2026-09-12T02-55-47-422Z
    │   ├── aggregate-result.json
    │   └── report.html
    ├── 2026-09-12T03-58-41-015Z
    │   ├── aggregate-result.json
    │   └── report.html
    ├── 2026-09-12T04-04-49-220Z
    │   ├── aggregate-result.json
    │   └── report.html
    └── 2026-09-12T04-29-53-821Z
        ├── aggregate-result.json
        └── report.html

Open the HTML report at evals/results/2026-09-12T04-29-53-821Z/report.html. It contains per-case cards, per-run grader judgments, judge votes, and the full text the judge reviewed. It has more information than the terminal summary, so it is worth checking.

Finally, address the items recorded in FOLLOWUPS.md. Fix the weaknesses identified at the end and re-measure the effect as needed.

Thoughts on claude plugin eval

Until now, writing SKILL.md and CLAUDE.md was a "write it and see how it goes" process, where quality depended on the author's intuition. Eval brings testing and baseline comparison to this process, and I see it as a tool that applies a testing mindset to what was previously a guesswork-driven activity. The comparison against a baseline without the plugin in particular is a concept not found in unit testing—it is closer to the thinking behind A/B testing.

The biggest takeaway from running through the whole thing was seeing in numbers what should be obvious: what you wrote is not guaranteed to be followed, and what you did not write is certainly not followed. Writing "return only one line" does not prevent fences from appearing. Conversely, there are things the base Claude can do even without writing them. Being able to separate these two is, I believe, the value of eval.

On the other hand, the upfront effort for skill development increases. What used to be "just write it and distribute it" now involves case design, grader design, and pilot iteration. I think this overhead is only worthwhile for things you intend to distribute. If you are the only one using a skill, being the only one affected by it not working is a minor concern. But if a skill distributed internally turns out to do nothing, it keeps consuming everyone's tokens. Being able to verify that with numbers is where I see the practical significance.

The impression is that the barrier to quickly validating a Skill/Plugin is fairly high. If you want to try it, please refer to the following documentation.

https://code.claude.com/docs/en/plugin-evals

Finally

On the plugin front, a new feature in the form of an evaluation framework and a security fix for permissions on extracted archives landed at the same time. For those distributing plugins internally, I would recommend incorporating the permissions fix before setting up evaluations.

For those enforcing operations with permission rules or using Claude Code on shared machines, consider updating and giving it a try.

References

https://github.com/anthropics/claude-code/blob/main/CHANGELOG.md

https://code.claude.com/docs/en/changelog

https://dev.classmethod.jp/articles/20260911-cc-updates-v2-1-268/