![[Report][NRI Secure Technologies, Ltd.] The Challenge of Building an Agentic SOC That Surpasses Skilled Analysts — Injecting Organizational Knowledge and Control Through a Harness - CODE BLUE 2026 #codeblue_jp #codeblue2026](https://devio2024-media.developers.io/image/upload/f_auto,q_auto/c_limit,w_1200/v1794835744/eyecatch/lwaeb6mt4ljjwzxqdkhi.jpg)
[Report][NRI Secure Technologies, Ltd.] The Challenge of Building an Agentic SOC That Surpasses Skilled Analysts — Injecting Organizational Knowledge and Control Through a Harness - CODE BLUE 2026 #codeblue_jp #codeblue2026
This page has been translated by machine translation. View original
Hello, this is Usuda.
Everyone, are you doing security measures? (Greeting)
This is a report on the following session held at CODE BLUE 2026.
It was a session that generously shared knowledge on the design and construction of a very well-made Agentic SOC!
[NRI Secure Technologies, Ltd.] The Challenge of an Agentic SOC That Surpasses Expert Analysts — Injecting Organizational Knowledge and Governance via a Harness
SOC first-line response has not escaped the structure of "number of alerts × manual human work." To entrust this to autonomous AI agents, it is necessary to give auditable governance to a probabilistically behaving subject and to consistently reproduce the level of judgment of experts. In this talk, regarding "AgenticBlue," the Agentic SOC platform independently developed by NRI Secure that overcame this difficult point and has been put into production operation, we will present the design philosophy of injecting organizational knowledge and governance via a harness, along with the development process so far.
Speakers: Yumehisa Haga
Report
- Originally moved from SOC analyst to R&D
- This time, a talk about the inside of the SOC
- Structural challenges of the SOC
- SOC first-line response involves triage, detailed investigation, severity determination, and notification creation
- It is hard work done by people
- Processing volume is proportional to the number of alerts
- There are problems such as quality varying by person and knowledge being buried
- It cannot be improved just by increasing or training staff
- Operational improvement is needed
- Attacks are getting faster with AI
- AI is now advancing even at the tactics stage
- Alerts increase, so human work cannot keep up
- The defending side also needs to use AI to speed up
- The question is how to make it work
- What is actually happening in NRI Secure Technologies' production SOC
- 40% reduction in response time
- Notifications contain twice the amount of information
- Accuracy is 84%
- Combined with humans, it approaches 100%
- This gap is not a matter of model performance but of how organizational knowledge is incorporated
- For an Agentic SOC to surpass expert analysts
- Produce expert judgment at the same level, with grounds
- Four conditions
- Keep quality constant
- The organization holds permissions and credentials
- Do not let confidential data out
- Be able to trace the reasons afterward
- These are done with the harness
- Built a platform called AgenticBlue
- There is an orchestrator
- Connects to SIEM/EDR and others via MCP
- Judgment knowledge is injected via Skills
- To inject judgment criteria deeply, it is developed in-house rather than using the AI on the side of each integrated solution
- Injecting organizational knowledge
- AI is smart but not knowledgeable
- Some judgment knowledge is not in the LLM's training data
- That is written up as procedures in Skills and tuned
- This raised accuracy to 84%
- Since they are md files, they can be adjusted each time
- The Fat Skills/Thin Harness approach
- Keep the way of running things and the enforcement of safety lightweight
- What happens if you put all the years of materials into the AI?
- The context overflows
- Compaction may occur
- Information in the middle gets dropped
- Context needs to be treated as a budget
- A design that reveals it in stages is needed
- Create narrower, deeper content the lower you go
- Always read, once entering a Phase, once the category is decided, one per detection rule name
- Protect the main agent's context
- Investigation is split into independent agents, so it does not easily get heavy
- The inspector, reporter, and evaluator roles are independent
- Thinking in output
- Analysis has five stages
- Observed facts, evaluation, rationale for judgment, alternative hypotheses, conclusion
- With this, the output does not easily bloat and is easy for humans to follow later
- Processing time was reduced by 30% while maintaining context
- Governance via the harness
- Lines that must not be crossed are managed outside the prompt
- Four places are consolidated into a common platform
- What to hand over
- Tools are permission-based
- A route where the LLM is not trained on the data
- Kept within Japan
- What to require approval for
- Where to stop
- OS commands only via designated scripts
- What to record
- Record reasoning and tool executions so that judgments can be reviewed later
- What to hand over
- Placed on the common platform so that everyone uses the same settings, the same Hooks, and the same version of Skills
- Where to fix things in code
- Can all branches be written out in advance?
- If they can be written out, fix them in code
- Do not have the LLM do standardized aggregation
- Matching against assets is also done by script
- Defanging IoCs too
- Notification destinations too
- If they cannot be written out, use the LLM
- Perspectives for deeper digging
- Perspectives for the next query
- If they can be written out, fix them in code
- Can all branches be written out in advance?
- Quality
- Prepare an evaluator model separate from the one that produced the work
- Send it back if it does not meet the criteria
- Verifying with the same criteria keeps quality constant
- However, there are limits, and errors inevitably occur
- Humans need to watch the AI's behavior closely and evaluate reliability
- Since hundreds of companies' environments are monitored on a single platform, it must be ensured that information from different customers does not get mixed up
- Writing it in guidelines is not enough
- Use Hooks to decide allow/deny outside the LLM
- Unexpected cases are also denied
- Enable users to use multiple agents
- Agents can be separated by route
- Human-on-the-loop agents are prohibited from asking humans for confirmation
- Internal report agents are prohibited from sending external emails
- Considering attacks against the SOC
- Indirect prompt injection
- Injection is possible in alert text, logs, and so on
- Assuming AI agents can be hijacked, what they can do is restricted
- Dual LLM: raw logs are read only by an isolated inspector
- Trace here logs are used to follow up later
- Real operation case
- A large-scale scan of the internal network occurred
- NTLMSSP authentication attempts
- Ultimately RDP RCE
- The agent classified it as the highest urgency category
- It matched the human judgment
- It was detected quickly
- Development story
- The idea of using AI for the SOC itself existed since 2021
- Consideration advanced after the concept of Agent Skills emerged
- The platform layer does not rely on off-the-shelf products; a small team builds it carefully from specifications together with AI agents
- Threat modeling against itself was also included
- The way of building follows the same principles
- The agent layer is grown while being used
- Do not predict everything from the start; thicken things when needed and remove them when no longer needed
- Field analysts understand best, so the environment is set up to make it easy for them to build
- Skill transfer was carried out
- Roadmap
- Want to expand in three directions
- Domain expansion
- Increase what is entrusted
- Autonomy expansion
- Want to gradually entrust even blocking and isolation
- Strengthening governance
- Make it possible to entrust with peace of mind
- Separation per customer, permissions and approvals, etc.
- Summary
- For an Agentic SOC to surpass expert analysts
- Quality, permissions and credentials, not letting confidential data out, being able to trace the reasons afterward
- Design principles
- Organizational knowledge goes into procedures
- Separate generation and evaluation
- Draw the lines with mechanisms
Impressions
I felt it was a very well-made Agentic SOC.
The mechanisms for ensuring and improving quality are solid. I want to imitate it!