The story of how assigning Jev the task of selecting automation scripts for routine work got it done in 52% of the time Codex would have taken
This page has been translated by machine translation. View original
Introduction
There are tasks assigned to Codex that follow the same steps every time. These steps can be prepared in advance as Python scripts or shell scripts. (In this article, these are called scripts.) Since Codex users request the same tasks in various ways, the judgment of selecting the script that matches the meaning of the request remains. If this small judgment is also left to Codex, even simple tasks require waiting for a GPT response each time.
Therefore, Jev, which answers from predefined choices, was placed upstream of Codex. For verification, 120 independent scripts were created, and the Codex-only approach was compared with the combined Jev and Codex approach.
In the verification, both approaches correctly selected the right process for all 270 cases. With the combined Jev and Codex approach, Codex calls were reduced from 270 to 90. Execution time was reduced from 21 minutes 38 seconds to 11 minutes 13 seconds, and cost was reduced from approximately 4.9 USD to approximately 1.8 USD.
What is Jev
Jev is a TypeSafe model that selects answers from predefined candidates. When given free-form text and the information needed for execution, it returns the selected candidate, the probability for each candidate, and the confidence level. In this case, 120 scripts and a handoff to Codex were provided as candidates, and it was used to select the execution target.
Target Audience
- Those who want to know specific use cases for Jev
- Those who have Codex use a large number of scripts
- Those who want to reduce the number of GPT calls and waiting time
Verification Environment
- Verification date: September 23, 2026
- Jev:
jev-1.13.0 - Codex:
gpt-5.6-sol, reasoning intensitylow
References
Configuration
Verification Scripts
The verification scripts were created in 8 fields, 15 scripts each, covering areas such as work queue updates and document inspection. The 120 scripts are each separate Python files that can be executed independently. Note that Jev does not read Python files directly; it reads the JSON that is sent to it. The JSON summarizes the processing each script performs and the conditions for use.
Among the 120 Python files, 3 with different roles are introduced. The first updates the work queue, the second creates a software bill of materials, and the third reads service status.
Script that updates work queue status
import argparse
from common import ok, read_csv, run, workspace_root, write_csv, Rejected
def main() -> int:
parser = argparse.ArgumentParser()
parser.add_argument("--root", required=True)
parser.add_argument("--queue", required=True)
args = parser.parse_args()
root = workspace_root(args.root)
rows, fields = read_csv(root, "queues/work_items.csv")
item = next((row for row in rows if row["queue"] == args.queue and row["status"] == "ready" and not row["blocked_by"]), None)
if item is None:
raise Rejected("no_ready_item")
item["status"] = "in_progress"
item["owner"] = "worker-1"
write_csv(root, "queues/work_items.csv", rows, fields)
return ok(item_id=item["id"], new_status=item["status"])
run(main)
Script that creates a software bill of materials from dependencies
import argparse
from common import Rejected, ok, read_json, run, workspace_root, write_json
def main() -> int:
parser = argparse.ArgumentParser(description='Generate a software bill of materials from dependencies.')
parser.add_argument("--root", required=True)
parser.add_argument("--target", required=True)
parser.add_argument("--output", required=True)
args = parser.parse_args()
root = workspace_root(args.root)
target = args.target
records = read_json(root, 'states/build-build_sbom.json')["records"]
record = records.get(target)
if record is None:
raise Rejected("target_not_registered")
if not record.get("ready"):
raise Rejected("build_sbom_source_not_ready")
write_json(root, args.output, {"source": target, "operation": "build_sbom", "items": record["items"]})
return ok(target=target, output=args.output, items=len(record["items"]))
run(main)
Script that retrieves service throughput and latency
import argparse
from common import Rejected, ok, read_json, run, workspace_root, write_json
def main() -> int:
parser = argparse.ArgumentParser(description='Retrieve throughput and latency time series for a specified service.')
parser.add_argument("--root", required=True)
parser.add_argument("--service", required=True)
args = parser.parse_args()
root = workspace_root(args.root)
target = args.service
records = read_json(root, 'states/diagnostic-collect_metrics.json')["records"]
record = records.get(target)
if record is None:
raise Rejected("target_not_registered")
if not record.get("items"):
raise Rejected("collect_metrics_data_unavailable")
return ok(target=target, items=record["items"])
run(main)
In contrast, the JSON that Jev reads takes the following form. requires is the condition under which the script can be used, and reject_when is the condition under which it cannot be used.
Information for the 3 scripts sent to Jev
{
"c001": {
"arguments": "queue",
"purpose": "Change the next available item in the specified work queue to in-progress.",
"reject_when": "There are no available items.",
"requires": "There is an item that has not started and has no dependencies."
},
"c080": {
"arguments": "target, output",
"purpose": "Generate a software bill of materials from dependencies.",
"reject_when": "The target is not registered, or the specific prerequisites are not met.",
"requires": "The target indicated by target is registered, and the state required to generate a software bill of materials from dependencies is in place."
},
"c098": {
"arguments": "service",
"purpose": "Retrieve throughput and latency time series for the specified service.",
"reject_when": "The target is not registered, or the specific prerequisites are not met.",
"requires": "The target indicated by service is registered, and the state required to retrieve throughput and latency time series for the specified service is in place."
}
}
Requests Tested
The three types of requests used for verification are as follows.
- Requests that directly express the purpose of a script: 90 cases
- Requests that express the same purpose in different words: 90 cases
- Requests where it is appropriate to hand off processing to Codex: 90 cases
Of the 180 cases for selecting scripts, 150 had multiple scripts that accept the same item as input. For example, even when receiving the same release version, there is a script that creates a phased rollout plan and a script that creates a recovery plan for incidents. Therefore, selecting the correct script requires reading the meaning of the requested task in addition to the input items.
Requests that directly express the purpose of a script
{
"request": "Please generate a plan that includes recovery procedures and decision criteria for release failures.",
"work_state": {
"arguments": {
"output": "outputs/requested-result.json",
"version": "2.0.0"
},
"domain": "release",
"execution_guard": {
"arguments_resolved": true,
"operation_state_eligible": true,
"output_path_allowed": true,
"referenced_inputs_readable": true,
"target_registered": true
}
}
}
Requests that express the same purpose in different words
{
"request": "Please create a plan that includes recovery procedures and decision criteria for release failures.",
"work_state": {
"arguments": {
"output": "outputs/requested-result.json",
"version": "2.0.0"
},
"domain": "release",
"execution_guard": {
"arguments_resolved": true,
"operation_state_eligible": true,
"output_path_allowed": true,
"referenced_inputs_readable": true,
"target_registered": true
},
"recent_event": "Target preparation completed"
}
}
Requests where it is appropriate to hand off processing to Codex
{
"request": "Please get the release preparation in good shape.",
"work_state": {
"arguments": {},
"domain": "release",
"request_scope": "not_fixed"
}
}
Comparison Method
The choices given to Jev were execution of one of the 120 scripts (c001~c120), or handoff to Codex (codex_required).
TypeSafe's Choice selects one from predefined choices and returns the probability for each choice and the confidence level.
Jev's response
{
"model": "jev-1.13.0",
"answers": {
"routing": {
"type": "choice",
"choice": "c001",
"probabilities": {
"c001": 0.99,
"c002": 0.0,
"c003": 0.0,
"c004": 0.0,
"c005": 0.0,
"c006": 0.0,
"c007": 0.0,
"c008": 0.0,
"c009": 0.0,
"c010": 0.0,
"c011": 0.0,
"c012": 0.0,
"c013": 0.0,
"c014": 0.0,
"c015": 0.0,
"c016": 0.0,
"c017": 0.0,
"c018": 0.0,
"c019": 0.0,
"c020": 0.0,
"c021": 0.0,
"c022": 0.0,
"c023": 0.0,
"c024": 0.0,
"c025": 0.0,
"c026": 0.0,
"c027": 0.0,
"c028": 0.0,
"c029": 0.0,
"c030": 0.0,
"c031": 0.0,
"c032": 0.0,
"c033": 0.0,
"c034": 0.0,
"c035": 0.0,
"c036": 0.0,
"c037": 0.0,
"c038": 0.0,
"c039": 0.0,
"c040": 0.0,
"c041": 0.0,
"c042": 0.0,
"c043": 0.0,
"c044": 0.0,
"c045": 0.0,
"c046": 0.0,
"c047": 0.0,
"c048": 0.0,
"c049": 0.0,
"c050": 0.0,
"c051": 0.0,
"c052": 0.0,
"c053": 0.0,
"c054": 0.0,
"c055": 0.0,
"c056": 0.0,
"c057": 0.0,
"c058": 0.0,
"c059": 0.0,
"c060": 0.0,
"c061": 0.0,
"c062": 0.0,
"c063": 0.0,
"c064": 0.0,
"c065": 0.0,
"c066": 0.0,
"c067": 0.0,
"c068": 0.0,
"c069": 0.0,
"c070": 0.0,
"c071": 0.0,
"c072": 0.0,
"c073": 0.0,
"c074": 0.0,
"c075": 0.0,
"c076": 0.0,
"c077": 0.0,
"c078": 0.0,
"c079": 0.0,
"c080": 0.0,
"c081": 0.0,
"c082": 0.0,
"c083": 0.0,
"c084": 0.0,
"c085": 0.0,
"c086": 0.0,
"c087": 0.0,
"c088": 0.0,
"c089": 0.0,
"c090": 0.0,
"c091": 0.0,
"c092": 0.0,
"c093": 0.0,
"c094": 0.0,
"c095": 0.0,
"c096": 0.0,
"c097": 0.0,
"c098": 0.0,
"c099": 0.0,
"c100": 0.0,
"c101": 0.0,
"c102": 0.0,
"c103": 0.0,
"c104": 0.0,
"c105": 0.0,
"c106": 0.0,
"c107": 0.0,
"c108": 0.0,
"c109": 0.0,
"c110": 0.0,
"c111": 0.0,
"c112": 0.0,
"c113": 0.0,
"c114": 0.0,
"c115": 0.0,
"c116": 0.0,
"c117": 0.0,
"c118": 0.0,
"c119": 0.0,
"c120": 0.0,
"codex_required": 0.01
},
"confidence": 0.99
}
},
"usage": {
"input_tokens": 17090,
"output_tokens": 1229
}
}
When Jev selected a script and its probability was 0.5 or higher, it was executed. When Jev selected handoff to Codex, or when the script probability was less than 0.5, processing was handed off to Codex.
| Approach | Selection Method |
|---|---|
| Codex only | Have Codex select all cases |
| Jev and Codex | Have Jev select all cases, hand off to Codex only when handoff is selected or selection probability is below 0.5 |
Verification Results
In this case, the request text, execution arguments, and execution conditions were placed in state.
{
"state": "{\"request\": \"Please change the next available item in the target work queue to in-progress.\", \"work_state\": {\"arguments\": {\"queue\": \"editorial\"}, \"domain\": \"queue\", \"execution_guard\": {\"arguments_resolved\": true, \"operation_state_eligible\": true, \"output_path_allowed\": true, \"referenced_inputs_readable\": true, \"target_registered\": true}, \"recent_event\": \"Target preparation completed\"}}"
}
In response to this input, Jev selected c001.
The results of the combined Jev and Codex approach are shown by request type.
| Request Type | Number of Requests | Correct Answers | Jev Script Selections | Jev Handoff Selections | Codex Calls |
|---|---|---|---|---|---|
| Requests that directly express the purpose of a script | 90 | 90 | 90 | 0 | 0 |
| Requests that express the same purpose in different words | 90 | 90 | 90 | 0 | 0 |
| Requests requiring handoff to Codex | 90 | 90 | 0 | 90 | 90 |
The aggregated results for 270 cases by approach are shown below.
| Approach | Correct Answers (out of 270) | Codex Calls | Incorrect Script Selections | Total Time to Determine Execution Target (min:sec) | Cost (USD) |
|---|---|---|---|---|---|
| Codex only | 270 | 270 | 0 | 21:38 | 4.946 |
| Jev and Codex | 270 | 90 | 0 | 11:13 | 1.779 |
The time to determine the target script was 51.8% of the Codex-only approach.
Cost was calculated by applying the GPT-5.6 Sol API unit price and Jev's official unit price to the recorded token counts. The breakdown for the combined Jev and Codex approach is approximately 1.585 USD for Codex and approximately 0.194 USD for Jev. The total cost was 36.0% of the original.
Discussion
Based on the results of this verification, it is considered that Jev can be used for selecting scripts that match routine tasks requested in natural language. In development environments where scripts for inspection, document generation, and state updates have grown in number, a configuration where Jev handles the comparison of candidates that Codex would otherwise read through, and only requests that existing scripts cannot handle are passed to Codex, can be considered.
Note that while Jev selected the correct script in all cases in this experiment, in practice there is a possibility that Jev may select a different script. It seems that mechanisms to mitigate the impact of incorrect selections will be necessary, assuming they can occur. For example, retaining input and state checks in each script, and preparing records of changes and means of reversal.
Summary
In the environment created for verification, while maintaining the number of correct answers, the total time to determine the execution target was reduced to 51.8% of the Codex-only approach, and cost was reduced to 36.0%. Jev can be applied in situations where choices, execution arguments, and execution conditions can be defined in advance. Before introducing it into a real project, verification using actual requests and existing scripts is necessary. We hope this article serves as a reference when considering the division of roles between Jev and Codex.
