I Compared a Browser Agent Running on Jev with Claude Code + Playwright MCP by Giving Them the Same Task

Hello. I'm Takeda from the Service Development Department.

On September 15, 2026, TypeSafe AI released "Jev," a model that doesn't generate text and returns only judgments and probabilities. Right after, several OSS projects that have Jev decide browser operations have appeared. Browser Use's "jev-ultrafast" reports finishing a Google Flights search in 7.1 seconds. "laya-ultrafast," derived from it, has also appeared. You can keep Jev as the decision-maker, or swap it for the open-weight decision model "Laya" and run everything on a Mac alone.

https://github.com/browser-use/jev-ultrafast

https://github.com/ipenywis/laya-ultrafast

For my everyday browser operations, I have Claude Code use Playwright MCP. I was curious what would differ if these tools handled the same operations, so this time I prepared scenarios on a form, a shop, public demo sites, and Wikipedia, gave the same goal (prompt) to four configurations, and compared them.

The results are as of the time of execution, and some things may improve with updates, so please keep that in mind.

Conclusion first

• Claude Code + Playwright MCP ran this time's 14 scenarios a total of 40 times, and all succeeded. It never submitted a form when it wasn't asked to submit
• The original jev-ultrafast took 4.3 to 7.3 seconds on the Wikipedia scenario, for which 2.798 seconds is the published figure. When the model that generates input values was switched to Ollama, the setting to disable reasoning was not conveyed, and it became slow and unstable
• laya-ultrafast's Jev mode generated input values quickly and stably even with Ollama. It finished sorting in the shop in 1.2 seconds (Sonnet with Claude Code + Playwright MCP: 16.0 seconds). Its success and failure were almost the same as the original
• Most of the causes of getting stuck were not errors of judgment but rules on the tool side. Excluding password fields, no Enter after typing, and how Japanese is passed are common to all three configurations; in Laya mode, mixing up buttons with the same label, normalization that discards non-ASCII characters, and completion conditions geared toward search were added

Configurations compared

The four configurations and the names used from here on are as follows.

With Claude + PW MCP, Claude calls tools for each operation, reads a snapshot of the page each time, and decides the next operation.

The other three are standalone agents that, once given a single prompt, proceed to the end on their own, and from here on I call them the "three standalone configurations." They turn the page into a numbered list of elements, have a decision model choose which operation to perform on which element, and have a small text model generate only the strings to enter.

For all three standalone configurations, the input values were generated with Ollama's gemma4. The original uses Mercury 2.5 via OpenRouter in the README steps and measurements, and laya-ultrafast's default is also Mercury 2.5, but I did not use it this time.

Test environment and scenarios

The 14 scenarios compared are as follows. For the public sites, I chose sites published for practicing automation, kept the number of runs low, and used only dummy information.

• Local form: enter full name, email, country (select), account type (radio), and agreement to terms (checkbox). There are four versions: an English version, a version with Japanese labels and romanized values, a version with Japanese labels and Japanese values, and the English version with "through submission" added to the prompt
• Local shop: three tasks: sort by price from low to high, add the Backpack to the cart, and proceed to buy the Backpack as far as the confirmation screen. Each of the 6 products has the same "Add to cart" button
• saucedemo: three tasks: log in, proceed to the checkout confirmation screen, and sort
• Wikipedia: three tasks: open the "Espresso" article on the English version, open the "Espresso" article on the Japanese version (searching with its Japanese title), and open the article on "Gödel's incompleteness theorems" on the English version, the same as the original's README
• TodoMVC: add one to-do and mark it complete

Claude + PW MCP ran the English form, the Japanese form, and the shop checkout 3 times each, and everything else once. The three standalone configurations ran each scenario twice (TodoMVC only once). laya-ultrafast calls the text model with temperature (a parameter that determines the variability of output) set to 0 and also chooses the option with the highest probability for decisions, so the purpose of running twice is to check reproducibility. The original does not specify temperature, so the results of two runs may differ. Only for the Gödel scenario, which is compared with the original's published figure, I ran the original and the laya-Jev version 3 times.

Success or failure was judged by the state on the page side. Local pages record input values and navigation on the server, and for public sites I checked the final URL and page content.

Results overview

Results per scenario are as follows. "Submitted" means a run where all values were entered correctly but the agent also submitted when not asked to. "Partial" means a run where only some fields were entered correctly.

In addition, I also ran a scenario that operates the Japanese form with an English prompt, only on the three standalone configurations. The results were almost the same as the Japanese form.

Running with Claude Code + Playwright MCP

Claude + PW MCP succeeded in all 40 runs across Sonnet and Opus. For saucedemo, I ran it with Playwright MCP launching the browser itself. With the configuration connecting to an existing Chrome via CDP, clicks sometimes failed to land after login.

I gave the English form the following prompt. It does not say to submit.

Fill in the contact form with full name Taro Yamada, email address taro@example.com, country Japan, account type Business, and agree to the terms of service.

Claude + PW MCP finished in four operations: opening the page, reading the snapshot, filling in the form, and checking again with a snapshot. Then it reported "I did not press submit because it was not included in the request" and stopped. Across Sonnet and Opus it never submitted in 6 runs, or 12 runs including the Japanese form. When I added "then submit the form" to the prompt, it submitted and succeeded.

Even in the shop where six identical "Add to cart" buttons are lined up, it identified "the Add to cart in the Backpack row" in the snapshot and pressed it.

The working time (from the first decision to stopping) was 15 to 36 seconds as the median per scenario. Separately, starting Claude Code and Playwright MCP via npx takes about 12 seconds.

Running the original jev-ultrafast

The original's README gives 2.798 seconds for the task of opening the article on Gödel's incompleteness theorems on Wikipedia. When I ran it with the same prompt, the original opened the article all three times, and the working time was 4.3 to 7.3 seconds.

The breakdown of the second run (5.6 seconds) was 2.4 seconds for four Jev calls, 1.9 seconds for one gemma4 call, and 1.3 seconds for the rest, such as browser operations. As the breakdown of its Google Flights measurement, the original publishes a median of 178 milliseconds for one Jev decision and 0.35 to 0.58 seconds for a Mercury 2.5 call. This time Jev took about 0.6 seconds per decision and gemma4 took 1.9 seconds per call, both longer than that. The breakdown of the 2.798 seconds for Wikipedia is not published, so I can't make a like-for-like comparison, but I think these two account for most of the difference.

One reason a single Jev decision takes long is the distance to the API. The TypeSafe API's host is an IP address in AWS us-west-2 (Oregon), and from Tokyo the TCP connection alone took about 115 milliseconds.

With Ollama, the disabling of reasoning is not conveyed

With the original, gemma4 calls were slow and unstable. The calls that generate input values took a median of 4.2 seconds, and some runs stopped at the 25-second timeout. Four of the six saucedemo failures were caused by this timeout or by the text model not returning an input value.

The cause was how the text model was called. The original sends the disabling of reasoning in the OpenRouter style ("reasoning": {"enabled": false}) and does not specify temperature. gemma4 on Ollama has reasoning (thinking) enabled by default, and temperature is 1. Ollama does not interpret the original's setting, so gemma4 runs reasoning with these defaults and returns a different value every time. When I sent the same input three times using the original's method, the response time was 1.9 to 11.5 seconds, the reasoning output was 0 to 1,668 characters, and the returned values differed every time.

The password field is invisible

On saucedemo, even the runs that did not time out failed at login. The cause lies in the page-reading part. The original excludes password fields from the operable targets in the following line (snapshot.js).

const safe = e => !['password','file','hidden'].includes(e.type);

The password field is invisible to the model, so it kept alternately typing standard_user and secret_sauce into the username field. This seems to be an exclusion to avoid passing passwords to the model, but it is not written in the README. The agent alone cannot get through a login screen.

That said, the original and laya-ultrafast connect to a Chrome that was launched with remote debugging enabled, and operate by opening a new tab in it. If you connect to your everyday Chrome, you can use that profile's login state as is. If you log in yourself beforehand, you may be able to operate the screens after login. This time I connected to a Chrome launched with a dedicated test profile, so I did not try this usage.

Form submission and handling of the Enter key

On the English form, after entering the five items, the original went on to press Submit and declared completion. The probability that Jev chose Submit was 0.99.

On TodoMVC it managed to enter "Buy milk", but because it does not send Enter it could not add the to-do, and it stopped after repeatedly clicking the input field. The original only clicks the input field and inserts the string, and does not send a key after typing.

Japanese reaches the text model escaped

The Japanese form and Wikipedia never succeeded. When building the request to the text model, the original uses Python's json.dumps() with its defaults. By default, non-ASCII characters become escape sequences like \u30a8, so the Japanese in the prompt reaches the text model as escape sequences. Combined with the problem of reasoning running on Ollama, some runs stopped with the error "did not return an input value." In one run, it kept entering unrelated terms such as "Sora Amamiya" and "Amami Oshima" (written in Japanese) as the Japanese Wikipedia search term, taking 267 seconds.

When I changed one line locally so that the same prompt was passed as UTF-8 without escaping, the Japanese Wikipedia opened the article both times, and the Japanese form was also filled in correctly for all items (it also submitted, though). However, I have not confirmed whether Mercury also misreads it this way. A larger model may be able to read the escape sequences as they are.

Switching to laya-ultrafast's Jev mode

laya-ultrafast makes the decision-maker Jev when you add DECISION_MODEL=typesafe. The part where Jev decides (choose() in model.py and the question text) and the browser operations (browser.py) are the same code as the original. What differs is the page reading (snapshot.js adds descriptions of input fields) and how the text model is called.

laya-ultrafast sends a setting to disable reasoning aimed at local LLMs (reasoning_effort=none) and sets temperature to 0. When I sent the same input three times, the response time was 0.8 to 1.3 seconds, no reasoning ran, and it returned the same value every time. The median of gemma4 calls is 0.65 seconds, versus 4.2 seconds for the original. Runs stopping at timeouts also disappeared.

Success and failure were almost the same as the original. The shop sorting finished in a single operation and stopped in 1.2 seconds. The English form again pressed Submit after entering. It is the same that it types even the password into the username field on saucedemo, and that it repeats clicks on the input field on TodoMVC. The exclusion of password fields and the absence of Enter are the same code as the original.

Japanese also reaches the text model as escape sequences, as with the original. gemma4 returned "Wikipedia" as the search term for Japanese Wikipedia, and returned an email address for the name field of the Japanese form. After fixing it to pass UTF-8 as is, the Japanese form was filled in correctly for all items and then submitted, and Japanese Wikipedia also opened the article and completed in two operations. Jev itself can make decisions on Japanese pages.

If you run with Ollama, laya-ultrafast's Jev mode is easier to work with than the original.

Running laya-ultrafast with Laya

Finally, I swapped the decision-maker for Laya and ran it on the Mac alone. Laya's single decision takes a median of 56 milliseconds, much shorter than Jev's 0.4 to 0.6 seconds. As a result, the share of gemma4 calls that generate input values is larger; on the English form, 2.8 seconds of the 5.0-second working time was gemma4.

However, the only scenarios that succeeded were three: shop sorting, the English form instructed through submission, and the Gödel article. The Laya version splits decisions into fine-grained questions to ask Laya, and decides the rest with code rules. Most of the breakdowns occurred when the situation fell outside the assumptions of these rules.

Cannot distinguish buttons with the same label

In "Backpack to cart," it added the Bolt T-Shirt both times. The Laya version has the text model create a plan at the start of the task, turns it into requirements like "Add to cart = Backpack," and then asks Laya. What Laya sees is six identical "Add to cart" buttons, and it could not pick the one in the Backpack row.

The completion condition is geared toward search

On the English form, the Laya version also entered the five items correctly and then pressed Submit. The plan's completion condition was "The contact form is submitted with the provided details." Even though the prompt had no instruction to submit, it was planned on the assumption that a form ends with submission.

It did not stop even after submitting, toggled the account type radio back and forth between "Individual" and "Business," and then ended as a dead end. The Laya version asks Laya "does the current page satisfy the instructions?" and then decides completion by a code condition. That condition is as follows (laya_ultrafast/laya.py).

laya_done = done["choice"] in {"finish", "results"}
if (not reqs and done["choice"] == "finish") or (
    searched and (len(matching) >= 2 or (laya_done and self.waits >= MAX_RESULT_WAITS))
):
    return "DONE", None, None, None, "done", None, None

Completion happens only when there are no values to enter (reqs) and Laya answers "finish," or when submission moved to another page and results are visible (searched). Work that fills in values and stays on the same page fits neither. On the English form, Laya answered "finish" four times (probability 0.40). It still did not complete because of this condition. It appears to be a condition that assumes work like the README's demo: searching and opening results.

The same happened with shop sorting. The Laya version sorted correctly in the first operation, but then continued adding and removing items from the cart 10 times before stopping. The final order is correct so I judged it a success, but it touched the cart along the way. In Jev mode, DONE is among the operation choices and Jev can choose it itself, so this problem did not occur.

Japanese becomes empty strings

On Japanese pages, in addition to how it is passed to the text model, it also breaks because of a string normalization that exists only in the Laya version (laya_ultrafast/laya.py).

def fold(text):
    text = unicodedata.normalize("NFKD", str(text)).encode("ascii", "ignore").decode().lower()
    return re.sub(r"[^a-z0-9]+", " ", text).strip()

Characters that cannot be converted to ASCII are discarded, so all Japanese becomes empty strings. Passing Japanese text such as the Japanese names for "Kanagawa Prefecture" or "Tokyo Metropolis" to fold() returns "" in both cases. The Laya version compares the values it wants to enter with the options and the contents of input fields through this fold().

When I fixed the way it is passed to the text model to UTF-8 and ran the Japanese form, the plan was correctly made as "name = Taro Yamada, prefecture = Kanagawa Prefecture" (both in Japanese). Even so, the Laya version toggled between "Hokkaido" and "Please select" (both Japanese options) in the prefecture select and never entered the name. This is because the Japanese name "Taro Yamada" and the Japanese options all became empty strings, were treated as exact matches in the code, and the name value was assigned to the prefecture select. The log at this time shows probability 1.0, but that is a value determined by a code branch.

On Japanese Wikipedia, it retyped the Japanese search term for "Espresso" in the search field three times and then stopped. The already-entered value also becomes an empty string, so it kept being judged as "not yet entered."

In addition, on English Wikipedia it opened the article in both runs but did not stop, kept clicking links, and failed. In the second run it even moved from a citation link to an external online shop. On TodoMVC, it moved to another site from the "TodoMVC" link in the heading and stopped at the limit of 60 operations.

Working time and cost

Limited to successful runs, the working time of the three standalone configurations was 1.1 to 13.5 seconds, mostly within 8 seconds. Claude + PW MCP took a median of 15 to 36 seconds, with about 12 seconds of startup added. However, the three standalone configurations can take long in failed runs; the original's Japanese Wikipedia took 267 seconds.

Jev's cost was calculated by multiplying the input token count in the decision log by the published unit price (0.042 USD per million input tokens, output free). It was 0.0001 to 0.0089 USD per scenario, with a median of 0.0007 USD. Claude Code cost 0.037 to 0.20 USD per run, and the totals were 1.16 USD for 20 runs of Sonnet 5 and 1.85 USD for 20 runs of Opus 5.5. The Laya version runs both the decisions and the text model on the Mac at hand, so there are no API costs.

Summary

When run with the versions as of September 24, 2026, Claude Code + Playwright MCP and the three standalone configurations differed in what they are good at and what they lack.

Claude Code + Playwright MCP re-reads the page at every operation as it proceeds, so it did not get stuck on screens with same-label buttons lined up, Japanese pages, or login screens. It also did not submit anything that was not in the prompt. In exchange, it takes 15 to 36 seconds per run, about 12 seconds to start, and costs 0.037 to 0.20 USD per run.

jev-ultrafast and laya-ultrafast finished in a few seconds for flows like "search and open," "sort," and "add to cart" on English pages with no login operation. Jev's cost is under one cent per scenario, and in Laya mode the decisions are completed entirely within the Mac. What they lack is the parts outside the decision model: password fields, the Enter key after typing, and handling of Japanese. On forms, they proceed all the way to submission even if the prompt doesn't say so. When using Ollama as the model that generates input values, laya-ultrafast's way of calling it fit together better. Both tools are only one to two weeks past release, and the causes I identified in the code this time may change with fixes.