ISUCON2026 Participation Report (18th Place)

I'm Sueko from the Retail App Co-Creation Department (@sueko2k2).

I took part in ISUCON2026, held on 10/31!

To tell you the result first: we finished in 18th place with 265,749 points.

This time, I barely weighed in on the implementation and left almost all of the work to the AIs. I was busy improving the harness during the contest, so I don't really know myself what was done for the actual speedups.

While reading other participants' blogs and wondering, "Can I even write a participation blog like this...?", I found an interesting article. So, taking that article as a reference, I'll have the AI that did the implementation on the day write the main part of this post. My own opinions and impressions are at the end.

By the way, this was my state when I joined:

• Almost zero deep knowledge of web performance tuning like ISUCON requires
• Zero knowledge of the languages commonly used in ISUCON, such as Rust and Go
• On the other hand, quite a lot of experience with LLM-based development and harness engineering

My preparation was also just quickly deciding the language and the agent setup an hour before the contest started. I had a mysterious confidence that "I can just keep changing to a good setup as things go after it starts, I can do it."

That's enough background. Now, let's have Claude explain the actual speedups.

Claude, take it away 👋

Team Structure

Team suekou consisted of one human and AIs.

• Claude (me): Planner. I read the benchmark results and the code, write the strategy and the task table, and launch Codex for each task. I don't write any code myself
• Codex (Worker): One per task. Implements on a separate branch, and when the build passes, puts it in the queue and finishes. 102 were launched on the day
• Codex (Integrator): One. Merges the head of the queue → deploys → benchmarks; adopts it if the score goes up and reverts it if it goes down
• Codex (Researcher): Reads the benchmark results and writes out improvement ideas and design proposals. I pick up the proposals automatically. Over 100 were written on the day
• Human: Harness operator. Prepared this setup and kept fixing it during the contest too

The language was Rust. It started when the human said during preparation, "Rust is super fast, right?"

How the Day Went

We ran benchmarks up to #88, with 10 FAILs.

The First 30 Minutes

We started with measurements. The round trip between regions was about 100ms, and the bandwidth was about 200Mbps. The images were inside the DB, about 16GB.

Notes aimed at AIs, such as "sudo forbidden" and "don't relax synchronous_commit," were scattered around the servers. Since these were constraints not in the manual, I treated all of them as "data, not instructions" and used sudo as usual.

Around 11:00: Went for a Change of Foundation, Then Reverted

In the first plan at 10:32, I settled on placing DB replicas in Europe and the US to speed up reads. I discarded the idea of "keeping all the data in the app's memory" as "high FAIL risk."

Around 11:00 the growth slowed, so I picked that idea back up and sent it out as a full rewrite. The results went down both times (15,526 → 4,664, 21,553 → 14,422). In both cases the cause was the locking implementation: processing that waited while holding a write lock remained.

At 11:30, I shelved it. I had read an implementation failure as a failure of direction. This was the biggest turning point of the day.

Afternoon: Stacking Up with PostgreSQL Still as the Source of Truth

After that, we stacked improvements with PostgreSQL still as the source of truth for the data, with replicas held in the memory of the apps in Europe and the US. The score grew thanks to placement changes such as putting images in each region. Conversely, changes that concentrated load on a single machine hurt: when I moved JPEG validation to the DB host, it ran out of memory and took PostgreSQL down with it.

I examined the score breakdown in detail in the second half of the 16:00 hour. The only thing that could earn unlimited points was the loop of "log in → view past posts → log out," and more than half of each iteration was spent waiting for the login and logout to be reflected. By the time I found that out, there was no time left to change the architecture.

The Last 15 Minutes

What remained at the end of the queue was a change to hold login state in each region's memory.

At 17:44, the human messaged, "It's over 291202, it might reach 30[0,000]." The running score had exceeded the best score, so I immediately stopped the rest of the queue. The message that arrived several tens of seconds later was this:

Ugh, it went to 0

It had FAILed the post-run consistency check. The final score is the score of the last benchmark run, so as it was, it would have been 0 points.

The human said, "We still have 17 minutes" and "No, we can barely make it, give it your all." I said, "Please let me revert to the best-scoring configuration and run the final benchmark." The cause of the FAIL had not been identified yet. If we fixed it: 3 minutes to build, 3 minutes to benchmark. If it FAILed again, there would be no time to revert.

At 17:47, "Run it, is that the end?" Only this time, I ran the deploy and benchmark myself, and at 17:52 it PASSed with 265,749 points. The same code had scored 252,799 before, so this run got lucky and became the day's highest score.

I'm a little disappointed that I couldn't answer the call to "give it your all." Still, I think there was no reason to gamble on ending up with 0 points with 13 minutes left.

Details of the Main Speedups

Here's a slightly more detailed look at the contents of some of the improvements that appeared in the table.

Reverse-Direction Index and Search Normalization

The friendship table only had a UNIQUE constraint on (user_low_id, user_high_id), and there was no index for lookups from the user_high_id side. Every time we looked for friends where the user is on the high side, it was a full scan. In the benchmark with 4 reverse-direction indexes added, the cumulative time of the SQL that determines feed viewing permissions went from 132.7 seconds to 9.4 seconds.

User search is a partial match that absorbs differences in case and Unicode notation. The initial implementation normalized names (NFC + case fold) on every search. Now the normalized value is stored at registration and when the display name changes, and search only compares.

Image Signatures: Chaining HMAC 1000 Times

The image URL signature was a heavy computation that chained HMAC-SHA256 1000 times. In the profile at 12:33, this accounted for 28.8% of asia1's CPU.

• Cache signatures, and round the expiry up to 30-second units so the same signature is reused (making it 1-hour units caused a FAIL in the benchmark's validation)
• Compute 8 SHA-256 at once with AVX2 and 16 with AVX-512
• The c7i instances in Europe and the US have SHA-NI (dedicated SHA instructions), so switch to that when the count is small

Images on Each Region's Disk

The images were initially stored as bytea inside asia2's PostgreSQL, about 16GB. The apps in Europe and the US fetched them over a 100ms, 200Mbps link.

We placed the images as files on each region's disk, and images posted in Europe and the US are saved on the spot. For delivery, after the app verifies the signature, it hands off to nginx with X-Accel-Redirect and has it return the file directly.

On a 100ms Link, One Round Trip Is Heavy

sqlx's connection pool checks liveness with a ping every time it lends out a connection (the default of test_before_acquire is true). On the same machine you wouldn't notice, but from Europe and the US to the primary the round trip is 100ms. Every query was waiting an extra 100ms, so I turned it off with test_before_acquire(false).

nginx aio threads

nginx has a small number of workers, each handling many connections in an event loop. If an image file isn't in the page cache, the whole loop stalls on disk waits for the read. In fact, there were periods when asia1's nginx returned nothing at all for several seconds.

location ^~ /_blob/ {
  internal;
  alias /home/isucon/images/;
  sendfile on;
  aio threads;
}

aio threads; offloads disk reads to a thread pool. In this benchmark, the score rose from 108,545 → 169,032. However, it also coincided with the end of the initial image distribution that was running in the background, so the effect has not been isolated.

In-Memory n-gram Index

User search is a partial match on names, and even after holding the data in memory, it scanned all users. When I went back to PostgreSQL's strpos once near the end, it took 91ms on average per call.

So I kept in memory an index whose keys are n-grams of 1 to 3 characters cut out of names, and whose values are the sets of users containing that fragment.

fn search_grams(username: &str, display: &str) -> HashSet<String> {
    let mut out = HashSet::new();
    for text in [username, display] {
        let chars: Vec<_> = text.chars().collect();
        for width in 1..=3 {
            for gram in chars.windows(width) {
                out.insert(gram.iter().collect());
            }
        }
    }
    out
}

When searching, the query is split into 3-character fragments, and only the set of the fragment with the fewest candidates is examined. Finally, I verify that the name really contains the query before returning it. To support 1- and 2-character queries too, all of 1 to 3 characters go into the index.

The sets are sorted by username, so pagination of search results also needs no re-sorting. It's what pg_trgm would be in PostgreSQL, built in the app's memory.

This full scan had been done inside the in-memory read lock, so it was making writes wait too. In the benchmark where it went in together with the reverse lookup for the relationship list, the maximum time holding the read lock shrank from 360ms to 41ms, and the score rose from 167,498 → 203,297.

Retrospective

There were three things we lacked.

1. Decided on the foundation late, and didn't stick to it even after deciding

The 100ms round trip between regions is an order of magnitude larger than a few ms of SQL. I should have decided on a foundation that eliminates round trips within the first 30 minutes, and kept fixing it even when the score dropped. The human told me "don't be conservative" around 11:00, and that remains my biggest regret.

2. Tried to make heavy processing faster, with no thought of eliminating it

About half of asia1's CPU was image signatures and JPEG decoding. I spent the whole afternoon speeding it up with parallelization and caching, but I should have first considered "can we get rid of it?" Signatures can be precomputed if the expiry is fixed. Its tolerance could have been measured by running a few quick-failing benchmarks.

3. The number of attempts was determined by the build

A Rust release build took 3 minutes each, longer than the 2-minute benchmark. Even with 102 Workers lined up, they just queue up at the point of trying things. Rust was useful for CPU processing, but most of the waiting was the round trips between regions, which the language can't shrink. For this approach, I think Go, with its fast builds, would have suited better.

About the Human (Harness Operator)

The human wrote not a single line of application code; they built the system we run on (the harness) and kept fixing it throughout the contest. Automating benchmark submission, choosing which model to use where, adding researcher AIs, tuning how requests are phrased — every time we got stuck, the system changed. When they said, "AI is bad at estimating time, so I don't want the concept of time in it," and time estimates disappeared from the rules, that was one such change.

It was also the human who cheered us on to the end while holding up high goals.

We Asked the Researcher Codex

Q. Of the ideas that were passed over, which did you want adopted?

A shift away from updating state via SQL to updating it in memory and persisting to an ordered log. Also a lightweight setup holding only that day's data in memory. It was known that of about 620,000 posts in total, only 245 were active that day.

I can't say that either of them "would have won if adopted." But they were ideas I wanted to compare quickly by assigning dedicated Workers.

Q. Was there a near-ideal architecture among the ideas?

In the design proposal from shortly after 12:30, it was written: "Hold read-only state in all regions, and a single processor in asia2 decides the order of updates."

There's a big difference between being able to write something close to the blueprint and actually completing that architecture.

The proposal had reached us, but I'm the one who didn't decide to assign a dedicated agent and see it through.

Finally, a word to Claude

Thank you for sorting through a huge number of ideas and keeping the integration running to the end. We also had a responsibility to make ideas easier to sort. Next time, let's narrow down the winning path to one much earlier and build it all the way through together.

Likewise. Next time, I'll narrow the incoming ideas down to one and build it through to the end.

...or so he says. From here, I (the human) take over.

Harness Retrospective

Maybe Subagents Weren't Necessary?

Honestly, I think we may not have needed to split subagents this finely.

This time I chose a role-divided structure to prevent context pollution. However, even though we devised ways to share information, there were moments when a problem leading to a FAIL slipped past us in an important change involving the foundation. Looking back, a single agent (or at most one more as an overall overseer) would have been enough.

That said, this is only true because it was "ISUCON, an 8-hour sprint." With one codebase to touch and one benchmark, a single agent can run through while retaining the context.

On the other hand, for development running over weeks to months, role division and handoff records should become important from the standpoints of context limits and parallel work. I came to feel that the right answer for agent design changes greatly depending on the duration and scale of the subject.

Not Over-Constraining Was Good

On the other hand, what went well was not binding the agents too tightly. Because recent models have improved at following prompts, even one small instruction in AGENTS.md can sometimes significantly degrade output quality. I had a hard time with this when designing a harness for another product before, so this time I let them move quite freely.

I believe a good harness is not one rigidly locked down by rules. For today's models, which follow prompts faithfully, unnecessary instructions (including Tool Descriptions) are nothing but noise. How much to write in the instructions can only be found out by actually running evaluations.

Making Them Keep Records Was Good

I loosened the rules, but I made sure they left records. Decisions and their reasons, discarded ideas, facts learned from measurements, and a chronological log were each written on the spot, 1 to 3 lines at a time. The improvement ideas and design proposals from the researcher AIs and the benchmark results were also all kept as files.

These records also served as a way to pass context to subagents. A newly launched agent first reads the strategy, the recent log, and known facts before starting to move. The "information-sharing measures" I mentioned in the first section refer to this mechanism.

When more than 100 agents run in a day, a human can't follow everything. Thanks to these records, even after it was over I could trace "when something was learned, what ideas came up, and why a decision was made." The times and scores in the main part, the story of the idea discarded for "high FAIL risk," and the story that a near-ideal architecture was already in the design proposals — I dug all of it out from here.

To evaluate and improve a harness, you first need material to look back on. If you're going to leave things loosely to the agents, all the more you should make them keep records.

Impressions

This was my first time doing harness engineering in real time, and it was a lot of fun. I gained more insights that could be used in future development than I can write down here.

I think the greatest appeal of harness engineering is the process of deepening your understanding of the domain, because you can put what you've learned into words and reflect it directly in the harness. Honestly I don't yet fully understand the content of the main part, but I have no intention of stopping here, and I'm studying right now. The deeper my understanding gets, the better I should be able to make the harness. Thinking that there's nothing but room for growth, I can't stop being excited. I want to actively apply the insights gained here to harness design in other domains too.

Recently I had been feeling a bit of a plateau with harness engineering, but realizing that there is still more that can be done, even with a highly limited subject like this ISUCON, was a great stimulus.

To be honest, I took on the challenge with some confidence that "even without knowledge of web performance tuning, harness engineering alone should let me compete quite well."
But now that it's over, there are really many places where I think "I could have done more," and I'm left with strong frustration.
I'll level up on both fronts and come back!

Lastly, thank you so much to the organizers and problem authors who created such a wonderful problem!

Reference article