[Session Report] Challenges Revealed Through the Operation of a Test Automation Environment, and Case Studies of Solutions Through Redesign: Case Studies of Scaling Up Test Frequency, Number of Devices, Test Content, and Platforms #CEDEC2026

Introduction

This is a report on a session I attended at CEDEC2026. A developer from SEGA Corporation presented a case study on a Unity-based competitive action game that has been in long-term operation, covering how they gradually expanded their test automation environment while continuing to operate it.

The session overview is as follows.

• Title: Challenges Revealed Through Test Automation Environment Operations and Case Studies on Resolution Through Redesign ～Case Studies on Scaling Up Test Frequency, Number of Devices, Test Content, and Platforms～
• Speakers: Miya Satonaka, Takeshi Hiroshima (both from SEGA Corporation, Technology Division, Development Technology Department, Technology Development Section)
• Date and time: July 24, 2026, 11:10–12:10
• Venue: Hall 8

This session is a follow-up to a presentation at CEDEC 2023 that covered environment setup, and focuses on challenges that became apparent during operations and how they were resolved.

Issues with the Old Environment and the Redesign Policy

The target game is a networked competitive action game with multiple rule sets. E2E tests that match devices against each other are run using the in-house framework ATC. The ATC team, which develops and operates the test automation environment, and the title team, which develops the game itself, carry out their development in parallel through independent flows.

There were constraints here. Jenkins and shared storage for the CI environment were shared with the title team, meaning the ATC team could not update the environment or add features on their own. The old environment was built on Jenkins, but the pipeline had grown bloated and complex, causing problems such as needing to stop all tests with every update and difficulty in debugging.

The solution was a redesign based on a policy of separating responsibilities. The idea was to limit Jenkins, the CI system, to the role of a trigger for launching processes, and to move execution control, state management, artifact management, notifications, and other processing outside of CI.

The In-House Pipeline Tool AutomagicPipeline

The in-house tool AutomagicPipeline (AmPl) was created to handle execution control outside of CI. It is a Windows console application written in C# and .NET8 that centrally controls each process in the test automation workflow.

AmPl is composed of jobs with a single responsibility and pipelines that combine them. Individual retry settings for failures and similar configurations can be set per job, and pipelines are defined by combining jobs in series or in parallel. Behavior can be switched via configuration files, and parameters such as branch and Unity version can be passed in. This made it possible to flexibly modify and verify the execution flow outside of CI.

After the redesign, the processes required for a single test are divided into three pipelines: build, test execution, and result aggregation. The overall system is composed of Jenkins, AmPl, a game server, Elasticsearch and Kibana for the logging infrastructure, OBS Studio for recording, and more.

Four Scale-Ups

In the second half, four case studies of scaling the environment during operations were introduced.

The first was the number of test executions. By preparing a dedicated game server for testing and managing its updates through AmPl as well, they were able to run tests without relying on the title team.

The second was the number of devices. They established a mechanism to aggregate logs from each device and deliver a summary of results from all devices to a chat tool every morning. They also introduced lifecycle management to delete artifacts such as recordings, which kept accumulating, after a certain period.

The third was test content. To match two devices against each other, they implemented a mechanism to synchronize the game launch timing of both devices using flag files on shared storage.

The fourth was platforms. For iOS, real devices are controlled using Appium and WebDriverAgent, and the screen and audio are mirrored on a Mac using OBS Studio. They mentioned that they are also considering future use of AVoIP, which transmits video and audio over an IP network. For Android, a configuration using adb and scrcpy was presented.

Impressions

This was a concrete and thought-provoking presentation about an approach of carving out responsibilities and centralizing them into an in-house tool, all within the realistic constraint of sharing the same CI and environment without being able to modify it freely. I came away feeling that for long-running titles, what is truly difficult is not the test automation itself, but the operational design that allows it to grow without ever being stopped.