![[Report] Too Close to the Metal: Attacks on GPU Neoclouds from Inside and Outside - CODE BLUE 2026 #codeblue_jp #codeblue2026](https://devio2024-media.developers.io/image/upload/f_auto,q_auto/c_limit,w_1200/v1794835744/eyecatch/lwaeb6mt4ljjwzxqdkhi.jpg)
[Report] Too Close to the Metal: Attacks on GPU Neoclouds from Inside and Outside - CODE BLUE 2026 #codeblue_jp #codeblue2026
This page has been translated by machine translation. View original
Hello, this is Usuda.
Everyone, are you taking security measures? (greeting)
This is a report on the following session held at CODE BLUE 2026.
It's a case showing that proper tenant isolation is necessary in multi-tenant services.
Too Close to the Metal: Attacks on GPU Neoclouds from Inside and Out
AI hasn't just created demand for GPUs. It has transformed GPU data centers into something anyone can rent by the hour.
GPU "neoclouds" (new cloud providers) sell compute resources, and often place tenants in bare-metal environments. Users may think they have simply rented GPUs, but in reality they may have rented a "foothold" inside the provider's data center.
We tested GPU providers from both sides of that boundary. From the internet side, through exposed BMCs (baseboard management controllers), we were able to access power control, remote consoles, boot settings, and firmware update flows. Also, unauthenticated GPU telemetry exposed fleet maps and monitoring endpoints that could be abused for reconnaissance and denial-of-service (DoS) attacks. Meanwhile, from inside a paid leased environment, we succeeded in breaking into backend layers that tenants should never be able to access (the InfiniBand fabric, shared storage, cluster DNS, and even the server's own BMC).
In this session, we examine the GPU data center as a single attack surface, component by component, and conclude by presenting a practical tenant-oriented checklist for finding the same vulnerabilities before attackers do.
Speakers: Michael Katchinskiy, Yakir Kadkoda
Report
- A talk about how to attack GPU neoclouds
- Starting with history
- In 1945, a computer occupied an entire room
- Computers came to be placed in data centers, which was a big undertaking
- People started using providers instead of owning their own infrastructure
- Customers only needed to focus on what they would use it for
- Then AI appeared
- This changed the situation
- Training models
- GPUs are in far shorter supply than demand
- Many providers entered the market
- The cloud split into two
- Building your own
- Renting from a neocloud
- New neoclouds emerged, and some companies even went public
- What happens to security as a result?
- They appeared in no time
- Security testing is lagging behind
- Most of the technology comes from HPC
- Or from data centers
- It comes from a single company
- They are deployed in places where parties don't trust each other
- Hyperscalers use their own isolation technologies, but the newcomers do not
- They are building as they run
- Security can't keep up
- What's inside a GPU server
- Lots of GPUs and a modest number of CPUs
- OS drivers, LLMs and so on, and monitoring
- Firmware and so on in the BMC
- There's also networking
- Connected to storage systems too
- Connected via management to the BMCs of other servers
- A single server is very complex
- Two approaches
- Routes from the public internet
- Renting and looking from the inside
- What we found
- NVIDIA DCGM
- Monitored with Prometheus
- You can see various things such as GPU status
- Can be used to map out attacks
- There are debugging tools
- Exposed to anyone
- An attacker can access it from the debugging console and stop it
- More than 2,000 open servers were found
- Neocloud customers had left them open
- NVIDIA DCGM
- BMC
- BMC stands for baseboard management controller
- Built into almost every server
- Runs even when the power is off
- Power, console startup, etc.
- Taking it over is the same as seizing all privileges over the physical machine
- Example of Dell iDRAC
- Administrators can view the screen and operate it
- Communication methods
- IPMI
- Old
- UDP/623
- Redfish
- New and secure
- over HTTP
- IPMI
- CVE-2013-4786
- Related information: 24,650 Internet-Exposed BMCs Disclose IPMI Password Hashes Before Login (The Hacker News)
- Very old
- A protocol issue that has not been fixed
- The BMC generates its own random number, ID, and hash value
- Since it returns the password hash to anyone, an attacker can crack it offline
- BMCs must not be reachable from the internet
- How easily can they be broken?
- Supermicro prints an individual password at shipment
- But it can be cracked in an hour
- HPE iLO passwords are 8 characters, uppercase and lowercase letters only
- Can be broken in 32 seconds
- The management screen of one real company had been rewritten with a message demanding a Bitcoin payment
- How to break in from the inside
- Results of investigating many neocloud vendors
- Investigated companies with silver or bronze track records
- Case1
- Rented one server
- There is no hypervisor in a bare-metal environment
- The kernel and drivers are dedicated
- But storage and so on are shared
- After use, it may be assigned to another customer
- IPMI over KCS
- A feature for the host OS that doesn't require a password
- Can communicate with the BMC
- Local communication is possible
- We could confirm the BMC's IP, and since there was no ACL, we could access it via Redfish
- We obtained details of the management controller
- We could access 14 subnets and the servers on them
- We shouldn't have been able to access management information, but we could obtain passwords using a vulnerability
- With access to the BMC, we could access router switches and crack passwords
- Case2
- Targets the high-speed network
- Rented servers and connected to InfiniBand
- Performs AI training
- Data is exposed through the servers
- Using RDMA can reduce overhead
- Accesses directly without going through the kernel
- HCA
- The subnet manager holds the information
- InfiniBand is supposed to separate tenants
- There is P_Key, with restrictions depending on configuration
- A full member can talk to anyone
- SA_Key + SMPs
- Communicate bypassing the subnet manager
- Even with a rented server, you can access any server
- M_Key
- This was 0
- Can shut down other tenants
- SM_Key
- Can control the subnet manager
- InfiniBand defaults
- Everyone can see everything
- Everything was visible
- Case3
- Shared storage
- Processed simultaneously on many GPU servers
- All the data can't be placed on the servers
- Each customer should see only their own data
- Weka is commonly used
- Connected to the storage network with the Weka client
- Also backed up to S3
- The tenant is the boundary
- The boundary is not the file system
- The one in the root org could be mounted without authentication
- Training results and so on were stored there
- Summary of the cases
- In the management plane there is a BMC, and its password is the default
- It can also be accessed from InfiniBand
- With shared storage, other tenants' data can be mounted
- What should be done
- Isolate KCS
- Isolate local BMC access
- Isolate the network
- Require a password for changes in InfiniBand
- NVIDIA has published that configuration
- Don't leave members at the default
- These countermeasures are the provider's responsibility
- Isolation must be done at every level
- It must be verified
- It needs to be done with every update
- There is also research by SemiAnalysis
- Related information: SemiAnalysisAI/ClusterMAX (GitHub)
- There are also OWASP activities
Impressions
With multi-tenant services, how far you can trust the provider is very important.
I want to use them carefully.