I tried bulk granting permissions to all GCP projects

Hello, I'm Harada, a non-engineer.
I manage Google Cloud (formerly known as GCP) used internally for testing purposes.

Introduction

GCP has a specification where "important notifications (such as billing-related or service termination notices) are only delivered to users who are directly registered as Owners on the project."
Even if you inherit the Owner role at the Organization or folder level, these kinds of important notifications will not be delivered.

From a management perspective, thinking "we can't afford to miss important emails," I decided to directly add two persons in charge, in addition to the existing users, as Owners to all existing GCP projects.

I casually thought "it should be possible immediately by running commands in CloudShell" and proceeded while consulting with AI, but when I actually opened the lid, it turned out to be several times more difficult than expected.

※ All actual person names and project names below have been replaced with pseudonyms and dummy values.

What I Wanted to Do

• Target: All existing Google Cloud projects under the organization
• Action: Directly grant roles/owner IAM to two people, Person A and Person B
• Constraint: There are too many to click through manually, so I want to run a script using gcloud commands from CloudShell

Stage 1: Painfully Slow Processing Speed

After consulting with AI, I first ran the following simple command.

USER_A_EMAIL="user-a@example.com"
USER_B_EMAIL="user-b@example.com"

for PROJECT_ID in $(gcloud projects list --format="value(projectId)"); do
  gcloud projects add-iam-policy-binding "${PROJECT_ID}" \
    --member="user:${USER_A_EMAIL}" \
    --role="roles/owner" \
    --condition=None

  gcloud projects add-iam-policy-binding "${PROJECT_ID}" \
    --member="user:${USER_B_EMAIL}" \
    --role="roles/owner" \
    --condition=None
done

The process was moving forward, but it seemed like it would never finish. (I think I waited about 30 minutes.)

Each time add-iam-policy-binding (= the command to add permissions) is executed once,
an exchange with Google's servers occurs: "Please give me the current permissions list" and "Add this person and send it back."
This exchange takes about 1 to 3 seconds each time, and since this was being done sequentially — twice per project for two people, one at a time in order — the waiting time accumulated directly with the number of projects, resulting in an enormous amount of time.

When I checked with AI, I found out that "the target projects are independent of each other, so parallelization is safe."
I switched to parallel execution using xargs -P.

add_owners() {
  local project_id="$1"
  gcloud projects add-iam-policy-binding "${project_id}" \
    --member="user:${USER_A_EMAIL}" --role="roles/owner" --condition=None --quiet \
    > "logs/${project_id}.log" 2>&1

  gcloud projects add-iam-policy-binding "${project_id}" \
    --member="user:${USER_B_EMAIL}" --role="roles/owner" --condition=None --quiet \
    >> "logs/${project_id}.log" 2>&1

  echo "done: ${project_id}"
}
export -f add_owners
export USER_A_EMAIL USER_B_EMAIL

mkdir -p logs
gcloud projects list --format="value(projectId)" | \
  xargs -P 20 -I{} bash -c 'add_owners "$@"' _ {}

I set it up to output a separate log file for each project.
Setting it up so you can search by file unit like grep -l ERROR logs/*.log makes subsequent investigation much easier.

Stage 2: Cloud Shell Disconnects

Since the target was on the scale of hundreds of projects, it still takes a considerable amount of time even with parallelization.
As expected, I encountered a situation several times where the browser-side Cloud Shell connection dropped during execution.

As a countermeasure, I combined nohup and disown to fully background the process.
Simply put, nohup and disown are instructions that say "keep working even after I hang up."

nohup bash -c '
  gcloud projects list --format="value(projectId)" | \
    xargs -P 20 -I{} bash -c "add_owners \"\$@\"" _ {}
' > run_summary.log 2>&1 &
disown

With this, even if the browser tab is closed or the connection is temporarily lost,
the process continues on the Cloud Shell virtual machine side (as long as the VM itself doesn't completely idle and shut down).
Since all execution results are saved in log files, after reconnecting I was able to follow along by running tail -f run_summary.log.

Stumbling Point 2: Running the Same Command Twice

After backgrounding it, since commands no longer ran as before, I thought "is it not responding?"
and ended up running the same launch command twice, causing two jobs to run simultaneously against the same target list.
There was no particular actual harm, but it doubled the load on the API and increased the risk of hitting rate limits.

Since I wanted to stop only one of the background jobs, I added a minus sign before the PID to stop the entire process group.

kill -- -<PID>

Simply using kill <PID> would leave child processes inside the pipeline (xargs and gcloud) alive.

Stage 3: The True Nature of "Thousands of Items"

When I checked the number of target projects, it turned out that the several hundred I had initially understood was actually over 1,000.
Looking at the contents, the majority were project IDs starting with sys- followed by long numbers.

These were "default Google Cloud projects" automatically issued behind the scenes by Google.
According to the official Apps Script documentation, Apps Script projects always use some Google Cloud project for authorization management, and if the user does not explicitly specify a GCP project, the Apps Script side automatically creates and assigns a default background project
(Reference: Apps Script - Cloud Platform projects).

Within the range actually confirmed in our internal environment, these default projects were created with long project IDs starting with sys-.

Every time an employee wrote a little GAS script for work or played around with AI Studio, a project was being automatically created behind the scenes. Considering the purpose this time (ensuring important notifications are reliably delivered), I judged that there was little point in adding Owners to these auto-generated projects as well, and decided to exclude them from the target.

gcloud projects list --format="value(projectId)" | \
  grep -Ev '^(sys-|gen-lang-client-)' > projects_target.txt

wc -l projects_target.txt
# => 1138

This narrowed down the target count to a realistic scale.

Side Story: Attempting Folder-Level Filtering and Failing

The GCP projects I manage are organized and managed by folder per department.
So I also considered whether it was possible to specify targets by folder unit.
The hypothesis was that shadow projects would mostly not be inside folders but scattered directly under the organization.

To achieve this, you can use Cloud Asset Inventory (gcloud asset search-all-resources) to retrieve projects under a folder.

gcloud services enable cloudasset.googleapis.com

gcloud asset search-all-resources \
  --scope="folders/${FOLDER_ID}" \
  --asset-types="cloudresourcemanager.googleapis.com/Project" \
  --format="value(displayName)"

However, PERMISSION_DENIED was returned for the majority of the 25 target folders.

ERROR: (gcloud.asset.search-all-resources) [user@example.com] does not have permission to access folders instance [xxxxxxxxxx:searchAllResources] (or it may not exist)

What I learned here is that "having Owner permissions at the project level" and "having permission to look inside at the folder level" are completely different things.
Even if you are the Owner of individual projects, without folder or Organization-level viewing permissions (such as roles/cloudasset.viewer), you cannot search within folders.

Since requesting permission grants from the organization administrator would incur waiting time, I gave up on this approach for now.

Final Results

The results of running the process against the 1,138 target projects were as follows.

• Succeeded: 1,129 projects (Owner grant to both Person A and B completed)
• Errors: 9 projects

For the 9 projects that resulted in errors, I checked each project individually and either put the response on hold or manually granted permissions from the console.

References

• 組織外のユーザーにプロジェクトレベルでオーナーを付与する際の注意点 - G-gen Tech Blog
• Apps Script - Cloud Platform projects（Google公式ドキュメント）

Closing

What was supposed to be a task of "just quickly running commands in CloudShell" turned out, upon opening the lid, to be a process full of various learnings about GCP's IAM specifications and Cloud Shell characteristics.
In the end, I was able to automatically process 1,129 out of 1,138 cases, and since the causes for the remaining 9 became clear, I was even able to establish a policy for handling them individually.

I hope this is helpful as a reference for someone.