I tried separating data visibility by tenant using Loki / Tempo / AMP with Grafana OSS edition

Hello, I'm sora from the Game Solutions Department.
This time, I tried separating the storage destinations for Loki / Tempo / AMP by tenant, linking them to Organizations in the Grafana OSS version, and separating the visible data.

Conclusion First

• Even if Grafana, Loki / Tempo, and S3 remain shared between tenants, by separating tenants on the storage side and linking them to per-Organization data sources, other tenants' data did not appear on the screen or through the API
• By keeping tenant-side users at Editor level or below, data sources cannot be modified, so separation is maintained
• By creating an Organization for operators, it was possible to view data across both tenants

Architecture

The following architecture was built this time.

In the following article, I confirmed that even when separated by Grafana Organizations, a configuration that directly reads CloudWatch / X-Ray cannot separate metrics and traces from the same AWS account.
https://dev.classmethod.jp/articles/grafana-tenant-separation-team-organization/

This time, the storage destinations are set to Loki / Tempo / AMP, and tenants are separated on the storage side.

Two sets of sample applications are placed, one for Tenant A (app-A) and one for Tenant B (app-B), sending logs, traces, and metrics from sidecars.
Grafana, Loki / Tempo, and S3 are shared between both tenants, and only AMP has separate workspaces per tenant.

The versions used are as follows.

• Grafana: 13.2.1
• Loki: 3.7.7
• Tempo: 3.0.3
• ADOT Collector: v0.50.0

Setup

This time I also built with Terraform.
Since there are many basic resources, I'll omit the code, but I'll explain only the parts where tenants are separated.

Loki / Tempo Tenant Configuration

When Loki has auth_enabled: true, all requests require the X-Scope-OrgID header, and its value becomes the tenant ID.

auth_enabled: true

querier:
  # Enable searching across multiple tenants like X-Scope-OrgID: tenant-a|tenant-b
  multi_tenant_queries_enabled: true

Tempo behaves the same way with multitenancy_enabled: true.

multitenancy_enabled: true

query_frontend:
  multi_tenant_queries_enabled: true

multi_tenant_queries_enabled is a setting for viewing data across both tenants in the operator Organization described later.
The default for Loki is false and for Tempo is true, but they are specified explicitly.
https://grafana.com/docs/loki/latest/operations/multi-tenancy/
https://grafana.com/docs/tempo/latest/operations/manage-advanced-systems/multitenancy/

When hitting Loki / Tempo directly from the Grafana task, requests without a header are rejected with 401.

$ wget -S -qO- http://loki.o11y.local:3100/loki/api/v1/labels
  HTTP/1.1 401 Unauthorized

$ wget -S -qO- --header "X-Scope-OrgID: tenant-a" http://loki.o11y.local:3100/loki/api/v1/labels
  HTTP/1.1 200 OK
{"status":"success","data":["container_name","job","service_name"]}

$ wget -S -qO- http://tempo.o11y.local:3200/api/search
  HTTP/1.1 401 Unauthorized

$ wget -S -qO- --header "X-Scope-OrgID: tenant-a" http://tempo.o11y.local:3200/api/search
  HTTP/1.1 200 OK

Sidecar Tenant Configuration

For logs, tenant_id is added to the FireLens Loki output.
This becomes the X-Scope-OrgID header (this is an excerpt from the task definition).

logConfiguration = {
  logDriver = "awsfirelens"
  options = {
    Name      = "loki"
    host      = "loki.o11y.local"
    port      = "3100"
    tenant_id = "tenant-a"   # tenant-b for Tenant B tasks
    labels    = "job=app"
  }
}

For traces and metrics, the ADOT Collector configuration is separated per tenant.
The only differences are the header attached to Tempo and the workspace destination for AMP writes.

exporters:
  otlp_grpc/tempo:
    endpoint: tempo.o11y.local:4317
    tls:
      insecure: true
    headers:
      X-Scope-OrgID: tenant-a   # tenant-b for Tenant B

  prometheusremotewrite:
    # Write to Tenant A's workspace
    endpoint: https://aps-workspaces.ap-northeast-1.amazonaws.com/workspaces/<Workspace A ID>/api/v1/remote_write
    auth:
      authenticator: sigv4auth
    resource_to_telemetry_conversion:
      enabled: true

The app task roles are also separated per tenant, and the app-A task role has aps:RemoteWrite only for Tenant A's workspace (app-B has only Tenant B's).

resource_to_telemetry_conversion is a setting that converts attributes like service.name into metric labels.
Without this, filtering by service_name is not possible.

S3 Prefixes

After accessing the app for a while, looking at S3 shows that both Loki and Tempo buckets are organized into per-tenant prefixes.

$ aws s3 ls s3://grafana-o11y-loki-<account-ID>/
                           PRE tenant-a/
                           PRE tenant-b/

$ aws s3 ls s3://grafana-o11y-tempo-<account-ID>/
                           PRE tenant-a/
                           PRE tenant-b/
2026-09-24 09:57:02       3084 work.json

The Loki index was uploaded after about ten-plus minutes, and it was also separated per tenant.

$ aws s3 ls s3://grafana-o11y-loki-<account-ID>/ --recursive | grep index/
2026-09-24 10:06:12 index/index_20720/tenant-a/...tsdb.gz
2026-09-24 10:06:12 index/index_20720/tenant-b/...tsdb.gz

IAM Roles for Reading AMP

The IAM roles for Grafana to read AMP are created as one per tenant and one for operators, totaling three.

data "aws_iam_policy_document" "grafana" {
  for_each = local.grafana_roles

  statement {
    actions = [
      "aps:QueryMetrics",
      "aps:GetLabels",
      "aps:GetSeries",
      "aps:GetMetricMetadata",
    ]
    resources = each.value.workspaces
  }
}

The Grafana task role does not have AMP read permissions, only sts:AssumeRole for these three roles.
This is because if read permissions are attached to the task role, all workspaces can be read even without specifying Assume Role.

Organization and Data Source Configuration

Creating Organizations and Users

Three Organizations were created: one for Tenant A, one for Tenant B, and one for operators.

Each user belongs to only one Organization and is not added to Main Org.
The method for creating Organizations and users is the same as in the following article.
https://dev.classmethod.jp/articles/grafana-tenant-separation-team-organization/

Loki Data Source Configuration

Switch to OrgA and create a Loki data source from Connections → Data sources.
The URL is the Loki Cloud Map name, which is the same for all Organizations.

What determines the tenant is X-Scope-OrgID in HTTP headers.
For OrgA the value is set to tenant-a, and for OrgB it is set to tenant-b.
Once saved, the value is masked as configured and cannot be read from the screen or the API.

Tempo Data Source Configuration

Tempo is the same as Loki — enter X-Scope-OrgID in HTTP headers.

AMP Data Source Configuration

For AMP, enter the tenant's workspace URL (https://aps-workspaces.ap-northeast-1.amazonaws.com/workspaces/<workspace ID>/) in Prometheus server URL.
OrgA uses Workspace A's URL and OrgB uses Workspace B's URL.

Set Authentication Provider to AWS SDK Default and enter the tenant's role (for OrgA, grafana-tenant-a) in Assume Role ARN.
Other options available for Authentication Provider are Access & secret key and Credentials file.

When pressing Save & test with Assume Role ARN left empty, a 403 Forbidden error occurred.
This confirms that with only the Grafana task role, no workspace can be read.

Note that when entering data sources via API or provisioning, the Assume Role ARN goes in sigV4AssumeRoleArn within jsonData.
The official documentation's provisioning description mentions assumeRoleArn, but when entered there, AssumeRole does not occur and the same 403 Forbidden as not entering an ARN was returned.
https://grafana.com/docs/plugins/grafana-amazonprometheus-datasource/latest/configure/

Verification

Only normal requests were sent to Tenant A (app-A), while Tenant B (app-B) received both normal requests and requests to an endpoint that returns errors.
This was done so that when viewing the screen, data from each tenant could be distinguished by the presence or absence of errors.

Verification with usera3

Log in as usera3 and run the following query against Loki in Explore.

sum by (ecs_task_definition) (count_over_time({job="app"} | json [1m]))

Only two series, grafana-o11y-app-a-frontend and grafana-o11y-app-a-backend, are shown, and app-B logs do not appear.

Opening Drilldown → Traces, the error/s in the upper right remains at 0.

Let's also check AMP.

sum by (http_response_status_code) (rate(app_requests_total[1m]))

Only status code 200 is shown, returning only values from Tenant A's workspace.

Verification with userb3

Let's do the same thing with userb3.

Only grafana-o11y-app-b-frontend and grafana-o11y-app-b-backend appear in Loki.

Since errors are being sent to Tenant B, a spike appears in error/s.

The 500 series also appears in AMP.
Even when using the same Grafana and the same Loki / Tempo, it has been confirmed that the visible data is separated per Organization.

Verifying Data Source Modification by Editor

Since the data source configuration is what separates tenants, if it can be rewritten, the separation breaks.
Since usera3 is an Editor, let's verify whether they can actually modify data sources.

There is no Connections in the left menu, and within Administration → Plugins and data there is only Correlations, with no entry point to the data source settings screen.
Even when directly opening the URL of the data source edit screen, it redirects back to the home page.

Attempting to rewrite the header value to tenant-b via the API also results in a 403.

$ curl -s -u usera3:<password> -H 'Content-Type: application/json' \
  -X PUT https://<GrafanaのURL>/api/datasources/uid/loki-a \
  -d '{"name":"loki","type":"loki","access":"proxy","url":"http://loki.o11y.local:3100","jsonData":{"httpHeaderName1":"X-Scope-OrgID"},"secureJsonData":{"httpHeaderValue1":"tenant-b"}}'
{"message":"You'll need additional permissions to perform this action. Permissions needed: datasources:write","title":"Access denied"}

Attempting to create a new data source also results in 403, since there is no datasources:create permission either.

On the other hand, an Organization Admin can modify data sources, so by rewriting the header value or Assume Role ARN, other tenants' data can also be viewed.
The premise of this separation is that tenant-side users are not given Organization Admin, and data sources are created by the operations side.

Cross-Tenant Viewing in the Operator Organization

When separated by Organizations, only your own tenant's data can be seen from within a single Organization.
For operators who want to see all tenants, Org-ops has the following data sources created.

For Loki / Tempo, listing tenant IDs separated by | in the header value enables searching across multiple tenants at once.
The multi_tenant_queries_enabled setting added in the setup section is for this purpose.
Since AMP can only specify one workspace per data source, a data source is created for each workspace.

Log in as ops3 and run the following query against Loki in Explore.

sum by (__tenant_id__) (count_over_time({job="app"}[1m]))

Two series, tenant-a and tenant-b, appeared.
__tenant_id__ is a label that Loki attaches when searching across multiple tenants, indicating which tenant the log belongs to.

Let's also check Drilldown → Traces.

Tenant B's errors appear in error/s, and the frontend Rate is about 3.1, roughly double what was seen when viewing with userb3 (about 1.6).
Traces from both tenants are visible together.

Incidentally, with Loki's cross-tenant search, depending on the search period, only one tenant's logs may be returned.
This has been reported as a known bug in Loki: when the search range spans both data in Loki's memory and data in S3, the results from the second tenant are missing (as of September 2026, this remains unresolved).
https://github.com/grafana/loki/issues/21936

Since no error is shown and only some tenants' results are missing, caution is needed when using cross-tenant search results for aggregation and similar purposes.

Supplementary: Loki / Tempo Basic Authentication and Forward OAuth Identity

In the Authentication of Loki / Tempo data sources, Basic authentication and Forward OAuth identity are also available.

However, since Loki / Tempo have no authentication mechanism, the sent credentials are not verified, and the tenant is determined solely by X-Scope-OrgID.
The Loki documentation states that an authenticating reverse proxy should be placed in front, and X-Scope-OrgID is to be added by that proxy.

Grafana Loki does not come with any included authentication layer. You must run an authenticating reverse proxy in front of your services.

https://grafana.com/docs/loki/latest/operations/authentication/

With a configuration where authentication is done by a reverse proxy and then X-Scope-OrgID is attached, the tenant is determined from the authenticated user or token rather than the header entered in Grafana's data source.
A proxy that maps Basic authentication users to tenants is also available as OSS.
https://github.com/giantswarm/grafana-multi-tenant-proxy

How to Separate When Used Across Multiple Projects

In this configuration, what separates the tenants is the data source configuration.
Based on the combination of how Organizations are divided and which roles are granted, I organized the verification results into the following three patterns.

• Pattern 1: Organization separation + Tenant side is Editor (this configuration)
  • Organization: Created per tenant
  • Tenant-side users: Editor
  • Data sources: Configured by the operations side's Organization Admin
  • Operators: View cross-tenant data from the operator Organization using data sources with multiple tenants listed in X-Scope-OrgID
  • Since tenant-side users cannot modify data sources, other tenants' data is not visible
    • Adding data sources and users is handled by the operations side
• Pattern 2: Organization separation + Tenant side has Organization Admin
  • Organization: Created per tenant
  • Tenant-side users: Organization Admin
  • Data sources: Configured by the tenant side
  • Operators: Same as Pattern 1
  • The tenant side can add data sources and users, but by rewriting the header value or Assume Role ARN, other tenants' data can also be viewed
    • To prevent this, additional configurations such as placing a reverse proxy in front of Loki / Tempo to overwrite X-Scope-OrgID are needed
• Pattern 3: Separate Grafana per tenant
  • Grafana: One instance per tenant
  • Tenant-side users: Organization Admin (Grafana Server Admin is also acceptable)
  • Data sources: Configured by the tenant side
  • Grafana task role: Permissions to read only that tenant's workspace are attached directly
  • Operators: Set up a separate Grafana for operators with data sources for all tenants
  • Since there are no other tenants' data sources within Grafana and the task role cannot read other tenants' workspaces, other tenants are not visible even when admin privileges are granted
    • This incurs the cost and operational overhead proportional to the number of Grafana instances

Closing

This time, I tried separating the storage destinations for Loki / Tempo / AMP by tenant, linking them to Organizations in the Grafana OSS version, and separating the visible data.
I hope this article is helpful to someone.