Can Azure Service Bus Duplicate Detection Prevent Double Processing? Testing Resends with the Same MessageId and Redelivery

Can Azure Service Bus Duplicate Detection Prevent Double Processing? Testing Resends with the Same MessageId and Redelivery

I tested Azure Service Bus duplicate detection hands-on. A resend with the same MessageId is dropped even though the send succeeds, even after the first is received, within the time window. If receiving stops before completion, the message is redelivered.
2026.11.22

This page has been translated by machine translation. View original

Hello! I'm Yuji Nishimura from the Operations Department!

When making order registration asynchronous with an Azure Service Bus queue, is it safe to assume that "if I enable duplicate detection, there's no need to worry about orders being registered twice"?

I checked on a real environment: when I resent with the same MessageId, the send succeeded but the second message did not enter the queue. On the other hand, when the receiving app stopped receiving before notifying completion, the same message was delivered once more. Duplicate detection prevents duplicate sends. You need to prepare separately a design in which the result is not doubled even if the receiving app processes the same order repeatedly.

Separating send retries from redelivery after receipt, I tested on a Standard namespace created in the East Japan region on November 10, 2026. I used Azure CLI 2.91.0, Python 3.13, and azure-servicebus 7.15.0.

Summary of what I tried

What I tried Result
Create a queue with duplicate detection enabled in a Basic namespace Could not be created (RequiresDuplicateDetection cannot be set in Basic)
Send twice with the same MessageId Both sends succeeded. Only the first entered the queue
For the same order, send a different operation (cancel) with a different MessageId Accepted
Send twice with a random MessageId each time Both messages were accepted
After receiving and completing the first message, send the same MessageId within the time window The send succeeded, but the message was discarded
Send the same MessageId outside the time window (1 minute) Accepted
Receive in PeekLock and stop receiving without notifying completion In a receive after the lock expired, the same message was redelivered (the Python SDK's delivery_count went from 0 → 1)

It couldn't be created in Basic, so I prepared a queue with a 1-minute time window in Standard

Duplicate detection is available in Standard and Premium. When I tried to create a queue with duplicate detection enabled in a Basic namespace, it resulted in an error.

SubCode=40000. The property 'RequiresDuplicateDetection' cannot be set when creating a Queue because the namespace '<namespace>' is using 'Basic' tier.

In a Standard namespace, I created the queue orders with duplicate detection enabled. To see results quickly, I set the duplicate detection time window to 1 minute and the lock duration to 30 seconds. Partitioning is disabled.

az group create --name <resource-group> --location japaneast
az servicebus namespace create --resource-group <resource-group> --name <namespace> --location japaneast --sku Standard
az servicebus queue create --resource-group <resource-group> --namespace-name <namespace> --name orders \
  --enable-duplicate-detection true --duplicate-detection-history-time-window PT1M \
  --lock-duration PT30S --enable-partitioning false

# 送受信に使う接続文字列(名前空間の RootManageSharedAccessKey)を環境変数に入れる
export SB_CONN="$(az servicebus namespace authorization-rule keys list --resource-group <resource-group> \
  --namespace-name <namespace> --name RootManageSharedAccessKey --query primaryConnectionString -o tsv)"

I did sending and receiving with Python's azure-servicebus. The scripts included in this article are the ones I actually used, with the log formatting removed.

A resend with the same MessageId was discarded even though the send succeeded

The sender resends the same request when it can't tell whether it got through, for example when no response comes back. To simulate that resend, I sent twice with the same MessageId (order-300/register).

send_twice.py
import os
from azure.servicebus import ServiceBusClient, ServiceBusMessage

with ServiceBusClient.from_connection_string(os.environ["SB_CONN"]) as client:
    with client.get_queue_sender("orders") as sender:
        for body in ["register order-300 (1st send)", "register order-300 (retry)"]:
            sender.send_messages(ServiceBusMessage(body, message_id="order-300/register"))

No exception was raised either time, and both sends succeeded. When I checked the queue's message count about 17 seconds later, it was 1.

az servicebus queue show --resource-group <resource-group> --namespace-name <namespace> --name orders \
  --query '{active:countDetails.activeMessageCount, dup:requiresDuplicateDetection, window:duplicateDetectionHistoryTimeWindow}'
{
  "active": 1,
  "dup": true,
  "window": "PT1M"
}

The queue overview in the portal also showed duplicate detection enabled and one active message (red frames added by me).

Overview screen of the Service Bus queue "orders" in the Azure portal. Duplicate detection Enabled, duplicate detection time window 1 minute, message lock duration 30 seconds, and active 1 message are each enclosed in red frames.

The official explanation of duplicate detection also says that Service Bus records the MessageId of messages sent to a queue or topic for the duration of the time window. A message resent with the same ID during that time is discarded after returning success to the sender. It is not a mechanism that compares the body to decide whether it's the same order. When partitioning is enabled, the decision is made on the combination of MessageId and PartitionKey (I tested with it disabled).

Service Bus duplicate detection is similar in mechanism to the deduplication ID of an Amazon SQS FIFO queue. The SQS documentation also says that subsequent messages with the same ID are accepted but not delivered to consumers. The length of the time window (5 minutes for SQS) and where it's configured differ, so I mention it only as an analogy for the concept.

How the ID was assigned changed how duplicates were handled

Before this test, I tried varying how IDs were assigned on an empty queue. After sending the following five and then receiving from the queue, I was able to retrieve 4 messages (all received messages were completed).

Message sent MessageId Result
Registration of order-123 (1st time) order-123/register Accepted
Registration of order-123 (resend) order-123/register Discarded
Cancellation of order-123 order-123/cancel Accepted
Registration of order-456 (1st time) Random ID Accepted
Registration of order-456 (resend) A different random ID Accepted (not treated as a duplicate)

If you assign a new random ID on every resend, Service Bus sees them as different messages. For resends of the same order registration, you need to keep the same ID, as in order-123/register. Conversely, even for the same order, "registration" and "cancellation" are different requests, so if you use only the order number as the shared ID, there is a risk that even necessary messages will be discarded.

Even after receipt, it was discarded within the time window

Immediately after emptying the queue (about 13 seconds after the first send), I sent the same order-123/register again. The send succeeded, but the only message I could retrieve in the subsequent receive was the one sent after the time window had passed (73 seconds after the first send). The log times are in Japan time.

01:49:28 send ok: order-123/register | register order-123 (1st send)
01:49:41 drained: 4
01:49:41 send ok: order-123/register | register order-123 (resend after consumed)
01:49:43 peek: 0 message(s) []
01:50:41 send ok: order-123/register | register order-123 (resend after window)
01:50:44 peek: 1 message(s) [('order-123/register', 'register order-123 (resend after window)')]
01:50:47 drained: 1

drained is the number of messages received and completed. A peek (an operation that looks at messages without taking their contents) doesn't necessarily return all messages at once, so I used the drained count as the basis for the numbers.

Service Bus remembers the same ID for the duration of the time window even after the first message has been processed. According to the official explanation, the time window defaults to 10 minutes, with a minimum of 20 seconds and a maximum of 7 days. Enabling duplicate detection and the size of the time window affect throughput, and the smaller the time window, the less the impact. The length of the time window needs to be decided based on the period during which resends may occur and the load.

Without notifying completion, the same message arrived again

With PeekLock, the message is locked on receipt, and when processing is finished you notify completion with Complete (complete_message in the Python SDK). If you don't specify a receive mode, receiving in the Python SDK uses PeekLock.

I sent order-789/register, received it in PeekLock, and closed the receiver without calling complete_message. I waited 35 seconds and then received again.

receive_without_complete.py
import os, time
from azure.servicebus import ServiceBusClient

with ServiceBusClient.from_connection_string(os.environ["SB_CONN"]) as client:
    with client.get_queue_receiver("orders", max_wait_time=5) as receiver:
        m = receiver.receive_messages(max_message_count=1)[0]
        print(m.message_id, m.delivery_count, m.locked_until_utc)
        # complete_message を呼ばずに受信側を閉じる
    time.sleep(35)
    with client.get_queue_receiver("orders", max_wait_time=10) as receiver:
        m = receiver.receive_messages(max_message_count=1)[0]
        print(m.message_id, m.delivery_count)
        receiver.complete_message(m)

The log times are in Japan time, and only locked_until is in UTC (01:51:19 in Japan time, 30 seconds after receipt).

01:50:49 received: order-789/register delivery_count: 0 locked_until(UTC): 16:51:19
01:50:49 simulate crash: close receiver without Complete
01:51:26 received again: order-789/register delivery_count: 1 | register order-789
01:51:26 completed

In a receive after the lock expired, the same order-789/register arrived once more, and delivery_count increased from 0 to 1. The sender did not resend anything. Duplicate detection is a send-time mechanism, so it cannot prevent this redelivery.

This time, I simulated a state in which the receiving app stopped midway through processing by closing the receiver's connection. I did not test abnormal process termination or network disconnection. I also did not test whether the message could be received before the lock expired, so I have not determined whether the lock was released when the receiver was closed or whether it waited until expiry.

Registering the order in the DB and notifying Service Bus of completion (Complete) are separate operations. If the receiving app stops right after the order registration succeeds and before Complete reaches Service Bus, the same message may arrive after the lock expires and the order registration may run again. When processing is long, you can renew the lock to extend its expiry, but if the app stops right after the DB update, the process that would renew it no longer exists, so the lock expires and the message is redelivered.

The following diagram is a conceptual diagram I made myself, not an official architecture diagram. It shows the boundary between duplicate detection on the sending side and redelivery on the receiving side.

A self-made conceptual diagram of send duplicate detection and receive redelivery. The sender resends to the queue with the same ID, and on the receiving side, if the completion notification after DB registration doesn't arrive, the message is redelivered after the lock expires.

The receiving side needs the property that the result is not duplicated even when the same request is reprocessed, that is, idempotency.

For order registration, consider ID consistency and the DB commit scope

The Idempotent Consumer pattern shows a method of committing the processed ID and the business update in the same transaction. Merely "checking whether it has been processed" cannot handle concurrent processing or mid-process stops.

For order registration, the following four points are items to check in the design. The first row, for the sending side, is based on this test's results; the three rows for the receiving side are design examples based on the official pattern and are not implementations verified on a specific DB.

Check point Approach for order registration State to avoid
Can the same request be sent with the same ID even on resend? (sending side) Keep an ID that combines the order and the operation, such as order-123/register, across resends Assigning a random ID on every resend. Using only the order number as the ID
Can the same request be identified? (receiving side) Use the ID of the order registration operation, which doesn't change on redelivery (the MessageId was the same in this redelivery too) Creating a different ID each time it's received
Even if two processes handle the same message at the same time, can double registration be prevented? (receiving side) Put a unique constraint or similar on the processed ID Both judge it "unprocessed" and register
Are the business update and the processed record committed together? (receiving side) Include the order registration and the processed record in the same DB transaction Only the registration succeeds and the process stops before recording

With this form, when a message is redelivered, you can look at the processed record and Complete it without repeating the same order registration. Conversely, if you commit only the processed record before the DB update, a stop right after that would cause an unregistered order to be skipped.

When to delete the processed records is also a separate decision. Simply matching the sending-side time window of Service Bus is not enough. In addition to the period during which receive retries continue, you also need to consider when an operator will resubmit from the dead-letter queue, where messages that exceeded the maximum delivery count (10 in this case) are set aside.

External payment APIs and email sending usually cannot be included in that DB transaction. Even if the order DB can prevent double registration, the external calls are not necessarily made just once. The idempotency key on the callee's side, and the methods of reconciliation and recovery when the result is unknown, need to be designed separately.

Summary

Duplicate detection discards send retries, but it does not stop redelivery after receipt. This time as well, a resend with the same MessageId was discarded within the time window, and a message for which receiving stopped without notifying completion was redelivered in a receive after the lock expired.

In design reviews, it seems good to check the two boundaries separately. On the sending side, check "when resending because the result is unknown, can the same request be sent with the same ID?"; on the receiving side, check "even if it stops right after the business update, do we commit the processed record together with the business update so that the next receive doesn't duplicate the result?"

Microsoft Azure の利用費割引・活用サポート提供中!

クラスメソッドは、Microsoft Azureのうち特に生成AI領域に強みがあります。AWSの総合支援をメインに5,000社以上支援してきた実績をベースに、特にマルチクラウドでビジネスを最大化したいお客様に引き合いを頂いております。

Microsoft Azure 請求代行・技術支援サービスを詳しく見る

Share this article