Duplicate messages when changing forward-to in Azure Service Bus

To handle blue/green of the services in much the same manner, we were going to have 2 queues, a blue and green.

As we move down the path of changing from RabbitMQ to Azure Service Bus we wanted to implement a blue/green deployment pattern for our services in Azure Service Bus. But we started to run into unexpected duplicate messages.

The thought process was that a message comes into a top-level topic, call it inbound. From there it is delivered to one of 2 topics via subscriptions to inbound-blue and inbound-green. We then have another topic, we'll call it processor, that has a subscription with filtering to both inbound-blue and inbound-green. When we need to make a change to the subscriptions/flow we make it on the inactive flow (in this case it's green), verify the change, then flip the flow to the green color and update blue.

To handle blue/green of the services in much the same manner, we were going to have 2 queues, a blue and green. We'll call these processor-blue and processor-green.  We were going to have the processor topic forward the messages to either processor-blue or processor-green, depending on the color taking the load.

The first thing we tried was topic inbound-> topic inbound-blue-> topic processor-> queue processor-green

This flow worked. Messages were flowing quickly and without any issue. The problems came when we tried to change the colors. There was a short period of time, about 5-10 seconds, where messages were going to both blue and green flows or queues.

We tried a number of approaches. None of them worked.

A few of the things we tried

  • Drop the subscription from processor to processor-color and create a new one, worked, but we lost messages because the subscription didn't exist.
  • Don't drop the subscription, create a new one then remove the old one. Duplicate messages while both subscriptions were in place.
  • Change forward-to on the subscription from blue to green. Duplicate messages for about 5-10 seconds.
  • Change the processor topic to a queue and set forward-to to blue/green, duplicate messages, though, the time dropped to 2-3 seconds.
  • Modify the subscription between inbound-color and processor and set a header to the appropriate color. Then the processor-color subscriptions had a filter looking for that color. We lost messages during the time the subscription was updating. This was also very complex and difficult to manage the different subscriptions all over the place. It was a nightmare to even try it. Even if it did work, we would have scrapped it.

The one that worked for us was to change the processor topic to a queue and set the forward-to to processor-blue or processor-green.  Here is the difference between when it did not work in my list above. Do not immediately set the forward-to to the color you want. Set it to nothing or disable it. Wait about 5 seconds and then set it to the correct queue or topic. When the forward-to is disabled the queue stores the messages, just like a good little queue. When it is set, any message currently stored in the queue are sent to the new destination.

This solution will only work with queues determining the flow. Reason being is that if a topic is not configured with a forward-to or any subscription the messages will end up in the dead letter queue or worse, completely dropped.

In conclusion

If your system can handle duplicate messages for a short period of time use the forward-to on either a subscription, queue or topic and just set it to the correct target and live with having duplicates during a change over.

If your system cannot handle duplicate messages, use a queue and forward-to. To flip the destination set it to nothing, wait 5 seconds, and set it to the correct target.