As we move down the path of changing from RabbitMQ to Azure Service Bus we wanted to implement a blue/green deployment pattern for our services in Azure Service Bus. But we started to run into unexpected duplicate messages.
The thought process was that a message comes into a top-level topic, call it
inbound. From there it is delivered to one of 2 topics via subscriptions to
inbound-green. We then have another topic, we'll call it
processor, that has a subscription with filtering to both
inbound-green. When we need to make a change to the subscriptions/flow we make it on the inactive flow (in this case it's green), verify the change, then flip the flow to the green color and update blue.
To handle blue/green of the services in much the same manner, we were going to have 2 queues, a blue and green. We'll call these
processor-green. We were going to have the
processor topic forward the messages to either
processor-green, depending on the color taking the load.
The first thing we tried was
This flow worked. Messages were flowing quickly and without any issue. The problems came when we tried to change the colors. There was a short period of time, about 5-10 seconds, where messages were going to both blue and green flows or queues.
We tried a number of approaches. None of them worked.
A few of the things we tried
- Drop the subscription from
processor-colorand create a new one, worked, but we lost messages because the subscription didn't exist.
- Don't drop the subscription, create a new one then remove the old one. Duplicate messages while both subscriptions were in place.
forward-toon the subscription from blue to green. Duplicate messages for about 5-10 seconds.
- Change the
processortopic to a queue and set
forward-toto blue/green, duplicate messages, though, the time dropped to 2-3 seconds.
- Modify the subscription between
processorand set a header to the appropriate color. Then the
processor-colorsubscriptions had a filter looking for that color. We lost messages during the time the subscription was updating. This was also very complex and difficult to manage the different subscriptions all over the place. It was a nightmare to even try it. Even if it did work, we would have scrapped it.
The one that worked for us was to change the
processor topic to a queue and set the
processor-green. Here is the difference between when it did not work in my list above. Do not immediately set the
forward-to to the color you want. Set it to nothing or disable it. Wait about 5 seconds and then set it to the correct queue or topic. When the
forward-to is disabled the queue stores the messages, just like a good little queue. When it is set, any message currently stored in the queue are sent to the new destination.
This solution will only work with queues determining the flow. Reason being is that if a topic is not configured with a forward-to or any subscription the messages will end up in the dead letter queue or worse, completely dropped.
If your system can handle duplicate messages for a short period of time use the
forward-to on either a subscription, queue or topic and just set it to the correct target and live with having duplicates during a change over.
If your system cannot handle duplicate messages, use a queue and
forward-to. To flip the destination set it to nothing, wait 5 seconds, and set it to the correct target.