Scale IoT backends horizontally with MQTT Shared Subscriptions

In IoT architectures, one core requirement is the ability to process all the messages sent by the connected devices in a scalable manner.

Choosing the right broker is a crucial part of the decision that will help achieve this goal, as discussed in my last post. However, it's equally important to select the appropriate strategy for consuming all the data processed by the message broker.

Whenever possible, I recommend using message broker rules. Depending on the message broker, they can be more or less customizable. From the power of the EMQX Rule Engine to the flexibility of AWS IoT.

One common use for message broker rules is to create different layers of message brokers, the will publish to other message brokers depending on the kind of message received from the devices. This allows the platform to scale the processing of each message type independently. For example, this is what we used when we collaborated on the Spin project.

When message broker rules do not fit your needs, you can always rely on messaging protocol strategies to scale the message processing.

MQTT Shared Subscriptions

MQTT 5.0 introduced shared subscriptions. All IoT message brokers that are fully compliant with MQTT 5.0, such as Mosquitto or EMQX, support shared subscriptions.

Shared subscriptions enable multiple MQTT clients to subscribe to the same topic. However, instead of each client receiving every message sent by a device, the message broker will load balance and distribute the messages among all subscribers using various configurable strategies.

Using shared subscriptions is quite straightforward. Here is the code of one of our applications, built with Node.js and TypeScript, using the mqtt.js client.

  const client = connect(brokerUrl, {
    username,
    password,
  });

  const topic = `$share/consumer-servers/${topicName}`;

  client.on('connect', () => {
    client.subscribe(topic, (error: unknown) => {
      error
        ? logError('Could not subscribe to topics', error)
        : logSuccess('Subscribed to topics');
    });
  }

All subscribers that start with $share/group-name/ will form a group that will be load balanced by the message broker. This allows us to scale all the nodes of the consumer-servers group as much as we need, depending on the devices connected and the time required to process every message.

If we need to scale different groups independently, we can do it by defining more groups, such as $share/expensive-consumer-servers/

Using MQTT Shared subscriptions instead of message broker rules has allowed us to implement complex behaviours in a multitenant application in a simple way, while reusing part of the data layer backend code developed in the frontend and the API of our application.

If you have any doubts on what strategy could be the most appropiate for your IoT application, write me a message on X and I can share my experience with you!