The modbee command-line tool

One binary that publishes, subscribes, benchmarks and runs the broker, built
on this library's own client. It reports what the broker said about every
operation, not just that something was sent.

Getting it

From the Docker image, with nothing installed:

docker run --rm ghcr.io/mod-bee/modbee-mqtt pub -h broker.example.com -t test -m hi -q 1
docker run --rm -it ghcr.io/mod-bee/modbee-mqtt sub -h broker.example.com -t 'test/#' -v

Given a command, the image runs modbee <command>; given none, it runs the
broker. Inside a running compose stack, docker compose exec modbee modbee ...
reaches that stack's broker on localhost. To reach a broker on your own
machine from docker run, add --network host.

From source:

cmake -S . -B build && cmake --build build -j
./build/cli/modbee --help
sudo cmake --install build          # optional: puts modbee in /usr/local/bin

On Linux every command is built. On macOS, pub, sub and req are, and
bench, serve and check are not, because the broker tier is Linux only.
serve includes the Kafka pipe when the librdkafka development
package (librdkafka-dev) is installed. Without it, serve is the broker
alone, and refuses to start on a modbee.yaml that has Kafka mappings rather
than quietly ignore them.

Why the Arduino library does not have it. The command lives in cli/ and
the server in server/, outside src/, which is the only folder the Arduino
IDE compiles. Nothing of either reaches a board, and neither costs a byte of
flash. A board can still be a broker - that is MqttBroker, in the library
(broker.md).

Exit codes, for scripts:

Connecting

Every client command takes the same options.

Publishing

modbee pub -t plant/tank-01/level -m 4.25 -q 1
#1 plant/tank-01/level: delivered to the broker

modbee pub -t nowhere/at/all -m x -q 1
#1 nowhere/at/all: accepted, but no matching subscribers

• QoS 1 and 2 publishes end with the broker's own answer. That includes
  the refusals MQTT 5 can express: not authorised, quota exceeded, payload
  format invalid. At QoS 0 there is no answer, so the tool reports the message
  as written.
• Payload sources: -m (text), -f (a file, sent as bytes), -l (one
  message per line of standard input) and -n (a zero-length payload, which
  clears a retained message).
• --count N --interval ms repeats a message, and every repeat is sent.
  It uses publishSeries(), because the ordinary publish() replaces a
  queued message to the same topic, which is right for a setpoint and wrong
  for "send this ten times".
• MQTT 5 properties:
  • --content-type, --response-topic, --correlation
  • --expiry (seconds)
  • --utf8 (Payload Format Indicator)
  • --user-property key=value, repeatable

Subscribing

modbee sub -t 'plant/#' -q 1 -v
subscribed plant/#: granted
plant/tank-01/level 4.25

• -t repeats for several filters.
• -C N exits after N messages, and -W s after N seconds.
• MQTT 5 subscription options: --no-local, --retain-as-published,
  --retain-handling 0|1|2.
• Binary payloads, such as Sparkplug B protobuf, print as base64 rather
  than wrecking the terminal.
• With --json, each message is one line. Text payloads appear as
  payload and binary ones as payload_base64. Properties are included, and
  user properties are an array of pairs, because MQTT 5 allows a key more than
  once:

{"topic":"demo/b","qos":1,"retain":false,"payload":"second","properties":{"content_type":"text/plain","user_properties":[["site","nz"],["site","au"]]}}

Request and response

modbee req -t svc/echo -m ping
echo:ping

1. Subscribes to a response topic (modbee/reply/<id> by default, or -e).
2. Waits for the SUBACK.
3. Publishes the request with that topic and a unique correlation value.
4. Prints the first reply carrying the same correlation value. A reply to
   somebody else's request is ignored.

It needs MQTT 5, because 3.1.1 has neither property.

Serving

modbee serve -C modbee.yaml                              # everything from the file
modbee serve -c 200000                                   # plain MQTT on 1883, one shard per core
modbee serve --tls-port 8883 --cert broker.pem --key broker.key
modbee serve --no-plain --tls-port 8883 --cert broker.pem --key broker.key --client-ca devices-ca.pem
modbee serve --tls-port 8883 --cert broker.pem --key broker.key --client-ca devices-ca.pem --client-certificates optional

• One broker per core, on one port. Each shard is an MqttBroker on its
  own thread with its own listener (SO_REUSEPORT), and a session lives on the
  shard that owns its Client Identifier. A connection that lands elsewhere is
  handed over at its CONNECT. Clients cannot tell. server/ShardedBroker.h
  has the design.
• -c and --max-sessions are for the whole server. Each shard is sized
  for its share, plus a quarter, because the hash spreads evenly on average.
• --username and --password require that login of every MQTT client.
  For a login per device, changed while the broker runs, use
  authentication: database: in the file (logins.md).
• --max-retained (10,000 by default) is how many retained topics the
  server keeps; every shard holds all of them.
• --retained-file /var/lib/modbee/retained.db keeps them in a file as
  well, so they survive a restart; --retained-store kafka keeps them in
  Kafka (with a kafka: section in the file). Memory only is the default.
  docker.md compares the three.
• TLS listens beside plain TCP on --tls-port, or alone with
  --no-plain. On OpenSSL the shards share one TLS context, so a session
  ticket from one is honoured by all of them; on wolfSSL each shard has its
  own (security.md). --client-ca makes it mutual TLS: every client
  must present a certificate that CA signed - or, with
  --client-certificates optional, may, and logs in with a password if it
  does not. The start-up line names the engine:
  tls 8883 (wolfSSL, client certificates required). With
  authentication: certificate-login: true the certificate is the login
  (logins.md), and the
  certificate, key and client CA can be changed while it runs
  (settings.md).
• Shared subscriptions work across shards. Each message to
  $share/<name>/<filter> reaches one member of the group in the whole
  server. The shard it was published on chooses which shard delivers it,
  rotating over every member, and that shard picks one of its own members.
• -C modbee.yaml runs a configuration file; flags given as well
  override it. The file's format, and the Kafka pipe it configures, are in
  kafka.md. modbee check -C modbee.yaml validates a file
  without starting anything, and exits 0 only if it would load.
• --metrics-port 9464 (or a metrics: section) serves Prometheus
  /metrics, /healthz (the process answers) and /readyz (it is ready
  for clients - kafka.md).
• Settings kept in the database. With authentication: database:, what
  modbee config has stored goes over the file, and under the environment
  and these flags (settings.md).

What a connection costs, measured on Linux with idle connections:

The TLS figures already include the engine releasing its idle record
buffers; without that, OpenSSL is 63 KB. At a million devices TLS is most of
the memory, so size the host for it.

Benchmarking

modbee bench conn -c 50000 -T 8 -R 2000 -d 30 --bind 127.0.0.2-127.0.0.5 --rss-pid $(pgrep -x my-broker)
modbee bench sub  -c 10 -t bench -q 1 -d 60
modbee bench pub  -c 1000 -r 10 -s 100 -q 1 -d 30

• conn opens N connections at -R per second and holds them for -d
  seconds. A few probe clients (--probes) publish at QoS 1 throughout and
  time the broker's answer, which is how loaded a broker feels.
  --rss-pid reports the broker's resident memory per connection.
• pub and sub: payloads carry the publisher id, a sequence number and
  the send time.
  • The subscriber reports p50, p99 and p999 latency, and counts loss and
    duplicates exactly, per subscriber and publisher.
  • Start subscribers first and stop publishers first, because loss is judged
    within the range of sequences each subscriber saw.
• Every client is a real MqttClient. Each thread runs a slice of them
  over one epoll set, and visits idle ones at least every --sweep-ms so
  keepalive runs.
• --bind spreads connections over several source addresses. Linux's
  default ephemeral port range gives about 28,000 connections per source
  address towards one broker port. On one machine, 127.0.0.2-127.0.0.12
  needs no setup, because all of 127/8 is local.
• Failures are reported by reason (connection lost, refused, timed out),
  never as a bare count.
• --lean (conn only) holds each connection as a raw MQTT 3.1.1
  CONNECT and PINGREQ of a few bytes, instead of a full client. That is how one
  machine opens a million connections. The probes stay full clients.
• --tls dials TLS and does not verify the broker, unless --pin gives
  its key fingerprint. It cannot be combined with --lean or --bind.