.. _rml-oob-label:

The OOB TCP Transport
=====================

The OOB ("out of band") transport is the layer beneath the RML API that moves a
message's bytes across a socket to the next daemon in the routing tree.  Where
the RML core (:ref:`rml-label`) decides *what* to send, *to whom*, and *which
way through the tree*, the OOB owns the sockets, the connection handshake, the
on-wire framing, and the retry logic.  The code lives in ``src/rml/oob/``; the
editing-oriented map is in ``src/rml/oob/AGENTS.md``.

One transport, not a framework
------------------------------

The OOB was once an MCA framework with pluggable transport *components*, each
with swappable *modules* — TCP was one, with notional room for others.  PRRTE
only ever shipped TCP, so the framework was collapsed into the single
``src/rml/oob`` directory.  Any language in the code about "trying the next
component," "another transport module," or a selection loop is historical
cruft.  There is exactly one transport: TCP.

Key state
---------

All transport state hangs off one global, ``prte_oob_base``
(``src/rml/oob/oob.h``, defined in ``oob_tcp.c``), and is owned by the progress
thread:

* the list of known **peers** (``prte_oob_tcp_peer_t``), each with its
  addresses, socket, connection state, and send queue;
* the local network **interfaces** the daemon may bind and connect from;
* the **listener** sockets;
* the many TCP tuning MCA parameters — buffer sizes, keepalives, retry limits,
  ``max_msg_size``, and the connection-retry knobs described below.

Each peer is a ``prte_oob_tcp_peer_t`` (``oob_tcp_peer.h``): the peer's name, a
list of candidate addresses, the active address, the socket, a state enum, the
send queue, and retry bookkeeping (``num_retries`` and ``first_attempt``).

The wire header
---------------

Every message on the wire is preceded by a ``prte_oob_tcp_hdr_t``
(``oob_tcp_hdr.h``): the ``origin``, the final ``dst``, the ``tag``, a sequence
number, the payload length, and a message ``type``:

* ``AUTH_HELLO`` / ``AUTH_CHALLENGE`` / ``AUTH_RESPONSE`` and ``IDENT`` /
  ``PROBE`` — the connection handshake, and
* ``USER`` — a normal message.

``origin`` and ``dst`` are **ranks**, not full process identifiers, and a data
message carries no namespace at all — 30 bytes in total.  There is only ever
one namespace to name: an OOB endpoint belongs to a daemon of this DVM and to
nothing else, and every send entry point takes a rank that is resolved in the
sender's own namespace, so the receiver rebuilds both identifiers with its
own.  The connection handshake is the exception, and the reason the rest can
be silent: it does carry a namespace, and a peer presenting any namespace but
ours is refused rather than adopted.

The header is exchanged **only** among daemons of the same DVM, all running the
same build, so it is **not** a stable ABI.  Its layout may change freely, but
every daemon in a build must agree; there is no versioning.

Interfaces and listeners
------------------------

``prte_oob_open`` (``oob_tcp.c``) chooses the local interfaces the transport
will use, and ``prte_oob_tcp_start_listening`` (``oob_tcp_listener.c``) opens
the sockets that accept connections on them.

**Selection.**  Every IPv4 interface (and IPv6, in a build configured with
``--enable-ipv6``) is a candidate, apart from those that look virtual (a name
beginning ``vir``).  The ``prte_if_include`` or ``prte_if_exclude`` MCA
parameter — interface names or CIDR subnets — then narrows the set.  A
loopback address is kept only by the DVM controller, since a daemon on another
node could never reach it, and only in two cases: ``prte_if_include`` selects
it and nothing else, or the host has no other interface at all.  So
``--prtemca prte_if_include lo`` (``lo0`` on macOS) confines a DVM that runs
on a single node to loopback.  An *exclude* list that removes every other
interface is not taken as such a request, and fails at startup.  A DVM that is
confined to loopback but asked to start a daemon on another node fails too:
that daemon finds no usable interface, and the controller ends the launch.

**Listeners.**  One listening socket is opened per selected address — never on
the wildcard address — and all of an address family's sockets share one port.
A peer learns where to connect only from a daemon's contact URI, which names
exactly the selected addresses and a single port for them, so nothing is lost
by binding narrowly; binding the wildcard instead would accept connections on
interfaces the user excluded, and expose a loopback-confined DVM on every
external interface of its host.  The port is the first entry of
``prte_static_ipv4_ports`` / ``prte_static_ipv6_ports`` if given, otherwise the
first port in ``prte_dynamic_ipv4_ports`` / ``prte_dynamic_ipv6_ports`` that
every address can take, otherwise one the kernel chooses.  A family on which no
socket can be opened is dropped from the contact URI.

Sending
-------

``PRTE_OOB_SEND(msg)`` (a macro in ``oob.h``) thread-shifts the send onto the
progress thread, where ``prte_oob_base_send_nb`` (``oob_base_stubs.c``) runs:

#. **Give-up checks.**  If the destination is known down, or the message's retry
   budget (``prte_rml_max_retries``) is exhausted, the message is dropped and
   the failure is reported back through its send callback.

#. **Route.**  ``prte_rml_get_route(dst)`` returns the rank of the next hop.

#. **Find or build the peer.**  If a ``prte_oob_tcp_peer_t`` for that hop already
   exists, use it.  Otherwise obtain the hop's contact URI — directly for the
   HNP, from the PMIx store otherwise, or (in a bootstrapped DVM) synthesized
   from configuration — and build the peer with ``process_uri`` / ``set_addr``.

#. **Queue.**  If the peer is connected, the message is queued for immediate
   transmission (``MCA_OOB_TCP_QUEUE_SEND``).  Otherwise it is queued as pending
   (``MCA_OOB_TCP_QUEUE_PENDING``) and the connection state machine is started.

Once the socket is up and the IDENT handshake has completed, the send handler in
``oob_tcp_sendrecv.c`` writes the header, then the payload, and - on a
connection that tags its messages (see `Message tagging`_ below) - the tag.

The connection state machine
----------------------------

``oob_tcp_connection.c`` drives a peer from unconnected to usable.

Authentication comes first.  A daemon's port answers whatever connects to it,
and a connected daemon is trusted with launch commands, so before anything
else each end proves it holds the *DVM key* - a 256-bit value every daemon of
one DVM shares and nothing else has.  The dialer sends ``AUTH_HELLO`` carrying
a fresh nonce; the acceptor answers ``AUTH_CHALLENGE`` with its own nonce and
its proof; the dialer checks that proof and answers ``AUTH_RESPONSE`` with its
own.  Each proof is an HMAC-SHA256, under the key, of both nonces, both ranks,
the namespace, which end made it, and whether that end tags its messages, so
no proof can be replayed, reflected back, or presented on another connection,
and the key itself never crosses the wire.  Nothing outside the handshake is
touched until a peer has proved itself: no peer object is created and no
existing connection is disturbed.

The key travels to each daemon by a channel private to the DVM's user - never
on a command line, where the namespace and the HNP's contact address already
are.
``prte`` generates it; ``plm/ssh`` writes it down the stdin of each ``ssh`` it
starts; ``plm/slurm``, ``plm/lsf`` and ``plm/pals`` put it in the launcher's
environment, which the resource manager hands to every daemon; and a
bootstrapped DVM reads it from the file ``DVMKeyFile`` names in ``prte.conf``.
A daemon removes it from its environment as soon as it has it, so nothing the
daemon starts inherits it.  The check is always made; nothing turns it off.
For the same reason ``plm/ssh`` refuses ``rsh`` and ``remsh`` as launch
agents: they would carry the key across the network as plain text.

Authentication establishes who is at the other end of a connection.  On its
own it says nothing about the messages that follow; that is what message
tagging adds (see `Message tagging`_).

Then the ``IDENT`` exchange that has always opened a connection: the daemon
that initiated it names itself (and must name the rank it proved); the
acceptor (``recv_handler`` in ``oob_tcp.c``, plus ``oob_tcp_listener.c`` for
the accept socket) checks the identity and either acks or nacks.  Because two
daemons can try to connect to each other simultaneously, the handshake resolves
which socket survives so a pair of peers does not end up with two half-open
connections.

The handshake runs on the progress thread, and the listening port answers
anything that connects to it, so the handshake is read as its bytes arrive
rather than by waiting for them.  A connection that sends part of one and
stops costs its socket, not the daemon - and only for so long, and only so
many of them:

* ``prte_oob_handshake_timeout`` (default 30 seconds) is how long a
  connection has to finish its handshake.  An inbound one is dropped when it
  runs out; one this daemon opened is treated as a failed address, so the
  daemon moves on to the peer's next address, or reports the peer
  unreachable when none is left, instead of waiting on a port that accepted
  the connection and never answered;
* ``prte_oob_max_pending`` (default: half the process's file-descriptor limit)
  bounds how many inbound connections may be part way through their handshake
  at once, counted from the moment ``accept()`` returns them, and
  ``prte_oob_max_pending_per_host`` (default 64) how many of those may come
  from any one host.  An IPv6 host is identified by the leading
  ``prte_oob_pending_ipv6_prefix`` bits of its address (default 64), since one
  host may use any address in its prefix.

Should ``accept()`` nonetheless run out of descriptors or memory, the listener
pauses for a second and tries again; it used to close for good, after which
no daemon could ever join, rejoin or reconnect.

Retry and backoff.  When a connect attempt finds no listener yet — a common race
during startup — ``prte_oob_tcp_peer_try_connect`` schedules a retry.  The base
behavior is a fixed ``retry_delay``-second wait, bounded by
``max_recon_attempts``.  Two MCA parameters modify this, both defaulting to the
original behavior:

* ``prte_retry_max_delay`` — when larger than ``retry_delay``, the delay backs
  off exponentially (``retry_delay``, 2×, 4×, …) capped at ``retry_max_delay``,
  so a daemon waiting on an absent peer polls quickly at first and then settles
  onto a steady rate rather than busy-spinning.  The exponent is clamped before
  the shift so an unbounded retry count cannot overflow.

* ``prte_connect_max_time`` — caps how long a **non-lifeline** peer is chased
  (measured from the peer's ``first_attempt`` timestamp) before giving up so the
  routing tree can heal to an ancestor.  ``0`` means retry forever.  The HNP is
  never subject to this cap; it is always retried forever.

Receiving and relaying
----------------------

When a peer's socket has delivered a complete message - and, on a connection
that tags its messages, its tag has checked out - the recv handler in
``oob_tcp_sendrecv.c`` inspects ``hdr.dst``:

* **For us.**  It calls ``PRTE_RML_POST_MESSAGE`` to hand the message up to the
  RML matching layer (:ref:`rml-label`).

* **Not for us.**  This daemon is an intermediate hop.  The handler rebuilds a
  ``prte_rml_send_t`` with the same ``origin`` / ``dst`` and re-enters
  ``PRTE_OOB_SEND``, so relaying is simply "send again from here," routed on
  toward the destination.

Connection loss.  When a socket drops, ``oob_tcp_component.c`` fires its
``lost_connection`` / ``failed_to_connect`` handlers, which turn a transport
event into a routing event: the RML is told to treat the peer as lost so the
tree can be repaired (see :ref:`rml-label`).

Message tagging
---------------

Every message on a daemon connection is tagged, so that the receiving daemon
acts on a message only once it has checked that it is the next message the
far end sent on that connection, unchanged.  The keys come from the
handshake above.  Tagging is always on: every hop computes a tag on each
message it sends and checks one on each message it receives, which costs a
few percent of a large launch and nothing measurable in a collective.

* **Keys.**  When a connection's handshake succeeds, both ends derive a
  connection key - an HMAC-SHA256, under the DVM key, of the same namespace,
  ranks and nonces the proofs cover, under a label of its own - and from it
  one key for each direction.  The nonces are fresh on both sides, so no two
  connections share a key, even between the same two daemons; and with a key
  per direction a message cannot be sent back to the end that tagged it.

* **Tags.**  Each direction numbers its messages from 0.  A message's tag is
  a 16-byte Poly1305 tag, under a key made from the direction's key and the
  message's number, over the header as it goes on the wire and then the
  payload; it follows the payload on the wire.  Because each message has a
  key of its own, a message that is repeated, sent out of order, or left out
  is checked under a key that did not tag it.

* **Checking.**  The receiver checks the tag before it does anything else
  with the message - before the incarnation check, before delivering it, and
  before relaying it.  A message whose tag does not match closes the
  connection, with a message naming both daemons and the message's number:
  once one message cannot be accounted for, nothing later on the same stream
  can be relied on either.

* **Relays.**  A daemon that relays a message sends it again as a new message
  on its own connection to the next hop, so every hop tags and checks under
  the keys of the connection the message is on.

Confidentiality
^^^^^^^^^^^^^^^

PRRTE authenticates its daemons' connections and their messages; it does
not encrypt them.  Where the traffic between nodes has to
be kept private, provide that at the network layer: an isolated fabric, or
IPsec or WireGuard between the nodes.

Which thread moves the bytes
----------------------------

A daemon runs a pool of **worker progress threads**, sized by
``prte_num_worker_threads`` (default ``8``).  It is one pool for the whole
process, shared with the local fork/exec path; the OOB does not have a pool of
its own.  Each peer is assigned a base from that pool, in rotation, when the
peer is created.

Only the peer's **send and recv socket handlers** run there, and only once the
connection is established.  Everything those handlers hand onwards —
delivering a message to the RML, relaying one on, running a send's completion
callback, reacting to a lost connection — comes back to the main progress
thread, as does the whole connection state machine.

The reason to want this is not raw bandwidth.  A single thread copies into the
kernel several times faster than a 10-25 GbE link drains, so the transport is
not thread-starved for throughput.  What it is short of is *occupancy*: while
the main thread is compressing a broadcast, registering a namespace, or forking
local processes, no socket is being serviced at all, and the link simply idles
for that fraction of the launch.  Giving the sockets their own threads keeps
them busy across those stalls.  It also lifts a real ceiling on 100 GbE and on
nodes with several NICs, where one thread's copy rate *is* the limit.

Setting ``prte_num_worker_threads`` to ``0`` puts every peer back on the main
progress thread — the behavior that predates the pool — and is the thing to
try when you suspect the extra threads are costing more than they earn, which
they can on a node with no spare cores.

Bootstrap specifics
-------------------

In a launcher-less (bootstrapped) DVM the daemons boot independently, so the
transport cannot rely on a launcher having pre-distributed everyone's contact
information:

* **URIs are synthesized on demand.**  With ``prte_bootstrap_setup`` set and no
  peer object present, ``prte_oob_base_send_nb`` derives the next hop's URI from
  configuration via ``prte_ess_base_bootstrap_peer_uri`` — the same synthesis a
  prted uses for its original parent — rather than reading a nidmap that was
  never distributed.  This also covers a healed lifeline whose new parent (a
  former grandparent) was never pre-synthesized.

* **A missing interface mask is tolerated.**  A synthesized URI cannot know the
  peer's interface mask, so ``set_addr`` treats a missing or empty mask as
  ``/0`` (universally reachable) instead of rejecting the address.

* **A not-yet-present parent is not fatal.**  Because a parent may simply not be
  up when a child times out on it,
  ``prte_mca_oob_tcp_component_failed_to_connect`` (in bootstrap mode) heals the
  tree the way a lost *live* parent would: ``prte_rml_route_lost`` promotes the
  daemon to the next ancestor and returns a ``COMM_FAILED`` recovery, instead of
  raising a fatal ``FAILED_TO_CONNECT``.  The HNP is the exception — it is
  retried forever and never allowed to time out.

Where to look
-------------

* Send entry and routing: ``oob_base_stubs.c``.
* Connection handshake, retry, backoff: ``oob_tcp_connection.c``,
  ``oob_tcp_listener.c``, and ``recv_handler`` in ``oob_tcp.c``.
* Socket I/O and the deliver-vs-relay decision: ``oob_tcp_sendrecv.c``.
* Transport-to-routing fault events: ``oob_tcp_component.c``.
* The wire header: ``oob_tcp_hdr.h``.
