Proxylity

Payload Formatters

A formatter decides how the bytes of a UDP payload are represented in the Data field that your AWS resources receive, and how the Data you send back is turned into bytes on the wire. Simple formatters encode the bytes as text. Protocol-aware formatters (coap, coap+ and syslog) parse each packet into a JSON object, so your code works with fields instead of bytes, and encode JSON replies back into valid protocol messages.

On this page

• Where formatters apply
• Choosing a formatter
• base64, hex, utf8 and ascii
• coap
• coap+ (content-type aware CoAP)
• syslog
• Errors and invalid data

Where formatters apply

Set the formatter with the Formatter property of a Destination, or per message in a Packet Source message. The same formatter is used in both directions:

• Inbound: each packet's payload is formatted into the Data field of the request packet. The envelope's Formatter field names the formatter that was used.
• Replies: for destinations that reply (Lambda, EXPRESS Step Functions and API Gateway), the Data of each reply is decoded with the destination's formatter before it is sent.
• Packet Sources: each message's Data is decoded with that message's Formatter (default base64) before it is sent.

With base64, hex, utf8 and ascii, Data is a JSON string. With the protocol-aware formatters (coap, coap+ and syslog), Data is a JSON object: read its fields directly, and return an object (not a string) as the Data of replies and Packet Source messages. Because the fields are part of the event itself, services can act on them without code: DynamoDB stores Data as a map attribute, EventBridge rules and SNS filter policies can match on fields such as Data.severity, and Athena can read Data as a struct. CloudWatch Logs destinations write each packet as a JSON request packet, so Logs Insights and metric filters can use the fields too.

When Formatter is omitted, CloudWatch Logs destinations use utf8 and all others use base64. A FilterExpression is evaluated against the raw packet bytes before formatting, so filters work the same way whichever formatter you choose.

The service-specific key formatters (TenantIdFormatter, MessageGroupIdFormatter, MessageDeduplicationIdFormatter, PartitionKeyExpressionFormatter and TopicSuffixFormatter) are separate settings. They convert bytes extracted with a BREX expression into a string and accept only hex, base64, utf8 and ascii.

Choosing a formatter

base64, hex, utf8 and ascii

These formatters encode the payload bytes without interpreting them. For a 6-byte payload 68 65 6C 6C 6F 0A (the text hello and a newline), Data is:

Hex is often the most convenient choice in Step Functions: Step Functions can't work with binary content, and JSONata can slice a hex string by byte offset (two characters per byte) with $substring, whereas base64 has to be decoded first.

coap

The coap formatter decodes each binary CoAP (RFC 7252) message into JSON and encodes JSON replies back into CoAP. Payload bytes are always base64; Content-Format is left in option 12 and not interpreted. Use it when your code decodes payloads itself, or when you need to forward payload bytes exactly.

Request shape

For example, a confirmable POST /readings carrying a CBOR payload (Content-Format 60) arrives as:

{
  "Version": 1,
  "Type": 0,
  "Code": 2,
  "MessageId": 4660,
  "Token": "qrs=",
  "Options": [
    { "Number": 11, "Value": "cmVhZGluZ3M=" },
    { "Number": 12, "Value": "PA==" }
  ],
  "Method": "POST",
  "Path": "/readings",
  "Payload": "ogBpc2Vuc29yLTAxAflNYA=="
}

This object is the value of the request packet's Data field. A message that isn't valid CoAP is still delivered, as {"Error": "...", "Raw": "..."} (see Errors and invalid data).

Reply shape

Replies use the same field names. Every field is optional: Version defaults to 1 and the others to zero or empty. Code can be the raw integer or a "class.detail" string such as "2.05". Options can be listed in any order; they are sorted and delta-encoded for you. Method, Path, ContentFormat, Error and PayloadError are ignored. If Raw is set (base64), those bytes are sent as-is and every other field is ignored. To answer a CON request with a piggybacked response, reply with an ACK (Type: 2) that echoes the request's MessageId and Token:

{
  "Type": 2,
  "Code": "2.04",
  "MessageId": 4660,
  "Token": "qrs=",
  "Options": [ { "Number": 12, "Value": "PA==" } ],
  "Payload": "oQBoYWNjZXB0ZWQ="
}

The CoAP Time Service and CoAP demo examples show complete services built on this formatter, including discovery, Observe, and block transfers.

coap+ (content-type aware CoAP)

coap+ uses the same message shape as coap, with two differences:

• When a message has a Content-Format option, its numeric value is also delivered as a top-level ContentFormat field.
• For the media types in the table below, Payload holds the decoded value (a JSON object, array or string) instead of base64. In replies, Payload is encoded back to the declared content type.

The CBOR request from the coap example arrives through coap+ as:

{
  "Version": 1,
  "Type": 0,
  "Code": 2,
  "MessageId": 4660,
  "Token": "qrs=",
  "Options": [
    { "Number": 11, "Value": "cmVhZGluZ3M=" },
    { "Number": 12, "Value": "PA==" }
  ],
  "Method": "POST",
  "Path": "/readings",
  "ContentFormat": 60,
  "Payload": { "0": "sensor-01", "1": 21.5 }
}

In a reply, set ContentFormat instead of building option 12 yourself, and give Payload as a JSON value. This reply produces exactly the same bytes on the wire as the coap reply example:

{
  "Type": 2,
  "Code": "2.04",
  "MessageId": 4660,
  "Token": "qrs=",
  "ContentFormat": 60,
  "Payload": { "0": "accepted" }
}

If a reply includes option 12 in Options as well as ContentFormat, the option wins. Property order does not matter.

Example: a Lambda handler

This Node.js handler accepts CBOR readings on POST /readings and replies with a piggybacked acknowledgement:

export const handler = async (event) => {
  const Replies = [];
  for (const message of event.Messages) {
    const request = message.Data;               // the decoded CoAP message
    if (request.Error || request.Type !== 0) continue;  // skip malformed messages; only answer CON requests

    let response;
    if (request.PayloadError) {
      response = { Code: "4.00", ContentFormat: 0, Payload: "bad payload" };
    } else if (request.Method === "POST" && request.Path === "/readings" && request.ContentFormat === 60) {
      const deviceId = request.Payload["0"];    // CBOR integer keys arrive as "0", "1", ...
      const temperature = request.Payload["1"];
      await saveReading(deviceId, temperature);
      response = { Code: "2.04", ContentFormat: 60, Payload: { "0": "accepted" } };
    } else {
      response = { Code: "4.04", ContentFormat: 0, Payload: "not found" };
    }

    Replies.push({
      Tag: message.Tag,
      Data: { Type: 2, MessageId: request.MessageId, Token: request.Token, ...response }
    });
  }
  return { Replies };
};

Text and JSON

Content-Format 0 is a JSON string holding the UTF-8 text, and replies must give a string. Content-Formats 50, 110 and 111 carry the JSON value in either direction. Inbound JSON payloads must be valid JSON without duplicate property names, and a reply's Payload value is sent as compact JSON.

{ "Type": 2, "Code": "2.05", "MessageId": 2, "ContentFormat": 0,  "Payload": "21.5 C" }
{ "Type": 2, "Code": "2.05", "MessageId": 3, "ContentFormat": 50, "Payload": { "status": "ok" } }

Link format (resource discovery)

RFC 6690 link-format, the payload of GET /.well-known/core, becomes an array with one object per link. The target URI is href. Quoted attributes become strings, bare numeric values become numbers, other bare values become strings, and valueless flags such as obs become true. This discovery reply:

{
  "Type": 2,
  "Code": "2.05",
  "MessageId": 1,
  "ContentFormat": 40,
  "Payload": [
    { "href": "/sensors/temp", "rt": "temperature-c", "if": "sensor", "obs": true },
    { "href": "/readings", "ct": 60 }
  ]
}

is sent with this payload:

</sensors/temp>;rt="temperature-c";if="sensor";obs,</readings>;ct=60

In replies, strings are always quoted, numbers are bare, true emits a flag, and false or null omits the attribute. The mapping is structural rather than byte-exact: attribute order and quoting may differ from an original document. If an attribute repeats within one link, only its last value is kept. Use coap if you need to serve a hand-written discovery document byte for byte.

XML

Content-Format 41 is delivered as a string with the original markup. It is not parsed, validated, reformatted or converted to JSON, so attributes, namespaces, whitespace and entities are left as they are. Payloads must be valid UTF-8, and replies (which must be strings) are always sent as UTF-8; make sure any encoding declaration agrees. For XML in other encodings, use coap with a base64 payload.

{ "ContentFormat": 41, "Payload": "<sensor unit=\"Cel\">21.5</sensor>" }

CBOR

CBOR (Content-Format 60) is mapped to the JSON an application would naturally expect:

• Text strings, booleans, null, arrays and finite floats map to ordinary JSON.
• Maps with text or integer keys become objects. Integer keys become decimal property names, so the CBOR map {0: "sensor-01", 1: 21.5} becomes {"0": "sensor-01", "1": 21.5}.
• Integers within JavaScript's exact range (±(2⁵³−1)) are JSON numbers.

In replies, property names that look like canonical integers (optional -, no leading zeros) are encoded as integer keys, so a decoded integer-keyed map echoes back unchanged. Other names are text keys. Whole-number JSON values encode as CBOR integers and fractional values as CBOR floats (using the narrowest exact width).

CBOR values with no plain JSON equivalent use reserved $cbor wrapper objects, both in what you receive and in what you send:

Integer wrappers cover −2⁶⁴ to 2⁶⁴−1; larger integers use bignum tags 2 and 3 wrapping a bytes wrapper. Tag numbers are decimal strings. Inbound maps use the map wrapper when their keys would collide or be ambiguous as JSON property names (for example integer 0 and text "0", duplicate keys, text keys that look like integers, or non-scalar keys). Use the same wrapper in a reply to send a text key that looks like an integer, or a literal $cbor key. Wrappers accept only their documented fields.

This mapping describes the data, not the exact encoding: float widths, NaN payloads and container encodings are not preserved. When your code needs the original bytes, use coap and work with the base64 payload.

SenML and SenSML

For SenML over CBOR (112) and SenSML over CBOR (113), Payload uses the JSON labels from RFC 8428 rather than generic CBOR maps. All standard labels (−6 to 8: bver, bn, bt, bu, bv, bs, n, u, v, vs, vb, s, t, ut, vd) are translated in both directions:

{
  "Type": 2,
  "Code": "2.05",
  "MessageId": 4711,
  "ContentFormat": 112,
  "Payload": [
    { "bn": "urn:dev:ow:10e2073a01080063:", "n": "temp", "u": "Cel", "v": 21.5 },
    { "n": "hum", "u": "%RH", "v": 40 }
  ]
}

On the wire, the first record is the CBOR map {-2: "urn:dev:ow:10e2073a01080063:", 0: "temp", 1: "Cel", 2: 21.5}.

• vd (data value) is unpadded base64url in JSON.
• Numeric values must be finite, and bver must be an unsigned integer. CBOR decimal fractions (tag 4) are decoded as numbers.
• Base values are not resolved: bn, bt and friends are delivered as sent, and the formatter does not interpret measurement semantics.
• Optional extension labels stay as text keys and use the CBOR mapping above. Unknown integer labels, mistyped standard fields, duplicate fields and mandatory extensions (names ending in _) are rejected.
• SenML must be a definite-length array. SenSML accepts either array form and is sent as an indefinite-length array. Whole payloads are processed, not incremental streams.

Moving from coap to coap+

coap stays exactly as it is, so existing integrations do not need to change. Switch a destination to coap+ when its code is ready to read decoded Payload values for the media types above, and to return JSON values (rather than base64) for those media types in replies. Messages with other Content-Formats, or none, behave the same under both formatters.

syslog

The syslog formatter parses each message into a JSON object with a fixed set of fields. It accepts RFC 5424 and BSD-style RFC 3164 messages and detects the format per message. Messages that cannot be parsed are still delivered, with the original text in raw, so syslog ingestion never silently drops data. In the other direction, it builds RFC 5424 or RFC 3164 messages from the same JSON shape, for replies and for Packet Sources that send syslog to devices or collectors.

Fields

Every field is always present, and is null when the message does not provide it. The shape is versioned by schema_version, and future changes will only add fields, so tables, crawlers and dashboards built on it keep working.

Examples

An RFC 5424 message with structured data:

<165>1 2026-11-02T10:15:30.123Z edge-01.example.com gateway 4321 LINK [origin@32473 site="hq" rack="r12"][meta@32473 seq="77"] link down on eth1

is delivered as:

{
  "schema_version": 1,
  "parse_ok": true,
  "format": "rfc5424",
  "pri": 165,
  "facility": 20,
  "severity": 5,
  "facility_name": "local4",
  "severity_name": "notice",
  "version": 1,
  "timestamp": "2026-11-02T10:15:30.123Z",
  "hostname": "edge-01.example.com",
  "app_name": "gateway",
  "proc_id": "4321",
  "msg_id": "LINK",
  "structured_data": {
    "origin@32473": { "site": "hq", "rack": "r12" },
    "meta@32473": { "seq": "77" }
  },
  "message": "link down on eth1",
  "raw": null
}

A BSD-style message, as sent by many network devices and older daemons:

<86>Nov  2 09:01:02 web-03 sshd[1234]: Accepted publickey for deploy

{
  "schema_version": 1,
  "parse_ok": true,
  "format": "rfc3164",
  "pri": 86,
  "facility": 10,
  "severity": 6,
  "facility_name": "authpriv",
  "severity_name": "info",
  "version": null,
  "timestamp": "2026-11-02T09:01:02Z",
  "hostname": "web-03",
  "app_name": "sshd",
  "proc_id": "1234",
  "msg_id": null,
  "structured_data": null,
  "message": "Accepted publickey for deploy",
  "raw": null
}

A message without a valid PRI cannot be parsed. It is still delivered, with every parsed field null:

{
  "schema_version": 1,
  "parse_ok": false,
  "format": null,
  "pri": null,
  "facility": null,
  "severity": null,
  "facility_name": null,
  "severity_name": null,
  "version": null,
  "timestamp": null,
  "hostname": null,
  "app_name": null,
  "proc_id": null,
  "msg_id": null,
  "structured_data": null,
  "message": null,
  "raw": "hello world"
}

How messages are parsed

• A message is treated as RFC 5424 when the PRI is followed by a one- or two-digit VERSION and a space. If such a message is malformed (for example, an invalid timestamp or structured data), it is reported with parse_ok: false rather than being reinterpreted as RFC 3164. A valid PRI is kept even then.
• RFC 3164 is parsed on a best-effort basis, as the RFC intends, and never fails once a valid PRI is found. The timestamp, hostname and tag are each optional: <13>just a message parses with only message set. An RFC 3339 timestamp in the BSD position is also accepted. A hostname is only recognized after a timestamp, and a token ending in : or containing [ is read as the tag.
• In structured data, repeated parameter names within one element are joined with , ([x@1 a="1" a="2"] gives {"a": "1,2"}), and repeated SD-IDs are merged into one object. Escaped characters (\", \\, \]) are unescaped. All values are strings.
• RFC 5424 NILVALUEs (-) become null.

Timestamps

Device clocks drift, and RFC 3164 timestamps carry neither a year nor a time zone. RFC 3164 times are assumed to be UTC, in the current year, or the previous year when that would put them more than a day in the future (a Dec 31 message received on January 1st is dated to the year before). For partitioning, retention and alarms, prefer the envelope's ReceivedAt; use timestamp for what the device reported. The sender's address and the listener are in the envelope's Remote and Local fields, not in Data.

Processing syslog

With no code at all, a Firehose or S3 destination stores one request packet per line, with the parsed syslog object in Data. In Athena, declare data as a struct with the fields you need and query them directly:

CREATE EXTERNAL TABLE syslog_archive (
  receivedat string,
  remote     struct<ipaddress:string, port:int>,
  data       struct<parse_ok:boolean, severity:int, severity_name:string, hostname:string,
                    app_name:string, message:string, raw:string>
)
ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'
LOCATION 's3://amzn-s3-demo-bucket/syslog/';

SELECT receivedat, data.hostname, data.app_name, data.severity_name, data.message
FROM syslog_archive
WHERE data.severity <= 3   -- err and worse
ORDER BY receivedat DESC
LIMIT 100;

On an EventBridge destination, rules can route by syslog fields without any code. This pattern matches messages of severity err or worse:

{
  "source": ["proxylity"],
  "detail-type": ["udp-gateway-packet-ingress"],
  "detail": {
    "Formatter": ["syslog"],
    "Data": { "severity": [{ "numeric": ["<=", 3] }] }
  }
}

A CloudWatch Logs destination writes one log event per message. With syslog, each event is the request packet as JSON, with no prefix, so CloudWatch sees structured events: Logs Insights discovers the fields automatically, and metric filters and subscription filters can match on them. For example, this Logs Insights query lists messages of severity crit or worse:

fields ReceivedAt, Remote.IpAddress, Data.hostname, Data.app_name, Data.message
| filter Data.severity <= 2
| sort ReceivedAt desc

and this metric filter pattern counts messages of severity err or worse:

{ $.Data.severity <= 3 }

If you prefer the original text in CloudWatch, give that destination utf8 (the default) and use syslog on the others. Text formatters write a bracketed prefix with the sender, listener and region, followed by the payload text.

In a Lambda function, read the fields from Data. Remember to handle unparsed messages:

def handler(event, context):
    for message in event["Messages"]:
        log = message["Data"]
        if not log["parse_ok"]:
            print("unparsed syslog from", message["Remote"]["IpAddress"], log["raw"])
            continue
        if log["severity"] is not None and log["severity"] <= 2:   # crit, alert, emerg
            notify_on_call(log["hostname"], log["app_name"], log["message"])
    return {"Replies": []}

In Step Functions, JSONata reads the fields directly: $states.input.Messages[Data.severity <= 2] selects the critical messages in a batch.

Sending syslog

When syslog is sent (a reply from a destination, or a Packet Source message with Formatter set to syslog), Data is a JSON object of the same shape. Every field is optional, and schema_version and parse_ok are ignored, so a delivered object can be sent back out unchanged.

For example, these objects:

{ "facility_name": "local4", "severity_name": "warning", "hostname": "edge-01", "app_name": "gateway",
  "structured_data": { "origin@32473": { "site": "hq" } }, "message": "link down on eth1" }

{ "format": "rfc3164", "facility": 10, "severity": 6, "timestamp": "2026-11-02T09:01:02Z",
  "hostname": "web-03", "app_name": "sshd", "proc_id": "1234", "message": "Accepted publickey" }

are sent as:

<164>1 - edge-01 gateway - - [origin@32473 site="hq"] link down on eth1
<86>Nov  2 09:01:02 web-03 sshd[1234]: Accepted publickey

To forward syslog to another collector with a Packet Source, publish a message like this to the Packet Source's SNS topic (with the EgressRegion message attribute set):

{
  "Messages": [
    {
      "Remote": { "Address": "203.0.113.10", "Port": 514 },
      "Formatter": "syslog",
      "Data": {
        "facility_name": "local0",
        "severity_name": "err",
        "hostname": "aws-bridge",
        "app_name": "forwarder",
        "message": "disk 92% full on vol-0abc"
      }
    }
  ]
}

Errors and invalid data

Every inbound packet is delivered. When a packet can't be fully decoded, the problem is recorded in Data and also reported as a destination error. Enable MetricsEnabled and set LogGroupName on the destination to see these errors in the Errors metric and its logs (see Destinations).

{
  "Version": 1, "Type": 0, "Code": 2, "MessageId": 4660, "Token": "qg==",
  "Options": [ { "Number": 11, "Value": "cg==" }, { "Number": 12, "Value": "PA==" } ],
  "Method": "POST",
  "Path": "/r",
  "ContentFormat": 60,
  "Payload": "/w==",
  "PayloadError": "the application/cbor payload could not be decoded, so it is base64: ..."
}

To forward a malformed CoAP message unchanged, reply with its Raw value. When PayloadError is present, Payload is base64, so don't echo that object back with the same Content-Format.

Replies and Packet Source messages are encoded one at a time. One that can't be encoded (an unknown severity_name, a hostname containing a space, a non-string application/xml payload, an invalid $cbor wrapper, invalid base64, or a string Data for a JSON formatter) is reported and not sent, and the others are still sent. A bad packet or reply never holds up the rest of its batch.

Related

• Input and Output Formats: the request and response envelopes that carry Data
• Destination Formatter property
• PacketSource message format
• Proxylity examples, including CoAP and syslog services