Protobuf Schemas

Protocol Buffers (Protobuf) is a language- and platform-neutral mechanism for serialising structured data.
In MAPS, Protobuf is used wherever you want:

• strong typing
• explicit schema evolution
• compact binary representation
• multi-language interoperability

MAPS integrates Protobuf schemas through SchemaConfig with format: "protobuf" and a Protobuf-specific configuration object stored in schema.

---

1. Description

Protobuf schemas are defined in .proto files, compiled to language-specific code and a binary descriptor set.
MAPS uses a small JSON wrapper plus the compiled descriptors to:

• identify the root message type
• resolve nested message types
• decode/encode payloads to/from Typed Events

Once decoded, Protobuf behaves like any other schema type in MAPS: all downstream processing sees a Typed Event, not raw Protobuf messages.

---

2. Protobuf Schema Definition

2.1 Example .proto definition

syntax = "proto3";

package maps.sensors;

message Bme688Reading {
  string device_id   = 1;
  int64  timestamp   = 2; // epoch millis
  double temperature = 3; // °C
  double humidity    = 4; // %RH
  double pressure    = 5; // hPa
  double gas         = 6; // Ω
  string heater_mode = 7;
  string gas_mode    = 8;
}

This is the canonical Protobuf schema.
MAPS does not rewrite this; it consumes the compiled descriptors.

---

3. Protobuf SchemaConfig in MAPS

Protobuf schemas in MAPS are stored as a SchemaConfig with:

• format = "protobuf"
• schema = a Protobuf-specific config object

The Protobuf config is represented as:

public static final class ProtobufConfig {
  private String messageName;
  private byte[] descriptorValue;
}

In the SchemaConfig, this is serialised into JSON as:

"schema": {
  "messageName": "fully.qualified.MessageName",
  "descriptorValue": "<base64-encoded descriptor bytes>"
}

Important:
Protobuf uses only the schema object.
For Protobuf, schemaBase64 is not used and must remain unset.

MAPS expects descriptorValue to contain a base64-encoded descriptor payload, typically:

• a FileDescriptorSet, or
• a descriptor for the root message and its dependencies.

---

4. Example SchemaConfig

{
  "versionId": "1",
  "name": "Bme688-Protobuf",
  "description": "BME688 sensor reading using Protobuf encoding.",
  "labels": {
    "uniqueId": "e63f4f0f-1ef9-4b6a-8c7a-7c7b27e9f111",
    "resource": "sensor",
    "interface": "sensor.bme688",
    "comments": "BME688 telemetry encoded as Protobuf"
  },
  "format": "protobuf",
  "schema": {
    "messageName": "maps.sensors.Bme688Reading",
    "descriptorValue": "<base64-encoded FileDescriptorSet bytes>"
  }
}

4.1 Fields

• format: must be "protobuf".
• schema.messageName: the fully-qualified Protobuf message name for the root message.
• schema.descriptorValue: base64 encoding of the descriptor bytes used to construct the Protobuf descriptors at runtime.

---

5. Java Integration (Conceptual)

At runtime, MAPS:

1. Looks up the SchemaConfig for the topic/context.
2. Deserialises the schema object into ProtobufConfig.
3. Decodes descriptorValue from base64 into a byte[].
4. Builds a descriptor set from this byte array.
5. Locates the root message type using messageName.
6. Decodes or encodes Protobuf messages based on that descriptor.

The result is exposed to the rest of MAPS as a Typed Event.

---

6. Event Flow with Protobuf

6.1 Decoding

1. Incoming payload: binary Protobuf message on a topic bound to a Protobuf schema.
2. MAPS:
   • finds the Protobuf SchemaConfig
   • loads/uses the ProtobufConfig
   • decodes the bytes into a dynamic Protobuf message
   • converts it into a Typed Event
3. The Typed Event flows through:
   • filtering
   • transformations
   • statistics
   • format conversion (for example, Protobuf → JSON / Avro / CSV)

6.2 Encoding

For outbound conversions:

1. MAPS starts from a Typed Event.
2. Maps fields into the Protobuf message structure defined by the descriptors.
3. Builds a Protobuf message dynamically.
4. Serialises it to a binary Protobuf payload.

---

7. Protobuf-Specific Notes

7.1 Strong Typing and Evolution

• Protobuf identifies fields by numeric tags.
• Schemas can evolve (add/remove fields) while remaining compatible.
• MAPS uses versionId and labels.uniqueId to track versions on the schema side.

7.2 Enums

• Protobuf enums are mapped to typed fields in the Typed Event.
• MAPS can preserve both the enum name and underlying numeric value.

7.3 Timestamps

Typical patterns:

• int64 epoch millis
• google.protobuf.Timestamp

Internally, MAPS normalises timestamps into a standard representation (for example, epoch millis) for filtering and statistics.

7.4 Oneof

Protobuf oneof fields are treated as mutually exclusive fields in the Typed Event:

• only the active field is populated
• inactive members are omitted or set to null

7.5 Nested / Embedded Messages

Nested messages become nested objects in the Typed Event and can participate in:

• filtering
• transformation rules
• statistics aggregation

---

8. When to Use Protobuf in MAPS

Protobuf is a good fit when you need:

• strict schemas and type checking
• compatibility across languages and platforms
• compact and efficient binary encoding
• more expressive structure than CSV or Native, but with smaller size than JSON/XML

For very simple single-value payloads, prefer Native schemas.
For JSON-centric systems or ad-hoc APIs, JSON schemas may be simpler to manage.

---

9. Example

This example loads a Protobuf descriptor and constructs a ProtoBufSchemaConfig for use

  public static SchemaConfig getProtobufSchema(byte[] protoDesc, String messageName, String title, String description, String matcher, String type) {
    UUID schemaId;
    try {
      schemaId = UuidGenerator.getInstance().generate(NamedVersions.SHA1,uuid, title );
    } catch (NoSuchAlgorithmException e) {
      schemaId = UuidGenerator.getInstance().generate();
    }
    ProtoBufSchemaConfig config = new ProtoBufSchemaConfig();
    ProtoBufSchemaConfig.ProtobufConfig protoConfig = new ProtoBufSchemaConfig.ProtobufConfig();
    protoConfig.setDescriptorValue(protoDesc);
    protoConfig.setMessageName(messageName);
    config.setProtobufConfig(protoConfig);
    config.setComments(description);
    config.setTitle(title);
    config.setVersion(1);
    config.setMatchExpression(matcher);
    config.setUniqueId(schemaId);
    config.setResourceType(type);
    return config;
  }

Example protobuf config

syntax = "proto3";

package aircraft;

option java_package = "io.mapsmessaging.proto.aircraft";
option java_outer_classname = "SensorReadingProto";

message SensorReading {
  string id = 1;             // Optional sensor or reading ID (e.g., "egt-sensor-1")
  uint32 engine_id = 2;      // Engine number (e.g., 1, 2, ...)
  float value = 3;           // The measured value
  string unit = 4;           // Unit of measurement (e.g., "C", "RPM", "PSI")
  int64 timestamp = 5;       // Epoch timestamp in milliseconds
}

syntax = "proto3";

package aircraft;

option java_package = "io.mapsmessaging.proto.aircraft";
option java_outer_classname = "FuelSystemProto";

message FuelSystem {
  float fuel_quantity_left_kg = 1;
  float fuel_quantity_right_kg = 2;
  float fuel_temperature_c = 3;
  bool crossfeed_enabled = 4;
}