Reader question: Are nested hashes in messages bad?

In response to yesterday’s message about [metadata.properties](http://metadata.properties) and metadata.local_properties, which mentioned that there are each hashes on the metadata attribute of a message, Practical Microservices Daily reader Connor Widtfeldt wrote in with a surprised response (shared with permission):

I thought nested hashes were taboo in the message-db world.

Thank you for bringing the point up, Connor!

In the Eventide documentation, you can read:

Messages are typically flat key/value structures, and by default, transformation of messages does not traverse a graph of attributes (emphasis added)

Why would we care about a flat message structure vs. a nested one? Messages with nested structures are a design smell. Remember that “smells” in programming aren’t necessarily wrong, but they invite a second or third look to make sure they’re correct.

When you have a nested structure in your message design, you may actually have two or more messages pretending to be a single message. For example, if you had a single event capturing users’ mailing and physical addresses in the same event, a nested structure might make that seem reasonable. Represented as JSON, you could have:

{
  "mailing": {
    "street": "",
    "zip": "",
  },
  "physical": {
    "street": "",
    "zip": ""
  }

(Yes, I left off a lot of address complication in the interest of space.)

The nested structure almost makes it seem like this is a natural fit.

If you tried not nesting this, you’d end up with something like:

{
  "mailing_street": "",
  "mailing_zip": "",
  "physical_street": "",
  "physical_zip": ""
}

I think this repetition of prefixes strongly signals, “Hey, you’re trying to cram two things into one thing. Danger lies ahead.”

The nature of a monolith is that things that ought to be separated aren’t, and you can arrive at monolithic design even if you use messaging. I’ve seen messages with Mariana Trench depths of nesting out in the wild that were megabytes in size. These messages would not pass my filter.

Now, as regards the point in the documentation and Connor’s surprise at how metadata is used in Eventide, I can offer two angles:

1. The distinction between MessageData and Messages in Eventide
2. The importance of separating the mechanical from your design

MessageData vs. Messages in Eventide

In Message DB, messages are stored as rows in the messages table. This table has 8 columns, including data and metadata. In truth, every column other than data is metadata about the message. Several of them are separated out for ease of querying.

When Eventide retrieves the raw rows from Message DB, it first injects their data into MessageData instances, which are low-level representations of messages. These are barely above the level of raw query results, and you don’t have different MessageData classes for each type of message you deal with.

When you write a message handler, you typically deal with Message instances. These are your commands and events found in lib/<component_name>/messages/**. When you declare something like:

handle Deposit do |deposit|
end

Deposit is a Message. If you look at one of your message definitions, it’ll be something like:

module AccountComponent
  module Messages
    module Commands
      class Deposit
        include Messaging::Message
      end
    end
  end
end

Notice the include Messaging::Message. Eventide’s handler constructs receive MessageData instances and build instances of your higher-level message classes for use in your handlers. So Message DB ⇒ MessageData instances ⇒ specific message classes that include Messaging::Message.

That link the documentation refers to these higher-level classes. You locate the design of your system in these classes, and this is where you have the greatest latitude in your design decisions. Specifically, your Messages are where you typically have flat structures.

Separating mechanical concerns from your design

A message’s metadata has different constraints and is a very mechanical field. First of all, at least in Eventide, you don’t choose anything that goes directly onto metadata. You can use Message DB without Eventide and do whatever you want, but if you’re using Eventide, you’ll have a much better time if you follow that rule.

Second, [metadata.properties](http://metadata.properties) and metadata.local_properties could have been extracted into separate columns. Remember that 7 of the 8 columns are all metadata. id, type, stream_name, position, global_position, and time have their own columns because these fields are used in queries. metadata.properties and metadata.local_properties are not, so they happen to be located in the metadata column. There’s no need for the database to be able to address them directly.

The risk of nested structures lies in your system design—it’s logical nesting that we scrutinize. That’s where run-away coupling will tax your productivity until your work comes to a grinding halt. Your properties and local_properties are unlikely to be nested themselves, and even though they are nested under metadata, logically I think of them as flat.

Connor, I hope that helps! If it doesn’t, please do respond.