remarks on Message Format

[stephanbracke]

metadata

What exactly do you mean with channel it isn't reflected in the sample.
According to me the startime and sample rate are not really metadata they are necessary to interprete the rest of the message and are part of the body of the message and not meta data as in the metadata database.
One element missing is the fields indicator: XYZ,DHF,F etc
Are sample rate and delta (in the example) the same thing? If so it is better to use the word Sample Period as in the CDF standard (5.1) and use the ISO8601 for durations.
Maybe it is better to make a chapter of needed metadata, afterwards we can decide if it is necessary to repeat it in the message (most will be already in the topic)

[leonro]

And here a completely different suggestion: What about separating meta information and data, completely? I see the benefit of publishing Json type formats. However, we broadcast a lot of unnecessary information along with every data block. My suggestion:
Use topics like
IAGACODE/INSTRUMENTCODE/data#
for sending numerical data blocks (eventually even binary coded)

and send (eventually far less frequent) meta information as key/value pairs:
IAGACODE/INSTRUMENTCODE/meta#

I personally would prefer to send timestamps along with each data set. This will increase the load but some of our data sets are not evenly spaced. Anyway, having the possibiilty to send starttime, sampling rate, and a block of evenly spaced data sets within the data block is certainly also favorable and mimics seismic formats like seed.

The meta block can be filled basically with any information. Yet there need to be some list of essential information on the data sent: like components (field indicators), units, setup, ...

[CharlesBlais]

What Jeremy is proposing shares a lot of similarities with seismic data. When you mean “duration” Stephan, are you referring to his “delta”? I would prefer number representation as opposed to string representation for duration so that decoding is simplified. In his case, delta is in seconds (should probably document that) and it’s the same lingo used in seismic data.
In regards to your metadata, Jeremy is not missing the channel indicator (or fields). It is the key on his data. I agree with what Jeremy is doing however, I believe the metadata should be associated with the data per component (much like miniSeed). I think this is what Roman is saying also. This opens up the format to variable time scale and sampling periods per component. For example, a scalar instrument may be sampling at 5 seconds as opposed to 1 for vector instrument but could be in the same stream (that is the case in Canada). In other words, to simplify decoding and open up to wider libraries, I would just reuse other libraries (ex: ~obspy.Stream in Python https://docs.obspy.org/packages/autogen/obspy.core.stream.Stream.html which is essentially a representation of miniSeed as an object). So to extend, on Jeremy’s example:

[{
	"header": {
		"observatory" : "BOU",
		"channel" : "X",
		"starttime" : "2017-09-05T00:00:00.000Z",
		"delta" : 1,
		…
	},
	"data": [20542.04, 20541.486, 20542.067]
},{
	"header": {
		"observatory" : "BOU",
		"channel" : "F",
		"starttime": "2017-09-05T00:00:00.000Z",
		"delta" : 5,
		…
	},
	"data": [20542.04, 20541.486, 20542.067]
}]

[stephanbracke]

Charles,
I used the terminology that Simon uses in CDF there the delta is a sample_period (see chapter 5.1 in the ImagCDF), you use almost the same word sampling_period in the webservice document. If you change terminology for each different standard it gets complicated. The same is true for the channel definition in Intermagnet standards we always talk about components ( also in your defined webservice and IMFV1.23) . Personally I think its better to use the same words used in geomagnetic standards (instead of seismic).

[jmfee-usgs]

@stephanbracke

Thanks for starting this issue for metadata. Let's add new issues as appropriate to discuss other sections of the document.

I think of the terms channel and component as interchangable, but maybe component or even element are more appropriate?

Similarly, I'm open to using the term rate or period instead of delta, although I prefer shorter keys ('period' instead of 'sample_period' since they are repeated within each message.

I like the idea of ISO8601 durations, although that leaves a lot of flexibility for representing equivalent values and may be more difficult to implement. I think a period represented as a number (or fractional number) of seconds covers most use cases. Are there other cases you think where the duration is more appropriate?

@leonro

I think that's an interesting idea to publish more detailed metadata for observatories and instruments via topics, particularly to notify users when things like pier correction values change. I think that may be outside the initial scope, and maybe we could reserve a type=metadata section within the topic format for future use?

I agree we should send time along with each message, as well as observatory, code, and period; enough information to identify the data and look up more detailed metadata elsewhere.

@CharlesBlais

I think that's useful to allow different sampling rates; particularly for later derived data sets. What if we make the overall message an array of smaller blocks and, for efficiency, allow multiple channels within the data for cases where the metadata is shared? It may be simpler to keep each component separate, and I can see advantages either way.

@stephanbracke

I think we should document the conventions we use, whatever they may be. I based some of the information on the Obspy conventions, which deal with (albeit seismic in most cases) timeseries data:
https://docs.obspy.org/packages/autogen/obspy.core.trace.Stats.html

While we are applying this to geomagnetism data, we are ultimately transmitting timeseries data and these standards may eventually be useful outside the geomagnetism community.