Failed processing on MT-TA stations

[jlmaurer]

Stations:

WVO54 = ["4P", "WVO54",  '2016-04-25 22:13:06', '2016-05-03 15:53:17'] 
WVO55 = ["4P", "WVO55", '2016-04-25 22:13:06', '2016-05-03 15:53:17']
WVO53 = ["4P", "WVO53", '2016-04-25 22:13:06', '2016-05-03 15:53:17']

Error 1 - WVO53 with WVO55 as reference

kernel_dataset = KernelDataset()
station_id = "WVO53"
remote_reference_id = "WVO55"
kernel_dataset.from_run_summary(run_summary, station_id, remote_reference_id)
kernel_dataset.mini_summary

Output:
survey station run start end duration
0 4P WVO53 b 2016-04-25 22:13:06+00:00 2016-04-25 22:44:38+00:00 1892.0
1 4P WVO55 a 2016-04-25 22:13:06+00:00 2016-04-25 22:44:38+00:00 1892.0
2 4P WVO53 b 2016-04-25 22:58:43+00:00 2016-05-03 15:53:17+00:00 665674.0
3 4P WVO55 b 2016-04-25 22:58:43+00:00 2016-05-03 15:53:17+00:00 665674.0
4 4P WVO53 c 2016-05-04 18:48:31+00:00 2016-05-14 05:25:17+00:00 815806.0
5 4P WVO55 c 2016-05-04 18:48:31+00:00 2016-05-14 05:25:17+00:00 815806.0

This resulted in ValueError: operands could not be broadcast together with shapes (6962,) (18454,)
NOTE: This same error happens when running WVO53 by itself (single-station processing)

Error 2 - WVO54 with WVO55 as reference

kernel_dataset = KernelDataset()
station_id = "WVO54"
remote_reference_id = "WVO55"
kernel_dataset.from_run_summary(run_summary, station_id, remote_reference_id)
kernel_dataset.mini_summary

Output:
survey station run start end duration
0 4P WVO54 b 2016-04-25 22:13:06+00:00 2016-04-25 22:44:38+00:00 1892.0
1 4P WVO55 a 2016-04-25 22:13:06+00:00 2016-04-25 22:44:38+00:00 1892.0
2 4P WVO54 b 2016-04-25 22:58:43+00:00 2016-05-03 15:53:17+00:00 665674.0
3 4P WVO55 b 2016-04-25 22:58:43+00:00 2016-05-03 15:53:17+00:00 665674.0
4 4P WVO54 c 2016-05-04 18:48:31+00:00 2016-05-14 05:25:17+00:00 815806.0
5 4P WVO55 c 2016-05-04 18:48:31+00:00 2016-05-14 05:25:17+00:00 815806.0

This results in numpy.linalg.LinAlgError: Singular matrix (!!!)
NOTE: This same error results when running WVO54 as single-station processing and WVO55 as single-station

[kkappler]

Thanks @jlmaurer for this detailed description. In trying to replicate this issue I have observed the following run summary table. I highlighted with a grey box the column of concern (has_data). In most applications these are all True, but here there are several False values.

What seems to be going on is that the archive is not completely formed. This means a couple TODOs are on my list

• Debug why the archive is not built as expected (this is an MTH5 & Earthscope data availability question)
  • Actually, the requested data are in the archive. What is perhaps a bit confusing is that non-requested runs associated with the stations are also listed in the archive. This extra information could be removed if we want, but after debugging, the requested data from 2016-04-25 22:13:06+00:00 to 2016-05-03 15:53:17+00:00 are indeed present.
• Add handling in aurora's pre-processing to verify that all time series are present before starting a processing task.