2016-03-25 validation data issues (depth of water temp)

[jordansread]

related to #87 , but we have seemingly errant depth data the is coming from the Water Quality Portal.

After transforming the units, here are the distributions of the data:

depth.units   p.25   p.50     p.75  count
        (chr)  (dbl)  (dbl)    (dbl)  (int)
1        feet 1.8288 5.4864 10.02792  57365
2          ft 0.9144 0.9144  1.82880  93566
3          in 0.2540 0.3810  0.45720     27
4           m 1.0100 4.0590  9.00000 730459
5      meters 3.0000 8.0000 16.00000  53368
6          mm 0.0130 0.0130  0.01300      1

note that the depth.units is the original unit, and the values in the p.25, p.50, etc are the transformed measurements, all in m

A couple of things could be going on here:

1. certain agencies/groups that use certain units may have errant data in here
2. certain groups may use a certain unit (ft for example) and collect mostly surface measurements
3. bad data is spread across agency identifiers and units randomly, or follows some kind of pattern we can use to ultimately make a judgement call to remove certain data.

Will be updating this issue w/ progress.

@lawinslow noted this difference between data coming from the WI DNR, and the WI section of the WQP:
WQP

DNR

[jordansread]

group_by(data.in, OrganizationFormalName) %>% summarize(unique.units=length(unique(ActivityDepthHeightMeasure.MeasureUnitCode)), count=length(ActivityDepthHeightMeasure.MeasureUnitCode)) %>% arrange(desc(count)) %>% data.frame
                                                 OrganizationFormalName unique.units  count
1                                    Minnesota Pollution Control Agency            3 692498
2                             Wisconsin Department of Natural Resources            4 461730
3                                   USGS Wisconsin Water Science Center            3  87138
4                        National Park Service Water Resources Division            3  20627
5                                    USGS Michigan Water Science Center            2  20059
6                          Michigan Department of Environmental Quality            2  11219
7                                             Grand Portage Reservation            1   6290
8                                     Fond du Lac Band of Chippewa (MN)            1   6233
9                                   USGS Minnesota Water Science Center            3   6158
10                                           Bay Mills Indian Community            2   3030
11 Lac Du Flambeau Band of Lake Superior Chippewa Indians Water Program            3   2795
12                       Forest County Potawatomi Community (Wisconsin)            2   2064
13                                  Little River Band of Ottawa Indians            1   1669
14                                EPA National Aquatic Resources Survey            1   1586
15                     MATCH-E-BE-NASH-SHE-WISH BAND OF POTAWATOMI (MI)            2   1477
16                                     The Stockbridge-Munsee Community            2   1232
17                       FOREST COUNTY POTAWATOMI COMMUNITY (Wisconsin)            2   1099
18                                                         Michigan DEQ            1    932
19                          GRAND PORTAGE Band of CHIPPEWA INDIANS (MN)            1    886
20                             STOCKBRIDGE-MUNSEE COMMUNITY (Wisconson)            2    811
21   Pokagon Band of Potawatomi Indians Department of Natural Resources            1    493
22                                                                 DENR            3    331
23                                        Prairie Island Community (MN)            1    234
24                         SD Dept of Environmental & Natural Resources            2    181
25                      MENOMINEE INDIAN TRIBE OF WISCONSIN (Wisconson)            2    162
26                                                         Red Lake DNR            1    122
27                                ONEIDA TRIBE OF WISCONSIN (Wisconsin)            1     54
28                              21IOWA-Iowa Dept. of  Natural Resources            1     49
29                              Sokaogon Chippewa Community (Wisconsin)            1     45
30                            Saginaw Chippewa Planning Department (MI)            1     17
31                                     Iowa Dept. of  Natural Resources            1     13
32                                    Lower Sioux Indian Community (MN)            1      7
33                               USGS North Dakota Water Science Center            1      7
34           Lac Du Flambeau Band of Lake Superior Chippewa Indians DNR            1      6
35                                     USGS Kansas Water Science Center            1      6
36                                                  GLEON Lake Observer            1      5
37         Sisseton-Wahpeton Sioux Tribe Lake Traverse Reservation (SD)            1      5
38                                    USGS Georgia Water Science Center            1      2

[lawinslow]

USGS Wisconsin Water Science Center 6 87138
Ha thought for sure the different units would be from an agency.

[jordansread]

I had screwed up on that, fixed

[lawinslow]

Ah, makes more sense.

[jordansread]

@lawinslow why are we seeing different things w/ the histograms?

note this is all three states, not just WI.

[lawinslow]

Probably because you're looking at all three states? The comparison I showed was just overlapping Wilma lakes to aid the contrast (didn't really know what to expect). If you look at all maybe the bad Vals are diluted. I'll re-visit my question from yesterday, how many bad depth Vals do we need to cause an issue with rmse? 5%?

[jordansread]

Looks like "Wisconsin Department of Natural Resources" is a bit odd:

[jordansread]

Focusing on WI-DNR, because of the large amount of near surface data.

Those data seem a little bit off for what we would expect in June:

July:

[jordansread]

vs

[lawinslow]

Whoa, those low temperature surface values in the "ft" data seem pretty bad. Could we have stream data in there?

[jordansread]

WI DNR apparently uses ResultDepthHeightMeasure.MeasureUnitCode instead of ActivityDepthHeightMeasure.MeasureUnitCode

see:

[jordansread]

Minnesota Pollution Control Agency uses ActivityDepthHeightMeasure.MeasureValue instead:

[lawinslow]

That's such an awesome figure.

[jordansread]

All data after flipping between result and activity depth based on agency:

(for all time)

[jordansread]

now the data look as we would expect: