coercion of "NA" for numeric variables on read

[nicholas-masel]

On read, "NA" is coerced.

datasetjson/R/read_dataset_json.R

Line 102 in 9a81c17

d[dbl_cols] <- lapply(d[dbl_cols], as.double)

Simple example:

Browse[1]> as.double(c("1", "0.2", "NA", "  NA"))
[1] 1.0 0.2  NA  NA
Warning message:
NAs introduced by coercion

Simple fix:
Store "NA" as ""

Browse[1]> as.double(c("1", "0.2", ""))
[1] 1.0 0.2  NA

[nicholas-masel]

@mstackhouse What do you think of this change? I'm not sure how it would impact the SAS and python implementations for reading datajson with a switch from "NA" to "" for numerics.

[mstackhouse]

I think I'm ok with this... but, what brought this up for you exactly? If the column is truly a float, then the "NA" imports make sense, but not " NA", because this would be driven by the yyjsonr interpretation of the empty values in the input dataset.

My concern is if this would inadvertently cause information loss. But if the column is supposed to be a float, then any non-numeric value should be NA to begin with.

[nicholas-masel]

In the datasetjson submission pilot, start and end day are numeric and being saved to json with padding that matches the longest value.

For example both start and end have padding to keep the strings to a length of 3 since the longest day is 100-999 days.
" 2"," NA"

So the thought is when writing numeric, if NA, write out as "".

[mstackhouse]

Ok. Yeah I think the update is worth it.