Data format

[nealf]

Here's what I'm proposing for our crime data format:
{
“_id”: String (autogenerated),
“AgencyID”: String,
“AgencyName”: String,
“CaseNumber”: String,
“CriminalOffense”: String,
“DateReported”: DateTime (YYYY-MM-DDTHH:MM:SS),
“Description”: String,
“Location”: String,
“OccurenceDate”: DateTime (YYYY-MM-DDTHH:MM:SS),
“Disposition”: String,
“Lat”: float,
“Lng”: float
}

• _id is autogenerated by CouchDB if you need, or could use a unique id like CrimeCodeID
• Coordinates will need to be converted or geocoded to lat/lng
• Location should have the city/state added to the street address
• CriminalOffense (CrimeCode) should probably be standardized to some extent

Does anybody have any thoughts on whether we should essentially keep all of the original data we scrape and then add the standardized fields separately?