Duplication control

[anderspeders]

We might need to have some type of duplication control as part of the upload process for ResourceProjects.

[mattfullerton]

Well... 🙂 we do... new companies and projects get flagged as being potential duplicates and can then be merged together, flagged as duplicates to be ignored or treated as independent (not duplicates). The system is also designed to maintain the knowledge for future imports.

However, the tough part is not flagging too many entries as duplicates (programming side), and also taking the time to do the reconciliation (human side). And there is perhaps the question of whether some of this can be moved into a pre-import step - at least for company names this sounds like something very plausible - mapping a company name to a fixed single company name and company group if appropriate by checking against our existing data or opencorporates.