Skip to content

Skip duplicate records when importing

Martin Burchell requested to merge ignore-duplicates into staging

Ideally we'd have unique IDs from the data source but we can't guarantee that so next best thing is to put a unique together validator for the body and timestamp fields. We do this at the REST API level as this is not possible for a text field in the database. Unfortunately because MySQL truncates microseconds we need to have a custom field in the serializer to truncate microseconds before validation. Shouldn't be a problem when we move to 5.6+ all round

Merge request reports

Loading