There is one problem that has plagued me for years, how to ensure that when i'm perform imports I don't create duplicate objects. In reality the problem is fairly simple conceptually, you create a hash table, and then hash the Incoming objects to test for their membership in the set, Easy.
But if you believe in DRY, the problem is then making this unique constraint checking code modular and portable across projects, and different Core Data Models.
The first issue we need to is identify what property of a NSManagedObject should be unique relative to the rest of the greater set. We can achieve this by adding a class method to the NSManagedObject class
+ (NSString *)uniqueKeypath {
return nil;
}
+ (NSArray *)uniqueKeypaths {
return nil;
}
Finding a Needle in a Haystack
Now it's time to make the magic happen. We apply the concept, we create a hash table, then lookup the unique value of the new objects in the table. If we get a 'hit', we don't do anything, if we 'miss' we create a new object. The sudo code is effectively.
FOR EACH object IN newObjects:
IF NOT object IN allObjects:
createObject(object)
So, the solution is GCD and batched fetching. Given that we can assume that only one import is happening at any one time. So once we have performed the initial fetch of all the existing objects, we can split the comparison operations for each of the new objects on to a concurrent GCD queue. We also need a place to store the objects that need to be created, rather than copying them into another collection, we can keep the current collection and gather the indexes of all the new objects, and then create a subset of the superset.
//Compare new hashes against all the known hashes on multiple threads
dispatch_apply([newObjects count], concurrentQueue, ^(size_t idx) {
//Note that everything that happens here is on a concurrent queue
if (![hashes member:[[newObjects objectAtIndex:idx] valueForKeyPath:aKeypath]]) {
//We have synchronize access to the mutable indexset
[lock lock]; //Lock the index set
[uniqueIndexes addIndex:idx]; //add the unique index
[lock unlock]; //Unlock the index set
}
});
The next part is to split the work of fetching the objects in to smaller batches so we can not only perform smaller units of work, but also have a lower high memory watermark. NSFetchRequest has support for batching requests so this is apparently handled transparently to the rest of the code using a special type of NSArray. However I haven't tested this to ensure it behaves as I expect.
I've posted an implementation as a abstract subclass of NSManagedObject on Github, fork away!
