Design Principles

The following are a set of principles that are crucial to understand and follow when creating a Loader as it will greatly improve the quality of Loaders being developed if they are followed:

• Mutually Exclusive - The choice of Partition Keys should guarantee that data from one Data Partition cannot exist in another Data Partition. This could indicate an improper choice of keys and could result in data duplication.

• Atomicity - Optimistically, the loader should be able to fetch data from the source system in a single request using the provided Partition Keys. This could mean too few Partition Keys have been defined.

• Symmetry - Optimistically, the loader should fetch data from the source system and realise the value of the entire response without filtering or adulting the data meaning the symmetry of the source and target is high. Low symmetry could mean too many Partition Keys have been defined and data is ingested from the source but being thrown away.