Data Loading
Typical Data Loading Practices
Databases for reporting and data analysis are typically loaded through a controlled process. Traditionally, large incremental data sets are added on daily, weekly, or even monthly basis and additional data is mostly inserted as opposed to updated or deleted. Nowadays data loads are often much more frequent and in some cases data arrives as a constant stream of inputs and there is no real point in time that no data comes in. With the increase in data load frequency, data volumes have grown as well.
Vector works very well for scenarios with large incremental volumes of data that are predominantly inserts. Streaming data can often be collected and presented in incremental batches every few seconds (depending on the query requirements), which works well as long as incremental batches have a relatively high volume. Ideally, multiple data blocks are filled for every incremental data load.