How to Achieve Optimal Loading with vwload
For optimal loading speed with vwload, follow these steps:
1. Split the data files (manually or with a third party tool) into separate files, preferably equal to the number of partitions but at least a multiple of the number of nodes.
2. Copy the files to HDFS using standard HDFS utilities.
3. Use the vwload operation in parallel (cluster) mode.