The program attempted to emulate a map-reduce paradigm in order to allow for horizontal scaling. The map part consists of the files cat,accept_only_five_columns.sh and mapper.py. Then, we sort the results to limit the memory usage in the reducer with the program sort_loans.sh and finally perform the reduction using the file reducer.py.

Bash and Python3 were used in coming up with this program. Bash was used because of the toolsets it has e.g. awk that enable fast processing of data especially in streams. The sort program was also used to perform fast sorting. Python was mainly used to allow for Object Oriented Programming style and also due to the fact that it's a more readable scripting language with many libraries for additional functions.

Input and output streams have been used to reduce the memory requirements of reading an entire file before processing. This usually results in faster programs that can handle data larger than the memory of the given machine before resorting to virtual memory.