MapReduce < Main

Tags: view all tags
---+ MapReduce defined

MapReduce is a framework for processing highly distributable problems across huge datasets using a large number of computers (nodes).

<strong>"Map" step:</strong> The master node takes the input, divides it into smaller sub-problems, and distributes them to worker nodes. A worker node may do this again in turn, leading to a multi-level tree structure. The worker node processes the smaller problem, and passes the answer back to its master node.

<strong>"Reduce" step:</strong> The master node then collects the answers to all the sub-problems and combines them in some way to form the output &ndash; the answer to the problem it was originally trying to solve.

MapReduce allows for distributed processing of the map and reduction operations. Provided each mapping operation is independent of the others, all maps can be performed in parallel &ndash; though in practice it is limited by the number of independent data sources and/or the number of CPUs near each source. Similarly, a set of 'reducers' can perform the reduction phase - provided all outputs of the map operation that share the same key are presented to the same reducer at the same time.

-- [[http://en.wikipedia.org/wiki/MapReduce][Wikipedia page on MapReduce]]
Topic revision: r1 - 2012-04-26 - brodbd