Map-Reduce is a programming paradigm used for processing large-scale datasets. It helps simplify the process of parallel computation and improves computational efficiency.
This article is first published in the medium MPP plan. If you are a medium user, please follow me in
In this example, we define a
MapFunctionthat takes a string array and converts each element to uppercase using a custom functionfn, returning a channel. TheReduceFunctiontakes a channel and a custom functionfnto concatenate the results and print them out.
The following image provides a metaphor that vividly illustrates the business semantics of Map-Reduce, which is very useful in data processing.
You may understand that Map/Reduce is just a control logic, and the real business logic is defined by the data and the function passed to them. Yes, this is a classic programming pattern of separating "business logic" from "control logic." Now let's take a look at a code example with meaningful business logic to reinforce the understanding of separating "control logic" and "business logic."
Business Example
Employee Information
First, we have an employee object and some data:
CODEtype Employee struct {
Name string
Age int
Vacation int
Salary int
}
var list = []Employee{
{"Hao", 44, 0, 8000},
{"Bob", 34, 10, 5000},
{"Alice", 23, 5, 9000},
{"Jack", 26, 0, 4000},
{"Tom", 48, 9, 7500},
{"Marry", 29, 0, 6000},
{"Mike", 32, 8, 4000},
}
Related Reduce/Filter Functions
CODEfunc EmployeeCountIf(list []Employee, fn func(e *Employee) bool) int {
count := 0
for i, _ := range list {
if fn(&list[i]) {
count += 1
}
}
return count
}
func EmployeeFilterIn(list []Employee, fn func(e *Employee) bool) []Employee {
var newList []Employee
for i, _ := range list {
if fn(&list[i]) {
newList = append(newList, list[i])
}
}
return newList
}
func EmployeeSumIf(list []Employee, fn func(e *Employee) int) int {
var sum = 0
for i, _ := range list {
sum += fn(&list[i])
}
return sum
}
Here's a brief explanation:
EmployeeCountIfandEmployeeSumIfare used to count the number of employees or calculate the total based on a certain condition. They represent the semantics of Filter + Reduce.
EmployeeFilterInfilters the employees based on a certain condition. It represents the semantics of Filter.
Now we can have the following code:
1) Count the number of employees over 40 years old:
CODEold := EmployeeCountIf(list, func(e *Employee) bool {
return e.Age > 40
})
fmt.Printf("Old people: %d\n", old)
//Old people: 2
2) Count the number of employees with a salary greater than 6000:
CODEhighPay := EmployeeCountIf(list, func(e *Employee) bool {
return e.Salary >= 6000
})
fmt.Printf("High Salary people: %d\n", highPay)
//High Salary people: 4
3) List employees who have not taken any vacation:
CODEnoVacation := EmployeeFilterIn(list, func(e *Employee) bool {
return e.Vacation == 0
})
fmt.Printf("People with no vacation: %v\n", noVacation)
The Map-Reduce programming paradigm divides the computational task into Map and Reduce phases. Although writing single-machine code may not be faster than a simple for loop and may appear complex, in the era of cloud-native computing, we can leverage parallel computation and shared data access to improve computational efficiency. It is a powerful tool suitable for handling large-scale data and parallel computing scenarios, such as the original Google PageRank algorithm. The main purpose of learning it is to understand its mindset.
↗ Original-Artikel auf dev.to lesenVollständiger Original-BerichtAusführliche Details, Code-Beispiele & Hersteller-Stellungnahme auf dev.to.
SOCIAL SHARE CARD GENERATOR