tinymr 0.1

截断输出

目前有两个MapReduce实现，一个包括排序，另一个不包括。以下示例将不会从排序中受益，因此我们可以利用不排序的固有优化。API是相同的，但tinymr.memory.MRSerial()在分区后和reducer()与final_reducer()之间再次进行排序。

import json
import re
import sys

from tinymr.memory import MRSerial


class WordCount(MRSerial):

    def __init__(self):
        self.pattern = re.compile('[\W_]+')

    def mapper(self, item):
        for word in item.split():
            word = self.pattern.sub('', word)
            if word:
                yield word.lower(), 1

    def reducer(self, key, values):
        yield key, sum(values)

    def final_reducer(self, pairs):
        return {k: tuple(v)[0] for k, v in pairs}


wc = WordCount()
with open('LICENSE.txt') as f:
    out = wc(f)
    print(json.dumps(out, indent=4, sort_keys=True))

截断输出

{
    "a": 1,
    "above": 2,
    "advised": 1,
    "all": 1,
    "and": 8,
    "andor": 1
}

开发中

$ git clone https://github.com/geowurster/tinymr.git
$ cd tinymr
$ pip install -e .\[dev\]
$ py.test tests --cov tinymr --cov-report term-missing

许可

见LICENSE.txt

变更日志

见CHANGES.md