serpextract 0.7.3

命令行

命令行用法，以逗号分隔并引号括起来的引擎名称和关键词组件

$ serpextract "http://www.google.ca/url?sa=t&rct=j&q=ars%20technica"
"Google","ars technica"

您也可以通过以下方式打印出您本地缓存中当前可用的所有SearchEngineParsers的列表：

$ serpextract -l

Python

from serpextract import get_parser, extract, is_serp, get_all_query_params

non_serp_url = 'http://arstechnica.com/'
serp_url = ('http://www.google.ca/url?sa=t&rct=j&q=ars%20technica&source=web&cd=1&ved=0CCsQFjAA'
            '&url=http%3A%2F%2Farstechnica.com%2F&ei=pf7RUYvhO4LdyAHf9oGAAw&usg=AFQjCNHA7qjcMXh'
            'j-UX9EqSy26wZNlL9LQ&bvm=bv.48572450,d.aWc')

get_all_query_params()
# ['key', 'text', 'search_for', 'searchTerm', 'qrs', 'keyword', ...]

is_serp(serp_url)
# True
is_serp(non_serp_url)
# False

get_parser(serp_url)
# SearchEngineParser(engine_name='Google', keyword_extractor=['q'], link_macro='search?q={k}', charsets=['utf-8'])
get_parser(non_serp_url)
# None

extract(serp_url)
# ExtractResult(engine_name='Google', keyword=u'ars technica', parser=SearchEngineParser(...))
extract(non_serp_url)
# None

简单检测

Matomo及其因此使用的搜索引擎解析器列表远非详尽无遗。如果您想让serpextract尝试猜测给定的引用URL是否是SERP，您可以指定use_naive_method=True给serpextract.is_serp或serpextract.extract。默认情况下，简单方法被禁用。

简单的搜索引擎检测尝试在URL的netloc中找到r'\.?search\.'的实例。如果找到，serpextract将尝试通过查找以下参数顺序查找URL的查询部分中的关键词

_naive_params = ('q', 'query', 'k', 'keyword', 'term',)

如果找到其中之一，将提取一个关键词并构建一个ExtractResult

ExtractResult(domain, keyword, None)  # No parser, but engine name and keyword

# Not a recognized search engine by serpextract
serp_url = 'http://search.piccshare.com/search.php?cat=web&channel=main&hl=en&q=test'

is_serp(serp_url)
# False

extract(serp_url)
# None

is_serp(serp_url, use_naive_method=True)
# True

extract(serp_url, use_naive_method=True)
# ExtractResult(engine_name=u'piccshare', keyword=u'test', parser=None)

自定义解析器

如果您的自定义搜索引擎不受Matomo/serpextract支持，您可以为serpextract.SearchEngineParser创建自己的实例，并将其显式传递给serpextract.is_serp或serpextract.extract，或者将其添加到解析器内部列表。

# Create a parser for PiccShare
from serpextract import SearchEngineParser, is_serp, extract

my_parser = SearchEngineParser(u'PiccShare',          # Engine name
                               u'q',                  # Keyword extractor
                               u'/search.php?q={k}',  # Link macro
                               u'utf-8')              # Charset
serp_url = 'http://search.piccshare.com/search.php?cat=web&channel=main&hl=en&q=test'

is_serp(serp_url)
# False

extract(serp_url)
# None

is_serp(serp_url, parser=my_parser)
# True

extract(serp_url, parser=my_parser)
# ExtractResult(engine_name=u'PiccShare', keyword=u'test', parser=SearchEngineParser(engine_name=u'PiccShare', keyword_extractor=[u'q'], link_macro=u'/search.php?q={k}', charsets=[u'utf-8']))

您还可以永久将自定义解析器添加到serpextract维护的内部解析器列表，这样您就不再需要显式传递解析器对象给serpextract.is_serp或serpextract.extract。

from serpextract import SearchEngineParser, add_custom_parser, is_serp, extract

my_parser = SearchEngineParser(u'PiccShare',          # Engine name
                               u'q',                  # Keyword extractor
                               u'/search.php?q={k}',  # Link macro
                               u'utf-8')              # Charset
add_custom_parser(u'search.piccshare.com', my_parser)

serp_url = 'http://search.piccshare.com/search.php?cat=web&channel=main&hl=en&q=test'
is_serp(serp_url)
# True

extract(serp_url)
# ExtractResult(engine_name=u'PiccShare', keyword=u'test', parser=SearchEngineParser(engine_name=u'PiccShare', keyword_extractor=[u'q'], link_macro=u'/search.php?q={k}', charsets=[u'utf-8']))