Python 结巴分词(jieba)Tokenize和ChineseAnalyzer的使用及示例代码

1､Tokenize的使用

返回词语在原文的起止位置

注意：输入参数只接受 unicode

1) 默认模式

esult = jieba.tokenize(u'永和服装饰品有限公司')
for tk in result:
    print("word %s\t\t start: %d \t\t end:%d" % (tk[0],tk[1],tk[2]))

输出：

word 永和                start: 0                end:2
word 服装                start: 2                end:4
word 饰品                start: 4                end:6
word 有限公司            start: 6                end:10

2) 搜索模式

result = jieba.tokenize(u'永和服装饰品有限公司', mode='search')
for tk in result:
    print("word %s\t\t start: %d \t\t end:%d" % (tk[0],tk[1],tk[2]))

输出：

word 永和                start: 0                end:2
word 服装                start: 2                end:4
word 饰品                start: 4                end:6
word 有限                start: 6                end:8
word 公司                start: 8                end:10
word 有限公司            start: 6                end:10

2､ChineseAnalyzer的使用

ChineseAnalyzer for Whoosh 搜索引擎

1) 引用ChineseAnalyzer

from jieba.analyse import ChineseAnalyzer

2) 用法

# -*- coding: UTF-8 -*-
from __future__ import unicode_literals
import sys,os
sys.path.append("../")
from whoosh.index import create_in,open_dir
from whoosh.fields import *
from whoosh.qparser import QueryParser
from jieba.analyse.analyzer import ChineseAnalyzer
analyzer = ChineseAnalyzer()
schema = Schema(title=TEXT(stored=True), path=ID(stored=True), content=TEXT(stored=True, analyzer=analyzer))
if not os.path.exists("tmp"):
    os.mkdir("tmp")
ix = create_in("tmp", schema) # for create new index
#ix = open_dir("tmp") # for read only
writer = ix.writer()
writer.add_document(
    title="document1",
    path="/a",
    content="This is the first document we’ve added!"
)
writer.add_document(
    title="document2",
    path="/b",
    content="The second one 你 中文测试中文 is even more interesting! 吃水果"
)
writer.add_document(
    title="document3",
    path="/c",
    content="买水果然后来世博园。"
)
writer.add_document(
    title="document4",
    path="/c",
    content="工信处女干事每月经过下属科室都要亲口交代24口交换机等技术性器件的安装工作"
)
writer.add_document(
    title="document4",
    path="/c",
    content="咱俩交换一下吧。"
)
writer.commit()
searcher = ix.searcher()
parser = QueryParser("content", schema=ix.schema)
for keyword in ("水果世博园","你","first","中文","交换机","交换"):
    print("result of ",keyword)
    q = parser.parse(keyword)
    results = searcher.search(q)
    for hit in results:
        print(hit.highlights("content"))
    print("="*10)
for t in analyzer("我的好朋友是李明;我爱北京天安门;IBM和Microsoft; I have a dream. this is intetesting and interested me a lot"):
    print(t.text)

官方文档：https://github.com/fxsjy/jieba

相关文档：

Python 结巴分词(jieba)使用方法文档及示例代码

Python 使用结巴分词(jieba)并行分词及示例代码

Python 使用结巴分词(jieba)提取关键词和词性标注方法及示例代码

Python 结巴分词(jieba)的延迟加载机制及示例代码

Python 使用结巴分词(jieba)调用命令行分词及示例代码