Go 语言高效分词, 支持英文、中文、日文等

词典用双数组 trie （ Double-Array Trie ）实现， 分词器算法为基于词频的最短路径加动态规划, DAG 和 HMM (Viterbi) 算法分词, 新增 API 基本和结巴分词保持一致.

支持普通、搜索引擎、全模式、精确模式和 HMM 模式多种分词模式，支持用户词典、词性标注，可运行 JSON RPC 服务。

项目地址: https://github.com/go-ego/gse

Snake River

Add

• [NEW] Add more common APIs and more examples
• [NEW] Export seg.Find() function
• [NEW] Add AddToken and AddTokenForce func
• [NEW] Update dictionary code, add LoadNoFreq and MinTokenFreq parameter configuration
• [NEW] Update examples code, add more examples and documents
• [NEW] Optimize benchmark test tools code
• [NEW] Add seg.Find() test
• [NEW] Add AddToken and AddTokenForce test code
• [NEW] Add crf.go file directory

Update

• [NEW] Add more hmm is bool judge
• [NEW] Update test dictionary and example code
• [NEW] Update segmenter test code
• [NEW] Add clear some internal api name and update godoc
• [NEW] Add go1.12.x remove go1.10.x
• [NEW] Add AddToken and AddTokenForce example code
• [NEW] Add custom dictionary example to example.go
• [NEW] Update godoc
• [NEW] Update CHANGELOG.md

Fix

• [FIX] Optimize code wait todo load model
• [FIX] Fixed some README.md benchmark
• [FIX] Fixed link break
• [FIX] Fixed goroutines benchmark code open path

See Commits for more details, after Jan 23.