jieba

package

v1.101.9 Latest Latest Go to latest Published: Jan 13, 2020 License: MIT Imports: 9 Imported by: 0

Details

View Source

const DictDirEnvName = "JIEBA_DICT_DIR"

View Source

const FilterName = "filter_jieba"

This section is empty.

func JiebaFilterConstructor(config map[string]interface{}, cache *registry.Cache) (analysis.TokenFilter, error)

type JiebaFilter struct {
	// contains filtered or unexported fields
}

JiebaFilter implements word segmentation for Chinese. It's a filter so that is can used with other tokenizer (e.g. unicode).

func NewJiebaFilter(dictDir string, searchMode, useHMM bool) *JiebaFilter

func (f *JiebaFilter) Filter(input analysis.TokenStream) analysis.TokenStream

func (f *JiebaFilter) JiebaInstance() *JiebaInstance

JiebaInstance returns the underly JiebaInstance.

type JiebaInstance struct {
	// contains filtered or unexported fields
}

JiebaInstance is a thread-safe *gojieba.Jieba for a given dict directory.

func FindAllJiebaInstances() []*JiebaInstance

FindAllJiebaInstances returns all exists JiebaInstances.

func FindJiebaInstance(dictDir string) *JiebaInstance

FindJiebaInstance returns an exists JiebaInstance for a given dict directory or nil if not found.

func NewJiebaInstance(dictDir string) *JiebaInstance

NewJiebaInstance creates a new JiebaInstance or returns an exists JiebaInstance for a given dict directory.

func (inst *JiebaInstance) DictDir() string

DictDir returns the dict directory.

func (inst *JiebaInstance) Get() (*gojieba.Jieba, func())

Get returns *gojieba.Jieba and a defer function which MUST be called after using.

func (inst *JiebaInstance) LoadTime() (t time.Time, dur time.Duration)

LoadTime returns the load time of data.

func (inst *JiebaInstance) Reload()

Reload reloads data.