放牧代码和思想
专注自然语言处理、机器学习算法
    时间有限,只有GitHub上的issue能及时处理,大约每周末一次。另外,不要叫我楼主,谢谢。

Lucene 4.6.1 java.lang.IllegalStateException: TokenStream contract violation

这是旧代码在新版本Lucene中出现的异常,异常如下:

Exception in thread "main" java.lang.IllegalStateException: TokenStream contract violation: reset()/close() call missing, reset() called multiple times, or subclass does not call super.reset(). Please see Javadocs of TokenStream class for more information about the correct consuming workflow.
at org.apache.lucene.analysis.Tokenizer$1.read(Tokenizer.java:110)
at java.io.Reader.read(Reader.java:140)
at org.wltea.analyzer.core.AnalyzeContext.fillBuffer(AnalyzeContext.java:124)
at org.wltea.analyzer.core.IKSegmenter.next(IKSegmenter.java:122)
at org.wltea.analyzer.lucene.IKTokenizer.incrementToken(IKTokenizer.java:78)
at com.hankcs.train.IKHelper.parse(IKHelper.java:36)
at com.hankcs.train.AnalysisAdjuster.handleFile(AnalysisAdjuster.java:44)
at com.hankcs.train.AnalysisAdjuster.main(AnalysisAdjuster.java:37)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)

Process finished with exit code 1

旧代码:

IKAnalyzer ss = new IKAnalyzer();
        StringReader reader = new StringReader(str);
        try
        {
            TokenStream tokenStream = ss.tokenStream("", reader);
            while (tokenStream.incrementToken())
            {
                CharTermAttribute termAttribute = tokenStream.getAttribute(CharTermAttribute.class);
                System.out.println(termAttribute.toString());

            }
        } catch (IOException e)
        {
            e.printStackTrace();
        }

根据新的API文档,调用TokenStream API的流程必须是:

The workflow of the new TokenStream API is as follows:

  1. Instantiation of TokenStream/TokenFilters which add/get attributes to/from the AttributeSource.

  2. The consumer calls reset().

  3. The consumer retrieves attributes from the stream and stores local references to all attributes it wants to access.

  4. The consumer calls incrementToken() until it returns false consuming the attributes after each call.

  5. The consumer calls end() so that any end-of-stream operations can be performed.

  6. The consumer calls close() to release any resource when finished using the TokenStream.

所以代码必须在incrementToken()之前调用一次reset()

  IKAnalyzer ss = new IKAnalyzer();
        StringReader reader = new StringReader(str);
        try
        {
            TokenStream tokenStream = ss.tokenStream("", reader);
            tokenStream.reset();
            while (tokenStream.incrementToken())
            {
                CharTermAttribute termAttribute = tokenStream.getAttribute(CharTermAttribute.class);
                System.out.println(termAttribute.toString());

            }
        } catch (IOException e)
        {
            e.printStackTrace();
        }

知识共享许可协议 知识共享署名-非商业性使用-相同方式共享码农场 » Lucene 4.6.1 java.lang.IllegalStateException: TokenStream contract violation

分享到:更多 ()

评论 欢迎留言

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址

我的开源项目

HanLP自然语言处理包基于DoubleArrayTrie的Aho Corasick自动机