Food Log with Speech Recognition and NLP

1. 分词 word segmentation

国内有jieba 分词

2. Named Entity Recognition

  1. 训练自己的Model

      

How can I train my own NER model

https://nlp.stanford.edu/software/crf-faq.html#a

C:\my_study\ML\NLP\stanford-ner--->java -cp stanford-ner.jar edu.stanford.nlp.ie.crf.CRFClassifier -prop chinese.meal.fpp.prop
Invoked on Thu Mar :: CST with arguments: -prop chinese.meal.fpp.prop
usePrevSequences=true
useClassFeature=true
useTypeSeqs2=true
useSequences=true
wordShape=chris2useLC
useTypeySequences=true
useDisjunctive=true
noMidNGrams=true
serializeTo=ner-model.ser.gz
maxNGramLeng=
useNGrams=true
usePrev=true
useNext=true
maxLeft=
trainFile=chinese.meal.fpp.tsv
map=word=,answer=
useWord=true
useTypeSeqs=true
numFeatures =
Time to convert docs to feature indices: 0.0 seconds
numClasses: [=O,=TIME,=QUANTITY,=UNIT,=FOOD]
numDocuments:
numDatums:
numFeatures:
Time to convert docs to data/labels: 0.0 seconds
numWeights:
QNMinimizer called on double function of variables, using M = .
An explanation of the output:
Iter The number of iterations
evals The number of function evaluations
SCALING <D> Diagonal scaling was used; <I> Scaled Identity
LINESEARCH [## M steplength] Minpack linesearch
-Function value was too high
-Value ok, gradient positive, positive curvature
-Value ok, gradient negative, positive curvature
-Value ok, gradient negative, negative curvature
[.. B] Backtracking
VALUE The current function value
TIME Total elapsed time
|GNORM| The current norm of the gradient
{RELNORM} The ratio of the current to initial gradient norms
AVEIMPROVE The average improvement / current value
EVALSCORE The last available eval score Iter ## evals ## <SCALING> [LINESEARCH] VALUE TIME |GNORM| {RELNORM} AVEIMPROVE EVALSCORE Iter evals <D> [M 1.000E-1] 9.068E2 .04s |4.550E1| {4.995E-1} 0.000E0 -
Iter evals <D> [M 1.000E0] 6.222E2 .05s |3.525E1| {3.870E-1} 2.287E-1 -
Iter evals <D> [M 1.000E0] 2.386E2 .07s |5.406E1| {5.935E-1} 9.334E-1 -
Iter evals <D> [M 1.000E0] 9.082E1 .08s |1.571E1| {1.724E-1} 2.246E0 -
Iter evals <D> [M 1.000E0] 7.031E1 .10s |1.181E1| {1.297E-1} 2.379E0 -
Iter evals <D> [M 1.000E0] 5.308E1 .11s |1.025E1| {1.125E-1} 2.681E0 -
Iter evals <D> [1M 2.740E-1] 2.988E1 .14s |7.586E0| {8.328E-2} 4.193E0 -
Iter evals <D> [1M 1.292E-1] 2.234E1 .16s |6.471E0| {7.105E-2} 4.949E0 -
Iter evals <D> [1M 1.801E-1] 1.615E1 .18s |5.573E0| {6.118E-2} 6.127E0 -
Iter evals <D> [1M 1.815E-1] 1.218E1 .24s |4.477E0| {4.915E-2} 7.346E0 -
Iter evals <D> [1M 3.119E-1] 8.873E0 .30s |4.694E0| {5.154E-2} 6.912E0 -
Iter evals <D> [1M 4.760E-1] 6.621E0 .31s |2.092E0| {2.296E-2} 3.504E0 -
Iter evals <D> [M 1.000E0] 6.093E0 .32s |1.906E0| {2.092E-2} 1.390E0 -
Iter evals <D> [M 1.000E0] 5.844E0 .33s |9.067E-1| {9.955E-3} 1.103E0 -
Iter evals <D> [M 1.000E0] 5.721E0 .33s |5.774E-1| {6.339E-3} 8.279E-1 -
Iter evals <D> [M 1.000E0] 5.660E0 .34s |3.535E-1| {3.881E-3} 4.279E-1 -
Iter evals <D> [M 1.000E0] 5.640E0 .35s |1.946E-1| {2.137E-3} 2.961E-1 -
Iter evals <D> [M 1.000E0] 5.632E0 .36s |7.832E-2| {8.599E-4} 1.868E-1 -
Iter evals <D> [M 1.000E0] 5.631E0 .38s |3.559E-2| {3.907E-4} 1.163E-1 -
Iter evals <D> [M 1.000E0] 5.631E0 .39s |2.149E-2| {2.359E-4} 5.758E-2 -
Iter evals <D> [M 1.000E0] 5.631E0 .41s |1.027E-2| {1.128E-4} 1.758E-2 -
Iter evals <D> [M 1.000E0] 5.631E0 .42s |3.631E-3| {3.986E-5} 8.218E-3 -
Iter evals <D> [M 1.000E0] 5.631E0 .44s |1.629E-3| {1.789E-5} 3.791E-3 -
Iter evals <D> [M 1.000E0] 5.631E0 .45s |9.548E-4| {1.048E-5} 1.596E-3 -
Iter evals <D> [M 1.000E0] 5.631E0 .45s |5.724E-4| {6.284E-6} 5.196E-4 -
Iter evals <D> [M 1.000E0] 5.631E0 .47s |1.578E-4| {1.732E-6} 1.686E-4 -
QNMinimizer terminated due to average improvement: | newest_val - previous_val | / |newestVal| < TOL
Total time spent in optimization: .49s
CRFClassifier training ... done [0.6 sec].
Serializing classifier to ner-model.ser.gz... done.

2. 使用训练好的Model来evaluate 一下,看看效果怎么样.

C:\my_study\ML\NLP\stanford-ner--->java -cp stanford-ner.jar edu.stanford.nlp.ie.crf.CRFClassifier -loadClassifier ner-model.ser.gz -testFile chinese.meal.fpp.test.tsv
Invoked on Thu Mar :: CST with arguments: -loadClassifier ner-model.ser.gz -testFile chinese.meal.fpp.test.tsv
testFile=chinese.meal.fpp.test.tsv
loadClassifier=ner-model.ser.gz
Loading classifier from ner-model.ser.gz ... done [0.1 sec].
我 O O
今天 O O
晚上 TIME TIME
吃 O O
了 O O
两 QUANTITY QUANTITY
盘 UNIT UNIT
回锅肉 FOOD FOOD CRFClassifier tagged words in documents at 88.89 words per second.
Entity P R F1 TP FP FN
FOOD 1.0000 1.0000 1.0000
QUANTITY 1.0000 1.0000 1.0000
TIME 1.0000 1.0000 1.0000
UNIT 1.0000 1.0000 1.0000
Totals 1.0000 1.0000 1.0000

还不错哦!

Ref:

1. Standford NLP NER: https://nlp.stanford.edu/software/CRF-NER.html

上一篇:通过yum安装mysql数据


下一篇:Git配合Tag的代码回滚