NLP Toolkit
8 methods分词、词性标注、命名实体识别与预训练模型推理的核心 API。
nltk.word_tokenize(text)将文本切分为单词级别的 token 列表。
Parameters
| Name | Type | Description |
|---|---|---|
| text | str | 待分词的文本 |
Returns
list[str] — 词法单元列表
Example
nlp
import nltk
nltk.download("punkt", quiet=True)
tokens = nltk.word_tokenize("Hello, world! NLP is fun.")
print(tokens)
# ['Hello', ',', 'world', '!', 'NLP', 'is', 'fun', '.']nltk.pos_tag(tokens)对词法单元进行词性标注。
Parameters
| Name | Type | Description |
|---|---|---|
| tokens | list[str] | 已分词的词列表 |
Returns
list[tuple[str, str]] — (词, 词性) 列表
Example
nlp
import nltk
nltk.download("averaged_perceptron_tagger", quiet=True)
tokens = nltk.word_tokenize("The quick brown fox jumps.")
tagged = nltk.pos_tag(tokens)
print(tagged)
# [('The', 'DT'), ('quick', 'JJ'), ('brown', 'JJ'), ('fox', 'NN'), ('jumps', 'VBZ'), ('.', '.')]nltk.ne_chunk(tagged)对已标注词性的文本进行命名实体识别。
Parameters
| Name | Type | Description |
|---|---|---|
| tagged_tokens | list[tuple] | pos_tag 的输出 |
Returns
Tree — 命名实体语法树
Example
nlp
import nltk
nltk.download("maxent_ne_chunker", quiet=True)
nltk.download("words", quiet=True)
text = "Barack Obama was born in Hawaii."
tagged = nltk.pos_tag(nltk.word_tokenize(text))
tree = nltk.ne_chunk(tagged)
print(tree)nltk.sent_tokenize(text)将文本切分为句子列表。
Parameters
| Name | Type | Description |
|---|---|---|
| text | str | 待切分的文本 |
Returns
list[str] — 句子列表
Example
nlp
import nltk
nltk.download("punkt", quiet=True)
text = "Hello there. How are you? I am fine."
sents = nltk.sent_tokenize(text)
print(sents)
# ['Hello there.', 'How are you?', 'I am fine.']nltk.corpus.stopwords.words(lang)返回指定语言的停用词列表。
Parameters
| Name | Type | Description |
|---|---|---|
| language | str | 语言,如 'english' |
Returns
list[str] — 停用词列表
Example
nlp
import nltk
nltk.download("stopwords", quiet=True)
from nltk.corpus import stopwords
words = stopwords.words("english")
print(len(words), words[:5])AutoTokenizer.from_pretrained(name)按模型名加载 Hugging Face 预训练分词器。
Parameters
| Name | Type | Description |
|---|---|---|
| model_name | str | 模型标识,如 'bert-base-uncased' |
Returns
PreTrainedTokenizer — 分词器实例
Example
nlp
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("bert-base-uncased")
enc = tok("Hello, transformers!", return_tensors="pt")
print(enc["input_ids"])AutoModel.from_pretrained(name)按模型名加载 Hugging Face 预训练模型。
Parameters
| Name | Type | Description |
|---|---|---|
| model_name | str | 模型标识 |
Returns
PreTrainedModel — 预训练模型实例
Example
nlp
from transformers import AutoTokenizer, AutoModel
tok = AutoTokenizer.from_pretrained("bert-base-uncased")
model = AutoModel.from_pretrained("bert-base-uncased")
enc = tok("Hello world", return_tensors="pt")
out = model(**enc)
print(out.last_hidden_state.shape)pipeline(task)创建端到端推理流水线,封装分 词、模型与后处理。
Parameters
| Name | Type | Description |
|---|---|---|
| task | str | 任务名,如 'sentiment-analysis' |
Returns
Pipeline — 推理流水线对象
Example
nlp
from transformers import pipeline
clf = pipeline("sentiment-analysis")
result = clf("I love using Hugging Face transformers!")
print(result)
# [{'label': 'POSITIVE', 'score': 0.9998}]