nori_part_of_speech token filter

The nori_part_of_speech token filter removes tokens that match a set of part-of-speech tags. The list of supported tags and their meanings can be found here: Part of speech tags

It accepts the following setting:

stoptags: An array of part-of-speech tags that should be removed.

and defaults to:

"stoptags": [
    "IC",
    "MAG", "MAJ", "MM",
    "SP", "SSC", "SSO", "SC", "SE",
    "XPN", "XSA", "XSN", "XSV",
    "UNA", "NA", "VSV"
]

Warning

The default stoptags include XPN (prefix). For Korean terms where a meaning-carrying prefix is emitted as XPN, the default nori analyzer removes the prefix. For example, 비급여 is analyzed as 급여:

GET _analyze
{
  "analyzer": "nori",
  "text": "비급여"
}

Which responds with:

{
  "tokens" : [ {
    "token" : "급여",
    "start_offset" : 1,
    "end_offset" : 3,
    "type" : "word",
    "position" : 1
  } ]
}

To preserve the full term, add entries such as 비급여 with user_dictionary_rules so they are emitted as a single noun token. To preserve prefix tokens globally, define a custom stoptags list that omits XPN; this preserves prefixes such as 비 but can add prefix noise.

For example:

PUT nori_sample
{
  "settings": {
    "index": {
      "analysis": {
        "analyzer": {
          "my_analyzer": {
            "tokenizer": "nori_tokenizer",
            "filter": [
              "my_posfilter"
            ]
          }
        },
        "filter": {
          "my_posfilter": {
            "type": "nori_part_of_speech",
            "stoptags": [
              "NR"
            ]
          }
        }
      }
    }
  }
}
GET nori_sample/_analyze
{
  "analyzer": "my_analyzer",
  "text": "여섯 용이"
}

Korean numerals should be removed (NR)
Six dragons

Which responds with:

{
  "tokens" : [ {
    "token" : "용",
    "start_offset" : 3,
    "end_offset" : 4,
    "type" : "word",
    "position" : 1
  }, {
    "token" : "이",
    "start_offset" : 4,
    "end_offset" : 5,
    "type" : "word",
    "position" : 2
  } ]
}