Elasticsearch Synonyms And Aliases

Aug 7, 2014 by Jonathan Slate
elasticsearch
author image
Aug 07 2014

You've probably heard of Post Traumatic Stress Disorder (PTSD). Recently, we had a visitor come to our office and talk about PTSD, and he argued that it should be referred to as "Post Traumatic Stress," without the "Disorder" part, because really this condition is a normal human response to extreme events.

We all thought that was a good point, but we don't think the name will change overnight, so we'd like to support both names. For most searches on our site, we are using ElasticSearch, and we wanted PTSD and PTS to be treated as synonyms. ElasticSearch provides synonyms to support just such a case. But implementing the change to use synonyms in production with minimal downtime proved to be harder than expected. Here's the process we followed:

Define the new settings

We created a JSON file to store our new configuration:

{ 
  "analysis": {
    "filter": {
      "english_stop": {
        "type":       "stop",
        "stopwords":  "_english_" 
      },
      "english_keywords": {
        "type":       "keyword_marker",
        "keywords":   ["pts"] 
      },
      "english_stemmer": {
        "type":       "stemmer",
        "language":   "english"
      },
      "english_possessive_stemmer": {
        "type":       "stemmer",
        "language":   "possessive_english"
      },
      "plm_synonyms" : {
        "type":       "synonym",
        "synonyms": [
          "ptsd, pts => ptsd"
        ]
      }
    },
    "analyzer": {
      "english_plm_synonyms": {
        "tokenizer":  "standard",
        "filter": [
          "english_possessive_stemmer",
          "lowercase",
          "plm_synonyms",
          "english_stop",
          "english_keywords",
          "english_stemmer"
        ]
      }
    }  
  }
}

Before this change, we didn't have a custom configuration, we just went with the ElasticSearch defaults. This new configuration defines a set of analysis filters, and then creates an analyzer called english_plm_synonyms that uses those filters. The filters are applied in order:

  1. english_possessive_stemmer: The handles possessives in search, so for example, if you search for Ernie's rubber ducky it will still match Ernie rubber ducky
  2. lowercase: This converts text to lower case for search, so search is case insensitive.
  3. plm_synonyms: Use our synonyms filter so PTSD and PTS are synonyms, which is the whole point!
  4. english_stop: Get rid of common English words.
  5. english_keywords: This prevents stemming of PTS. We needed this because, when searching for PTS on our site, especially in forums, you would often turn up matches for PT (Physical Therapy), with the "S" being considered a stem. But we don't want that, PTS should match PTS and PTSD, but not PT.
  6. english_stemmer: Handle stems like -ing and -s.

-

Create a reindexing rake task

We also created a rake task to copy an existing index over to a new index, here it is slightly simplified.

desc 'reindex from existing index to new index'
task :es_reindex, [:new_index] => :environment do

  [Condition, Symptom, Lab].each do |klass|

    # a method we have that saves a mapping, 
    # which we modified to take an index argument
    klass.es_save_mapping(index: args[:new_index]) 

    client = klass.es_client

    # Open the "view" of the index with the `scan` search_type
    r = client.search index: klass.es_index, 
      search_type: 'scan', scroll: '5m', size: 100, type: klass.name, 
      fields:["_parent","_source","_routing"]

    # Call the `scroll` API until empty results are returned
    loop do
      r = client.scroll(scroll_id: r['_scroll_id'], scroll: '5m') 
      break if r['hits']['hits'].empty?

      # getting this right was a bit tricky...
      body = r['hits']['hits'].map { |hit| { index: { _id: hit['_id'], data: hit['_source'] }
        .merge((hit['fields'].reject{|k,v|k=='fields'} rescue {}))}} 
      client.bulk(index: args[:new_index], type: klass.name, body: body)
    end
  end

end

Moving to the new index

Once those pieces were in place, we were ready for the next step. To minimize downtime, the plan was to:

  1. Create an alias for the ElasticSearch index
  2. Bring down the site, change the configuration to point to the alias, and bring the site back up
  3. Build the new index
  4. Change the alias to point to the new index

-

Change ElasticSearch to use an alias

  • Create an alias called plm_production_alias that points to plm_production
curl -XPUT 'localhost:9200/plm_production/_alias/plm_production_alias'
  • Bring the site down
  • Change elastic_search.yml use the new plm_production_alias
  • Bring site back up

-

Build the new index

  • Create a new index called using a date stamp in the name, we used plm_20140721
curl -XPUT 'http://localhost:9200/plm_20140721/'
  • Apply the new settings we created above to the new index
curl -XPOST localhost:9200/plm_20140721/_close
curl -XPUT -d @config/elastic_index.json localhost:9200/plm_20140721/_settings
curl -XPOST localhost:9200/plm_20140721/_open
  • Run the rake task we made in this spike to reindex all the data from the old index (plm_production) to the new one (plm_20140721)
bundle exec rake es_reindex[plm__20140721]

Make the alias point to the new index

  • Change the plm_production_alias alias to point to plm_20140721. This did not require any down time.
curl -XPOST 'http://localhost:9200/_aliases' -d '
{
    "actions" : [
        { "remove" : { "index" : "plm_production", "alias" : "plm_production_alias" } },
        { "add" : { "index" : "plm_20140721", "alias" : "plm_production_alias" } }
    ]
}
  • We ran the rake task again, to catch any data that came in during the operation
  • Once we were sure all was well, we deleted the old index
curl -XDELETE 'http://localhost:9200/plm_production/'

Handling the change in other environments

The changes to use the new analyzer involves changes to our code base, and those changes assume the new analyzer exists. So if a developer checks out the new code without creating a new index, they will get errors that the analyzer does not exist. They must then either apply the change to their index (adapting the "Apply the new settings.." steps above), or drop and recreate their index. The same would apply for QA environments, etc.

Future changes

In the future, if we want to make additional changes, we will need to:

  • Modify elastic_index.json with our new settings
  • Create a new index
  • Apply the settings to the new index
  • Run the reindex rake task for the new index
  • Change the plm_production_alias alias to point to the new index

-

Automation

Much of this could be automated. I imagine that you would run a rake task that would generate a new configuration JSON file, with a timestamp in the file name. Then you would make your changes to that configuration file, and then run another task that would create a new index, using the same timestamp, apply the new settings to the index, run the reindex, and reassign the alias. Perhaps that will be the topic for a future blog post.