Open In App

Elasticsearch – Painless is Really Painless

Improve
Improve
Like Article
Like
Save
Share
Report

Elasticsearch has been a really cool kid in the town for quite a long time when it comes to full-text search. So many companies like Uber, Slack, Udemy, etc. uses it for the search. There are many more advantages of Elasticsearch other than just full-text search like powerful tools for analysis of data for scaling with ease. We are living in an era where the data is flowing like anything, and data in the system often changes with the requirements. We might come across situations where we need to update or modify the data.

In this article, we are going to update the documents in the Elasticsearch as following:

  1. Add a new field in all the records.
  2. Update a field on condition.
  3. Add a field on condition.
  4. Remove a field.
  5. Remove a field on condition.

To update the data, we are going to use a scripting language that is particularly designed for Elasticsearch, Painless.
To perform the above-mentioned operations, we’ll use Kibana. 

1. Create Index

To insert and update data, first, we’ll have to create an index on which we’ll do all our operations. To do so follow the below statement:

Syntax : PUT /employee

The above query is going to create an empty index (called employee). After creating the index successfully, we’ll insert some records in the employee index.

 

2. Insert Data.

Before inserting the data, let’s have a look at the structure of a document. Below are the fields and field type of documents.

# sample fields and field types
{
   "id":"number",
   "firstName":"string",
   "lastName":"string",
   "address":[
      {
         "Street":"string",
         "City":"string",
         "State":"string"
      },
      {
         "Street":"string",
         "City":"string",
         "State":"string"
      }
   ],
   "email":"string"
}

We can also insert a single record using the below query:

PUT employee/_doc/4
{
  "id": 4,
  "firstName": "Paul",
  "lastName": "walker",
  "email": "paulwalker@org.com",
  "address": [
    {
      "Street": "42 Oberoy street",
      "City": "Ohio",
      "State": "Nestville"
    }
  ]
}

Output:

3. Bulk Insert Data:

The below query performs bulk insertion operation on the index. The below three records will be inserted, after successfully inserting the data, let’s check what we have till now.

Bulk Insert (use autoindent option to indent query)

POST _bulk
{"create":{"_index":"employee","_id":"1"}}
{"id":1,"firstName":"Alice","lastName":"White","email":"alicewhite@org.com",
"address":[{"Street":"2446 McDowell Street","City":"Palmyra","State":"Tennessee"},{"Street":"4809 Blackwell Street","City":"Dry Creek","State":"Alaska"}]}

{"create":{"_index":"employee","_id":"2"}}
{"id":2,"firstName":"Andrew","lastName":"Dunn","email":"andrewdunn@org.com",
"address":[{"Street":"101 Ramsgate Rd","City":"Wildboarclough","State":"Alaska"}]}

{"create":{"_index":"employee","_id":"3"}}
{"id":3,"firstName":"Louis","lastName":"Hale","email":"louishale@org.com",
"address":[{"Street":"84 Main St","City":"Archamore","State":"Las Vegas"}]}

Output:

4. Fetch Records

The below query is used to get the data from the specified index(employee). We’ll get the records which we’ve inserted(3 records):

GET employee/_search

Output:

5. Add a new field in all the records.

We forgot to add the role field in employee records. Now what? Do we have to remove all the records and add the records with the role field? We don’t have to do that. We are going to add just one more field in all the records.

Let’s add a new field role with a value emp. The below query will add a new field inside all our documents:

POST /employee/_update_by_query/
{
  "script": "ctx._source.role = 'emp'"
}

Output:

After adding a new field, obviously, the structure of our document is going to be changed. This will be the revised structure after adding a new field.

{
   "id":"number",
   "firstName":"string",
   "lastName":"string",
   "address":[
      {
         "Street":"string",
         "City":"string",
         "State":"string"
      },
      {
         "Street":"string",
         "City":"string",
         "State":"string"
      }
   ],
   "email":"string",
   "role":"string"
}

6. Update field value based on condition.

Now, it’s not necessary that all the employees have the same role. Some of them are admin or manager etc. Let’s say we encounter a case where we want to change the role of some employees based on some condition. The below query will change the role of the employees who have in-state as Alaska in at least one of the addresses. We’re looping over all the addresses array of the employee object and checking condition if the state is Alaska, if it evaluates to true, we’re setting the role as an admin. 

POST /employee/_update_by_query
{
  "script": {
    "source": "for(a in ctx._source.address){if(a.State == 'Alaska'){ctx._source.role = 'admin'}}",
    "lang": "painless"
  }
}

Output:

7. Add a field based on condition.

In query no. 4, we added the role field in all the records. But let’s say we don’t want to set the field value the same in all the documents. Now, to add the different values in some records, there will be some conditions upon which we’re going to decide what to set? Let’s add one more field isAdmin based on the role’s value. If the role is admin then set isAdmin to true else false.

POST /employee/_update_by_query
{
  "script": {
    "source": "if(ctx._source.role == 'admin'){ctx._source.isAdmin = true} else{ctx._source.isAdmin = false}",
    "lang": "painless"
  }
}

Here we’re checking simple if/else condition. The above query will set isAdmin to true if the role is admin otherwise false.

Output:

After again adding a new field, schema also gets changed,

{
   "id":"number",
   "firstName":"string",
   "lastName":"string",
   "address":[
      {
         "Street":"string",
         "City":"string",
         "State":"string"
      },
      {
         "Street":"string",
         "City":"string",
         "State":"string"
      }
   ],
   "email":"string",
   "role":"string",
   "isAdmin":"boolean"
}

8. Rename a field

We humans often make some errors, typos which often leads to production go down. But more important is getting the system running back to normal. Let’s say we are referring to the state in addresses array in our code, but our records have State, which will produce errors. Now, we want to change the field name from State -> state.

Let’s rename a field inside an address array. State to address state. Sometimes it occurs we insert data with a typo in the fields name, will use the below query to fix the typo, and rename the key:

POST /employee/_update_by_query
{
  "script": {
    "source": "for(a in ctx._source.address){a.state = a.State; a.remove('State')}",
    "lang": "painless"
  }
}

 In the above query, we’re assigning State‘s value to state and then removing the State field from all the elements of the array of all the records.

Output:

9. Remove a field from all the records.

Data gets larger day by day, so it’s storing cost. We as developers should store what is necessary only, removing unnecessary fields which often creates discrepancy or irregularities becomes essential. Let’s remove the role field from all the records:

POST /employee/_update_by_query
{
  "script": {
    "source": "ctx._source.remove('role')",
    "lang": "painless"
  }
}

The remove function accepts the field name to remove. The above query removes the role field from all the records. 

Output:

After removing the field, we left with the below schema,

{
   "id":"number",
   "firstName":"string",
   "lastName":"string",
   "address":[
      {
         "Street":"string",
         "City":"string",
         "State":"string"
      },
      {
         "Street":"string",
         "City":"string",
         "State":"string"
      }
   ],
   "email":"string",
   "isAdmin":"boolean"
}

Add, update, remove, fetch operations are the crux of any system. These operations are used in day to day life of a developer. 



Last Updated : 18 Jan, 2021
Like Article
Save Article
Previous
Next
Share your thoughts in the comments