Category: elasticsearch

Elasticsearch – Pie Charts Pie charts are one of the simplest and famous visualization tools. It represents the data as slices of a circle each coloured differently. The labels along with the percentage data values can be presented along with the circle. The circle can also take the shape of a donut. Visualize In Kibana Home screen, we find the option name Visualize which allows us to create visualization and aggregations from the indices stored in Elasticsearch. We choose to add a new visualization and select pie chart as the option shown below. Choose the Metrics The next screen prompts us for choosing the metrics which will be used in creating the Pie Chart. Here we choose the count of base unit price as the metric and Bucket Aggregation as histogram. Also, the minimum interval is chosen as 20. So, the prices will be displayed as blocks of values with 20 as a range. The result below shows the pie chart after we apply the selection. Please note the shades of the colour and their values mentioned in the label. Pie Chart Options On moving to the options tab under pie chart we can see various configuration options to change the look as well as the arrangement of data display in the pie chart. In the following example, the pie chart appears as donut and the labels appear at the top.

Elasticsearch – Testing Elasticsearch provides a jar file, which can be added to any java IDE and can be used to test the code which is related to Elasticsearch. A range of tests can be performed by using the framework provided by Elasticsearch. In this chapter, we will discuss these tests in detail − Unit testing Integration testing Randomized testing Prerequisites To start with testing, you need to add the Elasticsearch testing dependency to your program. You can use maven for this purpose and can add the following in pom.xml. <dependency> <groupId>org.elasticsearch</groupId> <artifactId>elasticsearch</artifactId> <version>2.1.0</version> </dependency> EsSetup has been initialized to start and stop Elasticsearch node and also to create indices. EsSetup esSetup = new EsSetup(); esSetup.execute() function with createIndex will create the indices, you need to specify the settings, type and data. Unit Testing Unit test is carried out by using JUnit and Elasticsearch test framework. Node and indices can be created using Elasticsearch classes and in test method can be used to perform the testing. ESTestCase and ESTokenStreamTestCase classes are used for this testing. Integration Testing Integration testing uses multiple nodes in a cluster. ESIntegTestCase class is used for this testing. There are various methods which make the job of preparing a test case easier. S.No Method & Description 1 refresh() All the indices in a cluster are refreshed 2 ensureGreen() Ensures a green health cluster state 3 ensureYellow() Ensures a yellow health cluster state 4 createIndex(name) Create index with the name passed to this method 5 flush() All indices in cluster are flushed 6 flushAndRefresh() flush() and refresh() 7 indexExists(name) Verifies the existence of specified index 8 clusterService() Returns the cluster service java class 9 cluster() Returns the test cluster class Test Cluster Methods S.No Method & Description 1 ensureAtLeastNumNodes(n) Ensures minimum number of nodes up in a cluster is more than or equal to specified number. 2 ensureAtMostNumNodes(n) Ensures maximum number of nodes up in a cluster is less than or equal to specified number. 3 stopRandomNode() To stop a random node in a cluster 4 stopCurrentMasterNode() To stop the master node 5 stopRandomNonMaster() To stop a random node in a cluster, which is not a master node. 6 buildNode() Create a new node 7 startNode(settings) Start a new node 8 nodeSettings() Override this method for changing node settings. Accessing Clients A client is used to access different nodes in a cluster and carry out some action. ESIntegTestCase.client() method is used for getting a random client. Elasticsearch offers other methods also to access client and those methods can be accessed using ESIntegTestCase.internalCluster() method. S.No Method & Description 1 iterator() This helps you to access all the available clients. 2 masterClient() This returns a client, which is communicating with master node. 3 nonMasterClient() This returns a client, which is not communicating with master node. 4 clientNodeClient() This returns a client currently up on client node. Randomized Testing This testing is used to test the user’s code with every possible data, so that there will be no failure in future with any type of data. Random data is the best option to carry out this testing. Generating Random Data In this testing, the Random class is instantiated by the instance provided by RandomizedTest and offers many methods for getting different types of data. Method Return value getRandom() Instance of random class randomBoolean() Random boolean randomByte() Random byte randomShort() Random short randomInt() Random integer randomLong() Random long randomFloat() Random float randomDouble() Random double randomLocale() Random locale randomTimeZone() Random time zone randomFrom() Random element from array Assertions ElasticsearchAssertions and ElasticsearchGeoAssertions classes contain assertions, which are used for performing some common checks at the time of testing. For example, observe the code given here − SearchResponse seearchResponse = client().prepareSearch(); assertHitCount(searchResponse, 6); assertFirstHit(searchResponse, hasId(“6”)); assertSearchHits(searchResponse, “1”, “2”, “3”, “4”,”5”,”6”);

Elasticsearch – Kibana Dashboard A Kibana dashboard is a collection of visualizations and searches. You can arrange, resize, and edit the dashboard content and then save the dashboard so you can share it. In this chapter, we will see how to create and edit a dashboard. Dashboard Creation From the Kibana Homepage, select the dashboard option from the left control bars as shown below. This will prompt you to create a new dashboard. To Add visualizations to the dashboard, we choose the menu Add and the select from the pre-built visualizations available. We chose the following visualization options from the list. On selecting the above visualizations, we get the dashboard as shown here. We can later add and edit the dashboard for changing the elements and adding the new elements. Inspecting Elements We can inspect the Dashboard elements by choosing the visualizations panel menu and selecting Inspect. This will bring out the data behind the element which also can be downloaded. Sharing Dashboard We can share the dashboard by choosing the share menu and selecting the option to get a hyperlink as shown below −

Discuss Elasticsearch Elasticsearch is a real-time distributed and open source full-text search and analytics engine. It is used in Single Page Application (SPA) projects. Elasticsearch is an open source developed in Java and used by many big organizations around the world. It is licensed under the Apache license version 2.0.

Elasticsearch – Time Series Time series is a representation of sequence of data in a specific time sequence. For example, the data for each day starting from first day of the month to the last day. The interval between the data points remains constant. Any data set which has a time component in it can be represented as a time series. In this chapter, we will use the sample e-commerce data set and plot the count of the number of orders for each day to create a time series. Choose Metrics First, we choose the index pattern, data field and interval which will be used for creating the time series. From the sample ecommerce data set we choose order_date as the field and 1d as the interval. We use the Panel Options tab to make these choices. Also we leave the other values in this tab as default to get a default colour and format for the time series. In the Data tab, we choose count as the aggregation option, group by option as everything and put a label for the time series chart. Result The final result of this configuration appears as follows. Please note that we are using a time period of Month to Date for this graph. Different time periods will give different results.

Elasticsearch – Useful Resources The following resources contain additional information on Elasticsearch. Please use them to get more in-depth knowledge on this topic. Useful Video Courses 11 Lectures 47 mins 47 Lectures 4 hours 61 Lectures 7.5 hours 20 Lectures 1 hours 194 Lectures 18.5 hours 38 Lectures 3.5 hours

Elasticsearch – Logs UI Kibana can also help in visualizing log data from various sources. Logs are important sources of analysis for infrastructure health, performance needs and security breach analysis etc. Kibana can connect to various logs like web server logs, elasticsearch logs and cloudwatch logs etc. Logstash Logs In Kibana, we can connect to logstash logs for visualization. First we choose the Logs button from the Kibana home screen as shown below − Then we choose the option Change Source Configuration which brings us the option to choose Logstash as a source. The below screen also shows other types of options we have as a log source. You can stream data for live log tailing or pause streaming to focus on historical log data. When you are streaming logs, the most recent log appears at the bottom on the console. For further reference, you can refer to our tutorial.

Elasticsearch – Heat Maps Heat map is a type of visualization in which different shades of colour represent different areas in the graph. The values may be continuously varying and hence the colour r shades of a colour vary along with the values. They are very useful to represent both the continuously varying data as well as discrete data. In this chapter we will use the data set named sample_data_flights to build a heatmap chart. In it we consider the variables named origin country and destination country of flights and take a count. In Kibana Home screen, we find the option name Visualize which allows us to create visualization and aggregations from the indices stored in Elasticsearch. We choose to add a new visualization and select Heat Map as the option shown below &mimus; Choose the Metrics The next screen prompts us for choosing the metrics which will be used in creating the Heat Map Chart. Here we choose the count as the type of aggregation metric. Then for the buckets in Y-Axis, we choose Terms as the aggregation for the field OriginCountry. For the X-Axis, we choose the same aggregation but DestCountry as the field to be used. In both the cases, we choose the size of the bucket as 5. On running the above shown configuration, we get the heat map chart generated as follows. Note − You have to allow the date range as This Year so that the graph gathers data for a year to produce an effective heat map chart.

Elasticsearch – Filtering by Field The discover functionality available in Kibana home page allows us to explore the data sets from various angles. You can search and filter data for the selected index patterns. The data is usually available in form of distribution of values over a period of time. To explore the ecommerce data sample, we click on the Discover icon as shown in the picture below. This will bring up the data along with the chart. Filtering by Time To filter out data by specific time interval we use the time filter option as shown below. By default, the filter is set at 15 minutes. Filtering by Fields The data set can also be filtered by fields using the Add Filter option as shown below. Here we add one or more fields and get the corresponding result after the filters are applied. In our example we choose the field day_of_week and then the operator for that field as is and value as Sunday. Next, we click Save with above filter conditions. The result set containing the filter conditions applied is shown below.

Elasticsearch – Frozen Indices The indices that are searched frequently are held in memory because it takes time to rebuild them and help in an efficient search. On the other hand, there may be indices which we rarely access. Those indices need not occupy the memory and can be re-build when they are needed. Such indices are known as frozen indices. Elasticsearch builds the transient data structures of each shard of a frozen index each time that shard is searched and discards these data structures as soon as the search is complete. Because Elasticsearch does not maintain these transient data structures in memory, frozen indices consume much less heap than the normal indices. This allows for a much higher disk-to-heap ratio than would otherwise be possible. Example for Freezing and Unfreezing The following example freezes and unfreezes an index − POST /index_name/_freeze POST /index_name/_unfreeze Searches on frozen indices are expected to execute slowly. Frozen indices are not intended for high search load. It is possible that a search of a frozen index may take seconds or minutes to complete, even if the same searches completed in milliseconds when the indices were not frozen. Searching a Frozen Index The number of concurrently loaded frozen indices per node is limited by the number of threads in the search_throttled threadpool, which is 1 by default. To include frozen indices, a search request must be executed with the query parameter − ignore_throttled=false. GET /index_name/_search?q=user:tpoint&ignore_throttled=false Monitoring Frozen Indices Frozen indices are ordinary indices that use search throttling and a memory efficient shard implementation. GET /_cat/indices/index_name?v&h=i,sth