Category: splunk

Splunk – Apps ”; Previous Next A Splunk app is an extension of Splunk functionality which has its own in-built UI context to serve a specific need. Splunk apps are made up of different Splunk knowledge objects (lookups, tags, eventtypes, savedsearches, etc). Apps themselves can utilize or leverage other apps or add-ons. Splunk can run any number of apps simultaneously. When you log in to Splunk, you land on an app which is typically, the Splunk Search app. So, almost everytime you are inside the Splunk interface, you are using an app. Listing Splunk Apps We can list the available apps in Splunk by using the option Apps → Manage Apps. Navigating this option brings out the following screen which lists the existing apps available in Splunk interface. Following are important values associated with the Splunk apps − Name − It is the name of the App and unique for each App. Folder name It is the name to use for the directory in $SPLUNK_HOME/etc/apps/. The name of the folder cannot contain “dot” (.) character. Version − It is the app version string. Visible Indicates whether the app should be visible in Splunk Web. Apps that contain a user interface should be visible. Sharing − It is the level of permissions (read or write) given to different Splunk users for that specific app. Status − Status: It is the current status of availability of the App. It may be enabled or disabled for use. App Permissions A proper setting of permissions for using the app is important. We can restrict the app to be used by a single user or by multiple users including all users. The below screen which appears after clicking on the permissions link in the above is used to modify the access to different roles. By default, the check marks for Read and Write option is available for Everyone. But we can change that by going to each role and selecting appropriate permission for that specific role. App Marketplace There is a wide variety of needs for which the Splunk search functionalities are used. So, there is a Splunk App market place which has come into existence show casing many different apps created by individual and organizations. They are available in both free and paid versions. We can browse those apps by choosing the option Apps → Manage Apps → Browse More Apps. The below screen comes up. As you can see, the App name along with a brief description of the functionality of the App appears. This helps you decide which app to use. Also, note how the Apps are categorized in the left bar to help choose the type of App faster. Print Page Previous Next Advertisements ”;

Splunk – Search Optimization ”; Previous Next Splunk already includes the optimization features, analyses and processes your searches for maximum efficiency. This efficiency is mainly achieved through the following two optimization goals − Early Filtering − These optimizations filter the results very early so that the amount of data getting processed is reduced as early as possible during the search process. This early filter avoids unnecessary lookup and evaluation calculations for events that are not part of final search results. Parallel Processing − The built-in optimizations can reorder search processing, so that as many commands as possible are run in parallel on the indexers before sending the search results to the search head for final processing. Analysing Search Optimisations Splunk has given us tools to analyse how the search optimization works. These tools help us figure out how the filter conditions are used and what is the sequence of these optimisation steps. It also gives us the cost of the various steps involved in the search operations. Example Consider a search operation to find the events which contain the words: fail, failed or password. When we put this search query in the search box, the built-in optimizers act automatically to decide the path of the search. We can verify how long the search took to return a specific number of search results and if needed can go on to check each and every step of the optimization along with the cost associated with it. We follow the path of Search → Job → Inspect Job to get these details as shown below − The next screen gives details of the optimization that has occurred for the above query. Here, we need to note the number of events and the time taken to return the result. Turning Off Optimization We can also turn off the in-built optimization and notice the difference in the time taken for the search result. The result may or may not be better than the in-built search. In case it is better, we may always choose this option of turning off the optimization for only this specific search. In the below diagram, we use the No Optimization command presented as noop in the search query. The next screen gives us the result of using no optimization. For this given query, the results come faster without using in-built optimizations. Print Page Previous Next Advertisements ”;

Splunk – Time Range Search ”; Previous Next The Splunk web interface displays timeline which indicates the distribution of events over a range of time. There are preset time intervals from which you can select a specific time range, or you can customize the time range as per your need. The below screen shows various preset timeline options. Choosing any of these options will fetch the data for only that specific time period which you can also analyse further, using the custom timeline options available. For example, choosing the previous month option gives us the result only for the previous month as you can see the in spread of the timeline graph below. Selecting a Time Subset By clicking and dragging across the bars in the timeline, we can select a subset of the result that already exists. This does not cause the re-execution of the query. It only filters out the records from the existing result set. Below image shows the selection of a subset from the result set − Earliest and Latest The two commands, earliest and latest can be used in the search bar to indicate the time range in between which you filter out the results. It is similar to selecting the time subset, but it is through commands rather than the option of clicking at a specific time line bar. So, it provides a finer control over that data range you can pick for your analysis. In the above image, we give a time range between last 7 days to last 15 days. So, the data in between these two days is displayed. Nearby Events We can also find nearby events of a specific time by mentioning how close we want the events to be filtered out. We have the option of choosing the scale of the interval, like – seconds, minutes, days and week etc. Print Page Previous Next Advertisements ”;

Splunk – Environment ”; Previous Next In this tutorial, we will aim to install the enterprise version. This version is available for a free evaluation for 60 days with all features enabled. You can download the setup using the below link which is available for both windows and Linux platforms. https://www.splunk.com/en_us/download/splunk-enterprise.html. Linux Version The Linux version is downloaded from the download link given above. We choose the .deb package type as the installation will be done in a Ubuntu platform. We shall learn this with a step by step approach − Step 1 Download the .deb package as shown in the screenshot below − Step 2 Go to the download directory and install Splunk using the above downloaded package. Step 3 Next you can start Splunk by using the following command with accept license argument. It will ask for administrator user name and password which you should provide and remember. Step 4 The Splunk server starts and mentions the URL where the Splunk interface can be accessed. Step 5 Now, you can access the Splunk URL and enter the admin user ID and password created in step 3. Windows Version The windows version is available as a msi installer as shown in the below image − Double clicking on the msi installer installs the Windows version in a straight forward process. The two important steps where we must make the right choice for successful installation are as follows. Step 1 As we are installing it on a local system, choose the local system option as given below − Step 2 Enter the password for the administrator and remember it, as it will be used in the future configurations. Step 3 In the final step, we see that Splunk is successfully installed and it can be launched from the web browser. Step 4 Next, open the browser and enter the given url, http://localhost:8000, and login to the Splunk using the admin user ID and password. Print Page Previous Next Advertisements ”;

Splunk – Pivot and Datasets ”; Previous Next Splunk can ingest different types of data sources and build tables which are similar to relational tables. These are called table dataset or just tables. They provide easy ways to analyse and filter the data and lookups, etc. These table data sets are also used in creating pivot analysis which we learn in this chapter. Creating a Dataset We use a Splunk Add-on named Splunk Datasets Add-on to create and manage the datasets. It can be downloaded from the Splunk website, https://splunkbase.splunk.com/app/3245/#/details. It has to be installed by following the instructions given in the details tab in this link. On successful installation, we see a button named Create New Table Dataset. Selecting a Dataset Next, we click on the Create New Table Dataset button and it gives us the option to choose from the below three options. Indexes and Source Types − Choose from an existing index or source type which are already added to Splunk through Add Data app. Existing Datasets − You might have already created some dataset previously which you want to modify by creating a new dataset from it. Search − Write a search query and the result can be used to create a new dataset. In our example, we choose an index to be our source of data set as shown in the image below − Choosing Dataset Fields On clicking OK in the above screen, we are presented with an option to choose the various fields we want to finally get into the Table Dataset. The _time field is selected by default and this field cannot be dropped. We choose the fields: bytes, categoryID, clientIP and files. On clicking done in the above screen, we get the final dataset table with all the selected fields, as seen below. Here the dataset has become similar to a relational table. We save the dataset with save as option available in the top right corner. Creating Pivot We use the above dataset to create a pivot report. The pivot report reflects aggregation of values of one column with respect to the values in another column. In other words, one columns values are made into rows and another columns values are made into rows. Choose Dataset Action To achieve this, we first select the dataset using the dataset tab and then choose the option Visualize with Pivot from the Actions column for that data set. Choose the Pivot Fields Next, we choose the appropriate fields for creating the pivot table. We choose category ID in the split columns option as this is the field whose values should appear as different columns in the report. Then we choose File in the Split Rows option as this is the field whose values should be presented in rows. The result shows count of each categoryid values for each value in the file field. Next, we can save the pivot table as a Report or a panel in an existing dashboard for future reference. Print Page Previous Next Advertisements ”;

Splunk – Source Types ”; Previous Next All the incoming data to Splunk are first judged by its inbuilt data processing unit and classified to certain data types and categories. For example, if it is a log from apache web server, Splunk is able to recognize that and create appropriate fields out of the data read. This feature in Splunk is called source type detection and it uses its built-in source types that are known as “pretrained” source types to achieve this. This makes things easier for analysis as the user does not have to manually classify the data and assign any data types to the fields of the incoming data. Supported Source Types The supported source types in Splunk can be seen by uploading a file through the Add Data feature and then selecting the dropdown for Source Type. In the below image, we have uploaded a CSV file and then checked for all the available options. Source Type Sub-Category Even in those categories, we can further click to see all the sub categories that are supported. So when you choose the database category, you can find the different types of databases and their supported files which Splunk can recognize. Pre-Trained Source Types The below table lists some of the important pre-trained source types Splunk recognizes − Source Type Name Nature access_combined NCSA combined format http web server logs (can be generated by apache or other web servers) access_combined_wcookie NCSA combined format http web server logs (can be generated by apache or other web servers), with cookie field added at end apache_error Standard Apache web server error log linux_messages_syslog Standard linux syslog (/var/log/messages on most platforms) log4j Log4j standard output produced by any J2EE server using log4j mysqld_error Standard mysql error log Print Page Previous Next Advertisements ”;

Splunk Tutorial PDF Version Quick Guide Resources Job Search Discussion Splunk is a software used to search and analyze machine data. This machine data can come from web applications, sensors, devices or any data created by user. It serves the needs of IT infrastructure by analyzing the logs generated in various processes but it can also analyze any structured or semi-structured data with proper data modelling. It has built-in features to recognize the data types, field separators and optimize the search processes. It also provides data visualization on the search results. Audience This tutorial targets IT professionals, students, and IT infrastructure management professionals who want a solid grasp of essential Splunk concepts. After completing this tutorial, you will achieve intermediate expertise in Splunk, and easily build on your knowledge to solve more challenging problems. Prerequisites The reader should be familiar with querying language like SQL. General knowledge in typical operations in using computer applications like storing and retrieving data and reading the logs generated by computer programs will be an highly useful. Print Page Previous Next Advertisements ”;

Splunk – Interface ”; Previous Next The Splunk web interface consists of all the tools you need to search, report and analyse the data that is ingested. The same web interface provides features for administering the users and their roles. It also provides links for data ingestion and the in-built apps available in Splunk. The below picture shows the initial screen after your login to Splunk with the admin credentials. Administrator Link The Administrator drop down gives the option to set and edit the details of the administrator. We can reset the admin email ID and password using the below screen − Further from the administrator link, we can also navigate to the preferences option where we can set the time zone and home application on which the landing page will open after your login. Currently, it opened on the Home page as shown below − Settings Link This is a link which shows all the core features available in Splunk. For example, you can add the lookup files and lookup definitions by choosing the lookup link. We will discuss the important settings of these links in the subsequent chapters. Search and Reporting Link The search and reporting link takes us to the features where we can find the data sets that are available for searching the reports and alerts created for these searches. It is clearly shown in the below screenshot − Print Page Previous Next Advertisements ”;

Splunk – Overview ”; Previous Next Splunk is a software which processes and brings out insight from machine data and other forms of big data. This machine data is generated by CPU running a webserver, IOT devices, logs from mobile apps, etc. It is not necessary to provide this data to the end users and does not have any business meaning. However, they are extremely important to understand, monitor and optimize the performance of the machines. Splunk can read this unstructured, semi-structured or rarely structured data. After reading the data, it allows to search, tag, create reports and dashboards on these data. With the advent of big data, Splunk is now able to ingest big data from various sources, which may or may not be machine data and run analytics on big data. So, from a simple tool for log analysis, Splunk has come a long way to become a general analytical tool for unstructured machine data and various forms of big data. Product Categories Splunk is available in three different product categories as follows − Splunk Enterprise − It is used by companies which have large IT infrastructure and IT driven business. It helps in gathering and analysing the data from websites, applications, devices and sensors, etc. Splunk Cloud − It is the cloud hosted platform with same features as the enterprise version. It can be availed from Splunk itself or through the AWS cloud platform. Splunk Light − It allows search, report and alert on all the log data in real time from one place. It has limited functionalities and features as compared to the other two versions. Splunk Features In this section, we shall discuss the important features of enterprise edition − Data Ingestion Splunk can ingest a variety of data formats like JSON, XML and unstructured machine data like web and application logs. The unstructured data can be modeled into a data structure as needed by the user. Data Indexing The ingested data is indexed by Splunk for faster searching and querying on different conditions. Data Searching Searching in Splunk involves using the indexed data for the purpose of creating metrics, predicting future trends and identifying patterns in the data. Using Alerts Splunk alerts can be used to trigger emails or RSS feeds when some specific criteria are found in the data being analyzed. Dashboards Splunk Dashboards can show the search results in the form of charts, reports and pivots, etc. Data Model The indexed data can be modelled into one or more data sets that is based on specialized domain knowledge. This leads to easier navigation by the end users who analyze the business cases without learning the technicalities of the search processing language used by Splunk. Print Page Previous Next Advertisements ”;

Splunk – Calculated Fields ”; Previous Next Many times, we will need to make some calculations on the fields that are already available in the Splunk events. We also want to store the result of these calculations as a new field to be referred later by various searches. This is made possible by using the concept of calculated fields in Splunk search. A simplest example is to show the first three characters of a week day instead of the complete day name. We need to apply certain Splunk function to achieve this manipulation of the field and store the new result under a new field name. Example The Web_application log file has two fields named bytes and date_wday. The value in the bytes field is the number of bytes. We want to display this value as GB. This will require the field to be divided by 1024 to get the GB value. We need to apply this calculation to the bytes field. Similarly, the date_wday displays complete name of the week day. But we need to display only the first three characters. The existing values in these two fields is shown in the image below − Using the eval Function To create calculated field, we use the eval function. This function stores the result of the calculation in a new field. We are going to apply the below two calculations − # divide the bytes with 1024 and store it as a field named byte_in_GB Eval byte_in_GB = (bytes/1024) # Extract the first 3 characters of the name of the day. Eval short_day = substr(date_wday,1,3) Adding New Fields We add new fields created above to the list of fields we display as part of the search result. To do this, we choose All fields options and tick check mark against the name of these new fields as shown in below image − Displaying the calculated Fields After choosing the fields above, we are able to see the calculated fields in the search result as shown below. The search query displays the calculated fields as shown below − Print Page Previous Next Advertisements ”;