Category: agile Data Science

Agile Data Science – SQL versus NoSQL ”; Previous Next The complete focus of this tutorial is to follow agile methodology with less number of steps and with implementation of more useful tools. To understand this, it is important to know the difference between SQL and NoSQL databases. Most of the users are aware of SQL database, and have a good knowledge on either MySQL, Oracle or other SQL databases. Over the last several years, NoSQL database is getting widely adopted to solve various business problems and requirements of project. The following table shows the difference between SQL and NoSQL databases − SQL NoSQL SQL databases are mainly called Relational Database Management system (RDBMS). NoSQL database is also called documentoriented database. It is non-relational and distributed. SQL based databases includes structure of table with rows and columns. Collection of tables and other schema structures called database. NoSQL database includes documents as major structure and the inclusion of documents is called collection. SQL databases include predefined schema. NoSQL databases have dynamic data and include unstructured data. SQL databases are vertical scalable. NoSQL databases are horizontal scalable. SQL databases are good fit for complex query environment. NoSQL do not have standard interfaces for complex query development. SQL databases are not feasible for hierarchal data storage. NoSQL databases fits better for hierarchical data storage. SQL databases are best fit for heavy transactions in the specified applications. NoSQL databases are still not considered comparable in high load for complex transactional applications. SQL databases provides excellent support for their vendors. NoSQL database still relies on community support. Only few experts are available for setup and deployed for large-scale NoSQL deployments. SQL databases focuses on ACID properties – Atomic, Consistency, Isolation And Durability. NoSQL database focuses on CAP properties – Consistency, Availability, and Partition tolerance. SQL databases can be classified as open source or closed source based on the vendors who have opted them. NoSQL databases are classified based on the storage type. NoSQL databases are open source by default. Why NoSQL for agile? The above-mentioned comparison shows that the NoSQL document database completely supports agile development. It is schema-less and does not completely focus on data modelling. Instead, NoSQL defers applications and services and thus developers get a better idea of how data can be modeled. NoSQL defines data model as the application model. MongoDB Installation Throughout this tutorial, we will focus more on the examples of MongoDB as it is considered the best “NoSQL schema”. Print Page Previous Next Advertisements ”;

Creating better scene with agile and data science ”; Previous Next Agile methodology helps organizations to adapt change, compete in the market and build high quality products. It is observed that organizations mature with agile methodology, with increasing change in requirements from clients. Compiling and synchronizing data with agile teams of organization is significant in rolling up data across as per the required portfolio. Build a better plan The standardized agile performance solely depends on the plan. The ordered data-schema empowers productivity, quality and responsiveness of the organization’s progress. The level of data consistency is maintained with historical and real time scenarios. Consider the following diagram to understand the data science experiment cycle − Data science involves the analysis of requirements followed by the creation of algorithms based on the same. Once the algorithms are designed along with the environmental setup, a user can create experiments and collect data for better analysis. This ideology computes the last sprint of agile, which is called “actions”. Actions involves all the mandatory tasks for the last sprint or level of agile methodology. The track of data science phases (with respect to life cycle) can be maintained with story cards as action items. Predictive Analysis and Big data The future of planning completely lies in the customization of data reports with the data collected from analysis. It will also include manipulation with big data analysis. With the help of big data, discrete pieces of information can be analyzed, effectively with slicing and dicing the metrics of the organization. Analysis is always considered as a better solution. Print Page Previous Next Advertisements ”;

Agile Data Science – Working with Reports ”; Previous Next In this chapter, we will learn about report creation, which is an important module of agile methodology. Agile sprints chart pages created by visualization into full-blown reports. With reports, charts become interactive, static pages become dynamic and network related data. The characteristics of reports stage of the data value pyramid is shown below − We will lay more stress on creating csv file, which can be used as report for data science analysis, and drawing conclusion. Although agile focusses on less documentation, generating reports to mention the progress of product development is always considered. import csv #———————————————————————- def csv_writer(data, path): “”” Write data to a CSV file path “”” with open(path, “wb”) as csv_file: writer = csv.writer(csv_file, delimiter=”,”) for line in data: writer.writerow(line) #———————————————————————- if __name__ == “__main__”: data = [“first_name,last_name,city”.split(“,”), “Tyrese,Hirthe,Strackeport”.split(“,”), “Jules,Dicki,Lake Nickolasville”.split(“,”), “Dedric,Medhurst,Stiedemannberg”.split(“,”) ] path = “output.csv” csv_writer(data, path) The above code will help you generate the “csv file” as shown below − Let us consider the following benefits of csv (comma- separated values) reports − It is human friendly and easy to edit manually. It is simple to implement and parse. CSV can be processed in all applications. It is smaller and faster to handle. CSV follows a standard format. It provides straightforward schema for data scientists. Print Page Previous Next Advertisements ”;

Collecting and Displaying Records ”; Previous Next In this chapter, we will focus on the JSON structure, which forms part of the “Agile methodology”. MongoDB is a widely used NoSQL data structure and operates easily for collecting and displaying records. Step 1 This step involves establishing connection with MongoDB for creating collection and specified data model. All you need to execute is “mongod” command for starting connection and mongo command to connect to the specified terminal. Step 2 Create a new database for creating records in JSON format. For now, we are creating a dummy database named “mydb”. >use mydb switched to db mydb >db mydb >show dbs local 0.78125GB test 0.23012GB >db.user.insert({“name”:”Agile Data Science”}) >show dbs local 0.78125GB mydb 0.23012GB test 0.23012GB Step 3 Creating collection is mandatory to get the list of records. This feature is beneficial for data science research and outputs. >use test switched to db test >db.createCollection(“mycollection”) { “ok” : 1 } >show collections mycollection system.indexes >db.createCollection(“mycol”, { capped : true, autoIndexId : true, size : 6142800, max : 10000 } ) { “ok” : 1 } >db.agiledatascience.insert({“name” : “demoname”}) >show collections mycol mycollection system.indexes demoname Print Page Previous Next Advertisements ”;

Agile Data Science – Data Visualization ”; Previous Next Data visualization plays a very important role in data science. We can consider data visualization as a module of data science. Data Science includes more than building predictive models. It includes explanation of models and using them to understand data and make decisions. Data visualization is an integral part of presenting data in the most convincing way. From the data science point of view, data visualization is a highlighting feature which shows the changes and trends. Consider the following guidelines for effective data visualization − Position data along common scale. Use of bars are more effective in comparison of circles and squares. Proper color should be used for scatter plots. Use pie chart to show proportions. Sunburst visualization is more effective for hierarchical plots. Agile needs a simple scripting language for data visualization and with data science in collaboration “Python” is the suggested language for data visualization. Example 1 The following example demonstrates data visualization of GDP calculated in specific years. “Matplotlib” is the best library for data visualization in Python. The installation of this library is shown below − Consider the following code to understand this − import matplotlib.pyplot as plt years = [1950, 1960, 1970, 1980, 1990, 2000, 2010] gdp = [300.2, 543.3, 1075.9, 2862.5, 5979.6, 10289.7, 14958.3] # create a line chart, years on x-axis, gdp on y-axis plt.plot(years, gdp, color=”green”, marker=”o”, linestyle=”solid”) # add a title plt.title(“Nominal GDP”) # add a label to the y-axis plt.ylabel(“Billions of $”) plt.show() Output The above code generates the following output − There are many ways to customize the charts with axis labels, line styles and point markers. Let’s focus on the next example which demonstrates the better data visualization. These results can be used for better output. Example 2 import datetime import random import matplotlib.pyplot as plt # make up some data x = [datetime.datetime.now() + datetime.timedelta(hours=i) for i in range(12)] y = [i+random.gauss(0,1) for i,_ in enumerate(x)] # plot plt.plot(x,y) # beautify the x-labels plt.gcf().autofmt_xdate() plt.show() Output The above code generates the following output − Print Page Previous Next Advertisements ”;

Agile Data Science – Role of Predictions ”; Previous Next In this chapter, we will earn about the role of predictions in agile data science. The interactive reports expose different aspects of data. Predictions form the fourth layer of agile sprint. When making predictions, we always refer to the past data and use them as inferences for future iterations. In this complete process, we transition data from batch processing of historical data to real-time data about the future. The role of predictions includes the following − Predictions help in forecasting. Some forecasts are based on statistical inference. Some of the predictions are based on opinions of pundits. Statistical inference are involved with predictions of all kinds. Sometimes forecasts are accurate, while sometimes forecasts are inaccurate. Predictive Analytics Predictive analytics includes a variety of statistical techniques from predictive modeling, machine learning and data mining which analyze current and historical facts to make predictions about future and unknown events. Predictive analytics requires training data. Trained data includes independent and dependent features. Dependent features are the values a user is trying to predict. Independent features are features describing the things we want to predict based on dependent features. The study of features is called feature engineering; this is crucial to making predictions. Data visualization and exploratory data analysis are parts of feature engineering; these form the core of Agile data science. Making Predictions There are two ways of making predictions in agile data science − Regression Classification Building a regression or a classification completely depends on business requirements and its analysis. Prediction of continuous variable leads to regression model and prediction of categorical variables leads to classification model. Regression Regression considers examples that comprise features and thereby, produces a numeric output. Classification Classification takes the input and produces a categorical classification. Note − The example dataset that defines input to statistical prediction and that enables the machine to learn is called “training data”. Print Page Previous Next Advertisements ”;

Agile Data Science – Useful Resources ”; Previous Next The following resources contain additional information on Agile Data Science. Please use them to get more in-depth knowledge on this. Useful Video Courses Agile Methodology Course for Beginners Best Seller 15 Lectures 1 hours Tutorialspoint More Detail Agile Project Management: Scrum Step by Step Course with Examples Most Popular 62 Lectures 1 hours Paul Ashun More Detail Agile for Security Teams 20 Lectures 1.5 hours Cristina Gheorghisan More Detail Scrum Testing: Learn Agile and Scrum Testing from A to Z NOW 26 Lectures 1.5 hours Dejan Majkic More Detail Agile Kanban: Kanban for Software Development Team 23 Lectures 2 hours Packt Publishing More Detail Agile & Scrum Fundamentals 22 Lectures 56 mins Asad Ur Rehman More Detail Print Page Previous Next Advertisements ”;

Agile Data Science – Data Processing in Agile ”; Previous Next In this chapter, we will focus on the difference between structured, semi-structured and unstructured data. Structured data Structured data concerns the data stored in SQL format in table with rows and columns. It includes a relational key, which is mapped into pre-designed fields. Structured data is used on a larger scale. Structured data represents only 5 to 10 percent of all informatics data. Semi-structured data Sem-structured data includes data which do not reside in relational database. They include some of organizational properties that make it easier to analyse. It includes the same process to store them in relational database. The examples of semi-structured database are CSV files, XML and JSON documents. NoSQL databases are considered semistructured. Unstructured data Unstructured data represents 80 percent of data. It often includes text and multimedia content. The best examples of unstructured data include audio files, presentations and web pages. The examples of machine generated unstructured data are satellite images, scientific data, photographs and video, radar and sonar data. The above pyramid structure specifically focusses on the amount of data and the ratio on which it is scattered. Quasi-structured data appears as type between unstructured and semi-structured data. In this tutorial, we will focus on semi-structured data, which is beneficial for agile methodology and data science research. Semi structured data does not have a formal data model but has an apparent, selfdescribing pattern and structure which is developed by its analysis. Print Page Previous Next Advertisements ”;

Discuss Agile Data Science ”; Previous Next Agile is a software development methodology that helps in building software through incremental sessions using short iterations of 1 to 4 weeks so that the development is aligned with the changing business needs. Agile Data science comprises of a combination of agile methodology and data science. In this tutorial, we have used appropriate examples to help you understand agile development and data science in a general and quick way. Print Page Previous Next Advertisements ”;

Agile Data Science – Quick Guide ”; Previous Next Agile Data Science – Introduction Agile data science is an approach of using data science with agile methodology for web application development. It focusses on the output of the data science process suitable for effecting change for an organization. Data science includes building applications that describe research process with analysis, interactive visualization and now applied machine learning as well. The major goal of agile data science is to − document and guide explanatory data analysis to discover and follow the critical path to a compelling product. Agile data science is organized with the following set of principles − Continuous Iteration This process involves continuous iteration with creation tables, charts, reports and predictions. Building predictive models will require many iterations of feature engineering with extraction and production of insight. Intermediate Output This is the track list of outputs generated. It is even said that failed experiments also have output. Tracking output of every iteration will help creating better output in the next iteration. Prototype Experiments Prototype experiments involve assigning tasks and generating output as per the experiments. In a given task, we must iterate to achieve insight and these iterations can be best explained as experiments. Integration of data The software development life cycle includes different phases with data essential for − customers developers, and the business The integration of data paves way for better prospects and outputs. Pyramid data value The above pyramid value described the layers needed for “Agile data science” development. It starts with a collection of records based on the requirements and plumbing individual records. The charts are created after cleaning and aggregation of data. The aggregated data can be used for data visualization. Reports are generated with proper structure, metadata and tags of data. The second layer of pyramid from the top includes prediction analysis. The prediction layer is where more value is created but helps in creating good predictions that focus on feature engineering. The topmost layer involves actions where the value of data is driven effectively. The best illustration of this implementation is “Artificial Intelligence”. Agile Data Science – Methodology Concepts In this chapter, we will focus on the concepts of software development life cycle called “agile”. The Agile software development methodology helps in building a software through increment sessions in short iterations of 1 to 4 weeks so the development is aligned with changing business requirements. There are 12 principles that describe the Agile methodology in detail − Satisfaction of customers The highest priority is given to customers focusing on the requirements through early and continuous delivery of valuable software. Welcoming new changes Changes are acceptable during software development. Agile processes is designed to work in order to match the customer’s competitive advantage. Delivery Delivery of a working software is given to clients within a span of one to four weeks. Collaboration Business analysts, quality analysts and developers must work together during the entire life cycle of project. Motivation Projects should be designed with a clan of motivated individuals. It provides an environment to support individual team members. Personal conversation Face-to-face conversation is the most efficient and effective method of sending information to and within a development team. Measuring progress Measuring progress is the key that helps in defining the progress of project and software development. Maintaining constant pace Agile process focusses on sustainable development. The business, the developers and the users should be able to maintain a constant pace with the project. Monitoring It is mandatory to maintain regular attention to technical excellence and good design to enhance the agile functionality. Simplicity Agile process keeps everything simple and uses simple terms to measure the work that is not completed. Self-organized terms An agile team should be self-organized and should be independent with the best architecture; requirements and designs emerge from self-organized teams. Review the work It is important to review the work at regular intervals so that the team can reflect on how the work is progressing. Reviewing the module on a timely basis will improve performance. Daily Stand-up Daily stand-up refers to the daily status meeting among the team members. It provides updates related to the software development. It also refers to addressing obstacles of project development. Daily stand-up is a mandatory practice, no matter how an agile team is established regardless of its office location. The list of features of a daily stand-up are as follows − The duration of daily stand-up meet should be roughly 15 minutes. It should not extend for a longer duration. Stand-up should include discussions on status update. Participants of this meeting usually stand with the intention to end up meeting quickly. User Story A story is usually a requirement, which is formulated in few sentences in simple language and it should be completed within an iteration. A user story should include the following characteristics − All the related code should have related check-ins. The unit test cases for the specified iteration. All the acceptance test cases should be defined. Acceptance from product owner while defining the story. What is Scrum? Scrum can be considered as a subset of agile methodology. It is a lightweight process and includes the following features − It is a process framework, which includes a set of practices that need to be followed in consistent order. The best illustration of Scrum is following iterations or sprints. It is a “lightweight” process meaning that the process is kept as small as possible, to maximize the productive output in given duration specified. Scrum process is known for its distinguishing process in comparison with other methodologies of traditional agile approach. It is divided into the following three categories − Roles Artifacts Time Boxes Roles define the team members and their roles included throughout the process. The Scrum Team consists of the following three roles − Scrum Master Product Owner Team The Scrum artifacts provide key information that each member should be aware of. The information includes details of product, activities