KDD Process in Data Mining

Data Mining – Knowledge Discovery in Databases(KDD). 

KDD (Knowledge Discovery in Databases) is a process that involves the extraction of useful, previously unknown, and potentially valuable information from large datasets. The KDD process in data mining typically involves the following steps:

1. Selection: Select a relevant subset of the data for analysis.
2. Pre-processing: Clean and transform the data to make it ready for analysis. This may include tasks such as data normalization, missing value handling, and data integration.
3. Transformation: Transform the data into a format suitable for data mining, such as a matrix or a graph.
4. Data Mining: Apply data mining techniques and algorithms to the data to extract useful information and insights. This may include tasks such as clustering, classification, association rule mining, and anomaly detection.
5. Interpretation: Interpret the results and extract knowledge from the data. This may include tasks such as visualizing the results, evaluating the quality of the discovered patterns and identifying relationships and associations among the data.
6. Evaluation: Evaluate the results to ensure that the extracted knowledge is useful, accurate, and meaningful.
7. Deployment: Use the discovered knowledge to solve the business problem and make decisions.

The KDD process is an iterative process and it requires multiple iterations of the above steps to extract accurate knowledge from the data.

Why do we need Data Mining?

Volume of information is increasing everyday than we can handle from business transactions, scientific data, sensor data, Pictures, videos, etc. So, we need a system that will be capable of extracting essence of information available and that can automatically generate report, 
views or summary of data for better decision-making. 

Why Data Mining is used in Business?
Data mining is used in business to make better managerial decisions by: 

• Automatic summarization of data
• Extracting essence of information stored.
• Discovering patterns in raw data.

Data Mining also known as Knowledge Discovery in Databases, refers to the nontrivial extraction of implicit, previously unknown and potentially useful information from data stored in databases. 

Steps Involved in KDD Process:

 KDD process

1. Data Cleaning: Data cleaning is defined as removal of noisy and irrelevant data from collection. 
   • Cleaning in case of Missing values.
   • Cleaning noisy data, where noise is a random or variance error.
   • Cleaning with Data discrepancy detection and Data transformation tools.
2. Data Integration: Data integration is defined as heterogeneous data from multiple sources combined in a common source(DataWarehouse). 
   • Data integration using Data Migration tools.
   • Data integration using Data Synchronization tools.
   • Data integration using ETL(Extract-Load-Transformation) process.
3. Data Selection: Data selection is defined as the process where data relevant to the analysis is decided and retrieved from the data collection. 
   • Data selection using Neural network.
   • Data selection using Decision Trees.
   • Data selection using Naive bayes.
   • Data selection using Clustering, Regression, etc.
4. Data Transformation: Data Transformation is defined as the process of transforming data into appropriate form required by mining procedure. 

   Data Transformation is a two step process: 

   • Data Mapping: Assigning elements from source base to destination to capture transformations.
   • Code generation: Creation of the actual transformation program.
5. Data Mining: Data mining is defined as clever techniques that are applied to extract patterns potentially useful. 
   • Transforms task relevant data into patterns.
   • Decides purpose of model using classification or characterization.
6. Pattern Evaluation: Pattern Evaluation is defined as identifying strictly increasing patterns representing knowledge based on given measures. 
   • Find interestingness score of each pattern.
   • Uses summarization and Visualization to make data understandable by user.
7. Knowledge representation: Knowledge representation is defined as technique which utilizes visualization tools to represent data mining results. 
   • Generate reports.
   • Generate tables.
   • Generate discriminant rules, classification rules, characterization rules, etc.

Note: 

• KDD is an iterative process where evaluation measures can be enhanced, mining can be refined, new data can be integrated and transformed in order to get different and more appropriate results.
• Preprocessing of databases consists of Data cleaning and Data Integration.

ADVANTAGES OR DISADVANTAGES:
Advantages of KDD:

1. Improves decision-making: KDD provides valuable insights and knowledge that can help organizations make better decisions.
2. Increased efficiency: KDD automates repetitive and time-consuming tasks and makes the data ready for analysis, which saves time and money.
3. Better customer service: KDD helps organizations gain a better understanding of their customers’ needs and preferences, which can help them provide better customer service.
4. Fraud detection: KDD can be used to detect fraudulent activities by identifying patterns and anomalies in the data that may indicate fraud.
5. Predictive modeling: KDD can be used to build predictive models that can forecast future trends and patterns.

Disadvantages of KDD:

1. Privacy concerns: KDD can raise privacy concerns as it involves collecting and analyzing large amounts of data, which can include sensitive information about individuals.
2. Complexity: KDD can be a complex process that requires specialized skills and knowledge to implement and interpret the results.
3. Unintended consequences: KDD can lead to unintended consequences, such as bias or discrimination, if the data or models are not properly understood or used.
4. Data Quality: KDD process heavily depends on the quality of data, if data is not accurate or consistent, the results can be misleading
5. High cost: KDD can be an expensive process, requiring significant investments in hardware, software, and personnel.
6. Overfitting: KDD process can lead to overfitting, which is a common problem in machine learning where a model learns the detail and noise in the training data to the extent that it negatively impacts the performance of the model on new unseen data. 

Difference Between KDD and Data Mining 

Parameter	KDD	Data Mining
Definition	KDD refers to a  process of identifying valid, novel, potentially useful, and ultimately understandable patterns and relationships in data.	Data Mining refers to a  process of extracting useful and valuable information or patterns from large data sets.
Objective	To find useful knowledge from data.	To extract useful information from data.
Techniques Used	Data cleaning, data integration, data selection, data transformation, data mining, pattern evaluation, and knowledge representation and visualization.	Association rules, classification, clustering, regression, decision trees, neural networks, and dimensionality reduction.
Output	Structured information, such as rules and models, that can be used to make decisions or predictions.	Patterns, associations, or insights that can be used to improve decision-making or understanding.
Focus	Focus is on the discovery of useful knowledge, rather than simply finding patterns in data.	Focus is on the discovery of patterns or relationships in data.
Role of domain expertise	Domain expertise is important in KDD, as it helps in defining the goals of the process, choosing appropriate data, and interpreting the results.	Domain expertise is less critical in data mining, as the algorithms are designed to identify patterns without relying on prior knowledge.

References: 
Data Mining: Concepts and Techniques

 There are many books available on the topic of data mining and KDD. Here are a few well-known books on data mining and KDD that you may find useful:

1. “Data Mining: Concepts and Techniques” by Jiawei Han, Micheline Kamber, and Jian Pei – This is a comprehensive book that covers the fundamental concepts and techniques of data mining, including data pre-processing, data warehousing, and data visualization.
2. “Data Mining: Practical Machine Learning Tools and Techniques” by Ian H. Witten, Eibe Frank, and Mark A. Hall – This book provides a practical guide to data mining, including real-world examples and case studies.
3. “Data Mining for Business Intelligence: Concepts, Techniques, and Applications in Microsoft Office Excel” by Galit Shmueli, Nitin R. Patel, and Peter C. Bruce – This book provides a hands-on guide to data mining using Microsoft Excel and the add-in XLMiner.
4. “Data Mining: The Textbook” by Charu Aggarwal – This book provides a comprehensive introduction to the field of data mining, including the latest techniques and algorithms, as well as real-world applications.
5. “Data Mining and Knowledge Discovery Handbook” by Oded Maimon and Lior Rokach – This book is a comprehensive handbook that covers the fundamental concepts and techniques of data mining and KDD, including data pre-processing, data warehousing, and data visualization.

These books provide a good introduction to the field of data mining and KDD and can be a good starting point for learning more about these topics.