Data Integrity Tests for R

Data integrity testing is a procedure to make sure the data is accurate. It ensures that the information in a database is what we planned to store and is not inadvertently altered during access. Any data consistency and accuracy are referred to as its integrity. It relates to the accuracy and dependability of the data.

Types of Data Integrity Tests

In R Programming Language There are two types of data integrity logical and physical. First, we must be understood in order to maintain data integrity. Data integrity is enforced by each of these sets of procedures and techniques.

• Physical integrity: Physical integrity refers to the protection of data accuracy and completeness during its maintenance, retrieval, and storage procedures. Natural catastrophes, power outages, and disk drive failures all result in the loss of physical data integrity. Aside from internal auditors, data processing managers, and system and applications programmers may not have access to correct data because of human error, storage degradation, and other issues.
• Logical integrity: Logical integrity in a relational database protects data when it’s used in different ways. Logical integrity provides protection against human error and hackers, unlike physical integrity.
• There are four kinds of logical integrity:
• Domain integrity: Domain integrity refers to the set of protocols that ensure that all data within a domain is accurate. In this context, a domain refers to the range of acceptable values that can exist in a column. The type, amount, and format of data that can be entered may be restricted by its restrictions and other constraints.
• Referential integrity: Referential integrity verifies the relationship between the primary key and foreign key in two tables to guarantee that the relationship is legitimate.
• Entity integrity: Entity integrity depends on the creation of primary keys, which are the unique values that identify individual pieces of data. These keys ensure that no column in a database is null and that data isn’t listed more than once. This characteristic is present in relational systems, which store data in tables that may be connected and used in a variety of ways.
• User-defined integrity: User-defined integrity is the set of rules and constraints that the user has created to suit their own needs. Data protection requires more than just entity, referential, and domain integrity. Many times, data integrity measures must take specific business standards into account and put them into practice.

Components of data integrity

1. Cyber Security: The main concern in cyber security is data access. The data should only be accessible to authorized individuals and programs in order to view or edit it. Otherwise, either malicious activity or human error will contaminate the data.
   Cybersecurity uses credentials, such as usernames and passwords, to restrict access to data. In certain cases, the data is even encrypted, which means that a decryption key is required to access it—even if it has been stolen or disclosed.
2. Physical Safety: Physical safety guarantees that the data storage equipment are protected from weather, fire, theft, and other similar occurrences. It also entails making sure the data storage devices are of a high caliber. Device malfunctions may result in an unanticipated data loss.
3. Database Integrity: Databases typically come with an established structure that makes it easier to identify the relationships between different kinds of data. They are implemented by using restrictions that were put into the tables at the time of construction and entity relationships, also known as foreign keys.

There are certain unique limitations that SQL offers that can be used for data checking during data entering. These are only a handful of the restrictions:

• Unique: First we have to confirms that every value entered in a column is unique. There is a place to store email addresses because each user must have their own.
• Not Null: Not null means a certain field may not be left empty.
• Foreign Key: A foreign key is used to relate data from one table to another. This ensures that data in one table won’t be accidentally removed without also affecting related tables.
• Check: Checks any data added into a database using a bespoke procedure. Checks for ranges or codes are part of the check constraint.

Benefits of Data Integrity Tests

• Improved Data Quality: Data integrity checks contribute to improved data quality by assisting in the detection and correction of mistakes and inconsistencies in your data. Efficient and precise data are necessary for significant analysis and trustworthy outcomes.
• Reduced Risk of Errors: Enums lower the likelihood of errors and problems in your code by assisting in the prevention of using inconsistent or inaccurate data.
• Time and Cost Savings: You can save time and resources that would otherwise be used troubleshooting issues that develop during analysis by recognizing and resolving data quality concerns early.
• Enhanced Reproducibility: Clear, well-documented data from data integrity checks improves transparency and reproducibility of your work and makes it easier for others to duplicate your research and conclusions.
• Better Decision-Making: High-quality data makes it feasible to make smarter decisions. By helping to verify the authenticity of the data you use for analysis, data integrity tests reduce the likelihood that judgments will be made based on false information.

Important of Data Integrity Testing

1. Improved insights and analytics.
2. Reliable data for use in artificial intelligence (AI) and machine learning (ML) projects.
3. It aids in preventing unintentional data loss.
4. It aids in maintaining the database’s ACID (Accuracy, Consistency, Integrity, Durability) feature.
5. Faster and more confident decision-making.
6. Enhanced business agility.
7. It aids in ensuring back-end and front-end synchronization. Both the activities on the front-end and the changes made in the database must have an impact on the front-end.

Data Integrity Tests in R

Finding Missing Data in R

We can for the the missing values using the built-in functions that R offers us. The details of these built-in features are provided below.

To check for NA values, we can utilize R’s built-in is.na() function. Only logical values (True or False) are contained in the vector that this function returns. The appropriate vector value should be True if the original dataset’s NA values are present, and False otherwise.

R

myVector <- c(NA, "TP", 4, 6.7, 'c', NA, 17)
sum(is.na(myVector))

Output:

 [1] 2

Duplicate Data Detection

Duplicate rows need to be found and eliminated in order to keep a dataset accurate and free of duplication. In this tutorial, we’ll look at how to spot and get rid of duplicate data in R. If duplicate data is found in our data, we will first determine if it has to be removed.

R

# Create a sample vector with duplicate elements
vector_data <- c(2,4,6,8,5,2)

# Identify duplicate elements
duplicated(vector_data)

# count of duplicated data
sum(duplicated(vector_data))

Output:

[1] FALSE FALSE FALSE FALSE FALSE  TRUE
[1]  1

Outlier Detection

An outlier is a value or observation that differs significantly from other observations, or from other data points in terms of distance from other observations.

R

# Sample data with outliers
data <- c(23, 22, 21, 28, 90, 39, 17, 200, 36, 38)

# Detect outliers using the IQR method
q1 <- quantile(data, 0.25)
q3 <- quantile(data, 0.75)
iqr <- q3 - q1
lower_bound <- q1 - 1.5 * iqr
upper_bound <- q3 + 1.5 * iqr

outliers <- data[data < lower_bound | data > upper_bound]

outliers

Output:

[1]  90 200

Data Type Test

The knowledge and items are categorized into several data categories. Numeric, String, List, Tuple, Set, and Dictionary are the six common data types in R.

R

# Create a sample data frame with a data type mismatch
data <- data.frame(
  ID = 1:5,
  Value = c(2, 3, "A", 4, 5)
)

# Check for data type mismatches in the 'Value' column
data$type_mismatch <- !sapply(data$Value, is.numeric)
data[data$type_mismatch, ]

Output:

  ID Value type_mismatch
1  1     2          TRUE
2  2     3          TRUE
3  3     A          TRUE
4  4     4          TRUE
5  5     5          TRUE

Consistency Test

The purpose of the consistency test is to evaluate one essential but insufficient component of resilience. That is, the capacity to arrive at the same conclusion independent of starting point.

R

# Create a sample data frame with values outside a specified range
data <- data.frame(
  ID = 1:5,
  Value = c(2, 3, 12, 4, 5)
)

# Check for values outside the specified range (e.g., 1 to 10)
inconsistent_values <- data$Value < 1 | data$Value > 10
data[inconsistent_values, ]

Output:

ID Value.
3  3    12

Conclusion

To conclusion, It is important for data to be accurate and full. Because it can impact the outcomes of any study done on the data, maintaining data integrity is vital. Many things, such as human mistake, system flaws, and hostile assaults, can threaten data integrity.