Analysis: Redundancy And Correlation Analysis

3.1 Redundancy and Correlation Analysis

Another important issue in data integration is redundancy. If an attribute is derived from another attribute it may be redundant. To detect redundancies you can use the correlation analysis, this method will detect inconsistencies in the resulting dataset.

3.2 Entity Identification Problem

There a number of issues during data integration. Two issues can be tricky which include schema integration and also object matching. Entity identification problem derives from how can real world entities from multiple data sources be matched up.

A special attention must be paid to the structure of the data when matching attributes from one database to another during integration. This is the case as it has to ensure

In this section, I will be presenting an overview of data reduction strategies, and also then I will have a closer look at individual techniques

4.1 Overview of Data Reduction Strategies

There are a number of data reduction strategies that includes: numerosity reduction, dimensionality reduction and finally data compression.

4.1.1 Numerosity reduction

I will firstly talk about numerosity reduction. Numerosity reduction techniques make the smaller forms of data representation then the original data volume, they can be parametric or nonparametric. The parametric methods is used to make an estimation of the data typically the data parameters that need to be stored, not the actual data. The methods for storing the reduced representations are known as the nonparametric methods, these methods include clustering, sampling, data cube aggregation and also histograms. With aggregation it is the combing of two or more objects into a single object. One way to aggregate transactions is to replace all the transactions of a single store with a single storewide transaction

Pre-processing data is a very important step in data mining, its techniques which I have discussed prove very valuable for the data quality in the database. All of the techniques in pre-processing have a very important role to play. Data mining with pre-processing with result in very inconsistent data sets, meaning for people analyzing data, they would get very inaccurate results. But with pre-processing this minimizes and almost defeats all inconsistent noisy data using the techniques as I have discussed such as Data cleaning,. Data cleaning provides a solution for missing values and noisy data in the database. As well as data cleaning data integration is a vital process as it gets the coloration of the data sets to detect data redundancies. Data reduction compresses all the data and reduces the volume of the data within the data, and it clears all of the unwanted data. Data transformation makes it easier for visualizing, and understand the data. All of these techniques within data pre-processing is very effective in the fact that they minimizes and avoids inconsistent, noisy and human error data. Pre-processing is a very important step in the process of data mining, when dealing with a lot of data make sure to use the pre-processing technique to make your database/ data ware house as accurate and as real as possible as predictive analytics would prove no value and would