Data Preprocessing is required because:
Real world data are generally:
Incomplete: Missing attribute values, missing certain attributes of importance, or having only aggregate data
Noisy: Containing errors or outliers
Inconsistent: Containing discrepancies in codes or names
Steps in Data preprocessing:
1. Data cleaning:
- Data cleaning, also called data cleansing or scrubbing.
- Fill in missing values, smooth noisy data, identify or remove the outliers, and resolve inconsistencies.
- Data cleaning is required because source systems contain “dirty data” that must be cleaned.
- Steps in Data cleaning:
- Parsing locates and identifies individual data elements in the source files and then isolates these data elements in the target files.
- Example includes parsing the first, middle and the last name.
- Correct parsed individual data components using sophisticated data algorithms and secondary data sources.
- Example includes replacing a vanity address and adding a zip code.
- Standardizing applies conversion routines to transform data into its preferred and consistent format using both standard and custom business rules.
- Examples include adding a pre name, replacing a nickname.
- Searching and matching records within and across the parsed, corrected and standardized data based on predefined business rules to eliminate duplications.
- Examples include identifying similar names and addresses.
- Analyzing and identifying relationships between matched records and consolidating/merging them into one representation.
1.6 Data cleansing must deal with many types of possible errors:
- These include missing data and incorrect data at one source.
1.7 Data Staging:
- Accumulates data from asynchronous sources.
- At a predefined cutoff time, data in the staging file is transformed and loaded to the warehouse.
- There is usually no end user access to the staging file.
- An operational data store may be used for data staging.
2. Data integration and Transformation:
Data integration: Combines data from multiple sources into a coherent data store e.g. data warehouse.
Sources may include multiple databases, data cubes or data files.
Issues in data integration:
- Integrate metadata from different sources.
- Entity identification problem: identify real world entities from multiple data sources, e.g. A cust-id=B.cust#.
Detecting and resolving data value conflicts:
- For the same real world entity, attribute values from different sources are different.
- Possible reasons: different representations, different scales.
- Redundant data occur often when integration of multiple databases:
- The same attribute may have different names in different databases.
- Data Transformation: Transformation process deals with rectifying any inconsistency (if any).
- One of the most common transformation issues is ‘Attribute Naming Inconsistency’. It is common for the given data element to be referred to by different data names in different databases.
- Eg Employee Name may be EMP_NAME in one database, ENAME in the other.
- Thus one set of Data Names are picked and used consistently in the data warehouse.
- Once all the data elements have right names, they must be converted to common formats.
3. Data Reduction:
- Obtains reduced representation in volume but produces the same or similar analytical results.
Need for data reduction:
- Reducing the number of attributes
- Reducing the number of attribute values
- Reducing the number of tuples
4. Discretization and Concept Hierarchy Generation(or summarization):
- Discretization: Reduce the number of values for a given continuous attribute by divide the range of a continuous attribute into intervals.
- Interval labels can then be used to replace actual data values.
- Concept Hierarchies: Reduce the data by collecting and replacing low level concepts(such as numeric values for the attribute age)by higher level concepts(such as young, middle-aged or senior).