Category: Data Quality

Data Reconciliation

Also known as: Dataset Reconciliation

Simply put

Data reconciliation is the process of comparing two or more datasets to find and resolve discrepancies between them. It is typically done to confirm that information is accurate, consistent, and complete across different systems. It is generally a data quality and integrity activity rather than a specific legal compliance requirement.

Formal definition

Data reconciliation is the systematic process of comparing, verifying, and harmonizing data across multiple sources or systems to surface discrepancies and support data accuracy, consistency, and completeness. As a data governance and data quality practice, it commonly addresses record matching, field-level comparison, and remediation of mismatches to maintain data integrity across pipelines, migrations, or integrated systems. This entry addresses the general governance and quality concept only; it does not define or scope security controls (such as confidentiality or availability protections), retention rules, cross-border transfer mechanics, or any jurisdiction-specific regulatory obligation. Reconciliation may support demonstrable accuracy in ways relevant to accountability under governance frameworks, but performing it does not by itself satisfy any statutory requirement, and its treatment as a data quality control should not be conflated with information security functions.

Why it matters

Data reconciliation underpins trust in the information an organization relies on to operate. When the same record exists across multiple systems, a source database, a data warehouse, a downstream application, unnoticed discrepancies can propagate into reporting, decision-making, and customer-facing outputs. Comparing datasets to surface and resolve those mismatches helps confirm that data is accurate, consistent, and complete, which is a foundational data quality and integrity concern rather than a standalone legal obligation.

Within a data governance program, reconciliation contributes to the demonstrable accuracy that accountability frameworks generally expect. Where a governance framework or a data protection regime treats accuracy as an obligation, reconciliation can be one of the practices that produces evidence of that accuracy over time. It is important, however, not to overstate its reach: performing reconciliation does not by itself satisfy any statutory requirement, and it should not be treated as a substitute for information security controls, retention rules, or jurisdiction-specific compliance measures.

Reconciliation is especially valuable during data migrations, system integrations, and pipeline changes, where the risk of silent data loss or corruption is elevated. Detecting a field-level mismatch or a missing record early is far less costly than remediating downstream errors after they have been consumed by end users. Treating reconciliation as an ongoing quality control, rather than a one-time migration check, helps maintain data integrity as systems evolve.

Who it's relevant to

Data Governance and Data Quality Leads

Those responsible for data ownership, stewardship, and quality use reconciliation to maintain accuracy, consistency, and completeness across systems. It supports demonstrable accuracy that governance accountability generally expects, though it does not by itself satisfy any statutory requirement and should be paired with clear ownership for remediating discrepancies.

Data and Integration Engineers

Engineers building pipelines, integrations, and migrations rely on reconciliation to detect silent data loss or corruption through record matching and field-level comparison. It is most effective as an ongoing control across evolving systems rather than a single migration checkpoint.

Compliance and Data Protection Officers

Where a governance framework or data protection regime treats data accuracy as an obligation, reconciliation can contribute evidence supporting that accuracy. This entry addresses only the general governance and quality concept; it does not cover security controls, retention, cross-border transfers, or jurisdiction-specific requirements, which must be assessed separately.

Information Security Teams

Reconciliation is a data quality control and should not be conflated with security functions. It does not provide confidentiality or availability protections. Security teams benefit from understanding the distinction so integrity-oriented quality work and security controls are each addressed on their own terms.

Inside Data Reconciliation

Source-to-Target Comparison

The core activity of comparing data between an originating system and a destination system to identify discrepancies in values, counts, or completeness. This is typically performed after data migration, integration, or transformation processes.

Record Count Validation

Verification that the number of records present in the target matches the expected number derived from the source, accounting for documented filtering or deduplication rules applied during processing.

Field-Level Matching

Comparison of individual attribute values across systems to confirm that transformations preserved intended meaning and that no corruption or truncation occurred during transfer.

Discrepancy Logging and Resolution

Documentation of identified mismatches together with the investigation and remediation steps taken. Under governance frameworks, accountability generally requires demonstrable evidence of resolution, not merely a stated intent to correct errors.

Governance Linkage

Reconciliation supports data quality and lineage objectives within a data governance program by evidencing that data remained accurate and complete as it moved between systems. This is distinct from information security controls, which address confidentiality, integrity, and availability rather than business-meaning correctness.

Control Totals and Checksums

Aggregate figures such as sums of numeric fields, or computed hashes, used to detect differences without comparing every record individually. These support integrity checks but do not by themselves establish that individual records are semantically correct.

Common questions

Answers to the questions practitioners most commonly ask about Data Reconciliation.

Is data reconciliation the same thing as data validation or a data quality check?

No. Reconciliation specifically compares two or more data sources, copies, or system-of-record versions to confirm they agree, whereas validation checks a single dataset against defined rules or constraints. Reconciliation is one technique within the broader data quality discipline, not a synonym for it. Treating them as interchangeable can leave gaps: a record can pass validation against local rules yet still fail reconciliation against an authoritative source, and vice versa.

Does successful data reconciliation mean the reconciled data is accurate or fit for compliance purposes?

Not necessarily. Reconciliation generally establishes that sources are consistent with one another, not that any of them reflects the true, current, or lawful state of the underlying facts. Two systems can agree and both be wrong. Consistency is evidence of alignment, not of accuracy, completeness, or compliance, and reconciliation results should not be presented as a standalone assurance of data correctness.

Where does data reconciliation typically fit within a data governance program?

It generally sits within the data quality and stewardship functions of data governance, supporting activities such as identifying the system of record, monitoring lineage, and demonstrating that controlled data remains consistent across environments. Assigning clear ownership for who performs reconciliation, who investigates discrepancies, and who signs off on resolutions is part of governance accountability, which typically requires demonstrable evidence rather than stated intent.

How should reconciliation discrepancies be handled once they are detected?

In most implementations, detection is only the first step; a defined exception-handling process should route discrepancies to accountable owners, capture the investigation, and record the resolution. Establishing which source is authoritative for a given field in advance helps determine the correct direction of any correction. Retaining an auditable trail of discrepancies and their disposition supports accountability, though the specifics of retention are out of scope for this entry.

How does reconciliation interact with personal data protection obligations?

Reconciliation processes typically operate on data that may include personal data, so the activity itself is generally a form of processing subject to applicable data protection principles such as purpose limitation, data minimization, and security controls. This entry does not address which lawful basis applies, cross-border transfer mechanics, or retention rules, and applying a technical control during reconciliation does not by itself remove data from the scope of personal data.

What should be considered when deciding how frequently to run reconciliation?

Frequency generally depends on the criticality of the data, how quickly sources diverge, the tolerance for inconsistency in downstream use, and operational cost. Some environments favor continuous or scheduled automated reconciliation, while others rely on periodic point-in-time checks. There is no single frequency that guarantees adequacy; the appropriate cadence is context-dependent and should be documented and justified rather than assumed.

Common misconceptions

Data reconciliation is a security control that protects data.

Reconciliation is primarily a data quality and governance activity focused on confirming accuracy and completeness of data across systems. It overlaps with the integrity dimension of information security but does not replace confidentiality or availability controls, and it is not itself a security safeguard.

Matching record counts between source and target proves reconciliation is complete.

Equal record counts indicate volume alignment only. Field-level values can still be corrupted, truncated, or incorrectly transformed while counts match, so count validation is generally one component rather than sufficient evidence on its own.

Reconciling personal data changes its regulatory status.

Reconciliation confirms accuracy across systems and does not alter whether data is personal data. Where reconciliation involves personal data, it typically remains subject to applicable obligations under regimes such as the EU GDPR or UK GDPR; treatment differs across jurisdictions and is out of scope for this definition.

Best practices

Define reconciliation rules and expected outcomes in advance, including documented handling of legitimate differences such as filtering, deduplication, or transformation logic, so that discrepancies can be distinguished from intended changes.

Combine record count validation with field-level matching and control totals rather than relying on any single technique, since each detects different classes of error.

Log every identified discrepancy along with its investigation and resolution to produce demonstrable evidence of data quality, as accountability under governance frameworks generally requires evidence rather than stated intent.

Assign clear ownership and stewardship for reconciliation outcomes so that responsibility for investigating and resolving mismatches is unambiguous.

Where reconciliation involves personal data, coordinate with the relevant data protection function to confirm that applicable obligations are met, and treat cross-border transfer mechanics, retention rules, and lawful basis as separate matters outside the reconciliation exercise itself.

Schedule reconciliation at appropriate points in the data lifecycle, typically after migration, integration, or transformation events, and document the scope so limitations of each run are explicit.