Data migration is one of the most difficult and complex portions of any digital transformation. To better understand this, I recently interviewed Daryl Crockett, the CEO of ValidDatum, which is our technology-agnostic data migration partner at Third Stage. She is a leader in the industry, so we work with Daryl and her team to help address the data migration components of our clients’ digital transformation initiatives.

Below are some excerpts from my interview with Daryl:

At Third Stage Consulting, we sometimes encounter clients treating data migration as a lower priority project. Thoughts on this?

Unfortunately, most companies (and even their hired ERP systems integrators) minimize the importance of data migration work that must precede a successful ERP implementation/digital transformation. Many companies don’t understand the multiple complexities or steps involved. Data migration is an iterative journey. Some believe that data can be moved with (auto-magical) “lift and shift” tools, or they mistakenly think that someone else on the team will do the planning and heavy lifting.

Anticipate a tedious ongoing technical process requiring a lot of testing (think multiple rounds). When owned by a project team that doesn’t have the tools, prior large-scale migration experience, or enough time – the results can be subpar and significantly impact the bigger initiative. When real data is run through the new system configurations it will highlight system inadequacies, leading to design and configuration changes.

When data readiness is not a well-planned pillar of the project, data is often introduced too late in the project cycle preventing early detection of system weaknesses. Late fixes to the new system can and will cause project delays and costly design/build/test rework cycles.

“Dirty data” is also a potential impact. When data readiness is not properly planned and executed in advance, a garbage in, garbage out result occurs. The promise of large-scale IT initiatives are business intelligence, data analytics, internet of things, etc. However, these tech tools will be very unimpressive if your business data is compromised, inconsistent or inaccurate.

You often speak about “data transformation.” How does this differ from data cleansing?

Data cleansing is typically performed in advance of the actual data migration because it requires more extensive pre-work or may require external validation. It’s not unusual for a data cleansing “phase” to be skipped or minimized, when it should begin (whenever possible) in advance of software selection.

Data transformation is a systematic, programmatic process where data is transformed in bulk according to predetermined business and technical rules. Transformation is designed to be performed iteratively and repeatedly, allowing the project team to introduce large amounts of transformed live data into the newly built system for effective/realistic testing. This means not only understanding the technical aspects but getting the right approvals (for the mapping) of who gets access to what data.

Are internal company SME’s the best resources to do data cleansing?

Data cleansing is often performed by internal SME’s, especially if the data is particularly messy and if the SME’s are familiar with the data. They should have access to and knowledge of information necessary to correct the data. However, it’s a misconception that the business must always take the lead in data cleaning. For example, some independent migration consultants have developed sophisticated semi-automated cleansing and validation tools that have been built and tested for this very purpose. A fresh perspective can also have the added benefit of identifying issues or opportunities long before the ERP or digital transformation project is even completed.

Why do so many companies wait until user acceptance testing to begin validating their data?

Unfortunately, most haven’t planned out a “data readiness track” well enough to have any significant data ready for testing earlier. Some of the biggest problems with data testing so late in the game is it’s likely not all the data will ever be tested before go-live. Downstream systems, reports and analytics will also not be run and tested with master and transactional data (purchase orders, invoices, goods movements, etc.). (Note: data readiness is one of the areas addressed as part of Third Stage’s ERP implementation readiness methodology and approach.)

The system integrator is typically only tasked with the main ERP system, therefore isn’t incentivized to ensure downstream systems/reporting are able to properly process the outputs. Many careers have been derailed when a CEO, COO or CFO is unpleasantly surprised by the poor condition of data appearing in their new executive dashboards and reports. Proper data readiness planning can significantly reduce post-project surprises.

Data security and data privacy are huge issues. Should they be considered part of data migration?

The exercises of data mapping and data cataloging are common to data migration and data privacy and security. To protect and secure data, you must know what data you have and how it flows and is used throughout your organization. If you have already completed your data privacy and security initiatives, then the data migration team can leverage that work. If you’re planning a data privacy or security initiative (highly recommended for ANY size business), then you should look for opportunities to combine some of the work with your data migration efforts. This can save significant time and money if your migration team can work with (and understand) the relational advantages of doing so.