10 Frequently Asked Questions (FAQs) about Data Migration for Permitting,  Inspections & Licensing Software

This post seeks to answer some common questions about data migration that we get from our customers when they are looking to move to a new Permitting & Inspections, Licensing or Code Enforcement Software.

1. What are the different options for Data Migration?

There are 3 options when it comes to Data Migration.  Please refer to this article our team has written that goes in depth into each option.  At a high-level here are the 3 options:

1. 1. 1. Custom Data Migration: A team from the new system's vendor does the migration for you.
      2. Data Migration via a Standardized Loader: This is a lower cost approach but there are shared responsibilities between your organization's IT department and the new system's vendor.
      3. Choosing not to migrate data: This is the lowest cost option that is meaningful in some situations.  This option may involve relying on your organization's IT department.

2. Is not migrating any data from my current system to the new system an option?

In some scenarios, it is important to have a continuity for your customers between the old system and the new.  For example, if your organization is responsible for Business Licensing, Alcohol Licensing, Short Term Rental Licenses, etc., it is important that the licenses issued in the previous system be brought over to the new one.  This is because your customers will be renewing their licenses via the new system.  And it makes more convenient for them to easily search for their previous license and renew it.  More convenience for your customers means lesser calls from unhappy customers to your organization.

But in some cases it does make sense to not migrate data at all.  Please refer to Option 3 in this article to learn more about this option.

3. What does the process look like at a high level?

The process depends on the Data Migration Option chosen.

Process for Custom Data Migration

1. 1. 1. Requirements Gathering: The vendor's team works with your staff to determine which data needs to be moved from your current system into the new system (Ex. SagesGov).
      2. Data Clean-up: This is the step where your staff cleans up unwanted data in your existing system and corrects any inconsistencies that they do not wish to carry forward to the new system.
      3. Script Development: In this step the vendor's development team writes custom scripts to export data from your current system's data sources.
      4. Data Export: In this step the vendor's development team executes the scripts developed in the previous step to export the data from the current system's data sources into a format more conducive for loading into the new system.
      5. Data Load: In this step the vendor's development team loads the data exported from the current system into the new system.
      6. Data Verification: In this step your organization's staff is given access to the data in the new system and they verify that the data has been brought over to the new system completely and accurately.  Sometimes this step is also called User Acceptance Testing (UAT).
      7. Fix data migration errors and repeat: In this step the vendor's development team looks into the feedback given by your organization's staff when they performed data verification and corrects any errors or deficiencies.
      8. Data Export for Final Load: In this step the vendor's development team exports the data for the final load before go-live.
      9. Final Data Load: In this step the vendor's development team loads the final data into the new system for go-live.

Process for Migration via Standardized Loader

1. 1. 1. Make Reference Documents Available: In this step, the vendor's implementation team makes reference documents available to your organization's IT department.  These reference documents will have the specifications for how data must be exported by your organization's IT department.
      2. Requirements Gathering: Your IT department works with your staff to determine which data needs to be moved from your current system into the new system (Ex. SagesGov).
      3. Data Clean-up: This is the step where your staff cleans up unwanted data in your existing system and corrects any inconsistencies that they do not wish to carry forward to the new system.
      4. Script Development: In this step your IT either writes custom scripts to export data from your current system's data sources or works with your current system's vendor who develops scripts to export data.
      5. Data Export: In this step either your IT department or the current system's vendor executes the scripts developed in the previous step to export the data from the current system's data sources into a format more conducive for loading into the new system.
      6. Data Load: In this step the new system's vendor's development team (Ex. the SagesGov development ream) loads the data exported from the current system into the new system.
      7. Data Verification: In this step your organization's staff is given access to the data in the new system and they verify that the data has been brought over to the new system completely and accurately.  Sometimes this step is also called User Acceptance Testing (UAT).
      8. Fix data migration errors and repeat: In this step either your IT department or the current system's vendor looks into the feedback given by your organization's staff when they performed data verification and corrects any errors or deficiencies.
      9. Data Export for Final Load: In this step your IT department or the current system's vendor exports the data for the final load before go-live.
      10. Final Data Load: In this step the new system's vendor's development team (Ex. SagesGov's development team) loads the final data into the new system for go-live.

4. What do you need from me/my team/my organization?

What the new system's vendor needs from you and your team/organization will depend on the Data Migration Option chosen (Please see answers to previous questions). Boxes that are colored dark blue in the previous answer indicates that your organization/team/you are responsible for that step.  Boxes in color green in the previous answer indicates that it is the new system's vendor's responsibility for that step.

5. Will the export and load happen once or will it be done multiple times?

• • • Export/load for Technical Teams:   The first few exports and loads will be done earlier in the implementation.  These activities are mainly meant for the technical teams in the implementation.  It is meant to allow them to go end to end with moving data and for them to perform high level sanity checks.  A few tries are typical for the export/load to be successful at this stage and for the technical teams to have a level of confidence that the migration has happened correctly. 
    • Export/Load for UAT: After the technical teams determine that the data has been migrated correctly, they release the data that has been loaded into the new system to be verified by your staff/line of business users.  At this point your staff perform User Acceptance Testing (UAT) and verify that the data has been moved over correctly.  If they find deficiencies at this point these are communicated back to the technical teams who will fix the errors, export and load another dataset and make that available to your staff for verification.

6. What if my organization cannot give you the data in the specific format your standardized loader needs?

Sometimes we see that IT organizations are unable to provide data exports so the vendor's development team can load that into the new system.  Such circumstances are handled on a case by case basis - in some situations it may be feasible for the current system vendor's development team to write scripts and export the data on your organization's behalf (for a fee).  In other scenarios, it may not be feasible for the current vendor's team to do so.  It may be feasible for the new vendor's team to write scripts to export the data (for a fee).  The SagesGov team has done this for our customers when our customers' IT organizations had too much on their plate and the previous system's vendor was unresponsive.

7. What kinds of data will be migrated - is it only records in the database or can you also move files from the old system to the new?

Typically data migrations involve moving only records in a database and not files.  This means you will be able to move information about permits, inspections, certificates and licenses such as important dates associated with the records, information about which customers/staff performed actions on those records and other data such as cost of construction, construction type, total amount of fees collected, etc.  While the files themselves are not moved into the new software, files can still be made accessible when going from the old system to the new one.  The way this is done is by exporting all the files from the old system to a shared network location inside your organization.  The new system can then point to these files where appropriate.

8. Why should I clean-up my data before migrating it?  Can I not move everything?

Over the lifecycle of a system's usage people usually take the system in different directions.  For example, in Year 1 of the current system's existence the staff in your organization at that point in time may have determined certain kinds of data capture as important to your processes and may have configured the system to collect that data.  In Year 2, different staff members may have decided to change the data collection requirements and stop collecting the data that was being collected in Year 1. Many such shifts in the kinds of being data captured, which of the fields being captured are mandatory/non-mandatory, the validity and quality of the data being captured over the years, etc. will mean that not all data that is present in your current system is present and useful to be moved over to the new system.

This is why a data clean-up effort is extremely important.  You can decide which attributes/fields associated with each record is no longer relevant for your organization and decide not to carry that forward to the new system.  You may spend some time on other fields/attributes to manually clean-up the information present.  Doing all this ensures that you are starting off at much better point in the new system because you can now rely in meaningful ways on the data that is present in the new system (for reporting and for automated decision making).

9. What do you need from the current system's vendor?

What you need from the current system's vendor depends on the data migration option that you choose. 

• • • If you decide that Custom Data Migration is the best path forward, you will need the current system's vendor to provide you the databases/data sources for your current system.  You can then forward the databases/data sources to the new system's vendor.
    • If you decide that Data Migration via the New System Vendor's Standardized Loader is the best path forward, then you can rely on the current system's vendor to either provide you with the databases/data sources and you can then have your IT department develop appropriate scripts to export the data in a format that the Standardized Loader expects.  Or you can engage the current system's vendor to develop the scripts for the data exports (for a fee).

10. I only have 1000 records.  Will that reduce the cost?

We are often asked by our customers if the number of records in the system impacts the cost for migrating data.  If the number of records is very low, then it may be feasible for a manual approach to migrating data - your organization's staff can open up the current system and the new system and manually move the data over by creating appropriate records.

But you may ask - "We have more than a few records so our staff cannot manually move data over.  But I do not have that many records - I have only a 1000 records.  Will that not reduce the cost compared to if I had 100,000?".  Unfortunately, when data is migrated via either the Custom Data Migration option or via the option where migration is performed via the Standardized Loader, developers still need to write the scripts that export the data from your old system to the new system.  The exported data must still be loaded into the new system by the technical team.  Since the effort behind these activities is not reduced if the number of records to be migrated is 1000 or 100,000, the cost for the migration does not go down if there are fewer records.