15 Jun 2013

15 Jun 2013

Quality Gates validates thousand of tables in Big Data migration in less than 1 hour!


Category: New Feature, News

0 Comment

Quality Gates validates thousand of tables in Big Data migration in less than 1 hour!

Apparently, some IT manager start shaking when it come to validating database migration. But actually…, it is much easier than you think. You just need to know how :)

In one very challenging migration project, we were required to validate consistency between Oracle and Greenplum databases.This was not the first time we had to validate this kind of migration, but the challenge here was, to make this validation over more than 2,000 tables and views!!

To do that efficiently, we decided to use Quality Gates API, and within less than 1 hour we automatically create thousand of tests and ran it against both database.
Not long after, Quality Gates was already providing the project manager with a clear picture of the migration state and the problematic areas that needed to be investigated.

We discovered that around 20% of the tables and view had a gap between the 2 sources. At this point we had to find what the cause for the gaps were, and which specific columns and rows out of milions of records were not matching. This was also an easy task; the relevant tests which failed could be modified in order to find the exact place where the gap is.

Until the end of the project all those thousands of test were scheduled to run at least once a day and keep on reflecting a clear picture of the migration state.

Easier done than said ;)

API dashboard

The image above showing the main dashboard of Quality Gates which gives an immediate overview of the migration status and progress.
Out of 1197 test implemented over the different tables, there were only 74 tests that failed (6%).