Migration between test management systems is the transfer of test data from one system to another: the structure of projects and test suites, test cases with their steps and attributes, custom properties, attachments, and, depending on the decision of the team, the history of test runs and the links to requirements and defects. Migration ends when the new system is used for testing and its data are confirmed to match the source.
No standard defines a format for exchanging test cases between test management systems. Systems export and import data in general-purpose formats: CSV and spreadsheets, their own XML or JSON formats, and their APIs. Two neighboring areas have common formats: JUnit XML for the results of automated tests and ReqIF, the Requirements Interchange Format of the Object Management Group, for requirements. The entities that are transferred are described in the article What is a test management system.
Why migration is not a copy
The core entities are similar in most test management systems, but their data models differ in details, and each detail affects the transfer:
- Steps. In one system a step is a separate record with an action and an expected result; in another the steps are one text field.
- Fields. The sets of standard fields differ, and so do their values: three levels of priority in one system, four in another.
- Custom properties. Field types (list, multiple choice, number, date, user) and their allowed values are defined per system and per project.
- Formatting. Text can be stored as HTML, Markdown or plain text, and tables, images and links inside the text are encoded differently.
- Identifiers. The target system assigns its own identifiers. Everything that referred to the old identifiers, including automated tests, defects in the bug tracker and documents, has to be updated or mapped.
- History. Results of past runs refer to versions of test cases, users and environments. Not every system can import results with their original dates and executors.
Migration is therefore a transformation of data from one model into another, with decisions about what is transferred, how and what is left behind.
What is transferred
The table lists the kinds of data and the questions that are decided for each of them before migration:
| Data | Questions to decide |
|---|---|
| Structure: projects, suites, folders | Is the tree kept as it is or reorganized? How are nested levels mapped if the depth is limited? |
| Test cases | Which statuses are transferred: only active ones or also deprecated? Which versions: the latest or all? |
| Steps and expected results | Are steps transferred as separate records or as text? How are shared steps and called test cases handled? |
| Standard fields | How are priorities, types and statuses mapped between the two sets of values? |
| Custom properties | Which are needed in the new system? Are their types and values created before import? |
| Attachments | Are files transferred with the test cases and results? How are images embedded in the text resolved? |
| Test runs and results | Is the history transferred, kept in the old system as an archive, or exported to files? |
| Links to requirements | Do the requirements stay in the same tool? Are the links recreated by identifier? |
| Links to defects | Do the defects stay in the bug tracker? Are the links from results to defects recreated? |
| Users | How are authors and executors mapped to accounts in the new system? |
Test runs and their history
History is the most expensive part of a migration. Results refer to test cases, environments, builds and users, and all of them have to exist in the target system before the results are imported. Some systems do not allow results to be created with an original date or on behalf of another user.
Three approaches are common:
- full transfer: runs and results are imported with their dates and executors where the target system allows it;
- summary transfer: only the latest result of each test case, or the results of the last release, are imported;
- archive: the history stays in the old system in read-only mode or is exported to files, and the new system starts with new runs.
The choice depends on what the history is used for: audits and regulatory requirements, analysis of unstable areas, comparisons with past releases.
Exchange formats
CSV and spreadsheets
CSV is the format that almost every test management system can export and import. Its common description is RFC 4180 (October 2005), which has the status of an informational document, not a standard. According to RFC 4180, records end with a line break CRLF, an optional first line contains the names of the fields, and a field that contains a comma, a double quote or a line break is enclosed in double quotes; a double quote inside such a field is written twice.
Test cases often contain multi-line text, so these rules matter. A test case with two steps in one CSV record:
ID,Title,Priority,Steps,Expected Result
TC-13,"Pay with an expired card",High,"1. Open the cart and click ""Checkout""
2. Click ""Pay""","The message ""Card expired"" is shown"RFC 4180 does not fix the character encoding: it names US-ASCII as common usage and allows other character sets through the charset parameter of the media type. In practice, the encoding, the separator (comma or semicolon) and the format of dates are agreed between the export and the import, otherwise non-Latin text and dates are corrupted.
Two limits of CSV follow from its structure. A record is flat, so steps as separate records are encoded either as one multi-line field or as several rows of one test case, depending on what the importing system expects. Attachments cannot be placed in CSV and are transferred separately.
Spreadsheets in the Office Open XML format (XLSX, standardized as ECMA-376 and ISO/IEC 29500) are used in the same way. They avoid the problems of separators and encodings but have the same flat structure.
XML and JSON exports
Many systems export a project in their own XML or JSON format, which keeps the structure: suites, test cases, steps as separate elements, custom properties, sometimes attachments. Such a format is documented by the vendor of the source system. A target system can import it if it supports the format of this particular source, or a script converts it into the import format of the target.
API
An API gives access to all data that the system stores, including the data that an export does not contain. A migration through the API reads the data from the source and writes it into the target with a script. Such a script takes into account the limits of the APIs: pagination of lists, limits on the number of requests per minute, the order of creation (a suite before its test cases, a test case before its results), and repeated attempts after errors.
Formats of neighboring areas
The results of automated tests are not migrated as history in most cases. After migration, the CI pipeline sends new results to the new system, usually in the JUnit XML format. Automated tests that contain identifiers of test cases in their names or attributes are updated to the new identifiers.
Requirements that are moved together with the test cases can be exchanged in ReqIF, if both tools support it. ReqIF is an XML format of the OMG; its current version 1.2 was published in July 2016.
Field mapping
A field mapping is a table that says for each field of the source where it goes in the target and how its value is transformed. It is written before the first trial import and is the main document of the migration.
| Source field | Target field | Transformation |
|---|---|---|
| ID | Custom property "Legacy ID" | Copied as is |
| Title | Title | Copied as is |
| Priority: Critical, High, Medium, Low | Priority: High, Medium, Low | Critical and High become High |
| Status: Draft, Ready, Obsolete | Status: Draft, Active, Deprecated | One to one by meaning |
| Component (text) | Custom property "Component" (list) | Values are normalized and created in the list before import |
| Steps (HTML) | Steps (separate records) | Text is split by numbered items, HTML is converted to the target markup |
| Automation (yes, no) | Custom property "Automation status" | yes becomes "Automated", no becomes "Manual" |
A mapping that merges values loses information: after Critical and High are merged, the difference between them cannot be restored from the target. If the difference matters, the original value is kept in a separate field.
Values that the mapping does not cover appear in every migration: an unknown status, a component written with a typo, an empty required field. The script reports them instead of filling in defaults silently, and each case gets a decision.
Identifiers and links
The target system assigns new identifiers. The old identifier is kept in a custom property, for example "Legacy ID", so that people and documents that refer to the old identifier find the test case. A mapping of identifiers from old to new is saved as a separate table after the import.
The mapping of identifiers is used to update everything that referred to the old system:
- links from results and test cases to defects in the bug tracker, and links from defects back to test cases;
- links between requirements and test cases;
- identifiers of test cases in the code of automated tests and in the configuration of the CI pipeline;
- references in documents: test plans, reports, wiki pages.
Links inside the migrated data, such as a test case that calls another test case or a result that belongs to a run, are recreated during the import in the order of dependencies: first the objects that are referred to, then those that refer to them.
Migration process
A migration of test data usually goes through the following stages:
- Inventory. The volume of data is counted: projects, suites, test cases by status, custom properties, attachments and their total size, runs and results.
- Scope. The team decides what is transferred and what is left behind: deprecated test cases, history, attachments of old runs.
- Mapping. The field mapping is written, the custom properties and their values are created in the target system.
- Trial migration. A part of the data, for example one project or one suite with test cases of all kinds, is migrated to a test space and checked.
- Corrections. The mapping and the script are corrected based on the trial, and the trial is repeated until the check passes.
- Freeze. Changes in the source system are stopped for the time of the full migration, or the changes made during the migration are recorded and transferred afterwards.
- Full migration. All data within the scope are migrated.
- Verification. The result is checked against the source, as described below.
- Switch. Testing moves to the new system, the CI pipeline sends results there, the old system is set to read-only or archived.
Verification after migration
Verification compares the target with the source at several levels.
Counts. The numbers of objects are compared per project and per suite: test cases by status, steps, attachments, runs, results. A difference in counts is found and explained before anything else is checked.
Sample comparison. A sample of test cases is compared field by field: the title, all steps and expected results, every custom property, the attachments. The sample includes the difficult cases: long texts, tables and images in the text, non-Latin characters, test cases with many steps, test cases with all custom properties filled in.
Attachments. The number and the total size of files are compared, and a sample of files is opened. Images embedded in the text of test cases are checked separately, because they are referenced from the text and are lost when the reference is not converted.
Links. For a sample of test cases, the links to requirements and defects are followed in both directions. For automated tests, a CI run is executed and its results are checked to arrive in the right test cases of the new system.
Use. Testers execute a small run in the new system before the switch. The run checks what counts and samples cannot: that the test cases are readable, that filters and reports work with the migrated custom properties.
Migration with and without history
The decision about history affects the cost and the duration of the migration more than any other.
| With history | Without history | |
|---|---|---|
| What is transferred | Test cases, structure, runs and results with dates and executors | Test cases and structure; runs start anew |
| Requirements to the target system | Import of results with original dates and users | Import of test cases only |
| Effort | High: environments, builds, users and versions of test cases are recreated | Lower |
| Where the history lives | In the new system | In the old system in read-only mode or in exported files |
| Reports across the switch | Possible in one system | Require combining data from two sources |
The history in the old system stays available only as long as the old system or its export is kept. When the old system is shut down, the export of the history is the only copy, and its format and readability are checked before the shutdown.
Terms
| Term | Meaning |
|---|---|
| Migration between test management systems | Transfer of test data from one system to another with transformation into the data model of the target |
| Field mapping | Table that says for each field of the source where it goes in the target and how its value is transformed |
| Mapping of identifiers | Table of correspondence between the identifiers of objects in the source and in the target |
| Trial migration | Migration of a part of the data to a test space to check the mapping and the script |
| CSV | Comma-separated values: text format of tables, described by RFC 4180 |
| ReqIF | Requirements Interchange Format: XML format of the OMG for exchanging requirements between tools |
Sources
- RFC 4180, Common Format and MIME Type for Comma-Separated Values (CSV) Files, October 2005(opens in a new tab)
- ECMA-376, Office Open XML file formats, Ecma International(opens in a new tab)
- Requirements Interchange Format (ReqIF), version 1.2, OMG(opens in a new tab)
- Common JUnit XML format and examples, Testmo(opens in a new tab)