AI can help translate a legacy COBOL program into Python, Java, Kotlin, C++, or Rust, but a successful build does not prove that the new program behaves like the original. A migration can compile, pass a handful of happy-path tests, and still calculate a financial amount differently, skip an exceptional record, or return a different status code.

The safest way to evaluate an AI-generated migration is to treat the existing program as a behavioral reference and compare it with the candidate implementation using the same controlled inputs. This is commonly called parity testing or differential testing. Regression tests make that comparison repeatable, so every change can be checked against behavior the business already depends on.

The workflow below is intended for engineering teams modernizing legacy applications. It does not assume that a particular AI model or migration tool is automatically reliable. For context on the recent model benchmark results, see our report on OpenAI’s COBOL coding and legacy software migration results.

1. Define what must remain the same

Before asking an AI coding tool to translate code, write down what the existing program is expected to do. Start with observable behavior rather than a line-by-line explanation of the source.

Record the program’s inputs, outputs, files read or written, database changes, return codes, error messages that downstream systems rely on, and any scheduled jobs or external services it calls. Identify which behaviors are contractual and which implementation details may safely change.

Create a short validation plan that answers four questions:

  • Which business operation is being migrated?
  • Which input cases represent normal and exceptional behavior?
  • What counts as an exact match between the old and new implementations?
  • Who has authority to approve a mismatch or an intentional behavior change?

OpenAI’s Codex code-modernization cookbook recommends scoping a pilot flow and documenting the existing system, target design, and validation approach. A narrow pilot is easier to inspect than a whole-system rewrite. Keep the first migration small enough that engineers can explain every relevant dependency and result.

2. Build a baseline from the original COBOL program

Run the original program in a controlled environment and preserve its outputs for a representative set of test cases. This is your baseline. Do not rely solely on screenshots, manually copied results, or a developer’s memory of what the system usually returns.

For each test case, record as applicable:

  • Input data and configuration, with sensitive fields removed or safely substituted.
  • Program output and generated files.
  • Relevant database state before and after execution.
  • Return code, error path, and meaningful log events.
  • Runtime or batch conditions that can affect behavior, such as date, locale, encoding, or sort order.

Use version control or an access-controlled artifact store to identify the exact source revision and test data used. If the legacy program depends on external systems, either use a safe test environment or document and control those dependencies. A baseline is only useful when another engineer can reproduce it.

Protect production data. Do not paste customer records, credentials, payment data, or confidential source code into a public AI service. Use approved enterprise controls and synthetic or de-identified test fixtures where possible. Preserve only the information needed to validate behavior.

3. Cover business rules and edge cases—not just typical inputs

A small test suite that contains only ordinary records can miss the most expensive migration failures. Build a test matrix with business owners and engineers who understand the legacy system.

Test category Example to include What it can reveal
Normal values A typical valid account or transaction Basic behavior and expected output
Boundaries Zero, maximum permitted value, one unit above the limit Off-by-one errors and overflow handling
Decimal arithmetic Amounts with extra fractional digits, rounding boundaries Different rounding or truncation behavior
Missing data Absent record, blank field, missing file Incorrect defaults or error handling
Invalid input Malformed date, unsupported code, negative amount where prohibited Validation gaps and changed failure behavior
Ordering Equal sort keys, duplicate records, different input order Unintended record ordering changes
Dates and time Month-end, leap day, year boundary Date parsing and calendar differences
Recovery Interrupted batch, failed write, retry, rollback Partial updates and non-idempotent retries
Volume Large batch and unusually long records Truncation, performance, and resource limits

For financial or accounting software, pay special attention to fixed-point arithmetic, implied decimal positions, packed-decimal fields, truncation versus rounding, sign handling, and overflow. A translation can produce plausible-looking values while differing by a cent or changing a downstream reconciliation. COBOL’s record layouts, file encodings, and error conventions may also need explicit treatment in the target language.

Do not invent expected outputs just to make tests pass. Derive expectations from the legacy program, documented business rules, independently calculated examples, or an approved reference implementation. Where the old behavior is itself a known defect, document the intended correction separately instead of silently changing the baseline.

4. Run both implementations against identical inputs

For each fixture, run the original COBOL program and the migrated program using equivalent inputs and controlled dependencies. Capture results in a consistent format, then compare the fields that matter.

The comparison should be strict by default. If the output is a fixed-format report or a protocol where spaces, record order, status codes, or exact decimal formatting are meaningful, compare those details. If a field is intentionally variable—such as a generated timestamp or a non-contractual trace identifier—exclude or normalize it only after documenting why it is not part of the required behavior.

Avoid broad normalization that hides defects. Trimming all whitespace, sorting every output record, or rounding every number before comparison can make a faulty migration appear correct. Normalize only known nondeterministic elements, and keep the original raw artifacts so reviewers can inspect any difference.

Here is a simple example of comparing two text outputs after each implementation has run. It is a starting point for a test harness, not a complete COBOL execution runner:

from pathlib import Path
import difflib
import sys

baseline = Path("baseline-output.txt").read_text(encoding="utf-8")
candidate = Path("candidate-output.txt").read_text(encoding="utf-8")

if baseline == candidate:
    print("PASS: outputs match exactly")
    sys.exit(0)

print("FAIL: output mismatch")
print("".join(difflib.unified_diff(
    baseline.splitlines(keepends=True),
    candidate.splitlines(keepends=True),
    fromfile="legacy-cobol",
    tofile="migrated-program",
)))
sys.exit(1)

For structured data, compare parsed fields and database state as well as text files. The right comparison method depends on the contract: byte-for-byte comparison is appropriate for exact output parity, while a schema-aware comparator may be more useful for structured responses. In either case, every ignored field should have an explicit reason.

5. Check side effects, errors, and repeatability

Matching standard output is not enough if the program also updates account balances, writes batch files, sends messages, or changes database records. Compare side effects in a test environment, including inserted, updated, and deleted records and any external calls that form part of the workflow.

Test failure behavior deliberately. What happens if an input record is missing? What happens if a write fails halfway through a batch? Does the program roll back, leave a partial result, or return a specific code? If a job is retried, does it duplicate transactions? These details often matter as much as the successful path.

Run the same test more than once where repeatability is expected. Use clean fixtures or a controlled reset between runs. A result that matches only because a previous test changed the database is not reliable evidence of parity.

Keep the test harness separate from the AI-generated implementation. The code being evaluated should not be allowed to rewrite expected results, disable failing tests, or silently alter the comparison rules. Require review for any test modification that weakens an existing assertion.

6. Use AI as an assistant, not as the final judge

A coding assistant can help explain unfamiliar COBOL sections, propose test cases, draft a target-language implementation, or investigate a failing comparison. It should not be the only authority deciding whether its own migration is correct.

A useful workflow is:

  1. Ask the assistant to summarize the selected program’s inputs, outputs, business rules, and dependencies.
  2. Have a domain expert correct that summary before implementation.
  3. Request a small migration with explicit constraints and a validation plan.
  4. Run the same test fixtures against the original and candidate programs.
  5. Share minimized, non-sensitive failure details with the assistant and ask for a diagnosis.
  6. Re-run the complete suite after each change and inspect the final diff manually.

For a broader view of agent quality controls, see our AI agent reliability evaluation checklist. For the recent OpenAI benchmark context, read OpenAI’s report on low-resource programming languages. Benchmark results can help identify models worth evaluating, but they cannot replace tests built around your own business rules and dependencies.

7. Set release gates before deployment

Decide in advance what must pass before the migration can proceed. A practical release gate can require:

  • All critical parity tests pass, with no unexplained differences.
  • Boundary, invalid-input, decimal, and recovery scenarios have been reviewed.
  • Database and file side effects match the agreed contract.
  • Security, privacy, dependency, and code-review checks are complete.
  • A named owner approves any intentional behavior change.
  • A rollback plan and monitoring are ready.

For high-impact systems, begin with a non-production pilot. Where architecture permits, compare the candidate’s results with the legacy system in a controlled shadow run without allowing the candidate to commit live side effects. Then expand gradually, monitor reconciliations and error rates, and retain a practical rollback path. The exact rollout method depends on system architecture, regulatory obligations, and operational risk; no checklist guarantees a risk-free migration.

Common mistakes to avoid

Treating compilation as proof. A program can compile and still calculate, sort, or handle errors differently.

Testing only happy paths. Missing records, unusual decimal values, boundary conditions, and recovery paths often expose migration defects.

Over-normalizing outputs. Comparison rules that erase meaningful differences can hide the very regressions the suite should detect.

Letting the model grade itself. Have a separate test harness and accountable human reviewers validate the result.

Changing expected results without approval. If the new program is supposed to fix an old defect, record that as a deliberate business decision with its own tests.

Migrating too much at once. A small pilot with explicit acceptance criteria is easier to diagnose and roll back than a broad rewrite.

Frequently asked questions

Are unit tests enough to validate an AI-generated COBOL migration?

No. Unit tests are useful, but they may not cover file layouts, batch sequencing, database changes, integrations, and end-to-end behavior. Combine unit tests with parity tests, integration tests, and business-level reconciliation appropriate to the system.

Should the migrated program match every output byte for byte?

Only where exact representation is part of the contract. Fixed-width records, protocol responses, and regulated output files may require exact matches. Other outputs can be compared structurally, but every normalization rule should be explicit and reviewed.

Can an AI coding agent generate the regression suite?

It can propose test cases and help draft test code. Treat those tests as candidates: verify expected values independently, add domain-specific edge cases, and ensure the suite cannot silently weaken its own assertions.

When is a COBOL migration ready for production?

When the agreed critical tests pass, all differences are explained and approved, side effects and failure paths are checked, security and operational reviews are complete, and the team has an appropriate rollout and rollback plan. The threshold should be defined by the system’s risk—not by the AI model’s benchmark score.

Sources and further reading

The core principle is straightforward: an AI-generated migration is a candidate implementation, not proof of equivalence. Establish a reproducible baseline, compare meaningful behavior on identical inputs, investigate every unexplained mismatch, and require accountable review before release.