OpenAI published new evaluation results on October 19, 2026, reporting progress in AI-assisted software development with programming languages that have comparatively limited training data and documentation. The work focuses on languages including COBOL, Fortran, and Visual Basic—technologies that remain important in banking, enterprise applications, scientific computing, and other long-lived systems.

The announcement, titled “Advancing Coding in Low-Resource Programming Languages”, is not the launch of a separate ChatGPT feature. It is a report on model performance across two evaluations: building a SQLite-compatible database engine in 20 programming languages, and migrating existing COBOL programs while preserving their behavior.

For developers and technology leaders, the key question is not simply whether an AI model can generate code. It is whether it can understand unfamiliar legacy systems, preserve business rules, pass meaningful tests, and do so at a cost that makes modernization more practical.

What OpenAI announced

OpenAI says it developed two evaluations to measure different software-engineering tasks.

MultiLangBench tests whether a model can build a SQLite-compatible database engine from scratch in a specified programming language. The task involves organizing a codebase, managing state and errors, and adding functionality while preserving existing behavior. The company says it adapted tests from SQLite’s existing test suite and evaluates results through database files and expected query outcomes.

The second evaluation focuses on COBOL migration. Models are asked to translate existing COBOL programs into C++, Java, Kotlin, Python, or Rust. The test checks whether the migrated program preserves the original program’s observable behavior, including details that can be easy to overlook when translating business software.

OpenAI says the evaluation setup and test suite for MultiLangBench are available on GitHub, allowing developers to inspect what is tested rather than relying only on headline scores.

The benchmark results: what the numbers say

OpenAI reported the following results in its October 19 publication:

  • GPT-6 Astra: an average test pass rate of 87.4% across 20 languages on MultiLangBench, compared with 26.8% for GPT-5.6 Sol at the maximum reasoning setting described by the company.
  • GPT-6 Luna: an average pass rate of 40.3% across the same languages at maximum reasoning effort. OpenAI estimates an average API cost of $0.77 per sample, compared with $3.48 for GPT-5.6 Sol—approximately 78% lower.
  • GPT-6.1 Sol on COBOL migration: 86.6% accuracy at low reasoning effort, compared with 5.4% for GPT-5.6 Sol at the same setting.
  • GPT-6.1 Sol at higher reasoning effort: 97.9% accuracy on the COBOL migration evaluation, according to OpenAI.

These are company-reported results from specific evaluation tasks, not independent proof that a model will deliver the same performance on every commercial codebase. The benchmark scores also should not be treated as interchangeable: building a database engine across multiple languages and migrating existing COBOL programs measure different capabilities.

OpenAI’s results suggest that its newer models may be more useful for unfamiliar or less commonly represented languages, but teams should examine the exact test conditions, reasoning settings, model versions, and costs before applying the figures to their own projects.

Why COBOL modernization matters

COBOL remains embedded in many established enterprise systems. These systems may process financial transactions, account records, billing, inventory, and other operations that organizations cannot simply replace overnight. The challenge is often not that the software has stopped working; it is that maintaining or changing it can be difficult.

Documentation may be incomplete, original developers may no longer be available, and important business rules may be encoded in decades-old source code. A migration that compiles successfully can still be wrong if it changes rounding, record handling, error behavior, or an edge case relied on by a downstream system.

OpenAI’s migration evaluation explicitly addresses behavior preservation. The company says it uses hidden tests and requires byte-for-byte parity on observable outputs for a migration to count as successful. One example in its explanation concerns fixed-point arithmetic: COBOL may discard excess decimal digits unless rounding is explicitly requested. A rewrite that rounds those values differently might run without crashing but still produce incorrect financial results.

That is why AI-generated migration code needs more than a syntax check. Teams need regression tests, representative data, domain-expert review, and a careful comparison of old and new outputs.

What developers can use this for today

The announcement may be relevant to developers maintaining legacy code, engineering teams planning modernization, and enterprise architects evaluating AI coding tools. OpenAI says its latest models can be tried through the API on coding tasks in the languages teams use.

A practical pilot can begin with a bounded, non-production component rather than an entire core system:

  1. Document the existing behavior. Collect available specifications, sample inputs and outputs, error cases, and dependencies before asking a model to change the code.
  2. Build a baseline test suite. Include normal cases, boundary values, invalid inputs, rounding behavior, and historical edge cases that the current system must preserve.
  3. Ask for small, reviewable changes. Have the model explain the relevant code paths and propose a migration plan before generating a large replacement.
  4. Run old and new implementations side by side. Compare outputs on the same test data and investigate every difference rather than assuming a passing build means equivalent behavior.
  5. Measure cost and review effort. Track API usage, time spent validating generated code, defects found, and the number of changes that survive human review.
  6. Keep production controls in place. Require code review, security checks, integration testing, rollback plans, and sign-off from owners who understand the business rules.

For broader context on AI-assisted development workflows, see our guide to Claude Code vs. Codex vs. Gemini CLI. The tools and models differ, but the same principle applies: evaluate them on your own repository and workflow rather than selecting one from a single benchmark.

What MultiLangBench does—and does not—prove

Benchmarks are useful when they expose a task clearly enough for others to inspect and reproduce. MultiLangBench’s focus on a working database engine is more demanding than a short code-completion prompt because a solution must preserve functionality across an extended task. The COBOL migration evaluation adds a separate challenge: preserving the behavior of existing software.

Still, a benchmark is only a proxy for production work. A real enterprise repository may include proprietary libraries, undocumented dependencies, unusual build systems, strict performance requirements, and business rules that are not captured in a public test set. A high score does not establish that an AI-generated migration is secure, maintainable, compliant, or ready for deployment.

The results are also published by the company whose models are being evaluated. Independent replication and testing by developers would provide additional evidence about how well the results generalize.

For teams comparing AI coding models, the useful takeaway is to inspect the benchmark and reproduce relevant tests where possible. The public OpenAI announcement provides the reported results and methodology; the MultiLangBench GitHub repository is the place to examine the released evaluation materials.

Is this a new feature in ChatGPT?

Not on the evidence in this announcement. OpenAI describes model improvements and evaluations, and says its latest models can be tried through the API on coding tasks. The post does not announce a new standalone button, dedicated COBOL migration interface, or automatic production-migration feature in the ChatGPT app.

Availability depends on the model and the access route. Developers should check the current OpenAI API model documentation and account access before planning an integration. The announcement should be understood as a capability and evaluation update, not a promise that every user has access to a specific coding workflow in every product.

Frequently asked questions

Which programming languages were tested?

MultiLangBench covers 20 languages, including COBOL, Fortran, Visual Basic, Python, JavaScript, C, C++, Go, Java, Rust, and others. OpenAI’s separate migration evaluation focuses on converting COBOL programs to C++, Java, Kotlin, Python, or Rust.

Does the result mean AI can safely replace a COBOL system?

No. The scores describe performance on defined evaluations. Production modernization still requires domain expertise, test coverage, security review, integration testing, and verification that the new implementation preserves business-critical behavior.

Where can developers inspect the benchmark?

OpenAI links to the MultiLangBench repository on GitHub, which contains the evaluation setup and test suite described in its announcement.

Is this an independent benchmark?

The results in the October 19 announcement are reported by OpenAI. The released evaluation materials can be inspected, but the reported figures should not be described as independently verified unless separate replication is available.

The bottom line

OpenAI’s October 19 results point to progress in AI-assisted coding for languages that are less common in modern training data but remain important to established software systems. The strongest practical opportunity is not to replace legacy infrastructure blindly; it is to help engineers understand, test, and incrementally modernize it.

For teams considering COBOL migration or other legacy-code projects, the right next step is a controlled pilot with explicit acceptance tests and human review. Benchmark scores can help identify promising models, but only testing against the behavior and constraints of the target system can establish whether the model is useful for that project.

Sources