OpenAI publishes coding evaluations for low-resource languages such as COBOL

On 19 October 2026 OpenAI published research on how its models handle low-resource programming languages such as COBOL, Fortran and Visual Basic, which have far less code and documentation to learn from than Python or JavaScript. The company says the work follows feedback from developers and enterprise customers that coding models often struggle with these languages.

MultiLangBench asks a model to build a SQLite-compatible database engine in a given language, checked with tests adapted from SQLite's own test suite. At maximum reasoning effort GPT-6 Astra scores above 83% in 19 of the 20 languages, with COBOL the hardest at 72.1%. GPT-6 Luna reaches 40.3% at an estimated average API cost of $0.77 per sample, against $3.48 for GPT-5.6 Sol. OpenAI is releasing the evaluation setup, test suite and per-language results on GitHub.

The second evaluation has models migrate COBOL programs to C++, Java, Kotlin, Python or Rust and counts a migration as successful only if hidden tests show byte-for-byte identical outputs. GPT-6.1 Sol scores 86.6% at low reasoning effort, compared with 5.4% for GPT-5.6 Sol, while costing 39% less, and 97.9% at higher effort. OpenAI notes that using this on production software still needs expert review and testing.