Skip to content

Adventures With PHP on IBM i

I spent my first two posts whining about JavaScript and its community, but I want you to know that I don’t hate it. Not really. There are things about the language that I will never understand and things about the ecosystem that make me laugh, sure, but the fact remains that good developers can write perfectly reasonable, maintainable software using JavaScript and its many third-party libraries and tools.

The truth is that all the things people like to tease the JavaScript devs about – the endless hype, the overabundance of tooling, the breathless Medium posts – are things I can only dream of in my day job. I don’t work for a startup, or even for a tech company. I write in-house software for a small (~60 employee) business in a very niche industry. It’s the software equivalent of being raised by wolves.

Here in the software hinterlands, out of the sight of God and Hacker News and everyone, the IBM Power Systems server is the king of computing. Formerly known as “System i” (which was formerly known as “AS/400”, which, I assume, was formerly known as “Prince”), the IBM Power Systems server is in theory capable of running multiple operating systems in parallel, but is most at home running a severely outdated copy of America’s favorite operating system, IBM i.

I joke, but IBM i honestly isn’t as obscure as you’d expect (or hope). Every time I’ve attended an IBM training session, I’ve been amazed by the number and variety of businesses that are represented there: aerospace, banking, logistics, retail… If you live in the US in 2018, odds are very good that some very important part of your life depends on an RPG or COBOL program that’s older than I am, running on whatever the oldest supported version of IBM i happens to be at that moment. I wish I were joking about that.

Of course, most people in the year of our lord 2018 don’t want to write RPG or COBOL. Dimly aware that the Internet exists, IBM has at various points attempted to bring popular web development languages to IBM i. You can run Java on IBM i, as well as Python and even very outdated versions of Node.js. My company did not use any of these. Until very recently we used PHP, by way of ZendServer.

ZendServer on IBM i is the only technology that I have ever felt personally victimized by. The day I was allowed to abandon it (and PHP) was the happiest day of my life. I baked a cake.

Here are some of my stories.

The One With the Empty Strings

Some context:

IBM i ships with a database engine called DB2 for i. Unlike, say, MySQL, DB2 is not a separate, third-party application that is installed on your server; DB2 is native to the operating system. One of the consequences of this is that every time you upgrade or patch your operating system, you are also upgrading and/or patching your database engine. That means that you also need to upgrade the drivers and libraries you use to connect clients to the database, and since the community of people who write those could probably fit together in a compact car, the drivers and libraries are perpetually buggy.

Suppose you’re a Good DBA™ working with DB2 for i. You want to create a generic table for storing North American mailing addresses for your clients, but you know that your records are really incomplete and a lot of the data is missing right now. You also want to avoid storing NULL values because you’ve convinced yourself that’s a best practice (or maybe you just really like Chris Date). Your table winds up looking like this:

CREATE TABLE myschema.address
                    (START WITH 1, INCREMENT BY 1, NO CACHE),
    client_id   INT NOT NULL,
    street_1    VARCHAR(255) NOT NULL WITH DEFAULT '',
    street_2    VARCHAR(255) NOT NULL WITH DEFAULT '',
    street_3    VARCHAR(255) NOT NULL WITH DEFAULT '',
    street_4    VARCHAR(255) NOT NULL WITH DEFAULT '',
    locality    VARCHAR(255) NOT NULL WITH DEFAULT '',
    region      VARCHAR(255) NOT NULL WITH DEFAULT '',
    postal_code VARCHAR(255) NOT NULL WITH DEFAULT '',
    country     VARCHAR(255) NOT NULL WITH DEFAULT '',

    CONSTRAINT client_address FOREIGN KEY (client_id)
        REFERENCES myschema.client ON DELETE CASCADE

This database is ASCII-encoded because that’s the character encoding your company used in the 1980s, when most of its important software was written. The contractors you’ve hired in the past have sounded confused when you asked them about using Unicode (or mentioned character encodings in general), so it’s probably wisest not to press the issue.

Now suppose you’re a developer who has been tasked with creating a simple CRUD interface for managing client addresses with ZendServer. Your PHP code needs to insert a new address. ZendServer ships with a DB2 driver for PDO, so you write something like this:


try {
    $dbh = new PDO($dsn, $username, $password);
    $dbh->setAttribute(PDO::ATTR_EMULATE_PREPARES, false);

    $query = "
        INSERT INTO myschema.address
        VALUES (?, ?, ?, ?, ?, ?)

    $params = [

    $sth = $dbh->prepare($query);
catch (PDOException $e) {

Seems pretty reasonable, right? Well, depending on the version of ZendServer you’re using, one of four things will happen:

  1. The data is inserted into the database as you expected. All is well. You sleep soundly at night, knowing that you live in a just and good world.
  2. Every field that should be left blank (like street_2) gets filled with 255 spaces, or whatever the maximum length of the VARCHAR field happens to be.
  3. Every field that should be left blank (like street_2) gets filled with garbage. Just nonsense, unprintable characters that display as �. It’s not ASCII and it’s not even UTF-8, it’s just gibberish.
  4. A PDOException is thrown with a completely generic, unhelpful SQL error that references a character encoding error.

Of these possibilities, 3 is by far the worst. It appeared in a version of ZendServer that shipped with PHP 5.4, whose json_encode function would silently drop anything that could not be encoded as UTF-8. So, if your PHP application was an API server that returned JSON, your database would slowly fill with garbage without your knowledge: the garbage data would be completely invisible to your UI or any other services that consumed the accidentally sanitized data from the PHP API.

When we upgraded ZendServer from that particular version (leaping all the way to PHP 7), the behavior of the json_encode function changed. Instead of dropping the invalid data, it would either throw an exception or display the gibberish, depending on the arguments passed to it. All of the sudden, we were unable to display much of our saved data. And since you need an actual, physical IBM server to run ZendServer for i, we had no way of testing this before we went live. It was an absolute nightmare.

Luckily, you can use DB2’s TRANSLATE function to get rid of the non-ASCII text:

x'000102030405060708090A0B0C0D0E0F' ||
x'101112131415161718191A1B1C1D1E1F' ||
x'7F' ||
x'808182838485868788898A8B8C8D8E8F' ||
x'909192939495969798999A9B9C9D9E9F' ||

Just run that on every column in every table that may have had an empty string inserted into it. Simple, right?

The One With the Database Migrations

I sort of mentioned this in passing already, but you really can’t virtualize IBM i on your laptop the way you can Windows or Linux. An IBM Power Systems server has a unique CPU architecture – that’s the whole point of it.

As far as I’m aware, most small- to medium-sized businesses will, at best, hang on to their last-gen server to keep it as a test unit. The resale value on those old servers is high enough that many don’t even bother, and people just write, run, and test code on the production box: no gods, no masters. And once you’re at the point of doing everything on the fly, in production, all those best practices you learned about with respect to responsibly deplying code tend to fall away as well.

As you can imagine, this is not a community of developers who do very much testing.

When I started at my current job, absolutely none of our software had automated tests of any kind. Testing is one of those subjects people in software have Very Strong Opinions about, largely based on hypotheticals. The practical reality I faced was that when you have very old codebases with no tests whatsoever, trying to refactor anything – or even to understand what the intended behavior was originally – was very difficult to do. One of my earliest priorities was to make sure that anything new I wrote included at least some basic tests.

Since I couldn’t just spin up an instance of IBM i with ZendServer and DB2 in a VM, I needed to figure out some way to test against the production database without touching any of the production data. I figured I would do something similar to what Django’s test suite does: spin up a test schema, populate it with data fixtures, run the tests, and tear it all down. And to do that, it would be really helpful to have some kind of a database migration tool that could automate creating and tearing down the schema based on a central file (or files) that defined it, rather than hand crafting SQL queries for every single database I wanted to test against.

Everything worked when my test suite created the schema. It also worked when I tore the entire schema down. If I applied migrations that involved altering columns, however, it would just hang until the query threw an error saying that the database connection timed out. What was going on?

It all goes back to the fact that DB2 for i is integrated into the operating system. If you perform an ALTER COLUMN operation that could result in data loss (such as changing VARCHAR(255) to VARCHAR(32)), DB2 issues an inquiry message to the system operator asking for confirmation. The system operator, of course, is a human who is sitting in front of a 5250 terminal and can reply to interactive requests.

If you have the audacity to use a script to run SQL queries, your only real option is to run this command in the 5250 terminal:


This sets a default reply to the offending message, but it does it globally. Remember, the database is part of the OS. What about other, older programs running on the server that might depend on the operating system’s default behavior in some way? What if, in another context, you really did want this safeguard against data loss? And, if the point is to prevent accidental data loss, how is it that a script can drop an entire schema (!!) without getting prompted like this?

Who knows! ¯\_(ツ)_/¯

The One With the Error Log

None of the Apache or PHP error logs that ZendServer produces are rotated. They just grow infinitely in size until they take full minutes to open and/or cause text editors to crash.

That’s it. That’s the entire story.