A few months ago, IBM slapped me on the wrist. No, this wasn’t really physical punishment or like hazing like at the Military Academy. Rather, it was a Notes Inbox slap named Action Required: Hercules Detection Report. It turns out that the CIO’s office had scanned my BYOD Macbook Air, and I was busted for breaking a major IBM taboo: installing the Hercules mainframe emulator on an IBM workstation. Reading the strongly-worded message, my mind conjured up the image of  TJ Watson Senior, the “Father of IBM,” scowling down at me from heaven.

Watson Mad
“McBride, you are an impish Scofflaw!!” -imaginary Watson

From a legalistic standpoint, I felt wrongly accused. Sure, I had downloaded Hercules and MVS/380 with the intention of checking it out, but due to Breaking Bad, Game of Thrones,  Bioshock: Infinite, and sundry other distractions, I never seemed to have the time to get it installed. I was incensed:

How could IBM judge me for a crime that I had been too lazy to commit? C’mon, this is starting to sound like thoughtcrime and Big Brother stuff.

So how did IBM and Hercules get to this point? Well, in many ways, this taboo is a more recent occurrence. Between 1999 and 2009, IBM seemed to have a fairly peaceful relationship with the hobbyist community that ran older versions of IBM operating systems (such as MVS 3.8 and VM/370) and z/Architecture Linux on their PCs using Hercules. Things changed in 2009 when Roger Bowler, the creator of Hercules sought to monetize his creation by forming a company called TurboHercules to sell Intel x86-powered boxes capable of emulating  z/Architecture to run modern IBM operating systems, such as z/OS.

IBM licenses mainframe software in such a way that end-users are only authorized to run mainframe software on IBM System z hardware. While it’s possible to run z/OS and z/VM on x86 hardware using Hercules, it’s a violation of the software license agreement to do so. Think of this act is akin to Hackintoshing (running Mac OS X on a non-Apple PC such as a ThinkPad).

TurboHercules thought  it found an opening to chip away at IBM’s licensing agreements in an ambiguous clause of the license that authorized customers to transfer their operating system license to “another machine” in the event that their mainframe suffers an outage. According to their LinkedIn pageTurboHercules is exactly this sort of “another machine,” enabling them to “provides Mainframe Disaster Recovery solutions” and acts as “a ‘stand-by’ mainframe” via “emulation of a mainframe or a partition.” Tom Lehman, the VP of TurboHercules product development expanded on this idea by saying that TurboHercules is “not an alternate mainframe,” but “more like that spare tire in the trunk of the car that’s going to get you home in the event of a flat.

Despite this seemingly benign self-characterization, TurboHercules represented a threat to IBM’s ability to maintain the critical mass needed to support ongoing mainframe hardware development. The intention of IBM’s software license agreement was to ensure that IBM software ran on IBM hardware, but if TurboHercules could exploit this chink in the license’s armor, then perhaps they could eventually lobby anti-trust authorities to force IBM to license their mainframe software for emulated mainframes on commodity x86 hardware. In a worst case scenario, siphoning off a portion of the mainframe hardware business might force IBM to abandon ongoing development of its mainframe processor family altogether and shift to Xeon-based systems, similar to those that Unisys sells to run their legacy MCP and OS2200 workloads. For these reasons, IBM likely considered it critical to protect their software licensing structure, leading them to send TurboHercules a cease-and-desist letter threatening a lawsuit over alleged patent violations.

“We were merely responding to TurboHercules’ surprise that IBM had intellectual property rights on a platform we’ve been developing for more than 40 years.”

-IBM spokesperson Steve Eisenstadt

When faced with this threat, TurboHercules retaliated by filing an anti-trust suit with the EU in March of 2010 in an attempt to force IBM to license z/OS and mainframe software for use on TurboHercules’ x86-based pseudo-mainframes. In a statement, Roger Bowler alleged that “by tying IBM’s mainframe operating system with IBM hardware,” IBM “prevents TurboHercules from providing its product to mainframe customers desiring an open-source solution.” This suggests that Roger Bowler intended to take a card from Dr. Gene Amdahl’s playbook and turn TurboHercules into a hardware competitor for mainframe software.

According to Roger Bowler, IBM’s allegation “that TurboHercules seeks a free ride on IBM’s ‘massive investments in the mainframe by marketing systems that attempt to mimic the functionality’ of its machines” was akin to the charge that “Linux is nothing more than an attempt to ‘mimic’ the functionality of Unix.” This statement clearly referred to the longstanding SCO v. IBM lawsuit over whether IBM’s work on GNU/Linux violated SCO’s intellectual property in UNIX, which was controversial because of Microsoft’s decision to fund SCO with the alleged aim of sowing  fear, uncertainty, and doubt in the Linux community. Drawing on this parallel, it became public in December of 2010 that Microsoft was funding TurboHercules and other companies in their anti-trust efforts against IBM’s mainframe software licensing.

“Microsoft shares TurboHercules’ belief that there needs to be greater openness and choice for customers in the mainframe market. Customers tell us that they want greater interoperability between the mainframe and other platforms, including systems that run Windows Server. For that reason, we continue to invest in companies like TurboHercules to develop new solutions for our mutual customers.” 

-Microsoft Spokesman

Despite Microsoft and TurboHercules’ best efforts,  the EU Antitrust suit worked out well for IBM. In 2011, the EU dismissed the suit after an in-depth investigation of IBM’s licensing practices. IBM committed to supply “spare parts and technical information swiftly available, under commercially reasonable and non-discriminatory terms, to independent mainframe maintainers,” but the core principles of IBM’s mainframe software licenses were found not to violate EU antitrust rules.

“I am pleased that we could find a swift solution with IBM to our competition concerns. Timely interventions are crucial in fast moving technology markets.”

-European Commission VP of Competition Policy Joaquín Almunia

This decision wiped out the potential business model of TurboHercules (and Neon Software and T3). If you go to the TurboHercules website, you’re now greeted by a terse message:

“System Offline: This Site is currently Offline.”

Given TurboHercules’ role in Microsoft’s legal proxy war on IBM, it’s unsurprising that Hercules has left a bad taste in IBM’s mouth. However, I believe that it is now fair to say that the two year legal battle with TurboHercules has concluded in IBM’s favor.

It’s time to draw more of a fine grained moral distinction between the antagonistic TurboHercules company of 2009-2011, which was a one time threat to the mainframe platform, and the open-source Hercules project of 1999-today, which has been a continuous asset to the mainframe platform by building up a community of mainframe enthusiasts. By enabling technologists to run older public domain mainframe software on their PCs free of charge, Hercules provides access to the mainframe style of computing to those that do not have access to mainframes through their employers. From my point of view, this is undoubtedly a good thing.

That said, I understand the position of IBM’s Chief Information Officer and the reasoning of the Action Required: Hercules Detection Report. Rather than write a philosophical manifesto, rocking the boat, and stressing out my management chain, I complied with IBM’s request and removed Hercules from my BYOD Macbook Air.

One reason that I have been so compliant is that I have discovered a good solution for experiencing the work of the Hercules enthusiast community without putting IBM’s IP in danger: I run Hercules on a computer other than my IBM workstation. While this might have been too expensive in the past, thanks to a post from Soldier of Fortran, I now run Hercules cheaply on a Raspberry Pi.

For those of you that have been living under a rock, the Raspberry Pi is a ARM-based computer capable of running small instances of Linux at an incredibly low price. In my case, $70 bought me the Pi circuit board, a plastic case, an SD card, and a wireless USB dongle. Because I am most interested in tinkering with the MVS/380 and VM/380 forks of public domain mainframe software from 1981, the Pi is more than sufficient for achieving levels of performance exceeding the 3081 Processor Complex of that era(Oh.. Ahh. thirty-two whole megabytes of memory).

Given the low cost of the Pi and the ease of installation of Hercules and the pre-configured “MVS Turnkey” and “VM Six Pack” environments, the Raspberry Pi has become a great playground for gaining initial exposure to mainframe computing.

In the near future, Millennial Mainframer will publish a series of guides to walk you step-by-step through the process of building a Raspberry Pi, installing the Raspbian operating system and Hercules emulator, and setting up and using the MVS Turnkey and VM Six Pack environments. My hope is to leverage the Raspberry Pi and the Hercules and the Turnkey projects to help college students and millennials gain access to the mainframe style of computing in a safe, cheap, and legal way.

If you are interested in following this process along with us, I suggest purchasing the following components. Please use the following links, as proceeds from the Amazon Associates program help defray the cost of hosting Millennial Mainframer:

Happy New Year!

Nicolas Raymond ~  Free Grunge Textures

How to Destroy a Sysplex

To say we had an interesting Business Recovery Exercise this week would be an understatement!

Since bringing our BR (business recovery) / DR (disaster recovery) solution in house, rather than performing offsite, we’ve had a total of five BR Exercises this year alone.  This is pretty impressive for our shop since we use to go YEARS between BR Exercises.  Now our clients can declare a BR Exercise without prior notice to ensure our infrastructure is sound and solid.

Our infrastructure IS sound and solid…provided no one  messes with it!

Two months earlier I was doing what I thought was helpful clean up on RACF.  I was adding a new PROFILE for a monitoring application.  Our RACF expert had just recently retired and our new RACF person was not quite trained and up to speed.

On occasion I would go in and “fix up” some things in RACF trying to helpful.  Although I had ADMIN rights to reset PASSWORDS when I’m on-call,  I’m not really suppose to mess around in RACF.

But what’s the worse that can happen?

I honestly thought I was doing something good by deleting a VERY suspicious * (G)ENERIC profile.

Disaster Recovery_RACF_profiles
(* I have my very own screen shot auditing script that captures my screen every minute on my workstation.  It was able to capture the quiet destruction of the sysplex.)

To me this generic profile seemed a security risk and decided to take matters into my own hands (since the new guy surely was not going to) and DELETED this profile!

Disaster Recovery_RACF_delete
Ah oh!!!
Disaster Recovery_RACF_warning
“You still have a chance to undo this Paul!!!”
Disaster Recovery_RACF_refresh
Nope.  Profile is deleted.

Quiet Disaster

What I didn’t realize what I had done is that instead of making the system more secure I delete a VERY important PROFILE that’s used at IPL.

As Michael Cairns excellently describes in his article “Addressing Common RACF Configuration Issues“, that * GENERIC PROFILE was the catchall profile.

[The] class SURROGAT profile consisting simply of "**" or "*.*" (sometimes called a catchall profile). It applies to all user IDs that aren't matched by a more specific profile and probably covers your user ID unless steps have been taken to avoid this.
Without a catchall generic profile of some kind in the class STARTED, a previously undefined started task will fall back to the contents of ICHRIN03. 
If fallback to ICHRIN03 can happen, you need to know what privileges it's granting.

That’s exactly what happened.

We started the Business Recovery Exercise and the system upon the first IPL came to a screeching halt.  Apparently JES2 (Job Entry Subsystem) did not have authority and the ICHRIN03 was poorly coded.

But…NOTHING has changed!!!!

Imagine the frustration my fellow colleagues (and myself before discovery) were experiencing.  Here we were doing our FIFTH BR exercise this year.  It always worked.  It never failed.  We had a perfect mirror of our working production.  Nothing had changed!

To make a long story (and painful one for me) short, we opened a Service Request with Severity 1 with IBM.  This is equivalent to calling 911 or pressing the nuclear panic button when you need IBM support and need it fast!

We were directed to a teleconference with their JES and RACF experts and with their AWE INSPIRING expertise guide us to the discovery that yes, we were missing that * GENERIC PROFILE in RACF.  Since JES2 at our shop started in a certain sequence we were unable to re-create this PROFILE on our BR system.

Since this was a mirror of our production we discovered that we were in fact vulnerable on our PRODUCTION SYSTEM!!!

If we had IPL’d any of our production LPARS, meaning recycling them, there was NO WAY they were coming back up.  JES2 would have ran into the same authority issue error and the entire system would be in a matter of speaking…toast!

Luckily we caught this and were able to RECREATE the profile on our PRODUCTION system so we could mirror it over to the BR SYSTEM and finish the exercise.

Take away lessons:

  1. NEVER…  EVER…   MESS WITH RACF! (At least without knowing what you’re doing.  My RACF roles have been relinquished to the appropriate people.)
  2. Business / Disaster Recovery Exercises are there for a REASON!  If you’re not doing it at your shop, how do you know you’re not vulnerable?


Cadet at Full-Screen Terminal

Today is December 1, 2013. Today I turn twenty-eight years old. Today is also approximately the ten year anniversary of the most interesting “birthday party” I’ve ever been to. As a millennial mainframer, I’ve occasionally had instances where I’ve figuratively considered my nuts in a vice (like last week, when I ran the wrong CMS commands and copied my broken z/VM environment over my backup DASD volumes instead of visa versa), but ten years ago, my nuts literally were in a vice. Just thinking back on this story makes my testicles writhe in pain and retract into my abdomen.

Ten years ago, I was a first-year cadet at the United States Military Academy at West Point. First-year cadets at West Point are called “plebes,” which is a Latin term that denotes the underclass. Much like the barbarism of antiquity, plebes at a military academy are treated like crap. They clean the barracks, deliver the newspapers, pick up the trash or recyclables, march not walk, and salute all of their superiors with a chipper attitude. They also are expected to memorize the meals of the day, the football schedule, the main articles of the New York Times, West Point lore, and errata about the heroes of the American Empire.  Failure of any of these tasks is never pretty.

The corps of cadets is organized into companies of around 120 cadets each, and each company has a slightly different culture. To my misfortune, I was a plebe in company A-2, known as the Spartans. Due to the historical legacy of Spartan military discipline and oppression of the enslaved Helot underclass, company A-2 was well known for their thorough and systematic hazing their Plebes.

At West point, there is one socially sanctioned avenue for plebes to exact revenge on especially brutal upperclassmen: the “Birthday Party.” In such a birthday party, the plebes break into the upperclassman’s room, tie them up, and demean them in some way. Often this might be tying them up, placing them in the communal open shower, and spraying they with ketchup, mustard, etc. stolen from the mess hall. Of course, the specific recipe of revenge would always slightly vary from upperclassman to upperclassman.

So there was one Spartan upperclassman that was considered especially mean and vindictive towards the plebes. He loved to ask obscure and seemingly impossible-to-memorize trivia in the hope of being able to scream at and bully a plebe. Considering that his grades weren’t great, his physical fitness was poor, and other aspects of his life likely sucked, bullying seventeen and eighteen year old plebes was likely the high point of his life. Given his general asshole-ish-ness, it was guaranteed that this guy was going to get one hell of a “birthday party.”

So the Spartan plebes geared up as if for the battle of Thermopylae. We put on football equipment used by the intramural football team. We drew battle plans and planned to stay in formation to be able to overwhelm the upperclassman and subdue him quickly and painlessly. Unfortunately, things didn’t work out to plan.

McBride Football Plebe
In my Intramural Football Garb

Once the plebes burst through the door, the upperclassman’s roommate attacked up from our right flank. This disoriented the biggest plebes leading the charge, and forced me to assume point and continue the attack. I charged at Cadet Sergeant Douchebag and tackled him facedown onto his desk. With his torso down pinned down, I considered him momentarily subdued.

However, the upperclassman still had one last gambit. I felt his hand reach up along my thigh shortly before I felt a strong hand close vice-like around my testicles. I let out a shriek several octaves higher than I had ever uttered before, and the upperclassmen let out an evil chuckle and made an ominous announcement to the other plebes:

“I have McBride’s nuts, and if you ever want him to be able to procreate, you’ll back the f*** off and get out of my room.”

Once everyone realized the gravity of the situation, the plebes stopped their attack and my comrades retreated away. Suddenly, I was alone with the two upperclassmen. My testicles throbbed with pain, my face was beat red, and a layer of persperations seeped through the football equipment.  “Damn it,” I thought to myself, “why the hell didn’t I wear a cup?”

With the situation diffused, the upperclassman opened his strong hand and released my family jewels. Looking at his face, the upperclassman had a sly and self-satisfied look on his face. He caught my glance and winked.

So with this being a mainframe blog and everything, you might be wondering why you’re reading about military academy hazing and Sean McBride’s “nuts in peril” anecdote. Well, it turns out that mainframe computers were a big deal at West point in the 1960s and 1970s that they became part of the hazing culture.

By the 1960, it was clear to the big brass that the Army needed savvy centurions ready to answer the President’s call to directly engage the forces of Global Communism in limited nuclear wars or “brush fire” conflicts in far off places like Vietnam. As a result, West Point changed it’s curriculum so that “every cadet should have practical exposure to computers, including the writing, running, and testing of computer programs to solve real problems.” In short, the West Point leadership voiced opinions quite similar to those now heard by Code.org: EVERYONE (or at least every cadet) should learn to code. Fifty years ago, these curriculum changes were downright revolutionary. At MIT and other computing powerhouses of the era, computers were used by highly specialized engineers, mathematicians, and asocial hacker types. At West Point, even the dumb jocks coded… Even the Football player coded… Even the future Infantry officers coded…

Sure West Point had “Hacker” types like Charles L. Baker of Sheffield, Alabama, who had an “admiration for force vectors, do-loops and [the] dear old GE 225 [mainframe that] know[ing] no bounds” and Larry Hartman of Tucsan, Arizona, who “spied a sweet young thing named the GE 225 and has been in love ever since,” but by the mid 1960s, all cadets had personal experience coding on a mainframe computer, making computing a common cultural touchstone for cadets decades before PCs made computing mainstream.  Sure, computing clubs at places like MIT coded, but at West Point, even the SCUBA Club coded, modeling an Underwater Navigation Course using the GE 225.

It didn’t take long before computing combined with the timeless West Point tradition of hazing Plebes (first-year cadets) to produce bizarre scenarios, such as this:

Upperclassman: “Plebe! Are you stupid or something?”
Plebe: “No, sir!”
Upperclassman: “Wrong answer, beanhead, are you stupid?”
Plebe: “Yes, sir!”
Upperclassman: “How stupid are you, crot?”
Plebe: “Sir, I make the incline plane look like a GE 225 digital computer!”

In fact, in many ways, the very act of learning to code became an additional element of West Point hazing during the sixties and seventies. At that time, plebes were required to take a programming course, where they would code in a simplified version of FORTRAN called CADETRAN using mark-sense cards (think number-two pencils and Scantron-style cards). This process was euphemistically called “squint and print,” “jot and plot,” and “punch and pray.” Whatever it was called, it surely must have been better than “nuts and crunch.” 🙂

Between 1968 and 1971, cadets transitioned to being able using teletype terminals that resembled typewriters, and between 1974 and 1983, cadets even used terminals with full-screen displays.

Cadet at Full-Screen Terminal
Cadet Frank Monaco (USMA Class of 1970) pecks at a keypunch while working diligently on his senior design project for EF489: a macro assembler for the GE 635 Mainframe. In a twist of fact, Frank later returned to West Point to act as the Academy’s Chief Information Officer during the 1990s. Frank now consults and teaches courses on computing. Follow him on Twitter at @ProfessorMonaco.

I’ve recently asked a handful of West Point graduates about their memories taking computer coursework on this mainframe, and here are few of the best anecdotes:

“Our mainframe was enclosed in a huge heated and air conditioned room in the bowels of Thayer Hall. We never interacted directly with “THE MACHINE.” We created flow charts and wrote code in an abbreviated version of fortran called cadetran. Then we created punch cards which represented each operation of our code. If there were no “hanging chads,” and if the damn cards were in the right order, and after waiting hours (or days) for everyone else whose stacks of cards were in line in front of us, our program would get run by “THE MACHINE.” Frustratingly, I rarely got my programs to work properly on the first run. I usually wrote good code. Failure to catch that “hanging chads” had caused the card reader to misread a card, or failure to have them all in the proper order was typically the culprit. But it took an inordinate amount of time to track down the culprit, create new cards and/or reorder cards (opportunities for more error – like inspection walking the area is an opportunity for more demerits!), place card stacks into the queue again, and wait. Oh yes, I remember it well….”

“My classmate had a program in his hands one day that was about four inches high. As he was crossing the road they slipped and the wind took punch cards to hell and back. It took us fifteen minutes to police them all up.”

“Some of us played Star Trek [one of the earliest computer games] while most were lost in space. Time was of the utmost importance. As if by magic, by the time our terminal got to the last line in a drawing, the last multiple guess, the final print-out, the 635 would disconnect.”

Prior to Cadetran we also did programs in a version of Basic which I think they called Cadet Basic. I was fortunate to take a couple of electives in Management Science that actually used computers to solve problems, rather than just running programs to learn computers… In spite of the operational hurdles it took to actually use USMA computers, we did learn the concepts of computer science. At least I like to think that foundation is what allowed me to go into CS for my graduate work.”

“I was enrolled in a Fortran course (known as FORTY). I am sure most remember working on teletype variants in which your input was typed as a new line on a very long sheet of paper and your output came back the same way. That and punch cards were the standard. As someone taking an elective, I was admitted to the outer rings of cognoscienti. At one class period, this semi-select group was taken down the hall by our Professor. He pulled out his keys and opened a locked door, ushering us into what was surely one of the sanctums of computer engineering. In that stronghold, I was shown a strange new device in which our interactions with the computer were displayed, a CRT! Of course it was monochrome with everything displayed in various shades of green. We emerged discussing our discovery in hushed and reverent tones.”

“I remember when we witnessed a demonstration of programs drawing vectors on the screen of a monitor. That was top notch stuff. FORTRAN was fun; it made me think I wanted to go into computer science. COBOL was so boring that I changed my mind. Our mainframe was a 635, Honeywell or GE. I can’t remember which company bought the sector from the other.”

“I just remember hand writing out the code, and then transferring it to the punch cards, and waiting – a common refrain from you all. As I recall, in the late 60’s, it was always a challenge to decide whether to use the faster automated punch card writing machines (that took whatever you gave them even if it had a typo) or to use the slower but potentially more accurate mark sense punch cards. I am still not sure how we ever identified the errors, much less corrected them.”

“Yes I remember the card punch room outside the glassed-in computer. That’s where I learned a phrase I still use today. The Do loop. 010 Do 20 020 Do 10 I’ve been in a do loop ever since.”

“The more important memory for me was the Star Trek game. I look at video game interfaces now and think about the mental math that was required to play that game and just shake my head in wonder that it was actually as much fun as it was. I also learned not to go up to the terminal room until somewhere around 10 at night. If I went up earlier, I would get so absorbed in the game that I wouldn’t notice the time. Taps would come and I would realize that I hadn’t gotten all the rest of my assignments done. I guess that qualifies as addictive behavior, but at least I worked out strategies to control it! Those memories also make it hard for me to get too far up on my high horse with my boys when they spend entire evenings playing video games!”

On that note, I’m going to sign off and play some video games myself.

Have a great week and remember to protect your nuts!