It’s time for the application layer (layer 3) stuff, the Python script for protocol conversion and session management.
I have uploaded the code (5250_terminal.py) as well as some basic instructions to run and configure it to the Github repository:
The first two layers give us the ability to send and receive frames to and from the terminals, but the 5250 terminals and the Linux compatible terminals, specifically DEC VT type terminals, are two completely different worlds:
- The 5250 terminals are block-oriented terminals. They provide a persistent buffer where we can write and read text characters to display on the screen. We have to interact with the buffer using commands that manage the displayed text itself as well as several internal registers to change text styles, control the cursor, copy data across the buffer, etc.
- The DEC VT terminals are character-oriented terminals. We interact with this type of terminals directly sending and receiving a serialized character stream, like in the good old times of mechanical teletypes. What these terminals receive they directly print it on the screen, and what you type in its keyboard is also directly sent to the computer as another character stream.
As we can only send plain text to a character based terminal, to have a greater control over what is displayed on the screen we need to include in the character stream what we call “control characters”. The control characters are analogous to the commands in the block oriented terminals, and using the we can make things like clearing the screen, positioning the text cursor, changing text style and many other things.
So with the block-oriented terminals we have a “buffer” and “commands” to manage that buffer, and with the character-oriented terminals we have a character stream where we mix standard displayable characters with non-displayable control characters to manage the display.
There are two main kinds of control characters in a DEC type terminal:
ASCII control chars
- ASCII control characters, that are simply the characters from the ascii table below the “space” character (character 32 or 0x20 in hexadecimal). Those are characters like line feed, carry return, tabulators, etc. They are also typically used as keyboard-generated control characters, usually pressing in the keyboard the Control key followed by another key (as an example, Control + c generates the character ETX or End of Text).
- Escape characters, that are extended control sequences that begin with the “ESC” character code (0x1B) followed by one or more characters as command modifiers. The meaning of those escape sequences are terminal-specific so we need to target an adequate terminal for the conversion.
For this project we are currently implementing only the Escape sequences for the VT52 terminal, as it it simple enough for a quick implementation, and somehow matches the capabilities of the IBM 5251 terminal. In the future it would be nice to have a full VT100 or ANSI X3.64 support for this converter, as this will allow to have a nearly total compatibility with most modern text-based software. Maybe this is a little overkill for a 5251 terminal but could be adequate for more modern terminals like the IBM 3196.
VT52 escape sequences
The structure of the Python program (5250_terminal.py) is as follows:
- The program starts a thread to manage the serial port communication, to and from the Teensy. That resource is shared between all the managed terminals.
- One thread is started for each managed terminal that starts a shell process and captures its standard input and output for that terminal.
- The output generated by the shell process (character stream) is captured and converted to a 5250 command sequence. For that, the program makes an ASCII to EBCDIC to character conversion for standard text characters, and for the control characters (ASCII and Escaped) it converts them to 5250 commands that emulate the control character actions in the 5250 terminal. So a character stream is converted to a command sequence for the 5250
- The input received from the 5250 keyboard in the form of keyboard scancodes is converted to ASCII character data, that can be regular text characters (uppercase, lowercase or "alt" depending on the status of “shift”, “alt” and “caps lock” keys) or escape sequences if we press the configured “control” key plus another key.
- The main thread ends up displaying a command prompt where the user can execute some administrative and debugging commands, like restarting a terminal, sending commands to it or sending text input to the shell (the same as typing it in the terminal)
I won’t give here a full description of the VT52 escape codes, but as an example this is how the ESC_E escape code is converted to 5250 commands (ESC_E is clear screen, that we will receive from the shell as the two-character sequence 0x1B 0x45 in Hex):
- Move address counter to upper left corner with LOAD_ADDRESS_COUNTER command
- Move reference counter to lower right corner with LOAD_REFERENCE_COUNTER command
- Send CLEAR command to erase screen contents between the address counter and the reference counter
- Position the cursor and address counter to the upper left corner with the LOAD_CURSOR_REGISTER and LOAD_ADDRESS_COUNTER commands
- Send a END_OF_QUEUE command to indicate the end of command sequence
Every escape code has to be translated to a similar command sequence to run it in the 5250 terminal.
You also have a brief description of the available 5250 commands in the “5251 Maintenance Information Manual”, previously linked in this thread.
One drawback of this adaptation is that not every ASCII character has a correspondence in EBCDIC, and even due to differences in the localized character set of some terminals there are some weird character mappings that could negatively affect the user experience. For that reason the Python script supports user defined character mappings that you can configure to better suit your needs. For example in my case for the spanish 5251 character set, I have redefined these character mappings that correspond to ASCII character that are absent from EBCDIC code page 037 or get weird mappings from the default conversion:
Code: Select all
Those mappings make sense in my case, but for other systems, different or no mappings at all should be configured by the user. I selected those mappings only because the mapped characters “look better” or “look more similar” to the original ASCII characters.
To have a feeling about the capabilities of the VT52 terminal and check if you could be really interested in running it over a 5250 you can use in Linux this command line to emulate it in a xterm window:
That's more or less what you will find in the real converted 5251 terminal.
My emulation is really slightly better than that, because I’ve added some very basic capabilities to detect VT100 Escape sequences, but that should give you a broad idea of its general compatibility.
So with the current state of the project you should consider those three main limitations of the conversion:
- Only V52 support, no VT100 or ANSI at the moment. This can change in the future.
- You will need to configure your keyboard mappings, maybe in the future this would be improved with the inclusion of more mappings, some kind of autodiscovery or a better interface.
- You will get some weird character conversions, others will be missing and you probably will have to configure some manual character mappings. This cannot be improved as it is a limitation of the terminal itself.
I still have to upload the PCB files, I want to make some final adjustments to them.