原创 Revisiting vintage multi-core

After several years of hype, multi-core and multiple-CPU embedded systems are now becoming mainstream. There are numerous articles about multi-core design that address different hardware architectures (homogeneous and heterogeneous multi-core) and software architectures (AMP: asymmetrical multi-processing and SMP: symmetrical multi-processing). In this article the development of an AMP system is outlined, highlighting various challenges that were addressed. What is unusual is that the design was completed in 1981!

It often appears that there is nothing truly new in the world. With technology, it is often a question of the time being right. The programming languages that we use nowadays are mostly developments of 30-40 year old technology. The current enthusiasm for multi-core designs is really nothing new either. Browsing the literature turns up titles like "Multi-core is Here Now" that have been appearing for at least 5 years.
But multi-core goes back further still. I was working on a multi-core system in 1980 ...

How it all started
It was my first job out of college and I worked for a company that made materials testing instruments – large machines and systems that stressed and broke things under controlled conditions. The use of computer or microprocessor control was new. Hitherto, the machines had been controlled by racks of analogue electronics, with meters and chart recorders providing results. I worked in the division that provided the computer control. Initially, the approach was simply to link a mini-computer to a traditional console. The next – and, at the time, brave – step was to replace the entire console with a microprocessor where a keypad enabled input of parameters and selection of settings from menus on a screen. Of course, a mouse or touch screen might have been better, but that technology would not appear for some years.

The project to which I was assigned was to facilitate the "user programmability" of the new microprocessor-controlled machines – the "User Programmability Option" or "UPO". It was decided that the best way to provide this capability would be to add an additional computer instead of potentially compromising the real-time behaviour of the controlling microprocessor. This is exactly how I might advise a customer today who is designing a multi-core system with real-time and non-real-time components.

The processors
The advanced console was built around a Texas Instruments 9900 microprocessor, which was one of the first true 16bit, single-chip devices on the market. It had an advanced architecture, with some interesting pros and cons: it could intrinsically support multi-threading in a very simple way, with context saving accommodated in hardware; but its registers were mostly RAM based, which, at the time, was a significant performance limiter. The instruction set and addressing modes bore some similarity to the 68000. I recall that the documentation was confusing, as the bits were numbered backwards, with the most significant bit being #0. This part of the system was programmed in Forth code (see code below). I have no idea why this design decision was made, but I found the language intriguing and my interest persists.

The UPO computer was an SBC-11. The "11" came from the internal processor, which was essentially a DEC PDP-11, a mini-computer which was familiar to us at the time. "SBC" was apparently short for "shoe box computer", because that is what it looked like. I have a suspicion that this was a joke and it actually stood for "single board computer", as it does today. We implemented user programmability using a variant of the BASIC language, with some extensions to access capabilities of the testing machine.

Interprocessor communications
Using multiple CPUs (or cores) presents a variety of challenges. One is the division of labour, which was reasonably straightforward in this case. Another is communication between the processors ...

In designing the UPO, we considered a number of means by which the two CPUs might be connected. As they were separate boxes, serial and parallel connections were considered. But we were concerned about any possible compromise of the real-time performance of the console microprocessor. Also, we did not want the user to be faced with the UPO freezing while it waited for attention from the console. So, clearly a buffering mechanism was needed and shared memory seemed to be a good option.

A small memory board was designed. I have no idea of the hardware architecture, except that I seem to recall that the TI-9900 had priority over the SB-11, as it could not afford to be delayed by slow memory access. If I remember correctly, the board was 2K (words, probably).

Protocol
It was down to us to define a protocol for communication, so we aimed to produce something that was simple and reliable. We divided the memory into two halves; one was a buffer for communication from the UPO to the console and the other for the opposite direction. The first word of each buffer was for a command/status code, which was simply a non-zero value. We did not use interrupts. The receiving CPU just polled the first word when appropriate, awaiting a non-zero value. When a command was found, any data could be copied and the command word cleared to zero, indicating that the processing was complete. So, the UPO sending a command to the console might go through a sequence like this:
* Write data to the buffer [second word onwards].
* Write a command to the first word.
* Poll the word, waiting for it to become zero.

If it was expecting a response, it would then start monitoring the other buffer. Of course, there were other facilities to handle a situation where one CPU did not respond after a timeout period.

Nowadays, multi-core and multi-chip systems have a variety of interconnection technologies, but shared memory is still common. A number of standardised protocols have been developed over the years, including derivatives of TCP/IP. In recent years, the Multicore Association produced the Multicore Communications API (MCAPI), which is rapidly gaining broad acceptance in multi-core embedded system designs.

Challenges
When we hooked up the shared memory and started to send test messages between the processors, we hit a problem: they seemed to get scrambled. At first we assumed that there was a problem with the memory board, but it was checked by the hardware guys who pronounced it healthy.
Then we spotted a pattern: the bytes of each 16bit word were getting swapped. We thought that it was a wiring problem with the board, but studying the schematics and the board layout showed no error.

Of course, the reason for the problem was that the two CPUs were different architectures, from different manufacturers, each of whom had a different idea about which byte went where in the word. Now I would describe one as big-endian and the other as little-endian, but I did not have the vocabulary back then.

An adjustment to the board design could have made this problem go away. But of course it was too late for that. So we had to resort to the age-old method to rectify a problem found late in the design process: fix it in the software. I simply put a byte swap on one side of the interface and the other side was none the wiser.

How would I do it now?
In the end, we got the UPO working to our satisfaction and I think we even sold a few of them. It is interesting to consider how I might build such a system now, thirty-odd years later.

First off, I would design the console as a multi-core system. There would probably be one core that would do non-real-time work (user interface, data storage, networking, etc.) and maybe two more to do real-time work like data sampling and control of the servo-hydraulics.ÿThe UPO would just be an app on a standard Windows PC with a USB interface to the console. I have no insight into how the company builds testing systems now, but it would be interesting to compare notes with my 2013 counterparts.

Colin Walls has over thirty years experience in the electronics industry, largely dedicated to embedded software. A frequent presenter at conferences and seminars and author of numerous technical articles and two books on embedded software, Colin is an embedded software technologist with Mentor Embedded (the Mentor Graphics Embedded Software Division), and is based in the UK.