High-speed Serial Buses in Embedded Systems

By Feng Zhang

This book describes the most frequently used high-speed serial buses in embedded systems, especially those used by FPGAs. These buses employ SerDes, JESD204, SRIO, PCIE, Aurora and SATA protocols for chip-to-chip and board-to-board communication, and CPCIE, VPX, FC and Infiniband protocols for inter-chassis communication. For each type, the book provides the bus history and version info, while also assessing its advantages and limitations. Furthermore, it offers a detailed guide to implementing these buses in FPGA design, from the physical layer and link synchronization to the frame format and application command. Given its scope, the book offers a valuable resource for researchers, R&D engineers and graduate students in computer science or electronics who wish to learn the protocol principles, structures and applications of high-speed serial buses.

• Language: English
• Publisher: Springer
• Release date: Jan 3, 2020
• ISBN: 9789811518683

Feng Zhang

Feng Zhang got his Ph.D. degree in Industrial Engineering from the State University of New York at Buffalo (UB), US, in September of 2018. From Nov. 2018 to Mar. 2019, he worked as a short term Postdoctoral researcher in the department of Mechanical and Aerospace Engineering with Dr. Deborah Chung. In May of 2019, he joined Nanjing Normal University, and is a member of Jiangsu Key Laboratory of 3D Printing Equipment and Manufacturing. He obtained his Bachelor degree from Wuhan University of technology in 2005. After several years’ experience in industry, He attended UB and obtained his master degree in 2014. His current research interests include developing novel 3D printing methods for fabrication of various porous objects. Beginning February of 2019, Dr. Zhang’s Ph.D. topic is supported by the National Science Foundation’s most prestigious award – CAREER. Dr. Zhang has served as the reviewer for many technique journals, including Rapid Prototyping, Additive Manufacturing, Journal of manufacturing process, Journal of manufacturing science and engineering. His work has been published in many high impact journal such as Nano Energy, Small, ACS Nano, and Nanotechnology.

Book preview

© Springer Nature Singapore Pte Ltd. 2020

F. ZhangHigh-speed Serial Buses in Embedded Systemshttps://doi.org/10.1007/978-981-15-1868-3_1

1. History and Development of Bus

Feng Zhang¹  

(1)

The 10th Research Institute of CETC, Chengdu, China

Feng Zhang

Email: [e-mail]

1.1 Appearance and Definition of Bus

The word bus originates from the computer whose history can be traced back to 1946. The military designed the first computer for the calculation of ballistic trajectory and gunnery and put the weapon-aimed equipment into operation in the peace-loved University of Pennsylvania on February 15, 1946. They call it ENIAC (Electronic Numerical Integrator And Computer) [1]. ENIAC consisted of 17,468 vacuum tubes, consuming a power of 174 kW, covering an area of 170 m², weighing 30 t, performing 5,000 addition operations and 400 multiplication operations per second, and costing USD 1 million.

What does the operation speed mean? Taking π as an example, it took Tsu Chung-Chi (a distinguished Chinese mathematician, born in A.D.429) 15 years to calculate π to 7 decimal places, and William Shanks (a British mathematician, born 1300 years later than Tsu Chung-Chi) a whole life to calculate it to 707 decimal places. Shanks created and held the precision record in the following 73 years until the appearance of the computer ENIAC, which took only 40 s to achieve the same precision and found that the 528th place was wrong in Shanks’ calculation [2].

So ENIAC brought the whole world a new world—the Information Revolution came and ENIAC was just the very beginning, marking the advent of computer era.

The 1st generation computers were usually called vacuum tube computers, and gave birth to the information technology. Vacuum tube computers had played an inharmonious role in the social community, since they consumed much of the precious power energy belonging to the inhabitants nearby. In 1956, they were replaced by the 2nd generation computers, which were made up of transistors, had higher operation speeds and less volumes, and could be accommodated in one room compared with a plant building for the older generation. Despite of these advantages, they were only applied in the military and government due to high cost of transistors until 1963. In parallel with the development and evolution of the 2nd generation computers during the 1950s, substantial efforts were put into the 3rd generation research. In 1958, three transistors were integrated into a single silicon chip, giving birth to the integrated circuit (IC) industry. Then the IC was firstly used for computer by IBM with its product named IBM360. With the IC used, the space required was again reduced from the size of a room to a desk. The computers produced by 1971 were called the 3rd generation computers, featuring medium- and small-scale ICs, while those produced after 1971 were called the 4th generation computers, characterized by mega-scale ICs. On August 12, 1981, IBM released a new computer called 5150 series, then the Personal Computer era came and computer became an everyday commodity [3].

The history of the four generations of computers is shown in Fig. 1.1.

../images/457493_1_En_1_Chapter/457493_1_En_1_Fig1_HTML.png

Fig. 1.1

Four generations of computers. a 1st generation computers (vacuum tubes) [3], b 2nd generation computers (transistors), c 3rd generation computers (medium and small-scale ICs) [3], d 4th generation computers (mega-scale ICs) [4]

The 1st and 2nd generation computers were mainly used for the professional and complex calculations. Apart from the need of huge amounts of vacuum tubes or transistors, these computers also required many welders, since the data transmission in these computers was accomplished by wires. With the IC’s rapid development later, the data transmission between IC chips on a PCB became a capability and served as a substitute for wires. IBM realized that it was a must to establish a standard that all the manufactories of CPU and other peripherals should stand by, because the data transmission rate between CPU and peripheral slaves became rapider with smaller but powerful ASICs. In 1981, IBM addressed its representative computer PC/XT, specifying a data transmission mechanism between CPU and its peripheral slaves. The mechanism was called PC/XT bus or PC bus. The major characteristic of PC/XT bus was that: data width 8 bit, address width 20 bit, with CPU as the only master and all the other peripherals as the slaves, including the DMA and coprocessor [5].

The PC/XT bus is an open architecture with several slots in the motherboard that can be used for connecting the peripheral I/O to CPU. Because of the cheapness, robustness, simpleness and a good compatibility of PC/XT bus, many manufactories became a part member the PC/XT family. So the IBM bus standard PC/XT played the No. 1 role in the computer market, and became the industrial standard in fact.

Based on the bus conception of IBM, we can make the definition of bus as follows:

In computer architecture, a bus is a communication system that transfers data between components inside a computer, or between computers. This expression covers all related hardware components (wire, optical fiber, etc.) and software, including communication protocols (www.​wikipedia.​com).

A typical computer bus structure is shown in Fig. 1.2, involving FSB, ISA, AGP, PCI, SCSI, ATA, USB and other buses to connect the CPU, chipset, South Bridge, North Bridge, hard disks, video card, sound card and other functional devices together.

../images/457493_1_En_1_Chapter/457493_1_En_1_Fig2_HTML.png

Fig. 1.2

A typical bus structure in PC

The introduction of bus into the computer has brought about obvious advantages:

simplifying system structure, benefiting system expansion.

easier for upgrading. Upgrading one module generally does not affect the other functions in the system slots.

easy for diagnosis and repairment.

1.2 Progress of Bus in PC

It is the development of ultra large scale integrated circuit computers that contributed to the emergence and the evolution of the bus. The bus has always been associated with the development of PC dependently in the Twentieth Century. The story starts from the ISA bus.

1.2.1 ISA

Three years later after the PC/XT bus models are launched, IBM launched the 16 PC named PC/AT (Personal Computer Advanced Technology) in 1984 based on the latest Intel 80286 processor, and introduced the PC/AT bus based on the PC/XT bus concept to meet the 8/16-bit data bus requirements. Also at this time, the PC industry had begun to take shape, and IBM allowed third-party vendors to supply compatible products, so various manufacturers developed IBM PC-compatible peripherals and PC/AT bus became the default standard. Based on the IBM PC/AT bus specification, an industry standard was gradually established, named the Industrial Standard Architecture (ISA) [6, 11]. The IEEE formally drafted the ISA bus standard in 1987.

The ISA-based computer architecture is shown in Fig. 1.3.

../images/457493_1_En_1_Chapter/457493_1_En_1_Fig3_HTML.png

Fig. 1.3

ISA-based PC structure

ISA bus had been running through the 286 and 386SX era, with the maximum transfer rate of 16 MBps, which is more than enough for the 16-bit X86 system. ISA did not encounter any trouble until the 32-bit 386DX processor appeared. ISA bus with 16 bit data width became a problem. The over-low transfer rate, high CPU occupancy rate and hardware interrupt resource occupancy severely constrained the performance of processors. In 1988, Compaq, HP and other nine manufacturers expand ISA to 32 bit, which was the famous EISA (Extended ISA) bus. The EISA bus operates at a frequency of up to 8 MHz and is fully compatible with the 8/16 bit ISA bus, doubling the bandwidth to 32 MBps due to the 32 bit bus, but EISA did not repeat the successes of ISA. Because of high cost, before becoming a standard bus, it was replaced by another new generation of bus—PCI (Peripheral Component Interconnect Local) bus [7] in the early 1990s. However the EISA bus did not disappear fast, it had been coexisting with the PCI bus in the computer for a long time. An ISA/PCI bus coexistence motherboard is shown in Fig. 1.4. It was until 2000, i.e. 19 years after the proposal of ISA standard, that EISA officially withdrew.

../images/457493_1_En_1_Chapter/457493_1_En_1_Fig4_HTML.png

Fig. 1.4

ISA/PCI bus coexistence motherboard

1.2.2 PCI/PCI-X

CPU had gotten a rapid development since 1992, and then Intel introduced the 80486 processor. At that time, the operation rate of CPU had been significantly higher than the bus. ISA and EISA obviously became a bottleneck in the transmission between CPU and peripherals. Because hard disks, graphics cards and other peripherals could only send and receive data through a slow and narrow ISA/EISA channel, the low speed of the bus was not synchronized with the high speed CPU, and the whole performance was seriously affected. For solving this problem, Intel launched a new technology in 1993—PCI (Peripheral Component Interconnect Local Bus), which was quickly recognized and became a new industry standard.

In terms of the structure of PCI local bus, one level of bus is added between the ISA bus and CPU bus to coordinate data transmission and provide bus interface. With the new bus structure, some high-speed peripherals, such as network adapter cards and disk controllers, can be unloaded from the ISA bus and then connected to the CPU bus directly through the PCI bus to match the transmission rate friendly, meanwhile the low-speed devices achieve interaction with the PCI bus through the traditional ISA bus controller.

Figure 1.5 shows the internal structure of the PCI—based computer.

../images/457493_1_En_1_Chapter/457493_1_En_1_Fig5_HTML.png

Fig. 1.5

PCI-based PC structure

The initial version of the PCI bus, operating at 33 MHz frequency, with data width of 32 bit and transmission bandwidth up to 133 MBps, well meets the transmission rate requirements in computer system, providing a performance much higher than the ISA and EISA bus. The PCI local bus allows up to 10 PCI add-on cards to be installed on a single computer, and allows the ISA and EISA expansion controller cards to be installed simultaneously, to achieve a better system bus synchronization. This approach not only meets the requirements of peripherals on interface and bus, but also provides a considerable flexibility, which contributes to today’s design philosophy. Subsequent PCI versions can use 32-bit or 64-bit (depending on the device) to exchange data with the system CPU.

The PCI bus has some unique advantages over the previous bus.

(1)

High-performance multimedia technology. High-performance graphics, video, and network require a high operation rate processor. PCI local bus provides a wide bandwidth, allowing these applications to operate quite smoothly.

(2)

Wide compatibility. Unlike any other bus standard, any device that follows the PCI bus protocol can be used in PCI systems, regardless of the type of processor.

Therefore, the PCI local bus provides a wider bandwidth and faster access to the CPU, effectively overcoming the bottleneck of data transmission. For a long period of time, the PCI local bus interface had been the preferred interface for many adapters, such as network adapters, built-in MODEM cards, and sound adapters. Most motherboards have PCI slots.

One year after of PCI specification release, Intel immediately put forward the 64-bit/33 MHz PCI bus version with 266 MBps bandwidth, primarily used for enterprise servers and workstations. In these areas, the requirements for bus performance become higher day by day. The bandwidth of 64-bit/33 MHz specification of the PCI soon became insufficient again, which was increased to 66 MHz later. With the X86 server markets continuing to expand, the 64 bit/66 MHz specification of the PCI bus undoubtedly became the standard for the servers/workstations containing SCSI (Small Computer System Interface) card, RAID (Redundant Array of Independent Disks) card, Gigabit Ethernet and other devices. Even today the 64bit/66 MHz PCI specification is widely used.

In 1999, PCI-SIG (Peripheral Component Interconnect Special Interest Group) launched PCIX (PCI eXtensions) [8] bus based on the inheritance and development of PCI bus. PCIX bus operation adopts separate transaction processing, eliminating the waiting state and greatly improving the utilization ratio of the bus. The bandwidth of PCIX was up to 1.066 GBps (133 MHz/64 bit). PCIX is the product during the transition from shared bus architecture to switched bus architecture. PCIX bus architecture has been used to improve the I/O throughput in high performance server field. A PCIX bus interface on the motherboard is shown in Fig. 1.6.

../images/457493_1_En_1_Chapter/457493_1_En_1_Fig6_HTML.png

Fig. 1.6

PCIX becomes server’s major standard

PCIX bus frequency was then increased to 133 MHz. In the year of 2002, PCI-SIG launched PCIX 2.0 (PCIX 266) version, supporting DDR (Double Date Rate) mode, in which data was transmitted at both the rising and falling edges of clock, making PCIX theoretic bandwidth increased to 2.1 GBps.

The development process of ISA/PCI/PCIX bus is shown in Table 1.1.

Table 1.1

Comparison among ISA/PCI/PCIX

Encouraged by the success of PCIX 2.0, the PCI-SIG organization announced in November 2002 that it would develop the PCIX 3.0 standard, which is called PCIX 1066. It is reported that the standard would clock at a frequency of 1066 MHz, sharing bandwidth of up to 8.4 GBps, with each peripheral device having at least 1.06 GBps bandwidth. But it was a pity that the plan came to an end before execution, and there was no further plan for PCIX, since synchronization for the 64-bit parallel data was extremely difficult at the frequency of 1066 MHz. Once again it proved that improvement alone was not enough; sometimes there must be a revolutionary change for solving the problem, just like the replacement of ISA bus by PCI bus and then the replacement of PCI by PCIE.

PCI/PCIX/PCIE bus has been coexisting for a long time, as shown in Fig. 1.7.

../images/457493_1_En_1_Chapter/457493_1_En_1_Fig7_HTML.png

Fig. 1.7

PCI/PCIX/PCIE coexistence

1.2.3 PCIE

With new technologies and equipment emerging one after another, especially in the increasingly wide field of games and multimedia applications, PCI operating frequency and bandwidth were unable to meet the demand. In addition, PCI also had some other problems, such as IRQ sharing confliction and only supporting a limited number of devices. After 10 years of tinkering, PCI bus was unable to meet the CPU’s performance requirements, and must be replaced by a new generation of bus with higher bandwidth, wider adaptability, and greater development potential. This was the mission of PCI Express bus.

In the spring of 2001, at the Intel Developer Forum (Intel Developer Forum), Intel proposed the concept of a 3GIO (Third Generation I/O Architecture) bus, which achieved high performance in a serial and high-frequency operation. The 3GIO’s design was forward-looking, aimed to replace the PCI bus and meet the requirement of PC for the next decade. The 3GIO plan got extensive responses, and later was submitted to the PCI-SIG organization by Intel. PCI-SIG organization renamed the 3GIO to PCI Express (Express means high speed or fast) [9], and launched it in April 2002. Indeed its performance is amazing; just X1 mode can achieve 2.5 Gbps bandwidth. PCI Express also supports X2, X4, X8, and X16 modes, with a bandwidth of up to 80 Gbps in X16 mode. More importantly, PCI Express has been improved in basic architecture, completely eliminating the parallel shared structure and adopting the industry’s popular peer-to-peer serial connection. Compared with the shared parallel architecture of PCI and earlier buses, each device has its own dedicated connection in PCI Express structure. This mechanism makes it unnecessary to request bandwidth from the entire system, and increases the data transfer rate to a very high frequency which PCI cannot achieve. So a new era began. The type of PCIE connector is shown in Fig. 1.8.

../images/457493_1_En_1_Chapter/457493_1_En_1_Fig8_HTML.png

Fig. 1.8

Typical PCIE module cards and system slots

Following the release of PCI Express version 1.0, practical development started. In June 2004, Intel launched its chipset i915/925x fully based on PCI Express, and nVIDIA and ATI, two graphics card manufacturers, launched their own PCI ExpressX16 interface graphics card at the earliest time. PCI Express era officially arrived. In 2005, PCI Express version 2.0 was released, with the unidirectional bandwidth of X1 link reaching 5 Gbps and the total bandwidth of X16 reaching 160 Gbps. PCI Express version 3.0 standard was released in 2010, with the unidirectional bandwidth reaching 8 Gbps. On January 9, 2012, the world’s first PCI-E 3 graphics card Radeon HD 7970 came out. Until now, the replacement of PCI by PCIE is in full swing.

The PCIE bus’s roadmap is shown in Table 1.2.

Table 1.2

Roadmap of PCIE [9]

The three revolutions of the PC bus are shown in Fig. 1.9.

../images/457493_1_En_1_Chapter/457493_1_En_1_Fig9_HTML.png

Fig. 1.9

Three Revolutions of the PC Bus

Refer to Chap. 7 for more details on the PCI/PCIX and PCIE bus.

1.2.4 ATA/SATA—Used for Storage

The abovementioned ISA, PCI and PCIE buses are all system buses or local buses used in PC, for connecting CPU and various peripherals based on a common bus structure, while the graphics cards, disks and other high-speed peripherals need to have a dedicated bus channel. Here is a typical dedicated bus in PC just for storage purpose.

With PC widely used in thousands of households, Compaq, Western Digital and several other companies saw great opportunities, and in order to share a piece of cake they jointly developed a storage interface, known as Advance Technology Attachment technology later for hard disks. The Advance Technology Attachment technology began to be applied to the desktop PC in the early 1990s, known as the famous ATA. In 1986, Compaq’s 386 PC was equipped with a hard disk based on the Western Data’s controller chip, which was the earliest record of ATA hard drives, supporting data rates up to 8.3 MBps [12].

ATA specification is a protocol family, with ATA-1 as the starting point. After years of development, ATA-5 was launched. The ATA-5 made use of two rows of 40-pin cables to connect the motherboard and hard disk interface, as shown in Fig. 1.10. When the ATA-6 protocol came, the traditional parallel interface and cables could not satisfy the higher data transmission rate as the signal integration became worse. So the traditional parallel transmission method was a technical bottleneck, and there was no hope in making a breakthrough with the later ATA-7 protocol. Many manufacturers gave up on the ATA standard development except that only Maxtor launched a series of ATA-7 standard in 2004, supporting data rates up to 133 MBps. The ATA protocol went on the same way as ISA/PCI which was replaced by PCI Express, ATA was replaced by Serial ATA [13, 14].

../images/457493_1_En_1_Chapter/457493_1_En_1_Fig10_HTML.png

Fig. 1.10

ATA interface

The transfer rates of ATA family are shown in Table 1.3.

Table 1.3

Roadmap of ATA standard

In February 2000, Intel for the first time proposed the SATA technology concept at the Intel Developer Forum (IDF) and set up the Serial ATA Working Group. In August 2001, Seagate announced the Serial ATA version 1.0 at the IDF Fall 2001 conference, and the Serial ATA (SATA) specification was formally established. By changing the data transmission mechanism from parallel to serial (the principle like PCIE), the crosstalk and synchronization problems caused by parallel transmission were avoided. SATA 1.0 as the starting point supported the rate of 150 MBps, faster than 133 MBps of ATA-7, SATA 2.0 supported 3.0 Gbps, SATA 3.0 supported 6.0 Gbps, and the latest SATA 3.2 supported 16 GBps. Figure 1.11 shows the SATA protocol development.

../images/457493_1_En_1_Chapter/457493_1_En_1_Fig11_HTML.png

Fig. 1.11

Roadmap of SATA specification [15]

In addition to solving the crosstalk and synchronization problems in the ATA specifications, SATA has fewer pins and better error correction capability and supports hot plug, as shown in Fig. 1.12. So far, SATA has successfully replaced ATA, and quickly become the primary mechanism in storage market.

../images/457493_1_En_1_Chapter/457493_1_En_1_Fig12_HTML.png

Fig. 1.12

SATA interface

ATA/SATA is primarily used in hard disk-based data storage field, mainly for individual users, while in the field of servers, SCSI/SAS standard is dominant.

1.3 Progress of Bus in Embedded System

1.3.1 The Emergence of Embedded Systems

Human desire is always endless, and the desire in turn promotes the development of technology and even the society. In view of the great success of bus in standardization in PC, the enterprisers consider two issues from the very beginning of the birth of the computer:

Can the computer substitute for human in some industrial control applications. So the embedded system appears.

Is it possible to use some bus specifications for embedded system only by referring to the bus protocol in PC. So the dedicated embedded system bus appears.

With the rapid development of computers, their high-speed numerical computing ability and high intelligent level arouse the interest of the control professionals. Would the computer be an alternative of human in some specific field to achieve the intelligent control of the specific system? For example, the computer can be installed on a large ship as an autopilot or turbine condition monitoring system by electrical alteration, mechanical reinforcement, and configuration of a variety of peripheral interface circuits. The properties of smallness, cheapness, and high-reliability make computer get out of the engine room quickly and embedded in a specific but different system, to achieve intelligent control since the birth of the embedded system. So, the embedded system should be defined as a dedicated computer system embedded in a specific system.

At the beginning, people tried to modify the general computer system to achieve embedded applications in large-scale equipment reluctantly. However restricted by volume, environmental requirements and other factors, the general computer system cannot be embedded in for a large number of systems (home appliances, instrumentation, industrial unit, etc.). The general computer system must be cut to meet the minimum requirements of the object with the proper configuration of software/hardware, to meet the environmental requirements of the target system, such as physical environment (small), electrical environment (reliable) and cost (inexpensive), and to meet the reliability requirement by control. Therefore, the embedded computer systems must be developed independently, and the applications in embedded system become an important branch of modern computers, including the rapid development of ASICs.

CPU in the computer corresponds to the micro-controller in embedded system and the first micro-controller was the 8048 microcontroller launched in 1976 by Intel. Motorola launched the 68HC05 at the same time. These early microcontrollers contained 256Byte RAM, 4KByte ROM, four 8-bit parallel ports, a full-duplex serial port and two 16-bit timer. Later in the early 1980s, Intel further improved the 8048, enhancing it to a series named 8051 [16], a milestone in the history of the microcontroller. So far, the 51 series microcomputers are still extensively used in a variety of products.

In 1990s, embedded system technology got a broader space for development. The famous American futurist Negroponte predicted in January 1999 that 4 to 5 years later, the embedded intelligent (computer) equipment will be the greatest invention after PC and the internet.

After 2005, with the development of information, intelligence and network, the application of embedded systems has been in full swing, and has now become the development trend of communication and consumer products. In the field of communications, digital technology is taking the place of analog technology. In the field of radio and television, the United States has begun to shift from analog TV to digital TV, and the Europe’s DVB (digital television broadcasting) technology has been promoted in most countries around the world. Digital audio broadcasting (DAB) has also entered the commercialization stage. It is because of the rapid development of embedded systems that smart phones affect and change the life in all aspects. All of the above products cannot be separated from the embedded