CROSS-REFERENCE TO RELATED APPLICATIONS

This patent application is a continuation-in-part of the following commonly-owned, U.S. patent application Ser. Nos.:

U.S. application Ser. No. 08/986,430, filed Dec. 7, 1997 now U.S. Pat. No. 6,065,077; and

U.S. application Ser. No. 09/163,294, filed Sep. 29, 1998 now U.S. Pat. No. 6,292,705;

all of which are incorporated by reference herein.

BACKGROUND

The system of FIG. 1 is a prototypical prior art symmetric multiprocessor (SMP) system 100. This traditional approach provides uniform access to memory 130 over a shared system bus 110. Each processor 120 has an associated cache and cache controller. The caches are individually managed according to a common cache coherency protocol to insure that all software is well behaved. The caches continually monitor (snoop) the shared system bus 110, watching for cache updates and other system transactions. Transactions are often decomposed into different component stages, controlled by different system bus signals, such that different stages of multiple transactions may be overlapped in time to permit greater throughput. Nevertheless, for each stage, subsequent transactions make sequential use of the shared bus. The serial availability of the bus insures that transactions are performed in a well-defined order. Without strong transaction ordering, cache coherency protocols fail and system and application software will not be well behaved.

A first problem with the above-described traditional SMP system is that the serial availability of the bus limits the scalability of the SMP system. As more processors are added, eventually system performance is limited by the saturation of the shared system bus.

What is needed is an SMP system architecture that provides greater scal ability by permitting concurrent use of multiple buses, while still providing a system serialization point to maintain strong transaction ordering and cache coherency. What is also needed is an SMP architecture that further provides increased transaction throughputs.

SUMMARY

A preferred embodiment of a symmetric multiprocessor system includes a switched fabric (switch matrix) for data transfers that provides multiple concurrent buses that enable greatly increased bandwidth between processors and shared memory. A high-speed point-to-point Channel couples command initiators and memory with the switch matrix and with I/O subsystems. Each end of a channel is connected to a Channel Interface Block (CIB). The CIB presents a logical interface to the Channel, providing a communication path to and from a CIB in another IC. CIB logic presents a similar interface between the CIB and the core-logic and between the CIB and the Channel transceivers. A channel transport protocol is is implemented in the CIB to reliably transfer data from one chip to another in the face of errors and limited buffering.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a drawing of a prior-art generic symmetric shared-memory multiprocessor system using a shared-bus.

FIG. 2 is a drawing of a preferred embodiment symmetric shared-memory multiprocessor system using a switched fabric data path architecture centered on a Flow-Control Unit (FCU).

FIG. 3 is a drawing of the switched fabric data path architecture of FIG. 2, further showing internal detail of an FCU having a Transaction Controller (TC), Transaction Bus (TB), and Transaction Status Bus (TSB) according to the present invention.

FIG. 4 is a drawing of a variation on the embodiment of FIG. 2, in which each CPU has its own CCU, and in which the channel interface and control is abstractly represented as being composed of a physical (PHY) link layer and a transport layer.

FIG. 5 is a timing diagram comparing the memory transaction performance of a system based on a flow control unit according to the present invention and a prior art shared-bus system.

FIG. 6 is another view of the embodiment of FIG. 4.

FIG. 7 is a drawing of a number of system embodiments according to the present invention. FIG. 7a illustrates a minimal configuration, 7b illustrates a 4-way configuration, 7c illustrates an 8-way high-performance configuration, and 7d illustrates a configuration for I/O intensive applications.

FIG. 8 is a drawing of a CPU having an integral CCU.

FIG. 9 illustrates a variation of the embodiment of FIG. 6 using the integrated CPU/CCU of FIG. 8.

FIG. 10 illustrates variations of the embodiments of FIG. 7 using the integrated CPU/CCU of FIG. 8.

FIG. 11 is a drawing of an 4-way embodiment of the present invention that includes coupling to an industry standard switching fabric for coupling CPU/Memory complexes with I/O devices.

FIG. 12 is a drawing of a 16-way embodiment of the present invention, in which multiple 4-way shared-bus systems are coupled via CCUs to the FCU, and which includes coupling to two instances of an industry standard switching fabric for coupling CPU/Memory complexes with I/O devices.

DETAILED DESCRIPTION

System Overview

FIG. 2 is a drawing of a preferred embodiment symmetric shared-memory multiprocessor system using a switched fabric data path architecture centered on a Flow-Control Unit (FCU) 220. In the illustrated embodiment, eight processors 120 are used and the configuration is referred herein as an 8P system.

The FCU (Flow Control Unit) 220 chip is the central core of the 8P system. The FCU internally implements a switched-fabric data path architecture. Point-to-Point (PP) interconnect 112, 113, and 114 and an associated protocol define dedicated communication channels for all FCU I/O. The terms Channels and PP-Channel are references to the FCU's PP I/O. The FCU provides Point-to-Point Channel interfaces to up to ten Bus Bridge Units (BBUs) 240 and/or CPU Channel Units (CCUS, also known as Chanel Interface Units or CIUs) and one to four Memory Control Units (MCUs) 230. Two of the ten Channels are fixed to connect to BBUs. The other eight Channels can connect to either BBUs or CCUs. In an illustrative embodiment the number of CCUs is eight. In one embodiment the CCUs are packaged as a pair referred herein as a Dual CPU Interface Unit (DCIU) 210. In the 8P system shown, the Dual CPU Interface Unit (DCIU) 210 interfaces two CPUs with the FCU. Throughout this description, a reference to a CCU is understood to describe the logical operation of each half of a DCIU 210 and a references to CCUs is understood to apply to equally to an implementation that uses either single CCUs or DCIUs 210. CCUs act as a protocol converter between the CPU bus protocol and the PP-Channel protocol.

The FCU 210 provides a high-bandwidth and low-latency connection among these components via a Data Switch, also referred herein as a Simultaneous Switched Matrix (SSM), or switched fabric data path. In addition to connecting all of these components, the FCU provides the cache coherency support for the connected BBUs and CCUs via a Transaction Controller and a set of cache-tags duplicating those of the attached CPUs' L2 caches. FIG. 5 is a timing diagram comparing the memory transaction performance of a system based on a flow control unit according to the present invention and a prior art shared-bus system.

In a preferred embodiment, the FCU provides support two dedicated BBU channels, four dedicated MCU channels, up to eight additional CCU or BBU channels, and PCI peer-to-peer bridging. The FCU contains a Transaction Controller (TC) with reflected L2 states. The TC supports up to 200M cache-coherent transactions/second, MOSEI and MESI protocols, and up to 39-bit addressing. The FCU contains the Simultaneous Switch Matrix (SSM) Dataflow Switch, which supports non-blocking data transfers.

In a preferred embodiment, the MCU supports flexible memory configurations, including one or two channels per MCU, up to 4 Gbytes per MCU (maximum of 16 Gbytes per system), with one or two memory banks per MCU, with one to four DIMMS per bank, of SDRAM, DDR-SDRAM, or RDRAM, and with non-interleaved or interleaved operation.

In a preferred embodiment, the BBU supports both 32 and 64 bit PCI bus configurations, including 32 bit/33 MHz, 32 bit/66 MHz, and 64 bit/66 MHz. The BBU is also 5 V tolerant and supports AGP.

All connections between components occur as a series of transactions. A transaction is a Channel Protocol request command and a corresponding Channel Protocol reply. For example, a processor, via a CCU, can perform a Read request that will be forwarded, via the FCU, to the MCU; the MCU will return a Read reply, via the FCU, back to the same processor. A Transaction Protocol Table (TPT) defines the system-wide behavior of every type of transaction and a Point-to-Point Channel Protocol defines the command format for transactions.

The FCU assumes that initiators have converted addresses from other formats to conform with the PP-Channel definitions. The FCU does do target detection. Specifically, the FCU determines the correspondence between addresses and specific targets via address mapping tables. Note that this mapping hardware (contained in the CFGIF and the TC) maps from Channel Protocol addresses to targets. The mapping generally does not change or permute addresses.

Summary of Key Components

Transaction Controller (TC) 400. The most critical coherency principle obeyed by the FCU is the concept of a single, system-serialization point. The system-serialization point is the funnel through which all transactions must pass. By guaranteeing that all transactions pass through the system-serialization point, a precise order of transactions can be defined. (And this in turn implies a precise order of tag state changes.) In the FCU, the system-serialization point is the Transaction Controller (TC). Coherency state is maintained by the duplicate set of processor L2 cache-tags stored in the TC.

The Transaction Controller (TC) acts as central system-serialization and cache coherence point, ensuring that all transactions in the system happen in a defined order, obeying defined rules. All requests, cacheable or not, pass through the Transaction Controller. The TC handles the cache coherency protocol using a duplicate set of L2 cache-tags for each CPU. It also controls address mapping inside the FCU, dispatching each transaction request to the appropriate target interface.

Transaction Bus (TB) 3104 and Transaction Status Bus (TSB) 3106. All request commands flow through the Transaction Bus. The Transaction Bus is designed to provide fair arbitration between all transaction sources (initiators) and the TC; it provides an inbound path to the TC, and distributes outbound status from the TC (via a Transaction Status Bus).

The Transaction Bus (TB) is the address/control highway in the FCU. It includes an arbiter and the Transaction Bus itself. The TB pipelines the address over two cycles. The extent of pipelining is intended to support operation of the FCU at 200 Mhz using contemporary fabrication technology at the time of filing of this disclosure.

Whereas the TB provides inputs to the Transaction Controller, the Transaction Status Bus delivers outputs from the Transaction Controller to each interface and/or target. The TSB outputs provide transaction confirmation, coherency state update information, etc. Note that while many signals on the TSB are common, the TC does drive unique status information (such as cache-state) to each interface. The Transaction Bus and Transaction Status Bus are discussed in detail later in this application.

Switched Fabric Data Path (Data Switch). The Data Switch is an implementation of a Simultaneous Switched Matrix (SSM) or switched fabric data path architecture. It provides for parallel routing of transaction data between multiple initiators and multiple targets. The Data Switch is designed to let multiple, simultaneous data transfers take place to/from initiators and from/to targets (destinations of transactions). Note that the Data Switch is packet based. Every transfer over the Data Switch starts with a Channel Protocol command (playing the role of a packet header) and is followed by zero or more data cycles (the packet payload). All reply commands (some with data) flow through the Data Switch. Both write requests and read replies will have data cycles. Other replies also use the Data Switch and will only send a command header (no payload).

IIF (Initiator InterFace) 3102. The IIF is the interface between the FCU and an initiator (a BBU or a CCU). The EIF transfers Channel Protocol commands to and from the initiator. The IIF must understand the cache coherency protocol and must be able to track all outstanding transactions. Note that the BBU/CCU can be both an initiator of commands and a target of commands (for CSR read/write if nothing else). Address and control buffering happen in the IIF; bulk data buffering is preferably done in the BBU/CCU (in order to save space in the FCU, which has ten copies of the IIF). The IIF needs configuration for CPU and I/O modes, and to handle differences between multiple types of processors that may be used in different system configurations.

Memory Interface (MIF) 3108. The Memory Interface (MIF) is the portal to the memory system, acting as the interface between the rest of the chipset and the MCU(s). The MIF is the interpreter/filter/parser that receives transaction status from the TB and TC, issues requests to the MCU, receives replies from the MCU, and forwards the replies to the initiator of the transaction via the Data Switch. It is a slave device in that it can never be an initiator on the TB. (The MIF is an initiator in another sense, in that it sources data to the Data Switch.) For higher performance, the MIF supports speculative reads. Speculative reads start the read process early using the data from the TB rather than waiting for the data on the TSB. There is one MIF (regardless of how many memory interfaces there are). The MIF contains the memory mapping logic that determines the relationship between addresses and MCUs (and memory ports). The memory mapping logic includes means to configure the MIF for various memory banking/interleaving schemes. The MWF also contains the GART (Graphics Address Remap Table). Addresses that hit in the GART region of memory will be mapped by the GART to the proper physical address.

Configuration Register Interface (CFGIF) 410. This is where all the FCU's Control and Status Registers (CSRs) logically reside. CFGIF is responsible for the reading and writing of all the CSRs in the FCU, as well as all of the diagnostic reads/writes (e.g., diagnostic accesses to the duplicate tag RAM).

Channel Interface Block (CIB). The CIBs are the transmit and receive interface for the Channel connections to and from the FCU. The FCU has 14 copies of the CIB, 10 for BBUs/CCUs, and 4 for MCUs. (The CIB is generic, but the logic on the core-side of the Channel is an IIF or the MIF.)

Embodiments overview. FIG. 3 is a drawing showing internal detail of the switched fabric data path architecture within the FCU of FIG. 2. A first key component of the FCU is the Transaction Controller (TC) 400. A second key component of the FCU is an address and control bus 3100, that is actually an abstraction representing a Transaction Bus (TB) 3104 and Transaction Status Bus (TSB) 3106. A third key component of the FCU is the Data Path Switch (also referred herein as the Data Switch, or the switched fabric data path). The Data Switch is composed of vertical buses 320, horizontal buses 340, node switches 380. The node switches selectively couple the vertical and horizontal buses under control of the Data Path Switch Controller 360 and control signals 370. Additional key components of the FCU include one or more Initiator Interfaces (IIFs) 3102; a Memory Interface (MIF) 3108; and Channel Interface Blocks (CIBs) 305 at the periphery of the various interfaces.

A number of alternate embodiments exist. FIG. 4 is a drawing of a variation on the embodiment of FIG. 2, in which each CPU has its own CCU. In this view the channel interface and control that make up the IIFs and CCUs are abstractly represented as being composed of a physical (PHY) link layer and a transport layer. FIG. 6 is another view of the embodiment of FIG. 4. FIG. 7 is a drawing of a number of application specific variations on the embodiment of FIG. 4. FIG. 7a illustrates a minimal configuration, 7b illustrates a 4-way configuration, 7c illustrates an 8-way high-performance configuration, and 7d illustrates a configuration for I/O intensive applications.

FIG. 8 is a drawing of a CPU having an integral CCU. FIG. 8 makes explicit a backside bus interface to an external cache (an L2 cache in the case illustrated). An IIF replaces the conventional CPU interface, such that the Channel is the frontside bus of the CPU of FIG. 8. The embodiments of FIGS. 9 and 10, are respective variations of the embodiments of FIGS. 6 and 7, with adaptation for the use of the integrated CPU/CCU of FIG. 8. The embodiments of FIGS. 9 and 10 offer system solutions with lower CPU pin counts, higher throughput, lower latency, hot plugable CPUs (if an OS supports it), and reduced PCB board layout complexity compared with non-integrated solutions.

FIG. 11 is a drawing of an 4-way embodiment of the present invention that includes coupling to an industry standard switching fabric for coupling CPU/Memory complexes with I/O devices. FIG. 12 is a drawing of a 16-way embodiment of the present invention, in which multiple 4-way shared-bus systems are coupled via CCUs to the FCU, and which includes coupling to two instances of an industry standard switching fabric for coupling CPU/Memory complexes with I/O devices.

Additional Descriptions

U.S. application Ser. No. 08/986,430, AN APPARATUS AND METHOD FOR A CACHE COHERENT SHARED MEMORY MULTIPROCESSING SYSTEM, filed Dec. 7, 1997, incorporated by reference above, provides additional detail of the overall operation of the systems of FIGS. 2 and 3. U.S. application Ser. No. 09/163,294, METHOD AND APPARATUS FOR ADDRESS TRANSFERS, SYSTEM SERIALIZATION, AND CENTRALIZED CACHE AND TRANSACTION CONTROL, IN A SYMMETRIC MULTIPROCESSOR SYSTEM, filed Sep. 29, 1998, incorporated by reference above, provides additional detail of particular transaction address bus embodiments. U.S. application Ser. No. 09/168,311, METHOD AN APPARATUS FOR EXTRACTING RECEIVED DIGITAL DATA FROM A FULL-DUPLEX POINT-TO-POINT SIGNALING CHANNEL USING SAMPLED DATA TECHNIQUES, filed Oct. 7, 1998, provides additional detail of particular transceiver embodiments, and is incorporated by reference herein. U.S. application Ser. No. 09/192,181, COMPACT MODULAR MULTI-PROCESSING ASSEMBLY WITH EFFICIENT PROCESSOR COOLING, filed Nov. 15, 1998, provides additional detail of particular multiprocessor assembly embodiments, and is incorporated by reference herein. To the extent to which any discrepancies exist between the descriptions in the above referenced applications and the instant application, the instant application should take priority in resolving such discrepancies as it provides architectural improvements and preferred embodiments not known at the time of the filing of the earlier applications.

Channel Overview and Terminology

The Channel of the present invention is a general-purpose, high-speed, point-to-point, full-duplex, bi-directional interconnect bus suitable for use in a broad range of applications including computer memory, I/O, CPU, switching/routing, or any other application where high bandwidth and low latency is required. The use of full-duplex, bi-directional signaling technology permits 1.6 GBps transfer rates simultaneously in each direction (3.2 GBps total throughput), using conventional system and board design technologies, and using only 25 signal pins. The CIB presents a simple, logical interface to a Channel, providing a communication path (a Channel) to and from a CIB in another IC.

A packet is the basic unit of transport over the Channel. In a preferred embodiment, conceptually a packet is a single 80-bit frame (information unit) exchanged between CIBs, the frame including:

64 bits of data exchanged core-to-core;

2 bits of control information exchanged core-to-core;

6 bits of control information exchanged CIB-to-CIB; and

8 bits of ECC exchanged CIB-to-CIB.

The 2 bits of control information from the core and the 6 bits of control information added by the CIB are encoded in to a single, 8-bit control field.

Each packet is sent across the CIB/transceiver interface in two phases of 40-bits each. Each packet is exchanged between CIB transceivers via a physical interconnect, which has 25 wires, including 16 wires dedicated to bi-directional core-to-core data and 4 wires dedicated to bi-directional core-to-core control information. Quad-Data-Rate (QDR) signaling is used to transmit and receive the 16-bit data and 4-bit control information four times, to effect the transfer of the full 80-bit packet within each 200 MHz clock.

The core-to-core data and control information is sent across the CIB/core interface along with additional signaling that is meaningful only to CIB/core interactions. The CIB-to-CIB signals are not sent across the CIB/core interface. In the context of the CIB/core interface, the term packet is used to refer to just the core-to-core data and control-information. Other than in conjunction with the CIB/core interface, the core-to-core control-information is not used within the core. In the context of the core functions, the term packet is used to refer to just the core-to-core data. The core-to-core data is the information unit primitive upon which transactions (discussed next) are built. Thus, the particular connotation for the term packet must be determined from the context in which it is used.

A transaction is a concept in the higher-level protocol sent over the Channel by the core logic. A transaction consists of a command header and zero or more data cycles. From the CIB's perspective, transactions are irrelevantthe CIB just guarantees to deliver packets reliably, and the core logic adds any interpretation that forms multiple packets in to transactions. Note that the core logic cannot expect packets within a transaction to arrive contiguouslydue to retries, there may be any number of IDLE cycles inserted at any point. In networking terms, the transceivers provide the physical layer (L1), the CIB logic provides the link-level layer (L2), and the core logic provides the transaction-level layer (L3). Each layer only has to know about the layers on either side, not about all layers in the stack.

A master IC is one which generates its own clock; a slave IC is one which receives a clock from the master (via the Channel).

Channel Interface Block

The Channel Interface Block (CIB) is a common block used in all the ICs in the 8P FCU-based chipset. The CIB presents a simple, logical interface to a Channel, providing a communication path (a Channel) to and from a CIB in another IC. From the IC core logic's point-of-view, the CIB acts as a pair of FIFOs with a non-zero (and variable) fall-through time which connect this IC to another. The CIB performs no interpretation of the data provided by the core logicthe higher-level (transaction-level) protocols used by the core logic are transparent to the CIB.

The CIB is divided into two parts: the CIB logic, and the transceivers. The CIB logic refers to the part of the CIB that forms the interface between the core logic in the IC and the Channel transceivers. In a preferred embodiment, the CIB logic and the transceivers are treated as a single top-level block in ICs.

CIB Signal Interface. The CIB interfaces to the core logic in the IC using a simple, generic interface that is frequency and bandwidth independent. This interface is common, no matter what CIB or transceiver design is used. The CIB has an internal interface to the transceivers; while this interface is not externally visible, it is important to define it here because much of the CIB logic's design is dependent upon this interface. The transceiver interface is discussed first because some signals just pass through to other parts of the IC. The CIB interface to the core logic is designed to be synchronous using the core logic clockfrom the core logic's perspective, the CIB is just another top-level logic block communicating via flip-flops on the core logic clock. The nature of the Channel is such that some asynchronous interfaces are necessary; these asynchronous interfaces are hidden inside the transceiversthe interface between the CIB logic and the transceivers is also synchronous on the core logic clock. This clean division allows issues with clock domains, clock skew, race conditions, etc., to be handled in a confined area (the transceivers) where the place and route variations can be minimized.

Nomenclature for signal tables. In the signal tables to follow, signals marked with an asterisk are pass through and are I/O pins of the CIB (and are not used in the CIB). Signals marked with a are signals that pass through (possibly with a name change) which are also used inside the CIB.

Signals in the CIB to Transceiver Interface (CIB-internal)

Clocks:

Signal nameDefinition* clkthe core clock (used to sampletransmitted data, to synchronizereceived data, etc.)* tr_clkclock output to clock block (slaveICs)

CIB to Transceivers Datapath:

Signal nameDefinitiontr_t_data_p079:40the first half of the packet to betransmittedtr_t_data_p239:0the second half of the packet to betransmitted* tr_test_modetest-mode enable; when active, allowstransceiver I/Os to be used as testI/Os* tr_t_test_data19:0test data to be drive in test mode* tr_pscan_data19:0parallel scan data for core scan mode* tr_t_entri-state enable for transmitted testOR parallel scan data (active meanstransmit; inactive means receive) power_on_resetpower-on reset from clock controlblock* pll_lockindication that the PLL is locked andis stable and can be used

Transceivers to CIB datapath:

Signal nameDefinitiontr_r_data79:0the received packet* tr_r_test_data19:0received test OR parallel scan data

CIB to transceivers control:

Signal nameDefinition tr_masteractive indicates this is a master IC tr_half_speedrun the Channel I/Os at half-speedtr_clk_en_inclock enable for transmitted packets;used to run at half speedtr_phase_adjust_grantgrant of request to adjust clock phasetr_force_recalibrationforce Channel to recalibrate itself

Transceivers to CIB control:

Signal nameDefinitiontr_clk_en_outclock enable for received packets;used to run at half speed tr_t dll_lockedstatus output: transmitter DLL lockachieved tr_r_dll_lockedstatus output: receiver DLL lockachieved tr_drvr_calibratedstatus output: drivers calibratedtr_phase_adjust_neededrequest to adjust clock phase

Transceiver JTAG interface:

Signal nameDefinition* tr_jtag_bs_enJTAG boundary scan enable* tr_jtag_pscan_enJTAG parallel scan enable* tr_jtag_tdiJTAG (boundary) scan in* tr_jtag_tdoJTAG (boundary) scan out* tr_jtag_clkJTAG scan clk* tr_jtag_captureJTAG clk enable for capture* tr_jtag_loadJTAG clk enable for load

Transceiver external I/Os:

Signal nameDefinition* PP_data15:016 full-duplex, bi-directional datawires* PP_ctl3:04 full-duplex, bi-directional controlwires* PP_clkoutforwarded output clock with trans-mitted data* PP_clkinforwarded input clock with receiveddata* PP_ref1:0high and low voltage reference lines* PP_calChannel calibration trace

The normal interface between the CIB and the transceivers is very simpletr_t_data_p0/tr_t_data_p2 and tr_r_data are the 80-bit datapath to and from the Channel. The clk signal is the core clock, and both tr_t_data_p0/tr_t_data_p2 and tr_r_data are synchronous with this clock. But, tr_t_data_p2 (providing the second half of the data packet to be transmitted) is (less than) one half clock cycle late This feature is used to allow ECC to be computed and sent on-the-fly after the data is already in progress.

The power_on_reset and pll_lock inputs to the transceivers control the wake-up sequencing that synchronizes the two ends of the Channel.

In test mode (when tr_t_en is asserted), the 20 external Channel I/O pins (PP_data and PP_ctl) are available for use as a test bus. (PP_ctl forms bits 19:16 of the test bus; PP_data forms bits 15:0.) This bus can be driven directly by tr_t_test_data or tr_pscan_data (as controlled by tr_jtag_pscan_en) and can be received by tr_r_test_data; the enable signal tr_t_en controls whether this bus drives or receives. Note that some transceiver designs require a clock in order to function as a receiver.

There are various control and status signals to and from the transceivers. The major mode controls to the transceivers are tr_master, tr_half_speed, and tr_clk_en_in. tr_master indicates that the IC is a master (a clock source), and tr_half_speed indicates that the transceivers are sending (and receiving) data at half of the core clock rate. tr_clk_en_in is used to control which core clock edge provides data to the transceivers. tr_clk_en_out from the transceivers is the corresponding output clock enable which controls when data from the transceivers is present. (When not running at half speed, both tr_clk_en_in and tr_clk_en_out should always be asserted.)

The Channel is ready (the CIB logic can begin to send data) when the following is true:

tr_t_dll_locked && tr_r_dll_locked && tr_drvr_calibrated

The transceivers may determine that the clock has drifted and a clock phase adjustment is needed; this is requested with the tr_phase_adjust_needed signal and granted with the tr_phase_adjust_grant signal. tr_phase_adjust_grant should be asserted for at least 32 cycles. The Channel should not be used by the CIB logic during the cycles when tr_phase_adjust_grant is asserted; the CIB logic should send IDLE cycles during this period. (Note that this definition of the Channel being ready refers to the transceivers being ready and is not necessarily the same as the CIB as a whole being ready to accept data from the core logic.)

If the CIB logic determines it has too high an error rate, it can request that the transceiver logic resynchronize the Channel with the tr_force_recalibration signal. The Channel is not usable during recalibration; the CIB logic should send IDLE cycles to the transceivers, and should assert backoff (ch_t_backoff) to the core logic, during this period. When recalibration starts, tr_drvr_calibrated is deasserted by the transceivers; the Channel is again ready for use when tr_drvr_calibrated is re-asserted. (tr_force_recalibration should be asserted until tr_drvr_calibrated is de-asserted; this forms a positive handshake with the transceivers.) The CIB logic should detect a timeout if the recalibration procedure takes longer than a predetermined interval.

The various JTAG scan modes, tr_jtag_bs_en and tr_jtag_pscan_en, over-ride all other modes and control scan operations of the transceiver circuitry.

Signals in the Core Logic to CIB Interface

The signals between the core logic and the CIB can be broken down into several major categories: clocks, reset, status, transmit, and receive.

Clock Signals to CIB

Signal nameDefinitionClkCore logic clock

The CIB interface to the core logic runs off of the core logic clock, nominally at 200 Mhz. From the core logic's point-of-view, the CIB appears to be purely synchronous on the core logic clock. Furthermore, from the CIB logic's point of view, the transceivers ALSO appear to be synchronous on the core logic clock. Any magic with phased clocks and higher frequencies is hidden inside of the transceivers.

Clock Control Block to CIB

Signal nameDefinitionpower_on_resetpower-on reset from clock controlblockpll_lockindication that the PLL is locked andis stable and can be used

The CIB logic ignores power_on_resetthis signal just gets passed through to the transceivers. The CIB logic uses pll_lock as its reset signal. Once pll_lock is achieved, the CIB comes out of reset and waits for ch_ready (Channel ready) to be asserted to begin operation. In an alternate embodiment hard and soft reset are communicated from master to slave via the CIB.

CIB Status to Core Logic Signals

Signal nameDefinitionch_readyindication that the Channel isoperationaltr_t_dll_lockedstatus output: transmitter DLL lockachievedtr_r_dll_lockedstatus output: receiver DLL lockachievedtr_drvr_calibratedstatus output: drivers calibrated

The CIB outputs a ch_ready (Channel Ready) signal to indicate that the Channel logic is operational (calibrated, in sync, etc.). The ch_ready signal should go active (high) some time after IC power is stable, and should stay active (unless there is a fatal error). If the signal does not go active, or ever goes inactive, then the Channel has been unable to calibrate itself and/or get in sync with the other end. (The assumption is that the ch_ready, tr_t_dll_locked, tr_r_dll_locked, and tr_drvr_calibrated signals will be passed through to the core logic and latched in a CSR which software can read in order to determine why a particular Channel failed to become ready. The core logic should also provide a timeout which will give up on a Channel if it does not become ready in a predetermined interval after reset.).

The ch_ready output of the CIB is a function of the transceivers being ready, and of the CIB logic being ready. The transceiver ready status is initially derived from status outputs of the transceivers:

trcvr_readytr_t_dll_locked && tr_r_dll_locked && tr_drvr_calibrated;

Note that tr_drvr_calibrated may become deasserted if the Channel is recalibrated. The ch_ready output of the CIB, on the other hand, must stay active once asserted; the CIB logic must ensure that ch_ready is held active during recalibration. (The ch_t_backoff signal should be asserted to the core logic during recalibration so that the core logic will not attempt to use the Channel.)

In addition to taking in to account trcvr_ready, the ch_ready output of the CIB must also account for any initialization protocols the CIB itself uses (such as the TC_RESET protocol); ch_ready should not be asserted until after the CIB is able to accept data from the core logic.

Configuration Signals to CIB from Core Logic

Signal nameDefinitionch_half_speedrun the Channel at half-speedch_masterif active, indicates that this is a master(as opposed to a slave) ICch_power_downnotification that the Channel shouldshut offch_retry_count1:0encoded indication of how manytimes to retry

The core logic sends a ch_power_down signal to the CIB to indicate that the Channel is not connected (or should be powered down for some other reason). ch_power_down can be used for configuration purposes (for Channels that are unused), or for dynamic (but relatively slow) power management. The ch_power_down signal should not be used in a dynamic fashion; the Channel can take msec. to become ready once it has been powered down. Note that when ch_power_down is asserted, any active communication on the Channel will be lost. When ch_power_down is asserted, the CIB will deassert ch_ready and will assert ch_t_backoff.

The ch_retry_count1:0 signal is a two-bit input to the CIB which controls how hard the CIB should attempt to recover in the event of an error; specifically it encodes how many retries should be attempted prior to attempting recalibration of the Channel. The encoding is:

ch_retry_count1:0definition00recalibrate every retry01recalibrate every 4 retries10recalibrate every 16 retries11never recalibrate

This signal should be driven from a CSR in the core logic. (Ideally, these signals will be set the same way on both sides of the Channel.)

The retry count controls the total number of retries (TC_SYNCs) generated before Channel recalibration is attempted. Channel recalibration will always be tried sixteen times (with ch_retry_count controlling the number of TC_SYNC attempts between each recalibration). If after all these retries/recalibrations, the Channel is still not functional, a fatal error (ch_r_fatal) is declared. Note that ch_retry_count3 will never recalibrate and will never signal a fatal