119 where connections to the first group of data/strobe signal lines 130-1 are disposed, and the i^thposition is between the i^thgroup of memory devices and an edge portion of the module board 119 where connections (not shown) to the i^thgroup of data/strobe signal lines 130-i are disposed. In one embodiment, the data buffers 118 are distributed along the edge 122 of the memory module 110. In one embodiment, each data buffer 118 is in a separate integrated circuit device package either by itself or together with at least some of the respective group of memory devices. In one embodiment, the module data/strobe signal lines 210, the module C/A signal lines 220, and the data buffer control signal lines 230 include signal traces formed on and/or in the module board 119.

As an option, memory module 110 may further include a serial-presence detect (SPD) device 240, which may include electrically erasable programmable read-only memory (EEPROM) for storing data that characterize various attributes of the memory module 110. Examples of such data include a number of row addresses, a number of column addresses, a data width of the memory devices, a number of ranks on the memory module 110, a memory density per rank, a number of memory device on the memory module 110, and a memory density per memory device, etc. A basic input/output system (BIOS) of system 100 can be informed of these attributes of the memory module 110 by reading from the SPD 240 and can use such data to configure the MCH 101 properly for maximum reliability and performance.

In certain embodiments, the SPD 240 and/or the control circuit 116 store module configuration information, such as: memory space translation code, memory address mapping function code, input and output signals timing control information for the control circuit 116, electrical and logical level control information for input and output signals for the control circuit 116, the data buffers 118, etc. In certain embodiments, the SPD 240 contains a system view of the module 110 which can be different from an actual physical construction of the module 110. For example, the SPD 240 stores at least one memory operation parameter that is different from a corresponding memory operation parameter in a system memory controller setting. The SPD 240 may also store at least on data buffer operation parameter that is different from a corresponding parameter in the system memory controller setting.

Thus, in certain embodiment, in the memory module 110, C/A signals representing a memory command are received and buffered by the module control circuit 116, so that the MCH sees only the module control circuit 116 as far as the C/A signals are concerned. Write data and strobe signals from the controller are received and buffered by the data buffers 118 before being transmitted to the memory devices 112 by the data buffers 118. On the other hand, read data and strobe signals from the memory devices are received and buffered by the data buffers before being transmitted to the MCH via the system data/strobe signal lines 130. Thus, MCH 101 does not directly operate or control the memory devices 112. As far as data/strobe signals are concerned, the MCH 101 mainly sees the data buffers 118, and the system 100 depends on the data buffers 118 to properly time the transmission of the data and strobe signals between the MCH 101 and the memory devices 112.

In certain embodiments, the memory module 110 is a dual in-line memory module (DIMM) and the memory devices are double data rate (DDR) dynamic random access memory devices (DRAM). In certain embodiments, the control circuit 116 includes a DDR register, and logic for memory space translation between a system memory domain and a module level physical memory domain. Such translation may produce address mapping, proper interface timing for the control signals to the module level physical memory domain, and a proper interface electrical and logical level for the control signals to the module level physical memory domain.

As shown in FIG. 2C, in certain embodiments, the control circuit 116 transmits registered C/A and clock signals to the memory devices 112 in a fly-by configuration, and transmits data buffer control signals and registered clock signal to the data buffers 118 in a fly-by configuration. In a fly-by signal routing configuration, signals are routed such that they connect to each a row of devices in a daisy chain topology. Thus, signals routed in a fly-by configuration would arrive skewed in time at different devices. As the speed of memory operations increase, issues can also arise with respect to signal alignment for input, output delay variation due to process, voltage and temperature (PVT) variations, lack of synchronization with system memory controller interface, and phase drift accumulation during operation, etc. Electrical interface calibration drift during operation caused by charge build up and timing interface calibration drift during operation caused by environment change can also create issues.

For example, to offset the skew in the registered C/A signals, certain memory devices (e.g., the DDR3 SDRAM) have a training mode that permits the MCH to effectively zero-out the skew and re-align the data in time by adding delay to the data paths. In certain embodiments, however, load reduction mechanism in the data buffers 118 would provide a single data bus interface for the memory devices, which hides the data interfaces between the memory devices 112 and the data buffers 118 from the system memory controller 101. Thus, a long sequence of interface timing training may be required because the system memory controller 101 has very limited control over the interface between the memory devices 112 and the data buffers 118. Furthermore, interface signal alignment-drift after the initial training would not be easily detected by the system memory controller 101, which may cause silent system failure.

Moreover, clock skew amongst the memory devices 112 and the data buffers 118 can cause synchronization issues. As the speed of memory operation increase, data period can become so short that variation of signal propagation time through I/Os becomes a very significant portion of a data period. As a result, the clock skew issues cannot simply be addressed by pipelining the data paths.

To address at least some of the above issues, in certain embodiments, as shown in FIG. 2D, the control circuit 116 transmits registered C/A signals to the memory devices 112 in a fly-by arrangement, and transmits the data buffer control signals and the registered clock signal to the data buffers 118, also in a fly-by arrangement. The memory devices 112 do not receive the registered clock signal from the control circuit 116. Instead, each data buffer 118 regenerates the clock that is used by a respective set of memory devices 112. Each Data buffer 118 is thus responsible for providing a correct data timing interface between the respective set of memory devices 112 and the system memory controller 101. Each data buffer 118 and its associated respective set of memory devices together form a group of locally synchronized components. The date buffer 118 in each group of locally synchronized components is also responsible for providing the correct control signal timing between the control circuit 116 and the group of locally synchronized components.

Thus, the memory module 110 in FIG. 2D allows a locally synchronized operation for each group of locally synchronized components, which can correspond to a nibble or a byte of a DDR data bus between the memory module 110 and the system memory controller 101. Also, signal interface between each data buffer 118 and the respective set of memory devices 112 can be synchronized. In one embodiment, each data buffer 118 has a set of configurable operations, including, for example: programmable phase relationship between the clock it receives and the clock it regenerates, programmable phase adjustment for the data and data-strobe signals sent to the memory devices 112, programmable phase adjustment for the data and data-strobe signals sent to the system memory controller 101, and/or programmable phase adjustment for one or more control signal from the control circuit 116. The locally synchronized operation also makes it easier for each data buffer 118 to perform self-testing of the associated set of memory devices 112, independent of the self-testing of other sets of memory devices performed by the other data buffers, as disclosed in commonly-owned U.S. Pat. No. 8,001,434, issued Aug. 16, 2011, entitled “Memory Board with Self-Testing Capability,” which is incorporated herein by reference in its entirety.

In certain embodiments, operations of the data buffers 118 are controlled by the data buffer control signals from the module control circuit 116, which generates the data buffer control signals according to the C/A signals received from the MCH. Thus, the data buffer control signals need to be properly received by the data buffers 118 to insure their proper operation. In one embodiment, the data buffer control signals are transmitted together with the registered clock signal, or the module clock signal CK, which is generated by the module control circuit 116 based on the system clock signal MCK. The isolation circuits 118 buffers the module clock signal, which is used to time the sampling of the data buffer control signals. Since the data buffers 118 are distributed across the memory module, the data buffer control signal lines 230 can stretch across the memory module 110, over a distance of several centimeters. As the data buffer control signals travel over such a distance, they can become misaligned with the module clock signal, resulting in metastability in the received data buffer control signals. Therefore, in one embodiment, the isolation circuits 118 includes metastability detection circuits to detect metastability condition in the data buffer control signals and signal adjustment circuits to adjust the data buffer control signals and/or the module clock signal to mitigate any metastability condition in the data buffer control signals, as disclosed in commonly-owned U.S. patent application Ser. No. 13/952,599, filed Jul. 27, 2013, entitled “Memory Module with Distributed Data Buffers and Method Of Operation,” the entirety of which is incorporated herein by reference.

Because the data buffers 118 are distributed across the memory module 110, during high speed operations, it may take more than one clock cycle time of the system clock MCK for the data buffer control signals to travel along the data buffer control signals lines 230 from the module control device 116 to the farthest positioned data buffers 118, such as data buffer ID-1 and data buffer ID-(n−1) in the exemplary configuration shown in FIG. 2. In other words, a same set of data buffer control signals may reach different data buffers 118 at different times across more than one clock cycle of the system clock. For example, when the clock frequency of the system clock is higher than 800 MHZ, the clock cycle time is less than about 1.2 ns. With a signal travel speed of about 70 ps per centimeter of signal line, a data buffer control signal would travel about 15 cm during one clock cycle. When the clock frequency increases to 1600 MHZ, a data buffer control signal would travel less than 8 cm during one clock cycle. Further, a data buffer control signal line can have multiple data buffer control signals on the line at the same time. Thus, before one data buffer control signal reaches an end of the signal line, another data buffer control signal can appear on the same signal line.

With the data buffers 118 receiving data buffer control signals at different times across more than one clock cycle, the data buffer control signals alone are not sufficient to time the transmission of read data signals to the MCH 101 from the data buffers 118. In one embodiment, the data buffers have programmable control of the timing of read/write data signals and the phase of the locally synchronized clock signal for each group of locally synchronized devices. In certain embodiments, each data buffer can further determine individually, during a write operation, a time interval between a time when one or more data buffer control signals are received from the module control circuit 116 and a time when a write strobe or write data signal is received from the MCH 101. This time interval is used to adjust the timing of the transmission of read data to the MCH 101 during a subsequent read operation, as explained in more detail below.

More illustrative information will now be set forth regarding various optional configurations, architectures, and features with which the foregoing framework may or may not be implemented, per the desires of the user. It should be strongly noted that the following information is set forth for illustrative purposes and should not be construed as limiting in any manner. Any of the following features may be optionally incorporated with or without the exclusion of other features described.

In one embodiment, as shown in FIG. 3, each group of signal lines 130 include a set of n data (DQ) signal lines 322 each for transmitting one of a set of bits DQ₀, DQ₁, . . . , DQ_n-1, in a data signal and at least one strobe (DQS) signal line 324 for transmitting at least one strobe signal DQS. Each set of module data/strobe lines Y include a set of n module data signal lines Y₀, Y₁, . . . , Y_n-1and at least one module strobe signal line Y_DQS. When the subsets of memory devices are coupled to the associated data buffer 118 via respective subsets of memory devices, each set of module data/strobe lines Y may include multiple subsets of module data/strobe lines, such as the subsets of module data/strobe lines YA and YB shown in FIG. 2B. Each subset of module data/strobe lines YA include a set of n first module data lines YA₀, YA₁, . . . , YA_nand at least one first module strobe signal line YA_DQS; and each subset of module data/strobe lines YB include a set of n second module data lines YB₀, YB₁, . . . , YB_nand at least one second module strobe signal line YB_DQS.

Each data buffer 118 includes a set of DQ routing circuits 320 coupled on one side to respective ones of the set of n DQ signal lines 322, and on another side to respective ones of the respective set of n module data lines, or respective ones of the respective subsets of module data lines, such as the first module data lines YA₀, YA₁, . . . , YA_nand the second module data lines YB₀, YB₁, . . . , YB_n. Each data buffer 118 further includes an DB Control circuit 310 coupled on one side to the at least one DQS signal line 324, on another side to the one or more module strobe signal lines Y_DQS, or the first module strobe signal line YA_DQSand second module strobe signal line YB_DQS.

The DB Control circuit 310 also receives the module clock signal CK and the data buffer control signals via the data buffer control signal lines 230, and outputs datapath control signals 330 to the DQ routing circuits 320, including, for example, one or more enable signals ENA and/or ENB, and some or all of the other received, decoded, and/or otherwise processed data buffer control signals. In certain embodiments, the DB Control circuit 310 also outputs one or more delay signals DS, a read DQS signal RDQS, a write DQS signal WDQS, and a buffer clock signal CK0. Each DQ routing circuit 320 is configured to enable data communication between the respective DQ signal line 322 with a selected subgroup of one or more memory devices in response to the data buffer control signals, as explained in more detail below.

In certain embodiments, the DB Control circuit 310 also provides one or more delay signals DS, which is used by the DQ routing circuits 320 to control the timing of read and/or write data output by the data buffer 118, as explained in further detail below. In certain embodiments, the DB Control circuit 310 regenerates clock signal CK0 from the module clock signal CK. In certain embodiments, the regenerated clock signal CK0 is phase-locked to the module clock signal CK. In certain embodiments, the regenerated clock signal CK0 has a programmable delay from the module clock signal CK. In certain embodiments, the regenerated CK0 is provided to the respective group of memory devices so that the data buffer 118 and the respective group of memory devices form a group of locally synchronized devices on the memory module 110.

The memory devices 112 in each group of locally synchronized devices are coupled to the data buffer 118 in the each group of locally synchronized devices via a same set of module data/strobe signal lines. Or the memory devices 112 in each group of locally synchronized devices include subgroups of memory devices that are coupled to the data buffer 118 via different subsets of module data/strobe signal lines. For example, as shown in FIG. 4A, memory devices M₁₁, M₁₂, M₁₃, and M₁₄in the first group of memory devices can be coupled to the data buffer ID-1 via a same set of module data lines Y-1₀, Y-1₁, . . . , Y-1_n-1and module strobe line Y-1_DQS. In such embodiment, a subgroup in the group of memory devices can be selected by the data buffers to communicated data with the MCH based on the phases of the data/strobe signals, which can be different with respect to different subgroups of memory devices.

Alternatively, as shown in FIG. 4B, memory devices M₁₁and M₁₃, which form a subgroup in the first group of memory devices, are coupled to the data buffer ID-1 via the module data lines YA-1₀, YA-1₁, . . . , YA-1_nand module strobe line YA-1_DQSand memory devices M₁₂and M₁₄, which form another subgroup in the first group of memory devices, are coupled to the data buffer ID-1 via the module data lines YB-1₀, YB-1₁, . . . , YB-1_nand module strobe line YB-1_DQS. Memory devices coupled to the same data buffers can be disposed on a same side or different sides of the memory board 119. Memory devices coupled to the same data buffers may be placed side-by-side, on opposite sides of the module boards 119, or stacked over each other, and/or over the associated data buffer.

Multiple memory devices having a data width that is less than a data width of the data buffers 118 may be used in place of one of the memory devices 112, which has the same data width as that of the data buffers. For example, as shown in FIG. 5A, two memory devices M_11-1and M_11-2may be used in place of the memory device M₁₁. Each of the two memory devices M_11-1and M_11-2has a data width of 4, and together they act like a memory device M₁₁of a data width of 8. Thus, memory device M_11-1is coupled to the data buffer ID-1 via module data lines YA-1₀, . . . , YA-1₃and module strobe line YA-1_DQS-1while memory circuit M_11-2is coupled to the data buffer ID-1 via module data lines YA-1₄, . . . , YA-1₇and module strobe line YA-1_DQS-2.

In another embodiment, as shown in FIG. 5B, four memory devices M_11-1to M_11-4may be used as the memory device M₁₁. Each of the four memory devices M_11-1to M_11-4has a data width of 4, and together they act like a memory device M₁₁of a data width of 16. Thus, memory device M_11-1is coupled to the data buffer ID-1 via module data lines YA-1₀, . . . , YA-1₃and module strobe line YA-1_DQS-1while memory device M_11-2is coupled to the data buffer ID-1 via module data lines YA-1₄, . . . , YA-1₇and module strobe line YA-1_DQS-2, and so on.

FIG. 6A is a diagram illustrating clock signals and data buffer control signals from a module control device to a plurality of data buffers in a memory module according to certain embodiments. As shown, the data buffer control signals may include, for example, eight signals received respectively by eight signal receiving circuits 630 on each data buffer. In certain embodiments, one or more of the eight signals are to specify a mode of operation (e.g., normal operation, configuration or test mode). The definitions of the other data buffer control signals are different in different modes of operation. For example, in the configuration mode, the module control circuit 116 uses the DBCS to configure the programmable features in the data buffers 118 based on configuration information received from the MCH 101; in the test mode, the DBCS is used to control the self-test processes run by the data buffers; in the normal operation mode, the DBCS includes the one or more enable signals and one or more ODT signals.

FIG. 6B illustrates the DB Control circuit 310 in an data buffer 118. As shown, the DB Control circuit 310 includes a clock buffer 610 to receive the module clock signal CK from the module control device 116, a phase-locked loop circuit (PLL) 611 that provides phase locking between the input clock CK and the locally synchronized clock CK0, a clock phase adjustment circuit 615 for adjusting the phase of the locally synchronized clock CK0, and a clock driver 618 that drives the locally synchronized clock CK0 to the corresponding group of memory devices. The DB Control circuit 310 further includes a strobe routing circuit 620 that are coupled on one side to a corresponding system DQS signal line 324 and on another side to corresponding module DQS signal lines YA_DQSand YB_DQS. The DB Control circuit 310 further includes receiver circuits 630 to receive respective ones of the DBCS.

The DB Control circuit 310 further includes a control processing circuit 640 that decodes and/or otherwise processes the data buffer control signals. In the normal operation mode, the decoded/processed data buffer control signals may include, for example, one or more data path enable signals ENA and/or ENB that are used by the DQ routing circuits 320 and the strobe routing circuit 620 to selectively enabling data communication between the MCH 101 and one of the subgroups in the respective group of memory devices, with which the data buffer is associated. In a configuration mode, the decoded/processed data buffer control signals are used to program configuration registers in the control processing circuit 640, which in turn control one or more delay signals DS1, DS2, DS3. The delay signal DS1 is used to control delay circuits in the read paths in the DQ/DQS routing circuits, the delay signal DS2 is used to control delay circuits in the write paths in the DQ/DQS routing circuits the data buffer, and the delay signal DS3 is used to control the phase of the locally synchronized clock CK0.

The strobe routing circuit 620 also buffers strobe signals received from either the MCH 101 or the memory devices 112, and outputs either a write strobe WDQS or read strobe RDQS, respectively, to the DQ routing circuits 320. In one embodiment, the DB Control circuit 310 further includes a dynamic delay control circuit 650 that provide dynamic adjustment of the timing of read data/strobe signals in the DQ routing circuit 320 and the strobe routing circuit 620.

In certain embodiments, as shown in FIG. 6C, the control processing circuit 640 includes control logic 660 that receives the data buffer control signals and the module clock signal CK and outputs in the normal operation mode the data path control signals 330. In a configuration mode, the control logic outputs configuration signals for programming configuration registers in the data buffer. For example, the control logic outputs configuration signals 681 to programmable registers 671 that provides static control of the timing of read data/strobe signals, configuration signals 682 to programmable registers 671 that controls the delay signals DS2, which used to control the timing of write data/strobe signals, and configuration signals 683 to programmable registers 673 that controls the delay signals DS3, which is used to control the phase of the locally synchronized clock signal CK0.

The programmable registers 671 is coupled via signal lines 691 to the dynamic delay control circuit 650, which also takes as input a DQS signal, clock signal CK, and one of the data path enable signals EN, and output the delay signal DS1. The dynamic delay control circuit 650 is described in further detail below.

FIG. 7A is a flowchart illustrating a method 700 performed by the memory module 110 according to certain embodiments. As shown in FIG. 7A, the method includes receiving 710 clock signal together with system C/A signals from the MCH 101, and generating 715 module clock signal CK. The module clock signal CK is transmitted 720 to a plurality of distributed data buffers 118, which generates 725 locally synchronized clock signals CK0 for each group of locally synchronized devices. Each data buffer then transmits 730 its locally synchronized clock signals CK0 to the memory devices corresponding to the each data buffer, and the memory devices perform 735 memory operations in accordance with the respective locally synchronized clock signals.

Since the module control circuit 116 generates the DBCS based on signals from the MCH 101, the MCH 101 can train the data buffers 118 to properly time the read/write data signals and to properly adjust the phase of the locally synchronized clock signals CK0. In certain embodiments, the module control circuit 116 generates the DBCS for setting the registers in the delay control circuit based on mode register command signals from the MCH 101. As shown in FIG. 7B, a method 750 for training the data buffers to provide a proper amount of phase shift to the locally synchronized clock signals CK0 includes performing a write operation by writing 751 data to a set of memory locations in the memory devices 112, which is followed by performing a read operation by reading 752 data from the set of memory locations. The read data is compared 753 with the write data to determine 754 if phase adjustment is needed for each locally synchronized clock signals CK0. For example, if the read data bits from data signal lines closer to the module control circuit 116 agree with the corresponding write data bits, while read data bits from the data signal lines further away from the module control circuit 116 do not agree with the corresponding write data bits. The disagreement is likely caused by the locally synchronized clock signals CK0 in the data buffers 118 further away from the module control circuit 116 being excessively delayed during transmission, and proper phase adjustment can be programmed 755 into the data buffers to add a phase-shift to the clock signals CK0 to compensate the skew among the data buffers. The programming can be done using, for example, a mode register write operation. The above process can be repeated until proper clock signal phase adjustment is achieved.

FIG. 8 illustrates a DQ or DQS routing circuit 320 or 620 according to an embodiment. As shown, the DQ/DQS routing circuit 320/620 includes a DQ/DQS pin 801 that is coupled to the corresponding DQ/DQS signal line 322/324, a set of one or more DQS pins 802 that is coupled to a corresponding module DQ/DQS line(s) Y/Y_DQS, or YA/YA_DQSand YB/YB_DQS. The DQ/DQS routing circuit 320/620 further includes a write strobe buffer 810 that buffers write data/strobe, a write data/strobe receiver 820 that samples the write data/strobe, and a write delay circuit 825 controlled by the delay signal DS2. The DQ/DQS routing circuit 320/620 further includes a plurality of write paths 830 that are selectable or can be selectively enabled by one or more of the data buffer control signals, such as the enable signals ENA and ENB.

The DQS routing circuit further includes a plurality of read paths 850 that are selectable in accordance with the one or more of the data buffer control signals. Output from the selected read path is delayed in a delay circuit 860 by an amount controlled by the delay signal DS1, and sampled by a sampler circuit 870. The sampled read data/strobe is transmitted by transmitter 880 onto the corresponding data/strobe signal line 322/324 via the DQ/DQS pin 801.

FIG. 9 illustrates a DQS routing circuit 620 according to an embodiment. As shown, the DQS routing circuit 620 includes a first DQS pin