Part 1: The OIF’s co-packaging initiative
Large-scale data centres consume huge amounts of power; one building on a data centre campus can consume 100MW. But there is a limit as to the overall power that can be supplied.
Jeff Hutchins
The challenge facing data centre operators is that networking, used to link the equipment inside the data centre, is consuming more and more of the power.
That means less power remains for the servers; the computing that does the revenue-generating work.
This is forcing a rethink regarding networking and explains the growing interest in co-packaged optics, a technique that effectively adds optical input-output (I/O) to a chip.
Two industry organisations - the OIF and The Consortium for On-Board Optics (COBO) - have each started work to identify the requirements needed for co-packaged optics adoption.
“We are seeing this activity because co-packaged optics is hard and requires prework to figure out how and when it is going to happen, and how the ecosystem changes,” says Nathan Tracy, TE Connectivity and the OIF’s vice president of marketing.
All change
Semiconductors and optics have always been separate domains but with a co-packaged design, silicon is suddenly only a handful of millimetres away from the optics, says Tracy: “It’s a very different environment.”
Hot chips sit next to the optics, so thermal characteristics must be shared and the cooling needs worked out. The electrical interface linking the optics to the chip will need to be optimised while there are new challenges such as how faults are dealt with.
“All these things come together and it changes what is done in the industry,” says Jeff Hutchins, Ranovus and OIF Physical and Link Layer (PLL) Working Group – Co-Packaging Vice-Chair.
“To be fair, there are companies that are not totally on-board with co-packaging,” says Hutchins. “But if you think about what is driving it, as you go to higher and higher electrical rates to connect things, you start to run more power and it is just more difficult to get a signal from Point A to Point B.”
For next-generation designs, companies are also considering ‘fly-over’ cables as well as the intermediate step of on-board optics, moving optics from the front panel onto the line card to be closer to the ASIC.
“But a good part of the industry thinks that, if you look forward, the only way to get there is co-packaging,” says Hutchins.
Using co-packaged optics will also impact the supply chain. The switch and pluggable modules are typically bought separately whereas a co-packaged design integrates the two. “Economically, it changes the way the industry works,” says Hutchins.
OIF and COBO
Hutchins, who is also a board member of COBO, says the co-packaging work of the two organisations will be complementary.
Co-packaged optics resides deep on the line card and fibre must connect the package to the system’s front panel. In turn, an external laser is commonly used as the light source for the optics. Such a laser is linked to the package using fibre.
Nathan Tracy“What COBO is doing is focussing on the optical connectivity part of this solution; the stuff outside the co-packaged assembly,” says Hutchins. “The OIF is concentrating on what the co-package assembly is, what goes inside, and what agreements can be made for interoperability for the whole assembly.”
The membership of the two organisations also differs: the OIF members include hyperscalers as well as optical and switch companies. “We have a good cross-section of the membership of this ecosystem,” says Tracy. COBO’s membership includes companies with connector and materials expertise.
Framework project
The OIF Framework Project will first study the applications where co-packaged optics will be used, identifying commonalities. It will then address the technology to determine what interoperability agreements are needed.
Applications for co-packaged optics besides Ethernet switches include machine learning and disaggregation. A disaggregated design refers to separating the chips found on a server motherboard - general processors (CPUs), graphics processor units (GPUs) and memory - into separate pools. A workload can then access the pools and configure the hardware elements it needs.
For each application, issues such as density, power, latency, and wavelength-count-per-fibre will be explored. “These must be understood as they differ as you go across the applications,” says Hutchins.
The OIF will identify what interoperability agreements to pursue and what should remain open for now before kicking-off specific Implementation Agreements.
Hutchins stresses that are many aspects that can be standardised such as the mechanical design, environmental issues, power, electrical interfaces and reliability. “That is enough work to keep the whole group busy for quite a while,” he says
As an example, such work could lead to a common socket design that would allow different optical specifications and reliability requirements, says Hutchins.
The OIF expects to complete the first two stages within the coming year.
“People are ready to go but they need to see the whole picture,” says Hutchins.
Roadmap
The OIF expects a gradual introduction of co-packaged Ethernet switches in the data centre with the technology spanning several generations.
Demonstrations could start with 25.6-terabit switches emerging now whereas many think the next-generation 51.2-terabit platforms will be the place to do initial demonstrations and small-scale deployments. After that, 100-terabit switches will likely be the sweet spot for co-packaged optics. And once 200-terabit switches appear, co-packaged optics will be a necessity.
This may be a wide range of entry points, says Hutchins, but technology is being put together in a new way.
“The industry has to learn how to make this cost-effectively and achieve good yields,” says Hutchins. “There has to be a starting point somewhere but where the intercept point is, I don’t know.”
“Pluggables have served the market really well; they are flexible and [optical module] innovation continues,” adds Tracy. “The methodology is working so the question is when does it no longer suit the market.”
Tracy does not rule out pluggables being used for 100-terabit switches but inevitably it will be much harder to satisfy that requirement. “That is when co-packaged optics starts to become compelling,” says Tracy.