The VUB will soon place the new Flemish Tier-1 supercomputer in a brand-new data center in Zellik, operated by Penta Infra. ITdaily visits the green data center, which is part of an energy research project.
The sun shines fiercly on the facade of the eight-story Penta Infra data center in Zellik. That’s good news for the power supply: the facade is entirely covered in solar panels. The panels generate up to 360 kW. Penta Infra Bru01 will soon need quite a bit more, but every little bit helps, and the panels illustrate the sustainable nature of the brand-new building.
Cream of the Crop
“The brand-new data center is the cream of the crop,” says Ward Poelmans, head of Scientific Data & Compute at the Free University of Brussels. The Nexus data center will soon house a Tier-1 supercomputer: the fourth for Flanders but the first under VUB management. “Such a system consumes a lot of electricity,” Poelmans continues. “If the cooling can be even a little more efficient, it helps a lot in the total consumption.”
Penta Infra BRU01, or Nexus to friends, exists thanks to the VUB. When the university was looking for a location for a new data center with room for a Tier-1 system, Ghelamco decided to take the bull by the horns and immediately build a larger data center where other customers are also welcome. After construction, Dutch data center specialist Penta Infra took over the site, thus gaining a foothold in Belgium for the first time.
Partner with Experience
“The fact that the data center is now in the hands of Penta Infra is very convenient for us”, says Karin Voets, CIO of the VUB. “It’s a player with experience, which, like the VUB, is strongly committed to sustainability.” “Running a data center requires specific knowledge”, Poelmans adds. “It’s impressive what goes into it. The company has all the specific knowledge about cooling systems and heat exchangers and generators.”
Those emergency generators are no pushovers. Currently, there are three immense examples next to the sun-drenched solar panel facade, but in the future, there’s room for six fixed and a seventh mobile one. The connections for these are already visible. The generators can not only provide power to the data center but can also supply the grid. This allows Penta Infra to deliver peak capacity at Elia’s request, but also to conduct emergency power tests under actual capacity.
Bees Welcome
Despite all that power, the data center is still built with sustainability in mind. That was essential for the university. Voets: “The building is part of the Green Energy Park, closely linked to the VUB.” The site where Penta Infra’s data center stands is actually a large laboratory where energy research will take place.
Along the large emergency power generators of the data center, the focus on sustainability is also literally visible. We see green berms, sown with a seed mixture that will produce wildflowers. There’s a pond to collect rainwater, with two nesting boxes on a pole behind it for local rodents, which have already come sniffing around. A bee hotel is also present.
Rainwater Tank and Heat Network
We enter the data center through the service entrance and make our way towards the room where the Tier-1 supercomputer will soon be located. We first pass through a hallway containing lids that provide access to the large underground rainwater reservoir. In a room further on, large machines that condition the air stand next to a large heat exchanger. There, the waste heat from the data center can be converted into usable heat.
The Green Energy Park aims to be a testing ground for, among other things, heat recovery.
Karin Voets, CIO VUB
“The Green Energy Park aims to be a testing ground for, among other things, heat recovery”, Voets explains. In the long term, new buildings will be connected to the data center’s heat network. In turn, solar panels on those roofs will provide power to the data center, allowing the supercomputer and other servers to run on solar energy as much as possible without burdening the wider electricity grid.
Walkway with Showcase
We ascend one floor and enter a beautiful reception hall. From there, we walk along a walkway that provides a view of a currently empty server room. We’ve visited many data centers, but this is truly unique. The server room will house the Tier-1 system and, at the request of the VUB, had to be clearly visible.
read also
Supercomputer in the Showcase: Visiting Penta Infra’s First Belgian Data Center
“The computer will have its own look”, Poelmans explains. “We’re going to decorate the device with a design and give it a name”. Unfortunately, we can’t extract any details about this. “Bringing the system to us does have a bit to do with prestige”, Poelmans continues. “We want to shed the label of being a small university.” Those passing by the Researchpark Zellik will be able to admire Flanders’ most powerful High Performance Computing (HPC) system through the glass wall.
Educational Role
The VUB wants to use this opportunity to play an educational role and bring HPC to a wider audience. “The visibility of HPC within our institution is also increasing”, Voets adds. “We will guide those who don’t spontaneously find their way to HPC but can make great strides by using it, and lead them to our systems.”
Although the unnamed supercomputer is not yet in place, we can already see some interesting details. On both sides of the room, redundant power connections come in, along with some plumbing that is less obvious. Flanders’ newest Tier-1 will be a water-cooled system.
Liquid cooling
“From the beginning, we assumed that the new system would not be exclusively air-cooled,” Poelmans explains. “If you see how much power one GPU uses and how much space is needed for air cooling, you’d have to keep server racks half empty. We now work with nodes of eight GPUs each and six nodes in a rack. To cool that efficiently, direct liquid cooling for the CPUs and GPUs is necessary.”
“The previous Tier-1 system is also partially water-cooled, but that wasn’t straightforward”, he continues. “We’ve learned a lot from it. There’s a lot involved. You need heat exchangers and there’s a liquid circuit running through the nodes.”
Complex Plumbing
Poelmans is glad that the knowledge for the underlying plumbing with valves, cables, and pumps can be outsourced to Penta Infra. The data center specialist then takes the opportunity to make water cooling possible elsewhere in the data center, should customers request it in the future. Penta Infra emphasizes that the data center is ready for liquid cooling and expects to house a mix of air- and liquid-cooled installations in the long term.
Despite its computing power, the system itself won’t take up that much space in the provided area. This is due to the high density, made possible in part by the water cooling.
GPUs for AI
“For this system, we looked at the previous Tier-1 computer and talked with researchers about their needs”, Poelmans further explains. “GPU ‘s and AI emerged as key points, but they are quite expensive. Some researchers only need CPU’ s. We sought a compromise with a balance between CPU and GPU that maximizes computing power within the budget.”
We step away from the HPC showcase and delve deeper into the data center. There we see air cooling installations that are somewhat surprisingly located under the data center floor. The air conditioners blow cold air directly into an enclosed cool corridor, where the distance between the cooler and the corridor is as small as possible, meaning the fans have to run as little as possible. The impact of such incremental improvements adds up, making this data center so efficient.
The VUB also relies on such expertise from Penta Infra and sees the collaboration as a two-way interaction. The university plans to consolidate its IT infrastructure, including existing Tier-2 systems and other servers, within Nexus. “If there are interventions that can promote sustainability, we also count on Penta Infra to inform us about them”, says Voets.
Rainwater Evaporation
For another example of sustainable data center innovation, we take the elevator to the top floor. This technical space immediately illustrates which jobs AI will not replace in the future: those of electricians or plumbers. We make our way through an impressive tangle of large and small pipes, pumps, and electrical wiring to a door leading to the roof.
Once outside, we try to keep our balance on the gigantic boulders that keep the roof covering in place, and regret not wearing our ankle-supporting hiking shoes. We look directly at an immense radiator, on which rainwater from the reservoir under the data center is misted. This evaporation allows Penta Infra to keep the cooling water at an efficient 19 degrees, even when it’s 22 degrees outside. Only above this temperature do more traditional and less economical chillers need to actively engage.
Grand Plans
From the roof, we have a beautiful view of the entire research park. The VUB wants to expand there quickly. “The former Roularta building is now in the hands of the VUB and the Smart Village Lab is already there”, Voets explains. “And Belnet is coming to install a Point of Presence (PoP) here.” “On the other side of the park, there will be an exceptionally powerful 7 Tesla MRI”, Poelmans adds. “This is a type of MRI scanner with a magnetic field strength of seven Tesla for very detailed medical imaging, which will be used by the five Flemish universities and four university hospitals.”
The first phase of the Tier-1 supercomputer should be online by the end of this year and run for at least four years. Two years later, a second phase will be added. The VUB initially expects a lifespan of six years for the system. During this period, not only researchers can get computing time, but also parties from industry and civil society.
Strong together
“Classic major users are researchers who want to make chemical calculations”, Poelmans clarifies. “Or applications from astrophysics and fluid dynamics. It’s also interesting for climate scientists, because earth simulations start with a bit of data but quickly explode into incredible complexity. Now AI is of course being added to the mix.”
read also
Sustainable and Modern BRU01 Data Center of Penta Infra Officially Opened
“It’s nice that this Tier-1 supercomputer is now coming to the VUB”, Voets adds. “But actually, the location doesn’t matter much to researchers in practice. We have good collaborations with other universities, and we will manage this system together with the HPC teams from the universities of Leuven, Ghent, and Antwerp. We’re not doing it alone.”
On the way down from the roof of the Nexus data center, the elevator passes many still empty halls. This will likely change soon, as the demand for efficient colocation space is rapidly increasing. Telecom providers have also provided their connections, making Penta Infra BRU01 provider-agnostic. The data center built for growth has an initial capacity of 7 MW, but can be further expanded.
On our way out, we take a look at the empty space where the VUB’s first Tier-1 will be. We’re curious about the name.