AI Inference Latency: Why Inference Runs On Interconnection, Not Just GPUs

|

 

AI inference latency is decided by the network path between a GPU and its users, not by the GPU alone. As AI clouds arrive in Malaysia, the exchanges, peers and cloud on-ramps beside them will decide how fast their services feel.

 

In October, one AI cloud provider signed letters of intent for 500MW of Tier III AI capacity across Malaysian sites, from the second half of 2027. Budget 2027 also named a planned sovereign AI. Both bring GPUs to Malaysia; neither brings the network paths with them.

 

 

Why Does AI Inference Depend On The Network?

 

Inference is the work a model does each time someone asks it something. It powers chatbots, searches, recommendations and agents, so every answer travels to a user and back.

 

Why AI inference depends on the network

 

McKinsey expects inference workloads to grow about 35% a year over the next five years, faster than training. The same analysis notes that inference sites are built for short round trips and strong interconnection, close to the people they serve.

 

 

How Is Inference Different From Training?

 

Training and inference need different things from a site. The network matters most for inference.

 

How inference differs from training

 

This is why a GPU cluster that trains well can still serve slowly. If its answers detour through another country before reaching a Malaysian user, the delay sits in the path, not the chip.

 

AI inference network path visual

 

 

What Should A GPU Cloud Data Centre Connect To Day One?

 

Power and cooling get an AI cluster running. These six connections get answers to users.

 

Essential connections for a GPU cloud data centre

 

DE-CIX is “one of the first Internet Exchange operators globally” with Google VPP Gold status. For a GPU cloud, that means one port can carry public peering and a verified Google path together.

 

What Is DE-CIX Malaysia’s View?

 

AI capacity should arrive connected, not be connected later. DE-CIX Malaysia offers access in Johor Bahru, Kuala Lumpur and Cyberjaya, and Penang, where Penang IX powered by DE-CIX runs at Open DC PE2. A node at Open DC D8-1 in Kedah is targeted for end-2026 or January 2027.

 

What Should GPU-Cloud Tenants Ask Before Going Live?

 

  • Which exchange can my cluster reach from its rack, and in how many regions?
  • Can I reach Google, AWS and Azure without leaving the exchange fabric?
  • Which ISPs and content networks will my traffic reach through public peering?
  • What happens to my traffic if one cable or site fails?

 

See who you can already reach on the connected networks page, or read how private cloud links work on the DirectCLOUD page. For Google traffic, read our October post, Google changed how networks reach it.

 

Connect your GPU cluster. Tell us where your cluster sits and which clouds and networks you need. A peering engineer will map the paths. Connect your GPU cluster →

 

 

FAQs

 

Why does latency matter for AI inference?
Inference answers live requests from users and applications, so each answer makes a round trip across the network. A longer path adds delay that the user feels, however fast the GPU is.

 

Is AI training sensitive to latency?
Much less than inference. Training moves large data sets in bulk and can run in remote, power-rich sites, while inference needs short paths to the people it serves.

 

What is DirectCLOUD?
DirectCLOUD is DE-CIX’s private cloud on-ramp service. In Malaysia it connects networks to AWS and Azure over the exchange, keeping data and storage traffic off the public internet.

 

 

Interconnect in Malaysia. Scale Across ASEAN. Connect Globally.

 

 

Working Hours: Monday – Friday, 9am – 6pm

Call Us: +603 9212 5961

Email Us: enquiry@de-cix.my

 

DE-CIX Malaysia Sdn Bhd