AI Inference Raises the Standard for Infrastructure Partners
By Scott Willis, President and CEO, DartPoints
When a business begins using AI to serve customers or support daily operations, it starts depending on several systems and providers to work together. A delay that was tolerable during testing can become a problem when a customer is waiting for an answer.
That changes what the business needs from its infrastructure partner. The conversation has to cover how the service will operate, who is responsible for each part of it and what happens when requirements change.
In my view, this is where providers will earn their customers’ confidence. Delivering on the date you commit to matters. So does being a partner the customer can depend on after the deployment is complete.
Start with the application
Inference is what happens when a trained AI model is used to produce a response or prediction. The work can take very different forms, and those differences should influence the infrastructure decision.
Consider a customer service application that uses AI to answer callers. Delays can interrupt the conversation. The business needs to understand how the application will perform when more people use it at once. An application that processes documents overnight has a different requirement. It may have more flexibility about response time, provided the work is finished when the business needs it.
Both use inference. They should not automatically get the same infrastructure recommendation.
A provider needs to understand the application well enough to explain how its part of the environment will support it. That includes knowing where the data sits and how the application will reach it. The customer and its technology partners should test performance under conditions that reflect expected usage.
A discussion that begins and ends with rack space or available power leaves too much unanswered.
Know what will be ready on the delivery date
Power availability and development schedules remain important constraints. For customers, the useful question is what the provider can commit to and what has to happen for that commitment to hold.
A facility handover, installed equipment and an application ready for users are separate milestones. A business needs to know which one the proposed date represents.
If power depends on utility work, that dependency should be understood. If the customer’s equipment installation or network connections follow the facility handover, those steps need to be included in the overall schedule. The customer should be able to see how the provider’s delivery plan connects to its own launch plan.
This is where a provider’s judgment matters. An aggressive date is useful only when there is a workable plan behind it. When something changes, the customer needs enough notice to make decisions about its own commitments.
Agree on responsibilities before launch
An AI application can depend on a data center operator, a cloud provider, network carriers and the team managing the model and application. Each may be doing its own job correctly while the overall service still falls short.
Customers need to understand where those responsibilities begin and end. In a colocation arrangement, for example, the facility operator may be responsible for power, cooling and physical security while the customer or another provider manages the servers and software. Managed services can shift some of that work, but the scope still needs to be clear.
Take a slow response from an AI application. The issue could involve the model, available compute or the connection to the data it needs. The customer needs an agreed process for investigating the problem and coordinating the people who can resolve it.
Those conversations belong before launch. Customers should know who monitors each part of the service, how a problem gets escalated and who keeps them informed. They should also understand what the provider’s service commitments cover and how performance will be measured.
Keep working as the requirements change
The first deployment gives the customer a starting point. Actual usage will help determine what comes next.
A business may add users, introduce another model or connect the application to more data. Before making those changes, it needs to understand the effect on performance, cost and the infrastructure supporting the service.
That calls for an ongoing conversation with the provider. It should cover whether the current environment is meeting expectations, what additional capacity would require and how much lead time the customer needs to allow.
I believe customers will place more value on providers who can have those conversations openly and follow through on what they agree to do. A trusted partner should be able to explain what it can support today, be candid about limitations and help the customer plan the next step.
As inference becomes part of daily operations, customers will judge us by how consistently we do that work. We have to deliver when we say we will and remain accountable for our part of the service once it is running.