While everyone is talking about new models and their possible use cases, their deployment aspect often gets overlooked. The journey from a trained model to a production-ready service is a complex and nuanced process that deserves more attention. From the perspective of a web API server, when a developer needs to access information like user profiles or services, we typically create a REST API service that interacts with the database. This API service also handles business logic, enabling the system to process and serve thousands of requests per minute efficiently. However, it is different when we talk about serving models.
In the pre-production phase, data scientists and machine learning (ML) engineers often test their models locally, loading model weights onto a Compute Unified Device Architecture (CUDA) device using ML libraries like PyTorch to showcase accuracy. While this local execution works excellently for testing, scaling that same model to handle real-time, production-level traffic is an entirely different challenge. Many engineers consider serving the model by wrapping it in a in production is quite different from monitoring the performance of traditional API servers. Inference requires specialized monitoring for aspects like latency, GPU utilization, and throughput—less relevant metrics for typical API services. This is where for a comprehensive overview.
In this cycle, inference and serving come into play in the latter half once a model has been trained and is ready for deployment. Though these terms are often used interchangeably, they refer to different stages in the lifecycle of a model in production.
What is inference?
Inference is when a trained model takes input data and produces predictions or outputs. In simpler terms, the actual computation happens when a model is asked to generate a result—like classifying an image, translating text, or generating a response in a chatbot. Inference happens locally when testing the model, often using a framework like PyTorch or TensorFlow, and can be run on either CPUs or GPUs.
What is model serving?
Serving, on the other hand, refers to making the model accessible as a service. This involves deploying the model in a way that allows it to handle real-time requests, often at scale. When a model is served, it’s not just about running inference but doing so in an optimized, scalable, and monitored environment where it can respond to multiple requests from users or applications in real time. Serving requires integrating the model with APIs, managing resources like GPU/CPU, and ensuring the service is stable and performant over time.
Since we’re talking about deploying for detailed understanding.
What is vLLM?
and serving libraries. As the name suggests, ‘virtual’ encapsulates the concept of virtual memory and paging from operating systems, which allows addressing the problem of maximum utilization of resources and providing faster token generation by utilizing , handling significantly significant traffic and reducing operational costs.
Why vLLM?
vLLM is a specialized and efficient library for large language models (LLMs) with several advantages:
Open source and highly adaptable: It’s an open source library, making it flexible and accessible for various use cases.
Broad model support: It supports a wide range of model architectures, which you can explore further in the is an open source unified framework for AI and Python applications built around the idea of simplified distributed computing. It allows users to run tasks in parallel across multiple nodes or machines, making it ideal for distributed machine learning, reinforcement learning, or parallel processing. One of Ray’s standout features is its high-level libraries, one of them is Ray Serve, designed to streamline model serving for machine learning applications. You can learn in detail in
and .
So, what we’re more interested in the scope of this blog is RayService CRD.
RayService
The RayService CRD allows you to deploy Ray Serve applications seamlessly on Kubernetes. By defining a RayService, you can specify your Ray Serve deployment's parameters, such as the model to be served, scaling options, and routing configurations. This abstraction simplifies the deployment process and allows you to manage your serving infrastructure through Kubernetes.
Example of a RayService CRD:
CODEapiVersion: serving.kubray.io/v1alpha1
kind: RayService
metadata:
name: audio-model
spec:
rayCluster: my-ray-cluster
deployment:
replicas: 3
model: AudioModel
routePrefix: "/audio"
In this example, the RayService CRD defines a deployment for the
AudioModel, specifying that three replicas should be created to handle incoming requests at the/audioendpoint. This structure simplifies the deployment and integrates with Kubernetes' existing capabilities.
Serving Model on Kubernetes
In this implementation, we will be deploying
Prerequisites
To get this working, we will need the following things beforehand.
- kubectl: Make sure you have kubectl installed on your local system.
- Kubernetes Cluster: It should have at least two worker nodes with 1 CPU node and 1 GPU node.
- Make sure the GPU node is tainted.
- Ray Serve library (optional): It is not required per se, but for local testing, it should be present.
- Helm: It will be used for installing charts.
Setting up
1.Install KubeRay via Helm on Kubernetes.
CODEhelm repo add kuberay https://ray-project.github.io/kuberay-helm/
helm repo update
CODEhelm install kuberay-operator kuberay/kuberay-operator --version 1.2.1
Output:
CODENAME: kuberay-operator
LAST DEPLOYED: Fri Sep 20 07:44:00 2024
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
2.Now, create a Ray Serve application.
We will wrap the deployment and serve in the same Python Class, VLLMInference. The vLLM engine will be created during initialization, and the tokenizer will be loaded. Upon getting a request on the REST API endpoint /generate, it will use the vLLM-provided chat template and pass the prompt self.engine.generate, which will queue the request if other requests are still being processed. Lastly, the Custom GenerateResponse PyDantic model will revert responses in a specified format.
CODE@serve.deployment(name='VLLMInference',
num_replicas=1,
max_concurrent_queries=256,
ray_actor_options={"num_gpus": 1.0}
)
@serve.ingress(app)
class VLLMInference:
def __init__(self, **kwargs):
super().__init__(app)
self.args = AsyncEngineArgs(**kwargs)
self.engine = AsyncLLMEngine.from_engine_args(self.args)
self.tokenizer = self._prepare_tokenizer()
def _prepare_tokenizer(self,):
from transformers import AutoTokenizer
if self.args.trust_remote_code:
tokenizer = AutoTokenizer.from_pretrained(self.args.model, trust_remote_code=True)
else:
tokenizer = AutoTokenizer.from_pretrained(self.args.model)
return tokenizer
@app.post("/generate", response_model=GenerateResponse)
async def generate_text(self, request: GenerateRequest, raw_request: Request) -> GenerateResponse:
logging.info(f"Received request: {request}")
try:
generation_args = request.dict(exclude={'prompt', 'messages'})
if generation_args is None:
# Default value
generation_args = {
"max_tokens": 500,
"temperature": 0.1,
}
if request.prompt:
prompt = request.prompt
elif request.messages:
prompt = self.tokenizer.apply_chat_template(
request.messages,
tokenize=False,
add_generation_prompt=True
)
else:
raise ValueError("Prompt or Messages is required")
sampling_params = SamplingParams(**generation_args)
request_id = self._next_request_id()
results_generator = self.engine.generate(prompt, sampling_params, request_id)
final_result = None
async for result in results_generator:
if await raw_request.is_disconnected():
await self.engine.abort(request_id)
return GenerateResponse()
final_result = result # Store the last result
if final_result:
return GenerateResponse(output=final_result.outputs[0].text,
finish_reason=final_result.outputs[0].finish_reason,
prompt=final_result.prompt)
else:
raise ValueError("No results found")
except ValueError as e:
raise HTTPException(HTTPStatus.BAD_REQUEST, str(e))
except Exception as e:
logger.error('Error in generate()', exc_info=1)
raise HTTPException(HTTPStatus.INTERNAL_SERVER_ERROR, 'Server error')
@staticmethod
def _next_request_id():
return str(uuid.uuid1().hex)
async def _abort_request(self, request_id) -> None:
await self.engine.abort(request_id)
@app.get("/health")
async def health(self) -> Response:
"""Health check."""
return Response(status_code=200)
def deployment_llm(args: Dict[str, str]) -> Application:
return VLLMInference.bind(**args)
Once the Ray Serve application is ready, push it to the repository.
3.Now, let’s define RayService CRD.
This CRD will help us deploy our Ray Serve application on Kubernetes and configure scaling and other Kubernetes-related parameters.
Here, I’m providing a name and route, import path, and the location of the binding function within the working directory (i.e., our .
4.Deploying monitoring stack.
To deploy the monitoring stack, you can use these docs: and checkout to master. Inside the local repo directory, run the below command.
CODE# Path: kuberay/
./install/prometheus/install.sh
Output:
CODE$ kuberay git:(master) ./install/prometheus/install.sh
+ set errexit
+ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
"prometheus-community" already exists with the same configuration, skipping
+ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "metrics-server" chart repository
...Successfully got an update from the "kuberay" chart repository
...Successfully got an update from the "prometheus-community" chart repository
Update Complete. ⎈Happy Helming!⎈
+++ dirname ./install/prometheus/install.sh
++ cd ./install/prometheus
++ pwd
+ DIR=/home/sudhanshu/Desktop/workspace/ray-demo/kuberay/install/prometheus
+ helm --namespace prometheus-system install prometheus prometheus-community/kube-prometheus-stack --create-namespace --version 48.2.1 -f /home/sudhanshu/Desktop/workspace/ray-demo/kuberay/install/prometheus/overrides.yaml
NAME: prometheus
LAST DEPLOYED: Mon Sep 23 07:53:55 2024
NAMESPACE: prometheus-system
STATUS: deployed
REVISION: 1
NOTES:
kube-prometheus-stack has been installed. Check its status by running:
kubectl --namespace prometheus-system get pods -l "release=prometheus"
Visit https://github.com/prometheus-operator/kube-prometheus for instructions on how to create & configure Alertmanager and Prometheus instances using the Operator.
+ monitor_dir=/home/sudhanshu/Desktop/workspace/ray-demo/kuberay/install/prometheus/../../config/prometheus
+ pushd /home/sudhanshu/Desktop/workspace/ray-demo/kuberay/install/prometheus/../../config/prometheus
~/Desktop/workspace/ray-demo/kuberay/config/prometheus ~/Desktop/workspace/ray-demo/kuberay
++ ls
+ for file in `ls`
+ kubectl apply -f podMonitor.yaml
podmonitor.monitoring.coreos.com/ray-workers-monitor created
+ for file in `ls`
+ kubectl apply -f rules
prometheusrule.monitoring.coreos.com/ray-cluster-gcs-rules created
+ for file in `ls`
+ kubectl apply -f serviceMonitor.yaml
servicemonitor.monitoring.coreos.com/ray-head-monitor created
+ popd
~/Desktop/workspace/ray-demo/kuberay
Check all the resources for monitoring up and running.
CODE$ kuberay git:(master) kubectl get all -n prometheus-system
NAME READY STATUS RESTARTS AGE
pod/alertmanager-prometheus-kube-prometheus-alertmanager-0 2/2 Running 0 114s
pod/prometheus-grafana-54cddddd76-r8jqp 3/3 Running 0 2m2s
pod/prometheus-kube-prometheus-operator-96f59f654-9vbxc 1/1 Running 0 2m2s
pod/prometheus-kube-state-metrics-786fbd7c69-9xdtk 1/1 Running 0 2m2s
pod/prometheus-prometheus-kube-prometheus-prometheus-0 2/2 Running 0 113s
pod/prometheus-prometheus-node-exporter-77kkn 1/1 Running 0 2m2s
pod/prometheus-prometheus-node-exporter-89dc5 1/1 Running 0 2m2s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 115s
service/prometheus-grafana ClusterIP 34.118.226.253 <none> 80/TCP 2m3s
service/prometheus-kube-prometheus-alertmanager ClusterIP 34.118.231.161 <none> 9093/TCP,8080/TCP 2m3s
service/prometheus-kube-prometheus-operator ClusterIP 34.118.234.87 <none> 443/TCP 2m3s
service/prometheus-kube-prometheus-prometheus ClusterIP 34.118.236.54 <none> 9090/TCP,8080/TCP 2m3s
service/prometheus-kube-state-metrics ClusterIP 34.118.232.116 <none> 8080/TCP 2m3s
service/prometheus-operated ClusterIP None <none> 9090/TCP 114s
service/prometheus-prometheus-node-exporter ClusterIP 34.118.225.149 <none> 9100/TCP 2m3s
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/prometheus-prometheus-node-exporter 2 2 2 2 2 kubernetes.io/os=linux 2m3s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/prometheus-grafana 1/1 1 1 2m3s
deployment.apps/prometheus-kube-prometheus-operator 1/1 1 1 2m3s
deployment.apps/prometheus-kube-state-metrics 1/1 1 1 2m3s
NAME DESIRED CURRENT READY AGE
replicaset.apps/prometheus-grafana-54cddddd76 1 1 1 2m3s
replicaset.apps/prometheus-kube-prometheus-operator-96f59f654 1 1 1 2m3s
replicaset.apps/prometheus-kube-state-metrics-786fbd7c69 1 1 1 2m3s
NAME READY AGE
statefulset.apps/alertmanager-prometheus-kube-prometheus-alertmanager 1/1 115s
statefulset.apps/prometheus-prometheus-kube-prometheus-prometheus 1/1 114s
5.Now, deploying RayService.
Note: Update the monitoring part in the YAML configuration under the ray-head container env with the correct and uncomment values.
CODEenv:
- name: RAY_GRAFANA_IFRAME_HOST
value: http://127.0.0.1:3000
- name: RAY_GRAFANA_HOST
value: http://prometheus-grafana.prometheus-system.svc:80
- name: RAY_PROMETHEUS_HOST
value: http://prometheus-kube-prometheus-prometheus.prometheus-system.svc:9090
To deploy, apply the YAML in the cluster.
CODEkubectl apply -f vllm-service-phi-3-mini-4k.yaml
Output:
CODErayservice.ray.io/vllm-service created
It will take some time to load the images since rayproject/ray-ml:2.30.0 image is oversized (you could try building a small image file using that one, as mentioned
8.Sending API requests to deployed model.
Send a request to deployed LLM model inference server. To do that, you would need to port-forward the service on local port 8000
CODEkubectl port-forward svc/vllm-service-serve-svc 8000
Now, send the curl request from the terminal or Postman application, whichever suits you the best.
CODEcurl --location --request POST 'http://127.0.0.1:8000/generate' \
--header 'Content-Type: application/json' \
--data-raw '{
"prompt": "<|user|>\n<|user|>\n What are Large Language Models?<|end|>\n<|assistant|>",
"messages": [],
"max_tokens": 500,
"temperature": 0.1
}'
Here, the number of tokens to be generated is 500 and temperature is set to 0.1, you can change it if you like and play around with it to reach optimal value.
Sending multiple messages/chat format
To send multiple messages similar to chat conversations with history as context, you could use the below curl.
CODEcurl --location --request POST 'http://127.0.0.1:8000/generate' \
--header 'Content-Type: application/json' \
--data-raw '{
"prompt": "",
"messages": [
{
"role": "user",
"content": "Can you provide ways to eat combinations of bananas and dragonfruits?"
},
{
"role": "assistant",
"content": "Sure! Here are some ways to eat bananas and dragonfruits together: 1. Banana and dragonfruit smoothie: Blend bananas and dragonfruits together with some milk and honey. 2. Banana and dragonfruit salad: Mix sliced bananas and dragonfruits together with some lemon juice and honey."
},
{
"role": "user",
"content": "What about solving an 2x + 3 = 7 equation?"
}
],
"max_tokens": 500,
"temperature": 0.1
}'
Monitoring model performance
With Ray Dashboard
Ray Dashboard serves as a comprehensive monitoring tool for Ray clusters, providing live updates on service health, application deployments, resource consumption, and node-level diagnostics, which are crucial for managing distributed workloads.
As you can see in the Serve tab, VLLMService is created with vLLM Inference as part of it, and there are logs in case you need to dive deep into something.
Serve Replica, which we set to 1 for our application, is deployed, and we can see its logs in the Ray Dashboard under Actors. As stated earlier, the stateful unit of work is the Actor.
With a monitoring stack
With Grafana and Prometheus in place, you can get more information, such as the QPS( Query Per Second ) of each service and replicas if you have more than one, and an overall view of the deployment we’ve deployed, i.e., VLLM Service. This monitoring setup reduces the burden and provides more than enough metrics for you when you start with Ray on Kubernetes.
.
If you found this post valuable and informative, subscribe to our weekly newsletter for more posts like this. I’d love to hear your thoughts on this post, so do start a conversation on
- Guide to GPU Sharing Techniques: vGPU, MIG and Time Slicing
SOCIAL SHARE CARD GENERATOR