Ensemble of tf models

AI Discussions
1

can we have any facility for serving ensemble of two or more models using tf-serve? also can i serve torch model using tf serving platform.? if yes could you share resource…

thanks

2

Have you checked out the MLOps specialization, it might answer your questions!

3

Hey Riyaj,
you can have look in MLOps Specialization, specifically in Course-03.

Also you can develop foundation towards ensemble learning by going through deeplearning Specilization courses, that helped me a lot, certainly it will help you as well.

Happy learning!!.

4

Hi @Riyaj_Atar

Yes, it is possible to serve an ensemble of two or more models using TensorFlow Serving (TF-Serving). One way to do this is to use the Multi-Inference API of TF-Serving, which allows you to send requests to multiple models in a single API call and receive the predictions from each model. You can also use the Ensemble API to combine the predictions of multiple models and make a final prediction.

Regarding serving torch models using TensorFlow Serving, it is possible but requires conversion of the model to TensorFlow format. One way to do this is by using the ONNX Runtime, which allows you to convert models from PyTorch to TensorFlow and then serve them with TensorFlow Serving.

Here’s a resource that might be helpful for converting torch models to TensorFlow format:

And also this resource is useful to use TensorFlow serving with ensemble models:

Please note that, You may also use other libraries such as “torch2tensorflow” or “onnxmltools” to convert Pytorch models to Tensorflow and serving them using TensorFlow-serving.

Regards
Muhammad John Abbas

5

Thanks for reply but the link you provided are not reachable.

6

Hi @Riyaj_Atar

Sorry for the wrong links
I didn’t cross checked
Links are updated now

7

Thanks but its not what i am looking ?
i am trying to run inferrence request for lab:C4W2 kubernets lab
curl.exe -d ‘{"instances": [1.0, 2.0, 5.0]}’ -X POST “http://127.0.0.1:58554/v1/models/half_plus_two:predict”

curl: (7) Failed to connect to 127.0.0.1 port 58554 after 2040 ms: Connection refused
getting this errror
=> service is up also tried to port forward but no luck
could u help what went wrong

8

I am trying on windows machine , service is up but unable to ping endpoint

9 
 minikube service tf-serving-service
|-----------|--------------------|----------------------|---------------------------|
| NAMESPACE |        NAME        |     TARGET PORT      |            URL            |
|-----------|--------------------|----------------------|---------------------------|
| default   | tf-serving-service | tf-serving-http/8501 | http://192.168.49.2:30001 |
|-----------|--------------------|----------------------|---------------------------|
* Starting tunnel for service tf-serving-service.
|-----------|--------------------|-------------|------------------------|
| NAMESPACE |        NAME        | TARGET PORT |          URL           |
|-----------|--------------------|-------------|------------------------|
| default   | tf-serving-service |             | http://127.0.0.1:64587 |
|-----------|--------------------|-------------|------------------------|
* Opening service default/tf-serving-service in default browser...
! Because you are using a Docker driver on windows, the terminal needs to be open to run it.
 curl.exe -d '{\"instances\": [1.0, 2.0, 5.0]}' -X POST http://127.0.0.1:64587/v1/models/half_plus_two:predict
curl: (7) Failed to connect to 127.0.0.1 port 64587 after 2020 ms: Connection refused
10 
kubectl get deploy -w
NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
tf-serving-deployment   0/1     1            0           14s
tf-serving-deployment   1/1     1            1           41s
tf-serving-deployment   0/1     1            0           55s

why my deployment is changing its state from on to off

11

kubectl decribe pods

Warning FailedMount 7s (x5 over 15s) kubelet MountVolume.SetUp failed for volume “tf-serving-volume” : hostPath type check failed: /var/tmp/saved_model_half_plus_two_cpu is not a directory

12

now mount is successfulbut still unable to hit the endpoint

13

also tried in postman still facing issue