C4W3 Graded Item: TFX on Google Cloud Vertex Pipelines

[lallimyl]

Hello!

When working on the assignment, running the cell that imports TF, TFX and KFP results in error “AttributeError: module ‘tensorflow._api.v1.experimental’ has no attribute ‘register_filesystem_plugin’”. Here’s the traceback:

AttributeError Traceback (most recent call last)
/tmp/ipykernel_11706/66828112.py in
3 import tensorflow as tf
4 print(‘TensorFlow version: {}’.format(tf.version))
----> 5 from tfx import v1 as tfx
6 print(‘TFX version: {}’.format(tfx.version))
7 import kfp

/opt/conda/lib/python3.7/site-packages/tfx/v1/init.py in
18 try:
19 # These modules may not be available in some environments.
—> 20 from tfx.v1 import components
21 from tfx.v1 import extensions
22 from tfx.v1 import orchestration

/opt/conda/lib/python3.7/site-packages/tfx/v1/components/init.py in
27 from tfx.components.statistics_gen.component import StatisticsGen
28 from tfx.components.trainer.component import Trainer
—> 29 from tfx.components.transform.component import Transform
30 from tfx.components.tuner.component import Tuner
31

/opt/conda/lib/python3.7/site-packages/tfx/components/transform/component.py in
17
18 from tfx import types
—> 19 from tfx.components.transform import executor
20 from tfx.components.util import udf_utils
21 from tfx.dsl.components.base import base_beam_component

/opt/conda/lib/python3.7/site-packages/tfx/components/transform/executor.py in
23 import tensorflow as tf
24 import tensorflow_data_validation as tfdv
—> 25 import tensorflow_transform as tft
26 from tensorflow_transform import impl_helper
27 import tensorflow_transform.beam as tft_beam

/opt/conda/lib/python3.7/site-packages/tensorflow_transform/init.py in
34 # filesystem.
35 try:
—> 36 import tensorflow_io as _ # pytype: disable=import-error # pylint: disable=g-import-not-at-top
37 except ModuleNotFoundError:
38 pass

/opt/conda/lib/python3.7/site-packages/tensorflow_io/init.py in
15 “”“tensorflow_io”“”
16
—> 17 from tensorflow_io.python.api import * # pylint: disable=wildcard-import
18 from tensorflow_io.python.api.version import VERSION as version

/opt/conda/lib/python3.7/site-packages/tensorflow_io/python/api/init.py in
17
18 # tensorflow_io.core.python.ops is implicitly imported (along with file system)
—> 19 from tensorflow_io.python.ops.io_dataset import IODataset
20 from tensorflow_io.python.ops.io_tensor import IOTensor
21

/opt/conda/lib/python3.7/site-packages/tensorflow_io/python/ops/init.py in
94 core_ops = LazyLoader(“core_ops”, “libtensorflow_io.so”)
95 try:
—> 96 plugin_ops = _load_library(“libtensorflow_io_plugins.so”, “fs”)
97 except NotImplementedError as e:
98 warnings.warn(“unable to load libtensorflow_io_plugins.so: {}”.format(e))

/opt/conda/lib/python3.7/site-packages/tensorflow_io/python/ops/init.py in _load_library(filename, lib)
62 for f in filenames:
63 try:
—> 64 l = load_fn(f)
65 if l is not None:
66 return l

/opt/conda/lib/python3.7/site-packages/tensorflow_io/python/ops/init.py in (f)
54 load_fn = lambda f: ctypes.CDLL(f, mode=ctypes.RTLD_GLOBAL)
55 elif lib == “fs”:
—> 56 load_fn = lambda f: tf.experimental.register_filesystem_plugin(f) is None
57 else:
58 load_fn = lambda f: tf.compat.v1.load_file_system_library(f) is None

/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/util/module_wrapper.py in getattr(self, name)

AttributeError: module ‘tensorflow._api.v1.experimental’ has no attribute ‘register_filesystem_plugin’

I have tried to upgrade different components (tf, tensorboard, etc), and I have followed instructions here to solve pip’s dependency issues.

I have tried to restart the kernel multiple times, and I have ended and restarted the entire lab, but the problem remains. I’m out of ideas.

Thank you in advance!

[lallimyl]

Okay, seems I managed after all. Ended up uninstalling and installing stuff. First cell looked like this:

!pip install --upgrade pip
!pip install tangled-up-in-unicode==0.1.0
!pip install numpy==1.20
!pip install google-api-python-client==1.8.0
!pip install xai-image-widget
!pip install --upgrade “tfx[kfp]<2”
!pip uninstall shapely -y # not sure if this actually did anything. in another practice notebook they say that this may fix an import error but they do not specify which one and when.

As that did not do the trick, I iterated a lot of stuff in the third cell that finally looked like this:

!pip uninstall tensorflow tensorflow-tensorboard tensorflow-io tensorflow-cloud -y
!pip install tensorflow==1.15.5

!pip uninstall tfx -y
!pip install --upgrade “tfx[kfp]<2”

import tensorflow as tf
print(‘TensorFlow version: {}’.format(tf.version))
from tfx import v1 as tfx
print(‘TFX version: {}’.format(tfx.version))
import kfp
print(‘KFP version: {}’.format(kfp.version))

I ended up with that by trying to dig which version of TF seemed to cause the least dependency issues in Pip that made sense (e.g., iirc, which tfx would like etc), and then uninstalled every package mentioned in the remaining issues before installing the 1.15.5 and tfx.

So, a bit of a brute force, lot’s of trial and error, and a pinch of luck.

[Peter_Grabner]

Thanks

[arvyzukai]

Thanks for sharing @lallimyl

I encountered the same problem (on 2023-10-24) and modifying just the first code cell is enough:

# Use the latest version of pip.
!pip install --upgrade pip
!pip uninstall tensorflow tensorflow-tensorboard tensorflow-io tensorflow-cloud -y
!pip install tensorflow==1.15.5

!pip uninstall tfx -y
!pip install --upgrade "tfx[kfp]<2"

Also, for future readers, the run was on the ‘us-east1’ Region (different than the default value) for me (in case you want to monitor Vertex AI > Pipelines during the run) .

Cheers