Problems with tensorflow/keras/hyperas within the cluster

Hi everyone,
I have been trying for more than a week now to set up a conda virtual environmnet to run my code. Unfortunately, I keep running in errors due to incompatibilities between TensorFlow packages and Keras packages. I have tried recreating new environments with different version of pythons, of keras and of tensorflow but nothing changed (except maybe the kind of error). I even tried the virtual environment from my own laptop (in which everything work) on the cluster, but then it would start throwing errors on hpc05 when importing keras or hyperas.
I have talked with a friend programmer and he thinks the problem might lie in the absence of NVIDIA drivers and problems with the installation of tensorflow-gpu?? Could this be it?

What can I do to get a working environment?
Let me know about any information needed.

Thank you!

1 Like

Please share a minimal environment.yml that reproduces the problem or the list of packages that you are installing. Are you using hpc05 cluster? I don’t remember it having GPUs, or is my information not up to date?

import pandas as pd 
import numpy as np
import h5py
import matplotlib.pyplot as plt 
import tensorflow as tf
from sklearn.model_selection import train_test_split
from matplotlib import pyplot
import random
from scipy.stats import truncnorm
import json
import csv

from hyperopt import Trials, STATUS_OK, tpe
from hyperas import optim
from hyperas.distributions import choice, uniform

these are the packages I need, and I am having problem with ‘from hyperas import optim’ .

Yes, I am using the hpc05 cluster.

Sorry, I mean the environment, so the specification of the packages that you install with conda.

like, versions and such? Is there a specific command I can run on terminal to have such info? (I am really new to this, sorry!)

I think conda list should work.

1 Like
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
_openmp_mutex             5.1                       1_gnu  
_tflow_select             2.3.0                       mkl  
absl-py                   0.15.0             pyhd3eb1b0_0  
aiohttp                   3.8.1            py38h7f8727e_1  
aiosignal                 1.2.0              pyhd3eb1b0_0  
argon2-cffi               21.3.0             pyhd8ed1ab_0    conda-forge
argon2-cffi-bindings      21.2.0           py38h0a891b7_2    conda-forge
astor                     0.8.1            py38h06a4308_0  
asttokens                 2.0.5              pyhd8ed1ab_0    conda-forge
astunparse                1.6.3                      py_0  
async-timeout             4.0.1              pyhd3eb1b0_0  
attrs                     21.4.0             pyhd3eb1b0_0  
backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
backports                 1.0                        py_2    conda-forge
backports.functools_lru_cache 1.6.4              pyhd8ed1ab_0    conda-forge
beautifulsoup4            4.11.1             pyha770c72_0    conda-forge
blas                      1.0                         mkl  
bleach                    5.0.1              pyhd8ed1ab_0    conda-forge
blinker                   1.4              py38h06a4308_0  
brotlipy                  0.7.0           py38h27cfd23_1003  
c-ares                    1.18.1               h7f8727e_0  
ca-certificates           2022.6.15            ha878542_0    conda-forge
cachetools                4.2.2              pyhd3eb1b0_0  
certifi                   2022.6.15        py38h578d9bd_0    conda-forge
cffi                      1.15.1                   pypi_0    pypi
charset-normalizer        2.0.4              pyhd3eb1b0_0  
click                     8.0.4            py38h06a4308_0  
cloudpickle               2.1.0              pyhd8ed1ab_0    conda-forge
colorama                  0.4.5              pyhd8ed1ab_0    conda-forge
cryptography              37.0.1           py38h9ce1e76_0  
dataclasses               0.8                pyh6d0b6a4_7  
dbus                      1.13.18              hb2f20db_0  
debugpy                   1.6.2                    pypi_0    pypi
decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
entrypoints               0.4                pyhd8ed1ab_0    conda-forge
executing                 0.8.3              pyhd8ed1ab_0    conda-forge
expat                     2.4.8                h27087fc_0    conda-forge
fastjsonschema            2.15.3                   pypi_0    pypi
flit-core                 3.7.1              pyhd8ed1ab_0    conda-forge
fontconfig                2.14.0               h8e229c2_0    conda-forge
freetype                  2.10.4               h0708190_1    conda-forge
frozenlist                1.2.0            py38h7f8727e_0  
future                    0.18.2           py38h578d9bd_5    conda-forge
gast                      0.3.3                    pypi_0    pypi
gettext                   hf34092f_1004    conda-forge
glib                      2.58.3          py38h73cb85d_1004    conda-forge
google-auth               2.6.0              pyhd3eb1b0_0  
google-auth-oauthlib      0.4.4              pyhd3eb1b0_0  
google-pasta              0.2.0              pyhd3eb1b0_0  
grpcio                    1.32.0                   pypi_0    pypi
gst-plugins-base          1.14.0               hbbd80ab_1  
gstreamer                 1.14.0               hb453b48_1  
h5py                      2.10.0           py38hd6299e0_1  
hdf5                      1.10.6               hb1b8bf9_0  
hyperas                   0.4.1              pyh8c360ce_1    conda-forge
hyperopt                  0.2.7              pyhd8ed1ab_0    conda-forge
icu                       58.2              hf484d3e_1000    conda-forge
idna                      3.3                pyhd3eb1b0_0  
importlib-metadata        4.11.3           py38h06a4308_0  
importlib_resources       5.8.0              pyhd8ed1ab_0    conda-forge
intel-openmp              2021.4.0          h06a4308_3561  
ipykernel                 6.15.1             pyh210e3f2_0    conda-forge
ipython                   8.4.0            py38h578d9bd_0    conda-forge
ipython_genutils          0.2.0                      py_1    conda-forge
ipywidgets                7.7.1              pyhd8ed1ab_0    conda-forge
jedi                      0.18.1           py38h578d9bd_1    conda-forge
jinja2                    3.0.3              pyhd3eb1b0_0  
jpeg                      9e                   h166bdaf_1    conda-forge
jsonschema                4.7.2              pyhd8ed1ab_0    conda-forge
jupyter                   1.0.0            py38h578d9bd_7    conda-forge
jupyter-client            7.3.4                    pypi_0    pypi
jupyter_client            7.0.6              pyhd8ed1ab_0    conda-forge
jupyter_console           6.4.4              pyhd8ed1ab_0    conda-forge
jupyter_core              4.11.1           py38h578d9bd_0    conda-forge
jupyterlab_pygments       0.2.2              pyhd8ed1ab_0    conda-forge
jupyterlab_widgets        1.1.1              pyhd8ed1ab_0    conda-forge
keras                     2.3.1                         0  
keras-applications        1.0.8                      py_1  
keras-base                2.3.1                    py38_0  
keras-preprocessing       1.1.2              pyhd3eb1b0_0  
ld_impl_linux-64          2.38                 h1181459_1  
libedit                   3.1.20210910         h7f8727e_0  
libffi                    3.2.1             hf484d3e_1007  
libgcc-ng                 11.2.0               h1234567_1  
libgfortran-ng            7.5.0               ha8ba4b0_17  
libgfortran4              7.5.0               ha8ba4b0_17  
libgomp                   11.2.0               h1234567_1  
libiconv                  1.16                 h516909a_0    conda-forge
libpng                    1.6.37               h21135ba_2    conda-forge
libprotobuf               3.20.1               h4ff587b_0  
libsodium                 1.0.18               h36c2ea0_1    conda-forge
libstdcxx-ng              11.2.0               h1234567_1  
libuuid                   2.32.1            h7f98852_1000    conda-forge
libxcb                    1.13              h7f98852_1004    conda-forge
libxml2                   2.9.14               h74e7548_0  
markdown                  3.3.4            py38h06a4308_0  
markupsafe                2.1.1            py38h7f8727e_0  
matplotlib-inline         0.1.3              pyhd8ed1ab_0    conda-forge
mistune                   0.8.4           py38h497a2fe_1005    conda-forge
mkl                       2021.4.0           h06a4308_640  
mkl-service               2.4.0            py38h7f8727e_0  
mkl_fft                   1.3.1            py38hd3c417c_0  
mkl_random                1.2.2            py38h51133e4_0  
multidict                 5.2.0            py38h7f8727e_2  
nbclient                  0.6.6              pyhd8ed1ab_0    conda-forge
nbconvert                 6.5.0              pyhd8ed1ab_0    conda-forge
nbconvert-core            6.5.0              pyhd8ed1ab_0    conda-forge
nbconvert-pandoc          6.5.0              pyhd8ed1ab_0    conda-forge
nbformat                  5.4.0              pyhd8ed1ab_0    conda-forge
ncurses                   6.3                  h5eee18b_3  
nest-asyncio              1.5.5              pyhd8ed1ab_0    conda-forge
networkx                  2.8.5              pyhd8ed1ab_0    conda-forge
notebook                  6.4.12             pyha770c72_0    conda-forge
numpy                     1.19.5                   pypi_0    pypi
numpy-base                1.22.3           py38hf524024_0  
oauthlib                  3.1.0                      py_0  
openssl                   1.1.1l               h7f98852_0    conda-forge
opt_einsum                3.3.0              pyhd3eb1b0_1  
packaging                 21.3               pyhd8ed1ab_0    conda-forge
pandoc                    2.18                 ha770c72_0    conda-forge
pandocfilters             1.5.0              pyhd8ed1ab_0    conda-forge
parso                     0.8.3              pyhd8ed1ab_0    conda-forge
pcre                      8.45                 h9c3ff4c_0    conda-forge
pexpect                   4.8.0              pyh9f0ad1d_2    conda-forge
pickleshare               0.7.5                   py_1003    conda-forge
pip                       22.1.2           py38h06a4308_0  
prometheus_client         0.14.1             pyhd8ed1ab_0    conda-forge
prompt-toolkit            3.0.30             pyha770c72_0    conda-forge
prompt_toolkit            3.0.30               hd8ed1ab_0    conda-forge
protobuf                  3.20.1           py38h295c915_0  
psutil                    5.9.1            py38h0a891b7_0    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
pure_eval                 0.2.2              pyhd8ed1ab_0    conda-forge
py4j                       pyhd8ed1ab_0    conda-forge
pyasn1                    0.4.8              pyhd3eb1b0_0  
pyasn1-modules            0.2.8                      py_0  
pycparser                 2.21               pyhd3eb1b0_0  
pygments                  2.12.0             pyhd8ed1ab_0    conda-forge
pyjwt                     2.4.0            py38h06a4308_0  
pyopenssl                 22.0.0             pyhd3eb1b0_0  
pyparsing                 3.0.9              pyhd8ed1ab_0    conda-forge
pyqt                      5.9.2            py38h05f1152_4  
pyrsistent                0.18.1           py38h0a891b7_1    conda-forge
pysocks                   1.7.1            py38h06a4308_0  
python                    3.8.1                h0371630_1  
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-fastjsonschema     2.16.1             pyhd8ed1ab_0    conda-forge
python-flatbuffers        2.0                pyhd3eb1b0_0  
python_abi                3.8                      2_cp38    conda-forge
pyyaml                    6.0              py38h7f8727e_1  
pyzmq                     19.0.2           py38ha71036d_2    conda-forge
qt                        5.9.7                h5867ecd_1  
qtconsole                 5.3.1              pyhd8ed1ab_0    conda-forge
qtconsole-base            5.3.1              pyha770c72_0    conda-forge
qtpy                      2.1.0              pyhd8ed1ab_0    conda-forge
readline                  7.0                  h7b6447c_5  
requests                  2.28.1           py38h06a4308_0  
requests-oauthlib         1.3.0                      py_0  
rsa                       4.7.2              pyhd3eb1b0_1  
scipy                     1.7.3            py38hc147768_0  
send2trash                1.8.0              pyhd8ed1ab_0    conda-forge
setuptools                61.2.0           py38h06a4308_0  
sip                       4.19.13          py38h295c915_0  
six                       1.15.0                   pypi_0    pypi
soupsieve                 2.3.2.post1        pyhd8ed1ab_0    conda-forge
sqlite                    3.33.0               h62c20be_0  
stack_data                0.3.0              pyhd8ed1ab_0    conda-forge
tensorboard               2.4.0              pyhc547734_0  
tensorboard-plugin-wit    1.6.0                      py_0  
tensorflow                2.4.1           mkl_py38hb2083e0_0  
tensorflow-base           2.4.1           mkl_py38h43e0292_0  
tensorflow-estimator      2.4.0                    pypi_0    pypi
termcolor                 1.1.0            py38h06a4308_1  
terminado                 0.15.0           py38h578d9bd_0    conda-forge
tinycss2                  1.1.1              pyhd8ed1ab_0    conda-forge
tk                        8.6.12               h1ccaba5_0  
tornado                   6.2                      pypi_0    pypi
tqdm                      4.64.0             pyhd8ed1ab_0    conda-forge
traitlets                 5.3.0              pyhd8ed1ab_0    conda-forge
typing-extensions                  pypi_0    pypi
typing_extensions         4.1.1              pyh06a4308_0  
urllib3                   1.26.9           py38h06a4308_0  
wcwidth                   0.2.5              pyh9f0ad1d_2    conda-forge
webencodings              0.5.1                    pypi_0    pypi
werkzeug                  2.0.3              pyhd3eb1b0_0  
wheel                     0.37.1             pyhd3eb1b0_0  
widgetsnbextension        3.6.1              pyha770c72_0    conda-forge
wrapt                     1.12.1                   pypi_0    pypi
xorg-libxau               1.0.9                h7f98852_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xz                        5.2.5                h7f8727e_1  
yaml                      0.2.5                h7b6447c_0  
yarl                      1.6.3            py38h27cfd23_0  
zeromq                    4.3.4                h9c3ff4c_1    conda-forge
zipp                      3.8.0            py38h06a4308_0  
zlib                      1.2.12               h7f8727e_2  

here there are

Can you try with python 3.10? You are using 3.8, which is very old by now, and packages might get obsolete.

I had tried with python 3.9 without success, now I am gonna try with python 3.10 as you suggested