Paths for loading functions from a python "library"

Programming questions
1

Question: I’ve been working on the “datacube” library.

I’ve never made a “library” before in python, so I’m a bit of a noob, and I’m a bit confused about paths.

In my library, I keep the code for the class in a file datacube.py

If I am working inside the repo directory, I can do the following: from datacube import Datacube

I have now also linked the repo directory to ~/lib on the JH (which is by default in the path for all users BTW)

However, working from outside the repo folder, I need to run from datacube.datacube import Datacube

Is there some way to be able to load the library using from datacube import Datacube when I am working outside the repo?

Thanks!

1 Like
2

Nice library!

There may be some trick to deal with the paths, but I think ultimately even in the short term, I think your best solution is to make your library pip-installable by adding a minimal setup.py. It sounds like a lot of extra work, but actually isn’t.

Here’s all you need to provide (source):

from setuptools import setup

setup(name='funniest',
      version='0.1',
      description='The funniest joke in the world',
      url='http://github.com/storborg/funniest',
      author='Flying Circus',
      author_email='flyingcircus@example.com',
      license='MIT',
      packages=['funniest'],
      zip_safe=False)

You can still create symlink the project folder with the installed version by using pip install . --editable.

3

But let’s say that I want to use the gitlab “bleeding edge” code in my notebooks that I pull to ~/lib.

Is there a solution for that?

4

Even then I recommend to install code not just in an arbitrary folder, but somewhere, where pip recognizes this, e.g. a shared conda environment.

This you can easily achieve installing bleeding edge code with pip install git+https://gitlab.tudelft.nl/steelelab/datacube.git@master (the part after @ can be any reference that git recognizes.

5

you’re strict! i’ll have a look at it

6

It’s not really about strictness; rather before long this actually saves you work through avoiding headache.

7

Thanks, it was indeed not that hard :slight_smile:

I guess though that for it to be PIP installable for anybody via PyPI (without downloading the source), I just have to make it a public repo and then register it.

Are there general guidelines for putting stuff on PyPI?

(This is a pretty silly and specific library, but it might be still handy to be able to just pip install it)

Thanks,
Gary

8

See the pip command I shared above: to make a repository installable without cloning you don’t need to publish it on pypi: pip understands how to deal with git sources directly.

Definitely. You’ll need to provide some extra information in the setup, and you’ll need to upload the resulting package to pypi using twine.

To upload a new version to pypi you’ll need to run these commands:

python setup.py sdist
python setup.py bdist_wheel
# check that the tarballs in ./dist have everything needed and that the wheel is universal
twine upload dist/*

But then you’ll also want to manage versioning of your package, to make that simple, I suggest to get started with miniver (written by Joe).

9

OK, thanks, for now I’ll stick to pip installs from the git dir :slight_smile:

1 Like