Python: Why Does Importing A Package Sometimes Grant Access To Modules Underneath It But Sometimes Not?

Python import mechanism is always a myth for me. Sometimes importing a package can grant access to the modules underneath it. For instance, import urllib urllib.parse.unquote give

Solution 1:

For an access to urllib.parse to work, the following two conditions must be true:

1. The urllib module object must be bound to the urllib name, whether in the local, global, or some enclosing scope.
2. The urllib.parse submodule must have been initialized and bound to the parse attribute of the urllib module object.

An import urllib in the current local or global scope (or any enclosing scope) satisfies the first condition.

An import urllib.parse executed anywhere in the program satisfies the second condition, since it loads the submodule and binds it to the parse attribute on the urllib module object, and there's only one urllib module object for the whole program.

In the environments where urllib.parse was accessible after a simple import urllib, some other code must have loaded urllib.parse, causing you to see it.

Solution 2:

Like user2357112 said it's getting imported; I believe these are the specific modules and statements.

Test:"import IPython"
 └─IPython:┐
      ┌────┘
      ├──"from core.application import Application"
      │   └──IPython.core.application: "from IPython.core import release, crashhandler"
      │      └──IPython.core.crashhandler: "from IPython.core import ultratb"
      │         └──IPython.core.ultratb: "import pydoc"
      │            └──pydoc: "import urllib.parse"
      └──"from terminal.embed import embed"
          └──IPython.terminal.embed:┐
                        ┌───────────┘
                        ├──"from IPython.core import magic_arguments"
                        │   └──IPython.core.magic_arguments: "from IPython.utils.text import dedent"
                        │      └──IPython.utils.text: "from pathlib import Path"
                        │         └──pathlib: "from urllib.parse import quote_from_bytes"
                        ├──"from IPython.core.magic import Magics, magics_class, line_magic"
                        │   └──IPython.core.magic: "from IPython.core import oinspect"
                        │      └──IPython.core.oinspect: "from IPython.core import page"
                        │         └──IPython.core.page: "from IPython.core.display import display"
                        │            └──IPython.core.display: "import mimetypes"
                        │               └──mimetypes: "import urllib.parse"
                        └──"from IPython.terminal.interactiveshell import TerminalInteractiveShell"
                            └──pygments.plugin: "import pkg_resources"
                               └──pkg_resources: "import email.parser"
                                  └──email.parser: "from email.feedparser import FeedParser, BytesFeedParser"
                                     └──email.feedparser: "from email._policybase import compat32"
                                        └──email._policybase: "from email.utils import _has_surrogates"
                                           └──email.utils: "import urllib.parse"

Solution 3:

Python 3 does not load the helper modules for urllib automatically. ( https://docs.python.org/2/library/urllib.html )

"Note The urllib module has been split into parts and renamed in Python 3 to urllib.request, urllib.parse, and urllib.error. The 2to3 tool will automatically adapt imports when converting your sources to Python 3."

"Note urllib also exposes certain utility functions like splittype, splithost and others parsing URL into various components. But it is recommended to use urlparse for parsing URLs rather than using these functions directly. Python 3 does not expose these helper functions from urllib.parse module."

---

If you attempt to query the urllib namespace dir(urllib) after import, there are no submodules. After you type urllib.parse.unquote and get the error, the urllib helper modules are loaded. (I'm serious, that sounds crazy and wrong, all things not Python, "he's a n00b", just try it.) You can see them in the namespace through dir(urllib) and can query use them as if they were all loaded initially. You will then get the function object return.

Python 3.5.2 (default, Aug 18 2017, 17:48:00)[GCC 5.4.0 20160609] on linuxType "help", "copyright", "credits" or "license" for more information.

>>> import urllib

>>> urllib.parse.unquote

Traceback (most recent call last):File "<stdin>", line 1, in <module>AttributeError: module 'urllib' has no attribute 'parse'

>>> urllib.parse.unquote

<function unquote at 0x7f559e4768c8>

---

In the six module, there are builtins.module(builtins.object)

    Module_six_moves_urllib
_LazyDescr(builtins.object)
    MovedAttribute
    MovedModule
_LazyModule(builtins.module)
    Module_six_moves_urllib_error
    Module_six_moves_urllib_parse
    Module_six_moves_urllib_request
    Module_six_moves_urllib_response
    Module_six_moves_urllib_robotparser

There is additional documentation (of course) such as

class Module_six_moves_urllib(builtins.module) " | Create a six.moves.urllib namespace that resembles the Python 3 namespace"

I suspect that terminal does not invoke the builtin to load the helper modules automatically where as Jupyter does though I honestly don't know.

Edit to add: Importing urllib, importing six and calling on it [even help("six")] will load the parse, request, response, robotparser modules to the urllib namespace. Additionally, importing urllib and calling help on it will load parse but not the other modules into the namespace. Jupyter may be loading help proactively causing it to load the parse module only. I don't have IPython/conda/Jupyter installed so can't help to test.