Discussion:
[Py2exe-users] Failing to identify subpackage module
David Bolen
2005-08-02 19:39:29 UTC
Permalink
I've been experimenting with converting our reasonably large system
over from Installer (which we've been using for a long time) to
py2exe, but I've been running into a strange problem with py2exe
failing to identify and include a particular module which seems to be
trivial to locate to me (and which Installer doesn't have a problem
with). The problem may be more an issue with the underlying
modulefinder module (since if I try to use that directly I get a
similar issue), but I'm not sure where else to inquire.

The problem showed up in a large scale system, but I've managed to
create a small example - it's a somewhat pointless hierarchy, but
realize that it's just my simplest form of what is actually a useful
hierarchy in my actual code.

If I have the following package structure:

a/__init__.py
resource.py

/c/__init__.py
resource.py

All of the package files are empty placeholders with the exception of
a/__init__.py which contains:

from resource import *

from a.c import resource

and a top level script (main.py) that contains:

import a

if __name__ == "__main__":
print 'hello'

and its matching setup.py:

from distutils.core import setup
import py2exe

setup(console=['main.py'])

Then running "python setup.py py2exe" processes the script and creates
the appropriate output in the "dist" directory, but fails to include
the a.c.resource module, although the rest of the a and c packages are
included. It does spit out at the end of processing:

The following modules appear to be missing
['a.c.resource']

so clearly it recognizes it's missing, but I can't figure out why it
can't find it.

In case it means anything, if I were to place an "import resource"
statement in the a/c/__init__.py module, then the above error message
changes to just 'resource' rather than 'a.c.resource', but in neither
case is the sub-package resource file included.

Can anyone offer any suggestions? This originally came about in our
application that has our own package which has an internal
"resource.py" module, but which also includes the twisted.web.resource
module (which I've replaced with the a.c module in the example here).

There are lots of sibling imports both in our packages and in twisted
(for example) none of which seem to cause a problem, except when one
is referenced in two ways in a single module as in this example.

Is there any way to get more verbose output from py2exe during the
scanning that is done and/or decisions made during the scanning
process?

Thanks for any suggestions.

-- David
Thomas Heller
2005-08-03 09:29:50 UTC
Permalink
Post by David Bolen
I've been experimenting with converting our reasonably large system
over from Installer (which we've been using for a long time) to
py2exe, but I've been running into a strange problem with py2exe
failing to identify and include a particular module which seems to be
trivial to locate to me (and which Installer doesn't have a problem
with). The problem may be more an issue with the underlying
modulefinder module (since if I try to use that directly I get a
similar issue), but I'm not sure where else to inquire.
The problem showed up in a large scale system, but I've managed to
create a small example - it's a somewhat pointless hierarchy, but
realize that it's just my simplest form of what is actually a useful
hierarchy in my actual code.
Can you please upload the example as zip-file somewhere the the bug
tracker? Either Python (for modulefinder) or py2exe. I promise to look
into it.

Thomas
Denis Barmenkov
2005-11-30 15:23:30 UTC
Permalink
Post by Thomas Heller
Can you please upload the example as zip-file somewhere the the bug
tracker? Either Python (for modulefinder) or py2exe. I promise to look
into it.
Thomas
Hi,

I found similar problem:

file test.py:
==========================
from Crypto.Cipher import *

print 'ok'
==========================

file setup_test.py:
==========================
from distutils.core import setup
import py2exe

setup(
version = "0.1",
description = "none",
name = 'none',

options = {"py2exe": {"compressed": 1,
"optimize": 2,
# "ascii": 1,
"bundle_files": 1}},

console = ["test.py"],
zipfile = None,
)
==========================

Crypto packages only:
...
skipping byte-compilation of \
d:\Python24\lib\site-packages\Crypto\Cipher\__init__.py to \
Crypto\Cipher\__init__.pyo
skipping byte-compilation of \
d:\Python24\lib\site-packages\Crypto\__init__.py to \
Crypto\__init__.pyo
...

And no occurences of d:\Python24\lib\site-packages\Crypto\Cipher\*.pyd
(compiled parts)



py2exe version 0.6.1 (0.6.3 was instaled before, and works wrong as now)
python 2.4 (ActivePython)

Is there a fiz availavble, or I should repeat install-check-remove triplets from
later py2exe's versions to earlier?

Thank you.

regards,
Denis Barmenkov
David Bolen
2005-08-03 18:02:19 UTC
Permalink
Post by Thomas Heller
Can you please upload the example as zip-file somewhere the the bug
tracker? Either Python (for modulefinder) or py2exe. I promise to look
into it.
Crud - while packaging for the bug report I found a bug in my
simplified version (misnamed module), so that my message as written
probably doesn't fail for anyone else. It still fails consistently
when using twisted, but I spent an hour or two trying to work my way
through the twisted base to again attempt a simpler example to no
effect yet. (twisted's imports remind me of Adventure's "a maze of
twisty little passages, all alike"). I'm guessing it's related to
recursive imports somewhere along the line, since a lot of that
happens in the Twisted core due to inter-dependencies between
sub-packages.

I don't suppose you have a copy of twisted (either 1.3.0 or 2.x seem
the same) on any system do you? If so, then replacing the "from a.c
import resource" with "from twisted.web import resource" in a's
__init__ should show the problem. If not, I'll keep digging to see
if I can create a simplified use case of some sort.

Also, for what it's worth, py2app under OS X doesn't exhibit the same
problem. I know it uses a modified version of modulefinder
(modulegraph) so it would seem like some of those modifications
resolved whatever is happening under the covers here. Not sure if
you've ever considered trying modulegraph in a subsequent release of
py2exe or not?

-- David
Thomas Heller
2005-08-04 20:23:14 UTC
Permalink
Post by David Bolen
Post by Thomas Heller
Can you please upload the example as zip-file somewhere the the bug
tracker? Either Python (for modulefinder) or py2exe. I promise to look
into it.
Crud - while packaging for the bug report I found a bug in my
simplified version (misnamed module), so that my message as written
probably doesn't fail for anyone else. It still fails consistently
when using twisted, but I spent an hour or two trying to work my way
through the twisted base to again attempt a simpler example to no
effect yet. (twisted's imports remind me of Adventure's "a maze of
twisty little passages, all alike"). I'm guessing it's related to
recursive imports somewhere along the line, since a lot of that
happens in the Twisted core due to inter-dependencies between
sub-packages.
I don't suppose you have a copy of twisted (either 1.3.0 or 2.x seem
the same) on any system do you? If so, then replacing the "from a.c
import resource" with "from twisted.web import resource" in a's
__init__ should show the problem. If not, I'll keep digging to see
if I can create a simplified use case of some sort.
David,

I have tried to build the a.c package as you described, and installed
twisted 1.3.0 for Python 2.4.1. Still, the main.exe file builds fine.
Here are the last lines of the py2exe output:

The following modules appear to be missing

['Crypto.Cipher', 'FCNTL', 'OpenSSL', 'jarray', 'java.io.IOException',
'java.io.InterruptedIOException', 'java.net', 'resource']

The missing 'resource' module may be related to the error that YOU get,
but basically it is correct. It refers to these lines in
twisted\internet\process.py file, near line 409:

try:
import resource
maxfds = resource.getrlimit(resource.RLIMIT_NOFILE)[1] + 1
# OS-X reports 9223372036854775808. That's a lot of fds to close
if maxfds > 1024:
maxfds = 1024
Post by David Bolen
Also, for what it's worth, py2app under OS X doesn't exhibit the same
problem. I know it uses a modified version of modulefinder
(modulegraph) so it would seem like some of those modifications
resolved whatever is happening under the covers here. Not sure if
you've ever considered trying modulegraph in a subsequent release of
py2exe or not?
The resource module that is imported here is the top-level stdlib
resource module, which is only available in Unix (and OSX, I assume).

So it may be that Unix and OSX doen't have this problem because the
builtin resource module is found, or it may be that py2app's modulegraph
works better than modulefinder.

Still, I would really like you to post a reproducible testcase for
Windows, because I HAVE seen cases where to many careless imports fail
in the py2exe'd app. I'm willing to install even other packages, if it
is required ;-)

Thomas
David Bolen
2005-08-11 00:11:11 UTC
Permalink
Post by Thomas Heller
Still, I would really like you to post a reproducible testcase for
Windows, because I HAVE seen cases where to many careless imports fail
in the py2exe'd app. I'm willing to install even other packages, if it
is required ;-)
Just for reference in the archives, I followed up on this with Thomas
directly with some of my original code base, and he tracked it down to
an issue with the Python modulefinder.py library module. The missing
import inside of twisted (per Thomas' last response) was flagged as a
badmodule (as the non-qualified "resource") that then filtered out any
other non-qualified "resource" imports even if they were in different
packages and did have matching modules.

The fix was to remove the 2 checks in _safe_import_hook that skipped
over modules in self.badmodules - hopefully that will make it into the
Python source tree at some point going forward.

-- David
Jimmy Retzlaff
2005-12-01 18:41:07 UTC
Permalink
Post by Denis Barmenkov
==========================
from Crypto.Cipher import *
print 'ok'
Because of the way Crypto.Cipher references those extension modules
(they are never explicitly imported), py2exe doesn't know they are being
imported. If you explicitly reference the ciphers you need then it
works:

file test.py:
==========================
from Crypto.Cipher import AES, Blowfish

print dir(AES)
print dir(Blowfish)


This works fine for me with your setup.py.

Jimmy
Denis Barmenkov
2005-12-02 09:24:00 UTC
Permalink
"Jimmy Retzlaff" <***@retzlaff.com>
01.12.2005 21:41


Кому: "Denis Barmenkov" <***@bpc.ru>, <py2exe-***@lists.sourceforge.net>
Копия:
Тема: RE: [Py2exe-users] Re: Failing to identify subpackage module
Post by Denis Barmenkov
==========================
from Crypto.Cipher import *
print 'ok'
Because of the way Crypto.Cipher references those extension modules
(they are never explicitly imported), py2exe doesn't know they are being
imported. If you explicitly reference the ciphers you need then it
works:

file test.py:
==========================
from Crypto.Cipher import AES, Blowfish

print dir(AES)
print dir(Blowfish)


This works fine for me with your setup.py.

Jimmy
==================================================

Thank you, this doesnt work:

1. file changed to:
--------------------------------
from Crypto.Cipher import AES, ARC2, ARC4, Blowfish, CAST, DES, DES3,
IDEA, RC5, XOR

print 'ok'
--------------------------------

2. exe created

3. run error occured:
Traceback (most recent call last):
File "test.py", line 2, in ?
ImportError: cannot import name AES

4. py2exe's output contain:
copying d:\Python24\lib\site-packages\Crypto\Cipher\AES.pyd ->
С:\projects\parse_log\build\bdist.win32\winexe\collect-2.4

Should py2exe copy AES.pyd to ..\collect-2.4\Crypto\Cipher directory as in
source Python/
Yaroslav Samchuk
2005-12-02 11:13:12 UTC
Permalink
Post by Denis Barmenkov
--------------------------------
from Crypto.Cipher import AES, ARC2, ARC4, Blowfish, CAST, DES, DES3,
IDEA, RC5, XOR
print 'ok'
--------------------------------
2. exe created
File "test.py", line 2, in ?
ImportError: cannot import name AES
copying d:\Python24\lib\site-packages\Crypto\Cipher\AES.pyd ->
С:\projects\parse_log\build\bdist.win32\winexe\collect-2.4
Your script with the very minimalistic setup.py (I'm sure generic one
will also show the same results) works fine for me.

=== setup.py ===
from distutils.core import setup
import py2exe
setup(console=["test.py"],)
================

actually, you could use from spam.eggs import *, but in this case you'll
have to play with `includes` and `packages` py2exe options. smth like

=========
hiddenimports = [
"_strptime",
"compileall",
"select",
"zlib",
...
]

hiddenpackages = [
"spam.eggs.larch",
...
]

...
def run_setup():
setup(
...,
options={"py2exe": {
"includes": hiddenimports,
"packages": hiddenpackages
}},
...,
)
...
=========

This was a part of setup.py (py2exe version 0.5.5) script I use for our
software product.
Post by Denis Barmenkov
Should py2exe copy AES.pyd to ..\collect-2.4\Crypto\Cipher directory as in
source Python/Lib tree?
Content of my E:\...st.win32\winexe\collect-2.4\Crypto\Cipher
=========
__init__.pyc
AES.pyc
ARC2.pyc
ARC4.pyc
Blowfish.pyc
CAST.pyc
DES.pyc
DES3.pyc
IDEA.pyc
RC5.pyc
XOR.pyc
=========

Note pyc files instead of pyd!
--
Best wishes,
Yaroslav
Jimmy Retzlaff
2005-12-02 14:21:43 UTC
Permalink
Post by Denis Barmenkov
01.12.2005 21:41
Post by Jimmy Retzlaff
Post by Denis Barmenkov
==========================
from Crypto.Cipher import *
print 'ok'
Because of the way Crypto.Cipher references those extension modules
(they are never explicitly imported), py2exe doesn't know they are being
imported. If you explicitly reference the ciphers you need then it
==========================
from Crypto.Cipher import AES, Blowfish
print dir(AES)
print dir(Blowfish)
This works fine for me with your setup.py.
Jimmy
==================================================
--------------------------------
from Crypto.Cipher import AES, ARC2, ARC4, Blowfish, CAST, DES, DES3,
IDEA, RC5, XOR
print 'ok'
--------------------------------
2. exe created
File "test.py", line 2, in ?
ImportError: cannot import name AES
Your new test.py still works for me (and apparently for Yaroslav as well). I'm using Python 2.4.2, py2exe 0.6.3, and PyCrypto 2.0.1 as compiled for Python 2.4 by Fuzzyman at:

http://www.voidspace.org.uk/python/modules.shtml#pycrypto

You might try deleting your build and dist folders as that can sometimes help.
Post by Denis Barmenkov
copying d:\Python24\lib\site-packages\Crypto\Cipher\AES.pyd ->
С:\projects\parse_log\build\bdist.win32\winexe\collect-2.4
Should py2exe copy AES.pyd to ..\collect-2.4\Crypto\Cipher directory as in
source Python/Lib tree?
Yes they should go into the Cipher folder (they do on mine). Try clearing build and dist and if it still doesn't work let us know how your versions of things differ from what I presented above.

Jimmy
Denis Barmenkov
2005-12-02 15:45:13 UTC
Permalink
"Jimmy Retzlaff" <***@retzlaff.com>
02.12.2005 17:21


Кому: "Denis Barmenkov" <***@bpc.ru>, <py2exe-***@lists.sourceforge.net>
Копия:
Тема: RE: [Py2exe-users] Re: Failing to identify subpackage module
Post by Denis Barmenkov
01.12.2005 21:41
Post by Jimmy Retzlaff
Post by Denis Barmenkov
==========================
from Crypto.Cipher import *
print 'ok'
Because of the way Crypto.Cipher references those extension modules
(they are never explicitly imported), py2exe doesn't know they are
being
Post by Denis Barmenkov
Post by Jimmy Retzlaff
imported. If you explicitly reference the ciphers you need then it
==========================
from Crypto.Cipher import AES, Blowfish
print dir(AES)
print dir(Blowfish)
This works fine for me with your setup.py.
Jimmy
==================================================
--------------------------------
from Crypto.Cipher import AES, ARC2, ARC4, Blowfish, CAST, DES, DES3,
IDEA, RC5, XOR
print 'ok'
--------------------------------
2. exe created
File "test.py", line 2, in ?
ImportError: cannot import name AES
Your new test.py still works for me (and apparently for Yaroslav as well).
I'm using Python 2.4.2, py2exe 0.6.3, and PyCrypto 2.0.1 as compiled for
Python 2.4 by Fuzzyman at:

http://www.voidspace.org.uk/python/modules.shtml#pycrypto

You might try deleting your build and dist folders as that can sometimes
help.
Post by Denis Barmenkov
copying d:\Python24\lib\site-packages\Crypto\Cipher\AES.pyd ->
С:\projects\parse_log\build\bdist.win32\winexe\collect-2.4
Should py2exe copy AES.pyd to ..\collect-2.4\Crypto\Cipher directory as
in
Post by Denis Barmenkov
source Python/Lib tree?
Yes they should go into the Cipher folder (they do on mine). Try clearing
build and dist and if it still doesn't work let us know how your versions
of things differ from what I presented above.

Jimmy
==========================

Thank you, Jimmy!

I re-installed 0.6.3, and yow all works fine!

Rega

Loading...