Corrupted ZIP files during download via PHP

Hello,

I am posting this rather lengthy description here in hope of getting at
least some insight into a problem that is really hurting my business,
since I have come to the conclusion that my problem is at least partly
PHP-related.

Some time now I have been having transfer problems, i.e. my customers
downloads of large binary (ZIP) files are often corrupt.

A corrupted file now and then can of course never be avoided, but I get
several complaints from customers every day. Of any specific file,
approx. 1 out of 4 downloads (larger than 100kB or so) are affected.
I have spent months trying to find the source of the problems without
success, but I have been able to rule out many possible causes. Here are
some facts:

1) The actual files are OK on the server. Direct (unencrypted) links are
fine, so it's not a general problem.

2) About 50% of the customers download (the exact same files) from
another source. I don't even get 1 complaint every two months from those
customers, while I get 1-3 complaints _every day_ from those who
download from my server.

3) The customers have various SW/HW/OS, so it's very unlikely that it's
a user problem. Plus, the same customer often download from both places,
but only have problems with the files from my server.

4) I have shut down all scripts for a while, so that only a static
website remained, but it didn't help.

5) I have had this problem on three servers: The present RaQ550, the
previous RaQ4 and the RaQ4 I had before that. So it's highly unlikely
that it is server-related.

6) I can easily reproduce the problem myself. As I looked into this I
found a systematic error:

Analyzing corrupt vs. clean ZIP file downloads with a binary/hex editor
clearly shows that the error is systematic. Clean ZIP files have a lot
of '00h' bytes in them. In ALL of the corrupt ZIP files ALL (well, most)
of these bytes have been exchanged for '5C30h', i.e. TWO bytes ( 00h ->
5C30h ) !

Since '5C30h' equals an escaped zero, i.e. "\0", my guess is that there
is a translation error somewhere.

So, what does PHP have to do with this? Well, my download links to
purchaseware are encrypted by the PHP "ewportal":
http://www.eliteweaver.co.uk/antifra....php?page=Home

From the few facts I have been able to uncover it seems there is
sometimes a problem with corrupted ZIP files depending on the
setting/compilation of ZLIB. If anyone on this forum have experienced
similar things I would very much appreciate if they could share.
My servers PHP configuration is at:
http://www.snith.com/phptest.php

Best regards, Lars

*** Swede escribió/wrote (Tue, 15 Feb 2005 21:16:18 +0100):
> Some time now I have been having transfer problems, i.e. my customers
> downloads of large binary (ZIP) files are often corrupt.

I suggest that you check the server's MIME type for *.zip files.


> From the few facts I have been able to uncover it seems there is
> sometimes a problem with corrupted ZIP files depending on the
> setting/compilation of ZLIB.

Do you compress the files on the fly?


--
-+ Álvaro G. Vicario - Burgos, Spain
+- http://www.demogracia.com (la web de humor barnizada para la intemperie)
++ Manda tus dudas al grupo, no a mi buzón
-+ Send your questions to the group, not to my mailbox
--

Hello Alvaro,

Alvaro G. Vicario wrote:

> *** Swede escribió/wrote (Tue, 15 Feb 2005 21:16:18 +0100):
>
>>Some time now I have been having transfer problems, i.e. my customers
>>downloads of large binary (ZIP) files are often corrupt.
>
> I suggest that you check the server's MIME type for *.zip files.

I have already tried that, it was set to some non-descript application.
I changed it to "octet-stream" together with "*.exe" and the other
binaries, but it did not help. Is there any more place than "mime.types"
where I could change this?

I have seen some brief comments about "commenting out the header which
gets the mimetype of the file in class.download.inc", but I have no idea
how/where to do that, and if it is related to my problem.

>> From the few facts I have been able to uncover it seems there is
>>sometimes a problem with corrupted ZIP files depending on the
>>setting/compilation of ZLIB.
>
> Do you compress the files on the fly?

Not intentionally...they are already ZIPs on the server disk.

Thanks for your input so far, I think you are definitely on the right
path! Any more good ideas?


Best regards, Lars

*** Swede escribió/wrote (Tue, 15 Feb 2005 21:52:39 +0100):
> I have already tried that, it was set to some non-descript application.
> I changed it to "octet-stream" together with "*.exe" and the other
> binaries, but it did not help. Is there any more place than "mime.types"
> where I could change this?

My suggestion only came because maybe the server was using the default type
(usually text/html) and was messing with character encoding or God knows
what.

Anyway, since you seem to be using a PHP script to serve files then it's
the script the one who should be setting the MIME type; otherwise you'll
get the default type. Something similar to this should do:

header('Content-Type: application/zip');

Internet Explorer is known for having problems with PDF files (esp. if you
use sessions) but I'm not sure about ZIPs.



--
-+ Álvaro G. Vicario - Burgos, Spain
+- http://www.demogracia.com (la web de humor barnizada para la intemperie)
++ Manda tus dudas al grupo, no a mi buzón
-+ Send your questions to the group, not to my mailbox
--

Hello Again,

Alvaro G. Vicario wrote:

> *** Swede escribió/wrote (Tue, 15 Feb 2005 21:52:39 +0100):
>
>>I have already tried that, it was set to some non-descript application.
>>I changed it to "octet-stream" together with "*.exe" and the other
>>binaries, but it did not help. Is there any more place than "mime.types"
>>where I could change this?
>
> My suggestion only came because maybe the server was using the default type
> (usually text/html) and was messing with character encoding or God knows
> what.
>
> Anyway, since you seem to be using a PHP script to serve files then it's
> the script the one who should be setting the MIME type; otherwise you'll
> get the default type. Something similar to this should do:
>
> header('Content-Type: application/zip');

Sounds like a good idea, should this go into "php.ini", or in the script
itself ?

>
> Internet Explorer is known for having problems with PDF files (esp. if you
> use sessions) but I'm not sure about ZIPs.

Yes, IE causes a lot of problems, but I see this also on my Mozilla/Firefox.


Best regards, Lars

The conversion of binary zeros to "\0"s is consistent with
magic_quotes_runtime = On.

*** Swede escribió/wrote (Tue, 15 Feb 2005 22:15:08 +0100):
>> header('Content-Type: application/zip');
>
> Sounds like a good idea, should this go into "php.ini", or in the script
> itself ?

Maybe it's a good idea not to mess with this if you are unsure of what it
does. This is just PHP code, not a configuration directive. How to use it
depends on how your script is coded.

>> Internet Explorer is known for having problems with PDF files (esp. if you
>> use sessions) but I'm not sure about ZIPs.
>
> Yes, IE causes a lot of problems, but I see this also on my Mozilla/Firefox.

If you use Firefox, please install the "Live HTTP Headers" extension and
find out yourself what MIME type is used.



--
-+ Álvaro G. Vicario - Burgos, Spain
+- http://www.demogracia.com (la web de humor barnizada para la intemperie)
++ Manda tus dudas al grupo, no a mi buzón
-+ Send your questions to the group, not to my mailbox
--

Hello,

Thanks for the tip. Unfortunately all of magic_quotes_gpc,
magic_quotes_runtime, magic_quotes_sybase are "off" in the php.ini, so
this should not be the cause as far as I can understand.

Best regards, Lars

chernyshevsky@hotmail.com wrote:

> The conversion of binary zeros to "\0"s is consistent with
> magic_quotes_runtime = On.
>

Hello Alvaro,

Thanks for your input, I really appreciate it!
Unfortunately the script is not my own to start with, I do many
languages but have not got around to PHP yet. I will try it out
(reversible...). But it will have to be tomorrow, it's bedtime for me
here now (Norway)!

Best regards, Lars


Alvaro G. Vicario wrote:

> *** Swede escribió/wrote (Tue, 15 Feb 2005 22:15:08 +0100):
>
>>>header('Content-Type: application/zip');
>>
>>Sounds like a good idea, should this go into "php.ini", or in the script
>>itself ?
>
>
> Maybe it's a good idea not to mess with this if you are unsure of what it
> does. This is just PHP code, not a configuration directive. How to use it
> depends on how your script is coded.
>
>
>>>Internet Explorer is known for having problems with PDF files (esp. if you
>>>use sessions) but I'm not sure about ZIPs.
>>
>>Yes, IE causes a lot of problems, but I see this also on my Mozilla/Firefox.
>
>
> If you use Firefox, please install the "Live HTTP Headers" extension and
> find out yourself what MIME type is used.
>
>
>

Don't know if it would help in this case, but Igor Tandetnik's
PassThruAPP is very useful for diagnosing IE issues:

http://home.nyc.rr.com/itandetnik/passthruapp.zip