PHP 5, XSL transformations of big files

Hi.

I'm going to make in auto some tasks that I'm doing manually every
day. I'm uploading big XML file (~20mb) and then I'm using XSLT schema
to transform it to MySQL queries file.

I have used 3rd tools to process XSLT and it takes few seconds to
process my big XML file.

But now I'm moving to PHP so I'm trying to use XSL PHP extension to
apply my XSL file. But... it takes minutes... I allowed 10 minutes
( set_time_limit(600) ) and it still crashes and says that time
excited.

So please somebody help me to find a way out of the impasse. I can't
move forward now because of slow XSL processing.

The code I'm using is:

*******
// Load the XML source
$xml = new DOMDocument;
$xml->load('extracted/marketplace_feed_v1.xml');

$xsl = new DOMDocument;
$xsl->load('cb_marketplace_feed_compact.xsl');

// Configure the transformer
$proc = new XSLTProcessor;
$proc->importStyleSheet($xsl); // attach the xsl rules

$sqlCommands = $proc->transformToXML($xml);
if ($sqlCommands == FALSE)
{
echo "XSLT transformation failed";
exit;
}

if (file_put_contents('cbfeed.sql', $sqlCommands) == FALSE);
{
echo 'Failed to save XSLT transformation result to the file
"cbfeed.sql"';
exit;
}
*******

You can upload files used in my script here:

http://megliosoft.org/meglio/cbfeed/...ce_feed_v1.xml
http://megliosoft.org/meglio/cbfeed/...ed_compact.xsl


Any information may help.

Thanks,
Anton

try to use javascript xslt transformation:
http://www.w3schools.com/xsl/xsl_client.asp

probaly faster

Meglio wrote:
> Hi.
>
> I'm going to make in auto some tasks that I'm doing manually every
> day. I'm uploading big XML file (~20mb) and then I'm using XSLT schema
> to transform it to MySQL queries file.
>
> I have used 3rd tools to process XSLT and it takes few seconds to
> process my big XML file.
>
> But now I'm moving to PHP so I'm trying to use XSL PHP extension to
> apply my XSL file. But... it takes minutes... I allowed 10 minutes
> ( set_time_limit(600) ) and it still crashes and says that time
> excited.
>
> So please somebody help me to find a way out of the impasse. I can't
> move forward now because of slow XSL processing.
>
> The code I'm using is:
>
> *******
> // Load the XML source
> $xml = new DOMDocument;
> $xml->load('extracted/marketplace_feed_v1.xml');
>
> $xsl = new DOMDocument;
> $xsl->load('cb_marketplace_feed_compact.xsl');
>
> // Configure the transformer
> $proc = new XSLTProcessor;
> $proc->importStyleSheet($xsl); // attach the xsl rules
>
> $sqlCommands = $proc->transformToXML($xml);
> if ($sqlCommands == FALSE)
> {
> echo "XSLT transformation failed";
> exit;
> }
>
> if (file_put_contents('cbfeed.sql', $sqlCommands) == FALSE);
> {
> echo 'Failed to save XSLT transformation result to the file
> "cbfeed.sql"';
> exit;
> }
> *******
>
> You can upload files used in my script here:
>
> http://megliosoft.org/meglio/cbfeed/...ce_feed_v1.xml
> http://megliosoft.org/meglio/cbfeed/...ed_compact.xsl
>
>
> Any information may help.
>
> Thanks,
> Anton

Thiago Macedo wrote:
> try to use javascript xslt transformation:
> http://www.w3schools.com/xsl/xsl_client.asp
>
> probaly faster
>
> Meglio wrote:
>> Hi.
>>
>> I'm going to make in auto some tasks that I'm doing manually every
>> day. I'm uploading big XML file (~20mb) and then I'm using XSLT schema
>> to transform it to MySQL queries file.
>>
>> I have used 3rd tools to process XSLT and it takes few seconds to
>> process my big XML file.
>>
>> But now I'm moving to PHP so I'm trying to use XSL PHP extension to
>> apply my XSL file. But... it takes minutes... I allowed 10 minutes
>> ( set_time_limit(600) ) and it still crashes and says that time
>> excited.
>>
>> So please somebody help me to find a way out of the impasse. I can't
>> move forward now because of slow XSL processing.
>>
>> The code I'm using is:
>>
>> *******
>> // Load the XML source
>> $xml = new DOMDocument;
>> $xml->load('extracted/marketplace_feed_v1.xml');
>>
>> $xsl = new DOMDocument;
>> $xsl->load('cb_marketplace_feed_compact.xsl');
>>
>> // Configure the transformer
>> $proc = new XSLTProcessor;
>> $proc->importStyleSheet($xsl); // attach the xsl rules
>>
>> $sqlCommands = $proc->transformToXML($xml);
>> if ($sqlCommands == FALSE)
>> {
>> echo "XSLT transformation failed";
>> exit;
>> }
>>
>> if (file_put_contents('cbfeed.sql', $sqlCommands) == FALSE);
>> {
>> echo 'Failed to save XSLT transformation result to the file
>> "cbfeed.sql"';
>> exit;
>> }
>> *******
>>
>> You can upload files used in my script here:
>>
>> http://megliosoft.org/meglio/cbfeed/...ce_feed_v1.xml
>> http://megliosoft.org/meglio/cbfeed/...ed_compact.xsl
>>
>>
>> Any information may help.
>>
>> Thanks,
>> Anton
>

How is that going to help? Javascript is client side - he'd have to
download those large files, then do it on the client and finally upload
the results. And clients machines are typically less powerful than servers.

A very bad suggestion, IMHO.

--
==================
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstucklex@attglobal.net
==================

On May 3, 1:05 am, Jerry Stuckle <jstuck...@attglobal.net> wrote:
> Thiago Macedo wrote:
> > try to use javascript xslt transformation:
> >http://www.w3schools.com/xsl/xsl_client.asp
>
> > probaly faster
>
> > Meglio wrote:
> >> Hi.
>
> >> I'm going to make in auto some tasks that I'm doing manually every
> >> day. I'm uploading big XML file (~20mb) and then I'm using XSLT schema
> >> to transform it to MySQL queries file.
>
> >> I have used 3rd tools to process XSLT and it takes few seconds to
> >> process my big XML file.
>
> >> But now I'm moving to PHP so I'm trying to use XSL PHP extension to
> >> apply my XSL file. But... it takes minutes... I allowed 10 minutes
> >> ( set_time_limit(600) ) and it still crashes and says that time
> >> excited.
>
> >> So please somebody help me to find a way out of the impasse. I can't
> >> move forward now because of slow XSL processing.
>
> >> The code I'm using is:
>
> >> *******
> >> // Load the XML source
> >> $xml = new DOMDocument;
> >> $xml->load('extracted/marketplace_feed_v1.xml');
>
> >> $xsl = new DOMDocument;
> >> $xsl->load('cb_marketplace_feed_compact.xsl');
>
> >> // Configure the transformer
> >> $proc = new XSLTProcessor;
> >> $proc->importStyleSheet($xsl); // attach the xsl rules
>
> >> $sqlCommands = $proc->transformToXML($xml);
> >> if ($sqlCommands == FALSE)
> >> {
> >> echo "XSLT transformation failed";
> >> exit;
> >> }
>
> >> if (file_put_contents('cbfeed.sql', $sqlCommands) == FALSE);
> >> {
> >> echo 'Failed to save XSLT transformation result to the file
> >> "cbfeed.sql"';
> >> exit;
> >> }
> >> *******
>
> >> You can upload files used in my script here:
>
> >>http://megliosoft.org/meglio/cbfeed/...ce_feed_v1.xml
> >>http://megliosoft.org/meglio/cbfeed/...ed_compact.xsl
>
> >> Any information may help.
>
> >> Thanks,
> >> Anton
>
> How is that going to help? Javascript is client side - he'd have to
> download those large files, then do it on the client and finally upload
> the results. And clients machines are typically less powerful than servers.
>
> A very bad suggestion, IMHO.
>
> --
> ==================
> Remove the "x" from my email address
> Jerry Stuckle
> JDS Computer Training Corp.
> jstuck...@attglobal.net
> ==================

sorry for not acquiring your "requisites".. but, if it will run on a
server, instead of download this large file, he'll upload it.. and
this is usually worst.
by the way, probaly the timeout is occurring in the file upload, not
in the transformation. already examined where it stops the execution?

regards,

Thiago

Jerry Stuckle wrote:

> Thiago Macedo wrote:
>> try to use javascript xslt transformation:
[...]
> How is that going to help?

By not crashing. This one's a quick & dirty automation task, so it ain't to
be pretty.

> Javascript is client side - he'd have to download those large files, then
> do it on the client and finally upload the results. And clients machines
> are typically less powerful than servers.

We can suppose the O.P. is inside a LAN and that semi-large files are not a
problem.




Meglio wrote:
> I have used 3rd tools to process XSLT and it takes few seconds to
> process my big XML file.
>
> But now I'm moving to PHP so I'm trying to use XSL PHP extension to
> apply my XSL file. But... it takes minutes...

I might get kicked out of here for saying here, but maybe PHP isn't the
right tool for this particular job.

You can consider using PHP as a web frontend, and use exec() to call a
command-line xsltproc, or any other 3rd-party tool. You can also do
client-side processing, given you have set the size limits for uploads
right.


> I allowed 10 minutes ( set_time_limit(600) ) and it still crashes and says
> that time excited.

Can we have a look at the *exact* error message?


--
----------------------------------
Iván Sánchez Ortega -ivansanchez-algarroba-escomposlinux-punto-org-

Y dijo el capitán Bloub: "Abordar el barco"... y les quedó precioso.

On May 3, 7:15 am, Thiago Macedo <thiago.ch...@gmail.com> wrote:
> On May 3, 1:05 am, Jerry Stuckle <jstuck...@attglobal.net> wrote:
>
>
>
> > Thiago Macedo wrote:
> > > try to use javascript xslt transformation:
> > >http://www.w3schools.com/xsl/xsl_client.asp
>
> > > probaly faster
>
> > > Meglio wrote:
> > >> Hi.
>
> > >> I'm going to make in auto some tasks that I'm doing manually every
> > >> day. I'm uploading big XML file (~20mb) and then I'm using XSLT schema
> > >> to transform it to MySQL queries file.
>
> > >> I have used 3rd tools to process XSLT and it takes few seconds to
> > >> process my big XML file.
>
> > >> But now I'm moving to PHP so I'm trying to use XSL PHP extension to
> > >> apply my XSL file. But... it takes minutes... I allowed 10 minutes
> > >> ( set_time_limit(600) ) and it still crashes and says that time
> > >> excited.
>
> > >> So please somebody help me to find a way out of the impasse. I can't
> > >> move forward now because of slow XSL processing.
>
> > >> The code I'm using is:
>
> > >> *******
> > >> // Load the XML source
> > >> $xml = new DOMDocument;
> > >> $xml->load('extracted/marketplace_feed_v1.xml');
>
> > >> $xsl = new DOMDocument;
> > >> $xsl->load('cb_marketplace_feed_compact.xsl');
>
> > >> // Configure the transformer
> > >> $proc = new XSLTProcessor;
> > >> $proc->importStyleSheet($xsl); // attach the xsl rules
>
> > >> $sqlCommands = $proc->transformToXML($xml);
> > >> if ($sqlCommands == FALSE)
> > >> {
> > >> echo "XSLT transformation failed";
> > >> exit;
> > >> }
>
> > >> if (file_put_contents('cbfeed.sql', $sqlCommands) == FALSE);
> > >> {
> > >> echo 'Failed to save XSLT transformation result to the file
> > >> "cbfeed.sql"';
> > >> exit;
> > >> }
> > >> *******
>
> > >> You can upload files used in my script here:
>
> > >>http://megliosoft.org/meglio/cbfeed/...ce_feed_v1.xml
> > >>http://megliosoft.org/meglio/cbfeed/...ed_compact.xsl
>
> > >> Any information may help.
>
> > >> Thanks,
> > >> Anton
>
> > How is that going to help? Javascript is client side - he'd have to
> > download those large files, then do it on the client and finally upload
> > the results. And clients machines are typically less powerful than servers.
>
> > A very bad suggestion, IMHO.
>
> > --
> > ==================
> > Remove the "x" from my email address
> > Jerry Stuckle
> > JDS Computer Training Corp.
> > jstuck...@attglobal.net
> > ==================
>
> sorry for not acquiring your "requisites".. but, if it will run on a
> server, instead of download this large file, he'll upload it.. and
> this is usually worst.
> by the way, probaly the timeout is occurring in the file upload, not
> in the transformation. already examined where it stops the execution?
>
> regards,
>
> Thiago

>> if it will run on a server, instead of download this large file, he'll upload it.. and
this is usually worst.

THIS is not worst because when I do it manually I still transform XML
to SQL (which is big file too) and upload it to the server. So I have
to upload SQL to the server in any case. The goal is to upload XML
directly to the server on auto and to transform it to SQL file using
PHP script.

>> by the way, probably the timeout is occurring in the file upload, not
in the transformation. already examined where it stops the execution?

I'm not so stupid to ask question before to examine where it stops. It
freezes on XSL transformation.

Fatal error: Maximum execution time of ... seconds exceeded in ... on
line ...

Meglio wrote:
> Hi.
>
> I'm going to make in auto some tasks that I'm doing manually every
> day. I'm uploading big XML file (~20mb) and then I'm using XSLT schema
> to transform it to MySQL queries file.
>
> I have used 3rd tools to process XSLT and it takes few seconds to
> process my big XML file.
>
> But now I'm moving to PHP so I'm trying to use XSL PHP extension to
> apply my XSL file. But... it takes minutes... I allowed 10 minutes
> ( set_time_limit(600) ) and it still crashes and says that time
> excited.
>
> So please somebody help me to find a way out of the impasse. I can't
> move forward now because of slow XSL processing.

Maybe there is some issue that causes PHP xslt processor to fail or go
into infinite loop. Try cutting the xsl template to few smaller parts
(just for test) and see if the error occurs in any specyfic part. Maybe
there is some syntax used that PHP doesn't really like..

Not an xsl expert, so if this is way off, just ignore..

best regards
Piotr N

On May 3, 11:29 am, Piotr <s...@poczta.onet.pl> wrote:
> Meglio wrote:
> > Hi.
>
> > I'm going to make in auto some tasks that I'm doing manually every
> > day. I'm uploading big XML file (~20mb) and then I'm using XSLT schema
> > to transform it to MySQL queries file.
>
> > I have used 3rd tools to process XSLT and it takes few seconds to
> > process my big XML file.
>
> > But now I'm moving to PHP so I'm trying to use XSL PHP extension to
> > apply my XSL file. But... it takes minutes... I allowed 10 minutes
> > ( set_time_limit(600) ) and it still crashes and says that time
> > excited.
>
> > So please somebody help me to find a way out of the impasse. I can't
> > move forward now because of slow XSL processing.
>
> Maybe there is some issue that causes PHP xslt processor to fail or go
> into infinite loop. Try cutting the xsl template to few smaller parts
> (just for test) and see if the error occurs in any specyfic part. Maybe
> there is some syntax used that PHP doesn't really like..
>
> Not an xsl expert, so if this is way off, just ignore..
>
> best regards
> Piotr N

Hi. Thanks for you advice. But i'm not XSLT expert too so it is not
simple way for me to find such bug. So I'm still looking for an expert
who can help me to understand where the problems is.

Thanks,
Anton

Meglio wrote:
>> Maybe there is some issue that causes PHP xslt processor to fail or go
>> into infinite loop. Try cutting the xsl template to few smaller parts
>> (just for test) and see if the error occurs in any specyfic part. Maybe
>> there is some syntax used that PHP doesn't really like..
>>
>> Not an xsl expert, so if this is way off, just ignore..
>>
>> best regards
>> Piotr N
>
> Hi. Thanks for you advice. But i'm not XSLT expert too so it is not
> simple way for me to find such bug. So I'm still looking for an expert
> who can help me to understand where the problems is.
>
> Thanks,
> Anton

Afaik, you should find tags like his:

<xsl:template match="some_tag">
[..]
</xsl:template>

if you have more then one, simply comment all but one, see if this times
out, if it doesn't, comment this template out and uncomment next one. Do
so as long as you find the one that times out.

When you find that one, comment out some functions, like xsl:for-each.
To comment, use <!-- --> syntax.

I suppose there is also xsl group for xslt specific problems.

best regards
Piotr N