Cron php & refresh

I'm working on an application that includes e-mail notifications of
certain events. Because the application will have hundreds or thousands
of users, I've designed it so that e-mail notifications are saved to a
MySQL table. Then a regular cron job runs a php page to select the data
from the table, put it into a mail() command and mail.

My original vision was to have the script do about 50 mails, then pause
for a 5 seconds, do a meta refresh and run through another 50 and so on
- with experimentation in numbers as necessary. This process was set up
because I recall reading somewhere - perhaps the manual - that it is not
a good thing to run too many php mail()s in succession. Also, I worry
about php time-outs.

If you know more about PHP and 'nix than I do, you already realise the
fatal flaw here: meta refresh doesn't work on a cron job.

Question: is there an alternative? I appreciate I could use sleep()
after every 50 mails - but there would still be the time-out problem.

I've searched the manual and via Google - but haven't found anything -
possibly I am searching on the wrong terms.

Your help will be much appreciated.

Thanks,

Jeffrey

Jeffrey wrote:

> I'm working on an application that includes e-mail notifications of
> certain events. Because the application will have hundreds or
> thousands of users, I've designed it so that e-mail notifications are
> saved to a MySQL table. Then a regular cron job runs a php page to
> select the data from the table, put it into a mail() command and mail..

Why not just send the notification at the time of the event? (I assume
you update the database at the time of the event).


/Per Jessen, Zürich

On Sunday 20 April 2008 11:27:38 Jeffrey wrote:
> I'm working on an application that includes e-mail notifications of
> certain events. Because the application will have hundreds or thousands
> of users, I've designed it so that e-mail notifications are saved to a
> MySQL table. Then a regular cron job runs a php page to select the data
> from the table, put it into a mail() command and mail.
>
> My original vision was to have the script do about 50 mails, then pause
> for a 5 seconds, do a meta refresh and run through another 50 and so on
> - with experimentation in numbers as necessary. This process was set up
> because I recall reading somewhere - perhaps the manual - that it is not
> a good thing to run too many php mail()s in succession. Also, I worry
> about php time-outs.
>
> If you know more about PHP and 'nix than I do, you already realise the
> fatal flaw here: meta refresh doesn't work on a cron job.
>
> Question: is there an alternative? I appreciate I could use sleep()
> after every 50 mails - but there would still be the time-out problem.
>
> I've searched the manual and via Google - but haven't found anything -
> possibly I am searching on the wrong terms.
>
> Your help will be much appreciated.
>
> Thanks,
>
> Jeffrey

What a waste. Is the MTA operational on the server? if so, forget mailing with
php and rather use php to access the mta. It is designed to actually send
mails, lots of lots of em. If not, set it up.
I recon a cluster with hundreds of thousands of users ought to have such a
thing... a couple of blades or some of those solaris niagara 8 core nicies
doing it.


--
---
Børge Holen
http://www.arivene.net

Per Jessen wrote:
> Jeffrey wrote:
>
>> I'm working on an application that includes e-mail notifications of
>> certain events. Because the application will have hundreds or
>> thousands of users, I've designed it so that e-mail notifications are
>> saved to a MySQL table. Then a regular cron job runs a php page to
>> select the data from the table, put it into a mail() command and mail.
>
> Why not just send the notification at the time of the event? (I assume
> you update the database at the time of the event).
>
>
> /Per Jessen, Zürich
>
>
Because in my experience, several hundred e-mails takes time to send,
hence either the user leaves the page before all the mails are sent or
the page times out before all mails are sent. And if there are thousands
of e-mails, it will only get worse.

Jeffrey

Jeffrey wrote:

> Per Jessen wrote:
>> Jeffrey wrote:
>>
>>> I'm working on an application that includes e-mail notifications of
>>> certain events. Because the application will have hundreds or
>>> thousands of users, I've designed it so that e-mail notifications
>>> are saved to a MySQL table. Then a regular cron job runs a php page
>>> to select the data from the table, put it into a mail() command and
>>> mail.
>>
>> Why not just send the notification at the time of the event? (I
>> assume you update the database at the time of the event).
>>
>>
>> /Per Jessen, Zürich
>>
>>
> Because in my experience, several hundred e-mails takes time to send,
> hence either the user leaves the page before all the mails are sent or
> the page times out before all mails are sent. And if there are
> thousands of e-mails, it will only get worse.

Sorry, I (wrongly) assumed 1 event = 1 email.

OK, then I'd stick to what you're doing - add the event to a queue (i.e..
your database), and poll this at regular intervals. I don't think you
need to worry about the number of emails sent per interval. Just let
your PHP script generate the sendmail commands and the email-text to
stdout, then pipe that to /bin/sh.

The output generated by your script would look like this:
sendmail -oi -r <fromaddr> <toaddr> ..... <<XXX
emailhdr
emailhdr

emailtxt
emailtxt
emailtxt
XXX
sendmail -oi -r <fromaddr> <toaddr> ..... <<XXX
etc
etc


/Per Jessen, Zürich

Børge Holen wrote:

> Is the MTA operational on the server? if so, forget
> mailing with php and rather use php to access the mta.

Yeah, that is what mail() does - it calls sendmail.


/Per Jessen, Zürich

On Sunday 20 April 2008, Jeffrey wrote:
> I'm working on an application that includes e-mail notifications of
> certain events. Because the application will have hundreds or thousands
> of users, I've designed it so that e-mail notifications are saved to a
> MySQL table. Then a regular cron job runs a php page to select the data
> from the table, put it into a mail() command and mail.
>
> My original vision was to have the script do about 50 mails, then pause
> for a 5 seconds, do a meta refresh and run through another 50 and so on
> - with experimentation in numbers as necessary. This process was set up
> because I recall reading somewhere - perhaps the manual - that it is not
> a good thing to run too many php mail()s in succession. Also, I worry
> about php time-outs.
>
> If you know more about PHP and 'nix than I do, you already realise the
> fatal flaw here: meta refresh doesn't work on a cron job.
>
> Question: is there an alternative? I appreciate I could use sleep()
> after every 50 mails - but there would still be the time-out problem.
>
> I've searched the manual and via Google - but haven't found anything -
> possibly I am searching on the wrong terms.
>
> Your help will be much appreciated.

Why not just have the cron task itself run every 5 minutes? Every 5 minutes,
hit a PHP script that will pull the next 50 messages to send and send them,
then exit. In 5 minutes minus the time that takes, cron will fire your
script again. Problem solved.

--
Larry Garfield AIM: LOLG42
larry@garfieldtech.com ICQ: 6817012

"If nature has made any one thing less susceptible than all others of
exclusive property, it is the action of the thinking power called an idea,
which an individual may exclusively possess as long as he keeps it to
himself; but the moment it is divulged, it forces itself into the possession
of every one, and the receiver cannot dispossess himself of it." -- Thomas
Jefferson

On Sun, 2008-04-20 at 13:32 -0500, Larry Garfield wrote:
> On Sunday 20 April 2008, Jeffrey wrote:
> > I'm working on an application that includes e-mail notifications of
> > certain events. Because the application will have hundreds or thousands
> > of users, I've designed it so that e-mail notifications are saved to a
> > MySQL table. Then a regular cron job runs a php page to select the data
> > from the table, put it into a mail() command and mail.
> >
> > My original vision was to have the script do about 50 mails, then pause
> > for a 5 seconds, do a meta refresh and run through another 50 and so on
> > - with experimentation in numbers as necessary. This process was set up
> > because I recall reading somewhere - perhaps the manual - that it is not
> > a good thing to run too many php mail()s in succession. Also, I worry
> > about php time-outs.
> >
> > If you know more about PHP and 'nix than I do, you already realise the
> > fatal flaw here: meta refresh doesn't work on a cron job.
> >
> > Question: is there an alternative? I appreciate I could use sleep()
> > after every 50 mails - but there would still be the time-out problem.
> >
> > I've searched the manual and via Google - but haven't found anything -
> > possibly I am searching on the wrong terms.
> >
> > Your help will be much appreciated.
>
> Why not just have the cron task itself run every 5 minutes? Every 5 minutes,
> hit a PHP script that will pull the next 50 messages to send and send them,
> then exit. In 5 minutes minus the time that takes, cron will fire your
> script again. Problem solved.

If you take this approach, just make sure you build a proper queuing
system or mark the mails you're processing as being processed in the DB.
Otherwise you could end up with a situation where the system takes
longer than 5 minutes to process the mails (it shouldn't, but you never
know - maybe there's a rogue process or the mail queue is backed up or
something) and then the PHP script starts at the beginning of those 50
mails again. It's an obvious error, but I've done it in an app where I
thought "There's no way the process won't finish before the next cron
job starts", and then it happened and I ended up sending a couple
thousand SMS's out twice at cost to my client. Oops.

J

On Sunday 20 April 2008 13:37:04 Per Jessen wrote:
> Børge Holen wrote:
> > Is the MTA operational on the server? if so, forget
> > mailing with php and rather use php to access the mta.
>
> Yeah, that is what mail() does - it calls sendmail.
>
>
> /Per Jessen, Zürich

What the point of 50 mails now and then, let it all be decided by the mta. you
don't need php to do stuff it isn't intended for.
the mta will send next mail as the previous is accepted and written off as
sent.

--
---
Børge Holen
http://www.arivene.net