question regarding image processing

On Fri, 23 May 2008 07:40:37 -0400, Jerry Stuckle wrote:
> Álvaro G. Vicario wrote:
>> Some guys adore storing binary data in databases but of course that's a
>> good old discussion. I particularly find no benefits in it (you can't do
>> searches like "WHERE category_id=20 AND picture PORTRAITS A 'black
>> cat'") and it add an annoying overhead in all related tasks.
>>
>>
>
> The ability to search is not the only reason to store things in
> databases. And if you have hundreds of thousands of images, a database
> can be more efficient.

It's a little rough on the incremental backups, though... (:

--
72. If all the heroes are standing together around a strange device and begin
to taunt me, I will pull out a conventional weapon instead of using my
unstoppable superweapon on them.
--Peter Anspach's list of things to do as an Evil Overlord

Álvaro G. Vicario wrote:
> Jerry Stuckle escribió:
>> If you have a large number of images, a database is often faster than
>> the file system. Databases can easily handle 100K images; file
>> systems not so much.
>
> If you mean "in the same directory", you're absolutely right.
>
>

But putting them in different directories means having to plan for those
directories in your code - adding more programming overhead (and places
to go wrong). Additionally, if you don't "guess" right the first time,
you then need to go back and redo that code to get additional
directories. Or you have lots of directories with just 1 or 2 files in
each one.

I've been storing binary data in SQL databases since the mid 80's, when
we stored documents on a DB2 mainframe. It works great.

Other advantages - backups are easier - just backup the database. And
if you use a database/engine with referential integrity, you don't have
to worry about images being deleted from the file system and still being
pointed to in the database.

Additionally, you can have things like the image size stored in the
database with the image, so you don't have to go looking for it later.
You can even have a caption for it. And it's all one retrieval - not a
retrieval plus a file system call.

Databases are made for handling data. It doesn't matter if that data is
text or binary; it does both quite well.

--
==================
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstucklex@attglobal.net
==================

Peter H. Coffin wrote:
> On Fri, 23 May 2008 07:40:37 -0400, Jerry Stuckle wrote:
>> Álvaro G. Vicario wrote:
>>> Some guys adore storing binary data in databases but of course that's a
>>> good old discussion. I particularly find no benefits in it (you can't do
>>> searches like "WHERE category_id=20 AND picture PORTRAITS A 'black
>>> cat'") and it add an annoying overhead in all related tasks.
>>>
>>>
>> The ability to search is not the only reason to store things in
>> databases. And if you have hundreds of thousands of images, a database
>> can be more efficient.
>
> It's a little rough on the incremental backups, though... (:
>

Only the first time. But it makes backups easier - you don't have to
back up files and database. Just the database.

--
==================
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstucklex@attglobal.net
==================

Jerry Stuckle wrote:
> Peter H. Coffin wrote:
>> On Fri, 23 May 2008 07:40:37 -0400, Jerry Stuckle wrote:
>>> Álvaro G. Vicario wrote:
>>>> Some guys adore storing binary data in databases but of course
>>>> that's a good old discussion. I particularly find no benefits in it
>>>> (you can't do searches like "WHERE category_id=20 AND picture
>>>> PORTRAITS A 'black cat'") and it add an annoying overhead in all
>>>> related tasks.
>>>>
>>>>
>>> The ability to search is not the only reason to store things in
>>> databases. And if you have hundreds of thousands of images, a
>>> database can be more efficient.
>>
>> It's a little rough on the incremental backups, though... (:
>>
>
> Only the first time. But it makes backups easier - you don't have to
> back up files and database. Just the database.
>
A database *is* files, Jerry.

On Sat, 24 May 2008 10:34:10 +0200, The Natural Philosopher <a@b.c> wrote:

> Jerry Stuckle wrote:
>> Peter H. Coffin wrote:
>>> On Fri, 23 May 2008 07:40:37 -0400, Jerry Stuckle wrote:
>>>> Álvaro G. Vicario wrote:
>>>>> Some guys adore storing binary data in databases but of course
>>>>> that's a good old discussion. I particularly find no benefits in it
>>>>> (you can't do searches like "WHERE category_id=20 AND picture
>>>>> PORTRAITS A 'black cat'") and it add an annoying overhead in all
>>>>> related tasks.
>>>>>
>>>>>
>>>> The ability to search is not the only reason to store things in
>>>> databases. And if you have hundreds of thousands of images, a
>>>> database can be more efficient.
>>>
>>> It's a little rough on the incremental backups, though... (:
>>>
>> Only the first time. But it makes backups easier - you don't have to
>> back up files and database. Just the database.
>>
> A database *is* files, Jerry.

.... but backing up the files of a database instead of using proper
mechanisms is asking for corruption. It's not the physical presence on a
hard drive that's the problem, it's having to apply 2 different processes.
As a bonus, keeping a database in sync in another location is far easier
(with more mature tools) then syncing files. (Allthough I could not
imagine life without rsync)
--
Rik Wasmus
....spamrun finished

..oO(The Natural Philosopher)

>Jerry Stuckle wrote:
>> Peter H. Coffin wrote:
>>> On Fri, 23 May 2008 07:40:37 -0400, Jerry Stuckle wrote:
>>>> Álvaro G. Vicario wrote:
>>>>> Some guys adore storing binary data in databases but of course
>>>>> that's a good old discussion. I particularly find no benefits in it
>>>>> (you can't do searches like "WHERE category_id=20 AND picture
>>>>> PORTRAITS A 'black cat'") and it add an annoying overhead in all
>>>>> related tasks.
>>>>>
>>>>>
>>>> The ability to search is not the only reason to store things in
>>>> databases. And if you have hundreds of thousands of images, a
>>>> database can be more efficient.
>>>
>>> It's a little rough on the incremental backups, though... (:
>>>
>>
>> Only the first time. But it makes backups easier - you don't have to
>> back up files and database. Just the database.
>>
> A database *is* files, Jerry.

Not necessarily. You can keep entire databases in memory (can't get any
faster) or even store multiple databases in a single file, which is what
InnoDB does for example. But for the user and often even for the admin
it's of no interest how the database actually stores the data. Messing
with that and directly accessing the container files in case of a disk-
based DB calls for trouble. Real trouble!

Micha

Rik Wasmus wrote:
> On Sat, 24 May 2008 10:34:10 +0200, The Natural Philosopher <a@b.c> wrote:
>
>> Jerry Stuckle wrote:
>>> Peter H. Coffin wrote:
>>>> On Fri, 23 May 2008 07:40:37 -0400, Jerry Stuckle wrote:
>>>>> Álvaro G. Vicario wrote:
>>>>>> Some guys adore storing binary data in databases but of course
>>>>>> that's a good old discussion. I particularly find no benefits in
>>>>>> it (you can't do searches like "WHERE category_id=20 AND picture
>>>>>> PORTRAITS A 'black cat'") and it add an annoying overhead in all
>>>>>> related tasks.
>>>>>>
>>>>>>
>>>>> The ability to search is not the only reason to store things in
>>>>> databases. And if you have hundreds of thousands of images, a
>>>>> database can be more efficient.
>>>>
>>>> It's a little rough on the incremental backups, though... (:
>>>>
>>> Only the first time. But it makes backups easier - you don't have
>>> to back up files and database. Just the database.
>>>
>> A database *is* files, Jerry.
>
> ... but backing up the files of a database instead of using proper
> mechanisms is asking for corruption. It's not the physical presence on a
> hard drive that's the problem, it's having to apply 2 different
> processes. As a bonus, keeping a database in sync in another location is
> far easier (with more mature tools) then syncing files. (Allthough I
> could not imagine life without rsync)

So in effect, C-ISAM apart, backing up a database is actually HARDER
than backing up ordinary files?

Michael Fesser wrote:
> .oO(The Natural Philosopher)
>
>> Jerry Stuckle wrote:
>>> Peter H. Coffin wrote:
>>>> On Fri, 23 May 2008 07:40:37 -0400, Jerry Stuckle wrote:
>>>>> Álvaro G. Vicario wrote:
>>>>>> Some guys adore storing binary data in databases but of course
>>>>>> that's a good old discussion. I particularly find no benefits in it
>>>>>> (you can't do searches like "WHERE category_id=20 AND picture
>>>>>> PORTRAITS A 'black cat'") and it add an annoying overhead in all
>>>>>> related tasks.
>>>>>>
>>>>>>
>>>>> The ability to search is not the only reason to store things in
>>>>> databases. And if you have hundreds of thousands of images, a
>>>>> database can be more efficient.
>>>> It's a little rough on the incremental backups, though... (:
>>>>
>>> Only the first time. But it makes backups easier - you don't have to
>>> back up files and database. Just the database.
>>>
>> A database *is* files, Jerry.
>
> Not necessarily. You can keep entire databases in memory (can't get any
> faster)

Cant get any worse if the system crashes..

> or even store multiple databases in a single file, which is what
> InnoDB does for example.

It's still a file ;-)

> But for the user and often even for the admin
> it's of no interest how the database actually stores the data. Messing
> with that and directly accessing the container files in case of a disk-
> based DB calls for trouble. Real trouble!
>

Actually I have backed up copied and restored C-ISAM files for years
with no problems.

I've even done it without shutting down the copied-from database engine.

On the basis that if I got a bunch of junk when using the copy, I'd do
it properly the longer away round..but in fact it worked.



> Micha

On Sat, 24 May 2008 11:22:54 +0200, The Natural Philosopher <a@b.c> wrote:
> Rik Wasmus wrote:
>> On Sat, 24 May 2008 10:34:10 +0200, The Natural Philosopher <a@b.c>
>> wrote:
>>
>>> Jerry Stuckle wrote:
>>>> Peter H. Coffin wrote:
>>>>> On Fri, 23 May 2008 07:40:37 -0400, Jerry Stuckle wrote:
>>>>>> Álvaro G. Vicario wrote:
>>>>>>> Some guys adore storing binary data in databases but of course
>>>>>>> that's a good old discussion. I particularly find no benefits in
>>>>>>> it (you can't do searches like "WHERE category_id=20 AND picture
>>>>>>> PORTRAITS A 'black cat'") and it add an annoying overhead in all
>>>>>>> related tasks.
>>>>>>>
>>>>>>>
>>>>>> The ability to search is not the only reason to store things in
>>>>>> databases. And if you have hundreds of thousands of images, a
>>>>>> database can be more efficient.
>>>>>
>>>>> It's a little rough on the incremental backups, though... (:
>>>>>
>>>> Only the first time. But it makes backups easier - you don't have
>>>> to back up files and database. Just the database.
>>>>
>>> A database *is* files, Jerry.
>> ... but backing up the files of a database instead of using proper
>> mechanisms is asking for corruption. It's not the physical presence on
>> a hard drive that's the problem, it's having to apply 2 different
>> processes. As a bonus, keeping a database in sync in another location
>> is far easier (with more mature tools) then syncing files. (Allthough I
>> could not imagine life without rsync)
>
> So in effect, C-ISAM apart, backing up a database is actually HARDER
> than backing up ordinary files?

Certainly not. Either point to another database server and say 'just keep
a most recent copy of that server', or use the tool (mysql would be
mysqldump) for the job. Easier to do it right then files, with a lot less
duplicate overhead in most cases. Mind the 'do it right', of course just
downloading / copying files to another point is toddler-easy, but on its
own that's not an ideal backup method. I had more troubles setting up
rsync then a slave mysql database.
--
Rik Wasmus
....spamrun finished

The Natural Philosopher wrote:
> Jerry Stuckle wrote:
>> Peter H. Coffin wrote:
>>> On Fri, 23 May 2008 07:40:37 -0400, Jerry Stuckle wrote:
>>>> Álvaro G. Vicario wrote:
>>>>> Some guys adore storing binary data in databases but of course
>>>>> that's a good old discussion. I particularly find no benefits in it
>>>>> (you can't do searches like "WHERE category_id=20 AND picture
>>>>> PORTRAITS A 'black cat'") and it add an annoying overhead in all
>>>>> related tasks.
>>>>>
>>>>>
>>>> The ability to search is not the only reason to store things in
>>>> databases. And if you have hundreds of thousands of images, a
>>>> database can be more efficient.
>>>
>>> It's a little rough on the incremental backups, though... (:
>>>
>>
>> Only the first time. But it makes backups easier - you don't have to
>> back up files and database. Just the database.
>>
> A database *is* files, Jerry.

Not necessarily. It depends on the implementation.

--
==================
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstucklex@attglobal.net
==================