How to detect locked files?

Hello!

If i wan't to know if a file is locked by another program, how do i go about
it in a shell-script? If thats not possible with shell-script, is there any
other way?

I have a shell script that every 5 minutes try to move files from an
incomming FTP directory to another directory. It might be so that the file
i'm trying to move is locked as it is currently being uploaded via the FTP
server. To keep out of various errors i wan't to detect if the file is
locked before i try to move it.

The filesystem i use is currently Reiser FS, but if there is a solution that
works on other as well (ext2, ext3) it would be really nice.

Mats

On Sat, 23 Apr 2005 22:02:46 GMT, Mackan staggered into the Black Sun
and said:
> If i want to know if a file is locked by another program, how do i go
> about it in a shell-script?

"lsof $FILENAME" returns 1 in $? if $FILENAME has not been opened or
does not exist, 0 in $? if the file is open. The text lsof returns will
tell you which process has the file open and various details; check the
man page and try it out yourself for more information.

There's probably a more efficient way to do this with fopen(), fileno(),
and fcntl(fd,F_GETLK,struck lock *my_lock), assuming the FTP server
places a write lock on the files it's writing.

> I have a shell script that every 5 minutes tries to move files from an
> incoming FTP directory to another directory. It might be so that the
> file i'm trying to move is locked as it is currently being uploaded
> via the FTP server.

Er... FTP servers may not lock the files they're writing to. If they
place an exclusive write lock on a file with fcntl(), that doesn't
prevent another process from moving the file to another directory.
Remember, moving/renaming files doesn't change the files themselves, but
the directory the file is in (and the directory the file's being moved
to, if you're moving a file.)

> The filesystem i use is currently Reiser FS

Not really relevant here; lsof works the same on ReiserFS as it does on
ext[23]. HTH,

--
Matt G|There is no Darkness in Eternity/But only Light too dim for us to see
Brainbench MVP for Linux Admin / mail: TRAP + SPAN don't belong
http://www.brainbench.com / Hire me!
-----------------------------/ http://crow202.dyndns.org/~mhgraham/resume

"Dances With Crows" <danSPANceswitTRAPhcrows@gmail.com> skrev i meddelandet
news:slrnd6lmq7.fvp.danSPANceswitTRAPhcrows@samant ha.crow202.dyndns.org...
> On Sat, 23 Apr 2005 22:02:46 GMT, Mackan staggered into the Black Sun
> and said:

> Er... FTP servers may not lock the files they're writing to. If they
> place an exclusive write lock on a file with fcntl(), that doesn't
> prevent another process from moving the file to another directory.
> Remember, moving/renaming files doesn't change the files themselves, but
> the directory the file is in (and the directory the file's being moved
> to, if you're moving a file.)

Oh!

So that means the file i move might not be complete? Does the write continue
after i moved the file in the case the FTP-application isn't done writing or
is it "cut" at the moment of the move, resulting in a corrupt file? What i
mean to ask is, is the move transparent (not affecting) the ongoing write
proccess of the FTP-application?

I really would like to know when the file is "done" so i can move it forward
for further processing. If i move a not completely downloaded file, the end
result will definitely be bad.


> --
> Matt G|There is no Darkness in Eternity/But only Light too dim for us to
> see
> Brainbench MVP for Linux Admin / mail: TRAP + SPAN don't belong
> http://www.brainbench.com / Hire me!
> -----------------------------/ http://crow202.dyndns.org/~mhgraham/resume

On Sun, 24 Apr 2005 01:08:38 GMT, Mackan staggered into the Black Sun
and said:
> "Dances With Crows" <danSPANceswitTRAPhcrows@gmail.com> skrev i
>> Er... FTP servers may not lock the files they're writing to. If they
>> place an exclusive write lock on a file with fcntl(), that doesn't
>> prevent another process from moving the file to another directory.
>> Remember, moving/renaming files doesn't change the files themselves,
>> but the directory the file is in
> So that means the file i move might not be complete?

If you move it before the FTP server has finished writing, yeah, it
might not be complete.

> Does the write continue after i moved the file in the case the
> FTP-application isn't done writing or is it "cut" at the moment of the
> move, resulting in a corrupt file?

Depends on how the FTP server handles it. IME, if you move the file
before the write's finished, it gets truncated. OTOH, the last time I
did that (several years ago), I was moving files from
/work/ftp/username/upload/ to /work/in-process/ , and username was
chrooted to /work/ftp/username/ . Since the instance of the FTP server
was chrooted, it couldn't access /work/in-process/ , so it *had* to
truncate.

> What i mean to ask is, is the move transparent (not affecting) the
> ongoing write proccess of the FTP-application?

If the FTP server has write access to the directory you moved the file
to, it should work. There are a number of variables though. Most FTP
servers are chrooted for obvious reasons. If yours is, and you move a
file-in-progress somewhere that isn't in the chroot environment, the FTP
server won't be able to continue writing and the file will be truncated.

> I really would like to know when the file is "done" so i can move it

"lsof $FILE" will tell you if $FILE is open or not, if the user running
lsof is root or the processes that have $FILE open are owned by the user
running lsof. FTP servers close() files after they've finished writing
to them. Combine those 2 things and you'll have an idea of how to
proceed.

If the FTP server is running as user bob, and you can't assume user
bob's identity with su or use su (or sudo) to run lsof as root, then
you'll have to fall back on something like this: Use stat on the file
every N seconds, check the file size, if file size is the same as it
was N seconds ago, it's probably done.

Certain types of files (zip files for one) have an alternate means of
checking; for zip files, the central directory signature is always in
the last few bytes, and if that signature is there, the file's finished
uploading. I use lsof, but then I have root on the FTP servers at ork.
YMMV. HTH anyway,

--
Matt G|There is no Darkness in Eternity/But only Light too dim for us to see
Brainbench MVP for Linux Admin / mail: TRAP + SPAN don't belong
http://www.brainbench.com / Hire me!
-----------------------------/ http://crow202.dyndns.org/~mhgraham/resume

"Mackan" <spamenot.mog.pettersson@telia.com> writes:


>"Dances With Crows" <danSPANceswitTRAPhcrows@gmail.com> skrev i meddelandet
>news:slrnd6lmq7.fvp.danSPANceswitTRAPhcrows@saman tha.crow202.dyndns.org...
>> On Sat, 23 Apr 2005 22:02:46 GMT, Mackan staggered into the Black Sun
>> and said:

>> Er... FTP servers may not lock the files they're writing to. If they
>> place an exclusive write lock on a file with fcntl(), that doesn't
>> prevent another process from moving the file to another directory.
>> Remember, moving/renaming files doesn't change the files themselves, but
>> the directory the file is in (and the directory the file's being moved
>> to, if you're moving a file.)

>Oh!

>So that means the file i move might not be complete? Does the write continue
>after i moved the file in the case the FTP-application isn't done writing or
>is it "cut" at the moment of the move, resulting in a corrupt file? What i
>mean to ask is, is the move transparent (not affecting) the ongoing write
>proccess of the FTP-application?

>I really would like to know when the file is "done" so i can move it forward
>for further processing. If i move a not completely downloaded file, the end
>result will definitely be bad.

The actual file reference is the inode. Once the file is opened, it will be
refered to by the inode, and renaming the file ON THE SAME PARTITION will
not change that.

A move of a file between partitions however is done by copying the file and
then erasing the old version. This would mean that the copy would be done
on the basis of whatever was there at the time, the file would be erased,
but since something had it opened, that open would keep the file until it
finally closed it. At that point since the file was erased, the file would
be removed.

Ie, it would depend on whether your move was between partitions or on the
same partition.

Mackan <spamenot.mog.pettersson@telia.com> wrote:
> So that means the file i move might not be complete? Does the write continue
> after i moved the file in the case the FTP-application isn't done writing or
> is it "cut" at the moment of the move, resulting in a corrupt file?

It depends whether you're moving the file across different filesystems
or not.

You also need to consider the possibility that there may be a network
fault while you're copying the file, resulting in a miscopy.

> I really would like to know when the file is "done" so i can move it forward
> for further processing. If i move a not completely downloaded file, the end
> result will definitely be bad.

The usual approach is to get your client (sender) to transmit the files
with a ".tmp" suffix. The client can detect a successful and complete
transfer, at which point it renames the file to remove the ".tmp"
suffix. The server (receiver) is coded to ignore all ".tmp" files. The
net effect is that files appear to arrive instantaneously (at the point
the rename occurs). One final point is to remember to clean up (i.e.
delete) all ".tmp" files that haven't been modified for more than a day,
on the basis that they got left over from an earlier transfer attempt.
If you're really fancy then include a unique identifier immediately before
the ".tmp" suffix, so that you can handle the unfortunate possibility
of two copies of the client transferring the same file at the same time.

Chris