File size/Number of file limits

[jambroo@gmail.com]

Hello,

We are having problems combining 2 directories under Windows 98.
Basically, we are appending a few hundred files to a larger directory
called 'Contracts'. When we copy over the files we get the following
error: 'The directory or file cannot be created.' Both directories are
on the primary C drive.

The target 'contracts' directory is 599MB and contains 16135 objects.
Is there a limit for FAT32 filesystems running Windows 98 which would
cause this error to happen. I thought maybe there was a 600MB cap and
it wasn't allowed to go over, however i copied a 5 MB file in there
instead of some of the smaller files and it went to 605MB.

I had a feeling it was the filenames, containing & symbols and spaces,
however they seem to copy fine elsewhere.

 

[Rod]

<jambroo@gmail.com> wrote in message
news:1193201315.741378.40400@t8g2000prg.googlegroups.com...
> Hello,
>
> We are having problems combining 2 directories under Windows 98.
> Basically, we are appending a few hundred files to a larger directory
> called 'Contracts'. When we copy over the files we get the following
> error: 'The directory or file cannot be created.' Both directories are
> on the primary C drive.

If you pop over to win XP Newsgroup,
I just asked a similar question with extensive replies
(inc FAT32)
search for "Folder limitations"
HTH
Rod.

 

[MEB]

Actually, it would be as easy, and require less downloading, to search
Google for this News group related to that information. We have done rather
extensive reviews.

The short is: Yes, you may have addressed issues related to the limits
imposed. You may also run across issues related to *like named* folders,
e.g., c:\Contracts\contracts and how these are saved in Fat.

--
MEB
http://peoplescounsel.orgfree.com
________


"Rod" <pookiethai@iprimus.com.au> wrote in message
news:uT5yYMgFIHA.936@TK2MSFTNGP06.phx.gbl...
|
| <jambroo@gmail.com> wrote in message
| news:1193201315.741378.40400@t8g2000prg.googlegroups.com...
| > Hello,
| >
| > We are having problems combining 2 directories under Windows 98.
| > Basically, we are appending a few hundred files to a larger directory
| > called 'Contracts'. When we copy over the files we get the following
| > error: 'The directory or file cannot be created.' Both directories are
| > on the primary C drive.
|
| If you pop over to win XP Newsgroup,
| I just asked a similar question with extensive replies
| (inc FAT32)
| search for "Folder limitations"
| HTH
| Rod.
|
|
|
|

 

[Jeff Richards]

The FAT32 file system has a limit of 65k filename entries. Each DOS 8.3
filename will require one entry - long filenames require more.

So it is quite likely that you have exceeded the filename limitation. AFAIK
there is no size limitation, other than the partition size.
--
Jeff Richards
MS MVP (Windows - Shell/User)
<jambroo@gmail.com> wrote in message
news:1193201315.741378.40400@t8g2000prg.googlegroups.com...
> Hello,
>
> We are having problems combining 2 directories under Windows 98.
> Basically, we are appending a few hundred files to a larger directory
> called 'Contracts'. When we copy over the files we get the following
> error: 'The directory or file cannot be created.' Both directories are
> on the primary C drive.
>
> The target 'contracts' directory is 599MB and contains 16135 objects.
> Is there a limit for FAT32 filesystems running Windows 98 which would
> cause this error to happen. I thought maybe there was a 600MB cap and
> it wasn't allowed to go over, however i copied a 5 MB file in there
> instead of some of the smaller files and it went to 605MB.
>
> I had a feeling it was the filenames, containing & symbols and spaces,
> however they seem to copy fine elsewhere.
>

 

[Alan Edwards]

There is a limit of 64K-1 (65,635) entries in folders but any folder
or filename over 13 characters will use 3 or more entries. One
normally hits the limit around 25-30,000 but it can be less with many
long file names.

....Alan
--
Alan Edwards, MS MVP Windows - Internet Explorer
http://dts-l.org/index.htm




On Tue, 23 Oct 2007 21:48:35 -0700, in
microsoft.public.win98.gen_discussion, jambroo@gmail.com wrote:

>Hello,
>
>We are having problems combining 2 directories under Windows 98.
>Basically, we are appending a few hundred files to a larger directory
>called 'Contracts'. When we copy over the files we get the following
>error: 'The directory or file cannot be created.' Both directories are
>on the primary C drive.
>
>The target 'contracts' directory is 599MB and contains 16135 objects.
>Is there a limit for FAT32 filesystems running Windows 98 which would
>cause this error to happen. I thought maybe there was a 600MB cap and
>it wasn't allowed to go over, however i copied a 5 MB file in there
>instead of some of the smaller files and it went to 605MB.
>
>I had a feeling it was the filenames, containing & symbols and spaces,
>however they seem to copy fine elsewhere.

 

[teebo]

> The FAT32 file system has a limit of 65k filename entries. Each DOS 8.3
> filename will require one entry - long filenames require more.

anyone know of a program that removes any unnecessary long filenames that
are only there to preserve the Case of the filenames ?

something to run at a directory hierarcy that removes all
long filenames for the files that not have more than 8+3 chars
(where only difference between the dos filename and the
long FileName is small and BIG letters)

 

[MEB]

"teebo" <no@mail.no> wrote in message newsp.t0o6opribr8ivg@300pl...
|
| > The FAT32 file system has a limit of 65k filename entries. Each DOS 8.3
| > filename will require one entry - long filenames require more.
|
| anyone know of a program that removes any unnecessary long filenames that
| are only there to preserve the Case of the filenames ?
|
| something to run at a directory hierarcy that removes all
| long filenames for the files that not have more than 8+3 chars
| (where only difference between the dos filename and the
| long FileName is small and BIG letters)

Search Google for things like *mass file name changer* and look at some of
the offerings. Some of them have some interesting potential uses.

--
MEB
http://peoplescounsel.orgfree.com
________

 

[jambroo@gmail.com]

Each file within the directory has around 40 characters in the name so
that would explain it.
Thanks everyone for your help.

On Oct 24, 7:44 pm, Alan Edwards <edwa...@southcom.com.au> wrote:
> There is a limit of 64K-1 (65,635) entries in folders but any folder
> or filename over 13 characters will use 3 or more entries. One
> normally hits the limit around 25-30,000 but it can be less with many
> long file names.
>
> ...Alan
> --
> Alan Edwards, MS MVP Windows - Internet Explorerhttp://dts-l.org/index.htm
>
> On Tue, 23 Oct 2007 21:48:35 -0700, in
>
> microsoft.public.win98.gen_discussion, jamb...@gmail.com wrote:
> >Hello,
>
> >We are having problems combining 2 directories under Windows 98.
> >Basically, we are appending a few hundred files to a larger directory
> >called 'Contracts'. When we copy over the files we get the following
> >error: 'The directory or file cannot be created.' Both directories are
> >on the primary C drive.
>
> >The target 'contracts' directory is 599MB and contains 16135 objects.
> >Is there a limit for FAT32 filesystems running Windows 98 which would
> >cause this error to happen. I thought maybe there was a 600MB cap and
> >it wasn't allowed to go over, however i copied a 5 MB file in there
> >instead of some of the smaller files and it went to 605MB.
>
> >I had a feeling it was the filenames, containing & symbols and spaces,
> >however they seem to copy fine elsewhere.

 

[98 Guy]

jambroo@gmail.com wrote:

> We are having problems combining 2 directories under Windows 98.

Win-98 - First edition or Second edition?

> Is there a limit for FAT32 filesystems running Windows 98 which
> would cause this error to happen.

Assuming you are running Win-98 second edition, then yes you probably
are dealing with FAT-32. But you should check that.

The claimed max number of files per directory is 65535.

http://www.microsoft.com/technet/technetmag/issues/2006/07/WindowsConfidential/

The max number of files possible using FAT-32: 268,435,437

http://en.wikipedia.org/wiki/File_Allocation_Table#Design

The number 268,435,437 (max number of files possible under FAT32) is
equal to 16k x 16k (ie, 16k directories, each with 16k files) or 4k x
64k (ie - 4k directories, each with 64k files).

For test purposes, I've filled a 500 gb volume on a win-98 system with
slightly over 3 million files across almost 200,000 directories.

The limiting factor for the number of files on a win-98 system (under
FAT-32) will be the number of allocation units available on the
volume. You can't have more files than you have allocation units (aka
clusters).

Micro$oft doesn't like Win-9x to have volumes with more than 2 million
clusters, and will allow the cluster count to rise above the 2 million
number only when the volume exceeds 64 gb. Typically, the cluster
count reaches a maximum of 4 million given a volume size of 128 gb.

As far as this being a long file name issue, I don't think so. The
size of the directory tables are variable under FAT-32 and can grow as
needed.

If you double-click the "my computer" icon and right-click on your C
drive and select properties, you will be told the type of file
system. If you right-click on a particular directory and select
properties, you will be told the number of files in that directory.

 

[John John]

98 Guy wrote:

> The claimed max number of files per directory is 65535.

You seem to think (or you are implying) that this is incorrect, by one
of your yet unknown methods are we to assume that you can also overcome
this limitation? In the Directory Table the object's first cluster is
held in a 16 bits wide feild so, being that objects cannot share
clusters, the maximum number of objects that can be contained in the
Directory Table is 65536.


> The max number of files possible using FAT-32: 268,435,437
>
> The number 268,435,437 (max number of files possible under FAT32) is
> equal to 16k x 16k (ie, 16k directories, each with 16k files) or 4k x
> 64k (ie - 4k directories, each with 64k files).

Sigh... Where did you get this mathematical explanation for the
268,435,437 file or cluster limit on FAT32? FAT32 is 32 bits wide but 4
bits are reserved or unused so it is really 28 bits wide and 2^28 less a
couple of bits (for something or other, that I am not sure why) =
268,435,437 clusters.


> The limiting factor for the number of files on a win-98 system (under
> FAT-32) will be the number of allocation units available on the
> volume. You can't have more files than you have allocation units (aka
> clusters).

Well duh! And you can't puts 3 pints in a quart!


> As far as this being a long file name issue, I don't think so. The
> size of the directory tables are variable under FAT-32 and can grow as
> needed.

Well think again! Long filenames require at lest 3 extra directory
object entries, so unless you figured out how to "widen" the 16 bit
field, long file names will reduce the number of available directory
objects.

I expect that you will soon tell us that you have blown FAT32's 4gb file
size limit!

John

 

[John John]

Typo correction in the before last paragraph. LFN's require at least
*2* extra entries, (requires 3 or more entries).

John John wrote:

> 98 Guy wrote:
>
>> The claimed max number of files per directory is 65535.
>
>
> You seem to think (or you are implying) that this is incorrect, by one
> of your yet unknown methods are we to assume that you can also overcome
> this limitation? In the Directory Table the object's first cluster is
> held in a 16 bits wide feild so, being that objects cannot share
> clusters, the maximum number of objects that can be contained in the
> Directory Table is 65536.
>
>
>> The max number of files possible using FAT-32: 268,435,437
>>
>> The number 268,435,437 (max number of files possible under FAT32) is
>> equal to 16k x 16k (ie, 16k directories, each with 16k files) or 4k x
>> 64k (ie - 4k directories, each with 64k files).
>
>
> Sigh... Where did you get this mathematical explanation for the
> 268,435,437 file or cluster limit on FAT32? FAT32 is 32 bits wide but 4
> bits are reserved or unused so it is really 28 bits wide and 2^28 less a
> couple of bits (for something or other, that I am not sure why) =
> 268,435,437 clusters.
>
>
>> The limiting factor for the number of files on a win-98 system (under
>> FAT-32) will be the number of allocation units available on the
>> volume. You can't have more files than you have allocation units (aka
>> clusters).
>
>
> Well duh! And you can't puts 3 pints in a quart!
>
>
>> As far as this being a long file name issue, I don't think so. The
>> size of the directory tables are variable under FAT-32 and can grow as
>> needed.
>
>
> Well think again! Long filenames require at least 2 extra directory
> object entries, so unless you figured out how to "widen" the 16 bit
> field, long file names will reduce the number of available directory
> objects.
>
> I expect that you will soon tell us that you have blown FAT32's 4gb file
> size limit!
>
> John
>

 

[Tim Slattery]

John John <audetweld@nbnet.nb.ca> wrote:

>98 Guy wrote:
>
>> The claimed max number of files per directory is 65535.
>
>You seem to think (or you are implying) that this is incorrect, by one
>of your yet unknown methods are we to assume that you can also overcome
>this limitation? In the Directory Table the object's first cluster is
>held in a 16 bits wide feild so, being that objects cannot share
>clusters, the maximum number of objects that can be contained in the
>Directory Table is 65536.

Actually, a FAT32 directory could hold many more entries than this.
The specification says that FAT32 drivers must have this limit. That's
because a FAT32 directory is always searched sequentially, so the
bigger it gets, the more time you spend searching it. NTFS
directories, by contrast, are stored as BTrees (balanced trees), which
makes searching *much* faster. (It does slow down the process of
retrieving all file names, but not hugely.)

The spec says this:

<quote>
* DIR_FileSize is a 32-bit field. For FAT32 volumes, your FAT
file system driver must not allow a cluster chain to be created that
is longer than 0x100000000 bytes, and the last byte of the last
cluster in a chain that long cannot be allocated to the file. This
must be done so that no file has a file size > 0xFFFFFFFF bytes. This
is a fundamental limit of all FAT file systems. The maximum allowed
file size on a FAT volume is 0xFFFFFFFF (4,294,967,295) bytes.

* Similarly, a FAT file system driver must not allow a directory
(a file that is actually a container for other files) to be larger
than 65,536 * 32 (2,097,152) bytes.
</quote>

and since each entry is 32 bytes long, that means no more than 65,536
entries per directory.

--
Tim Slattery
MS MVP(DTS)
Slattery_T@bls.gov
http://members.cox.net/slatteryt

 

[John John]

Tim Slattery wrote:

> John John <audetweld@nbnet.nb.ca> wrote:
>
>
>>98 Guy wrote:
>>
>>
>>>The claimed max number of files per directory is 65535.
>>
>>You seem to think (or you are implying) that this is incorrect, by one
>>of your yet unknown methods are we to assume that you can also overcome
>>this limitation? In the Directory Table the object's first cluster is
>>held in a 16 bits wide field so, being that objects cannot share
>>clusters, the maximum number of objects that can be contained in the
>>Directory Table is 65536.
>
>
> Actually, a FAT32 directory could hold many more entries than this.
> The specification says that FAT32 drivers must have this limit. That's
> because a FAT32 directory is always searched sequentially, so the
> bigger it gets, the more time you spend searching it. NTFS
> directories, by contrast, are stored as BTrees (balanced trees), which
> makes searching *much* faster. (It does slow down the process of
> retrieving all file names, but not hugely.)
>
> The spec says this:
>
> <quote>
> * DIR_FileSize is a 32-bit field. For FAT32 volumes, your FAT
> file system driver must not allow a cluster chain to be created that
> is longer than 0x100000000 bytes, and the last byte of the last
> cluster in a chain that long cannot be allocated to the file. This
> must be done so that no file has a file size > 0xFFFFFFFF bytes. This
> is a fundamental limit of all FAT file systems. The maximum allowed
> file size on a FAT volume is 0xFFFFFFFF (4,294,967,295) bytes.
>
> * Similarly, a FAT file system driver must not allow a directory
> (a file that is actually a container for other files) to be larger
> than 65,536 * 32 (2,097,152) bytes.
> </quote>
>
> and since each entry is 32 bytes long, that means no more than 65,536
> entries per directory.

Isn't the DIR_FileSize field used to record the size of the file? Hence
the 4,294,967,295 bytes maximum?

Shouldn't the maximum number of objects in the directory be limited by
DIR_FstClusLO or DIR_FstClusHI fields? Isn't the file location just
kept by the first cluster in the Directory Table, and isn't the rest of
the cluster map outside of the Directory Table, just sort of "daisy
chained" within the actual and successive clusters?

Much of this is new territory to me -) so I'm just being inquisitive, I
always appreciate getting good solid information on some of these
scantly documented (and complicated!) subjects.

John

 

[Tim Slattery]

John John <audetweld@nbnet.nb.ca> wrote:


>> The spec says this:
>>
>> <quote>
>> * DIR_FileSize is a 32-bit field. For FAT32 volumes, your FAT
>> file system driver must not allow a cluster chain to be created that
>> is longer than 0x100000000 bytes, and the last byte of the last
>> cluster in a chain that long cannot be allocated to the file. This
>> must be done so that no file has a file size > 0xFFFFFFFF bytes. This
>> is a fundamental limit of all FAT file systems. The maximum allowed
>> file size on a FAT volume is 0xFFFFFFFF (4,294,967,295) bytes.
>>
>> * Similarly, a FAT file system driver must not allow a directory
>> (a file that is actually a container for other files) to be larger
>> than 65,536 * 32 (2,097,152) bytes.
>> </quote>
>>
>> and since each entry is 32 bytes long, that means no more than 65,536
>> entries per directory.
>
>Isn't the DIR_FileSize field used to record the size of the file? Hence
>the 4,294,967,295 bytes maximum?

Exactly

>Shouldn't the maximum number of objects in the directory be limited by
>DIR_FstClusLO or DIR_FstClusHI fields?

As the quote shows, the 65,536 entries limit is purely arbitrary. The
spec says that any conforming driver must impose this limit. It's not
the result of anything in the structure/

> Isn't the file location just
>kept by the first cluster in the Directory Table, and isn't the rest of
>the cluster map outside of the Directory Table, just sort of "daisy
>chained" within the actual and successive clusters?

Yes, the directory entry points to the first cluster (or allocation
unit). To find where the rest of the file is, you have to look in the
File Allocation Table itself, where the entry for a given cluster will
point to the next cluster in the file. And another chain keeps track
of free space.

>Much of this is new territory to me -) so I'm just being inquisitive, I
>always appreciate getting good solid information on some of these
>scantly documented (and complicated!) subjects.

There is solid documentation of FAT32 (unlike NTFS). It's here:
http://www.microsoft.com/whdc/system/platform/firmware/fatgen.mspx

MS does not (AFAIK) make that kind of documentation of NTFS available
- at least without paying for it. There is an open source program
that's trying to make NTFS available for Linux. They have published
documentation at http://www.linux-ntfs.org/doku.php

--
Tim Slattery
MS MVP(Shell/User)
Slattery_T@bls.gov
http://members.cox.net/slatteryt

 

[John John]

Tim Slattery wrote:

> As the quote shows, the 65,536 entries limit is purely arbitrary. The
> spec says that any conforming driver must impose this limit. It's not
> the result of anything in the structure/

I see, the limit is imposed be the file system driver. Thanks for
steering me straight on this.

John

 

[98 Guy]

I performed a series of serial file generations to see how many files
could be created in a subdirectory while changing the length of the
file name.

The following table shows the file-name length and the corresponding
number of files that could be created before this eror was generated:

The directory or file cannot be created

6.0 - 65,533
6.3 - 32,767
12.3 - 21,845
15.3 - 21,845
17.3 - 21,845
23.3 - 16,384
47.3 - 13,107
63.3 - 9,362
96.3 - 7,282

So in the first case, with a filename composed of 6 characters (and no
suffix) I was able to create 65,533 files. In the second case, the
filename was composed of 6-characters.3-characters, and the directory
would only hold 32,767 of those files. So as the filename length
increases, there is a decrease in the number of possible files.

According to the OP:

> The target 'contracts' directory is 599MB and contains 16135
> objects. We are appending a few hundred files. We get the
> following error:
> 'The directory or file cannot be created.

For the above to happen, the existing 16,135 files would have to have
long file names (about 20 characters, more likely 23 characters,
possibly slightly more).

 

[Franc Zabkar]

On Thu, 25 Oct 2007 12:44:07 -0400, Tim Slattery <Slattery_T@bls.gov>
put finger to keyboard and composed:

>* Similarly, a FAT file system driver must not allow a directory
>(a file that is actually a container for other files) to be larger
>than 65,536 * 32 (2,097,152) bytes.
></quote>
>
>and since each entry is 32 bytes long, that means no more than 65,536
>entries per directory.

I'm thinking that this restriction could in extreme circumstances
result in file system corruption. For example, let's assume that the
directory is full (65,536 entries) and that the machine has been
booted in Win9x real DOS mode. What happens if you try to create an
additional file? AFAICS, DOS sees the existing LFNs as deleted files
(each entry has a leading 0xE5 flag byte), so wouldn't DOS then
overwrite one of these LFNs? Or does Win9x DOS understand that these
entries are LFNs even though it doesn't support them?

- Franc Zabkar
--
Please remove one 'i' from my address when replying by email.

 

[teebo]

> I performed a series of serial file generations to see how many files
> could be created in a subdirectory while changing the length of the
> file name.
>
> 6.0 - 65,533
> 6.3 - 32,767

strange that it starts to use LFN-names when you add extension to
the filenames? are you sure you didn't got lowercase in the
fileextensions somehow?

> 12.3 - 21,845
> 15.3 - 21,845
> 17.3 - 21,845
> 23.3 - 16,384
> 47.3 - 13,107
> 63.3 - 9,362
> 96.3 - 7,282

 

[teebo]

> Actually, a FAT32 directory could hold many more entries than this.
> The specification says that FAT32 drivers must have this limit. That's

nice to hear that it is just a rule with a number they chosed to be
"enough"
and not something forced by the format. I guess then you could relatively
easily
modify your filesystem drivers to a number you like more.
(but some of your file-utilities may not see all your files then, and
verry sloppily coded ones could perhaps crash)

That said, I must say that I think 65536 is way more than enough that it
is,
if you want more than 10000 files in the same directory without hierachy,
your program should really use some database instead.

 

[teebo]

>> * Similarly, a FAT file system driver must not allow a directory
>> (a file that is actually a container for other files) to be larger
>> than 65,536 * 32 (2,097,152) bytes.

good to know that I never need more than 2MB to hold a directory list in
memory

> I'm thinking that this restriction could in extreme circumstances
> result in file system corruption. For example, let's assume that the
> directory is full (65,536 entries) and that the machine has been
> booted in Win9x real DOS mode. What happens if you try to create an
> additional file? AFAICS, DOS sees the existing LFNs as deleted files

nope! the LFNs are not marked as deleted files.
The first char in the filename-part in the LFN-posts are a letter
that describes number or LFN-posts the LFN-filename have. A is one LFN-post
B is two, C is three etc.

the special marking of the LFN-posts is instead by using fileattribute
DiskLabel-System-Hidden-ReadOnly

> (each entry has a leading 0xE5 flag byte), so wouldn't DOS then
> overwrite one of these LFNs? Or does Win9x DOS understand that these
> entries are LFNs even though it doesn't support them?

dos without LFN-support understands them as "bad" and don't touch them.

(And if you run old scandisk type of diskutilities, like norton disk
doctor etc,
and the find these "errorus lines" and removes them, nothing more than the
long filenames is destroyed. the files can allways be used with the short
names)