[Etherlab-users] Best filesystem for DLS?

[Sebastien BLANCHET]

Dear Wilhelm,

You are right, there was a configuration error in job.xml the block size 
was only 10. I have increased it to 2000 to get fewer files.

I will do some tests with other filesystems that can handle more files.

I am sampling 20K signals at 1Hz without trigger and I keep them for one 
year for technical forensic analysis.

I do not know if dls is designed to operate at such scale, but it is the 
easiest archiving tool I have ever found and it seems to support the 
workload.

Best Regards,
---
Sebastien BLANCHET



On 3/23/21 5:30 PM, Dr.-Ing. Wilhelm Hagemeister wrote:
> Hi Sebastien,
> 
> increasing the number of inodes helps.
> 
> dumpe2fs -h /dev/sdxx helps to see what are your limits now.
> 
> Regarding dls please check, if you get restarts of the sampling
> processes due to sample time constrains. Usually this is the case with
> data sources form realtime processes with high jitter. This results in
> creating new chunks (and a lot of new files). So have a look in your
> channel directory (or in dlsgui) if you have lots of chunks which have a
> breaks of only a few seconds.
> Also you get a log entry in your messages (or journalctl) saying
> something like: "Time diff of xxx us (expected xxx us, error is percent
> xxx %)"
> 
> Also increasing the numbers for blocksize and reduction in the channel
> setup decreases the number of files.
> 
> Are you sampling constantly or just small chunks with a trigger for your
> job? If later is the case, try to increase chunksize by constant
> sampling over longer periods.
> 
> "btrfs" and "xfs" have no inode limitation but we have no experience
> with that.
> 
> Regards Wilhelm Hagemeister
> 
> Am 23.03.21 um 17:05 schrieb Merkel, Amos:
>> Hi,
>>
>>
>> I have never tried, but couldn't you simply manually define a
>> higher bytes-per-inode ratio with mkfs.ext4 -i ?
>>
>> Arch-Wiki has some explanations on the topic:
>> https://wiki.archlinux.org/index.php/Ext4#Bytes-per-inode_ratio
>> <https://wiki.archlinux.org/index.php/Ext4#Bytes-per-inode_ratio>
>>
>>
>> Greetings,
>>
>> Amos
>>
>> ------------------------------------------------------------------------
>> *Von:* Etherlab-users <etherlab-users-bounces at etherlab.org> im Auftrag
>> von blanchet at iram.fr <blanchet at iram.fr>
>> *Gesendet:* Dienstag, 23. März 2021 16:35:28
>> *An:* etherlab-users at etherlab.org
>> *Betreff:* [Etherlab-users] Best filesystem for DLS?
>>   
>> Hi,
>>
>> Short version:
>> -------------------
>> What is the best filesystem for DLS data ?
>>
>> Long version:
>> -------------------
>> I am running DLS to archive a lot of signals (about 20K), and I am
>> facing an issue with the number of files.
>>
>> I have a 1TB EXT4 filesystem to store the data. It is 50% full but there
>> is no more free inodes, because DLS creates many files and directories.
>> According to the output of “df -i”, DLS has already created about 78M of
>> files.
>>
>> I understand that I have to switch to another filesystem that supports a
>> bigger number of files.
>> And I wonder, what is the best filesystem for such a case.
>>
>> Regards,
>> —
>> Sebastien BLANCHET
>> -- 
>> Etherlab-users mailing list
>> Etherlab-users at etherlab.org
>> https://lists.etherlab.org/mailman/listinfo/etherlab-users
>> <https://lists.etherlab.org/mailman/listinfo/etherlab-users>
>>
> 
>