File Writer
Quote from AdminTC on 11/05/2021, 11:00File Writer block:
Stream/Write any RTSA data as there is .rtsa", ".tag" and ".dat" (I/Q, Spectra, Video, Tracking etc.) to disk:
Left hand side input:
- Stream
Right hand side output:
- Monitor
The File Writer block records any incoming data stream to disk. It offers a Monitor output to check the data in real-time, which can also be used to connect additional control blocks like the Control Sequencer.
When writing raw IQ data make sure you have a well performing SSD as the data rate can exceed several hundred megabytes per second (depending on settings)! You might consider to use the IQ Demodulator to reduce the data size.
You can set a lot of useful parameter so it will perfectly fit your needs:
- Single or Multi Stream support
- Autostart feature
- Automatically include index and/or timestamp in filenames
- Auto Split file (reduce single file size)
- Auto rotate (prevent a folder to overflow by overwriting the old data first | FIFO feature incl. adjustable file rotations)
- Prevent overwriting old files
- Adjustable data compression
- Adjustable folder memory limitation
- Adjustable record delay
BTW: We already have a nice posting helping you to setup up a Circular Buffer to prevent a full harddisk. It will limit the harddisk space by automatically overwriting the old data.
While writing the data you get some additional information e.g. a data preview via a small histogram screenshot, a compact waterfall display and some statistics as for instance the Write Rate (MBytes/s), the Buffer Fill and the Compression Ratio as graphs:
Some typical missions:
File Writer block:
Stream/Write any RTSA data as there is .rtsa", ".tag" and ".dat" (I/Q, Spectra, Video, Tracking etc.) to disk:
Left hand side input:
- Stream
Right hand side output:
- Monitor
The File Writer block records any incoming data stream to disk. It offers a Monitor output to check the data in real-time, which can also be used to connect additional control blocks like the Control Sequencer.
When writing raw IQ data make sure you have a well performing SSD as the data rate can exceed several hundred megabytes per second (depending on settings)! You might consider to use the IQ Demodulator to reduce the data size.
You can set a lot of useful parameter so it will perfectly fit your needs:
- Single or Multi Stream support
- Autostart feature
- Automatically include index and/or timestamp in filenames
- Auto Split file (reduce single file size)
- Auto rotate (prevent a folder to overflow by overwriting the old data first | FIFO feature incl. adjustable file rotations)
- Prevent overwriting old files
- Adjustable data compression
- Adjustable folder memory limitation
- Adjustable record delay
BTW: We already have a nice posting helping you to setup up a Circular Buffer to prevent a full harddisk. It will limit the harddisk space by automatically overwriting the old data.
While writing the data you get some additional information e.g. a data preview via a small histogram screenshot, a compact waterfall display and some statistics as for instance the Write Rate (MBytes/s), the Buffer Fill and the Compression Ratio as graphs:
Some typical missions:
Quote from nothingman1020 on 03/05/2022, 20:59I am trying to create a series of missions that each automatically start, write a single file (based on a time duration or file size limit) and then automatically move on to the next mission. I'm sure there must be a way to do this but have not been able to figure out the correct File Writer and Control Sequencer settings in order to accomplish this. I have the missions chained together via control sequencer, but the operator has to actually stop the file writing or pause the Spectran to make it move on to the next mission.
For file writer, regardless of chaining missions with control sequencer, I cannot find a combination of settings that have it just write a single file of a fixed time/size and then stop.
Also, I played a little with the folder size limit settings and it did not seem to work correctly.
I am trying to create a series of missions that each automatically start, write a single file (based on a time duration or file size limit) and then automatically move on to the next mission. I'm sure there must be a way to do this but have not been able to figure out the correct File Writer and Control Sequencer settings in order to accomplish this. I have the missions chained together via control sequencer, but the operator has to actually stop the file writing or pause the Spectran to make it move on to the next mission.
For file writer, regardless of chaining missions with control sequencer, I cannot find a combination of settings that have it just write a single file of a fixed time/size and then stop.
Also, I played a little with the folder size limit settings and it did not seem to work correctly.
Quote from mm_dev on 04/05/2022, 09:38Starting and Stopping the FileWriter based on time just needs a Start, Delay and Stop command in the Command Sequencer:
And the use the "Start" button in the Control Sequencer to start the recording. You may need to add a few seconds extra for the device connection to be established though (delay is clock time, not measurement time). Repeat that sequence (or use a "Load Mission" command) afterwards for any additional recording (and maybe add a delay in between for safety). FileWriter doesn't need any special settings, except maybe disabling "Check Overwrite" to prevent confirmation dialogs.
Quote from nothingman1020 on 03/05/2022, 20:59Also, I played a little with the folder size limit settings and it did not seem to work correctly.
The size check is only applied to the file set currently being recorded (when using the rotation feature and indices or timestamps), other files in the folder are not accounted for (for performance reasons).
Starting and Stopping the FileWriter based on time just needs a Start, Delay and Stop command in the Command Sequencer:
And the use the "Start" button in the Control Sequencer to start the recording. You may need to add a few seconds extra for the device connection to be established though (delay is clock time, not measurement time). Repeat that sequence (or use a "Load Mission" command) afterwards for any additional recording (and maybe add a delay in between for safety). FileWriter doesn't need any special settings, except maybe disabling "Check Overwrite" to prevent confirmation dialogs.
Quote from nothingman1020 on 03/05/2022, 20:59Also, I played a little with the folder size limit settings and it did not seem to work correctly.
The size check is only applied to the file set currently being recorded (when using the rotation feature and indices or timestamps), other files in the folder are not accounted for (for performance reasons).
Quote from AdminTC on 08/03/2023, 09:24The File Writer now offers a Discontinuities/Second chart & free disk space information.
This will help a lot to keep control over the recording:
The File Writer now offers a Discontinuities/Second chart & free disk space information.
This will help a lot to keep control over the recording:
Quote from Leo on 22/05/2023, 09:07premise:IQ Rate is 92MHz;
1.When I set the compression=0,the write rate was 737MB/s,and the compression ratio is 1.0:1.The IQ data collected at this time was complete.
2.When I set the compression=1(default setting),the write rate was 250MB/s,and the compression ratio is 2.8:1.Was the IQ data collected at this time not complete?
3.what is the meanning/use of compression and how should I set it up correctly?
premise:IQ Rate is 92MHz;
1.When I set the compression=0,the write rate was 737MB/s,and the compression ratio is 1.0:1.The IQ data collected at this time was complete.
2.When I set the compression=1(default setting),the write rate was 250MB/s,and the compression ratio is 2.8:1.Was the IQ data collected at this time not complete?
3.what is the meanning/use of compression and how should I set it up correctly?
Uploaded files:Quote from mm_dev on 22/05/2023, 20:32Compression does the same as in any other context: (repeated) long sequences of data are replaced by shorter versions and a lookup table, on decompression the short version is expanded again to the original long sequence. This can save significant amount of bandwidth at the expense of slightly higher CPU load and memory consumption.
It is usually safe to just leave the default value of 1: Disabling compression yields no real benefit as the way higher IO speeds will also need more CPU power as well, and the bandwidth requirements can easily exceed your systems IO capacity (even high end desktop SSDs will usually struggle to maintain transfer speeds above 1 GB/s over longer periods of time, and transfer speed requirements scale basically linear with RTBW bandwidth and/or number of streams). On the other hand higher compression settings will quickly run into diminishing returns, but that also depends quite a bit on the nature of the recorded data. Still might be useful if you're really short on disk space or IO bandwidth.
Compression does the same as in any other context: (repeated) long sequences of data are replaced by shorter versions and a lookup table, on decompression the short version is expanded again to the original long sequence. This can save significant amount of bandwidth at the expense of slightly higher CPU load and memory consumption.
It is usually safe to just leave the default value of 1: Disabling compression yields no real benefit as the way higher IO speeds will also need more CPU power as well, and the bandwidth requirements can easily exceed your systems IO capacity (even high end desktop SSDs will usually struggle to maintain transfer speeds above 1 GB/s over longer periods of time, and transfer speed requirements scale basically linear with RTBW bandwidth and/or number of streams). On the other hand higher compression settings will quickly run into diminishing returns, but that also depends quite a bit on the nature of the recorded data. Still might be useful if you're really short on disk space or IO bandwidth.
Quote from Sofon on 23/05/2023, 10:39Quote from mm_dev on 22/05/2023, 20:32Compression does the same as in any other context: (repeated) long sequences of data are replaced by shorter versions and a lookup table, on decompression the short version is expanded again to the original long sequence.
I had recorded the IQ data using File Writer ever before. but I don't pay attention that the compression level default value is "1" not "0". So how can I decompress the short version to expand again to the original long sequence on RTSA?
Quote from mm_dev on 22/05/2023, 20:32Compression does the same as in any other context: (repeated) long sequences of data are replaced by shorter versions and a lookup table, on decompression the short version is expanded again to the original long sequence.
I had recorded the IQ data using File Writer ever before. but I don't pay attention that the compression level default value is "1" not "0". So how can I decompress the short version to expand again to the original long sequence on RTSA?