C++ Just IQ Data Streaming
Quote from DevSF on 02/03/2021, 07:59The easiest way to get a real-time data stream from the Aaronia RTSA Suite PRO into your custom C++ application, is by running the RTSA Suite PRO and just asking for data with the HTTP Server block.
No configuration or device parameters changes are necessary since the default values within the RTSA Suite PRO mission are ok. Asking remotely for the status, looking at or changing the configuration of a RTSA Suite PRO mission will be posted in a separate topic.
In the topic RTSA Suite PRO HTTP Stream Server is the interface documentation for the HTTP Server block which already contains a C++ example called „RESTStreamer.zip".
Now lets take a closer look to that example.
If a RTSA Suite PRO mission contains a data source block connected to a HTTP Server block, the stream HTTP REST endpoint provides a continuous data stream of JSON objects. The data blocks are separated by a line feed (ASCII 10) and a record separator (ASCII 30) character.
For higher data rates the binary data values should be used. As described in the HTTP interface documentation, the format in this example is set to binary integer values, 16 bit each, by the HTTP Parameter „format=raw16".
For a higher dynamic range and for more accuracy the float values, 32 bit each, should be selected via the parameter "format=raw32".
The complete HTTP stream gets requested, for example, at this address: localhost, port 54664.
The result in the HTTP get request is: http://localhost:54664/stream?format=raw16
This will request an endless data stream of alternating JSON headers and data blocks.
It's repeated all the time as long as data is available. The JSON Object in the stream is the header of one data packet, followed by a payload of binary IQ data
( I and Q values interleaved).
{
"startTime":1614338274.734474,
"endTime":1614338274.734695,
"startFrequency":197609375,
"endFrequency":202390625,
"minPower":-2,
"maxPower":2,
"sampleSize":2,
"sampleDepth":1,
"payload":"iq",
"unit":"volt",
"antenna":{"name":""},
"scale":16384,
"samples":1448
}
��� ...binary data block... ���������������������������
The easiest way to get a real-time data stream from the Aaronia RTSA Suite PRO into your custom C++ application, is by running the RTSA Suite PRO and just asking for data with the HTTP Server block.
No configuration or device parameters changes are necessary since the default values within the RTSA Suite PRO mission are ok. Asking remotely for the status, looking at or changing the configuration of a RTSA Suite PRO mission will be posted in a separate topic.
In the topic RTSA Suite PRO HTTP Stream Server is the interface documentation for the HTTP Server block which already contains a C++ example called „RESTStreamer.zip".
Now lets take a closer look to that example.
If a RTSA Suite PRO mission contains a data source block connected to a HTTP Server block, the stream HTTP REST endpoint provides a continuous data stream of JSON objects. The data blocks are separated by a line feed (ASCII 10) and a record separator (ASCII 30) character.
For higher data rates the binary data values should be used. As described in the HTTP interface documentation, the format in this example is set to binary integer values, 16 bit each, by the HTTP Parameter „format=raw16".
For a higher dynamic range and for more accuracy the float values, 32 bit each, should be selected via the parameter "format=raw32".
The complete HTTP stream gets requested, for example, at this address: localhost, port 54664.
The result in the HTTP get request is: http://localhost:54664/stream?format=raw16
This will request an endless data stream of alternating JSON headers and data blocks.
It's repeated all the time as long as data is available. The JSON Object in the stream is the header of one data packet, followed by a payload of binary IQ data
( I and Q values interleaved).
{
"startTime":1614338274.734474,
"endTime":1614338274.734695,
"startFrequency":197609375,
"endFrequency":202390625,
"minPower":-2,
"maxPower":2,
"sampleSize":2,
"sampleDepth":1,
"payload":"iq",
"unit":"volt",
"antenna":{"name":""},
"scale":16384,
"samples":1448
}
��� ...binary data block... ���������������������������
Quote from DevSF on 02/03/2021, 08:12The RESTStreamer example is using the Qt framework, which is bringing up some advantages. Here especially the Qt framework simplifies the network access and the JSON data de- and encoding. Network operations and JSON object handling are common operations, if you have the requirement to use another framework, it is very likely it is providing similar features. Without a framework, well you have a bit more work to do, handling network sockets of the operating system, parsing the JSON data, ect..
(The Qt framework is available at their website https://www.qt.io/ and relatively easy to use. It is not content here to explain the Qt framework, have a look at their official Qt documentation.)
Example Preparation:
- Create a RTSA Suite mission with a HTTP Server block connected to a data source, here it is the IQ data output of the Spectran V6 block and configure your measurement paramaeters in the Spectran V6 block.
- To automatically start the Spectran V6 block when the mission gets loaded, add a Control Sequencer block to the Spectran V6 block, activate the Auto Start checkbox and add a start command in the hamburger menu of the Commands configuration. Save the mission under your preferred name, for example RemoteV6.rmix:
- To load the created mission every time the RTSA Suite gets started, go to the general RTSA Suite configuration menu Extras → Configuration at the left top of the RTSA Suite. In section Environment->Startup select Load Mission for the At startup parameter and select previously saved mission for the Mission parameter:
- Download and install the Qt framework, build and run your first „hello world“ program.
- Download and open the RESTStreamer example project with the Qt Creator, and run the example the first time.
The RESTStreamer example is using the Qt framework, which is bringing up some advantages. Here especially the Qt framework simplifies the network access and the JSON data de- and encoding. Network operations and JSON object handling are common operations, if you have the requirement to use another framework, it is very likely it is providing similar features. Without a framework, well you have a bit more work to do, handling network sockets of the operating system, parsing the JSON data, ect..
(The Qt framework is available at their website https://www.qt.io/ and relatively easy to use. It is not content here to explain the Qt framework, have a look at their official Qt documentation.)
Example Preparation:
- Create a RTSA Suite mission with a HTTP Server block connected to a data source, here it is the IQ data output of the Spectran V6 block and configure your measurement paramaeters in the Spectran V6 block.
- To automatically start the Spectran V6 block when the mission gets loaded, add a Control Sequencer block to the Spectran V6 block, activate the Auto Start checkbox and add a start command in the hamburger menu of the Commands configuration. Save the mission under your preferred name, for example RemoteV6.rmix:
- To load the created mission every time the RTSA Suite gets started, go to the general RTSA Suite configuration menu Extras → Configuration at the left top of the RTSA Suite. In section Environment->Startup select Load Mission for the At startup parameter and select previously saved mission for the Mission parameter:
- Download and install the Qt framework, build and run your first „hello world“ program.
- Download and open the RESTStreamer example project with the Qt Creator, and run the example the first time.
Quote from DevSF on 02/03/2021, 08:17The example contains the class RTSARestStreamer which is processing the network request and reacting on incoming data:
class RTSARestStreamer : public QObject
{
Q_OBJECT
public:
RTSARestStreamer(quint32 httpPort);
~RTSARestStreamer(void);
protected:
// QT htttp client
QNetworkAccessManager * mNetworkAccessManager;
// Reply from the HTTP server
QNetworkReply * mReply;
// Input buffer
QByteArray mBuffer;
// Number of IQ sample pairs in the current packet
int mPacketSamples;
// Previous sample end time to check for packet loss
double mPrevTime;
protected slots:
void onError(QNetworkReply::NetworkError code);
void onFinished(void);
void onReadyRead(void);
};
To handle network operations the QNetworkAccessManager class is used. The QNetworkManager returns a QNetworkReply and is handling the network requests and replies. The QNetworkReply class will asynchronously raise Qt Signals(Qt Signals & Slots) depending on its progress. A Qt Signal is used by Qt to handle for example asynchronous events. You can connect a method of a class (declared as Qt Slot) to a Qt Signal, which is then called if the related signal was raised.
The QNetworkReply raise three different Qt Signals, which are used here. In the RTSARestStreamer constructor the network manager is initialized to start the HTTP get request to the mentioned HTTP stream endpoint and connecting the three Qt Slots.
mNetworkAccessManager = new QNetworkAccessManager(this);
QUrl url("https://localhost:" + QString::number(httpPort) + "/stream?format=raw16");
QNetworkRequest req(url);
QNetworkReply mReply = mNetworkAccessManager→get(req);
connect(mReply, SIGNAL(error(QNetworkReply::NetworkError)), this, SLOT(onError(QNetworkReply::NetworkError)));
connect(mReply, SIGNAL(finished()), this, SLOT(onFinished()));
connect(mReply, SIGNAL(readyRead()), this, SLOT(onReadyRead()));
The onError(QNetworkReply::NetworkError) and the onFinished() slots are straight forward and not further described here. Data is processed in the onReadyRead() slot. And because of the endless stream the Qt Signal readyRead() will be triggered periodically as long the streaming is running and data is received. It is possible there is no complete JSON header or data packet received when the onReadyRead() is called, therefore the received data needs to be buffered (in a FIFO like manner) until a complete JSON header was received. If one complete JSON header was received, the related data gets removed from the buffer again and it continuous collecting the next header and data.
The example contains the class RTSARestStreamer which is processing the network request and reacting on incoming data:
class RTSARestStreamer : public QObject
{
Q_OBJECT
public:
RTSARestStreamer(quint32 httpPort);
~RTSARestStreamer(void);
protected:
// QT htttp client
QNetworkAccessManager * mNetworkAccessManager;
// Reply from the HTTP server
QNetworkReply * mReply;
// Input buffer
QByteArray mBuffer;
// Number of IQ sample pairs in the current packet
int mPacketSamples;
// Previous sample end time to check for packet loss
double mPrevTime;
protected slots:
void onError(QNetworkReply::NetworkError code);
void onFinished(void);
void onReadyRead(void);
};
To handle network operations the QNetworkAccessManager class is used. The QNetworkManager returns a QNetworkReply and is handling the network requests and replies. The QNetworkReply class will asynchronously raise Qt Signals(Qt Signals & Slots) depending on its progress. A Qt Signal is used by Qt to handle for example asynchronous events. You can connect a method of a class (declared as Qt Slot) to a Qt Signal, which is then called if the related signal was raised.
The QNetworkReply raise three different Qt Signals, which are used here. In the RTSARestStreamer constructor the network manager is initialized to start the HTTP get request to the mentioned HTTP stream endpoint and connecting the three Qt Slots.
mNetworkAccessManager = new QNetworkAccessManager(this);
QUrl url("https://localhost:" + QString::number(httpPort) + "/stream?format=raw16");
QNetworkRequest req(url);
QNetworkReply mReply = mNetworkAccessManager→get(req);
connect(mReply, SIGNAL(error(QNetworkReply::NetworkError)), this, SLOT(onError(QNetworkReply::NetworkError)));
connect(mReply, SIGNAL(finished()), this, SLOT(onFinished()));
connect(mReply, SIGNAL(readyRead()), this, SLOT(onReadyRead()));
The onError(QNetworkReply::NetworkError) and the onFinished() slots are straight forward and not further described here. Data is processed in the onReadyRead() slot. And because of the endless stream the Qt Signal readyRead() will be triggered periodically as long the streaming is running and data is received. It is possible there is no complete JSON header or data packet received when the onReadyRead() is called, therefore the received data needs to be buffered (in a FIFO like manner) until a complete JSON header was received. If one complete JSON header was received, the related data gets removed from the buffer again and it continuous collecting the next header and data.
Quote from DevSF on 02/03/2021, 08:24If the C++ application is not fetching up enough data from the HTTP Server block, the buffer of the HTTP Server block gets full. If the buffer is full, it gets an overflow and data is lost.
If you read until here, I guess you have the example already running. First make sure you do not have any data lost. As described in the HTTP interface document, this example compares the start time of each JSON header against the stop time of the previous received JSON header and dumps out any missed time slots (i.e. data lost) in the application output:
Okay, lets produce a buffer overflow by adding a break point at the beginning of the onReadyRead() method and run the application in debug mode:
When the program stops at the breakpoint, have a look to the HTTP Server block in the RTSA Suite. In the view area of the HTTP Server block you should see the buffer load and the dropped/s value are increasing:
Having a good trimmed hardware and software system, the buffer load should mostly be around zero and the dropped/s value in the connection statistics should always be zero. Remove the breakpoint and continue to debug the application to empty the buffer again.
If the C++ application is not fetching up enough data from the HTTP Server block, the buffer of the HTTP Server block gets full. If the buffer is full, it gets an overflow and data is lost.
If you read until here, I guess you have the example already running. First make sure you do not have any data lost. As described in the HTTP interface document, this example compares the start time of each JSON header against the stop time of the previous received JSON header and dumps out any missed time slots (i.e. data lost) in the application output:
Okay, lets produce a buffer overflow by adding a break point at the beginning of the onReadyRead() method and run the application in debug mode:
When the program stops at the breakpoint, have a look to the HTTP Server block in the RTSA Suite. In the view area of the HTTP Server block you should see the buffer load and the dropped/s value are increasing:
Having a good trimmed hardware and software system, the buffer load should mostly be around zero and the dropped/s value in the connection statistics should always be zero. Remove the breakpoint and continue to debug the application to empty the buffer again.
Quote from DevSF on 16/03/2021, 10:07The data format set by the parameter "format=raw16" with its 16 bit integer values has a limited dynamic range. Since RTSA Suite version 1.5.152.8366 the HTTP get request parameter scale=[value] for the 16 bit integer format is supported. So you can set the scale of the received data values from the RTSA Suite. The default scale for the 16bit integer values is 16384. To scale it by half the HTTP get request will look like this:
http://localhost:54664/stream?format=raw16&scale=8192
We have also added a 16bit float value format set by the HTTP parameter "format=float16". It reduces the data rate nearly by half compared to the 32 bit float values.
The data format set by the parameter "format=raw16" with its 16 bit integer values has a limited dynamic range. Since RTSA Suite version 1.5.152.8366 the HTTP get request parameter scale=[value] for the 16 bit integer format is supported. So you can set the scale of the received data values from the RTSA Suite. The default scale for the 16bit integer values is 16384. To scale it by half the HTTP get request will look like this:
http://localhost:54664/stream?format=raw16&scale=8192
We have also added a 16bit float value format set by the HTTP parameter "format=float16". It reduces the data rate nearly by half compared to the 32 bit float values.