Forum
Please or Register to create posts and topics.

Read http stream spectra in raw format with c++ using boost library

I am trying to read streamed spectra from the http server block.

  • The request is localhost:54664/stream?format=int16
  • I performe an async read with boost, find the record separator of the packet and decode the json header which reads similar to this:{"startTime":1606211620.580862,"endTime":1606211620.597246,"startFrequency":2409500000,"endFrequency":2410500000,"minPower":-95,"maxPower":5,"sampleSize":64,"sampleDepth":1,"payload":"spectra","unit":"dbm","scale":100,"samples":256}
  • Then I read the binary part of the package which contains in the above case 256 spectra with 64 bins. This works for 16 bins, but not for the entire number of samples. For 256 bins the data is scrambled alltogether.

Do you have any hints on how to correctly decode the package further? Help is very much appreciated.

I work on windows 64 bit with VS2022 (Version 17.3.5) and the latest boost library.

Quote from streamwise on 10/10/2022, 16:38
  • Then I read the binary part of the package which contains in the above case 256 spectra with 64 bins. This works for 16 bins, but not for the entire number of samples. For 256 bins the data is scrambled alltogether.

What do you mean with "scrambled alltogether" ? Can you show an example?

Also please provide the mission you're using so we can check all settings and try to reproduce the issue without guessing.

Dear Team

Thank you for your fast reply. Please find the mission attached. It is a simple read of a demo IQ data and a html stream of the spectrum.

My code sends this request: localhost:54664/stream?format=int16

The relevant code section of the async read is attached in the on_read.cpp file (please note this is development level). It does the following:

  • search for the record separator.
  • parse the json header file and decode relevant data.
  • Read the (binary) data part from the buffer. Here is my problem. If I set the FFT to 16 bin the data I get seem correct. However, I cannot read the entire data block of the receive buffer. Roughly one third of the data looks OK, the rest is filled with unreasonable data.
  • If I increase the number of bins (on the v6 side), I cannot manage to get any reasonable data.

Any idea what I am doing wrong?

 

Uploaded files:

Unfortunately a FileReader based mission without the associated record is of no use for reproducing the issue. Really need to see the data that is being fed to your program. So please either provide the record you're using, or a Spectran V6 based mission using the measurement parameters you're using that cause the problem.

One small issue that I've spotted so far is that you're only looking for the record separator for splitting of the JSON header, not the newline that should preceed it. Granted it is extremely unlikely for the record separator to be contained within the JSON header, but could lead to some other issues over time.

Not seeing any obvious parsing issues, but then that is an incomplete codesample, so there might be issues in the calling code.

Before sending any big records: Have you tried to verify with a second instance of the RTSA?

Mission 1: Basically your mission with the HTTP server block
Mission 2: A mission with HTTP client block, streaming the data over 127.0.0.1?

If that works, you can verify your local network traffic for e.g. with Wireshark to track down the issue with the code. I'm not an expert in Boost, so, you might stick to our code examples provided in the forum.

Dear fw_dev and mm_dev

Thank you both for your comments.

  1. I tried with a second rtsa, I can successfully read the streamed data with the http client.
  2. I also tried to split the data with the newline and the record separator. This does, however, not work. I used the splitting I found in your reststreamer example.
  3. In my example below, I read a buffer of at least 4096 bytes. Then I split the header  and parse the json. The output of my code is below. To me it looks as if sample 0 to 57 is fine (sample # after the *), and after that the data is incorrectly decoded.
  4. The input file to my mission is attached (its from you examples).
  5. The entire code is attached as well.

 

Split / offset 381 / 0
****{"startTime":1606211623.17363,"endTime":1606211623.180862,"startFrequency":2409500000,"endFrequency":2410500000,"minPower":-95,"maxPower":5,"sampleSize":16,"sampleDepth":1,"payload":"spectra","unit":"dbm","scale":100,"samples":452}

# of samples: 452
# of bins : 16
# of bytes reads : 4449
offset : 382
scale : 100
* 0 -73.32 -93.5 -70.48 -63.69 -59.06 -53.29 -47.51 -38.26 -11.76 -5.1 -11.76 -38.29 -47.41 -53.44 -59 -63.52
* 1 -70.83 -88.44 -71.21 -63.84 -58.48 -53.56 -47.35 -38.32 -11.76 -5.1 -11.76 -38.3 -47.34 -53.71 -58.16 -64.98
* 2 -71.07 -86.58 -71.15 -63.57 -58.51 -53.51 -47.45 -38.26 -11.76 -5.1 -11.76 -38.29 -47.46 -53.4 -58.78 -63.74
* 3 -72.95 -89.94 -70.41 -65.49 -57.75 -53.74 -47.37 -38.3 -11.76 -5.1 -11.76 -38.21 -47.62 -53.51 -58.24 -63.73
* 4 -69.56 -77.12 -74.05 -63.11 -58.9 -53.38 -47.57 -38.22 -11.76 -5.1 -11.76 -38.29 -47.42 -53.6 -58.49 -63.6
* 5 -74.03 -85.14 -71.39 -64.13 -58.07 -53.74 -47.37 -38.28 -11.75 -5.1 -11.76 -38.26 -47.38 -53.74 -58.44 -63.83
* 6 -70.21 -81.02 -70.09 -63.61 -58.88 -53.44 -47.36 -38.33 -11.76 -5.1 -11.76 -38.25 -47.59 -53.41 -58.54 -63.41
* 7 -70.27 -82.47 -71.84 -64.08 -58.38 -53.67 -47.41 -38.29 -11.76 -5.1 -11.76 -38.33 -47.29 -53.65 -58.47 -64.18
* 8 -72.17 -78.77 -71.83 -64.04 -58.43 -53.75 -47.35 -38.27 -11.76 -5.1 -11.76 -38.29 -47.39 -53.45 -59.14 -64.04
* 9 -69.33 -83.12 -71.03 -62.94 -59.02 -53.75 -47.22 -38.35 -11.76 -5.1 -11.76 -38.27 -47.5 -53.25 -58.74 -63.77
* 10 -70.3 -79.43 -71.54 -63.89 -58.57 -53.6 -47.35 -38.31 -11.76 -5.1 -11.76 -38.26 -47.53 -53.39 -58.58 -63.95
* 11 -71.44 -91.08 -70.12 -64.24 -58.63 -53.41 -47.44 -38.29 -11.76 -5.1 -11.76 -38.27 -47.39 -53.52 -58.47 -64.14
* 12 -69.75 -85.16 -70.71 -64.65 -58.25 -53.69 -47.36 -38.27 -11.76 -5.1 -11.76 -38.26 -47.36 -53.54 -58.89 -63.65
* 13 -71.09 -86 -69.27 -63.3 -59.24 -53.46 -47.4 -38.28 -11.76 -5.1 -11.76 -38.25 -47.32 -53.68 -58.42 -63.87
* 14 -68.13 -82.31 -72.66 -64.28 -57.96 -53.98 -47.27 -38.28 -11.76 -5.1 -11.76 -38.3 -47.39 -53.62 -58.52 -63.42
* 15 -70.69 -83.51 -71.37 -63.75 -58.47 -53.6 -47.38 -38.28 -11.76 -5.1 -11.76 -38.24 -47.58 -53.3 -59.06 -63.18
* 16 -70.15 -84.3 -73.05 -63.2 -58.69 -53.69 -47.38 -38.29 -11.76 -5.1 -11.75 -38.27 -47.23 -54.12 -58.14 -63.24
* 17 -70.16 -82.58 -70.04 -64.15 -58.37 -53.69 -47.3 -38.33 -11.76 -5.1 -11.76 -38.27 -47.51 -53.21 -58.94 -63.41
* 18 -71.85 -84.71 -72.65 -63.42 -58.64 -53.65 -47.39 -38.29 -11.76 -5.1 -11.76 -38.31 -47.34 -53.31 -59.15 -63.83
* 19 -74.55 -84.13 -68.31 -64.49 -58.89 -53.44 -47.35 -38.3 -11.76 -5.1 -11.76 -38.33 -47.37 -53.4 -58.39 -65.37
* 20 -70.26 -82.83 -69.12 -64.71 -58.21 -53.47 -47.41 -38.33 -11.76 -5.1 -11.76 -38.29 -47.47 -53.34 -58.73 -63.94
* 21 -71.79 -90.61 -70.99 -63.82 -58.61 -53.5 -47.42 -38.27 -11.76 -5.1 -11.76 -38.24 -47.53 -53.42 -59 -62.98
* 22 -71.03 -86.93 -68.69 -64.95 -58.36 -53.53 -47.42 -38.3 -11.76 -5.1 -11.76 -38.31 -47.42 -53.29 -59.03 -64.4
* 23 -69.33 -83.57 -71.43 -62.75 -59.01 -53.54 -47.32 -38.34 -11.76 -5.1 -11.76 -38.31 -47.4 -53.51 -58.59 -63.86
* 24 -70.31 -90.74 -71.14 -62.97 -59.47 -53.27 -47.47 -38.27 -11.76 -5.1 -11.76 -38.25 -47.37 -53.96 -58.1 -63.18
* 25 -68.67 -81.23 -73.25 -63.02 -59.12 -53.55 -47.39 -38.25 -11.76 -5.1 -11.76 -38.25 -47.41 -53.68 -58.66 -63.45
* 26 -69.37 -75.61 -71.51 -64 -58.13 -53.76 -47.42 -38.24 -11.76 -5.1 -11.76 -38.23 -47.54 -53.33 -58.74 -64.05
* 27 -70.69 -93.88 -70.76 -63.78 -58.47 -53.96 -47.25 -38.26 -11.76 -5.1 -11.76 -38.27 -47.42 -53.78 -58.4 -63.4
* 28 -70.32 -87.21 -70.14 -64.66 -58.66 -53.35 -47.47 -38.27 -11.76 -5.1 -11.76 -38.28 -47.4 -53.54 -58.4 -64.46
* 29 -74.66 -78.96 -72.42 -63.26 -58.92 -53.36 -47.5 -38.27 -11.76 -5.1 -11.76 -38.32 -47.35 -53.22 -59.56 -62.72
* 30 -69.61 -87.56 -70.67 -63.63 -58.67 -53.7 -47.23 -38.38 -11.76 -5.1 -11.76 -38.28 -47.4 -53.53 -58.62 -64.07
* 31 -71.76 -81.19 -71.46 -63.85 -59.03 -53.48 -47.35 -38.29 -11.76 -5.1 -11.76 -38.31 -47.33 -53.59 -58.49 -64.34
* 32 -71.36 -81.95 -72.65 -63.1 -58.85 -53.51 -47.39 -38.26 -11.76 -5.1 -11.76 -38.27 -47.37 -53.51 -59.19 -63.34
* 33 -74.11 -94.11 -70.87 -63.37 -58.68 -53.73 -47.33 -38.27 -11.76 -5.1 -11.76 -38.27 -47.43 -53.37 -59.24 -62.93
* 34 -70.96 -82.6 -69.68 -63.75 -59.07 -53.38 -47.44 -38.24 -11.76 -5.1 -11.76 -38.32 -47.44 -53.35 -58.79 -63.41
* 35 -71.89 -85.93 -72.01 -63.86 -57.82 -54.14 -47.27 -38.27 -11.76 -5.1 -11.76 -38.27 -47.32 -53.96 -58.2 -63.84
* 36 -70.89 -77.63 -72.83 -63.48 -58.75 -53.54 -47.3 -38.28 -11.76 -5.1 -11.76 -38.25 -47.53 -53.42 -58.61 -64.1
* 37 -70.56 -78.76 -70.99 -64.12 -58.52 -53.61 -47.35 -38.32 -11.76 -5.1 -11.76 -38.25 -47.4 -53.65 -58.58 -63.67
* 38 -71.17 -84.68 -70.95 -63.39 -59.12 -53.39 -47.38 -38.3 -11.76 -5.1 -11.76 -38.29 -47.39 -53.64 -58.44 -63.91
* 39 -70.82 -90.27 -71.25 -63.74 -58.93 -53.41 -47.34 -38.29 -11.76 -5.1 -11.76 -38.27 -47.41 -53.74 -58.55 -63.34
* 40 -71.25 -81.03 -68.95 -64.79 -58.25 -53.45 -47.44 -38.29 -11.76 -5.1 -11.76 -38.28 -47.49 -53.41 -59.02 -63.09
* 41 -70.11 -85.45 -70.4 -64.07 -58.08 -53.9 -47.27 -38.31 -11.76 -5.1 -11.76 -38.25 -47.54 -53.27 -58.89 -63.66
* 42 -70.36 -83.86 -71.14 -62.7 -59.09 -53.68 -47.38 -38.24 -11.76 -5.1 -11.76 -38.24 -47.66 -53.36 -58.4 -63.69
* 43 -68.27 -88.06 -70.02 -65.08 -58.2 -53.51 -47.39 -38.31 -11.76 -5.1 -11.76 -38.27 -47.41 -53.51 -58.78 -63.64
* 44 -70.82 -85.2 -69.41 -65.49 -58.44 -53.26 -47.5 -38.29 -11.76 -5.1 -11.76 -38.32 -47.46 -53.25 -59.3 -62.77
* 45 -70.29 -106.22 -72.45 -63.3 -58.6 -53.61 -47.2 -38.37 -11.76 -5.1 -11.76 -38.27 -47.49 -53.58 -58.52 -63.65
* 46 -71.22 -110.87 -69.6 -65.14 -57.93 -53.69 -47.51 -38.24 -11.76 -5.1 -11.76 -38.28 -47.49 -53.27 -58.87 -64.19
* 47 -70.58 -80.37 -70.18 -64.3 -57.99 -53.92 -47.33 -38.27 -11.76 -5.1 -11.76 -38.29 -47.47 -53.37 -58.87 -63.54
* 48 -70.43 -82.18 -71.11 -63.55 -58.93 -53.37 -47.47 -38.2 -11.75 -5.1 -11.76 -38.28 -47.46 -53.3 -59.26 -63.09
* 49 -71.1 -85.01 -68.31 -65.04 -58.54 -53.31 -47.54 -38.28 -11.76 -5.1 -11.76 -38.27 -47.44 -53.58 -58.51 -64.07
* 50 -72.09 -86.97 -71.33 -63.41 -58.85 -53.69 -47.32 -38.31 -11.76 -5.1 -11.76 -38.36 -47.28 -53.59 -58.42 -64
* 51 -68.98 -91.88 -69.48 -64.79 -58.31 -53.52 -47.55 -38.23 -11.76 -5.1 -11.76 -38.27 -47.37 -53.61 -58.69 -63.78
* 52 -71.25 -81.91 -72.12 -64.02 -58.13 -53.76 -47.36 -38.28 -11.76 -5.1 -11.76 -38.26 -47.45 -53.37 -58.99 -63.55
* 53 -69.78 -80.68 -70.14 -65.23 -58.26 -53.41 -47.57 -38.24 -11.76 -5.1 -11.76 -38.32 -47.43 -53.44 -58.6 -64.03
* 54 -71.24 -88.46 -69.99 -63.33 -59.14 -53.33 -47.42 -38.31 -11.76 -5.1 -11.76 -38.26 -47.43 -53.48 -58.79 -64.4
* 55 -69.84 -84.05 -71.12 -64.15 -58.3 -53.57 -47.44 -38.28 -11.76 -5.1 -11.76 -38.32 -47.23 -53.87 -58.31 -64.6
* 56 -70.93 -83.6 -70.96 -63.28 -58.88 -53.74 -47.31 -38.31 -11.76 -5.1 -11.76 -38.24 -47.39 -53.66 -58.49 -63.85
* 57 -70.18 -89.99 -69.73 -64.29 -58.37 -53.32 -47.59 -38.25 -11.76 -5.1 -11.76 -38.35 -47.34 -53.56 -58.57 -63.93
* 58 -11.76 -5.1 -11.76 -38.29 -47.41 -53.44 -59 -63.52 -70.19 -78.92 -70.53 -63.28 -58.79 -53.43 -47.48 -38.26
* 59 -11.76 -5.1 -11.76 -38.3 -47.34 -53.71 -58.16 -64.98 -68.92 -84.31 -68.9 -64.18 -58.9 -53.33 -47.48 -38.28
* 60 -11.76 -5.1 -11.76 -38.29 -47.46 -53.4 -58.78 -63.74 -70.59 -81.86 -70.03 -63.04 -59.03 -53.52 -47.33 -38.34
* 61 -11.76 -5.1 -11.76 -38.21 -47.62 -53.51 -58.24 -63.73 -71.01 -81.98 -70.74 -63.26 -59.02 -53.42 -47.42 -38.29
* 62 -11.76 -5.1 -11.76 -38.29 -47.42 -53.6 -58.49 -63.6 -69.16 -82.99 -69.22 -63.96 -58.99 -53.41 -47.39 -38.27
* 63 -11.75 -5.1 -11.76 -38.26 -47.38 -53.74 -58.44 -63.83 -70.27 -91.85 -70.98 -63.29 -59.19 -53.42 -47.35 -38.29
* 64 -11.76 -5.1 -11.76 -38.25 -47.59 -53.41 -58.54 -63.41 -72.52 -84.53 -69.56 -64.78 -58.36 -53.64 -47.34 -38.31
* 65 -11.76 -5.1 -11.76 -38.33 -47.29 -53.65 -58.47 -64.18 -69.47 -93.09 -71.47 -63.63 -58.63 -53.4 -47.45 -38.3
* 66 -11.76 -5.1 -11.76 -38.29 -47.39 -53.45 -59.14 -64.04 -68.4 -83.93 -71.99 -63.41 -58.69 -53.67 -47.41 -38.25
* 67 -11.76 -5.1 -11.76 -38.27 -47.5 -53.25 -58.74 -63.77 -70.88 -92.66 -69.93 -64.33 -58.57 -53.67 -47.28 -38.35
* 68 -11.76 -5.1 -11.76 -38.26 -47.53 -53.39 -58.58 -63.95 -70.47 -93.27 -69.94 -64.31 -58.55 -53.58 -47.39 -38.26
* 69 -11.76 -5.1 -11.76 -38.27 -47.39 -53.52 -58.47 -64.14 -70.89 -93.59 -69.54 -64.23 -58.59 -53.37 -47.39 -38.31
* 70 -11.76 -5.1 -11.76 -38.26 -47.36 -53.54 -58.89 -63.65 -70.26 -89.33 -70.57 -64.85 -57.88 -53.75 -47.34 -38.31
* 71 -11.76 -5.1 -11.76 -38.25 -47.32 -53.68 -58.42 -63.87 -71.2 -91.47 -69.77 -63.97 -58.85 -53.51 -47.42 -38.29
* 72 -11.76 -5.1 -11.76 -38.3 -47.39 -53.62 -58.52 -63.42 -72.18 -85.69 -69.28 -64.89 -58.19 -53.78 -47.38 -38.28
* 73 -11.76 -5.1 -11.76 -38.24 -47.58 -53.3 -59.06 -63.18 -70.12 -85.19 -72.1 -63.06 -58.83 -53.85 -47.11 -38.34
* 74 -11.76 -5.1 -11.75 -38.27 -47.23 -54.12 -58.14 -63.24 -71.6 -88.24 -70.61 -63.09 -58.93 -53.5 -47.5 -38.24
* 75 -11.76 -5.1 -11.76 -38.27 -47.51 -53.21 -58.94 -63.41 -73.53 -90.38 -69.91 -64.27 -58.45 -53.75 -47.32 -38.3

Uploaded files:

Ok, processing the data in python does not show the behavior you see, so the data seems fine (number of samples varies between packets, but that's to be expected).

In your last example you have "offset=382" which seems odd, as from what I get that should be the size of the JSON header, which for me is around 232 bytes (+-2). So apparently your buffer doesn't start on a packet boundary but with data from an older packet. Not sure how Boost buffers work, so can't say if this is correct or not.

Also with 452 samples and 16 bins you should should have way more than 4449 bytes to process. That number also doesn't line up with your 75 samples output (75*16*2 = 2400). So there is some issue with how you assemble / split the data into pieces.

What I suggest is that you actually download a stream (e.g. using the browser and the "limit" parameter to stop it automatically) to a file on disk and test your parser on that to see if it works on static data (ideally without using async IO first to reduce complexity). If the parser works there, but not with a live stream, then the problem is likely in your http or buffer handling. Which woudl be more related to your usage of Boost, which we can't really help with. I kinda suspect some kind of race condition so your buffer is partially overwritten while you're processing it (as your "wrong" samples appear to simply be shifted), but that's just speculation.

Ok, so if the second RTSA works my guess would be to take a look at this example here: https://github.com/hb9fxq/libspectranstream/blob/master/lib/spectranstream.cpp , maybe it gives you some hints on the record seperator / chunked transfer.

 

Thank you both. Yes, indeed the buffer was too short to contain all samples. I was trying to read as long as data was available in the buffer. However, I got meaningless data before I reached that end. I am trying now with data on disk, as recommended. Coming back ...

 

fw_dev has reacted to this post.
fw_dev