Riegl DR + waveform, aquired over Hyytiälä in 15.11.2011 Processing the data into hectare -files and error-checking. AH, 13.4.2012 ****** *Data* ****** Original data from Finnmap contains: -LAS1.1_wrong_FL_id: LAS1.1 -files by trajectory (DR data, NOTE: flight line id differs from those in the .trj-files) -LAS1.3: LAS1.3 -files by trajectory (DR, waveforms in external .wdp-files) -RawLaser: Riegl raw waveform data (.sdf) -waveView: Riegel waveform viewer -riwavelib_sdf-1.3.51: Riegl dll-library for accessing the raw waveform data -trj_256Hz_tm35fin: trajectory data (.trj) -trj_ascii_256Hz: trajectory data (ascii) -images: aerial images during the aquisition Processed hectare-files are in the directory "Processed_hectares/" ************************** *Processing into hectares* ************************** Data was delivered BOTH in LAS1.3 -format (with external wdp-files for waveform data) AND original Riegl waveform format (.sdf -files). Processing into hectares was made using LAS -files, which were generated from the original raw waveform data using Riegl's RiProcess software. If more control on processing steps or access to the system waveform are needed, the sdf-files can be accessed through Riegl "wavelib" -dll. ********************** *Hectare -file format* ********************** Echoes belonging to the same pulse were first searched and stored in a pulse format described below. Data was divided into hectares according to the coordinates of the last return in a pulse. For each hectare, there is one file (###_###.bin) for discrete return data. Each echo in a pulse contains a pointer to waveform data (###_###_wf.bin). Pointer indexing starts at 0. Structure of ###_###.bin: In the beginning of the file, there is the number of pulses (signed int32, bytes 0-4). The pulse data starts at byte nr 4. Each pulse record contains general information on the pulse (gps-time, trajectory etc.), and variable number of echoes. Pulse echoCount (signed 8-bit integer) gpsTime (64-bit float) byteLAS (signed 8-bit integer, NOTE: has errors in the case of more-than-7-return-pulses) echoes[] (36 bytes each, from last to first) scanAngle (signed 8-bit integer) poslidar.x (64-bit float) poslidar.y (64-bit float) poslidar.z (64-bit float) stripNum (trajectory number, signed 8-bit integer (not necessarily same as in original files)) Echo pos.x (64-bit float) pos.y (64-bit float) pos.z (64-bit float) intensity (unsigned 16-bit integer) pointer (pointer to waveform data, signed 32-bit integer) waveformSampleSize (number of waveform samples, signed 16-bit integer) retPointWFLoc (distance of the DR echo from start of the waveform recording in pikoseconds, 32-bit float) Structure of ###_###_wf.bin: No headers, waveform data starts from the beginning of the file. Waveforms are stored using unsigned 16-bit integers. Sampling frecuency was 1 ns. ************************************ *Additional notes on the processing* ************************************ -Riegl scanner stores waveform data into blocks. When the data is exported to LAS1.3 -format, each block is written into .wdp -file as separate waveform packet. The length of the packet is 80 samples. If the original Riegl block is larger, it is split into separate waveform packets. One waveform packet can contain many peaks. So several echoes in a pulse can point to the same waveform packet. -The retPointWFLoc is not accurate. It is rounded into the nearest nanosecond in RiProcess software. The more accurate value, if needed, can probably calculated using original waveform data in the .sdf -files. -There is a problem considering return numbering when exporting to LAS1.3 using RiProcess: The return number and number of returns in a pulse are coded using 3-bit integers. If the pulse contains more than 7 returns, the return number indexing starts over at 8th return ie. the 8th return has index 0, 9th index 1 etc. Also the "number of returns" -flag is erroneous if pulse contains more than 7 returns. These pulses were corrected: the echoes were sorted by z-coordinate and relabeled. The number of pulses containing more than 7 echoes is extremely low. For example, in the first trajectory there were 23812840 pulses, of which 162 had more than 7 returns. -There were small number of echoes that were missing the pointer to a waveform packet. The reason for this was a bug in Riegl software. If the echo is located exactly in between two LAS -waveform packets ie. its distance from the start of waveform recording rounds to 80 ns, it is left without pointer when exported to LAS1.3. If there is no subsequent echo after the one with missing pointer, the latter waveform packet does not exist in the LAS -file. -In addition, there were larger areas with missing pointers at the end of 4 of the flight lines. The map showing these areas can be found in "Processed_hectares/" -directory (err_wfpacket_pointer.tif), red means > 0.05 erroneous pulses per m2. There is also an excel file, in which the spatial distribution of the erroneous pulses is analysed (WavePacketSize_analysis.xlsx). -log file (Loki.txt) and the Java code (NetBeans project) for hectare-processing (Processed_hectares/HectareProcessing2011Riegl) are also in the "Processed_hectares/" -directory *************************** *Assessment of the quality* *************************** The data (DR and WF) was superimposed on aerial images on buildings and other clearly defined structures. The geometry seems to be ok, and waveforms are correctly associated with the discrete return echoes, except for the small amount that have the missing pointer. Intensity information differs somewhat from those in summer conditions, but appears to be logical, considering the conditions. Examples of the state of surfaces (grass, asphalt, buildings) at the aquisition can be seen in photographs: (Processed_hectares/IMG_0991.zip). ************************************ *Corrections to the data, 21.9.2012* ************************************ The bug in RiProcess was corrected (see "RiPROCESS-1.5.7-changes.pdf"), and Finnmap re-processed the data into LAS1.3 format. New LAS-files are in directory "LAS1.3_201209/". The new files were processed into hectares. Same metadata and file format apply as before, but handling of waveform data packets was changed a little. Now, RiProcess connects sample blocks, if there is an echo between them. The LAS data can therefore contain waveform packets with 80, 160 or 240 samples. There were still small areas at the end/beginning of the flight lines that contained echoes missing pointer to WF data. The reason for this is unknown. These can be excluded from the data by "waveformSampleSize" which is always 0 for these echoes.