Spiking and Despiking

I have used ADVs for measuring velocities in a laboratory model harbour under a tidal flow. Recently I came up to an article by Goring and Nikora (2002), which describes an efficient way to detect the spikes from ADVs. Wahl (2002), has updated the WinADV according their work with some modification. In the article by Goring and Nikora has been mentioned that if the data were undergoing a sustained change in velocity (e.g.tidal cycle), then the suggested method described in the paper would not work. I wonder if anyone has experienced spikes in a tidal measurment and whether has come to a sloution, or any comments or suggestions on this subject? Thanks " />'>

Hi Kourosh-

I am not sure the details of Goring and Nikora's despiking method, but you can eliminate tides or other varying currents by using an algorithm that works on the first difference. First differences are particularly good at amplifying spikes, which of course makes it easier to find them.

If you use Matlab, the attached Matlab file DESPIKE.M gives you an example. It seems to work reasonably well, finding and interpolating spikes.

-Lee Gordon

Hi

I have checked in ExploreV (written by Alex Sukhodolov) and he has implemented four forms of despiking in his program:

- Reject data point based on weak amplitude
- Reject data point based on low correlation
- Reject data when dv/dt is larger than a user selectable value
- Reject data for |v-vmean| larger than n*standard deviation where n is user selectable

I am not quite sure how this relates to the criteria you mention but the four criteria seem to cover most situations. " />'>

Thanks Lee

I used your filter in MATLAB environment, and it detects and eliminates spikes according the Lim number introduced to the programme. It obviously is dependent to Lim number you choose, and certainly experinece would have an essential role to choose the right value. Can you share your experience with me please?

Regards
Kourosh

Hi Kourosh-

When I use the despiking m-file, I use lim in the range of 3-6. I plot the data both before and after (often plotting after in a different color, directly on top of before), and then choose a value for lim that removes the spikes without messing up the rest of the data too much.

-Lee

p.s. I have used this filter to look at a lot of velocimeter data, both from Nortek and a competitor. I am learning that the Nortek velocimeters have quite a bit less spiking, especially the Vector. I believe this is because Nortek has implemented better signal processing algorithms.

Hi,

First of all I would like cite a couple of sentences from a conclusion by Goring and Nikora (2002): "...Spike replacement is an arbitrary procedure. There are many different strategies available and none of them has more validity than any other...". I just suppose that despite the findings of Goring and Nikora the same is still true for spikes detection.

For my opinion the problem of despiking should be considered in the layout of the whole data measurement and treatment program. Prior to measurements one should arrange appropriate use of the device: in different environments there may be different sources for spikes and hence different strategies to treat the data. It is helpful prior to measurements to simulate spikes appearance and study it by using for example two devices sampling close (or even the same point of flow simultaneously) but with artificial and controlable spikes production in one device (due to over-range or other factors). My personal experience with ADVs show that if a reasonable care is taken upon measurements the number of spikes in the series would be negligibly low (see example in Sukhodolov and Rhoads, 2001) and the very simple algorithms like those implemented in ExploreV treat data quite good. A great shortcoming of Goring and Nikora (2002) work is that they missed the comparison for methods for detecting bad samples with SNR and correlation thresholds which seems to be especially important in laboratory studies. As Atle Lohrmann has pointed in his reply: Nortek has managed precise measurements for for velocities and quality scores simultaneously, so that empirical thersholds for SNR and correlations are provided by manufacturer and documented in user's manuals. Spikes replacement should be also considered with further data processing in mind. There are methods of the spectral analysis that allows treatment of unevenly spaced data or data with gaps (see Press et al., 1997), so replacement in many cases may be not necessary at all.

Literature:

Goring D., and Nikora V., 2002. Despiking Acoustic Doppler Velocimeter Data. Journal of Hydraulic Eng., 128(1), 117-126.
Press W.H., Teukolsky S.A., Vettering W.T., and Flannery B.P., 1997. Numerical Recepies in C. The Art of Scientific Computing, Second Edition, Cambridge University Press.
Sukhodolov A.N., and Rhoads B.L., 2001, Field investigation of three-dimensional flow structure at stream confluences. Part II Turbulence, Water resources Research, 37(9), 2411-2424.

Previously Kourosh wrote:

Hi Kourosh-

I am not sure the details of Goring and Nikora's despiking method, but you can eliminate tides or other varying currents by using an algorithm that works on the first difference. First differences are particularly good at amplifying spikes, which of course makes it easier to find them.

If you use Matlab, the attached Matlab file DESPIKE.M gives you an example. It seems to work reasonably well, finding and interpolating spikes.

-Lee Gordon

Hi Lee Gordon

 

I see you have Despike.M to process the ADV data, recently I am trying to despike some data,  can you send it to me to help me?   I am from China,  this is my e-mail address here: antelope_maria@yeah.net 

Hi Katherine,

We unfortunately don't have access to Lee's despite.m file anymore. There are several other threads on the forums which deal with removing spikes and at least one Matlab toolbox posted to The Mathworks code exchange site.

 

P.J.

Previously Kourosh wrote:

... I wonder if anyone has experienced spikes in a tidal measurment and whether has come to a sloution, or any comments or suggestions on this subject? Thanks " />'>

 

I have used ADVs in both gradually-varied flows like tidal flow and rapidly-varied flows likes bores and tidal bores. In a tidal flow, the basic despiking techniques may be used suitably, ncl. the phase-space desking method developed by Goring and Nikora (2002) and discussed by wahl (2003) [1]. On the hand, the despiking techniques are totaly unsuitable to very rapidly varied flows like tidal bores [2].

Hope that the comment may help.

[1] CHANSON, H., TREVETHAN, M., and AOKI, S. (2008). "Acoustic Doppler Velocimetry (ADV) in Small Estuary : Field Experience and Signal Post-Processing." Flow Measurement and Instrumentation, Vol. 19, No. 5, pp. 307-313 (DOI: 10.1016/j.flowmeasinst.2008.03.003) {http://espace.library.uq.edu.au/view/UQ:152793}
[2] KOCH, C., and CHANSON, H. (2009). "Turbulence Measurements in Positive Surges and Bores." Journal of Hydraulic Research, IAHR, Vol. 47, No. 1, pp. 29-40 (DOI: 10.3826/jhr.2009.2954) {http://espace.library.uq.edu.au/view/UQ:164015}