Audacity- Need a math geek

[ConradH]

No pay, just fame and your name in genuine print, plus thanks from a lot of Audacity users. I want a plug-in that does Savitzky-Golay smoothing. This is a noise filter that doesn't shift or distort the underlying waveform as much as standard LP filters. There are lots of references for it, and even some code, on-line, but it's beyond my limited abilities and I don't know if the nyquist plug-in language lends itself to that kind of filter.

The Wave Repair program (http://www.delback.co.uk/wavrep/), that's actually very good, has a smoothing function, and it would be great if a similar thing can be done for Audacity. The purpose is smoothing sections of a waveform with high frequency noise, something that comes up with LP rips quite often. The various standard filters work very poorly for this.

If the S-G filter is impractical, a moving average or mean filter would probably be close to as good. Any math geeks here?

 

[guiller]

GramoFile is a GPL (free) software that may be used under Linux and perhaps under Windows and Mac OSs. There is a discussion about the nine provided "filters" and how to add new ones in here: http://www.opensourcepartners.nl/~costar/gramofile/Signproc.txt . Perhaps some of the discussion may be useful for you.

 

[ConradH]

Interesting, thanks- I hadn't seen it before and looked at the source code, but I'm not much on C and am not sure how to accomplish the same thing in Nyquist, the typical language for Audacity plug-ins. I'm just hoping somebody here has written some plug-ins and can at least tell me how to do the core function of S-G.

 

[lilsatyre]

I guess I fall into math geek territory (economics and statistics BA's), so here's my input. I'm never used / coded for nyquist / audacity, but nyqist does have the commands you would need to carry out the filtering from taking a peak in the reference manual, as it has a linear regression module and general matrix commands. The ease (or not) of programming it would mostly come from what kind of control you would like to give the end user in the smoothing fit (ie. selecting polynomial order, and the smoothing window), and what kinds of signals you are intending to process.

It appears that a major advantage of the s-g filter is that the polynomial coefficients used to generate the smoothed data are not related to the data values themselves. There are a couple of resources that tabulate those values that I found. The wiki article actually does a fairly good job of describing the generation of the smoothed values using matrix notation, although it does get screwy in failing to highlight the (subjective) simplicity of how it is done. If you are familiar with matrix operations / notation and know basic nyquist syntax it would be straightforward to program. PM me if you want some feedback or post here and see if we can't generate some more input, I can at the very least walk you through how the filter does its thing in very clear steps.

 

[ConradH]

Greatly appreciate the help! This is the second time I've had a need (or thought I did) for an S-G filter, so I should really figure it out. I'm mostly a cookbook programmer, so if I can get it broken down into "do this", "do that", "step through these", "rinse and repeat" terms, I can probably deal with it. The explanation in the Numerical Recipes for Fortran77 is good and I can sort of follow the code (never did Fortran), but the C code version is Greek to me. Alas, I also have to brush up on matrix stuff. Math has always been difficult for me but I can slog through most anything, given enough time. For anybody that's interested in why I want this, download a demo copy of Wave Repair and try the smoothing filter (they hide it under the edit dropdown) on some high frequency noise. Use it multiple times if necessary. Then try various filters in Audacity. I don't know the method Wave Repair uses, maybe median, maybe S-G, but it's way better than any lowpass filter I've found, and the author specifically says it isn't implemented as that type of filter. I'd just use Wave Repair, but I'm Audacity-centric and am more comfortable with the program.

 

[dodog]

Conrad-

Not sure if you have, or have access to, matlab, but it has S-G stuff built in (in the Signal Processing toolbox (if you have matlab, it would be under ...matlabpath/toolbox/signal/signal/sgolay.m). Take a look at these links:

https://www.mathworks.com/matlabcen...ky-golay-smoothing-and-differentiation-filter

https://www.mathworks.com/help/signal/ref/sgolay.html

I took a quick peek at the .m file for sgolay and there's not much to it. If you can get access to the code, you could probably borrow and create your own version for Audacity.

 

[audiotemp]

Conrad, the Nyquist language is a Lisp derivative, and if C is "Greek" then Lisp is probably Martian. However, Audacity can use a number of types of standard audio plug-ins as well as Nyquist ones. See here for the various types, http://www.audacityteam.org/download/plug-ins/ although I can't tell you what languages (Fortran?) can be used with what types.

Also, do you have a reference for the usefulness of S-G filters to audio applications? It's not obvious to me what the merits of it are. Unlike LP filtering, S-G (apparently, I didn't do any math...) is a non-causal filter but there are many types of digital filters that are non-causal. S-G's marquee feature seems to be that it doesn't shift features (peaks, etc.) in time but I don''t see how that is necessarily a useful feature here. Is it just a flat phase filter?

 

[ConradH]

My assumption, as a mathphobe, is that not shifting the basic waveform shape will make corrections less audible. Applying a window like is used with FFT could also help with that as it would reduce the effect near the ends. I do know that the standard LP implementations give a fairly painful shift that would take me extra time to fix. I'm trying to restore a very noisy recording and a painless abrasion reducer would really help. This is something where I'd expect a typical correction would be maybe 200 mS or less. Not specific to my problem, but here's a reference- http://research.ijcaonline.org/volume57/number21/pxc3883876.pdf

 

[lilsatyre]

I'm guessing the peak timing would be important for music where a shift could be audible and result in "off" timing, although I have no idea what differences would be audible to a normal person. The filter does not effect the start / end data points I believe, but that can be dealt with in a couple of different (easy) ways according to wikipedia. From what I can tell, nyquist can take a sample and turn it into an array (not sure what the array elements would look like), with which you can likely carry out the filter. I can draft up some pseudo-code with comments this weekend to detail what the filter code would look like, and maybe that can get the ball rolling on an implementation in nyquist. I'm guessing you would want to play with a couple of different filter window sizes and polynomial orders to see what sounds best. If the goal is to get a working prototype up and running a fixed window length and polynomial order would be fastest (the coefficients for the filter equation can be calculated outside of nyquist and inserted into whatever code.)

Audiotemp does raise the interesting observation about add-ons and different languages. Is there a place to ask audacity programmers what the best language to implement this for Audacity would be?

 

[sKiZo]

Something like this? I use it all the time for vinyl, and it was indispensable for cooling down 78 rips. There's a couple variations built into the "classic" filter set along with configurable Butterworth filters. Not sure if that ships with Audacity or is an addon ...

https://en.wikipedia.org/wiki/Chebyshev_filter

Oh. Here's a writeup on the Audacity wiki for nyquist that might help or point you in the right direction ...

http://wiki.audacityteam.org/wiki/Nyquist_Audio_Programming

Ah. Also ... Audacity's chain feature might be handy if you want to take it a step at a time. Build and verify sections of code, then chain them together so they process as one. Should help debugging as sections fail before you build a big bouncy filter. Leaving the code as small segments might also be nice for tweaking the individual parameters during use as well.

 

[audiotemp]

My assumption, as a mathphobe, is that not shifting the basic waveform shape will make corrections less audible. Applying a window like is used with FFT could also help with that as it would reduce the effect near the ends. I do know that the standard LP implementations give a fairly painful shift that would take me extra time to fix. I'm trying to restore a very noisy recording and a painless abrasion reducer would really help. This is something where I'd expect a typical correction would be maybe 200 mS or less. Not specific to my problem, but here's a reference- http://research.ijcaonline.org/volume57/number21/pxc3883876.pdf

OK, I hadn't understood that you wanted to process in small snippets. In that case, not changing the endpoint values and not shifting/moving things around would both seem to be important.

I won't be able to look at the paper until later.

 

[loopstick]

It appears that a major advantage of the s-g filter is that the polynomial coefficients used to generate the smoothed data are not related to the data values themselves.

I'm not familiar with this filter but the Wiki article suggests to me at least that the filtering is done by using a low-order polynomial to "least-squares" approximate the waveform. IOW it's too "low - order" to fit rapid / random-ish changes. I'm guessing it's not linear - kinda like a simple median filter used to remove speckle noise in images - but it is frequency limited I imagine by the order of the polynomial.

 

[ConradH]

I'm going to have to take this in steps. First I'll get the filter working in the one language I know and love, Bob Zale's PowerBASIC. Most of the utilities on my site are written with that. Then, once I understand the best ways to code it, I'll give the Nyquist a shot. There are some other Audacity filters that get done as dlls, and generating dlls in PB is trivially easy, so that might be another avenue. This won't be fast, but I'll check in if I make any progress.

 

[JoeESP9]

 

[audiotemp]

Page 4 (Figure 14.8.1) here gives a comparison of S-G to moving-average filtering. There's some Greek in there, too. Copyrighted, with permission granted to make one copy for personal use.

Also gonna get me some for this one.

 

[ConradH]

Thanks, I've got that one. The S-G just seems like it should be better, but the proof will be in the listening. I'm a bit worried because the code is starting to make sense.

I need a butter emoji.

 

[ConradH]

Progress report- I haven't programmed anything in a few years and can't believe how much I've forgotten, assuming I ever knew anything to begin with. I've got a little utility going where I can load and display wav data, and then operate on it. So far I'm just finding the best ways to do plain averaging filters, but will tackle the S-G soon. I've a ways to go before I understand things well enough to transfer the ideas to an Audacity filter, but I think it will be worth it.

I don't do VB, but there's a wonderful bit of work here that's helped me immensely- https://whenimbored.xfx.net/2011/01/fast-fourier-transform-written-in-vb/

 

[lilsatyre]

are you using nyquist for the visualization utility? if so, can you use the function to transform sound to an array type file? depending on what that looks like, I'd be happy to hop on the phone / skype and help you get the steps figured out for writing the s-g filter.

 

[ghazzer]

. . can't believe how much I've forgotten, assuming I ever knew anything to begin with.

I read you '5 X 5'! Those older neurons and synapses don't seem to fire as well as they used to, until they have been substantially refreshed. Old neurons never die, they are just hard to find.

Keep up the good work - - -

 

[ConradH]

Ha, at this point I just have to slog through the stuff myself. It's the only way I learn. I've already got some working FFT routines in other programs, but it's the SG I really want, and I really want it as an Audacity plug-in. That page I linked shows how to code it quite well in VB. What I need to determine is how far to take it- do I need to calculate coefficients for various types, or is one or two hard-coded sets enough for audio use? That demo uses a lot of helper functions I'd rather not bother with. I also need to understand matrix stuff and how it might help, or not. I just realized that my present averaging filter needs to process in 32 bit, as trying to do everything in 16 bit with 16 bit data is going to overflow. Oddly, I used to work with image processing software that did FFTs, processed the result and then converted back to an image, so I'm pretty familiar with what it can do. Wasn't involved with the code though.