HE v2 "sampling" mode brainstorm

Is a glitchy "sampling" mode on the new HE better than no "sampling" mode at all? The intent is to have a mode that records sound input and turns it into an envilope. Here's how I'm doing it so far:

20 sub-samples are taken from the input at 5ms intervals, then averaged. the average becomes the main "sample" that lasts 100ms. this occurs 10 times a second.

so it samples at a total of 200hz. the problem is when I hit a note like low E (82-ish hz), or any note really, that the played note will get pretty close in-phase with my sampling frequency from time to time as it "drifts". because of that, for an entire 100-200 milliseconds, it'll only be sampling the "dips" in-between the sound waves and will average close to 0, even though to our ears the note rings loud and clear. as clear as it was when it was sampling at the peaks of the same waves 300ms earlier! the result is a stuttery glitchy envelope instead of smooth envelope following.

adding circuitry is a last resort (there's barely any room left on the PCB). so far I've tried:

1. adding offsets of 2.5ms every 2 samples to try and average an equal number of wave dips and peaks.

2. only keeping the very highest sample out of 20

3. ignoring sudden drops of envelope if the next one goes right back up again (loss of detail)

Increasing the sampling frequency is difficult at the moment since the whole foundation of the bare-bones design is built around 20 sub-samples per "real" sample, because of the interpolation I'm doing. I think I'll try adding a timer interrupt that samples much more/faster and independantly of the rest of the program.