Erroll Dietz is a remarkable fellow. He started out at National Semiconductor as Bob Pease's
technician, and rose to become Chief Technology Officer.
Feedback amplifiers generally have an output impedance that rises linearly with frequency---in other words, it's
inductive. As Dietz's short paper from Electronic Design shows, this effective inductance can
resonate with the output bypass capacitor to cause really nasty noise peaks in the 1-100 kHz region. If you have an inexplicable noise peak in that range, a small resistor (a few tenths of an ohm to an ohm or two) in series with the regulator output can be just the ticket. You can also put it in series with the cap, but if you do that, make sure it's a pulse-withstanding type, or it's liable to blow up by an output short-circuit, or even the inrush transient, e.g. if somebody runs your gizmo off batteries.
Modern high-value multilayer ceramic caps (MLCCs) and aluminum-polymer electrolytics can have very low effective series resistance (ESR). This makes them amazing for filtering, but when used with voltage regulators, there's a catch: they make these noise peaks much worse, and frequently cause instability: the regulator's feedback loop goes into oscillation. This is really bad news; not only does your gizmo not work right, but the peaks of the oscillation are often way out of regulation, and so may destroy parts of the circuit. Low dropout (LDO) regulators, negative linear regulators, and most switchers are especially vulnerable to this.
Parallelling the Al-poly or MLCC filter with a tantalum or aluminum electrolytic of about the same value will often fix the problem, or you can use the pulse-withstanding resistor trick.