**A faint stain can stay; actual glue residue needs to be removed or sealed before you skim coat.** You don’t need to scrape until the drywall looks spotless, especially if that starts damaging its paper facing.

Check a stubborn spot with a damp sponge: if it becomes slippery, sticky, or gummy, there’s still adhesive there. Moisture from the joint compound can soften that adhesive and compromise the skim coat’s bond.

For your situation:

1. Remove the remaining loose glue and any raised deposits.
2. Wipe with clean water to remove the vinegar and dissolved paste, then let the wall dry thoroughly.
3. Apply a sealer specifically rated for wallpaper adhesive residue, such as **Zinsser Gardz**, and let it dry according to the label before skim coating. The manufacturer explicitly states that it seals old wallpaper paste and helps prevent bubbling when damaged drywall is coated with joint compound. [Manufacturer guidance](https://rustoleumsupport.zendesk.com/hc/en-us/articles/4424094153101-Zinsser-Gardz-Problem-Surface-Sealer)

With only faint, firmly attached traces remaining, sealing first is a reasonable way to proceed.