I wouldn't call it a ranking signal. Chances are good removing such links would have an effect on rankings by altering other signals, but it's not an effect you can guarantee. A site that hasn't been penalized (manual action) or downgraded (algorithmically evaluated, resulting in some action - technically, the algorithm is looking at the sites linking to yours but I'm trying to be concise and flailing) might actually harm itself by removing backlinks that were helping.
☰