Topological superfluid ³He, with unconventional spin-triplet p-wave pairing, provides a model system for topological superconductors, which have attracted significant interest through potential applications in topologically protected quantum computing. In topological insulators and quantum Hall systems, the surface/edge states, arising from bulk-surface correspondence and the momentum space topology of the band structure, are robust. Here we demonstrate that in topological superconductors the surface Andreev bound states, which depend on the momentum space topology of the emergent order parameter, are fragile with respect to the details of surface scattering. We confine superfluid 3He within a cavity of height comparable to the Cooper pair diameter. We precisely determine the superfluid transition temperature T_c and the suppression of the superfluid energy gap, for different scattering conditions tuned in situ, and compare to the predictions of quasi-classical theory. We discover that surface magnetic scattering leads to unexpectedly large suppression of T_c, corresponding to an increased density of low energy bound states.