In a world that is recognizably not our 18th or early 19th century, the teller of this tale is a man with an interesting talent. He can take memories from your mind so neatly that you'll never know the stolen memory is gone. He's made a nice business of this. Some people will always have memories that are too painful and they wish to be free of. Others will have memories that it would make their lives easier to have someone else forget. Unfortunately, once he's removed a memory from your mind, he has it in his--forever.
And some of the people he does business with are dangerous men, who might want to eliminate the last "last witness." Eventually, he outwits the gambling compulsion he picked up somewhere along the line, and manages to retire in relative comfort, with little need to associate with other people at all.
That's when he discovers there's someone else in the world who has his talent, and e'sven fewer ethical standards. It's someone to whom he has a connection that's painfully close.