Peter Singer and a reductio argument against morality?

Yesterday, I was talking to someone who thinks it is morally permissible to consciously let a nearby child drown, for no other reason than preference (say, "I would prefer not to."). The reason given was that inaction is a morally neutral category. The argument for why inaction is a morally neutral category went as follows: If you can be blamed for inaction, then you can be blamed for going to the movies instead of donating to charity. But clearly it is not condemnatory for one to simply not give to charity. So inaction cannot be condemnatory. So it's okay to let a child drown.

I think many people have an intuition that something is wrong with Peter Singer's argument that we are acting wrongly by not directing large amounts of our assets to charity.

What do you think typically grounds this intuition, if anything? Is it that we cannot be blamed for inaction? Or is it something else?

My own view is that insofar as we have a set of good and bad choices at any moment, we are at least obligated to choose one of the good ones. And the virtuous person tries to choose as close to the best option as is reasonable, given resources, life plans, etc. I don't feel like fleshing out my whole view, which ends up being close to Singer's. But I am just wondering if any of the non-participating authors on this blog, or someone from the infinitely large set of our readership, has a view on the original position above. What is the moral status of inaction? If inaction is not a morally relevant category, is there anything at our disposal that shows letting a child drown to be wrong?

A word about the title of this post: It seems to me that Singer-like arguments are inescapable on deeply plausible assumptions regarding morality. What struck me about the person who thinks it's okay to let children drown is that the argument went in the other direction. Singer's conclusions were taken to be absurd, and so deeply plausible moral principles were abandoned!