Is it ethical to fail to report rule violations by a co-worker? Why?

I don't see the problem. Unless those rule violations would result in harm to another person, mind your business. Nobody likes a rat. Start keeping a list, copy the paper trail, and once you have overwhelming evidence, then you can use it. Without that, it will backfire, and they'll look at you as the problem.