The authorities seem divided on this subject, some texts even appear to contradict themselves. After reading many sets of rules and prohibitions, it is my impression that the beads must be treated with care and respect, avoid touching with index finger, don't allow to touch the ground or unclean surface. Otherwise go by what you feel is right. For example I wear my panchmukti beads at all times during the day, leaving them on the altar while I sleep. As long as you hold your rudraksha sacred you should have no fear wearing them.