Yes, but unless you can get the condom to completely cover the ivory phallus, I think you're still asking for trouble. A condom is designed to stop seminal fluid from breaking through or around it. Condoms aren't engineered to control possible infection around the base of the penis, which is going to be an issue with an ivory phallus.

Personally, while I find the term distasteful and fads are lame, I might consider a "steampunk" alternative, using newer materials and an older visual appeal.