This is how yu can set up custom/private nameservers in WHM.
1. Log in to WHM Account.
2. Go to Server Configuration >> Basic WebHost Manager Setup or type Basic in the menu filter box.
The option will appear. Click on it.
3. Under Nameservers, select Explicitly Set the Nameservers.
Enter or change the nameservers. Then click on Save Changes.
Keep in mind: Domain propagation can take up to 24 hours.