To convert dmg file to iso, such as osx InstallESD.dmg to iso so you can install it on vmware, then run the following command in mac terminal:
hdiutil convert InstallESD.dmg -format UDTO -o osx.iso
This will create a file called, osx.iso.cdr. Then run the following command to rename it to .iso file:
mv osx.iso.cdr osx.iso