This is the work around that worked for me.
Create a new VM
Choose > Custome (advanced)
Choose > I will install the operating system later
For the Operating System Choose > Linux
Version> Red Hat Linux Enterprise Version 4 64-Bit
Keep going thru the setup for your VM and when you get to the end Click on the Customize Hardware button
Click the the Processors and choose Preferred mode: Intel VT-x or AND-V (leave the other two options uncheked)
Click on Display and uncheck 3D graphics if checked
Click on the CD Rom and mount the ESX iso
Remove any un-needed hardware
At this point click finish and make sure you dont turn on the VM.
Close vmworkstation (make sure vmworkstation is not running before going to the next step)
Go edit the VMX file of your new VM and add the line below and save the file.
monitor_control.restrict_backdoor="true"
Run vmwaorstation and install esx!
This worked for me and I'm able to run 64 Bit OS's inside my ESX VM on vmworkstation 8.
I hope this helps someone...