What is DEP/NX? DEP is the abbrevation for Data Execution Prevention aka No execute. It is a feature of processors that allows to mark memory segments as data or code. Memory segments marked as data will cause an exception when trying to execute code there. The operating system will manage this automatically for code loaded from executeable files. If an application dynamically creates executable code (JIT) it must tell the operating system which segments contain code and which contain data.
Application programmers for Windows should read this document from Microsoft: http://msdn.microsoft.com/en-us/library/aa366553(VS.85).aspx
How does this affect Visual Smalltalk?
Visual Smalltalk byte codes are dynamically translated into machine code, which is then executed. When using Callbacks the VM also creates a tailored code thunks, which are executed when the callback occurs. If running on a DEP enabled hardware and operating system, the VM will cause a protection violation when trying to executed the dynamically translated machine code, because the memory segments where not allocated with the flags allowing to execute code.
How can I resolve this issue?
- disable DEP in the operating system (not recommended) : http://technet.microsoft.com/en-us/library/cc700810.aspx
- patch the virtual machine and fix callbacks: VMPatchNX
- Contact Cincom in order to get the new VM that solves this problem.