FEX

Read the blog post at FEX-Emu's Site!

We were a bit distracted this month with the X.Org Developer's Conference happening but we still managed to get some work done. Let's jump in and see what has changed.

JIT changes

This month brought some improvements to the JIT although less than typical. We added support for AVX-VNNI which is starting to get used by some game libraries. We also optimized the SSE4.2 string operations so these are quite a bit faster. 3DNow! has been enabled again since we fixed the last known bug with it. We also improved emulation on the x87 FXAM instruction since apparently Mono uses it to detect denormals for some reason. Additionally a peephole optimization to improve a common case of an unmasked AVX gather instruction which some sound libraries rely on.

We also now have a software fallback for RDRAND and RDSEED instructions. This isn't enabled by default as the performance isn't as good as a hardware implementation but it's useful for programs that mandate these instructions to exist. Also useful for platforms with Qualcomm Oryon where their RNG implementation is bugged and doesn't return real values.

Plus a couple of other minor changes for future work. In total a fairly light month for JIT improvements.

Disk cache improvements

This month also had a large smattering of disk cache improvements. Now when the disk cache is enabled it will automatically limit its cache size to 1GB, where we will then stop caching at that point. This is configurable in the INI options for more or less space to be consumed, but because some edge cases allow it to have unbounded growth, it seems like a good balance at 1GB. We also added an optional memory buffer for caching; this allows an in-memory cache for hot blocks. This is only worth using if a game stutters frequently from JIT invocations and you have spare RAM to allow some additional caching.

Another important change that landed is that when FEX updates, it will automatically remove old disk caches that aren't relevant to the new version. This makes sure users don't have multiple 1GB cache files laying around as they upgrade over time.

A small improvement on the Linux side ensures that for ELF files being cached, that if there is an embeded GNU build-id hash, we will use that as a partial key for the file. This helps the case where a file could have been replaced and we would incorrectly use an outdated cache file.

Minor Vulkan thunk fixes

We upgraded our Vulkan thunks this month to fix a few minor bugs with newer Vulkan drivers. This fixes some games running on DGX Spark where they were supporting newer vulkan extensions primarily.

Raw Changes

FEX Release FEX-2610

• CMake

  • Stop searching for a bunch of projects when building for Steam (https://github.com/FEX-Emu/FEX/commit/902d433a00b8236ce60d7f10fb71b53c6d3cd419)

• CPUInfo

  • Adds new AVX extensions (https://github.com/FEX-Emu/FEX/commit/48d71752e2a363d5535181f803b0b0e9666c1825)

• Config

  • Add check to see if a config option is default (https://github.com/FEX-Emu/FEX/commit/177542e673b1f7d2179307c176ac8dd696784bed)

• DiskCache

  • Support FileID hash with gnu build-id (https://github.com/FEX-Emu/FEX/commit/d2b6e371e22a4fd29e0540fc52f57ecb563e2f31)
  • adjust key strategy (https://github.com/FEX-Emu/FEX/commit/72b2ff88aec844dbff88a87ab16ae6324b09cb6a)
  • configurable max file size (https://github.com/FEX-Emu/FEX/commit/08f451d3bce62a68aa6307dced5a8eaeb85c39bc)
  • optional memory buffer for faster lookups (https://github.com/FEX-Emu/FEX/commit/06b7f3b23222d4b5748f04e3758456e054f63ec5)
  • patch a bunch of disp32s in anon code (https://github.com/FEX-Emu/FEX/commit/98516b539e4a678de6d57f89263dbbf4711716ea)
  • allow relocations within the same ELF image (https://github.com/FEX-Emu/FEX/commit/e431bfe0fa80d44207eafc6c2c798cda70d41266)

• Externals

  • Update v

Read the blog post at FEX-Emu's Site!

"Kept you waiting, huh?" - Solid Snake, professional sneaky man.

We're not going to sugar coat it, this last month was very busy and time just kept moving faster than expected. A ton of work landed this month so we're going to just need to pick some of the highlights. Let's hop in!

Jit work you say?

A bunch of random bug fixes and performance improvements landed this month. Primary things here are that there were various behaviour fixes for some instructions. Undefined and defined flag fixes that FEX was messing up, some deadlock fixes, handling more cases that were previously asserts. There was an optimization for the PMULHRSW instruction family that increased some Geekbench benchmark numbers. Surely some game uses it somewhere to notice the 2x perf uplift. We also made the "full" self-modifying-code detection faster, which is used for Unity games in some instances. So that's nice. We also reduced some lock-contention in the JIT so multiple threads jitting code at the same time should block each other less frequently.

There's more of course but come on we gotta go!

Removed hacky unixlib fallbacks

For Wine we had some fallbacks for when our unixlib helper library wasn't installed. This last month we removed these fallbacks and require the unixlib for these features now. Currently all of these features technically are optional, so it isn't a big deal if unixlib is missing. We also discovered when removing these that we would accidentally be doing Windows syscalls if someone was replacing XTAJIT/Prism with FEX. Oops! Well those fallbacks are removed so that won't happen anymore at least.

Disk-cache as an option

Through an absolute flurry of commits this last month, disk caching of JIT code is in a place where we think it can be tinkered with more readily. What does disk caching mean for FEX? Once the configuration option is enabled; any code that FEX's JIT generates is cached to the disk in a FOZ file format database. Then when FEX needs to JIT code subsequently, it will attempt to look up code from that FOZ database first before going through the work of actually jitting. This database lookup happens during the initial run but also subsequent runs of FEX to reduce JIT stutter when running games.

Some downsides today are that if the disk caching is enabled on applications that JIT code itself, then the cache can technically grow unbounded. We haven't yet implemented any caching size limits or any way to remove stale cache entries. Additionally if any of the file contents change, there might be conflicts today where the cache doesn't get invalidated in those instances. So there's still some growing pains but we don't think these are a showstopper for tinkering today. To give it a try you can set the `FEX_DISKCACHE=1` environment variable or the `"DiskCache": "1"` json option to enable everything. This works with both the FEX linux interpreter program and also WINE, it maybe be a bit more stable in WINE for now. The cache files will end up in WINE's %APPDATA% directory or $HOME/.cache/fex-emu/ by default. So if you are running out of space make sure to delete those directories to recover some space!

Let us know how it goes and if there are any games that completely break when this feature is enabled! We'll be spending continued effort making this feature better as time goes on.

Oh right, the new logo

If you missed the random post, we got a new logo! You'll find that it's not dramatically different, but the rainbox gear has definitely changed!

Raw Changes

FEX Release FEX-2609

• ARMEmitter

  • Fill in bfloat API holes (https://github.com/FEX-Emu/FEX/commit/52cc5689dba6052c9de7a28e9a4a39a4d84ccb08)

• CMake

  • Fix MINGW compilation under ArchLinux (https://github.com/FEX-Emu/FEX/commit/fd141ed6d721d03062619e4702bca1a0c93b6dd9)

• CPUID

  • Adds a few new bits (https:

Read the blog post at FEX-Emu's Site!

After what felt like a very quick month, it's time for another FEX release. We're busy writing improvements for next month's release already, so let's not distract for too long and smash through these changes!

Removal of deprecated FEXInterpreter binary

Back in the FEX-2510 release we completely removed a FEXLoader binary with a deprecation of the FEXInterpreter binary, renaming it to just FEX. We kept the FEXInterpreter binary around for a while because downstream projects had hardcoded the binary in their scripts and we wanted to give some time for them to migrate. After nearly a year we're hoping that everyone has migrated to the new binary name. If your project is still using the old binary name then we recommend fix them!

The typical JIT improvements

As typical there are a bunch of JIT changes this month. Let's breakdown what was fixed.

• Use SVE XAR for SHA1 emulation as a minor optimization.
• Fix JIT telemetry value unconstrained predictable problem on ARMv8.0
  • Would only cause some missed telemetry on ARMv8.0 platforms, very minor.
• Optimize VMOVMASKPD/PS instructions.
  • Cutting from 11/10 instructions down to 7.
• Fix incorrect emulation of BFP store instruction
• Fixes crash if ProcessorID IR op was encountered on WINE
  • Silly little assert check that crashed.
• A bunch more improvements to hardware with 256-bit SVE
• Actually a ton of little bug fixes and edge cases getting patched up.

Fix VirtualProtect on win32

This change is fairly spicy because it means some of our memory protections were silently failing under WINE. With this fixed, it should fix some weird edge case crashes that weren't able to be explained.

Fix overwriting memory region in allocation tracking

This is a bit of a weird case where FEX's memory allocations on 48-bit VA systems were silently only ever using one allocation region. It was accidentally overwriting all regions with whatever the smallest allocation happened to be. Luckily on 48-bit systems, the smallest region would still be very large and unlikely to run out, but a bug fix is a bug fix.

Fix inactive JIT guard pages not being ignored

A funny little quirk of our JIT is that we rely on SIGSEGV to handle memory overflows safely. Where we try to calculate a size of code memory that gets generated from IR through a heuristic and if the size is calculated incorrectly we safely fault and longjump to try again. The times in which the heuristic is incorrect isn't very frequent so the performance win of not range checking as heavily is a big win.

One problem is we forgot to turn off the JIT guard page checking when outside of the JIT, so if an application happens to access from [0, 4096), we would long jump back in to our JIT and hilarious crashes would occur. Oops!

Fixes AVX signal state saving and restoring

While signals are fairly rare, we implementing saving the AVX state incorrectly, which could cause us the fail to restore it if the application jumps to a new location. This is now fixed which should probably resolve a few crashes.

Spinloop using WFE on WINE

On WINE we had some spinloops that are required when an application is in the process of suspending a thread. We now spin using the WFE instruction which puts the ARM CPU in lower-power state while waiting on an atomic. This can shave off watts of power in this edge case, so it's worth supporting!

Fix untracked thread handling

When a process creates a new thread using the Clone syscall API, there's a lot of tricky edge cases to support. In one of those paths we were accidentally losing tracking of a thread in our frontend. Usually this is pretty benign but it could result in some broken TLS tracking due to a new thread data structure actually pointing to the parent thread. Oops!

Raw Changes

FEX Release FEX-2608

• 64BitAllocator

Read the blog post at FEX-Emu's Site!

Time certainly goes by quicker than you'd expect. We even managed to skip last months release because we were busy doing other things. Let's take that as an example and fly through the changes that we did over these past two months!

Optimizations and fixes for 256-bit SVE2 hardware

While this hardware doesn't exist yet, we know it is an inevitability that it will at some point. While we switched gears a couple years ago to implement AVX using 128-bit operations, we never removed this code. In-fact we've been spending a bunch of effort on it fixing bugs and optimizing it so that once hardware ships it won't be broken. We have now validated extensively that all AVX instructions now zero extend their results as expected and optimized a bunch of the instructions so they generate faster code. We think it's now at a point that in the common case the code will be faster than our 128-bit emulation, but there is definitely still some more work todo. SVE2.1 provides a significant improvement to how shuffles operate and we haven't gotten those implemented yet. Because there aren't any 256-bit SVE2 hardware on the market, we might even require SVE2.1 or SVE2.2 for this class of hardware. We'll of course still maintain our 128-bit path for lower-end hardware of course.

Various JIT fixes/changes

Once again, too many to go through individually, let's throw it in a list. Most of these are just bug fixes but there are a handful of optimizations as well.

• Fixed corruption with back to back PMOVMSKB instructions and full SMC detection
  • Fixes Vivado in this situation
• Handle incorrect LOCK prefixed instructions correctly
• Allow larger CPU context state
  • Fixes compilation error with musl
• Fixes vsyscall page tracking
  • Was accidentally a NOEXEC page
• Only use DC ZVA for Ampere when clearing AVX state
  • It's the same or slower on other hardware
• Fix zero extension in VCVTPS2PH
• Fixes CRC32 with high 8-bit registers
• Fixes 64-bit LODS instruction with address size override
• Fix Mafia 3 in Arm64ec Proton
• Fix Bioshock and other 32-bit games that disable DEP on Arm64ec Proton
  • Requires bleeding-edge Proton or WINE
• Optimize x87 FYL2X/FPREM/FPREM1
• Fix incorrect RSP update on 16-bit LEAVE instruction
• Handle some new CPL0 only instructions correctly
• Fix JIT allocations ending up in 32-bit VA space
  • Fixes some 32-bit games under Proton

Implement support for a unixlib under Proton/WINE

For a long time FEX's WINE support works by building all of FEX as a DLL file which WINE loads at runtime. This has worked for us pretty well but constrained some of our design decisions at times, causing us to take less optimal or hacky paths. We also have some upcoming changes that would be dramatically harder to implement without a unixlib so we decided to start using one. Now FEX for WINE ships two DLL files, and two unixlib SO files.

Right now we are duplicating some code between the unixlib and the FEX WINE DLL file to make sure everything works correctly but within a few months we are likely to remove the duplicated code and rely entirely on the unixlib. Currently all the functionality provided is optional, but we would recommend using it because we may eventually not have it be optional.

Partial support for CUDA thunking

Some people were asking for CUDA thunking on the DGX Spark and we decided to implement partial support for it. It's not 100% coverage but depending on the workload it can potentially work. We know today that if you try to execute applications with the static CUDA runtime linked, it won't work well. Other than that give it a try and you might be surprised at what works.

Raw Changes

FEX Release FEX-2607

• AVX
  • Remove unnecessary moves from VPSHUF{D, HW, LW} (https://github.com/FEX-Emu/FEX/co