Hayden

Have you ever encountered the frustrating “Error 10 (Out of resources)” message that prevents VMware ESXi from booting properly during installation or upgrade? This issue has become increasingly common, particularly with ESXi 7.0 and later versions, typically caused by memory resource limitations or BIOS configuration problems. Today, we’ll walk through systematic approaches to resolve this troublesome issue using proven solutions from real-world deployments.

 

The ESXi “Error 10 Out of resources” error occurs when the system cannot allocate sufficient memory resources during the boot process. VMware has been gradually deprecating Legacy BIOS support starting with ESXi 8.0, which frequently leads to boot failures due to memory limitations in legacy BIOS mode.

Common error messages include:

  • “Error 10 (Out of resources) while loading module”
  • “Requested malloc size failed”
  • “No free memory”
  • “gzip extract failed for [filename]”

 

 

1. [Solution] Switching from Legacy BIOS to UEFI Boot Mode

The most reliable and effective solution is switching the boot mode to UEFI. VMware officially recommends UEFI mode transition as the primary resolution for this issue.

UEFI Mode Configuration Steps

  1. Server Reboot and BIOS Access
    • Restart the server and press F2, Delete, or F12 to enter BIOS setup.
  2. Boot Mode Configuration
    • Navigate to the Boot menu and locate “Boot Mode” or “UEFI/Legacy Boot” options.
    • Change from Legacy or CSM to UEFI.
  3. Secure Boot Settings (Optional)
    • UEFI mode supports Secure Boot activation.
    • Initially disable this setting for boot testing purposes.
  4. Save and Restart
    • Press F10 to save settings and reboot.

Benefits of UEFI Transition

Feature Legacy BIOS UEFI
Boot Speed Slower Faster
Memory Support Limited Large capacity support
Security Features Limited Secure Boot support
Partition Size 2TB limit 9.4ZB support

 

 

2. [Solution] BIOS Memory Configuration Optimization

When UEFI transition isn’t possible, adjusting BIOS memory-related settings can resolve the issue.

MMCFG (Memory Mapped Configuration) Settings

For specific hardware like Cisco UCS C240 M3 servers, adjusting MMCFG settings can resolve the problem.

Configuration Method:

  1. Navigate to “PCI Configuration” in BIOS settings
  2. Change “MMCFG Base” from Auto to 2GB
  3. Set “Memory Mapped IO above 4G” to Enabled

Enable Above 4G Decoding

Modern servers require enabling memory decoding above 4GB.

Configuration Path: Advanced → PCI Configuration → Above 4G Decoding → Enabled

 

 

3. [Solution] Disable Thunderbolt Controller

On compact systems like Intel NUC Skull Canyon, Thunderbolt controllers can cause resource conflicts during ESXi installation.

Disable Procedure

  1. Access BIOS Settings
    • Press F2 to enter BIOS setup.
  2. Locate Thunderbolt Settings
    • Navigate to Advanced → Devices → Thunderbolt or
    • Devices → USB Configuration → Thunderbolt
  3. Disable Controller
    • Set Thunderbolt Controller to Disabled
  4. Re-enable After Installation
    • Can be re-enabled after ESXi installation if needed

 

 

4. Version-Specific Solutions

ESXi 8.0 and Later

Starting with ESXi 8.0, Legacy BIOS support is officially deprecated, and VMware plans to completely discontinue Legacy BIOS support in future releases.

Recommended Solutions:

  • Complete transition to UEFI mode
  • Verify hardware compatibility using VMware HCL

ESXi 6.x and 7.x Versions

For older hardware, try the following approaches:

Memory Allocation Adjustment:

# Check memory settings in ESXi shell
esxcli system settings kernel list -o heapFreeThreshold

Boot Option Addition:

  • Press Shift+O during ESXi boot to edit boot options
  • Add limitation options like maxLogicalCPUs=8 or mem=4096M

 

 

5. Hardware-Specific Considerations

Dell PowerEdge Servers

For Dell servers, verify the following settings:

  1. System Profile
    • Change Performance Per Watt (DAPC) to Performance
  2. Memory Operating Mode
    • Set Advanced ECC to Optimizer Mode
  3. Processor Settings
    • Enable Logical Processor
    • Enable Virtualization Technology

HP ProLiant Servers

For HP servers:

  1. Power Management
    • Set Dynamic Power Savings to Static High Performance
  2. Memory Settings
    • Configure Advanced Memory Protection to Advanced ECC

SuperMicro Servers

For SuperMicro boards:

  1. BIOS Version Check
    • Update to latest BIOS version recommended
  2. CPU Configuration
    • Enable Hyper-Threading
    • Enable Execute Disable Bit

 

 

6. Troubleshooting Checklist

Step-by-step checklist for resolving ESXi “Error 10 Out of resources” issues:

Basic Verification

  • [ ] Confirm server has minimum 4GB RAM
  • [ ] Verify hardware is listed in VMware HCL
  • [ ] Check BIOS/UEFI firmware is latest version

Boot Mode Verification

  • [ ] Determine current boot mode (Legacy BIOS vs UEFI)
  • [ ] Verify UEFI mode transition capability
  • [ ] Check Secure Boot settings (initially disable for testing)

BIOS Settings Optimization

  • [ ] Change MMCFG setting to 2GB
  • [ ] Enable Memory Mapped IO above 4G
  • [ ] Enable Above 4G Decoding
  • [ ] Disable Thunderbolt controller (if applicable)

Alternative Methods

  • [ ] Try installing previous ESXi version then upgrade
  • [ ] Test installation with some memory modules removed
  • [ ] Recreate USB installation media

Log Monitoring

Check the following logs when issues occur:

# Check boot-related logs in ESXi shell
/var/log/boot.gz
/var/log/vmware.log
/var/log/vmkwarning.log

 

 

VMware ESXi “Error 10 Out of resources” issues are primarily caused by Legacy BIOS memory limitations. The most reliable solution is transitioning to UEFI mode, while environments that cannot make this transition can resolve issues through BIOS setting optimization. Particularly for ESXi 8.0 and later versions, UEFI transition is becoming mandatory rather than optional. We recommend proactively transitioning to UEFI environments to build stable virtualization infrastructure.

If all the above methods fail to resolve the issue, it’s likely a hardware compatibility problem. Consider contacting VMware technical support or upgrading to compatible hardware. 🙂

 

Leave a Reply