This last week we spent the better part of a day chasing down a space. A single white-space character that was breaking an Internet Information Service (IIS) deployment for a website. Debugging the issue would have been straight forward except that all of the errors were pointing us in the wrong direction. To start let me provide details on our deployment.

The website was a .NET 5 Web API hosted in IIS. The project had been modified so that it would generate a DLL and not an EXE upon being compiled. This was done by adding <UseAppHost>false</UseAppHost> within the <PropertyGroup> to the .CSPROJ of the Web API. By default the UseAppHost value is true which will cause the build to generate a framework dependent executable. For our situation the default configuration wasn’t desired. Additionally, since we were no longer generating an EXE that IIS could use to launch the application we also needed to update the web.config.

Within the web.config the launch definition is contained in the <aspNetCore> element. Initially the element was defined with the processPath attribute pointing to the framework built EXE of the project. Now that we weren’t generating this executable we needed to define how IIS would launch the site. To do this we updated the processPath attribute to point to the dotnet.exe binary that was installed with the .NET Core Hosting Bundle. We also added the arguments attribute passing in the exec command and path to the project’s main DLL. In the end the <aspNetCore> element looked like this.

Broken web.config

<aspNetCore processPath="dotnet.exe"
            arguments=" exec .\WebApi.dll"
            stdoutLogEnabled="false"
            stdoutLogFile=".\logs\stdout"
            hostingModel="inprocess">
  <environmentVariables>
    <environmentVariable name="ASPNETCORE_ENVIRONMENT" value="Development" />
  </environmentVariables>
</aspNetCore>

With these updates in place we deployed the app to our development environment and found that the site was displaying a 500.31 error.

Check the system event log for error messages
Enable logging the application process' stdout messages
Attach a debugger to the application process and inspect

The guidance from the Microsoft page tells us to look at the Application Event logs for further details. In examining the events we see three separate error events reported against IIS.

1. Unable to locate application dependencies. Ensure that the versions of Microsoft.NetCore.App and Microsoft.AspNetCore.App targeted by the application are installed.
2. Could not find ‘aspnetcorev2_inprocess.dll’. Exception message:
3. Failed to start application ‘/LM/W3SVC/1/ROOT/WebApi’, ErrorCode ‘0x8000ffff’.

We verified the installation, and re-installed, the .NET Hosting Bundle, checked to make sure that all of the files needed for the site were being deployed, ran a check for all dependencies, and even ran the dotnet.exe command and arguments from the command line. Everything checked out and looked good. Performed the standard exhaustive search on StackOverflow but didn’t have any success with the suggested fixes.

Finally, since the application worked locally from Visual Studio we started walking through the startup process. Since no logs were generated during startup we took a look at our web.config and transformation files. What we found was that in the arguments definition of the <aspNetCore> element there was an extra space between the quote and exec command.

<aspNetCore processPath="dotnet.exe" arguments=" exec .\WebApi.dll" …/>

It seemed improbable that the extra space would be causing such havoc on the IIS deployment but it was in fact the problem that broke the website. We removed the extra space and the fixed configuration file was deployed. To our surprise all errors went away.

Fixed web.config

<aspNetCore processPath="dotnet.exe"
            arguments="exec .\WebApi.dll"
            stdoutLogEnabled="false"
            stdoutLogFile=".\logs\stdout"
            hostingModel="inprocess">
  <environmentVariables>
    <environmentVariable name="ASPNETCORE_ENVIRONMENT" value="Development" />
  </environmentVariables>
</aspNetCore>

So pay attention to the spaces in your web.config deployment and transformation files. The error details provided by IIS and the event logs certainly did not indicate the underlying issue in this instance. But through a process of elimination we were able to track down the cause of issue and fix it.