The compilation pipeline

The purpose of this document is to give an overview of the different stages of the WebSharper compilation pipeline, from F#/C# source code (or F# quotation) to the final JavaScript/TypeScript output, and point to the relevant parts of the codebase for each stage.

The main stages are:

• Interpreting project argunents and WebSharper configuration from wsconfig.json files, and loading referenced assemblies' WebSharper metadata to create a combined metadata object for the compilation.
• Parsing and checking the source code with F# Compiler Service or Roslyn (or F# quotation via Reflection), and setting up a WebSharper Compilation object with all the information and ASTs needed. A project reader module deals with recognizing the types/members that need to be translated based on WebSharper attributes, and a code reader module deals with transforming member expressions from source to WebSharper AST.
• Some extra F#/C# specific processing is done to handle language-specific features and optimizations, like tail calls and C# asyncs.
• The name resolver creates a mapping of .NET types/members to their JavaScript equivalents, based on WebSharper attributes, respecting method overrides and interface implementations, and avoiding name conflicts (raising errors if necessary).
• The translator converts the .NET-specific AST forms to JavaScript-compatible AST forms.
• Various optimization passes are run on the JavaScript AST, and statement-based code inside expressions are eliminated so that JavaScript-compatible statement-based code is achieved.
• At this point, all the code is in JavaScript AST form, and depending on project type, various bundling/packaging steps are done to create standalone or interdependent modules.
• Then finally another transformation is done to convert the main WebSharper AST statements to JavaScript/TypeScript AST, resoving local variable names to be non-conflicting, and this is can be written out as JavaScript/TypeScript code. Also, the compilation results are persisted into metadata inside the dll, so that it can be used as a reference in other WebSharper projects.

Main compiler logic

The WebSharper.FSharp and WebSharper.CSharp projects are the main entry points for the WebSharper F# and C# compilers respectively. For F#, it also handles calling into the WebSharper.FSharp.Service background build service if configured so.

The WebSharper.Compiler.FSharp and WebSharper.Compiler.CSharp projects contain the language-specific compilation logic. They both have some similarly named files, with same goals:

• Compile.fs handles the main compilation pipeline logic, calling into the other modules for each stage and handling different WebSharper project types.
• Main.fs is a helper class for running the central stages of the compilation pipeline, like creating the Compilation object, populating it with code from source, running the translator and optimizations.
• ErrorPrinting.fs handles formatting and printing errors/warnings from the compilation process.
• ProjectReader.fs handles reading a checked F#/C# project and extracting the WebSharper-relevant types/members to populate the Compilation object.
• CodeReader.fs handles reading F#/C# member bodies and transforming them to WebSharper AST expressions/statements.

The WebSharper.Compiler.FSharp project also has:

• ArgCurrying.fs that analyzes tupled/curried F# function arguments, and automatically simplifying them to simpler JavaScript form (function with flat arguments) if possible (the function is always called with all arguments).
• TailCalls.fs that analyzes tail calls in F# functions and transforms them to loops when possible.

The WebSharper.Compiler.FSharp project also has:

• RoslynHelpers.g.fs, which is a generated file that wraps all Roslyn syntax forms into F# discriminated unions for easier pattern matching. This file is generated by the genHelper script, using the Syntax.xml file as input which is taken from the Roslyn repo.
• Scoping.fs that analyzes C# code scoping rules to determine variable lifetimes for proper JavaScript translation.
• Continuation.fs that analyzes C# awaits, yield returns, and gotos and transforms them to state machines for proper JavaScript translation. (Note: try/finally in state machines are not yet supported: issue #542)

Project and code readers

These steps are separate for C# and F# although their structure is similar, and the goal is the same to produce WebSharper.Core AST.

They use a couple shared helpers from the WebSharper.Compiler project, some of these are:

• assemblyResolution/AssemblyResolver.fs, which have a separate .NET 4x and modern .NET implementation (the former for the currently defunct C# analyzer). It handles finding references of current compilation to be loaded for metaprogramming, and loading assemblies into an AssemblyLoadContext so that they can be unloaded.
• Extra.fs handles parsing and executing on extra.files files which enable pattern-based file embedding and copying to output directory.
• AttributeReader.fs, which is the common logic for reading and handling WebSharper-defined attributes.
• Recognize.fs parses Inline strings, verified for correctness. Not all JavaScript forms are supported yet.
• Stubs.fs creates inline expressions for members marked with the [<Stub>] attribute.

Also there is a third path for creating a WebSharper Compilation, from an F# assembly using Reflection on FShapr.Quotations.Expr expressions and expressions created and persisted by the compiler in an assembly by the [<ReflectedDefinition>] attribute. If an F# member has a ReflectedDefinition, its expression will be used instead of its source code by the WebSharper F# compiler. The related files are:

• QuotationReader.fs handles transforming a single FShapr.Quotations.Expr.
• ReflectedDefinitionReader.fs handles interpreting an expression stored as a ReflectedDefinition.
• QuotationCompiler.fs handles transforming a whole assembly using its ReflectedDefinitions, and allowing to compile individual expressions on top of that. This is not used by any of the standard WebSharper tooling, but available as public API, and is used by the Warp Interactive.

Name resolver

The WebSharper.Compiler project contains the compiler parts that are independent of the source language. The project readers populate the WebSharper.Compiler.Compilation object with types/members to translate, and its Resolve method assigns JavaScript names to all members, in this order:

• Members with explicit JavaScript names via WebSharper Name attribute are assigned those names.
• Interface members are named in order of inheritance to avoid conflicts.
• Abstract members on classes are named to avoid conflicts. Overrides and implementations get the same name as the base/abstract member, and they cannot be explicitly named.
• Remaining class members are named to avoid conflicts.

Translator

The WebSharper.Compiler.Translator.DotNetToJavaScript.CompileFull method handles the main translation logic. It iterates through all pending members, and eliminates all .NET-specific AST forms, converting them to JavaScript-compatible AST forms. Then a couple optimization steps are run to simplify the code further:

• Remove Let bindings, simplifying them to direct expressions where possible, local variables otherwise.
• Remove most calls to WebSharper's Runtime.js methods when their functionality can be inlined directly.
• Break down statements inside expressions to proper JavaScript statements.
• Clean up unused variable declarations that has been made unnecessary by previous optimizer steps.
• Use .apply instead of Apply helper from Runtime.js.
• Then there is a safety check to ensure no .NET-specific AST forms remain.

Finally, the member is added back to the Compilation object as part of the compiled dictionaries instead of the compiling queues.

Helpers

The translator uses some helper files with a bunch of utilities. These are:

• CompilationHelpers.fs contains various utilities, analyzers, transformers, visitors, and active patters that help the translation and optimization process.
• Optimizations.fs focuses on optimizing AST forms for shorter and cleaner output.
• Breaker.fs contains the main logic for breaking expressions that internally need to use JavaScript statements (the StatementExpr form), often arising from F# because it's an expression-based language.

Bundling and packaging

The JavaScriptPackager.fs file in the WebSharper.Compiler project handles bundling and packaging of JavaScript code, meaning creating a full AST.Statement[] outputs that will form an output file. This involves resolving imports and in case of dead code eliminated bundles, sorting the defined classes in right order of inheritance. See the JavaScript translation documentation for output modes.

The Bundle.fs file is the main logic for writing SPA projects, which only create a single dead code eliminated .js output and potentially extra bundles for any web workers needed.

Writing output

The JavaScriptWriter.fs file in the WebSharper.Compiler project handles writing JavaScript/TypeScript AST from the WebSharper JavaScript. This step resolves local variable names to be non-conflicting, and doing some simple last-catch optimizations, like flattening nested statement blocks if not necessary.

Then the Writer.fs file in the WebSharper.Compiler.JavaScript project handles writing the final JavaScript/TypeScript code as text. This is a straightforward write into a StringBuilder. If source mapping is turned on, the CodeWriter instance also creates .map files by using the source position information that was preserved through AST transformations.

Creating WebSharper metadata

The Frontend.fs file in the WebSharper.Compiler project handles creating and serializing WebSharper metadata from the compilation results as well as all other resources that gets embedded into the output dll.

There are two types of metadata, the standard one will get embedded as a zippped WebSharper.meta file, this is used by WebSharper projects building atop the current project. The other is WebSharper.runtime.meta, only created for sitelet projects, where the sitelet runtime will use it for resolving remoting, dependencies, and page initialization code.

WebSharper Interface Generator

WIG is a tool to create JavaScript bindings defined declaratively in F#. The API and model types for this is defined in the WebSharper.InterfaceGenerator project, while the WIGCompile.fs file in WebSharper.Compiler can take an assembly definition created using these model types and transform it into a dll with Mono.Cecil. For observability, it creates all the [<Inline>] attributes on members. It does not create WebSharper metadata based on the original definition, but in a second step uses the produced assembly and inline string to create the WebSharper metadata, this is done by the Reflector.fs module.