Modularity

Modularity is a key concepts in software development. Being able to extend an application without modifying it, is the holy grail of many developers. A way to reach this quest is to use the plugin architecture.

Developers are used to the concept of SOLID principles at an atomic level with classes, the same concepts are applied with the plugin architecture but this time at the application level. Plugins provide extensibility, flexibility, and isolation of application features.

The plugin architecture is divided in two parts :

• Core: is the main application which is extended with new features encapsulated inside Plugin. The core keeps track of plugins (registry), and manage their lifecycle.
• Plugin: contains the feature logic. Plugins could be added or removed from the core at any time.

Module Federation

So now that we have this tiny definition of a plugin architecture how can we implemented it in the front end world.

Unless you’ve been living in a basement for a year, you should be aware of Module Federation and how this technology is a game-changer in achieving micro front-end architecture.

If not please refer you to Zack Jackson the creator of Module Federation. (Hero don’t wear cape, what did this guy is really amazing)

Zack said: “Module Federation gives us a new method of sharing code between frontend applications. It allows a Javascript application to dynamically run code from another bundle/build, on client and server“.

In module federation you will find the concept of

Host: Think of this as the wrapper for the other micro-apps. Its primary purpose is to load micro-apps.

Remote: Remote modules are modules that are not part of host build and loaded from a so-called container at the runtime. Remotes exposes their code thanks to the exposes property inside webpack.config.js (module-federation.config.js for Nx project)

From the definition that we have, we can see that Module Federation has lot of things in common with plugin architecture:

• Remote module could be assimilate to a Plugin
• Host app to the Core application.

The base are already there: code isolation with remote app, extending feature by loading code dynamically at runtime.

Implementation

So what are we waiting for to start.

In our implementation, instead of hosting each remote app separately we will serve all plugins statically in the same webserver (In production, it will be preferable to publish your plugins on a cloud storage like GCS).

Our implementation will look something like that:

Create the Workspace

To speed up the development I will use NX Mono Repo.

• First step lets creates the repository that will contains all projects (host, Plugin, Plugins host server, cli…).

> npx create-nx-workspace@latest

I chose the integrated style and choose an empty workspace to create apps and as workspace name pluggy.

• Install and add new dependencies.

> cd pluggy-nx
> yarn
> yarn add --dev @nrwl/react -W

“@nrwl/react “ simplifies the creation of Host and Remote application. Nx has predefined template generators which bootstrap react projects for us.

> nx g @nrwl/react:host host/pluggy --remotes=plugins/plugme --e2eTestRunner=none

• Start the project

Running the following command will start the host application, it serves also the remote application automatically (the plugins-plugme app).

> nx serve host-pluggy

On the right you will see a menu. The home link point to the host app and the second link to the remote App which is downloaded dynamically into the host application.

You can find the code of this first step here.

Load code with dynamic API

The default configuration of module federation that has been generated by NX for react ‘@nrwl/react’ is static.

Note: Nx for Angular ‘@nrwl/angular ‘ can generates host application which is configured to download remote Application dynamically at runtime by adding — dynamic at the end of the generator command.

> nx g @nrwl/angular:host host/pluggy — remotes=plugins/plugme — dynamic

But someone can argue, Module Federation loads code at runtime is not static, and you are right.

So, what I mean by static is the fact that urls of remote application are hardcoded inside the webpack.config.json (nx uses a seperated file to configure module federation parameters in module-federation.config.js)

// apps/host/pluggy/module-federation.config.js
const moduleFederationConfig = {
  name: 'host-pluggy',
  remotes: ['plugins-plugme'], // remotes application are referenced here
};

In our plugin architecture we would like to extend the host application at runtime without relaunching it or rebuild it. We wanted to add plugins which host has no knowledge about them.

What we should do, it’s to dynamically import remote code from url that will be defined at runtime. Module-Federation-Examples shows the way to did. If you are interested in this technology, I advise you to look at the other examples.

We have some code to write to import ES module at run time, we will store this code in a separate library. (Thanks to Nx, creating a library is a breeze :)

Let’s create one:

> nx g @nrwl/react:lib core

create a file in the libs/core/src/lib/dynamic-module.ts and put the following code in it.

import React from 'react';

interface Container {
  init(shareScope: string): void;

  get(module: string): () => any;
}

declare const __webpack_init_sharing__: (shareScope: string) => Promise<void>;
declare const __webpack_share_scopes__: { default: string };

function loadModule(url: string) {
  try {
    return import(/* webpackIgnore:true */ url);
  } catch (e) {

  }
  return null;
}

function loadComponent(remoteUrl: string, scope: string, module: string) {
  return async () => {
// eslint-disable-next-line no-undef
    await __webpack_init_sharing__('default');
    const container: Container = await loadModule(remoteUrl);
    console.log(container);
// eslint-disable-next-line no-undef
    await container.init(__webpack_share_scopes__.default);
    const factory = await container.get(module);
    const Module = factory();
    return Module;
  };
}

const componentCache = new Map();
// Hook to cache downloaded component and which encapsulates the logic to load 
// module dynamically
export const useFederatedComponent = (
  remoteUrl: string,
  scope: any,
  module: string,
) => {
  const key = `${remoteUrl}-${scope}-${module}`;
  const [Component, setComponent] = React.useState<any>(null);

  React.useEffect(() => {
    if (Component) setComponent(null);
    // Only recalculate when key changes
  }, [key]);

  React.useEffect(() => {
    if (!Component) {
      const Comp = React.lazy(loadComponent(remoteUrl, scope, module));
      componentCache.set(key, Comp);
      setComponent(Comp);
    }
    // key includes all dependencies (scope/module)
  }, [Component, key]);

  return {Component};
};

One thing important that you should retain is that we use dynamic import to load ES module at runtime. You can see also that we tell Webpack to not parse dynamic url and reinterpret them like for code splitting functionnality.

import(/* webpackIgnore:true */ url);

In Module-Federation-Examples repo, they use another approach to load dynamic code by injecting dynamically the the url of the remote bundle inside a script tag in the html header of the host application.

We can now remove in the host application the remotes section in module-federation.config.js as we will resolve remotes module (aka Plugin) at runtime.

// apps/host/pluggy/module-federation.config.js
const moduleFederationConfig = {
  name: 'composer-client',
  remotes: [],
};

We change the host application to load external module with dynamic api. Instead of importing the remote code like we did for static api (where webpack uses url defined in webpack.config.js to determine remotes plugin name and their url).

const PluginsPlugme = React.lazy(() => import('plugins-plugme/Module'));

We will use the hook useFederactedComponent that we’ve created just before in the core library inside dynamic-module.ts. We wrapped it in the DynamicComponent to simplify the code that can be called with props like this.

You can remarks that the url points to a remotreEntry.js file. This is a bundled file generated by Module Federation Webpack plugin, that serves as an entry point for hosting apps (Host apps will get many information from it like dependencies of the Remote App).

<DynamicComponent url="http://localhost:4201/remoteEntry.js" 
    scope="plugme" 
    module="./Module" />

Let see the modified code inside the host App

host/pluggy/src/app/app.tsx

export const DynamicComponent = ({url,scope,module}: { url:string, scope: string, module:string }) => {
  const {Component: DynComponent} = useFederatedComponent(url, scope, module);
  return DynComponent && <DynComponent/>;
};

export function App() {
  return (
    <React.Suspense fallback={null}>
      <ul>
        <li>
          <Link to="/">Home</Link>
        </li>

        <li>
          <Link to="/plugins-plugme">PluginsPlugme</Link>
        </li>
      </ul>
      <Routes>
        <Route path="/" element={<NxWelcome title="host-pluggy" />} />

        <Route path="/plugins-plugme" element={ DynamicComponent && <DynamicComponent url="http://localhost:4201/remoteEntry.js" scope="plugme" module="./Module" />} />
      </Routes>
    </React.Suspense>
  );
}

You can find the code of this second step here

Hosting / Registering / Publishing Plugins

Ok now that we have the plumbing, we need a way to register our plugin and publish it to be able to have a kind of plugin locator. For that we can use two kind of architectures that I have in mind:

• The first one is to let each remote app to be hosted in their own Webserver and register their hosted url to a centralized server.
• The second one is to centralize the deployment and the hosting of the Plugin inside a unique Webserver (in production a cloud storage is a good solution like GCS, S3,…)

In our case, we will use the second one. We will create a NestJs server that will host plugins, and manages the registration and publication.

The server will expose an Api to publish plugin to it and get a list of the plugins that have been published on it.

We will use Nx generator to generate for us a NestJs project. For that we install the following dependency

 > yarn add -D @nrwl/nest -W  

For generating the nestjs project run the following command

 >  nx g @nrwl/nest:app server/pluggy-locator  

We will add two apis in the app.controller.ts:

• ‘/plugins’: for getting the list of plugins
• ‘/upload’: for uploading a plugin archive to the server. The archive is a zip which contains the Remote App bundle.

import {Controller, Get, Post, UploadedFile, UseInterceptors} from '@nestjs/common';

import {AppService} from './app.service';
import {FileInterceptor} from "@nestjs/platform-express";
import {Express} from 'express';

@Controller()
export class AppController {
  constructor(private readonly appService: AppService) {
  }

  @Get('plugins')
  getPluginsList(): Promise<string[]> {
    return this.appService.getPluginsList();
  }

  @Post('upload')
  @UseInterceptors(FileInterceptor('file'))
  async uploadFile(@UploadedFile() file: Express.Multer.File) {
    const response = {
      originalname: file.originalname,
      filename: file.filename,
    };
    await this.appService.UploadedFile(file);
    return response;
  }
}

The implementation of these two apis are contained inside the app.service.ts

import { Injectable } from '@nestjs/common';
import { Express } from 'express';
import { Multer } from 'multer';
import {join, basename} from "path";
import { readdir, stat, existsSync } from 'fs-extra';
import {execute} from "@pluggy/core";
import {mkdirSync} from "fs";

@Injectable()
export class AppService {

  async  getDirectories(path): Promise<string[]> {
    let dirs = [];
    if(existsSync(path)) {
      const filesAndDirectories = await readdir(path);

      const directories = [];
      await Promise.all(
        filesAndDirectories.map(name => {
          return stat(join(path, name))
            .then(stat => {
              if (stat.isDirectory()) directories.push(name)
            })
        })
      );
      dirs = directories;
    }
    return dirs;
}

  async getPluginsList(): Promise<string[]> {
    return (await this.getDirectories(join(__dirname, 'plugins')));
  }

  async UploadedFile(file: Express.Multer.File) {

    const pluginName  = basename(file.originalname, '.zip');
    const pluginsDir = join(__dirname, 'plugins');
    const pluginDir = join(pluginsDir, `${pluginName}`);

    if(!existsSync(pluginsDir)) {
      mkdirSync(pluginsDir);
    }
    await execute(`cd ${join(__dirname, 'plugins')} && unzip -o ${pluginName}.zip -d ${pluginDir}`);
  }
}

For uploading files we uses the Multer library. To use it we install the following dependencies inside the server/pluggy-locator

> cd apps/server/pluggy-locator
> yarn add multer @types/multer

The multer should be configured inside the app.module.ts

import { Module } from '@nestjs/common';

import { AppController } from './app.controller';
import { AppService } from './app.service';
import {join} from "path";
import {MulterModule} from "@nestjs/platform-express";
import {diskStorage} from "multer";

@Module({
  imports: [MulterModule.register({
    storage: diskStorage({
      destination: join(__dirname, 'plugins'),
      filename: (req, file, cb) => {
        cb(null, file.originalname);
      }}),
  })],
  controllers: [AppController],
  providers: [AppService],
})
export class AppModule {}

To transform the Nest server to a Webserver that hosts and serves static files, we change the main.ts file and use the express.useStaticAssets() function which points to the plugins directory.

async function bootstrap() {
  const app = await NestFactory.create<NestExpressApplication>(AppModule);
  const port = process.env.PORT || 3333;
  app.useStaticAssets(join(__dirname,'plugins'), {prefix: '/plugins', setHeaders: (res, path, stat) => {
    res.set('Access-Control-Allow-Origin', '*');
  }});
  app.enableCors();
  await app.listen(port);
  Logger.log(
    `🚀 Application is running on: http://localhost:${port}/`
  );
}

to launch the server run

> nx serve server-pluggy-locator

You can find the code of this part here

Facilitate Plugin Creation, Packaging and Publishing

To facilitate building, publishing and creating plugins, we can use custom plugins of Nx:

• Local executors: to launch custom commands inside Nx projects (kind of cli dedicated to Nx project)
• Local generators: to generate custom projects from files templates.

Nx plugins facilitate greatly the creation of a mono repo CLI and generators in a professional way.

First we will have to create a custom Nx plugin:

> yarn add -D @nrwl/nx-plugin@latest -W
> nx g @nrwl/nx-plugin:plugin plugin --e2eTestRunner=none --unitTestRunner=none --minimal

Next we will generate two executors:

• a pack executor: to create plugin archive
• a publish executor: to publish archive on the pluggy-locator server

> nx generate @nrwl/nx-plugin:executor pack --project=plugin 
> nx generate @nrwl/nx-plugin:executor publish --project=plugin 

This is the code of the executor.ts file of Pack (libs/plugin/src/executors/pack).

import { ExecutorContext } from '@nrwl/devkit';
import { PackExecutorSchema } from './schema';
import {join, basename} from 'path';
import {execute} from "@pluggy/core";

export default async function runExecutor(
  options: PackExecutorSchema,
  context: ExecutorContext
) {

  console.log(`🚧 Packing ${context.projectName}...`);
  const projectOutputPath = context.workspace.projects[context.projectName].targets.build.options.outputPath;
  const pluginDir = join(context.root, projectOutputPath,'..');

  const pluginName = basename(projectOutputPath);
  const archive = join(pluginDir, `${pluginName}.zip`);

  await execute(`rm -rf ${archive} && cd ${pluginDir}  && zip -r -j ${archive} ${pluginName}/* && cd -`);

  return {
    success: true
  };
}

And the code of the Publish executor (libs/plugin/src/executors/pack),

import { ExecutorContext, runExecutor as run} from '@nrwl/devkit';
import { PublishExecutorSchema } from './schema';
import {execute} from "@pluggy/core";
import {basename, join} from 'path';

export const runExec = async (projectName:string, target:string, context: ExecutorContext) => {
  console.log(`✨ ${target} ${projectName}...`);
  const result = await run({project:projectName, target: target}, {watch:false, progress:false}, context);
  for await (const res of result) {
    if (!res.success) throw new Error(`Failed to run ${projectName} ${target}`);
  }
  return {
    success: true
  };
};

export default async function runExecutor(
  options: PublishExecutorSchema,
  context: ExecutorContext
) {
  const projectOutputPath = context.workspace.projects[context.projectName].targets.build.options.outputPath;
  const pluginDir = join(context.root, projectOutputPath,'..');
  const pluginName = basename(projectOutputPath);
  const archive = join(pluginDir, `${pluginName}.zip`);

  await runExec(context.projectName, 'build', context);
  await runExec(context.projectName, 'pack', context);
  console.log(`✨ publish ${context.projectName}...`);
  await execute(`curl --fail --location --request POST '${options.url}/upload' \
  --form 'file=@${archive}'`);
  return {
    success: true
  };
}

Now you can use these executors in the plugins projects. For this you will have to create new targets inside the Nx project.json configuration.

apps/plugins/plugme/project.json

{
  "name": "plugins-plugme",
  "$schema": "../../../node_modules/nx/schemas/project-schema.json",
  "sourceRoot": "apps/plugins/plugme/src",
  "projectType": "application",
  "targets": {
   ......
    "publish": {
      "executor": "@pluggy/plugin:publish"
    },
    "pack":{
      "executor": "@pluggy/plugin:pack"
    }
  },
  "tags": []
}

The publish executor will build, pack, and publish the plugin. You will only have to run one command:

> cd apps/plugins/plugme
> nx publish

For the creation of a plugin project, we will use an Nx generator. It’s a kind of template engine coupled to workspace functionalities.

 > nx generate @nrwl/nx-plugin:generator create --project=plugin

Our generator reuses the @nwrl/react:remote generator to bootstrap a remote application.

We’ve just added to the ‘project.json’ the pack and publish executor.

import {
  names,
  Tree,
  readProjectConfiguration,
  updateProjectConfiguration
} from '@nrwl/devkit';
import { Schema } from './schema';
import { remoteGenerator } from '@nrwl/react';
import { runTasksInSerial } from '@nrwl/workspace/src/utilities/run-tasks-in-serial';

export function normalizeDirectory(options: Schema) {
  return options.directory
    ? `${names(options.directory).fileName}/${names(options.name).fileName}`
    : names(options.name).fileName;
}

export function normalizeProjectName(options: Schema) {
  return normalizeDirectory(options).replace(new RegExp('/', 'g'), '-');
}

export default async function (tree: Tree, schema: Schema) {

  const remoteTask = await remoteGenerator(tree, schema);
  
  const appName = normalizeProjectName(schema);
  const appConfig = readProjectConfiguration(tree, appName);
  appConfig.targets['pack'] = {
    executor: '@pluggy/plugin:pack' 
  };
  appConfig.targets['publish'] = {
    executor: '@pluggy/plugin:publish' 
  };
  updateProjectConfiguration(tree, appName, {...appConfig});
  
  return runTasksInSerial(remoteTask);
}

Now to generate a new plugin project, you will just have to run:

> nx g @pluggy/plugin:create plugins/my-plugin

Fetching plugins List and displaying it

The last thing that is still remaining is to use and display the list of plugins.

For calling pluggy-locator Apis, we will use redux toolkit (I know it’s too much for a tutorial but I want to use this project for my personal usage also :)

> yarn add react-redux @reduxjs/toolkit

Let’s do some redux toolkit

• Create the Api

import { createApi, fetchBaseQuery } from '@reduxjs/toolkit/query/react'
import {environment} from "../../../environments/environment";

export const pluginsApi = createApi({
  reducerPath: 'pluginsApi',
  baseQuery: fetchBaseQuery({ baseUrl: environment.baseUrl }),
  endpoints: (builder) => ({
    getPlugins: builder.query<string[], void>({
      query: () => `${environment.pluginsUrl}/plugins/`,
    }),
  }),
})

export const { useGetPluginsQuery } = pluginsApi

• Create the Store

import { configureStore } from '@reduxjs/toolkit';
import { pluginsApi } from './plugins/services/plugin';
import { setupListeners } from '@reduxjs/toolkit/query';

export const store = configureStore({
  reducer: {
    [pluginsApi.reducerPath]: pluginsApi.reducer,
  },
  middleware: (getDefaultMiddleware) => getDefaultMiddleware().concat(pluginsApi.middleware),
});

setupListeners(store.dispatch);

// Infer the `RootState` and `AppDispatch` types from the store itself
export type RootState = ReturnType<typeof store.getState>
// Inferred type: {posts: PostsState, comments: CommentsState, users: UsersState}
export type AppDispatch = typeof store.dispatch

• Finally Provide the store context to the App

import { StrictMode } from 'react';
import * as ReactDOM from 'react-dom/client';
import { BrowserRouter } from 'react-router-dom';

import App from './app/app';
import {store} from "./app/store";
import {Provider} from "react-redux";

const root = ReactDOM.createRoot(
  document.getElementById('root') as HTMLElement
);
root.render(
  <StrictMode>
    <BrowserRouter>
      <Provider store={store}>
        <App />
      </Provider>
    </BrowserRouter>
  </StrictMode>
);

Create a component that displays the list of plugins

import {useGetPluginsQuery} from "./services/plugin";

import * as React from 'react';
import {Link, Route, Routes} from 'react-router-dom';
import {DynamicComponent} from "../components/dynamic/DynamicComponent";
import {environment} from "../../environments/environment";


export const PluginsList = () => {
    const { data: plugins, error, isLoading } = useGetPluginsQuery();

    if (isLoading) {
        return <div>Loading...</div>;
    }

    if (error) {
        return <div>error</div>;
    }
  return (
    <React.Suspense fallback={null}>
            <header style={{display:'inline-block'}}>
            <div>
                {plugins?.map((plugin,index) => (
                    <li key={index}>
                        <Link to={plugin}>{plugin}</Link>
                    </li>
                ))}
            </div>
            </header>
            <Routes>
               {plugins?.map((plugin,index) => (
                    <Route path={plugin} element={<DynamicComponent url={`${environment.pluggyLocatorUrl}/plugins/${plugin}/remoteEntry.js`}  scope="plugme" module="./Module"/>} />
                ))}
            </Routes>

    </React.Suspense>
  );
}

You can find the whole code here

Wrap everything

• run the Host application and the server

> nx run-many --target=serve --projects=server-pluggy-locator,host-pluggy
or
> yarn start

• add a new plugin

> nx g @pluggy/plugin:create plugins/my-plugin

• publish the plugin

> cd apps/plugins/my-plugin
> nx publish

• Refresh your browser , you should see your new plugin :)

This project was only to show how to bootstrap a plugin architecture, there are still lot of things to do like:

• Real React UI
• Managing the lifecycle of the plugins
• The communication between core and plugins (ex: through bus event or postmessage)
• refreshing browser client with Websocket when adding new plugin
• Create a SDK in the Core application which plugin can access for adding functionalities in a centralized maner or accessing the core functionalities
• Add unit testing
• Add Swagger for NestJs and verify Apis parameters types
• For production Usage publish plugins on a cloud Storage
• ….

But it’s already a good start :)