Use Zip to hurry up CocosWeb loading – Cocos Creator

Use Zip to hurry up CocosWeb loading

A while in the past, I used Cocos3.8 to do a cloud showroom mission that required publish to the Internet platform (WeChat H5 & browsers).

This mission makes use of gltf fashions, the gltf fashions are cut up into many meshes and supplies.

In Cocos, gltf is parsed and cut up to Cocos property. After publishing to the Internet, loading reminiscent of this gltf will use a whole lot of community requests. Regardless of having quick web pace, the loading continues to be sluggish as a result of there’s to many community requests.

• Create a brand new mission, take a gltf from a ThreeJS demo, place it straight within the assets folder for simple preloading within the demo. This gltf comprises a complete of 28 supplies, 32 meshes, and a few bones & textures.

• Create a Begin scene for preloading assets, and a Sport scene for displaying the mannequin.

• Create a Begin script to carry out a easy preloading of assets, when loaded then change to the Sport scene.

import {_decorator, Part, director, Label, settings, ProgressBar, assets, assetManager, Settings} from 'cc';

const {ccclass, property} = _decorator;

@ccclass('Begin')
export class Begin extends Part {

    @property(ProgressBar)
    progressBar: ProgressBar;

    @property(Label)
    barLab: Label = null;

    async begin() {
    
        // Load the assets root listing
        await this.preload([
            {
                path: "https://forum.cocosengine.org/",
                type: "dir",
            },
        ]);

        director.loadScene("Sport");

    }

    /**
     * Preloaded useful resource
     */
    preload = (pkg) => {
        return new Promise((resolve, reject) => {
            const pathArr = [];

            pkg.forEach((asset) => {
                if (typeof asset == "string") {
                    return pathArr.push(asset);
                }
                change (asset.kind) {
                    case "dir":
                        assets.getDirWithPath(asset.path).forEach((v) => pathArr.push(v.path));
                        break;
                        
                    default:
                        pathArr.push(asset.path);
                }
            });

            assets.load(
                pathArr,
                (end: quantity, whole: quantity, _) => {
                    const professional = end / whole;
                    if (professional  {
                    if (error) console.error(error);
                    resolve();
                }
            );
        });
    }
}

• Launch sport and checking the Community Panel within the localhost, the variety of community requests is 379, with 216 associated to the gltf. Packaged and printed to the Internet with all JSON recordsdata merged, load the gltf solely required 35 community requests.

localhost

Packaged & merged JSON

• Now we have just one gltf, nevertheless it required 35 community requests to load.

• The reason being that Cocos converts gltf assets to Cocos property, disintegrates Mesh, supplies, and many others., and every useful resource has a Json file that data attribute dependencies along with the useful resource itself

• Package deal your complete bundle, reminiscent of packaging it into a zipper file. Within the sport, load the required bundle’s zip file for decompression. Subsequently, retrieve assets straight from the decompressed file.

No want for additional clarification since everybody is probably going accustomed to it.

Refer on to the npm platform documentation for JsZip.

• By inspecting the Community, it may be noticed that Cocos downloads assets by way of a download-file. ts file, which may be traced by shifting the mouse over download-file. ts to view its operate name stack. The primary elements concerned are download-file. ts and downloader. ts, that are a part of the useful resource obtain pipeline. Then opening the supply code, we will delve into this course of.

• As you may see within the code, most recordsdata are downloaded utilizing the downloadFile operate, which is the operate in download-file.ts, which makes use of XMLHttpRequest to obtain the file

• The present apply in signifies that almost all of useful resource downloads depend on XMLHttpRequest in Cocos, To optimize this course of and decrease time consumption, we will intercept the requests and redirect them to the corresponding zip bundle.

• Create a ZipLoader.ts and use it as a singleton.
• The zip is loaded straight utilizing the Cocos built-in API
• Use Promise along with exterior async/await to simplify the management movement.

import {assetManager} from "cc";
import JSZIP from "jszip";

export default class ZipLoader {

    static _ins: ZipLoader;
    static get ins() {
        if (!this._ins) {
            this._ins = new ZipLoader();
        }
        return this._ins;
    }

    /**
     * Obtain a single zip file as buffer
     * Why is there a '.zip' suffix right here that we'll speak about later, it is for automation
     * @param path filePath
     * @returns buffer
     */
    downloadZip(path: string) {
        return new Promise((resolve) => {
            assetManager.downloader.downloadFile(
                path + '.zip',
                {xhrResponseType: "arraybuffer"},
                null,
                (err, information) => {
                    resolve(information);
                }
            );
        });
    }

    /**
     * load and unzip file
     * @param path filePath
     */
    async loadZip(path: string) {
        const jsZip = JSZIP();

        // donwload
        const zipBuffer = await this.downloadZip(path);

        // unzip
        const zipFile = await jsZip.loadAsync(zipBuffer);
    }
}

• Add code to Begin.ts earlier than
• Notice the next factors
• I take advantage of computerized compression add plug-in, will auto modify the server area, server is the mission printed root listing with protocol and area identify, reminiscent of https://xxx.com/cc_project/model/
• The plug-in will inject the bundle that requires zip loading into the window
• Equivalent to window["zipBundle"] = ["internal", "main", "resources"];
• All of the bundles listed below are distant so simply load the recordsdata in distant and the zip file is in the identical listing because the bundle folder

/* ... */

@ccclass('Begin')
export class Begin extends Part {

    /* ... */
    
    async begin() {

        // computerized compression add plug-in, will auto modify the server area
        // and inject the bundle that requires zip loading into the window
        // Equivalent to
        // window["zipBundle"] = ["internal", "main", "resources"];
        const remoteUrl = settings.querySettings(Settings.Class.ASSETS, "server");
        const zipBundle = window["zipBundle"] || [];

        // All of the bundles listed below are distant so simply load the recordsdata in distant and the zip file is in the identical listing because the bundle folder
        // The zip file and bundle folder are in the identical listing
        const loadZipPs = zipBundle.map((identify: string) => {
            return ZipLoader.ins.loadZip(`${remoteUrl}distant/${identify}`);
        });
        
        // wait zip loaded
        await Promise.all(loadZipPs);

        // load assets dir
        await this.preload([
            {
                path: "https://forum.cocosengine.org/",
                type: "dir",
            },
        ]);

        director.loadScene("Sport");

    }
    
    /* ... */
    
}

After referring to the Cocos paperwork, there is no such thing as a environment friendly technique obtainable to attain this requirement in batches, and since the Cocos engine is up to date regularly, I personally like to make use of new engines and new features, so a private choice to not customise the engine, and straight undertake the strategy of intercepting Cocos loading to exchange the loaded assets with my very own zip bundle.

The examination of the supply code aside from the picture useful resource, different assets are loaded by way of the XMLHttpRequest, Subsequently, it’s a simple course of, we straight intercept the XMLHttpRequest on the road straight.

So that you ask me how one can intercepting a browser Native object, that is JavaScript, JavaScript can do something!

• With out additional ado, one can intercept an XMLHttpRequest utilizing the next technique to carry out sure operations:

// Intercepting 'open'
const oldOpen = XMLHttpRequest.prototype.open;
// @ts-ignore
XMLHttpRequest.prototype.open = operate (technique, url, async, person, password) {
  return oldOpen.apply(this, arguments);
}

// Intercepting 'ship'
const oldSend = XMLHttpRequest.prototype.ship;
XMLHttpRequest.prototype.ship = async operate (information) {
  return oldSend.apply(this, arguments);
}

• Including code for parsing zip recordsdata, extracting code from the zip to finish paths:

/* ... */;

const ZipCache = new Map();

export default class ZipLoader {

    /* ... */

    constructor() {
        this.init();
    }

    /* ... */

    /**
     * Loading and parsing Zip recordsdata
     * @param path filaPath
     */
    async loadZip(path: string) {

        const jsZip = JSZIP();

        const zipBuffer = await this.downloadZip(path);

        const zipFile = await jsZip.loadAsync(zipBuffer);
        // Parsing the zip file, concatenating paths, bundle names, and file names, and storing them straight in a Map
        zipFile.forEach((v, t) => {
            if (t.dir) return;
            ZipCache.set(path + "https://discussion board.cocosengine.org/" + v, t);
        });
    }
    
    init() {
        // Intercepting 'open'
        const oldOpen = XMLHttpRequest.prototype.open;
        // @ts-ignore
        XMLHttpRequest.prototype.open = operate (technique, url, async, person, password) {
            return oldOpen.apply(this, arguments);
        }

        // Intercepting 'ship'
        const oldSend = XMLHttpRequest.prototype.ship;
        XMLHttpRequest.prototype.ship = async operate (information) {
            return oldSend.apply(this, arguments);
        }
    }

• Inside the intercepted open and ship features, cancel community requests and redirect them to cached zip assets.
• Because the response property of XMLHttpRequest is read-only, we make the most of Object.getOwnPropertyDescriptor and Object.defineProperty.
• Let’s dive straight into the code:

/* ... */

const ZipCache = new Map();
const ResCache = new Map();

export default class ZipLoader {

    /* ... */

    constructor() {
        this.init();
    }

    /* ... */

    init() {

        const accessor = Object.getOwnPropertyDescriptor(XMLHttpRequest.prototype, 'response');
        Object.defineProperty(XMLHttpRequest.prototype, 'response', {
            get: operate () {
                if (this.zipCacheUrl) {
                    const res = ResCache.get(this.zipCacheUrl);
                    return this.responseType === "json"
                        ? JSON.parse(res)
                        : res;
                }
                return accessor.get.name(this);
            },
            set: operate (str) {
                // console.log('set responseText: %s', str);
                // return accessor.set.name(this, str);
            },
            configurable: true
        });

        // Intercepting 'open'
        const oldOpen = XMLHttpRequest.prototype.open;
        // @ts-ignore
        XMLHttpRequest.prototype.open = operate (technique, url, async, person, password) {
            // File useful resource if it exists
            if (ZipCache.has(url as string)) {
                this.zipCacheUrl = url;
            }
            return oldOpen.apply(this, arguments);
        }

        // Intercepting 'ship'
        const oldSend = XMLHttpRequest.prototype.ship;
        XMLHttpRequest.prototype.ship = async operate (information) {
            if (this.zipCacheUrl) {
                // Skip parsing if cached
                if (!ResCache.has(this.zipCacheUrl)) {

                    const cache = ZipCache.get(this.zipCacheUrl);

                    if (this.responseType === "json") {
                        const textual content = await cache.async("textual content");
                        ResCache.set(this.zipCacheUrl, textual content);
                    } else {
                        // Parsing utilizing cocos set responseType straight for zip
                        const res = await cache.async(this.responseType);
                        ResCache.set(this.zipCacheUrl, res);
                    }
                }

                // Name onload after parsing and keep away from making actual community requests
                this.onload();
                return;
            }

            return oldSend.apply(this, arguments);
        }
    }
}

• Bundling the mission and manually compressing the bundle folder for testing reveals three downloaded zip assets, with quite a few JSON and binary folders not downloaded.
• Solely two texture pictures have been obtained for testing GLTF-related recordsdata, permitting profitable entry into the Sport scene.
• This demonstrates the effectiveness of the sooner code modifications, lowering the variety of requests for loading the GLTF file from 35 instances to three instances.

• Creating a Cocos plugin for bundling and mechanically compressing bundle recordsdata into zip codecs is an easy endeavor.
• Merely create a constructing plugin and script file, as proven under:

import * as fs from "fs";
import JSZIP from "jszip";

//Learn directories and recordsdata
operate readDir(zip, nowPath) {
    const recordsdata = fs.readdirSync(nowPath);
    recordsdata.forEach(operate (fileName, index) {
        console.log(fileName, index);
        const fillPath = nowPath + "https://discussion board.cocosengine.org/" + fileName;
        const file = fs.statSync(fillPath);
        if (file.isDirectory()) {
            const dirlist = zip.folder(fileName);
            readDir(dirlist, fillPath);
        } else {
            // Exclude picture recordsdata, as defined under
            if (fileName.endsWith(".png") || fileName.endsWith(".jpg")) {
                return;
            }
            zip.file(fileName, fs.readFileSync(fillPath));//压缩目录添加文件
        }
    });
}

// Provoke file compression
export operate zipDir(identify, dir, dist) {
    return new Promise((resolve, reject) => {
        const zip = new JSZIP();
        readDir(zip, dir);
        zip.generateAsync({
            kind: "nodebuffer",
            compression: "DEFLATE",
            compressionOptions: {
                stage: 9
            }
        }).then(operate (content material) {
            fs.writeFileSync(`${dist}/${identify}.zip`, content material, "utf-8");
            resolve();
        });
    });
}

• Within the hooks onAfterBuild part
• insert the compression script content material to run put up different operations and previous to useful resource add
• significantly for internet templates:

export const onAfterBuild: BuildHook.onAfterBuild = async operate (choices: ITaskOptions, outcome: IBuildResult) {

    // Carry out operation solely on particular templates
    if (choices.platform !== "web-mobile") return;

    // Modify scripts, obfuscate code, compress assets, and many others.
    / ... /
    if (fs.existsSync(outcome.dest + "/distant")) {
        await Promise.all(
            fs.readdirSync(outcome.dest + "/distant")
                .map((dirName) => {
                    return zipDir(dirName, outcome.dest + "/distant/" + dirName, outcome.dest + "/distant");
                })
        )
    }
    / ... /

    // Add

};

Upon inspecting the supply code and using the Community device,
it was famous that Cocos engine hundreds pictures by creating Picture objects as a substitute of utilizing XMLHttpRequest.
have avoided delving into changing picture loading with customized zipping strategies for now,
as no such implementation has been carried out but.

In consequence, throughout zip packaging, PNG/JPG recordsdata are filtered out to scale back the scale of the zip bundle,
containing solely mandatory recordsdata throughout the zip.