Use Zip to hurry up CocosWeb loading – Cocos Creator

Use Zip to hurry up CocosWeb loading

A while in the past, I used Cocos3.8 to do a cloud showroom mission that required publish to the Internet platform (WeChat H5 & browsers).

This mission makes use of gltf fashions, the gltf fashions are cut up into many meshes and supplies.

In Cocos, gltf is parsed and cut up to Cocos property. After publishing to the Internet, loading equivalent to this gltf will use lots of of community requests. Regardless of having quick web velocity, the loading remains to be sluggish as a result of there’s to many community requests.

• Create a brand new mission, take a gltf from a ThreeJS demo, place it straight within the sources folder for simple preloading within the demo. This gltf accommodates a complete of 28 supplies, 32 meshes, and a few bones & textures.

• Create a Begin scene for preloading sources, and a Sport scene for displaying the mannequin.

• Create a Begin script to carry out a easy preloading of sources, when loaded then change to the Sport scene.

import {_decorator, Part, director, Label, settings, ProgressBar, sources, assetManager, Settings} from 'cc';

const {ccclass, property} = _decorator;

@ccclass('Begin')
export class Begin extends Part {

    @property(ProgressBar)
    progressBar: ProgressBar;

    @property(Label)
    barLab: Label = null;

    async begin() {
    
        // Load the sources root listing
        await this.preload([
            {
                path: "https://forum.cocosengine.org/",
                type: "dir",
            },
        ]);

        director.loadScene("Sport");

    }

    /**
     * Preloaded useful resource
     */
    preload = (pkg) => {
        return new Promise((resolve, reject) => {
            const pathArr = [];

            pkg.forEach((asset) => {
                if (typeof asset == "string") {
                    return pathArr.push(asset);
                }
                change (asset.sort) {
                    case "dir":
                        sources.getDirWithPath(asset.path).forEach((v) => pathArr.push(v.path));
                        break;
                        
                    default:
                        pathArr.push(asset.path);
                }
            });

            sources.load(
                pathArr,
                (end: quantity, whole: quantity, _) => {
                    const professional = end / whole;
                    if (professional  {
                    if (error) console.error(error);
                    resolve();
                }
            );
        });
    }
}

• Launch recreation and checking the Community Panel within the localhost, the variety of community requests is 379, with 216 associated to the gltf. Packaged and revealed to the Internet with all JSON recordsdata merged, load the gltf solely required 35 community requests.

localhost

Packaged & merged JSON

• Now we have just one gltf, however it required 35 community requests to load.

• The reason being that Cocos converts gltf sources to Cocos property, disintegrates Mesh, supplies, and so forth., and every useful resource has a Json file that data attribute dependencies along with the useful resource itself

• Bundle your complete bundle, equivalent to packaging it into a zipper file. Within the recreation, load the required bundle’s zip file for decompression. Subsequently, retrieve sources straight from the decompressed file.

No want for additional clarification since everybody is probably going accustomed to it.

Refer on to the npm platform documentation for JsZip.

• By inspecting the Community, it may be noticed that Cocos downloads sources by means of a download-file. ts file, which might be traced by transferring the mouse over download-file. ts to view its perform name stack. The principle elements concerned are download-file. ts and downloader. ts, that are a part of the useful resource obtain pipeline. Then opening the supply code, we will delve into this course of.

• As you possibly can see within the code, most recordsdata are downloaded utilizing the downloadFile perform, which is the perform in download-file.ts, which makes use of XMLHttpRequest to obtain the file

• The present follow in signifies that almost all of useful resource downloads depend on XMLHttpRequest in Cocos, To optimize this course of and reduce time consumption, we will intercept the requests and redirect them to the corresponding zip package deal.

• Create a ZipLoader.ts and use it as a singleton.
• The zip is loaded straight utilizing the Cocos built-in API
• Use Promise at the side of exterior async/await to simplify the management circulate.

import {assetManager} from "cc";
import JSZIP from "jszip";

export default class ZipLoader {

    static _ins: ZipLoader;
    static get ins() {
        if (!this._ins) {
            this._ins = new ZipLoader();
        }
        return this._ins;
    }

    /**
     * Obtain a single zip file as buffer
     * Why is there a '.zip' suffix right here that we'll discuss later, it is for automation
     * @param path filePath
     * @returns buffer
     */
    downloadZip(path: string) {
        return new Promise((resolve) => {
            assetManager.downloader.downloadFile(
                path + '.zip',
                {xhrResponseType: "arraybuffer"},
                null,
                (err, knowledge) => {
                    resolve(knowledge);
                }
            );
        });
    }

    /**
     * load and unzip file
     * @param path filePath
     */
    async loadZip(path: string) {
        const jsZip = JSZIP();

        // donwload
        const zipBuffer = await this.downloadZip(path);

        // unzip
        const zipFile = await jsZip.loadAsync(zipBuffer);
    }
}

• Add code to Begin.ts earlier than
• Word the next factors
• I take advantage of automated compression add plug-in, will auto modify the server area, server is the mission revealed root listing with protocol and area title, equivalent to https://xxx.com/cc_project/model/
• The plug-in will inject the package deal that requires zip loading into the window
• Resembling window["zipBundle"] = ["internal", "main", "resources"];
• All of the bundles listed here are distant so simply load the recordsdata in distant and the zip file is in the identical listing because the bundle folder

/* ... */

@ccclass('Begin')
export class Begin extends Part {

    /* ... */
    
    async begin() {

        // automated compression add plug-in, will auto modify the server area
        // and inject the package deal that requires zip loading into the window
        // Resembling
        // window["zipBundle"] = ["internal", "main", "resources"];
        const remoteUrl = settings.querySettings(Settings.Class.ASSETS, "server");
        const zipBundle = window["zipBundle"] || [];

        // All of the bundles listed here are distant so simply load the recordsdata in distant and the zip file is in the identical listing because the bundle folder
        // The zip file and bundle folder are in the identical listing
        const loadZipPs = zipBundle.map((title: string) => {
            return ZipLoader.ins.loadZip(`${remoteUrl}distant/${title}`);
        });
        
        // wait zip loaded
        await Promise.all(loadZipPs);

        // load sources dir
        await this.preload([
            {
                path: "https://forum.cocosengine.org/",
                type: "dir",
            },
        ]);

        director.loadScene("Sport");

    }
    
    /* ... */
    
}

After referring to the Cocos paperwork, there isn’t any environment friendly technique accessible to realize this requirement in batches, and since the Cocos engine is up to date steadily, I personally like to make use of new engines and new features, so a private resolution to not customise the engine, and straight undertake the strategy of intercepting Cocos loading to exchange the loaded sources with my very own zip package deal.

The examination of the supply code aside from the picture useful resource, different sources are loaded by means of the XMLHttpRequest, Due to this fact, it’s a easy course of, we straight intercept the XMLHttpRequest on the road straight.

So that you ask me the way to intercepting a browser Native object, that is JavaScript, JavaScript can do something!

• With out additional ado, one can intercept an XMLHttpRequest utilizing the next technique to carry out sure operations:

// Intercepting 'open'
const oldOpen = XMLHttpRequest.prototype.open;
// @ts-ignore
XMLHttpRequest.prototype.open = perform (technique, url, async, consumer, password) {
  return oldOpen.apply(this, arguments);
}

// Intercepting 'ship'
const oldSend = XMLHttpRequest.prototype.ship;
XMLHttpRequest.prototype.ship = async perform (knowledge) {
  return oldSend.apply(this, arguments);
}

• Including code for parsing zip recordsdata, extracting code from the zip to finish paths:

/* ... */;

const ZipCache = new Map();

export default class ZipLoader {

    /* ... */

    constructor() {
        this.init();
    }

    /* ... */

    /**
     * Loading and parsing Zip recordsdata
     * @param path filaPath
     */
    async loadZip(path: string) {

        const jsZip = JSZIP();

        const zipBuffer = await this.downloadZip(path);

        const zipFile = await jsZip.loadAsync(zipBuffer);
        // Parsing the zip file, concatenating paths, bundle names, and file names, and storing them straight in a Map
        zipFile.forEach((v, t) => {
            if (t.dir) return;
            ZipCache.set(path + "https://discussion board.cocosengine.org/" + v, t);
        });
    }
    
    init() {
        // Intercepting 'open'
        const oldOpen = XMLHttpRequest.prototype.open;
        // @ts-ignore
        XMLHttpRequest.prototype.open = perform (technique, url, async, consumer, password) {
            return oldOpen.apply(this, arguments);
        }

        // Intercepting 'ship'
        const oldSend = XMLHttpRequest.prototype.ship;
        XMLHttpRequest.prototype.ship = async perform (knowledge) {
            return oldSend.apply(this, arguments);
        }
    }

• Inside the intercepted open and ship features, cancel community requests and redirect them to cached zip sources.
• Because the response property of XMLHttpRequest is read-only, we make the most of Object.getOwnPropertyDescriptor and Object.defineProperty.
• Let’s dive straight into the code:

/* ... */

const ZipCache = new Map();
const ResCache = new Map();

export default class ZipLoader {

    /* ... */

    constructor() {
        this.init();
    }

    /* ... */

    init() {

        const accessor = Object.getOwnPropertyDescriptor(XMLHttpRequest.prototype, 'response');
        Object.defineProperty(XMLHttpRequest.prototype, 'response', {
            get: perform () {
                if (this.zipCacheUrl) {
                    const res = ResCache.get(this.zipCacheUrl);
                    return this.responseType === "json"
                        ? JSON.parse(res)
                        : res;
                }
                return accessor.get.name(this);
            },
            set: perform (str) {
                // console.log('set responseText: %s', str);
                // return accessor.set.name(this, str);
            },
            configurable: true
        });

        // Intercepting 'open'
        const oldOpen = XMLHttpRequest.prototype.open;
        // @ts-ignore
        XMLHttpRequest.prototype.open = perform (technique, url, async, consumer, password) {
            // Document useful resource if it exists
            if (ZipCache.has(url as string)) {
                this.zipCacheUrl = url;
            }
            return oldOpen.apply(this, arguments);
        }

        // Intercepting 'ship'
        const oldSend = XMLHttpRequest.prototype.ship;
        XMLHttpRequest.prototype.ship = async perform (knowledge) {
            if (this.zipCacheUrl) {
                // Skip parsing if cached
                if (!ResCache.has(this.zipCacheUrl)) {

                    const cache = ZipCache.get(this.zipCacheUrl);

                    if (this.responseType === "json") {
                        const textual content = await cache.async("textual content");
                        ResCache.set(this.zipCacheUrl, textual content);
                    } else {
                        // Parsing utilizing cocos set responseType straight for zip
                        const res = await cache.async(this.responseType);
                        ResCache.set(this.zipCacheUrl, res);
                    }
                }

                // Name onload after parsing and keep away from making actual community requests
                this.onload();
                return;
            }

            return oldSend.apply(this, arguments);
        }
    }
}

• Bundling the mission and manually compressing the bundle folder for testing reveals three downloaded zip sources, with quite a few JSON and binary folders not downloaded.
• Solely two texture photographs have been obtained for testing GLTF-related recordsdata, permitting profitable entry into the Sport scene.
• This demonstrates the effectiveness of the sooner code modifications, lowering the variety of requests for loading the GLTF file from 35 occasions to three occasions.

• Growing a Cocos plugin for bundling and routinely compressing bundle recordsdata into zip codecs is a simple endeavor.
• Merely create a constructing plugin and script file, as proven beneath:

import * as fs from "fs";
import JSZIP from "jszip";

//Learn directories and recordsdata
perform readDir(zip, nowPath) {
    const recordsdata = fs.readdirSync(nowPath);
    recordsdata.forEach(perform (fileName, index) {
        console.log(fileName, index);
        const fillPath = nowPath + "https://discussion board.cocosengine.org/" + fileName;
        const file = fs.statSync(fillPath);
        if (file.isDirectory()) {
            const dirlist = zip.folder(fileName);
            readDir(dirlist, fillPath);
        } else {
            // Exclude picture recordsdata, as defined beneath
            if (fileName.endsWith(".png") || fileName.endsWith(".jpg")) {
                return;
            }
            zip.file(fileName, fs.readFileSync(fillPath));//压缩目录添加文件
        }
    });
}

// Provoke file compression
export perform zipDir(title, dir, dist) {
    return new Promise((resolve, reject) => {
        const zip = new JSZIP();
        readDir(zip, dir);
        zip.generateAsync({
            sort: "nodebuffer",
            compression: "DEFLATE",
            compressionOptions: {
                degree: 9
            }
        }).then(perform (content material) {
            fs.writeFileSync(`${dist}/${title}.zip`, content material, "utf-8");
            resolve();
        });
    });
}

• Within the hooks onAfterBuild part
• insert the compression script content material to run put up different operations and previous to useful resource add
• significantly for net templates:

export const onAfterBuild: BuildHook.onAfterBuild = async perform (choices: ITaskOptions, end result: IBuildResult) {

    // Carry out operation solely on particular templates
    if (choices.platform !== "web-mobile") return;

    // Modify scripts, obfuscate code, compress sources, and so forth.
    / ... /
    if (fs.existsSync(end result.dest + "/distant")) {
        await Promise.all(
            fs.readdirSync(end result.dest + "/distant")
                .map((dirName) => {
                    return zipDir(dirName, end result.dest + "/distant/" + dirName, end result.dest + "/distant");
                })
        )
    }
    / ... /

    // Add

};

Upon inspecting the supply code and using the Community device,
it was famous that Cocos engine masses photographs by creating Picture objects as an alternative of utilizing XMLHttpRequest.
have kept away from delving into changing picture loading with customized zipping strategies for now,
as no such implementation has been carried out but.

In consequence, throughout zip packaging, PNG/JPG recordsdata are filtered out to scale back the scale of the zip package deal,
containing solely vital recordsdata throughout the zip.