问题背景

Astro的Content Layer API很好，但是我接入notion-content-loader后却不断因为图片的问题而报错。

• 报错Cannot
• 本质问题：缓存链接过期导致无法获取图片
  • notion中上传文件都是s3存储，Get链接存在一个过期时间
  • astro中content layer都只会在构建时更新，导致获取的链接在一段时间后过期
  • assets API并不会持久缓存，而是随每次开发服务器启动重建本地资源服务器，这导致资源服务器无法从一个过期链接中获取存储
• 解决方案
  • 使用 pnpm astro sync 或 pnpm dev --force 强制刷新内容层缓存，但需要注意不要太过频繁，否则会触发Notion API的频控
  • 线上不会出现该问题，因为资源服务器会一次性构建完毕

分析处理流程

接下来将以官方的glob-loader来作为演示，整体分为内容处理流程和渲染流程两大板块：

• 内容处理流程：主要负责内容集合（content collection）的 sync 指令，对应获取内容集合的数据信息，生成content类型等，在编译或启动开发服务器之前进行，属于独立的生命周期
• 渲染流程：集成在Vite编译流程中，将md预处理得到的html变成最终的产物html，这里以 build 指令为入口进行探究

渲染流程

• CLI中 build 指令的入口在 cli/build/index.js 中，它负责获取配置，然后调用core中的 build/index.js

import _build from "../../core/build/index.js";

async function build({ flags }) {
  const inlineConfig = flagsToAstroInlineConfig(flags);
  await _build(inlineConfig, { devOutput: !!flags.devOutput });
}

• core/build的核心则是 class AstroBuilder，其执行顺序是 run → setup → build，其中run函数是入口，负责调用其他函数

• setup函数主要做两件事：创建vite服务器和执行 sync 函数更新content-layer，为后面的构建流程做准备

• build函数需要区分static模式和server模式，这里我们只看static模式，不考虑构建node服务的产物

    async build({ viteConfig }) {
      await runHookBuildStart({ config: this.settings.config, logging: this.logger });
      this.validateConfig();
      this.logger.info("build", `output: ${blue('"' + this.settings.config.output + '"')}`);
      this.logger.info("build", `directory: ${blue(fileURLToPath(this.settings.config.outDir))}`);
      if (this.settings.adapter) {
        this.logger.info("build", `adapter: ${green(this.settings.adapter.name)}`);
      }
      this.logger.info("build", "Collecting build info...");
      this.timer.loadStart = performance.now();
      // 1. 收集编译入口
      const { assets, allPages } = collectPagesData({
        settings: this.settings,
        logger: this.logger,
        manifest: this.manifest
      });
      this.logger.debug("build", timerMessage("All pages loaded", this.timer.loadStart));
      const pageNames = [];
      this.timer.buildStart = performance.now();
      this.logger.info(
        "build",
        green(`\\u2713 Completed in ${getTimeStat(this.timer.init, performance.now())}.`)
      );
      const hasKey = hasEnvironmentKey();
      const keyPromise = hasKey ? getEnvironmentKey() : createKey();
      const opts = {
        allPages,
        settings: this.settings,
        logger: this.logger,
        manifest: this.manifest,
        runtimeMode: this.runtimeMode,
        origin: this.origin,
        pageNames,
        teardownCompiler: this.teardownCompiler,
        viteConfig,
        key: keyPromise
      };
      const { internals, ssrOutputChunkNames, ssrOutputAssetNames, contentFileNames } = await viteBuild(opts);
      const hasServerIslands = this.settings.serverIslandNameMap.size > 0;
      if (hasServerIslands && this.settings.buildOutput !== "server") {
        throw new AstroError(AstroErrorData.NoAdapterInstalledServerIslands);
      }
      await staticBuild(opts, internals, ssrOutputChunkNames, ssrOutputAssetNames, contentFileNames);
      this.timer.assetsStart = performance.now();
      Object.keys(assets).map((k) => {
        if (!assets[k]) return;
        const filePath = new URL(`file://${k}`);
        fs.mkdirSync(new URL("./", filePath), { recursive: true });
        fs.writeFileSync(filePath, assets[k], "utf8");
        delete assets[k];
      });
      this.logger.debug("build", timerMessage("Additional assets copied", this.timer.assetsStart));
      await runHookBuildDone({
        settings: this.settings,
        pages: pageNames,
        routes: Object.values(allPages).flat().map((pageData) => pageData.route).concat(hasServerIslands ? getServerIslandRouteData(this.settings.config) : []),
        logging: this.logger
      });
      if (this.logger.level && levels[this.logger.level()] <= levels["info"]) {
        await this.printStats({
          logger: this.logger,
          timeStart: this.timer.init,
          pageCount: pageNames.length,
          buildMode: this.settings.buildOutput
          // buildOutput is always set at this point
        });
      }
    }
  
  

  1. 收集页面入口，即要编译的所有页面

  2. viteBuild：编译出网页的静态资源，主要是js文件

     

  3. staticBuild：负责一件事，生成所有的页面入口html，调用函数 generatePages 来实现

     

  4. generatePages

     // 代码已经删除ssr相关内容
     async function generatePages(options, internals) {
       const generatePagesTimer = performance.now();
       const ssr = options.settings.buildOutput === "server";
       let manifest;
       const baseDirectory = getOutputDirectory(options.settings);
       const renderersEntryUrl = new URL("renderers.mjs", baseDirectory);
       const renderers = await import(renderersEntryUrl.toString());
       const middleware = internals.middlewareEntryPoint ? await import(internals.middlewareEntryPoint.toString()).then((mod) => mod.onRequest) : NOOP_MIDDLEWARE_FN;
       manifest = createBuildManifest(
         options.settings,
         internals,
         // 这个renderers我猜是类似react.render()这样的，ssg模拟ssr渲染
         renderers.renderers,
         middleware,
         options.key
       );
       const pipeline = BuildPipeline.create({ internals, manifest, options });
       const { config, logger } = pipeline;
       const verb = ssr ? "prerendering" : "generating";
       logger.info("SKIP_FORMAT", `
     ${bgGreen(black(` ${verb} static routes `))}`);
       const builtPaths = /* @__PURE__ */ new Set();
       const pagesToGenerate = pipeline.retrieveRoutesToGenerate();
       // 调用generatePage挨个生成每个页面
       for (const [pageData, filePath] of pagesToGenerate) {
         const entry = await pipeline.retrieveSsrEntry(pageData.route, filePath);
         await generatePage(pageData, entry, builtPaths, pipeline);
       }
       logger.info(
         null,
         green(`\\u2713 Completed in ${getTimeStat(generatePagesTimer, performance.now())}.
     `)
       );
       // 调用外部函数处理图片资源
       const staticImageList = getStaticImageList();
       if (staticImageList.size) {
         logger.info("SKIP_FORMAT", `${bgGreen(black(` generating optimized images `))}`);
         const totalCount = Array.from(staticImageList.values()).map((x) => x.transforms.size).reduce((a, b) => a + b, 0);
         const cpuCount = os.cpus().length;
         const assetsCreationPipeline = await prepareAssetsGenerationEnv(pipeline, totalCount);
         const queue = new PQueue({ concurrency: Math.max(cpuCount, 1) });
         const assetsTimer = performance.now();
         for (const [originalPath, transforms] of staticImageList) {
           await generateImagesForPath(originalPath, transforms, assetsCreationPipeline, queue);
         }
         await queue.onIdle();
         const assetsTimeEnd = performance.now();
         logger.info(null, green(`\\u2713 Completed in ${getTimeStat(assetsTimer, assetsTimeEnd)}.
     `));
         delete globalThis?.astroAsset?.addStaticImage;
       }
       await runHookBuildGenerated({ settings: options.settings, logger });
     }
     
     

  5. generatePage：生成单个页面，每个页面可能会有多个path，如动态参数、重定向配置等等，因此generatePage会生成path配置，并为每个path调用generate函数

  6. generatePathWithLogs和generatePath：生成每个path，每个path都有一个自己的 renderContext ，这些route/path的类型定义如下

     /**
      * - page: a route that lives in the file system, usually an Astro component
      * - endpoint: a route that lives in the file system, usually a JS file that exposes endpoints methods
      * - redirect: a route points to another route that lives in the file system
      * - fallback: a route that doesn't exist in the file system that needs to be handled with other means, usually the middleware
      */
     export type RouteType = 'page' | 'endpoint' | 'redirect' | 'fallback';
     

  7. RenderContext：位于 core/render-context.js，简单来说就是为每个path的渲染模拟成一次在服务端渲染的请求，用于处理数据获取、重定向等逻辑，重点关注其中的render函数

     1. 在middleware的最后（lastNext）进行渲染
     2. routeType进行渲染，重点关注page

       async render(componentInstance, slots = {}) {
         const { cookies, middleware, pipeline } = this;
         const { logger, serverLike, streaming, manifest } = pipeline;
         const props = Object.keys(this.props).length > 0 ? this.props : await getProps({
           mod: componentInstance,
           routeData: this.routeData,
           routeCache: this.pipeline.routeCache,
           pathname: this.pathname,
           logger,
           serverLike,
           base: manifest.base
         });
         const apiContext = this.createAPIContext(props);
         this.counter++;
         if (this.counter === 4) {
           return new Response("Loop Detected", {
             // <https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/508>
             status: 508,
             statusText: "Astro detected a loop where you tried to call the rewriting logic more than four times."
           });
         }
         const lastNext = async (ctx, payload) => {
           if (payload) {
     	      // 处理rewrite逻辑，略
           }
           let response2;
           switch (this.routeData.type) {
             case "endpoint": {
               response2 = await renderEndpoint(
                 componentInstance,
                 ctx,
                 this.routeData.prerender,
                 logger
               );
               break;
             }
             case "redirect":
               return renderRedirect(this);
             // 重点关注page的情况
             case "page": {
               const result = await this.createResult(componentInstance);
               try {
                 response2 = await renderPage(
                   result,
                   componentInstance?.default,
                   props,
                   slots,
                   streaming,
                   this.routeData
                 );
               } catch (e) {
                 result.cancelled = true;
                 throw e;
               }
               response2.headers.set(ROUTE_TYPE_HEADER, "page");
               if (this.routeData.route === "/404" || this.routeData.route === "/500") {
                 response2.headers.set(REROUTE_DIRECTIVE_HEADER, "no");
               }
               if (this.isRewriting) {
                 response2.headers.set(REWRITE_DIRECTIVE_HEADER_KEY, REWRITE_DIRECTIVE_HEADER_VALUE);
               }
               break;
             }
             case "fallback": {
               return new Response(null, { status: 500, headers: { [ROUTE_TYPE_HEADER]: "fallback" } });
             }
           }
           const responseCookies = getCookiesFromResponse(response2);
           if (responseCookies) {
             cookies.merge(responseCookies);
           }
           return response2;
         };
         const response = await callMiddleware(middleware, apiContext, lastNext);
         if (response.headers.get(ROUTE_TYPE_HEADER)) {
           response.headers.delete(ROUTE_TYPE_HEADER);
         }
         attachCookiesToResponse(response, cookies);
         return response;
       }
     

内容处理流程

astro到底是怎么神出鬼没地在content layer的生命周期里用glob loader处理好md里的图片的？