Sitemap Optimization for Agentic Web Portal — September 23, 2026

Today, I focused on optimizing the sitemap for my agentic web portal, ensuring it adheres to Google's URL limits while also improving the overall performance of the platform. In about three hours, I completed six commits that addressed critical issues with URL sharding, allowing for better management of the sitemap and enhancing its effectiveness in search engine indexing. This work is crucial for my goal of building a one-man-show company with a billion-dollar valuation, as a well-optimized sitemap is a foundational element for achieving high visibility in search results.

The first major change I implemented was sharding the sitemap by URL count rather than by combination count. Previously, the system was set to use dense combinations, which caused some shards to overflow the 50k URL limit imposed by Google. I adjusted the shard configurations to ensure that no single file exceeds this limit, which is vital for maintaining compliance and avoiding errors in Google Search Console. This adjustment involved reducing the number of combinations from 150 to 90, and then further down to 40 as I refined the parameters. It’s a bit of a balancing act to find the sweet spot, but after a few iterations, I feel confident that I’ve struck the right balance.

One of the more tedious aspects of this optimization was dealing with the stale shard 404 errors. These occurred when the system attempted to access shards that were out of range due to the previous combination settings. I had to ensure that any out-of-range shard would return a proper error message instead of a 404, improving the user experience and reducing confusion. This involved modifying the logic in the local/sitemap.ts file to handle these cases more gracefully, which took some time to test and verify.

Another significant change was in the marketplace area of the platform. I switched the rendering behavior to a dynamic approach, which means that the off-box build will no longer prerender this section. Instead, it will fetch the necessary data at request time. This is crucial because the marketplace relies on both database queries and plugin manifests that must be available in real time rather than at build time. This change not only enhances performance but also ensures that users always see the most up-to-date information when they access the marketplace.

Throughout this process, I had to stay vigilant about the overall architecture of the sitemap and its interaction with other components of the platform. I spent a lot of time making sure that everything was well-integrated and that the changes I made did not inadvertently break any existing functionality. It’s a constant reminder of how interconnected everything is in a complex system like this. Each small change can have ripple effects that need careful consideration.

Reflecting on the day, I’m proud of the progress I made. Each commit felt like a step closer to a more robust and efficient platform. I’m building this almost entirely solo, relying heavily on AI tools like Claude Code in VS Code to assist with coding and debugging. While it sometimes feels overwhelming to manage all these aspects alone, having AI as my 'team' helps alleviate some of the workload. I’m committed to building a platform that stands out in the crowded agentic web space, and today’s improvements are a solid step in that direction.

Looking ahead, I’ll need to continue monitoring the sitemap’s performance and ensure that it scales effectively as the number of URLs grows. I also plan to run some tests to see how these changes impact search engine visibility and user experience. As I drive this project forward, each optimization brings me closer to my vision of a billion-dollar valuation for this one-man-show company. It’s a journey filled with challenges and learning experiences, but the incremental progress keeps me motivated to push through the grind.