#AWS Analytics | Explore Tumblr posts and blogs

jcmarchi · 1 day ago

Text

WordPress Multi-Multisite: A Case Study

New Post has been published on https://thedigitalinsider.com/wordpress-multi-multisite-a-case-study/

WordPress Multi-Multisite: A Case Study

The mission: Provide a dashboard within the WordPress admin area for browsing Google Analytics data for all your blogs.

The catch? You’ve got about 900 live blogs, spread across about 25 WordPress multisite instances. Some instances have just one blog, others have as many as 250. In other words, what you need is to compress a data set that normally takes a very long time to compile into a single user-friendly screen.

The implementation details are entirely up to you, but the final result should look like this Figma comp:

Design courtesy of the incomparable Brian Biddle.

I want to walk you through my approach and some of the interesting challenges I faced coming up with it, as well as the occasional nitty-gritty detail in between. I’ll cover topics like the WordPress REST API, choosing between a JavaScript or PHP approach, rate/time limits in production web environments, security, custom database design — and even a touch of AI. But first, a little orientation.

Let’s define some terms

We’re about to cover a lot of ground, so it’s worth spending a couple of moments reviewing some key terms we’ll be using throughout this post.

What is WordPress multisite?

WordPress Multisite is a feature of WordPress core — no plugins required — whereby you can run multiple blogs (or websites, or stores, or what have you) from a single WordPress installation. All the blogs share the same WordPress core files, wp-content folder, and MySQL database. However, each blog gets its own folder within wp-content/uploads for its uploaded media, and its own set of database tables for its posts, categories, options, etc. Users can be members of some or all blogs within the multisite installation.

What is WordPress multi-multisite?

It’s just a nickname for managing multiple instances of WordPress multisite. It can get messy to have different customers share one multisite instance, so I prefer to break it up so that each customer has their own multisite, but they can have many blogs within their multisite.

So that’s different from a “Network of Networks”?

It’s apparently possible to run multiple instances of WordPress multisite against the same WordPress core installation. I’ve never looked into this, but I recall hearing about it over the years. I’ve heard the term “Network of Networks” and I like it, but that is not the scenario I’m covering in this article.

Why do you keep saying “blogs”? Do people still blog?

You betcha! And people read them, too. You’re reading one right now. Hence, the need for a robust analytics solution. But this article could just as easily be about any sort of WordPress site. I happen to be dealing with blogs, and the word “blog” is a concise way to express “a subsite within a WordPress multisite instance”.

One more thing: In this article, I’ll use the term dashboard site to refer to the site from which I observe the compiled analytics data. I’ll use the term client sites to refer to the 25 multisites I pull data from.

My implementation

My strategy was to write one WordPress plugin that is installed on all 25 client sites, as well as on the dashboard site. The plugin serves two purposes:

Expose data at API endpoints of the client sites

Scrape the data from the client sites from the dashboard site, cache it in the database, and display it in a dashboard.

The WordPress REST API is the Backbone

The WordPress REST API is my favorite part of WordPress. Out of the box, WordPress exposes default WordPress stuff like posts, authors, comments, media files, etc., via the WordPress REST API. You can see an example of this by navigating to /wp-json from any WordPress site, including CSS-Tricks. Here’s the REST API root for the WordPress Developer Resources site:

The root URL for the WordPress REST API exposes structured JSON data, such as this example from the WordPress Developer Resources website.

What’s so great about this? WordPress ships with everything developers need to extend the WordPress REST API and publish custom endpoints. Exposing data via an API endpoint is a fantastic way to share it with other websites that need to consume it, and that’s exactly what I did:

Open the code

<?php [...] function register(WP_REST_Server $server) $endpoints = $this->get(); foreach ($endpoints as $endpoint_slug => $endpoint) register_rest_route( $endpoint['namespace'], $endpoint['route'], $endpoint['args'] ); function get() $version = 'v1'; return array( 'empty_db' => array( 'namespace' => 'LXB_DBA/' . $version, 'route' => '/empty_db', 'args' => array( 'methods' => array( 'DELETE' ), 'callback' => array($this, 'empty_db_cb'), 'permission_callback' => array( $this, 'is_admin' ), ), ), 'get_blogs' => array( 'namespace' => 'LXB_DBA/' . $version, 'route' => '/get_blogs', 'args' => array( 'methods' => array('GET', 'OPTIONS'), 'callback' => array($this, 'get_blogs_cb'), 'permission_callback' => array($this, 'is_dba'), ), ), 'insert_blogs' => array( 'namespace' => 'LXB_DBA/' . $version, 'route' => '/insert_blogs', 'args' => array( 'methods' => array( 'POST' ), 'callback' => array($this, 'insert_blogs_cb'), 'permission_callback' => array( $this, 'is_admin' ), ), ), 'get_blogs_from_db' => array( 'namespace' => 'LXB_DBA/' . $version, 'route' => '/get_blogs_from_db', 'args' => array( 'methods' => array( 'GET' ), 'callback' => array($this, 'get_blogs_from_db_cb'), 'permission_callback' => array($this, 'is_admin'), ), ), 'get_blog_details' => array( 'namespace' => 'LXB_DBA/' . $version, 'route' => '/get_blog_details', 'args' => array( 'methods' => array( 'GET' ), 'callback' => array($this, 'get_blog_details_cb'), 'permission_callback' => array($this, 'is_dba'), ), ), 'update_blogs' => array( 'namespace' => 'LXB_DBA/' . $version, 'route' => '/update_blogs', 'args' => array( 'methods' => array( 'PATCH' ), 'callback' => array($this, 'update_blogs_cb'), 'permission_callback' => array($this, 'is_admin'), ), ), );

We don’t need to get into every endpoint’s details, but I want to highlight one thing. First, I provided a function that returns all my endpoints in an array. Next, I wrote a function to loop through the array and register each array member as a WordPress REST API endpoint. Rather than doing both steps in one function, this decoupling allows me to easily retrieve the array of endpoints in other parts of my plugin to do other interesting things with them, such as exposing them to JavaScript. More on that shortly.

Once registered, the custom API endpoints are observable in an ordinary web browser like in the example above, or via purpose-built tools for API work, such as Postman:

PHP vs. JavaScript

I tend to prefer writing applications in PHP whenever possible, as opposed to JavaScript, and executing logic on the server, as nature intended, rather than in the browser. So, what would that look like on this project?

On the dashboard site, upon some event, such as the user clicking a “refresh data” button or perhaps a cron job, the server would make an HTTP request to each of the 25 multisite installs.

Each multisite install would query all of its blogs and consolidate its analytics data into one response per multisite.

Unfortunately, this strategy falls apart for a couple of reasons:

PHP operates synchronously, meaning you wait for one line of code to execute before moving to the next. This means that we’d be waiting for all 25 multisites to respond in series. That’s sub-optimal.

My production environment has a max execution limit of 60 seconds, and some of my multisites contain hundreds of blogs. Querying their analytics data takes a second or two per blog.

Damn. I had no choice but to swallow hard and commit to writing the application logic in JavaScript. Not my favorite, but an eerily elegant solution for this case:

Due to the asynchronous nature of JavaScript, it pings all 25 Multisites at once.

The endpoint on each Multisite returns a list of all the blogs on that Multisite.

The JavaScript compiles that list of blogs and (sort of) pings all 900 at once.

All 900 blogs take about one-to-two seconds to respond concurrently.

Holy cow, it just went from this:

( 1 second per Multisite * 25 installs ) + ( 1 second per blog * 900 blogs ) = roughly 925 seconds to scrape all the data.

To this:

1 second for all the Multisites at once + 1 second for all 900 blogs at once = roughly 2 seconds to scrape all the data.

That is, in theory. In practice, two factors enforce a delay:

Browsers have a limit as to how many concurrent HTTP requests they will allow, both per domain and regardless of domain. I’m having trouble finding documentation on what those limits are. Based on observing the network panel in Chrome while working on this, I’d say it’s about 50-100.

Web hosts have a limit on how many requests they can handle within a given period, both per IP address and overall. I was frequently getting a “429; Too Many Requests” response from my production environment, so I introduced a delay of 150 milliseconds between requests. They still operate concurrently, it’s just that they’re forced to wait 150ms per blog. Maybe “stagger” is a better word than “wait” in this context:

Open the code

async function getBlogsDetails(blogs) let promises = []; // Iterate and set timeouts to stagger requests by 100ms each blogs.forEach((blog, index) => if (typeof blog.url === 'undefined') return; let id = blog.id; const url = blog.url + '/' + blogDetailsEnpointPath + '?uncache=' + getRandomInt(); // Create a promise that resolves after 150ms delay per blog index const delayedPromise = new Promise(resolve => setTimeout(async () => try const blogResult = await fetchBlogDetails(url, id); if( typeof blogResult.urls == 'undefined' ) console.error( url, id, blogResult ); else if( ! blogResult.urls ) console.error( blogResult ); else if( blogResult.urls.length == 0 ) console.error( blogResult ); else console.log( blogResult ); resolve(blogResult); catch (error) console.error(`Error fetching details for blog ID $id:`, error); resolve(null); // Resolve with null to handle errors gracefully , index * 150); // Offset each request by 100ms ); promises.push(delayedPromise); ); // Wait for all requests to complete const blogsResults = await Promise.all(promises); // Filter out any null results in case of caught errors return blogsResults.filter(result => result !== null);

With these limitations factored in, I found that it takes about 170 seconds to scrape all 900 blogs. This is acceptable because I cache the results, meaning the user only has to wait once at the start of each work session.

The result of all this madness — this incredible barrage of Ajax calls, is just plain fun to watch:

PHP and JavaScript: Connecting the dots

I registered my endpoints in PHP and called them in JavaScript. Merging these two worlds is often an annoying and bug-prone part of any project. To make it as easy as possible, I use wp_localize_script():

<?php [...] class Enqueue function __construct() add_action( 'admin_enqueue_scripts', array( $this, 'lexblog_network_analytics_script' ), 10 ); add_action( 'admin_enqueue_scripts', array( $this, 'lexblog_network_analytics_localize' ), 11 ); function lexblog_network_analytics_script() wp_register_script( 'lexblog_network_analytics_script', LXB_DBA_URL . '/js/lexblog_network_analytics.js', array( 'jquery', 'jquery-ui-autocomplete' ), false, false ); function lexblog_network_analytics_localize() $a = new LexblogNetworkAnalytics; $data = $a -> get_localization_data(); $slug = $a -> get_slug(); wp_localize_script( 'lexblog_network_analytics_script', $slug, $data ); // etc.

In that script, I’m telling WordPress two things:

Load my JavaScript file.

When you do, take my endpoint URLs, bundle them up as JSON, and inject them into the HTML document as a global variable for my JavaScript to read. This is leveraging the point I noted earlier where I took care to provide a convenient function for defining the endpoint URLs, which other functions can then invoke without fear of causing any side effects.

Here’s how that ended up looking:

The JSON and its associated JavaScript file, where I pass information from PHP to JavaScript using wp_localize_script().

Auth: Fort Knox or Sandbox?

We need to talk about authentication. To what degree do these endpoints need to be protected by server-side logic? Although exposing analytics data is not nearly as sensitive as, say, user passwords, I’d prefer to keep things reasonably locked up. Also, since some of these endpoints perform a lot of database queries and Google Analytics API calls, it’d be weird to sit here and be vulnerable to weirdos who might want to overload my database or Google Analytics rate limits.

That’s why I registered an application password on each of the 25 client sites. Using an app password in php is quite simple. You can authenticate the HTTP requests just like any basic authentication scheme.

I’m using JavaScript, so I had to localize them first, as described in the previous section. With that in place, I was able to append these credentials when making an Ajax call:

async function fetchBlogsOfInstall(url, id) let install = lexblog_network_analytics.installs[id]; let pw = install.pw; let user = install.user; // Create a Basic Auth token let token = btoa(`$user:$pw`); let auth = 'Authorization': `Basic $token` ; try let data = await $.ajax( url: url, method: 'GET', dataType: 'json', headers: auth ); return data; catch (error) console.error('Request failed:', error); return [];

That file uses this cool function called btoa() for turning the raw username and password combo into basic authentication.

The part where we say, “Oh Right, CORS.”

Whenever I have a project where Ajax calls are flying around all over the place, working reasonably well in my local environment, I always have a brief moment of panic when I try it on a real website, only to get errors like this:

Oh. Right. CORS. Most reasonably secure websites do not allow other websites to make arbitrary Ajax requests. In this project, I absolutely do need the Dashboard Site to make many Ajax calls to the 25 client sites, so I have to tell the client sites to allow CORS:

<?php // ... function __construct() add_action( 'rest_api_init', array( $this, 'maybe_add_cors_headers' ), 10 ); function maybe_add_cors_headers() // Only allow CORS for the endpoints that pertain to this plugin. if( $this->is_dba() ) add_filter( 'rest_pre_serve_request', array( $this, 'send_cors_headers' ), 10, 2 ); function is_dba() $url = $this->get_current_url(); $ep_urls = $this->get_endpoint_urls(); $out = in_array( $url, $ep_urls ); return $out; function send_cors_headers( $served, $result ) // Only allow CORS from the dashboard site. $dashboard_site_url = $this->get_dashboard_site_url(); header( "Access-Control-Allow-Origin: $dashboard_site_url" ); header( 'Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept, Authorization' ); header( 'Access-Control-Allow-Methods: GET, OPTIONS' ); return $served; [...] }

You’ll note that I’m following the principle of least privilege by taking steps to only allow CORS where it’s necessary.

Auth, Part 2: I’ve been known to auth myself

I authenticated an Ajax call from the dashboard site to the client sites. I registered some logic on all the client sites to allow the request to pass CORS. But then, back on the dashboard site, I had to get that response from the browser to the server.

The answer, again, was to make an Ajax call to the WordPress REST API endpoint for storing the data. But since this was an actual database write, not merely a read, it was more important than ever to authenticate. I did this by requiring that the current user be logged into WordPress and possess sufficient privileges. But how would the browser know about this?

In PHP, when registering our endpoints, we provide a permissions callback to make sure the current user is an admin:

<?php // ... function get() $version = 'v1'; return array( 'update_blogs' => array( 'namespace' => 'LXB_DBA/' . $version, 'route' => '/update_blogs', 'args' => array( 'methods' => array( 'PATCH' ), 'callback' => array( $this, 'update_blogs_cb' ), 'permission_callback' => array( $this, 'is_admin' ), ), ), // ... ); function is_admin() $out = current_user_can( 'update_core' ); return $out;

JavaScript can use this — it’s able to identify the current user — because, once again, that data is localized. The current user is represented by their nonce:

async function insertBlog( data ) let url = lexblog_network_analytics.endpoint_urls.insert_blog; try await $.ajax( url: url, method: 'POST', dataType: 'json', data: data, headers: 'X-WP-Nonce': getNonce() ); catch (error) console.error('Failed to store blogs:', error); function getNonce() if( typeof wpApiSettings.nonce == 'undefined' ) return false; return wpApiSettings.nonce;

The wpApiSettings.nonce global variable is automatically present in all WordPress admin screens. I didn’t have to localize that. WordPress core did it for me.

Cache is King

Compressing the Google Analytics data from 900 domains into a three-minute loading .gif is decent, but it would be totally unacceptable to have to wait for that long multiple times per work session. Therefore I cache the results of all 25 client sites in the database of the dashboard site.

I’ve written before about using the WordPress Transients API for caching data, and I could have used it on this project. However, something about the tremendous volume of data and the complexity implied within the Figma design made me consider a different approach. I like the saying, “The wider the base, the higher the peak,” and it applies here. Given that the user needs to query and sort the data by date, author, and metadata, I think stashing everything into a single database cell — which is what a transient is — would feel a little claustrophobic. Instead, I dialed up E.F. Codd and used a relational database model via custom tables:

In the Dashboard Site, I created seven custom database tables, including one relational table, to cache the data from the 25 client sites, as shown in the image.

It’s been years since I’ve paged through Larry Ullman’s career-defining (as in, my career) books on database design, but I came into this project with a general idea of what a good architecture would look like. As for the specific details — things like column types — I foresaw a lot of Stack Overflow time in my future. Fortunately, LLMs love MySQL and I was able to scaffold out my requirements using DocBlocks and let Sam Altman fill in the blanks:

Open the code

<?php /** * Provides the SQL code for creating the Blogs table. It has columns for: * - ID: The ID for the blog. This should just autoincrement and is the primary key. * - name: The name of the blog. Required. * - slug: A machine-friendly version of the blog name. Required. * - url: The url of the blog. Required. * - mapped_domain: The vanity domain name of the blog. Optional. * - install: The name of the Multisite install where this blog was scraped from. Required. * - registered: The date on which this blog began publishing posts. Optional. * - firm_id: The ID of the firm that publishes this blog. This will be used as a foreign key to relate to the Firms table. Optional. * - practice_area_id: The ID of the firm that publishes this blog. This will be used as a foreign key to relate to the PracticeAreas table. Optional. * - amlaw: Either a 0 or a 1, to indicate if the blog comes from an AmLaw firm. Required. * - subscriber_count: The number of email subscribers for this blog. Optional. * - day_view_count: The number of views for this blog today. Optional. * - week_view_count: The number of views for this blog this week. Optional. * - month_view_count: The number of views for this blog this month. Optional. * - year_view_count: The number of views for this blog this year. Optional. * * @return string The SQL for generating the blogs table. */ function get_blogs_table_sql() $slug = 'blogs'; $out = "CREATE TABLE $this->get_prefix()_$slug ( id BIGINT NOT NULL AUTO_INCREMENT, slug VARCHAR(255) NOT NULL, name VARCHAR(255) NOT NULL, url VARCHAR(255) NOT NULL UNIQUE, /* adding unique constraint */ mapped_domain VARCHAR(255) UNIQUE, install VARCHAR(255) NOT NULL, registered DATE DEFAULT NULL, firm_id BIGINT, practice_area_id BIGINT, amlaw TINYINT NOT NULL, subscriber_count BIGINT, day_view_count BIGINT, week_view_count BIGINT, month_view_count BIGINT, year_view_count BIGINT, PRIMARY KEY (id), FOREIGN KEY (firm_id) REFERENCES $this->get_prefix()_firms(id), FOREIGN KEY (practice_area_id) REFERENCES $this->get_prefix()_practice_areas(id) ) DEFAULT CHARSET=utf8mb4;"; return $out;

In that file, I quickly wrote a DocBlock for each function, and let the OpenAI playground spit out the SQL. I tested the result and suggested some rigorous type-checking for values that should always be formatted as numbers or dates, but that was the only adjustment I had to make. I think that’s the correct use of AI at this moment: You come in with a strong idea of what the result should be, AI fills in the details, and you debate with it until the details reflect what you mostly already knew.

How it’s going

I’ve implemented most of the user stories now. Certainly enough to release an MVP and begin gathering whatever insights this data might have for us:

It’s working!

One interesting data point thus far: Although all the blogs are on the topic of legal matters (they are lawyer blogs, after all), blogs that cover topics with a more general appeal seem to drive more traffic. Blogs about the law as it pertains to food, cruise ships, germs, and cannabis, for example. Furthermore, the largest law firms on our network don’t seem to have much of a foothold there. Smaller firms are doing a better job of connecting with a wider audience. I’m positive that other insights will emerge as we work more deeply with this.

Regrets? I’ve had a few.

This project probably would have been a nice opportunity to apply a modern JavaScript framework, or just no framework at all. I like React and I can imagine how cool it would be to have this application be driven by the various changes in state rather than… drumroll… a couple thousand lines of jQuery!

I like jQuery’s ajax() method, and I like the jQueryUI autocomplete component. Also, there’s less of a performance concern here than on a public-facing front-end. Since this screen is in the WordPress admin area, I’m not concerned about Google admonishing me for using an extra library. And I’m just faster with jQuery. Use whatever you want.

I also think it would be interesting to put AWS to work here and see what could be done through Lambda functions. Maybe I could get Lambda to make all 25 plus 900 requests concurrently with no worries about browser limitations. Heck, maybe I could get it to cycle through IP addresses and sidestep the 429 rate limit as well.

And what about cron? Cron could do a lot of work for us here. It could compile the data on each of the 25 client sites ahead of time, meaning that the initial three-minute refresh time goes away. Writing an application in cron, initially, I think is fine. Coming back six months later to debug something is another matter. Not my favorite. I might revisit this later on, but for now, the cron-free implementation meets the MVP goal.

I have not provided a line-by-line tutorial here, or even a working repo for you to download, and that level of detail was never my intention. I wanted to share high-level strategy decisions that might be of interest to fellow Multi-Multisite people. Have you faced a similar challenge? I’d love to hear about it in the comments!

0 notes

sanjanabia · 1 month ago

Text

Why AWS is Becoming Essential for Modern IT Professionals

In today's fast-paced tech landscape, the integration of development and operations has become crucial for delivering high-quality software efficiently. AWS DevOps is at the forefront of this transformation, enabling organizations to streamline their processes, enhance collaboration, and achieve faster deployment cycles. For IT professionals looking to stay relevant in this evolving environment, pursuing AWS DevOps training in Hyderabad is a strategic choice. Let’s explore why AWS DevOps is essential and how training can set you up for success.

The Rise of AWS DevOps

1. Enhanced Collaboration

AWS DevOps emphasizes the collaboration between development and operations teams, breaking down silos that often hinder productivity. By fostering communication and cooperation, organizations can respond more quickly to changes and requirements. This shift is vital for businesses aiming to stay competitive in today’s market.

2. Increased Efficiency

With AWS DevOps practices, automation plays a key role. Tasks that were once manual and time-consuming, such as testing and deployment, can now be automated using AWS tools. This not only speeds up the development process but also reduces the likelihood of human error. By mastering these automation techniques through AWS DevOps training in Hyderabad, professionals can contribute significantly to their teams' efficiency.

Benefits of AWS DevOps Training

1. Comprehensive Skill Development

An AWS DevOps training in Hyderabad program covers a wide range of essential topics, including:

AWS services such as EC2, S3, and Lambda

Continuous Integration and Continuous Deployment (CI/CD) pipelines

Infrastructure as Code (IaC) with tools like AWS CloudFormation

Monitoring and logging with AWS CloudWatch

This comprehensive curriculum equips you with the skills needed to thrive in modern IT environments.

2. Hands-On Experience

Most training programs emphasize practical, hands-on experience. You'll work on real-world projects that allow you to apply the concepts you've learned. This experience is invaluable for building confidence and competence in AWS DevOps practices.

3. Industry-Recognized Certifications

Earning AWS certifications, such as the AWS Certified DevOps Engineer, can significantly enhance your resume. Completing AWS DevOps training in Hyderabad prepares you for these certifications, demonstrating your commitment to professional development and expertise in the field.

4. Networking Opportunities

Participating in an AWS DevOps training in Hyderabad program also allows you to connect with industry professionals and peers. Building a network during your training can lead to job opportunities, mentorship, and collaborative projects that can advance your career.

Career Opportunities in AWS DevOps

1. Diverse Roles

With expertise in AWS DevOps, you can pursue various roles, including:

DevOps Engineer

Site Reliability Engineer (SRE)

Cloud Architect

Automation Engineer

Each role offers unique challenges and opportunities for growth, making AWS DevOps skills highly valuable.

2. High Demand and Salary Potential

The demand for DevOps professionals, particularly those skilled in AWS, is skyrocketing. Organizations are actively seeking AWS-certified candidates who can implement effective DevOps practices. According to industry reports, these professionals often command competitive salaries, making an AWS DevOps training in Hyderabad a wise investment.

3. Job Security

As more companies adopt cloud solutions and DevOps practices, the need for skilled professionals will continue to grow. This trend indicates that expertise in AWS DevOps can provide long-term job security and career advancement opportunities.

Staying Relevant in a Rapidly Changing Industry

1. Continuous Learning

The tech industry is continually evolving, and AWS regularly introduces new tools and features. Staying updated with these advancements is crucial for maintaining your relevance in the field. Consider pursuing additional certifications or training courses to deepen your expertise.

2. Community Engagement

Engaging with AWS and DevOps communities can provide insights into industry trends and best practices. These networks often share valuable resources, training materials, and opportunities for collaboration.

Conclusion

As the demand for efficient software delivery continues to rise, AWS DevOps expertise has become essential for modern IT professionals. Investing in AWS DevOps training in Hyderabad will equip you with the skills and knowledge needed to excel in this dynamic field.

By enhancing your capabilities in collaboration, automation, and continuous delivery, you can position yourself for a successful career in AWS DevOps. Don’t miss the opportunity to elevate your professional journey—consider enrolling in an AWS DevOps training in Hyderabad program today and unlock your potential in the world of cloud computing!

0 notes