drupal

ComputerMinds.co.uk: Putting 1000 sites behind Cloudflare

That’s a lot of sites!

Yes, it is! 10+ years ago we migrated 200 sites to a new server - and in 2024 we set up Cloudflare protection for well over 1000 sites.

Aegir is a hosting system built in Drupal, for Drupal. It lets you create and manage Drupal sites and all their databases, filesystems and virtual hosts. With Aegir, it’s easy to manage hundreds or thousands of sites via a simple UI. Each site has a node to represent it, and this project stored a whole bunch of additional Cloudflare metadata against the Site Nodes.

Keeping a PaaS product online at all times comes with a high level of responsibility. After code quality assurance and testing, DDOS attacks of all sizes and types are a high risk threat. The cost of protecting our availability, unsurprisingly, was non-trivial and became a point requiring fresh research and investment. Reducing the general load and the potential attack load on our servers would serve to support our quality of service.

In the Spring of 2024 we set up a proof of concept using Cloudflare, which would allow us to make a significant ongoing cost saving whilst also playing with some really cool APIs.

The plan

In order to put all our sites behind Cloudflare, we needed to:

* Get Aegir talking to Cloudflare via their API, and build the automatic processes to support the setup process
* Create a clear interface for starting and tracking setup per site
* Create a clear dashboard for tracking progress overall
* Go! Change the nameserver records for every domain, to point to Cloudflare

Here are some of the key interesting parts of our story (which had negligible downtime, btw!)

Interacting with Cloudflare’s API

It was greatly pleasing to find that Cloudflare let you do almost everything over an API (given the right authentication!). Given the scale of their company, it shouldn’t be a huge surprise I suppose!

With 1000 sites approx. to deal with, there was a strong drive to automate as much of this process as possible. So there was much celebration when it became clear that we would be able to do almost every step automatically, including:

* create everything we needed for the site domain, with an appropriate plan under our chosen account
* upload DNS records, query them and update them
* perform real-time ownership verification and SSL validation

Simple admin setup - Obtain DNS records for hundreds of domains, store them against Site Nodes, press GO

A large portion of the domains (about 500) are under our control, so we were able to bulk export the DNS records, process and save them against their Aegir Site Nodes. Some simple processing removed the SOA and NS records so that we would be able to send the records straight to Cloudflare when setup started.
These ‘easy’ sites, for which we had the DNS records, would be processed in bulk with a lot of Go! button clicks, and then making the relevant nameserver changes with the domain provider.
(The domain provider did offer to do bulk updates for us, but there seemed to be a 24h delay before action was taken - so it was quicker to do these changes ourselves.)
 

Image removed. Cropped screen grab of our progress pie chart

 

 

Creating a self-service setup mechanism via Drupal config + settings.php

Domains that we only had nameserver control for would have to be updated by the customer. But how would they know what nameservers to set up, and how would they trigger the setup process? Stats to the rescue!
As part of the Drupal 10 rebuild of the platform we added a Statistics module, which collects a selection of data points from each site and passes them to the corresponding node on Aegir for storage. They’re then aggregated and sent back to the sites so that customers can compare their performance to the cohort averages.
We created a form interface for the user to trigger setup when ready, and then smuggled the outputs over to Aegir in amongst the performance stats 🤭 

This self service route did still require a lot of chasing, but generally performed well as it allowed the users to perform the nameserver change at their choice of timing, rather than requiring scheduled calls and appointments on top of an already high administrative load.

Surfacing useful data, stats and buttons for the team

When developing a technical tool that ideally needs a single fire-and-forget button to kick things off, you not only need that one button but also a lot of clear visual cues to help others understand what’s going on.

We tied the setup steps to interface outputs, with clear dependency messaging and reporting.
Reporting eventually included a message in our Slack channel ⭐️
 

Image removed. Our custom Aegir Site dashboard, with various setup steps all nice and green

The results

After just a few weeks, we had 990 sites set up on Cloudflare - 90% of the most important setups. It turned out to be very difficult to get hundreds of different people, groups and stakeholders to make DNS nameserver changes quickly (even when you tell them it’s urgent!), so the process would continue a little longer.

Already we can measure the success - thanks to Cloudflare’s caching, we’ve seen decent reductions in bandwidth use and the number of requests hitting the server.
 

Image removed. Our Cloudflare Dashboard on the Production Aegir server. Inactive sites didn't need our fuller protection setup steps, so the blue and pink bars don't match exactly.

Matt Glaman: Restrict Composer dependency updates to only patch releases

I was doing website maintenance and checked for outdated dependencies with composer outdated. I usually filter with -D for checking direct dependencies and -p for packages with patch releases. These are typically easy pickings. I saw I was on 2.1.3 of the Honeypot module and 2.1.4 was available. So I ran composer update drupal/honeypot. I noticed the module was updated to 2.2.0, because my Composer constraint is drupal/honeypot: ^2.0, allowing minor updates. I figured that was fine enough. Turns out it wasn't.

MidCamp - Midwest Drupal Camp: Session Submission Now Open for MidCamp 2025!

Session Submission Now Open for MidCamp 2025!

As the season of gratitude approaches, we’re excited to celebrate you—our future speakers! If you've got an idea for a session, now's the time to get involved in MidCamp 2025, happening May 20-22 in Chicago.

Call for Speakers

Since 2014, MidCamp has hosted over 300 amazing sessions, and we’re ready to add your talk to that legacy. We’re seeking presentations for all skill levels—from Drupal beginners to advanced users, as well as end users and business professionals. Check out our session tracks for full details on the types of talks we’re looking for.

Not quite ready? No worries! Join us for one of our Speaker Workshops:

  • December 2024 (TBD): Crafting an Outstanding Proposal
  • March 2025 (TBD): Polishing Your Presentation (Open to both MidCamp and DrupalCon Atlanta 2025 presenters!)

Key Dates

  • Session Proposals Open: November 25, 2024

Submit your session now!

  • Proposal Deadline: January 12, 2025
  • Speakers Notified: Week of February 17, 2025
  • MidCamp Sessions: May 20-21, 2025

P.S.

We hear Florida is also calling for submissions—but let’s be real, we know where your heart lies. 😊

Sponsor MidCamp

Looking to connect with the Drupal community? Sponsoring MidCamp is the way to do it! With packages starting at $600, there are opportunities to suit any budget. Whether you’re recruiting talent, growing your brand, or simply supporting the Drupal ecosystem, MidCamp sponsorship offers great value.

Act early to maximize your exposure!

Learn more about sponsorship opportunities

Stay in the Loop

Don’t miss a beat!

Keep an eye on our news page and subscribe to our newsletter for updates on the venue, travel options, social events, and speaker announcements.

Ready to submit your session? Let’s make MidCamp 2025 unforgettable!

Simon's Blog: Drupal Settings.php Snippets for Debug

Usage (TLDR;)

You can simply copy <CTRL-C> + <CTRL-V> the following into your public_html/site/default/settings.php, and toggle on/off those that are relevant to you based on your need:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 $config['system.logging']['error_level'] = 'verbose'; // [A] (For Drupal 8+) Turn on verbose debug message reporting //$conf['error_level'] = 2; // [A] (For Drupal 7) Turn on verbose debug message reporting (Equivalant to navigate to Administration→ Configuration→ Development → logging and errors and select "All messages".) // [A] ------------------------------------------- error_reporting(E_ALL); // [A] Enable PHP errors (For local Drupal development, you can also enable error reporting, display errors and display startup error to help you further debugging and fixing major runtime errors ini_set('display_errors', TRUE); // [A] Enable PHP errors (For local Drupal development, you can also enable error reporting, display errors and display startup error to help you further debugging and fixing major runtime errors ini_set('display_startup_errors', TRUE); // [A] Enable PHP errors (For local Drupal development, you can also enable error reporting, display errors and display startup error to help you further debugging and fixing major runtime errors $settings['twig_debug'] = TRUE; // [B] Twig Debug - Turn on twig debug mode $settings['twig_auto_reload'] = TRUE; // [B] Twig Debug - Turn on twig template auto reload $settings['twig_cache'] = FALSE; // [B] Twig Debug - Turn off twig cache // [B] ------------------------------------------ $settings[‘cache’][‘bins’][‘render’] = ‘cache.backend.null; // [B] Disable Caching - Disable render caching. $settings[‘cache’][‘bins’][‘page’] = ‘cache.backend.null; // [B] Disable Caching - Disable page cache. $settings[‘cache’][‘bins’][‘dynamic_page_cache’] = ‘cache.backend.null; // [B] Disable Caching - Disable dynamic page cache. $settings[‘cache’][‘default’] = ‘cache.backend.null; // [B] Disable Caching - Disable backend cache. $config['system.performance']['css']['preprocess'] = FALSE; // [C] Turn off agrregated css (see: https://www.drupal.org/docs/develop/development-tools/disabling-and-debugging-caching) $config['system.performance']['js']['preprocess'] = FALSE; // [C] Turn off agrregated js (see: https://www.drupal.org/docs/develop/development-tools/disabling-and $settings['update_free_access'] = FALSE; // [D] Enable access to /update.php $settings['rebuild_access'] = TRUE; // [D] Enable access to /rebuild.php (This setting can be enabled to allow Drupal's php and database cached storage to be cleared via the rebuild.php page. Access to this page can also be gained by generating a query string from rebuild_token_calculator.sh and using these parameters in a request to rebuild.php. $settings['skip_permissions_hardening'] = TRUE; // [E] Skip file system permissions hardening. (The system module will periodically check the permissions of your site's site directory to ensure that it is not writable by the website user. For sites that are managed with a version control system, this can cause problems when files in that directory such as settings.php are updated, because the user pulling in the changes won't have permissions to modify files in the directory. $settings['extension_discovery_scan_tests'] = TRUE; // [E] Allow test modules and themes to be installed. (Drupal ignores test modules and themes by default for performance reasons. During development it can be useful to install test extensions for debugging purpose. $settings['trusted_host_patterns'] = array(); // [E] Turn off trusted host

(for instance verbose debug required then turn comment all lines except for $config['system.logging']['error_level'] = 'verbose';)

Talking Drupal: Talking Drupal #477 - Drupal Association CTO Then & Now

Today we are talking about being the CTO of the Drupal Association, How the job has changed, and How its impacted Drupal with guests Josh Mitchell & Tim Lehnen. We’ll also cover Automatic Anchors as our module of the week.

For show notes visit: https://www.talkingDrupal.com/477

Topics
  • How long ago were you CTO Josh
  • Tim when did you take over
  • DA infrastructure
  • Drupal Credit System
  • Josh's proudest moment
  • Tim's proudest moment
  • Growth
  • Josh if you could do one thing differently
  • Tim if you could make one change
  • Future of the CTO job
Resources Guests

Tim Lehnen - aspenthornpress.com hestenet

Hosts

Nic Laflin - nLighteneddevelopment.com nicxvan John Picozzi - epam.com johnpicozzi Joshua "Josh" Mitchell - joshuami.com joshuami

MOTW Correspondent

Martin Anderson-Clutz - mandclu.com mandclu

  • Brief description:
    • Have you ever wanted headings on your Drupal site to have unique id values, so links can be created to take users to specific parts of any page? There’s a module for that.
  • Module name/project name:
  • Brief history
    • How old: created in Jun 2020 by Chris Komlenic (komlenic) of Penn State
    • Versions available: 2.1.1-beta1, which supports Drupal 8.8, 9, and 10
  • Maintainership
    • Test coverage
    • Number of open issues: x open issues, y of which are bugs against the current branch
  • Usage stats:
    • 137 sites
  • Module features and usage
    • By default, the module automatically generates ids on , , , , and elements within the page content
    • Even if two headings have the same content, the module will make sure their ids are unique, as well as making sure they are i18n-friendly, use hyphens instead of spaces, and are short enough to be useful
    • The module won’t interfere with or change manually-added or already-existing HTML ids
    • There’s a permission to view helpful links on each heading that the ids obvious and easy to copy
    • Configuration options include the root element it should look within (defaults to the body tag), which elements should get ids, what content to use for the displayed links, and whether or not generate ids on admin pages

Golems GABB: Twig & PHP Templating in Drupal 11

Twig & PHP Templating in Drupal 11 Editor Mon, 11/25/2024 - 10:11

Welcome here! This is your complete guide to Twig and PHP templating in Drupal 11. As you understand,  Twig and PHP are important for the frontend and backend development in Drupal. If you know how they work, you can create beautiful, effective, and easy-to-maintain websites. 
Today, Golems Drupal company explores Twig's smooth template engine and PHP's strong backend logic. Our blog will be helpful for every kind of person, whether you are a skilled Drupal developer, someone who owns a website or business owner, or simply starting your path in this field. We will dive into the details of Twig and PHP in Drupal to help you better understand how they work together so that your digital experiences can be crafted more effectively.

Brian Perry: Two Modules to Help Tame Large Drupal Menus

Stop me if you've heard this one before. At some point in the life of your Drupal site, you have a menu that has gotten out of control. Dragging and dropping is basically a lost cause, your hand hurts from scrolling, and a sense of dread approaches every time you find yourself in the menu administration screen. If it isn't possible to re-structure the menu to address the root cause, you'll need to turn to other solutions to make menu administration more manageable.

I recently used two modules to address this issue for a client. They may not be a huge surprise to those who have run into this problem repeatedly, but it seemed worth documenting for both future me and also our search engine and LLM overlords.

Big Menu

The first module is Big Menu. The project page on this one seems to be describing the Drupal 7 implementation of the module, which is quite a bit different. The 'modern Drupal' version of the module essentially re-works the menu administration page to focus on a single level of the menu tree at a time. Any menu item that has children will have an 'Edit child items' link that you can drill into. This results in more clicks to get to the item you want to edit, but it makes the menu administration page much more manageable and reduces cognitive load quite a bit.

You can also configure the module to use a different depth for the menu tree, which can be useful if wanted to see more of the menu in a single view. Personally I prefer to go all the way with this one and stick with the single level view that is used by default.

Menu Select

The Menu Select module addresses the experience of selecting a parent menu item in the menu settings for a node or menu item. By default, this is a select list containing the entire menu, which can get very long. Menu Select replaces this with an autocomplete search and a hierarchal collapsible unordered list.

Bonus: Menu Firstchild

Menu Firstchild is a little less about the admin experience, but can be useful in cases where a large menu needs some additional grouping but you don't want to turn to a full mega menu style approach. The module provides an option to have a menu item that doesn't have it's own path, but instead links to its first direct child.

Used together, these modules made a substantial difference in addressing the client's menu administration related feedback.

This was also a reminder of the impact that the ongoing work on Drupal CMS will hopefully have. I'm looking forward to a Drupal CMS future that can theoretically pre-package user experience improvements like these. Or in cases where it might not be the right choice for Drupal CMS, opinionated community developed recipes can be created to address common use cases like this one.