Jeff Geerling recently announced that the DrupalVM project is officially archived. DrupalVM was crucial in helping provide standardized local development for Drupal developers. DrupalVM was a customized Vagrant setup with batteries-included tooling powered by Jeff's many Ansible roles.

For me, it's wild to think back to 2014 when DrupalVM was first created. The ways we handled local development and all of the emerging technologies. Virtual machines ruled the land, and Vagrant empowered a new way to manage headless virtual machine instances. To use Docker on macOS, you had to use boot2docker to create a virtual machine.

Get the Most Out of Apache Solr: A Technical Exploration of Search Indexing Saranya Ashok Kumar 21 Feb, 2023 Subscribe to our Newsletter Now Subscribe Leave this field blank

A search feature enhances the user experience of a website by allowing the user to find what they’re looking for easily and quickly. More so for large websites, e-commerce sites, and sites with dynamic content (news sites, blogs).

Apache Solr is one of the most popular search platforms used by websites of all sizes. It is an open-source search engine based on Java that lets you search through large amounts of data, like articles, products, customer reviews and more. Take a deeper look into Apache Solr in this article.

Check out this article to learn how to configure Apache Solr in Drupal

Why is Apache Solr so popular?

Apache Solr is fast and flexible and allows for full-text search, hit highlighting (highlights the matching search term), faceted search (a more refined search), real-time indexing (allows new content to be indexed immediately), dynamic clustering (organizes search results into groups), database integration, NoSQL features (non-relational database) and rich document handling (to index a wide variety of document formats like PDF, MS Office, Open office).

Some good-to-know facts about Apache Solr:

• It was initially developed by CNET networks, inc. as a search engine for their websites and articles. Later, it was open-sourced and became a top-level Apache project.
• Supports multiple programming languages like PHP, Java, Python, and Ruby. It also provides APIs for these languages.
• Has built-in support for geospatial search, allowing to search content based on its location. Especially useful for sites like real estate websites, travel websites, etc.
• Supports advanced search features like spell checking, autocomplete, and custom search via APIs and plugins.
• Uses Lucene for indexing and searching.

What is Lucene

Apache Lucene is an open-source Java search library that lets you easily add search or information retrieval to the application. It is versatile, powerful, accurate, and works on an efficient search algorithm.

Although known for its full-text search capabilities, Lucene can also be used for document classification, data analysis and information retrieval. It also supports many languages other than English like German, French, Spanish, Chinese, Japanese, and more.

What is Indexing?

All search engines begin with indexing. Indexing is the processing of original data into highly efficient cross-reference lookup to facilitate rapid search.

Search engines don't index data directly. The texts are first broken into tokens (atomic elements). Searching is the process of consulting the search index and retrieving the doc matching the query.

Advantages of indexing

• Fast and accurate information retrieval (collects, parses and stores)
• Without indexing, the search engine requires more time to scan every document

Indexing flow

First, the document will be analyzed and split into tokens. All those tokens will be indexed to the inverted index. Inverted index is a way in which Solr builds the index.

How inverted indexing works

Lets consider we have 3 documents:

1. I love chocolate (D 1)
2. I ordered chocolate cake (D 2)
3. I prepared big vanilla cake (D 3)

The way it is tokenized is as shown in the 2nd column of the below table.

“Chocolate” is available in D1 and D2
“Cake” is available in D2 and D3
“Big” is available in D3
“Ordered” is available in D2
“Prepared” is available in D3
“Vanilla” is available in D3

You will notice that words like “I”, “love” are not tokenized. These are called Stop words which will not be indexed or searchable by Solr.

So when someone searches for the term “Chocolate Cake”, the engine looks into the index. Instead of looking for the document, it first looks into the index to see which documents do the words “Chocolate” and “Cake” fall under. This makes it easy and faster to fetch the particular document only. This is called inverted indexing.

Storage Schema

Apache Solr uses a document-based storage schema and stores every piece of data as a separate document within a collection. This allows for efficient and flexible storage and retrieval of data.

In Drupal, each node is considered as a document. So when you index your node to Apache Solr, it is considered as a document. Each document can contain multiple fields. Lucene does not have common global schema. Which means you can index any type of field in each document in Apache Solr.

How to Install Apache Solr

• First, make sure you have Java installed on your system.
• Next, let’s install Solr from here: https://solr.apache.org/downloads.html
• Download and extract Solr.
• Run this command on the Solr folder.

           ◦ bin/solr -e techproducts

             This will create a dummy core for demonstration and it will also start the Solr server.

• Once the server has started, go to your browser and type “http://localhost:8983/”.
• Make sure Solr is installed successfully with dummy core.

Directory Structure

Once you have installed Solr, you will see many folders like:

Docs - contains documentation about Solr
Dist - Solr main .jar file
Contrib - contains add-on plugins and specialised features of Solr
Bin - scripts of Solr
Example - contains demonstrate solr capabilities
Server - heart of Solr. Contains Solr web application, logs, Solr core

Config files

To create a core, we need two files mandatory.

• Schema.xml
• Solrconfig.xml

Schema.xml

• It will contain the types of fields you plan to support and how those types should be analyzed.

Solrconfig.xml

• Contains various settings that controls the behavior of a Solr core like request handler, request dispatcher, query components, update handlers, etc.

Querying in Solr

Now lets see how to query the Solr results in the Solr admin UI.

Query Parameter

• Local parameters are arguments in a Solr request that are specific to a query parameter.

For example: cat: electronics

 

Query Parameter with operations

• We can query multiple fields with operation.

For example: cat: electronics id:TWINX2048-3200PRO with q.op AND
[OR]
cat: electronics AND id:TWINX2048-3200PRO

[OR]

 

Filter Query

A filter query helps narrow down the results of a search. A query can be specified by the fq parameter to restrict which documents are returned in the superset, without affecting the score.

 

Sort Parameter

The sort parameter arranges search results in either ascending (asc) or descending (desc) order. Depending on the content, the parameter can be used either numerically or alphabetically.

 

Rows Parameter

The rows parameter allows you to paginate results from a query.

 

Field List Parameter

The fl parameter limits the information included in a query response to a specified list of fields.

 

Default field Parameter

Default field parameter is the default field for query parameter.

 

Highlights Parameter

The highlight feature in Solr enables the inclusion of fragments of documents that match a query.

 

Some of the most common highlight parameters are:

• Hl.fl - Highlights a list of fields.
• Hl.simple.pre - Specifies which "tag" should be used before a highlighted word.
• Hl.simple.post - Specifies which “tag” should be used after a highlighted term.
• hl.highlightMultiTerm - If it is set to true, Solr will highlight wildcard queries. If false, they won’t be highlighted at all.

 

Facet:

Facets enable users to explore and refine large sets of search results. They’re displayed in a UI as checkboxes, dropdowns or other controls. The two general parameters to control facets are:

1. Facet parameter

Using the facet parameter, users can generate facets based on the values of one or more fields in their search index. In the search results, the facet parameter can be configured to control how facets are generated and displayed.

      2. Facet.query paramater

When a user includes a facet.query parameter in their Solr query, Solr will generate a list of facet counts that correspond to the number of documents in the index that match each query. Facet.query is useful when you want to generate facets based on complex search criteria that can't be easily represented using a simple field value. 

There are several other facet parameters like the facet.field (to specify the fields that should be used to generate facets), facet.limit (max number of facets to display for each field), facet.mincount (min number of document needed for the facet to be included in the response), facet.sort (specifies the order in which the facet values should display).

 

Final Thoughts

Apache Solr is a highly versatile search engine that comes with many interesting features which can be customized as per your requirements. Drupal works extremely well with Apache Solr. If you’re looking for Drupal experts to configure a powerful search engine for your new project, we would love to take it further!

Author: Saranya Ashok Kumar

Meet Saranya Ashok Kumar, Drupal Specialist, who’s extremely fond of coding and Drupal and likes sharing valuable content through her YouTube channel. Saranya likes tapping her toes to her favorite music and dreams of traveling to the Maldives.

Email Address Subscribe Leave this field blank Drupal Development Drupal Planet Drupal

Leave us a Comment

 

Recent Blogs

Image

Get the Most Out of Apache Solr: A Technical Exploration of Search Indexing

Image

From Mother to Manager - Shreevidya’s Career Story

Image

How to Integrate Google Tag Manager with Drupal 9 - An Easy Step-by-Step Tutorial

Want to extract the maximum out of Drupal? TALK TO US

Featured Case Studies

Great Southern Homes, one of the fastest growing home builders in the US, sees greater results with Drupal

Explore

A reimagined digital solution for Abaco, a global leader in embedded computing systems for the defense industry

Explore

Upgrading the web presence of IEEE Information Theory Society, the most trusted voice for advanced technology

Explore View all Case Studies

Today we are talking about ChatGPT with Ezequiel Lanza.

For show notes visit: www.talkingDrupal.com/387

Topics

• What is ChatGPT?
• What is AI?
• What is Machine Learning?
• Common misconceptions
• How does it work?
• Accuracy
• Programmer bias
• Use cases
• Impressiveness
• Drupal
• Significance of Open Source

Resources

Hey GitHub - Coding with your voice ChatGPT Wolfram Alpha

Guests

Ezequiel Lanza - github.com/ezelanza @eze_lanza

Hosts

Nic Laflin - www.nLighteneddevelopment.com @nicxvan John Picozzi - www.epam.com @johnpicozzi Katherine Druckman - katherinedruckman.com @katherined

MOTW Correspondent

Martin Anderson-Clutz - @mandclu Search API Solr Boost By User Term Allows your site to boost search results that share taxonomy term references with your users.

Starting a newsletter with a rant about ageism is not my intent. I am not against developers gaining age or aged developers building things. I respect them. 

Last week, TheDropTimes (TDT) published a few interviews, which the larger Drupal Community read. In one of those interviews, I remarked on a question pointed towards Mike Herchel that 'the mean age of Drupal developers is increasing.' 

Mike carefully avoided confronting me on the veracity of my claim so that I wouldn't feel intimidated. I am thankful for that. I do not have any data to prove the statement or to disprove it. It remains an unconfirmed allegation until we have the numbers to substantiate it. 

But this is my perception of the Drupal community and almost all Free, Libre, and Open-Source Software communities. The leaders I see in the communities I once frequented are all in the 35-55 age bracket or even older. 

Some newer technologies have many takers, some of which are JS frameworks. They might still be FLOSS. What I am saying is about established technologies that have not entirely lost their sheen. 

Down the lane, these new technologies might turn legacy, and they, too, would face a similar drag in the influx of new developers. I think of it as a bane of the commons. 

For the sake of argument, consider this averment at face value. The engine to drive the enthusiasm to sustain a project and attract newbies shouldn't be the freshness of something. It should be the ideal tooling that makes everyone comfortable, gradually reducing the learning curve through the years and nurturing an accepting and receptive community that would reward the efforts. 

In the Interview with Mike Herchel, a community at large member of the Drupal Association's board of directors, he responds that 'most newer developers flock to the JavaScript-based ecosystem because of better developer tooling and marketing.' He also points out that 'many smart people are actively working on better developer tooling within Drupal core.' And the 'Association has many plans to do a better job marketing Drupal.' As a vibrant community, we must be thankful for that. TDT is also an independent attempt to build strong muscle in marketing Drupal to the world. And we say it with utmost responsibility. We say marketing Drupal, which includes the whole ecosystem, including the companies and developers working in the realm. 

That is why we chose to interview Jurriaan Roelofs, the founder and senior product manager at DXPR, a company known for its SaaS offering with which one could build Drupal websites without knowing how to code. It is a detailed interview about the development of Drupal Layout Builder, one of the best drag-and-drop experiences available in Drupal. 

We continued our interviews with the organizers and speakers of the recently concluded Florida DrupalCamp. Aubrey Sambor, a senior Front-end Developer at Lullabot, speaks about her work in web accessibility following her dad becoming a quadriplegic. Mark Shropshire, Senior Director of Development at Mediacurrent, reflects on the regulatory and standards requirements at organizations that fostered his interest in security, eventually ensuring the creation of Guardr. He also talks about Cypress testing framework for functional testings of Drupal and related headless front-ends, the subject of his session. In his interview with Alethia Braganza, Jonathan Daggerhart weighs the strengths and weaknesses of Drupal and WordPress. He talks about 'sustainable web development practice' and explains 'transparency,' a cultural pillar around which he built his company: these interviews and the interview with Mike Herchel were part of our FLDC series. 

Last week, TDT published a listicle with 12 email marketing apps integrated with Drupal and its modules. Add these tools to your MarTech quiver.

We routinely report on awards and accolades. An article published about the German Splash Awards had discrepancies that someone pointed out through our contact form. We have since taken down the report and will republish it with the corrected info. In Vol.01, Issue 04 of the newsletter published on February 13, 2023, the nutgraf had a significant mistake altering the sentence's meaning. Instead of revered, which means respected (the past tense of the word revere), we wrongly wrote rever, which means 'the upper part of some upper garments that folds back at or near the neck to give the appearance of a collar.' By the time we noticed it, the newsletter was already gone. We regret such mistakes and will strive to correct them as far as possible. If you see any such shortcomings, please get in touch with us immediately. 

Last week, we reported on Promet Source winning the Web Excellence Award, Joe Shindelar of Drupalize.me updating the Drupal User Guide for D10, Drupal Academy's video tutorial on passing variables into twig templates, Drupal Partner's case study on NIFTEP, an institute under Georgia State University, reducing spam signups on their website, Patti Cardiff's article in Promet Source's blog about why and how to measure web performance, Russel Jones video tutorial on Style Headers in Drupal with CXX Flex and Tailwind CSS, Magic Logix's blog post on benefits of Drupal Development Servies, an update on the upcoming DrupalCamp NJ, call for speakers in OpenSource North, the opening of registrations for DrupalCon Pittsburgh, Early Bird discount for DrupalCamp Ruhr,  Drupal 10 Masterclass book by Adam Bergstein, the announcement of DrupalSouth 2023, deadline of feedbacks for Project Browser Initiative, Lullabot's webinar on structured content, and other stories. 

This is for the week. Happy reading. 

Sincerely, 
Sebin A. Jacob
Editor-in-Chief

The upcoming interview will feature Senior Software Engineers April Sides and Melissa Bent from Red Hat.

In previous articles I have written about injecting context into context aware plugins and creating custom context providers and wanted to complete the series by writing about creating context aware custom plugins.

The context system in Drupal is a powerful way of injecting dynamic data into plugins without writing code to add that data directly to the plugin itself. Instead of adding custom code to find the current user or the node from the route of the page you can inject the context into the plugin using the context system and add code to make use of that data. Although most commonly used in blocks it can be found in a couple of other plugin types in Drupal core, like the condition plugin for example.

In this article I will go through how to create a context aware plugin, including how to create custom plugins and how to allow that plugin to understand the context_definitions annotation. Once the custom plugin is complete we will render it using a Drupal controller action to prove that the context works correctly.

Let's start by creating a custom plugin, we'll call this plugin ContextThing and it will be used to print out the context passed to it. The first step in creating custom plugins is to create an Annotation class.

Plugin Annotation Class

Annotations are special kinds of comments that have a number of functions in Drupal, but in this case we are using them to inform Drupal that a particular class is plugin.

As an example of annotations in action we can look at defining custom Blocks. To define a block plugin you would start the class annotation with @Block and then add the fields you need to the annotation definition. This would look something like this.

Read more

The 15th annual Florida DrupalCamp 2023 is here! Florida DrupalCamp is an annual conference that brings web professionals worldwide to learn, network, and talk about web development and Drupal. Florida DrupalCamp is called a “camp” to emphasize the unconference style of the event.

“Happiness is not the right metric by which to measure someone’s work life; “low-stress” might be better" Says Jonathan Daggerhart in his interview with TDT. Drupal Camp Florida is here, and as TDT is an in-kind sponsor we took this opportunity to speak to the FLDC session speakers.

DrupalCons are for everyone who loves to build and grow with Drupal. These events bring together community leaders, developers, designers, content creators, and users to share their knowledge, skills, and experiences. Whether you're a seasoned pro or just starting out, DrupalCons are a perfect platform to expand your horizons, network with like-minded individuals, and have fun in the process!

And what could be a more effective way to interact with as many people as possible than to volunteer? The highly anticipated DrupalCon Pittsburgh 2023 is just around the corner, and we’re looking for event volunteers to make it a resounding success!

Why volunteer?

For starters, you get a free ticket when you volunteer for a minimum of 8 hours! If that doesn’t cut it for you, discover more reasons to volunteer and be a part of this exciting event:

• Scientific effects of giving - When you’re volunteering for an event you care about, you are giving your time and a part of yourself to it. This has multiple psychological, social, and physiological benefits for you. Studies show that giving can trigger the release of feel-good chemicals like oxytocin, dopamine, and serotonin, resulting in a feeling of happiness, satisfaction and well-being.
• Personal development - After contributing to DrupalCon for the three years in a row, I am able to personally attest to my growth and the impact it has had on me. Volunteering helps you gain new skills, knowledge and experience. This can boost your self-confidence and enhance your personal growth.
• Building connections - This is one of my favorite reasons to contribute. Volunteering gives you a chance to interact with people from diverse backgrounds who have a very different skillset than yours. Learning opportunities are immense when you collaboratively work as a team.
• Community involvement - If you’re passionate about Drupal, you should love giving back to the community that helps build Drupal. It is simply the right thing to do if you’re using Drupal in any way. Being a part of an event as huge as DrupalCon is a great way to contribute and get involved with the community.
• Career advancement - Gain practical experience and demonstrate your commitment to the industry when you volunteer for DrupalCon. You are exhibiting your willingness to take the initiative and that’s a quality all employers are looking out for. Additionally, when you earn credits for your contributions, it benefits both you and your employer.

Open positions and How you can help

Out of the many opportunities to volunteer at DrupalCon Pittsburgh, we now have 3 open positions remaining. The Speaker Review committee, Marketing & outreach committee, Local Ambassador, Summit Program committees positions have been filled. Check out more details here.

Translation Assistant

DrupalCons are for everyone from anywhere! We strive to provide a fantastic experience for all DrupalCon attendees. We are seeking volunteers to make English second language attendees feel comfortable and at home. Do you speak a language other than English and want to help others have a better DrupalCon experience? This could be a perfect opportunity for you!

Time commitment: 15-20 hours till June 2023 (including 4 hours on-site at DrupalCon).

Scholarship Mentor

This is a great opportunity for you to pay it forward and support the next generation of leaders. As a Scholarship mentor, you will guide scholarship recipients through their DrupalCon journey. This being their first time at an event as big as DrupalCon, you will help them understand expectations like the Code of Conduct and also suggest programs they should attend. You will help them build better connections with members of the community who share similar interests. So basically, for them, you will be a friendly face in an unfamiliar environment.

Time commitment: 5-10 hours in May and June (including on-site at DrupalCon)

Logistics Contributor

Logistics Contributors are the event superheroes, saving the day with their high-octane energy and smooth communication style! You will work with the event staff to manage the registration desk, warmly greet attendees, guide them to their destinations, and help with set-up/tear-down tasks. To ensure the smooth running of the event, you will also assist the Conference staff as needed.

Time commitment: 4-8 hours of on-site time at DrupalCon

Interview with Mark Shropshire, one of the maintainers of Guardr Project, a security focussed Drupal Distribution. This interview is part of a series prior to the Florida DrupalCamp 2023.