Skip to content

SEO Tips: Mining Competitive Pages for Keyword Insights

I’ve personally been going to Pubcon and other SEO conferences for years, and as a leading Chicago SEO agency, we send our SEO consultants to these shows on a regularly basis.

Recently, we’ve sent five agency team members to Pubcon, three to MozCon and six to Content Marketing World. We do this because we need to be smarter than other marketing, PR and SEO agencies and we need to stay on top of the latest SEO trends, tools and techniques.

At every SEO conference, we get a few practical tips that improve our capabilities in  solving our clients’ problems – including getting them more leads, more visibility or whatever the task at hand is.

I’m just back from Pubcon and have many great marketing and SEO learnings that I can’t wait to use. Here’s one of them…although it’s a bit embarrassing to talk about.

Using Term Frequency Analysis for SEO

There’s a concept called TF-IDF, which stands for term frequency-inverse document frequency.

I’m no expert on TF-IDF, but effectively it’s a statistical calculation that looks at word and phrase frequency in a set of documents and determines which words are the most important words.

This is highly relevant for SEO because, whereas back in the good old days, I could write a page that was optimized for, say, “Chicago SEO agency”, and simply by using good on-page SEO tactics and creating decent content that got some good external links, I had a very good chance of quickly ranking well for that phrase. And, in fact, the Walker Sands website used to rank very high in Google’s search results when people were trying to find a good SEO partner.

But those days of easy SEO are long gone. Google’s algorithm is much more sophisticated now. It is powered by machine learning and it has very sophisticated algorithms that figure out similar words and word groupings that convey whether a page is well-targeted to a searcher’s intent.

The Google Hummingbird algorithm, rolled out a couple of years ago, is a big part of that change, evaluating keyword synonyms and theming sites and pages based on semantic analysis and other stuff that you can read about in Google’s patents if you are really curious.

It’s all good. For the most part, we’re all getting more relevant and more useful search results than we used to.

Ash Nallawalla’s Lateral Keywords Tool

One of the Pubcon speakers that I met, Ash Nallawalla, talked a lot about how to use TF-IDF for SEO. Ash introduced me to his tool, Lateral Keywords, and I decided to try it out.

Based on my little experiment, and, again, I’m a bit embarrassed to tell this story in its entirety, you should try it out. Right now, it’s in beta and totally free. It’s pretty intuitive and you can try it out in 20 minutes max.

Google Patents That Reference TF-IDF

Before I get to my experiment with TF-IDF, I just want to reiterate why anybody involved with SEO should try to learn a little bit about the TF-IDF calculation.

You can skip this section unless you like to geek out on Google’s patents. But, lest you doubt how important TF-IDF is to Google’s algorithms, here is just a small subset of Google patents that use, or reference, TF-IDF:

Wow. Based on this list of Google patents, I think you should be as interested in TF-IDF as I was when I decided to try out this tool.

TF-IDF and My Embarrassing Story About Ranking for the Phrase “Chicago SEO Agency”

As background, I’ve been doing SEO for over a decade. Walker Sands, our integrated marketing agency, was a pretty small business long ago and we relied heavily on leads powered by search engine optimization. We used to joke that our niche was catering to marketing executives who didn’t know many people and who had to use Google to find their providers.

It was a time when many were skeptical about using SEO to get B2B leads, and we were doing incredibly well with SEO.

Guess who did all the SEO work on our site back then? Me. I very meticulously wrote copy to rank for specific phrases related to marketing services, and it paid off. We ranked at the top for “Chicago PR firm,” “Chicago SEO agency” and other phrases. Back then, it was pretty easy if you knew what you were doing.

Fast forward to today. We’re a much bigger agency with 70+ people. I rarely touch our website. We have done good work for many clients and have a great track record. We now specialize in B2B tech marketing, whereas we used to take anything that came in. Referrals now come in regularly based on our growing reputation, and SEO is less of a lead driver for us than it used to be. In addition, Chicago customers are now a small part of our business. We serve clients all over the world. So, yeah, very different now.

Over the years, many of my old SEO-intensive pages have been watered down, deleted, re-written and reinvented by other more talented folks at Walker Sands. That’s fine. I think we attract bigger clients and get more challenging assignments because of our revised and improved positioning,” but it has always irked me that our rankings for the phrases we used to rank well for have fallen. In a recent offsite training session we had, I was quietly steaming when I realized we were ranking around #9 for “Chicago SEO Agency” when we used to be #1.

As a quick aside, our SEO methodology that we execute for clients doesn’t obsess about individual rankings for specific phrases. We are much more focused on overall results, e.g. are we getting more leads and business from organic traffic? And we’ve talked about why SEO agencies (and clients) that talk only about ranking reports may be misguided. In fact, we rarely just do SEO services for a client now, because we preach a holistic approach that blends SEO, PPC, social, content marketing and the like. So, I know that I shouldn’t care too much about any given SEO-related phrase and how we rank.

But the old-school SEO guy in me still cares a little bit about search rankings and is chagrined to see others outranking us.

All of this relates to my experience applying Ash’s Lateral Keywords Tool because I decided to apply it to our Walker Sands Digital page that now, sadly, ranks around #9 for “Chicago SEO Agency.”

Applying TF-IDF Keyword Analysis to “Chicago SEO Agency”

Applying Ash’s tool to this particular search phrase – Chicago SEO Agency – is as simple as keying in the fields shown in the image below.

After you enter your page and your competitor sites’ pages, Ash’s behind-the-scenes Python code creates a spreadsheet that may give you some insights that you’ve not considered with respect to the language you use on your web pages.

Note that it’s important to use your Google SERPs (search engine results page) competitors, not your real-world competitors. So, for whatever phrase you are focusing on, enter the search in Google and copy those links and enter them on the form. For your site’s entry, don’t put your home page URL in; put the page that ranks highest for the target keyphrase.

The TF-IDF Keyword Analysis output spreadsheet looks like this:

It’s a full inventory of all the one-word, two-word, and three-word phrases on your site page and the competitor site pages you’ve entered. (I’ve hidden some of the 9,559 rows in total, but you can see that URL0 is our site, and URL1 through URL5 are the competitor websites.)

When you look at your results, focus in on the phrases that are highlighted. These are the keyphrases that seem to be used extensively by the sites that rank well for the keyword phrase you are focused on.

So, in the picture above, you’ll see that “Google” is highlighted. The zero in the URL0 column tells me that we don’t mention “Google” once on our page. The sites that rank better for “Chicago SEO agency” mention “Google” multiple times. So, maybe, just maybe, having this particular word on a page, in combination with all the other words, helps Google’s search algorithms to believe that you are a better SEO agency than other sites that don’t use that particular phrase.

Needless to say, this is not a silver bullet. You can’t just add a missed word onto your page and zoom to a #1 ranking. But it’s highly likely that if you use some of the words that the tool identifies (and you’ll soon see that you need to think through the words because some of them are not all that useful), then you’ll better “theme” your site as being about your target keyphrase. These words are used by the people who rank better than you, so maybe you should try them.

Here’s the List of Keywords from My Little Experiment

It’s tough to scan through all the keyphrases and see the highlighted ones, so I did my own Excel filter to show the phrases that were identified by the term frequency calculation, and you can see them in the image below:

And here they are in a three-column list:

  • agency
  • avg
  • back
  • campaign
  • can
  • capabilities
  • com
  • commitment
  • companies
  • data
  • educo
  • engines
  • excels
  • find
  • founded
  • free
  • google
  • help
  • hourly
  • keywords
  • learn
  • length
  • links
  • local
  • location
  • more
  • online
  • out
  • page
  • popcorn
  • projects
  • retention
  • service
  • size
  • style
  • total
  • type
  • websites
  • who
  • worked
  • year
  • years
  • avg engagement
  • avg monthly
  • back to
  • chicago il
  • chicago style
  • client retention
  • digital marketing
  • engagement length
  • excels with
  • hourly rate
  • learn more
  • monthly commitment
  • of client
  • projects worked
  • retention rate
  • search engines
  • style seo
  • team size
  • to your
  • total projects
  • type of
  • we are
  • we can
  • worked on
  • year founded
  • your website
  • avg engagement length
  • avg monthly commitment
  • chicago style seo
  • client retention rate
  • projects worked on
  • total projects worked
  • type of client

As mentioned above, you’ll get some phrases that are just weird. “Popcorn” is a good example in the list above. No, using the word “popcorn” on a page won’t convince Google’s Hummingbird algorithm that you are good at SEO. Correlation is not causation, as they say. In this case, the word “popcorn” was used extensively by one Chicago SEO agency that outranks Walker Sands; on their ranking page, they listed many phrases that they helped a popcorn company to rank well for.

But “popcorn” wasn’t the phrase in the list that really jumped out at me. One other keyphrase jumped out right away.

And Now The Embarrassing Part…

It took me all of two seconds to see that many competing site pages that rank better than our page use the world “agency” on their page and we don’t.

Are you freaking kidding me?! Our page that we’d like to rank well for “Chicago SEO agency” – that currently is a pathetic #8 or #9 position when I do an incognito search in Chrome – does not include the word “agency” in it?

Needless to say, I quickly corrected this. I made a few quick edits and added some text to our page such that we now have the word “agency” on the page. Duh.

It’s a “cobbler’s shoes” situation, to be sure. We are busy doing good work for clients and have less time to focus on our marketing agency’s SEO. But this was a reminder (SEO team, are you reading this?) that we can’t lose sight of these things and probably need to increase our focus on our site’s SEO. With SEO, the devil is in the details.

I can’t even believe I’m telling you this. Freaking embarrassing! Competitors, you have every right to tell people, “Are you sure you want to use Walker Sands for SEO? They are in Chicago, focus on SEO, but don’t rank well for “Chicago SEO agency” and guess what? The CEO says the page they wanted to rank for that phrase didn’t even contain the word “agency” in it!

I mean it’s not just that we don’t have that word on the page and instead have words like “consultants”, “firm” and “company”, the Lateral Keywords tool, as depicted in the image below, shows me that our page doesn’t have much content compared to the pages that outrank us.

This is another punch to the gut. I know – we all know unless we are asleep – that Googe is now loving very long, high quality web pages with informative and useful content (like this one I’m writing now, I hope).

Short, shallow content that uses target keywords in middling-quality content doesn’t get very far these days. It used to work, back in 2007, but not now. So, for me, Ash’s tool reminded me that we’ve dropped the ball on creating good content for our website. Yes, we are busy doing this work for clients, but we can’t not do it for own site.

If you have an SEO agency, having a “cobbler’s shoes” problem means you will lose business. Would you be willing to get a haircut from somebody whose hair looked like the picture below? No! I rest my case.

More Takeaways from the Lateral Keywords Tool

OK, aside from the blinding glimpse of the obvious, what does this particular keyword frequency tool tell me?

Aside from “Google” and “agency” (mentioned above), I see a few other phrases that seem relevant. In this list, “search engines,” “keywords” and “your website” kind of make sense to me as phrases that might theme a page for SEO services. The other terms flagged by the TF-IDF formula seem a bit off to me.

But Ash had told me to expect that. You may run it and not have any epiphanies. But if you keep trying, you probably will.

It’s also worth noting that you don’t have to limit yourself to looking at only the highlighted phrases. Scanning the full list of phrases or applying your own Excel formulas to the data may allow you to find something you can use.

Overall, Lateral Keywords  is a great SEO tool. Yes, there are plenty of other good  SEO tools out there, but this is one that I’ll likely be using a fair bit in the future.

Get in Touch