In the past few years, Online Competitive Intelligence has slowly become a field of its own. As more third party data became available, companies have realized that if they won’t be taking advantage of such data, their competitors will. And the same goes for SEO teams, since in SEO, absolutely everything is in comparison to who you compete with.
Today’s tools allow companies to easily identify their SEO competitors, even if these are different from the company’s business competitors. They allow companies to see what content works better for their rivals, which keywords are responsible for their organic traffic, and even the links the competition is getting that help them drive referring traffic and rank on Google.
The potential behind uncovering these insights is only in line with how far you dig. With larger sites, the amount of data you can now pull is so substantial that an SEO team can drive dozens, if not hundreds of insights and task ideas from every single competitor.
I’ve been keeping an eye on the progress of competitive intelligence for SEO for the past decade, and I truly believe that doing it well isn’t about copying someone else, as much as about finding shortcuts to learning what works in your very own niche.
During 2019, I had the honor of speaking on the topic twice, first at SMX London, and then at an SEMRush Meetup in Tel Aviv.
On top of the very popular gap analysis for content, keywords, and links–here are some of my additional, less common, competitive intelligence tactics for SEO:
1 – Identifying Competitors Per Topic
No matter which SEO tool you’re using, it will be identifying your SEO competitors by the keywords both you and them are ranking for.
But not all of your competitors are direct competitors that offer the exact same solution as you do. Very often in SEO, we’ll see competitors who could be competing with you only for a single group of keywords, or a large topic. Perhaps on a subfolder basis.
You should map all the topics you cover, and make sure you aren’t missing the relevant competitors for each of them. The insights behind each of these competitor groups might just be as powerful as the insights uncovered from your direct competitors.
Ahrefs: Kayak.com SubFolders
2 – Understanding Each Competitor’s Product Strengths and Weaknesses
SEO tools focus on URLs, not really on site features or the Product itself. One of the first things you want to do when approaching a new competitor is simply trying what it has to offer. Use its features, register to the site, even buy from them if that’s what’s required to get the feeling of the full experience. You don’t want to be the type of SEO who studies competitors only through third party tools.
I was recently helping a Recipes website with their SEO. Instead of just telling them to improve their recipe pages, I decide to give one of their top competitors a try and make a pancake with one of their recipes myself. Fine, so it’s only a pancake, but it was enough. Only when I went through the whole experience as a user, I was able to understand how that recipe page could have been easily improved.
For example; adding alternative ingredients that could have worked just as well, adding ingredient quantities to the actual instructions once again, so that I won’t have to scroll up on my phone every time just to make sure I remember the exact number of teaspoons required, or even Cup to Gram conversions for the non-American users like myself.
While SEOs are not officially Product managers, Google has been pushing us in that direction for quite some time now. To a point I (almost) like calling it “SEO is the new Product”. In short, as Google’s focus is providing the best possible results to satisfy their users, making awesome products is now the key.
And as Kevin Indig recently wrote, “SEOs have to identify these features and bring them to product teams.”
Identifying your competitors’ Product strengths and weaknesses, then pushing your Product teams to build and improve your features when compared with your rivals, is now a big part of the SEO game.
To help you identify these strengths and weaknesses, you may use a concept often used by UX experts before launching new things. The Usability testing. In our case, you may simply start with the search query and try accomplishing specific tasks, while asking:
How do I accomplish this?
Does it require registering or paying?
How long did it take to find X ?
And in the end of the day… was this actually helpful?
Then, how do you rate the overall experience per competitor, per query?
3 – Running Content Audits
The idea behind SEO content audits is simple. You check your indexed content and analyze its performance with popular metrics. This concept can similarly work with competitors too.
The purpose is not only to understand what works for them, but mainly why, and find correlation between traffic, rankings and specific things they do differently. No doubt that auditing the content of competitors goes hand in hand with your competitors’ product strengths and weaknesses.
When you’re analyzing a specific SERP and you simply don’t understand how come this website outranks you, a content audit per competitor can really help you understand what’s going on and what needs to be improved on your end.
Here’s my list of metrics to check per competitor. Remember, you may rate these as you wish, since it’s all relative and in comparison to your site/page.
Content Type – What type of page comes up? – (This might be different from one industry to another).
Freshness (Frequency of updates)
Topic’s On-Page Optimization and URL structure
Total Referring Domains/Total Backlinks
Shares/Traffic/Popularity
Internal Linking – Information Architecture
Page Speed (Desktop/Mobile), other technical elements
Satisfaction (And whatever you define as satisfaction as a user)
Mobile Look and Feel: Both appearance and functionality
Volume of Branded Searches (Mainly for the topic itself)
4 – Crawling Through Competitors
Obviously, you won’t have access to your competitors’ GA or GSC accounts. But on top of today’s tools that may partially replace them, you can also crawl your competitors’ sites.
While some sites will be blocked for 3rd party crawling, many will not be. For those sites that are blocked, cloud base Crawlers such as DeepCrawl offer solutions for that too. DeepCrawl’s ‘Stealth Mode’ feature crawls slowly using a large pool of IPs and user agents as a work around to such blocking.
DeepCrawl Stealth Mode
Here are a few examples for the insights you may get by crawling competitors:
Their set up for page titles, descriptions – on a large scale.
URL structure, length, and crawl depth (Meaning, how ‘far’ are your competitor’s pages from their homepage or other relevant main sections?).
Pages they decide to Noindex, Nofollow.
Number of external links, number of internal links, word count, HTML size.
Linked Domains (domains they are linking to). Total number + which sites.
All broken URLs (then see who links there)
Number of unique pages.
How do they handle canonicals, duplicate titles, descriptions, pagination, 301s, 5xx errors, Non-HTML Pages, sitemaps, mobile, their site speed and fetch time compared to you, and many more.
Generally speaking, the more competitors’ sites you’ll crawl, the better you will understand how Google ranks sites in your specific industry. You’ll stumble upon pages that rank well despite having thin content, bad on-page structure, or a low number of links, and you’ll see the opposite–pages that are technically “Perfect”, yet nowhere to be found on Google. Also, just seeing how good or bad your rivals are at Technical SEO is both insightful and fun.
5 – Asking Your Own Audience
When it comes to understanding what works best for your competitors, there’s nothing like asking their own users. And if your site is already big enough to have a substantial audience, one of the easiest ways to go would be to ask your own users, since at least some of them are already using your competitors’ sites too.
There are many advantages to having a direct way of communicating with your top users, those users that you can call brand ambassadors. Your product’s inner circle.
One of these advantages is simply the fact that you can ask them directly what works better for your competitors, what features to improve, what content, inventory, or tools you must be adding as soon as possible. Ask for their help to make your site is better than what your competitors are able to offer.
6 – Understanding How Strong Their Brand is
You Search Console’s branded searches data can tell you a lot about what your users are thinking of when having your brand in mind. Or in other words, what your brand is strongly known for. Try that. You will also be fascinated by the correlation with the topics your rank well for.
If you have access to keyword search volume, you can track your competitors’ branded searches too. See if you can answer the following questions:
What’s the share of your branded searches compared with your main competitors?
What happens to this metric over time?
What are your competitors really known for by their existing users?
Whose brand is stronger, per topic?
SEMRush – Amazon.com’s Top Keywords
7 – Tracking Their Traffic to Spot New Trends and Opportunities
Looking at your competitors from time to time to see what changes they are going through is always a good idea. But this isn’t a tactic we can trust over time.
With today’s tools on the other hand, tracking the total traffic of your competitors is very simple. By checking that traffic number once a month, you’ll be able to see if there’s something interesting to dig into. New trends your competitors are on, new pages, sections, keywords, or any other opportunities they might have their hands on, will be revealed if you’ll just pay close attention.
Of course, the more competitors you track that more interesting it becomes, and with larger sites–you’ll have to break down their traffic. Otherwise, some trends might get muddled with other traffic changes including fluctuations in rankings. For example, you could break the traffic of such sites by subfolders or countries for a better view of what’s going on.
8 – Following Them to Predict Their Plans
Many of the interesting things you’d want to know about your competitors and their future plans are announced directly by them.
You can easily pay attention to who they are hiring, to the type of reviews they are getting, and to their surveys, newsletters, blog posts, social media posts or product beta versions.
These will often bring you insights about the things they are planning and are already working on.
For example, we knew that one of our top competitors is working on an app to compete with ours eight months prior to their release date. Simply because they published that they’re looking for an iOS developer.
9 – Finding Their Top Traffic Referrals
While the main reason SEOs focus on getting links is to improve rankings, getting large referrals has the potential for some very substantial amount of traffic.
Some interesting examples that worked very well for us at Investing.com in the past few years:
Drudge Report: This one-pager news aggregation website is sending tons of traffic to news publications. I’ll never forget the day Trump was elected and Drudge Report added a link to one of our US Index Futures pages. We received over 200K new users in less than 8 hours.
Flipboard: Another news aggregator platform, with a similar concept to Feedly. Still sending us hundreds of thousands of users every month.
Wikipedia: While a bit tricky and you really need to know what you’re doing when editing Wikipedia pages, if your competitors are getting significant traffic from Wikipedia, you want to know how and why. The References and External links areas have a lot of potential if done correctly, as these will be sending you traffic for a very long time.
For referrals, there are two interesting Features by SimilarWeb worth mentioning.
A. Outgoing traffic: So instead of just listing the sites that refer traffic to each of your competitors, you can look at top global referrals (or by country of interest), and see if any of your competitors are already there. If yes, you definitely want to be aware of it and see if you can do that too. This Outgoing Traffic report saves tons of time.
SimilarWeb – Outgoing Traffic
B. Comparison Mode: This is where you can compare your own website with up to 4 competitors. Here, SimilarWeb offers a visualization that helps you spot which sites are sending traffic to a group of your competitors, but not to you!
Of course, with some Excel work and without this visualization, you can still get the same data with other tools.
SimilarWeb – Referrals Comparison
So which competitive intelligence tactics work best for your SEO?