The robots.txt file is an important part of search engine optimization (SEO) that can help improve your web pages’ visibility in search engines. This article will describe what the robots.txt file is, how it works and how you can use it to improve your site’s SEO. We’ll also look at some best practices and tips for making the most of this powerful tool.

What Is Robots.txt?

Robots.txt is a text file that tells bots, including search engine crawlers, which pages and files on a website they should be able to see and index. Its main job in SEO is to instruct search engine crawlers like Googlebot not to look at certain pages. 

For example, you might want to tell them not to look at the category pages of your blog or online store because you don’t want those pages in search results. Other uses include preventing the indexing of duplicate content and other files you may not want indexed, including scripts and images.

Robots.txt can help ensure that your site’s crawl budget is used efficiently. Search engines tend to limit the number of pages they crawl each day. Site owners often want to focus their crawl budget on specific pages, and the robots.txt file can be used to ask Google not to use it on less important pages.

Does Robots.txt Stop Bots from Indexing Pages?

Well-behaved bots follow the directives in your robots.txt file. However, not all bots are well-behaved, and those that aren’t crawl whatever they like. Fortunately, Google and other major search engines usually obey robots.txt instructions.

But even if you use your robots.txt file to stop crawling, Google will still be able to see the pages via external links, so they may end up in search engine results anyway. If you want pages to be on the web but invisible, Google suggests password-protecting them. 

You can also use the noindex tag to prevent indexing. But don’t use robots.txt and noindex together to block pages—if Google can’t crawl a page because it is blocked in robots.txt, it can’t see the noindex directive and therefore may index the page and show it in search results. 

Useful Robots.txt Directives for SEO

Here are four robots.txt directives you may find useful for search engine optimization:

1. User-agent: * – Directives are divided into blocks, each of which begins with a user-agent line indicating which user-agent (crawlers, browsers, and so on) it applies to. The asterisk (*) indicates all user-agents, but you can name specific agents like “Googlebot.”
2. Disallow: /cgi-bin/ – This directive prevents search engine bots from crawling and indexing the files in the cgi-bin folder. 
3. Allow: /images/ – This directive tells search engine bots that they are allowed to crawl and index the files within the images folder. Allow directives are used to override disallow directives. For example, you may want to allow access to a folder inside a disallowed parent folder. 
4. Sitemap: http://example.com/sitemap.xml – This directive specifies which sitemap should be used, helping bots discover new content faster and more accurately.

In a robots.txt file, these directives would look like the following:

User-agent: *

Disallow: /cgi-bin/

Allow: /images/

Sitemap: http://example.com/sitemap.xml

Make Sure Your Robots.txt File Is Error-free

A word of warning: Make sure your robots.txt directives precisely follow the specification. Mistakes result in unexpected behavior that could hurt your site’s SEO. To be sure, run your robots.txt file through a validator.

Boxing week is just about here. You’ve sprinkled your store with decorations, you’ve decided which products you want to put on sale, and you’ve done the necessary legwork to spread the word about your upcoming promotions via email, paid advertising, and social media. But is your website ready for the influx of traffic it’s likely to receive?

Don’t panic if the answer is no. If you’ve laid the groundwork with promotions and sales, you’ve already done the heavy lifting. All that’s left is the digital equivalent of making sure a brick-and-mortar store is clean and accessible the night before a major event. 

At the end of the day, that’s really all search engine optimization (SEO) involves. With that in mind, we’ve put together a checklist of last-minute SEO practices and tactics to prepare your store for its boxing week sale. 

Imagery

• File names and alt text both use keywords describing the product as plainly as possible. 
• All image files are properly sized (per Shopify).
  • A maximum of 1 megabyte (MB) for header and background images.
  • A maximum of 300 kilobytes (KB) for product photos.
  • 100 KB or less for product thumbnails.
  • A maximum of 10 MB for hero images/feature photos. 
• All image files are uploaded in the proper aspect ratio.
  • 16:9 for hero images.
  • 1:1 for product photos and thumbnails.
• Files are uploaded as jpegs. 
• All product photos are of reasonable quality and give an idea of how each product looks. 

Product Pages

• Product titles include all relevant information, including manufacturer name, model/color, and SKU.
• Product descriptions provide a complete overview of what each product is and how it works. 
• Each product page has a unique description.
• Product page URLs are simple and easy to remember. 
• All product pages use the Product schema. 
• Each product page is tagged with all relevant keywords for internal searches. 
• User reviews are enabled. 
• You’ve used a keyword research tool to determine the best keywords for each product. 

General Performance and Usability

• You’ve tested your website with Google PageSpeed Insights and made any recommended adjustments.
• You have asked your web host if it’s possible to scale up your bandwidth for the week. 
• All content is optimized for mobile devices. 
• Site navigation is streamlined and intuitive, allowing users to find what they’re looking for in as few clicks as possible. 
• There are minimal interruptions to the user experience—no pop-ups or pop-over prompts. 
• You offer multiple payment and shipping options at checkout.
• Your website features an SSL certificate during checkout. 

Conclusion

Too many e-commerce website owners kill it when it comes to off-site promotion while neglecting the basics of on-site optimization. With the checklist above, you can ensure that you aren’t one of them—for boxing week as well as every event that follows. 

We’re going to assume you’re already familiar with the four major search intents—transactional, informational, navigational, and brand.  What you may not know is that as search engine optimization has continued to evolve, those intents have slowly started to lose relevance. To put it another way, a general idea of what a searcher wants is no longer sufficient. 

You need specifics. What precise content does someone with informational search intent want to find? At what stage of the buyer’s journey is a user with transactional/commercial intent? 

That’s where micro-intents come in. As noted by Search Engine Land, these subcategories of traditional search and user intents offer a better understanding of audience expectations. And that, in turn, can be used to create more targeted, higher-value content.

We’ve listed the different micro-intents below for posterity.  

Informational Micro-Intents

• Entertainment. Entertainment-focused content is all about passing the time. It can take many different forms but is typically easily digestible. 
• Definition. If you want an example of definition-focused content, you’re reading it. You came here because you wanted to know what micro-intents are. 
• Expansional. This type of content is similar to definition-focused but takes a much deeper, more comprehensive dive. 
• Enablement. Typically, enablement content takes the form of how-to articles and videos. 
• Overview/Aggregation. Similar to definition/expansional, overview content takes a high-level look at a topic. Examples include infographics and listicles.  

Transactional/Commercial Micro-Intents

• Comparison. The user is looking to buy but wants to determine which product/brand is the best before they do so. 
• Category. The user has a general idea of the product or service they want but is still trying to decide on a specific solution. 
• Product. The user is at the bottom of the funnel and is about to make a purchase—they’re looking to find out a bit more about a product or service before they finalize their decision.  

Brand Micro-Intents

There are actually no specific brand micro-intents to speak of. Generally, branded searches can be positioned under one of the other three categories. With that said, branded searches are typically looking for reviews, testimonials, or other information—anything that one would typically use to build trust. 

Navigational Micro-Intents

• Support. The user is currently a customer and is looking for help with their product or service. This could take the form of instructional articles, a product-related knowledge base, or contact information for a support professional.
• Website. Self-explanatory. The user wants to find a specific page, blog post, or social channel. 
• Location. Similar to the previous intent, except that they’re looking for a location in the real world rather than online. 

How Do You Determine Micro-Intents? 

The good news is that micro-intents are no more challenging to determine than high-level intents. Simply pay attention to specific phrases or keywords, and use your best judgment. From there, it’s simply a matter of creating content that’s more directly targeted. Couple this with the information we provided on identifying keyword targets, and you’re sure to find success in your SEO efforts.

With over one billion monthly active users, TikTok is among the most popular social networks in the world, eclipsed only by Facebook, YouTube, and Instagram. It’s also rapidly becoming the social app of choice for millennials and generation Z, who together make up roughly 75% of its global base. Suffice it to say, if your brand is looking to target either of those audiences, it’s in your best interest to gain a foothold. 

That’s actually not as difficult to do as you might expect—or at least, it’s no more or less challenging than any other social network. As is usually the case, success starts with a solid marketing strategy and a decent understanding of search engine optimization.  With that in mind, here are the three most important things you must understand about TikTok SEO.  

The Basics of Topic Ideation Remain the Same

Plenty of brands treat social media as some sort of arcane, unknowable entity, as if they’re convinced they’ll need to reinvent the wheel in order to succeed. The reality, however, is that the foundation of TikTok SEO, topic ideation, is the same as on any other site. To summarize: 

• Understand your audience—who they are, what they’re looking for, and why they might be interested in your brand. 
• Leverage sites like Quora and tools such as Answer the Public to research the questions your audience is asking and the topics that interest them.  
• Pay attention to how any topics you identify relate to your brand’s niche. 
• Identify the specific keywords in those questions and topics, then leverage them in your TikTok videos, captions especially. 
• Be respectful of your audience’s time. There’s rarely a reason for a TikTok video to be longer than a minute. 

Trends and Hashtags Are Crucial for Discoverability

Especially with Google’s eventual plan to index TikTok video content, choosing the right hashtags and trends goes a long way toward increasing your brand’s reach and visibility. Successfully leveraging more popular trends can generate a torrent of views and engagement, potentially even causing your page to go viral. Provided you’ve targeted the right users and linked your TikTok channel back to your website, that also has the potential to lead to a high volume of conversions. 

Quality and Consistency Always Win

The phrase ‘content is king’ may be overused, but that doesn’t make it any less true. High-quality, relevant content is the most important factor in your success on TikTok—or really, on any channel. That means crystal-clear audio and video and a well-written script geared towards your audience. 

Without those characteristics, it doesn’t really matter how well you optimize your TikTok videos from a search perspective. Reach is all well and good, but only if it’s attached to content that people actually want to watch. Always keep that in mind.

And above all, remember to always create content with purpose. 

If there’s one thing about any industry that always holds true, it’s that people love their buzzwords. And while such industry parlance can make things easier for someone who’s been around for a while, it can be downright overwhelming for newcomers to the field. That’s why we intend to keep doing our part to clear the air—to walk people through all the definitions bandied about by marketing professionals.

To that end, today we’re going to discuss the difference between horizontal content and vertical content. As you’ve probably guessed, these two phrases have a lot in common with business markets, at least in that they define scope in a very similar manner. 

What Is Vertical Content? 

The clue is in the name—vertical content is, quite fittingly, a catch-all term for any piece of content targeted at a specific industry, market, or niche. Typically, when someone discusses vertical content, they’re specifically focused on business to business marketing. However, this need not always be the case.

A vertical is any specialized niche with a highly-defined audience. A healthcare cybersecurity publication and a Portland-based craft brewing business will both publish vertical content as part of their inbound marketing efforts. The thing that defines vertical content is specificity.

Its audience is incredibly well-defined, and it’s incredibly unlikely that anyone outside of that demographic will have any interest in a vertical-focused business or its content. 

What Is Horizontal Content?

While vertical content is highly specific, horizontal content is a little more generalized. Rather than targeting an exclusive market or sector, horizontal content focuses on broader problems or needs. Again, although horizontal content is frequently targeted at the consumer sector, there are plenty of business audiences for which this content is beneficial, as well.

With that said, since horizontal content is by its nature a bit more nebulous than vertical content, we’ll offer up a few examples of topics that might typically have a horizontal focus:

• General cybersecurity.
• Consumer electronics.
• Grocery. 
• Renovations.
• Home decor. 
• Health and wellness. 
• Productivity. 

Choose the Best Content for Your Needs

It’s important to understand that neither of these content types is superior to the other. Some businesses are far better suited for horizontal content, while others, by their nature, will see the most success with vertical content. It’s also worth noting that in most cases, you can take a horizontal topic and narrow it down to a particular vertical and vice-versa. 

The best advice we can give you here is to first define your customer base and their needs. Once you understand who you’re marketing to and what their interests are, it will be a far simpler matter to figure out what time of content to publish. Just remember that at the end of the day, what truly matters is that you provide value to your audience—everything else is secondary to that. 

As the old saying goes, if it ain’t broke, don’t try to fix it. 

Unfortunately, that applies to the tactics used by cybercriminals just as much as it applies to legitimate businesses. It’s why distributed denial of service (DDOS) attacks have been around for decades. And it’s why ransomware has changed so little over the years.

Sure, we’re seeing more sophisticated distribution tactics, but ultimately modern ransomware works nearly the same as ransomware from a decade ago.

With that in mind, it appears another blast from the past has started to re-emerge. At least a few of you probably remember the early days of search engine optimization (SEO). Back when search engines were akin to a virtual wild west, black hat tactics were the best way to rank. 

What Is SEO Poisoning? 

You likely also recall how many malicious websites rose to the top of the search engine results page (SERP), abusing SEO to serve poisoned results to users, hence the term SEO poisoning. It didn’t take long for Google and other search engines to release algorithm updates that shut down most of the shadier tactics. But just like life, criminals always find a way. 

Today, SEO poisoning takes a different form.

First, criminals create a website and take great pains to make it look as legitimate as possible. From there, they begin ‘trend chasing,’ leveraging their understanding of SEO in an effort to gain a prominent position on the SERP. There’s no limit to the number of keywords a single malicious domain may target in this manner.

Security firm Websense Security Labs estimates that these malicious websites represent as much as a quarter of the first page of search results for trending topics.

Generally, the objective is fraud or identity theft. Hackers will use the poisoned sites to steal the personal details of unsuspecting users. They might also inject ransomware onto a victim’s system, add another node to a botnet, or — if they’re lucky enough to infect a PC belonging to a webmaster — hijack another website to add to their malicious network. 

How Do I Protect Myself From SEO Poisoning? 

As with many types of cyberattacks, a little mindfulness goes a long way.

• Be incredibly wary of opening a website you’ve never heard of before, particularly if you’re searching for a trending topic.
• We also strongly recommend installing an up-to-date antivirus.
• Use a password manager for both your personal accounts and your business accounts.
• Keep all your software and systems up to date. 
• It may be worthwhile to use a VPN or invest in a router that has built-in encryption. 
• Consider installing an ad blocker and blocking scripts, as ad networks and malicious scripts are two prevalent delivery mechanisms.

If you own or operate a website, the same rules apply — remain vigilant, and put in the necessary work to keep your personal files and your professional data safe. 

As with any business initiative,  it’s crucial that you understand how to quantify the success of your search engine optimization efforts. Not only do you need to show leadership that your budget is generating tangible results, but you also need to calculate SEO’s return on investment for your own purposes. Consistent measurement of ROI can also help you identify weaknesses and shortcomings, and help you determine where your time (and money) should be directed. 

But how exactly do you measure the ROI of SEO? 

Nebulous Returns

From an ROI perspective, the issues with SEO are similar to those with marketing. Namely, while there are certain fixed costs and returns, as a whole, the core goals of SEO are difficult to express in concrete numbers. Concepts like brand awareness and organic visibility are inherently abstract.

With that said, it is possible to at least approximate them. 

SEO Metrics That Measure ROI

The first step in determining ROI is to figure out how much you’re spending on SEO. If you’ve hired a third-party agency, this is relatively easy to calculate. Just look at whatever you’re paying them. 

If you’re managing SEO internally, things get a bit more complicated, and you’ll need to look at a few different factors. 

• How much time your staff is spending on SEO. This includes developers, designers, marketing specialists, etc. 
• How much per hour each staff member is paid, on average. 
• Subscription costs for any tools or platforms you’re using to inform your SEO efforts. 

With those numbers in mind, measure the following key performance indicators (KPIs) from the beginning of your SEO campaign to its end: 

• Organic traffic. Self-explanatory. Traffic generated from the search engine results page (SERP).
• Bounce rate. The number of people who visit your site and leave without performing any actions. Can be paired with time on site to identify potential bottlenecks. 
• Organic impressions. How many people have seen your site on the SERPs. 
• Organic click-through rate. The number of users who clicked your site on the SERP, measured against total impressions. 
• Pages per session. How many pages a user views, on average. 
• Conversions. Here’s where things get a bit complicated, as there are multiple ways you might define conversions. 
  • Sales. 
  • New subscribers. 
  • Sign-ups for mailing lists/asset downloads. 
  • Downloads. 
  • Social shares. 
  • Phone calls. 
  • Demo/proof of concept requests. 

One way to monitor the above is through conversion tracking. Google allows you to define certain actions as conversions, while also assigning a dollar value to each. Although this is typically intended for Google Ads, it can easily be applied to your site. 

Expressing the ROI of SEO

So, in light of the above, you can express the value of your SEO efforts in a few different ways: 

• Percent increase/decrease. Applies to bounce rate, organic traffic, impressions, clickthrough, etc. 
• Spend vs. Revenue. Specifically applies to conversions that involve monetary transactions. 
• Volume. How much more traffic/how many more phone calls you’re receiving now versus when you started.