# How search engines and AI crawlers index your hosted social experience

Genuin generates the metadata, sitemap, and robots.txt that let search engines and AI crawlers index a hosted social experience, built from the content each brand supplies.

Your hosted social experience lives on its own domain, either `<subdomain>.begenuin.com` or the domain you white-labeled. Every page on it is served to crawlers the same way it's served to people.

Genuin generates the page metadata, a sitemap, and a robots.txt for the domain. You supply the names, descriptions, and images they're built from, and you submit the sitemap to the search engines you care about.

That's how content in your social experience reaches traditional search engines such as Google Search and AI-driven discovery systems such as ChatGPT and Perplexity. Onsite is the product behind your hosted social experience; for the whole product, see [Onsite: what it does and where to start](https://begenuin.com/developers/browse-by-products/onsite/onsite-what-it-does-and-where-to-start). For the domain setup, see [White-label your hosted social experience](https://begenuin.com/developers/browse-by-products/onsite/how-can-you-white-label-your-community-under-your-own-domain).

## What each page exposes

Every page carries a `<title>`, a `description`, Open Graph tags, Twitter card tags, and an `x-brand-id` tag set to your brand username. Video pages carry considerably more, because that's where search engines and link previews get the most value.

| Page | URL | Title and description built from | Open Graph and Twitter | Also |
| --- | --- | --- | --- | --- |
| Home (subdomain page) | /home (the root redirects here) | Brand name and brand bio | og:title, og:description, og:image (your brand preview image); twitter:card summary_large_image | — |
| Community | /community/<handle> | Community name and description | Same as home plus og:url; og:image is generated from the community's name and image | Server-rendered links to the community's videos |
| Group | /group/<handle> | Group name and description | Same as community | Server-rendered links to the group's videos |
| Video | /video/<slug> | The post's caption, the poster's name, and your brand name | og:type video.other; og:video, og:video:url, og:video:secure_url (the MP4), og:video:type, og:video:width and height (1080×1920); og:image (a generated 1084×546 share image); twitter:card player with twitter:player and its dimensions | Canonical URL; robots index, follow, max-video-preview:-1, max-image-preview:large, max-snippet:-1; JSON-LD VideoObject with Person, WatchAction, and InteractionCounter; links to the next videos in the feed |
| User profile | /profile/<nickname> | The member's name and @nickname | og:title, og:description, og:url, og:image | — |
| Brand profile | /brand/<handle> | The brand's name and bio | og:title, og:description, og:url, og:image | — |

### Text of the title and description tags

Genuin fills the tags from templates. The description template also fills og:description and twitter:description, and the title template fills og:title and twitter:title.

| Page | Description | Title |
| --- | --- | --- |
| Home (subdomain page) | Brand's bio; twitter:description uses the meta description for sharing the brand URL | Brand's full name; twitter:title uses the page title for sharing the brand URL |
| Community | Community description | Community name |
| Group | <Group Description> \| Join <Group Name> to talk about it | Group name |
| Video | Watch videos from <Full Name of the poster's profile> on <Brand Name> | <Video Description> • Watch and react on <Brand Name> |
| Brand profile | Meta description for sharing the brand URL | Page title for sharing the brand URL |
| User profile | <User Full Name> (@username) on <Brand Name> | <User Full Name> (@username) is on <Brand Name> |

Where a page has og:url, it's the page's own URL: the community, group, video, brand page, or user profile URL. On video pages, og:video, og:video:url, og:video:secure_url, and twitter:player all point at the video. Together, these tags prepare content for search indexing, social sharing, and rich media previews.

Two consequences follow. First, the video page is the unit of search visibility. It's the only page with a canonical URL, structured data, and a `robots` directive. Its server-rendered HTML holds the caption and links to neighboring videos, so a crawler can walk from one video to the next.

Second, the home, community, and group pages render most of their content in the browser. A crawler that doesn't run JavaScript sees their metadata and video links but little text, so the sitemap, not the home page, is how crawlers find your content.

## Page performance

Genuin tests the pages of your hosted social experience with [PageSpeed Insights](https://pagespeed.web.dev/) against search engine performance benchmarks. PageSpeed measures two of the metrics that matter for SEO:

- **Largest Contentful Paint (LCP)**: loading performance.
- **Cumulative Layout Shift (CLS)**: visual stability during page load.

Run the same test on your own domain to see current values for your pages.

## The sitemap

Every domain serves a sitemap index at `/sitemap/index.xml`, and robots.txt points to it. All community content is crawlable, and the index lists one sitemap per page type, paged at 1,000 URLs per file.

| Sitemap | Lists | changefreq / priority |
| --- | --- | --- |
| /sitemap/static/index.xml | /home, /popular, /latest, /explore | daily / 1.0 |
| /sitemap/community/index.xml?page=N | Every active community | weekly / 0.8 |
| /sitemap/group/index.xml?page=N | Every active group | weekly / 0.8 |
| /sitemap/video/index.xml?page=N | Every published video, as a video sitemap entry with video:thumbnail_loc and video:content_loc pointing at the thumbnail and the MP4 | — |

The video entries follow Google's [video sitemap](https://developers.google.com/search/docs/crawling-indexing/sitemaps/video-sitemaps) format. `lastmod` is the date the community, group, or video was last updated.

Profile and brand pages aren't listed; crawlers reach them through links. Genuin generates the sitemap on request from the live database, so a new community or video appears the next time a crawler fetches the file. There's nothing to rebuild.

If you need the sitemap turned off, Genuin can disable it for your brand, and the files then return 404.

## robots.txt

Every domain serves `/robots.txt`. The default allows every crawler and names the sitemap:

```text
User-agent: *
Disallow:
Sitemap: https://<your-domain>/sitemap/index.xml
```

There's no per-path disallow by default, so search engines and AI crawlers can fetch every page. For how robots.txt works, see Google's [introduction to robots.txt](https://developers.google.com/search/docs/crawling-indexing/robots/intro).

Genuin keeps a per-brand list of extra robots.txt rules. To block a crawler or a path, send the rules you want to [support@begenuin.com](mailto:support@begenuin.com), because Business Account has no robots.txt editor.

Rules name a crawler by its user agent. For example, this set blocks every crawler from `/private/` and allows OpenAI's GPTBot everywhere else:

```text
User-agent: *
Disallow: /private/

User-agent: GPTBot
Allow: /
```

This set blocks `/admin/` and allows PerplexityBot:

```text
User-agent: *
Disallow: /admin/

User-agent: PerplexityBot
Allow: /
```

OpenAI lists its crawlers in [Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots), and Perplexity in [Perplexity crawlers](https://docs.perplexity.ai/docs/resources/perplexity-crawlers). Crawlers re-read robots.txt on their own schedule, so a change can take up to 24 hours to take effect.

## Submit the sitemap

Search engines find sitemaps on their own, but submitting yours gets the first crawl sooner and gives you an index report. Submit it to Google Search Console and Bing Webmaster Tools; Bing's index also covers DuckDuckGo, and Brave takes URLs.

Submit `https://<your-domain>/sitemap/index.xml` once per domain. If you white-label later, submit it again for the new domain.

### Submit to Google Search Console

1. Go to [Google Search Console](https://search.google.com/search-console/about) and sign in with a Google account.
2. Add your domain as a property.
3. Verify ownership with a DNS record. Search Console also offers HTML file upload and Google Analytics as verification methods.
4. Go to **Indexing** > **Sitemaps**.
5. Enter `sitemap/index.xml` as the sitemap URL.
6. Click **Submit**.

Google processes the sitemap and shows its status. Search Console shows the discovered URL count per sitemap within a day or two, and the **Pages** report then shows which URLs are indexed.

### Submit to Bing Webmaster Tools

1. Go to [Bing Webmaster Tools](https://www.bing.com/webmasters) and sign in with a Microsoft account.
2. Go to **Sitemaps**.
3. Click **Submit Sitemap**.
4. Enter the same sitemap URL.
5. Click **Submit**.

### Import your site from Google Search Console (Optional)

Bing can import properties you've already verified in Google Search Console instead of verifying them again.

1. In Bing Webmaster Tools, click **Import your sites from GSC**.
2. Allow Bing to access Google Search Console.
3. Authenticate with your Google account.
4. Select the websites to import.
5. Confirm the import.

Bing then fetches the sitemap automatically.

### Submit to Brave Search

1. Open the [Brave submission page](https://search.brave.com/submit-url).
2. Enter your website URL.
3. Click **Submit**.

Brave accepts URLs through this form but doesn't take sitemaps.

### Get indexed by DuckDuckGo

DuckDuckGo doesn't accept sitemap submissions. It takes its web index from Bing, so submitting your sitemap to Bing covers it, and backlinks from trusted websites help too.

## Build links from other sites

Your hosted social experience supports offsite SEO strategies, the work beyond its own pages: backlink strategies, social sharing, cross-platform content distribution, and the referral traffic they bring. These build your Domain Authority (DA), which strengthens organic rankings.

## AI crawlers and answer engines

AI assistants discover content the same way search engines do: by crawling pages that robots.txt allows and reading their text and structured data. ChatGPT uses OpenAI's GPTBot, and Perplexity uses PerplexityBot.

All six page types are open to AI crawlers: the home (subdomain) page and the community, group, post, user profile, and brand profile pages. The default allow-all rule applies to them, and the video pages, with their captions and `VideoObject` data, are what they read.

Making content crawlable and structured for LLMs is sometimes called LLM SEO (LEO). On the hosted side, nothing more is needed. Learn what AI engines read on each video page in [Make your social experience discoverable by AI assistants](https://begenuin.com/developers/browse-by-products/onsite/getting-discovered-on-llm).

### Publish llms.txt on your own website

Hosted social experience domains don't serve `/llms.txt` or `/llms-full.txt`; a request for either redirects to the home page. Publish the files on your own website and link your hosted social experience URLs from them.

The /llms.txt file is an industry convention, defined at [llmstxt.org](https://llmstxt.org/), that gives AI systems a short guide to a website. The [llms.txt directory](https://directory.llmstxt.cloud/) lists websites that publish one, and [an explainer on Towards Data Science](https://towardsdatascience.com/llms-txt-414d5121bcb3/) covers the format.

To publish llms.txt:

1. Create a file named `llms.txt` in your website's root directory.
2. Add a brief description of your website.
3. Add instructions for LLM systems.
4. Link to detailed documentation.
5. Publish the file at `yourdomain.com/llms.txt`.
6. Update the file regularly.

### Publish llms-full.txt on your own website

The /llms-full.txt file carries more complete information for AI systems, such as complete documentation, content guidelines, technical references, and detailed sitemap information. To publish llms-full.txt:

1. Create the file in your website's root directory.
2. Add detailed background information.
3. Add extensive instructions for AI systems.
4. Link supporting documentation.
5. Publish the file at `yourdomain.com/llms-full.txt`.
6. Maintain and update it regularly.

### Add the files to your sitemap

The two files aren't traditional web pages, but listing them in your website's sitemap.xml helps AI systems find them. It also gives LLMs clearer instructions and a better picture of your website's structure. For example:

```xml
<url>
<loc>https://yourwebsite.com/llms.txt</loc>
<lastmod>2026-01-01</lastmod>
<changefreq>monthly</changefreq>
<priority>0.8</priority>
</url>

<url>
<loc>https://yourwebsite.com/llms-full.txt</loc>
<lastmod>2026-01-01</lastmod>
<changefreq>monthly</changefreq>
<priority>0.8</priority>
</url>
```

### Write for generative and answer engines

Generative Engine Optimization (GEO) optimizes content for AI search systems that generate summarized answers. In a hosted social experience, it means well-structured content, organized groups, clear topic categories, and high-quality discussions. Learn more in Search Engine Land's [generative engine optimization strategies](https://searchengineland.com/generative-engine-optimization-strategies-446723).

Answer Engine Optimization (AEO) aims at appearing as a direct answer in search engines and AI assistants: featured snippets, AI-generated responses, and voice search answers. An llms-full.txt file supports AEO by giving AI systems structured content to extract answers from. Learn more in Surfer's [guide to answer engine optimization](https://surferseo.com/blog/answer-engine-optimization/).

## What you control

Genuin generates the tags, but their content is yours:

- **Brand name and bio** (**Settings** > **Brand Profile**) become the home and brand page title and description.
- The bio is optional and takes up to 150 characters.
- **Brand preview image** (**Brand Elements**) is the og:image for the home page and the image link previews show.
- The page title and meta description for sharing the brand URL come from **Image for Shared URL**. It's under **Customize Web CTAs** in **Settings** > **Brand Elements**, where **Preview Metadata** shows the card a shared brand URL produces.
- **Community and group names and descriptions** become their page titles and descriptions, and Genuin builds the generated share image from the name and image. A one-word description produces a one-word search snippet.
- In your Business Account, a community's description is its **Purpose of this community** field. A group's **Description** takes up to 300 characters, and its name up to 40.
- **Video captions** become the video page's title, its description, and the headline in search results. Write them as sentences, not hashtags.
- **Your domain**: every URL in the sitemap and every og:url uses the white-label domain once it's validated. Submit the sitemap after switching, not before.

## Limitations

- **Client-rendered listing pages**: home, community, and group pages expose metadata and video links to crawlers, but not their visible feed text.
- **No canonical tags outside video pages**: a community reached through both the subdomain and the white-label domain is two URLs to a search engine. That lasts until the white-label domain is validated and links switch.
- **Profile and brand pages aren't in the sitemap.**
- **No self-serve robots.txt or sitemap settings**: changes go through Genuin.
- **No llms.txt or llms-full.txt** on hosted social experience domains.

## See also

- [Make your social experience discoverable by AI assistants](https://begenuin.com/developers/browse-by-products/onsite/getting-discovered-on-llm)
- [Optimize your social experience for search and LLM discovery](https://begenuin.com/developers/browse-by-products/onsite/how-does-genuin-optimize-brand-communities-for-seo-and-llm-discoverability)
- [White-label your hosted social experience](https://begenuin.com/developers/browse-by-products/onsite/how-can-you-white-label-your-community-under-your-own-domain)
- [Point your domain to your hosted social experience](https://begenuin.com/developers/browse-by-products/onsite/can-you-host-your-brand-community-on-your-own-domain-with-a-white-label-setup)
- [Generate an embed code in your Business Account](https://begenuin.com/developers/browse-by-products/onsite/how-do-brands-generate-embed-codes-from-business-account)