Though the robot.txt cannot be edited per site (See 2820583 - Is it possible to Prevent Indexing of the Career Site in Recruiting - Recruiting Marketing), it is possible to refine search-engine optimization settings in Career Site Builder.
Image/data in this KBA is from SAP internal systems, sample data, or demo systems. Any resemblance to real data is purely coincidental.
SAP SuccessFactors Recruiting Marketing
Please check the implementation guide page : Using Search Engine Optimization Settings
SEO, HTML, Pages, Google, Index, RMK, Jobs, Search Engine , KBA , LOD-SF-RMK-CSB , Career Site Builder , Product Enhancement