On the Public view of the "My Site" page you will notice there is a web part called "My Recent Documents". This is generally liked however people often have concerns about the following:
1. It "can" display sensitive documents to end users
2. It is difficult to remove
So, just though it would be useful to:
1. Explain why it does not pose any security risk at all:
a) What is "My Recent Documents", well it is just a web part, just it is a static web part, which means it is embedded into the SPS template making it impossible to remove via the browser.
b) The results displayed in the web part come from SharePoint Search indexes, it just does a query that looks for any results where Author = name of the user whose profile is being displayed (in reality it may be a little more fancy than that.
c) Because it uses SharePoint search the results are security trimmed so that a user will only see those documents they have access to, this should eliminate any security issues you may have.
2. Document how this web part can be removed:
It can be removed by doing the following:
a) Open the following file: <drive>:\Program Files\Common Files\Microsoft Shared\web server extensions\60\TEMPLATE\<Locale ID>\SPSMSITE\Public.aspx
b) Search for a delete the following string: <SPSWC:ProfileDocuments FrameType="TitleBarOnly" id="ProfileDocuments" runat="server" />
c) Save, and refresh the public page.