The "little arrow thingie" works exactly as intended. If you hover your cursor over the arrow you will see - Go To Last Read Reply".
So if any post you are clicking the "little arrow thingie" on is the first time you have read any comments then you will be directed to the first page of the thread.
Yes, the little arrow thingie works for logged-in users just like you describe: it goes to the last reply you read. The thing is for not-logged-in users it goes to the first page which is technically correct (since a guest doesn't "mark things read") but it is also utterly useless. Why would I click on that link as a guest since it will always take me to the first page of the thread. What would be useful for guests is for the forum to take you to the last post of the thread instead.
Therefore, I will still call this a "bug" (even though it is technically not a bug) because pretty much every other forum on the Internet works like I described (for guests the link takes you to the last post).